[Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Peter Maydell]

Hi; this is a bug report I'm forwarding for somebody else.

The problem is that QEMU's tun/tap support in net/tap-linux.c only
supports the legacy /dev/net/tun interface (the filename is hardcoded).
If you created the tap device via macvtap, then this is the wrong
filename, but there's no code in QEMU to get the correct /dev/tapN
filename from the user-provided ifname=whatever option.

Talking on IRC, the answer suggested was that we ought to
do a SIOCGIFINDEX ioctl passing in the user-specified ifname
string, which then gives you the index N to construct the
/dev/tapN filename. There is probably complexity in working out
whether we should do that or use the legacy interface (or try both
always?) This is probably overall not a very large amount of code,
though.

The usual suggested workaround is to use the -netdev fd option, like
  fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
(which gets the shell to open the right /dev/tap device).
Unfortunately this isn't compatible with multi-queue support
because netdev complains
"ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
and vhostfds= are invalid with fd="
so you can't pass options like "queues=4"...

Detailed repro case below if required (uses aarch64 but none
of this is arch-specific as far as I know).

------------<Begin Reproducer>--------------

To reproduce, here are the steps I followed.

Create a tap device on the host
===============================

    # ip link add link eth0 name tap0 type macvtap mode bridge

To create the tap device the host kernel needs CONFIG_MACVTAP enabled.

You can check that the device is created by running

    $ ip a

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state
UNKNOWN group default qlen 1
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
UP group default qlen 1000
        link/ether 02:a1:a2:a3:a4:a5 brd ff:ff:ff:ff:ff:ff
        inet 10.1.194.30/23 brd 10.1.195.255 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 fe80::a1:a2ff:fea3:a4a5/64 scope link
           valid_lft forever preferred_lft forever
    [...]
    8: tap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP group default qlen 500
        link/ether 5e:fc:d2:28:f2:f9 brd ff:ff:ff:ff:ff:ff
        inet6 fe80::5cfc:d2ff:fe28:f2f9/64 scope link
           valid_lft forever preferred_lft forever

Creating a tap device creates a character device. The character device
created is "/dev/tap" followed by the interface index (8 in our example above).

The interface index can also be discovered by reading a node in sysfs.

    $ cat /sys/class/net/tap0/ifindex
    8

Modify tap device permissions
=============================

By default, the device node is created with permissions root:root.

    $ ls -l /dev/tap8
    crw------- 1 root root 242, 1 Nov 30 11:57 /dev/tap8

Update the permissions so that qemu-system-aarch64 can access the node.

    # chgrp kvm /dev/tap8
    # chmod g+rw /dev/tap8

    $ ls -l /dev/tap8
    crw-rw---- 1 root kvm 242, 1 Nov 30 11:57 /dev/tap8

Run qemu
========
I am using a locally built qemu (tag: v2.8.0-rc2). When attempting to
start a guest I get the following complaint -

    $ qemu-system-aarch64 -M virt -m 4096 -cpu host -smp 4 -enable-kvm
-nographic \
         -pflash flash0.img -pflash flash1.img \
         -drive if=none,file=hda.qcow2,id=hd0" \
         -device virtio-blk-device,drive=hd0   \
         -netdev tap,id=eth0,script=,ifname=tap0,queues=4 \
         -device
virtio-net-pci,netdev=eth0,mq=on,vectors=12,mac=22:22:3a:57:07:ed

    qemu-system-aarch64: -netdev
tap,id=eth0,script=,ifname=tap0,queues=4: could not open /dev/net/tun:
No such file or directory

------------<End of Reproducer>--------------

thanks
-- PMM

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Daniel P. Berrange]

On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
> Hi; this is a bug report I'm forwarding for somebody else.
> 
> The problem is that QEMU's tun/tap support in net/tap-linux.c only
> supports the legacy /dev/net/tun interface (the filename is hardcoded).
> If you created the tap device via macvtap, then this is the wrong
> filename, but there's no code in QEMU to get the correct /dev/tapN
> filename from the user-provided ifname=whatever option.
> 
> Talking on IRC, the answer suggested was that we ought to
> do a SIOCGIFINDEX ioctl passing in the user-specified ifname
> string, which then gives you the index N to construct the
> /dev/tapN filename. There is probably complexity in working out
> whether we should do that or use the legacy interface (or try both
> always?) This is probably overall not a very large amount of code,
> though.
> 
> The usual suggested workaround is to use the -netdev fd option, like
>   fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
> (which gets the shell to open the right /dev/tap device).
> Unfortunately this isn't compatible with multi-queue support
> because netdev complains
> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
> and vhostfds= are invalid with fd="
> so you can't pass options like "queues=4"...

FWIW you should be able to instead do

   fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)

note 'fds' plural, instead of 'fd'

It would be nicer to make it "just work" though when giving a device
name for macvtap

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Peter Maydell]

On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
>> The usual suggested workaround is to use the -netdev fd option, like
>>   fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>> (which gets the shell to open the right /dev/tap device).
>> Unfortunately this isn't compatible with multi-queue support
>> because netdev complains
>> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
>> and vhostfds= are invalid with fd="
>> so you can't pass options like "queues=4"...
>
> FWIW you should be able to instead do
>
>    fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>
> note 'fds' plural, instead of 'fd'

What's the difference between that and the "usual suggested workaround"
I described above that doesn't work with queues=... ?

thanks
-- PMM

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Daniel P. Berrange]

On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
> > On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
> >> The usual suggested workaround is to use the -netdev fd option, like
> >>   fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
> >> (which gets the shell to open the right /dev/tap device).
> >> Unfortunately this isn't compatible with multi-queue support
> >> because netdev complains
> >> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
> >> and vhostfds= are invalid with fd="
> >> so you can't pass options like "queues=4"...
> >
> > FWIW you should be able to instead do
> >
> >    fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
> >
> > note 'fds' plural, instead of 'fd'
> 
> What's the difference between that and the "usual suggested workaround"
> I described above that doesn't work with queues=... ?

It just seems the 'queues' param always wants you to use 'fds' instead
of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
and 'fd' only takes a single FD.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Peter Maydell]

On 7 December 2016 at 14:07, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
>> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
>> > On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
>> >> The usual suggested workaround is to use the -netdev fd option, like
>> >>   fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>> >> (which gets the shell to open the right /dev/tap device).
>> >> Unfortunately this isn't compatible with multi-queue support
>> >> because netdev complains
>> >> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
>> >> and vhostfds= are invalid with fd="
>> >> so you can't pass options like "queues=4"...
>> >
>> > FWIW you should be able to instead do
>> >
>> >    fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>> >
>> > note 'fds' plural, instead of 'fd'
>>
>> What's the difference between that and the "usual suggested workaround"
>> I described above that doesn't work with queues=... ?
>
> It just seems the 'queues' param always wants you to use 'fds' instead
> of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
> and 'fd' only takes a single FD.

Oh, I see. That seems a bit obscure :-)

thanks
-- PMM

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Daniel P. Berrange]

On Wed, Dec 07, 2016 at 02:21:24PM +0000, Peter Maydell wrote:
> On 7 December 2016 at 14:07, Daniel P. Berrange <berrange@redhat.com> wrote:
> > On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
> >> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
> >> > On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
> >> >> The usual suggested workaround is to use the -netdev fd option, like
> >> >>   fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
> >> >> (which gets the shell to open the right /dev/tap device).
> >> >> Unfortunately this isn't compatible with multi-queue support
> >> >> because netdev complains
> >> >> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
> >> >> and vhostfds= are invalid with fd="
> >> >> so you can't pass options like "queues=4"...
> >> >
> >> > FWIW you should be able to instead do
> >> >
> >> >    fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
> >> >
> >> > note 'fds' plural, instead of 'fd'
> >>
> >> What's the difference between that and the "usual suggested workaround"
> >> I described above that doesn't work with queues=... ?
> >
> > It just seems the 'queues' param always wants you to use 'fds' instead
> > of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
> > and 'fd' only takes a single FD.
> 
> Oh, I see. That seems a bit obscure :-)

And pointless, because QemuOpts would have allowed use of 'fd' multiple
times instead of inventing a new arg.  fd=1,fd=3,fd=6 could have worked
fine with multi-queue :-(

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Peter Maydell]

On 7 December 2016 at 14:24, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Wed, Dec 07, 2016 at 02:21:24PM +0000, Peter Maydell wrote:
>> On 7 December 2016 at 14:07, Daniel P. Berrange <berrange@redhat.com> wrote:
>> > On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
>> >> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
>> >> > note 'fds' plural, instead of 'fd'
>> >>
>> >> What's the difference between that and the "usual suggested workaround"
>> >> I described above that doesn't work with queues=... ?
>> >
>> > It just seems the 'queues' param always wants you to use 'fds' instead
>> > of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
>> > and 'fd' only takes a single FD.
>>
>> Oh, I see. That seems a bit obscure :-)
>
> And pointless, because QemuOpts would have allowed use of 'fd' multiple
> times instead of inventing a new arg.  fd=1,fd=3,fd=6 could have worked
> fine with multi-queue :-(

I guess the best we could do now would be to make fd= and fds= synonyms
with both supporting either comma-separated or being specified
multiple times ?

It's particularly confusing in this case that fd= doesn't work with
queues=, because the user isn't trying to pass multiple fds, just
the one is fine.

thanks
-- PMM

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Jason Wang]

On 2016年12月07日 22:32, Peter Maydell wrote:
> On 7 December 2016 at 14:24, Daniel P. Berrange <berrange@redhat.com> wrote:
>> On Wed, Dec 07, 2016 at 02:21:24PM +0000, Peter Maydell wrote:
>>> On 7 December 2016 at 14:07, Daniel P. Berrange <berrange@redhat.com> wrote:
>>>> On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
>>>>> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
>>>>>> note 'fds' plural, instead of 'fd'
>>>>> What's the difference between that and the "usual suggested workaround"
>>>>> I described above that doesn't work with queues=... ?
>>>> It just seems the 'queues' param always wants you to use 'fds' instead
>>>> of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
>>>> and 'fd' only takes a single FD.
>>> Oh, I see. That seems a bit obscure :-)
>> And pointless, because QemuOpts would have allowed use of 'fd' multiple
>> times instead of inventing a new arg.  fd=1,fd=3,fd=6 could have worked
>> fine with multi-queue :-(
> I guess the best we could do now would be to make fd= and fds= synonyms
> with both supporting either comma-separated or being specified
> multiple times ?

When multiqueue were introduced, qemu does not support such kind of 
parameters. This maybe useful (or use fd[0],fd[1]), but I'm not sure 
this is really needed consider libvirt is in charge of doing such things 
now.

>
> It's particularly confusing in this case that fd= doesn't work with
> queues=, because the user isn't trying to pass multiple fds, just
> the one is fine.

Since fd can only accept on file descriptor, so it implies 1 queue. For 
"fds", specifying queues seems redundant, since qemu can count the #fds 
by itself.

Thanks

>
> thanks
> -- PMM
>

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Jason Wang]

On 2016年12月07日 22:24, Daniel P. Berrange wrote:
> On Wed, Dec 07, 2016 at 02:21:24PM +0000, Peter Maydell wrote:
>> On 7 December 2016 at 14:07, Daniel P. Berrange <berrange@redhat.com> wrote:
>>> On Wed, Dec 07, 2016 at 12:30:31PM +0000, Peter Maydell wrote:
>>>> On 7 December 2016 at 12:04, Daniel P. Berrange <berrange@redhat.com> wrote:
>>>>> On Wed, Dec 07, 2016 at 12:01:14PM +0000, Peter Maydell wrote:
>>>>>> The usual suggested workaround is to use the -netdev fd option, like
>>>>>>    fd=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>>>>>> (which gets the shell to open the right /dev/tap device).
>>>>>> Unfortunately this isn't compatible with multi-queue support
>>>>>> because netdev complains
>>>>>> "ifname=, script=, downscript=, vnet_hdr=, helper=, queues=, fds=,
>>>>>> and vhostfds= are invalid with fd="
>>>>>> so you can't pass options like "queues=4"...
>>>>> FWIW you should be able to instead do
>>>>>
>>>>>     fds=3 3<>/dev/tap$(< /sys/class/net/tap0/ifindex)
>>>>>
>>>>> note 'fds' plural, instead of 'fd'
>>>> What's the difference between that and the "usual suggested workaround"
>>>> I described above that doesn't work with queues=... ?
>>> It just seems the 'queues' param always wants you to use 'fds' instead
>>> of 'fd' - 'fds' takes a comma-separated list of FDs - one per queue
>>> and 'fd' only takes a single FD.
>> Oh, I see. That seems a bit obscure :-)
> And pointless, because QemuOpts would have allowed use of 'fd' multiple
> times instead of inventing a new arg.  fd=1,fd=3,fd=6 could have worked
> fine with multi-queue :-(
>
> Regards,
> Daniel

Yes, but not the time when multiqueue is introduced. I'm not sure how 
much value we can gain from this, consider libvirt has already use fds 
for years.

Thanks

Re: [Qemu-devel] qemu -netdev tun/tap support can't handle macvtap type devices

[Peter Maydell]

On 8 December 2016 at 01:52, Jason Wang <jasowang@redhat.com> wrote:
> On 2016年12月07日 22:32, Peter Maydell wrote:
>> It's particularly confusing in this case that fd= doesn't work with
>> queues=, because the user isn't trying to pass multiple fds, just
>> the one is fine.
>
>
> Since fd can only accept on file descriptor, so it implies 1 queue. For
> "fds", specifying queues seems redundant, since qemu can count the #fds by
> itself.

Opening /dev/net/tun like net/tap-linux.c does also only gives
you one file descriptor, but it still supports multiple queues,
unless I'm misunderstanding how this works (quite possible).

thanks
-- PMM