Re: [Qemu-devel] [PATCH 10/12] virtio-net: multiqueue support

[Wanlong Gao <gaowanlong@cn.fujitsu.com>]

On 01/09/2013 05:30 PM, Jason Wang wrote:
> On 01/09/2013 04:23 PM, Wanlong Gao wrote:
>> On 01/08/2013 06:14 PM, Jason Wang wrote:
>>> On 01/08/2013 06:00 PM, Wanlong Gao wrote:
>>>> On 01/08/2013 05:51 PM, Jason Wang wrote:
>>>>> On 01/08/2013 05:49 PM, Wanlong Gao wrote:
>>>>>> On 01/08/2013 05:29 PM, Jason Wang wrote:
>>>>>>> On 01/08/2013 05:07 PM, Wanlong Gao wrote:
>>>>>>>> On 12/28/2012 06:32 PM, Jason Wang wrote:
>>>>>>>>> +    } else if (nc->peer->info->type !=  NET_CLIENT_OPTIONS_KIND_TAP) {
>>>>>>>>> +        ret = -1;
>>>>>>>>> +    } else {
>>>>>>>>> +        ret = tap_detach(nc->peer);
>>>>>>>>> +    }
>>>>>>>>> +
>>>>>>>>> +    return ret;
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>> +static void virtio_net_set_queues(VirtIONet *n)
>>>>>>>>> +{
>>>>>>>>> +    int i;
>>>>>>>>> +
>>>>>>>>> +    for (i = 0; i < n->max_queues; i++) {
>>>>>>>>> +        if (i < n->curr_queues) {
>>>>>>>>> +            assert(!peer_attach(n, i));
>>>>>>>>> +        } else {
>>>>>>>>> +            assert(!peer_detach(n, i));
>>>>>>>> I got a assert here,
>>>>>>>> qemu-system-x86_64: /work/git/qemu/hw/virtio-net.c:330: virtio_net_set_queues: Assertion `!peer_detach(n, i)' failed.
>>>>>>>>
>>>>>>>> Any thoughts?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Wanlong Gao
>>>>>>> Thanks for the testing, which steps or cases did you met this assertion,
>>>>>>> migration, reboot or just changing the number of virtqueues?
>>>>>> I use the 3.8-rc2 to test it again, I saw this tag has the multi-tap support.
>>>>>>
>>>>>> I just can't start the QEMU use  -netdev tap,id=hostnet0,queues=2,fd=%d,fd=%d -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:ce:7b:29,bus=pci.0,addr=0x3
>>>>>>
>>>>>> I pre-opened two tap fds, did I missing something?
>>>>> Nothing missed :) It should work.
>>>>>
>>>>> Could you please try not use fd=X and let qemu to create the file
>>>>> descriptors by itself? Btw, how did you create the two tap fds?
>>>> Can it create descriptors itself? I get 
>>>> qemu-system-x86_64: -netdev tap,id=hostnet0,queues=2: Device 'tap' could not be initialized
>>> You need prepare an ifup script which default at /etc/qemu-ifup (like
>>> following). Or you may try to add a script=no after:
>>>
>>> #!/bin/sh
>>>
>>> switch=kvmbr0
>>>
>>> /sbin/ifconfig $1 0.0.0.0 up
>>> /usr/sbin/brctl addif $switch $1
>>> /usr/sbin/brctl stp $switch off
>>>
>>> This will let qemu create a tap fd itself and make it to be connected to
>>> a port of the bridge caled kvmbr0.
>> But how to support multi-queue in this way?
> 
> Qemu will create the necessary multiqueue tap by itself, see patch 0/12.
>> I got guest kernel panic when using this way and set queues=4.
> 
> Does it happens w/o or w/ a fd parameter? What's the qemu command line?
> Did you meet it during boot time?

The QEMU command line is 

/work/git/qemu/x86_64-softmmu/qemu-system-x86_64 -name f17 -M pc-0.15 -enable-kvm -m 3096 \
-smp 4,sockets=4,cores=1,threads=1 \
-uuid c31a9f3e-4161-c53a-339c-5dc36d0497cb -no-user-config -nodefaults \
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/f17.monitor,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc -no-shutdown \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0xb,num_queues=4,hotplug=on \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 \
-drive file=/vm/f17.img,if=none,id=drive-virtio-disk0,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-drive file=/vm2/f17-kernel.img,if=none,id=drive-virtio-disk1,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1 \
-drive file=/vm/virtio-scsi/scsi3.img,if=none,id=drive-scsi0-0-2-0,format=raw \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi0-0-2-0,id=scsi0-0-2-0,removable=on \
-drive file=/vm/virtio-scsi/scsi4.img,if=none,id=drive-scsi0-0-3-0,format=raw \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=3,drive=drive-scsi0-0-3-0,id=scsi0-0-3-0 \
-drive file=/vm/virtio-scsi/scsi1.img,if=none,id=drive-scsi0-0-0-0,format=raw \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \
-drive file=/vm/virtio-scsi/scsi2.img,if=none,id=drive-scsi0-0-1-0,format=raw \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0 \
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 \
-chardev file,id=charserial1,path=/vm/f17.log \
-device isa-serial,chardev=charserial1,id=serial1 \
-device usb-tablet,id=input0 -vga std \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 \
-netdev tap,id=hostnet0,vhost=on,queues=4 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:ce:7b:29,bus=pci.0,addr=0x3 \
-monitor stdio

I got panic just after booting the system, did nothing,  waited for a while, the guest panicked.

[   28.053004] BUG: soft lockup - CPU#1 stuck for 23s! [ip:592]
[   28.053004] Modules linked in: ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables uinput joydev microcode virtio_balloon pcspkr virtio_net i2c_piix4 i2c_core virtio_scsi virtio_blk floppy
[   28.053004] CPU 1 
[   28.053004] Pid: 592, comm: ip Not tainted 3.8.0-rc1-net+ #3 Bochs Bochs
[   28.053004] RIP: 0010:[<ffffffff8137a9ab>]  [<ffffffff8137a9ab>] virtqueue_get_buf+0xb/0x120
[   28.053004] RSP: 0018:ffff8800bc913550  EFLAGS: 00000246
[   28.053004] RAX: 0000000000000000 RBX: ffff8800bc49c000 RCX: ffff8800bc49e000
[   28.053004] RDX: 0000000000000000 RSI: ffff8800bc913584 RDI: ffff8800bcfd4000
[   28.053004] RBP: ffff8800bc913558 R08: ffff8800bcfd0800 R09: 0000000000000000
[   28.053004] R10: ffff8800bc49c000 R11: ffff880036cc4de0 R12: ffff8800bcfd4000
[   28.053004] R13: ffff8800bc913558 R14: ffffffff8137ad73 R15: 00000000000200d0
[   28.053004] FS:  00007fb27a589740(0000) GS:ffff8800c1480000(0000) knlGS:0000000000000000
[   28.053004] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   28.053004] CR2: 0000000000640530 CR3: 00000000baeff000 CR4: 00000000000006e0
[   28.053004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   28.053004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   28.053004] Process ip (pid: 592, threadinfo ffff8800bc912000, task ffff880036da2e20)
[   28.053004] Stack:
[   28.053004]  ffff8800bcfd0800 ffff8800bc913638 ffffffffa003e9bb ffff8800bc913656
[   28.053004]  0000000100000002 ffff8800c17ebb08 000000500000ff10 ffffea0002f244c0
[   28.053004]  0000000200000582 0000000000000000 0000000000000000 ffffea0002f244c0
[   28.053004] Call Trace:
[   28.053004]  [<ffffffffa003e9bb>] virtnet_send_command.constprop.26+0x24b/0x270 [virtio_net]
[   28.053004]  [<ffffffff812ed963>] ? sg_init_table+0x23/0x50
[   28.053004]  [<ffffffffa0040629>] virtnet_set_rx_mode+0x99/0x300 [virtio_net]
[   28.053004]  [<ffffffff8152306f>] __dev_set_rx_mode+0x5f/0xb0
[   28.053004]  [<ffffffff815230ef>] dev_set_rx_mode+0x2f/0x50
[   28.053004]  [<ffffffff815231b7>] __dev_open+0xa7/0xf0
[   28.053004]  [<ffffffff81523461>] __dev_change_flags+0xa1/0x180
[   28.053004]  [<ffffffff815235f8>] dev_change_flags+0x28/0x70
[   28.053004]  [<ffffffff8152ff20>] do_setlink+0x3b0/0xa50
[   28.053004]  [<ffffffff812fb6b1>] ? nla_parse+0x31/0xe0
[   28.053004]  [<ffffffff815325de>] rtnl_newlink+0x36e/0x580
[   28.053004]  [<ffffffff811355cc>] ? get_page_from_freelist+0x37c/0x730
[   28.053004]  [<ffffffff81531e13>] rtnetlink_rcv_msg+0x113/0x2f0
[   28.053004]  [<ffffffff8117d973>] ? __kmalloc_node_track_caller+0x63/0x1c0
[   28.053004]  [<ffffffff8151526b>] ? __alloc_skb+0x8b/0x2a0
[   28.053004]  [<ffffffff81531d00>] ? __rtnl_unlock+0x20/0x20
[   28.053004]  [<ffffffff8154b571>] netlink_rcv_skb+0xb1/0xc0
[   28.053004]  [<ffffffff8152ea05>] rtnetlink_rcv+0x25/0x40
[   28.053004]  [<ffffffff8154ae91>] netlink_unicast+0x1a1/0x220
[   28.053004]  [<ffffffff8154b211>] netlink_sendmsg+0x301/0x3c0
[   28.053004]  [<ffffffff81508530>] sock_sendmsg+0xb0/0xe0
[   28.053004]  [<ffffffff8113a45b>] ? lru_cache_add_lru+0x3b/0x60
[   28.053004]  [<ffffffff811608b7>] ? page_add_new_anon_rmap+0xc7/0x180
[   28.053004]  [<ffffffff81509efc>] __sys_sendmsg+0x3ac/0x3c0
[   28.053004]  [<ffffffff8162e47c>] ? __do_page_fault+0x23c/0x4d0
[   28.053004]  [<ffffffff8115c9ef>] ? do_brk+0x1ff/0x370
[   28.053004]  [<ffffffff8150bec9>] sys_sendmsg+0x49/0x90
[   28.053004]  [<ffffffff81632d59>] system_call_fastpath+0x16/0x1b
[   28.053004] Code: 04 0f ae f0 48 8b 47 50 5d 0f b7 50 02 66 39 57 64 0f 94 c0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 <53> 80 7f 59 00 48 89 fb 0f 85 90 00 00 00 48 8b 47 50 0f b7 50 


The QEMU tree I used is git://github.com/jasowang/qemu.git

Thanks,
Wanlong Gao

> 
> Thanks
>>
>> Thanks,
>> Wanlong Gao
>>
>>>> I create the tap fd like this, and dup create the second fd, third fd, right?
>>> The second and third fd should be created with TUNSETIFF with the same
>>> tap_name also. Btw, you need to specify a IFF_MULTI_QUEUE flag to tell
>>> the kernel you want to create a multiqueue tap device, otherwise the
>>> second and third calling of TUNSETIFF will fail.
>>>
>>> Thanks
>>>> 	int tap_fd = open("/dev/net/tun", O_RDWR);
>>>> 	int vhost_fd = open("/dev/vhost-net", O_RDWR);
>>>> 	char *tap_name = "tap";
>>>> 	char cmd[2048];
>>>> 	char brctl[256];
>>>> 	char netup[256];
>>>> 	struct ifreq ifr;
>>>> 	if (tap_fd < 0) {
>>>> 		printf("open tun device failed\n");
>>>> 		return -1;
>>>> 	}
>>>> 	if (vhost_fd < 0) {
>>>> 		printf("open vhost-net device failed\n");
>>>> 		return -1;
>>>> 	}
>>>> 	memset(&ifr, 0, sizeof(ifr));
>>>> 	memcpy(ifr.ifr_name, tap_name, sizeof(tap_name));
>>>> 	ifr.ifr_flags = IFF_TAP | IFF_NO_PI;
>>>>
>>>> 	/*
>>>> 	 * setup tap net device
>>>> 	 */
>>>> 	if (ioctl(tap_fd, TUNSETIFF, &ifr) < 0) {
>>>> 		printf("setup tap net device failed\n");
>>>> 		return -1;
>>>> 	}
>>>>
>>>> 	sprintf(brctl, "brctl addif virbr0 %s", tap_name);
>>>> 	sprintf(netup, "ifconfig %s up", tap_name);
>>>> 	system(brctl);
>>>> 	system(netup);
>>>>
>>>> Thanks,
>>>> Wanlong Gao
>>>>
>>>>
>>>>> Thanks
>>>>>> Thanks,
>>>>>> Wanlong Gao
>>>>>>
>>>>>>>>> +        }
>>>>>>>>> +    }
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>> +static void virtio_net_set_multiqueue(VirtIONet *n, int multiqueue, int ctrl);
>>>>>>>>> +
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>