Re: [Qemu-devel] vhost-net issue: does not survive reboot on ppc64

[Alexey Kardashevskiy <aik@ozlabs.ru>]

On 12/25/2013 02:43 AM, Michael S. Tsirkin wrote:
> On Wed, Dec 25, 2013 at 01:15:29AM +1100, Alexey Kardashevskiy wrote:
>> On 12/24/2013 08:40 PM, Michael S. Tsirkin wrote:
>>> On Tue, Dec 24, 2013 at 02:09:07PM +1100, Alexey Kardashevskiy wrote:
>>>> On 12/24/2013 03:24 AM, Michael S. Tsirkin wrote:
>>>>> On Mon, Dec 23, 2013 at 02:01:13AM +1100, Alexey Kardashevskiy wrote:
>>>>>> On 12/23/2013 01:46 AM, Alexey Kardashevskiy wrote:
>>>>>>> On 12/22/2013 09:56 PM, Michael S. Tsirkin wrote:
>>>>>>>> On Sun, Dec 22, 2013 at 02:01:23AM +1100, Alexey Kardashevskiy wrote:
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> I am having a problem with virtio-net + vhost on POWER7 machine - it does
>>>>>>>>> not survive reboot of the guest.
>>>>>>>>>
>>>>>>>>> Steps to reproduce:
>>>>>>>>> 1. boot the guest
>>>>>>>>> 2. configure eth0 and do ping - everything works
>>>>>>>>> 3. reboot the guest (i.e. type "reboot")
>>>>>>>>> 4. when it is booted, eth0 can be configured but will not work at all.
>>>>>>>>>
>>>>>>>>> The test is:
>>>>>>>>> ifconfig eth0 172.20.1.2 up
>>>>>>>>> ping 172.20.1.23
>>>>>>>>>
>>>>>>>>> If to run tcpdump on the host's "tap-id3" interface, it shows no trafic
>>>>>>>>> coming from the guest. If to compare how it works before and after reboot,
>>>>>>>>> I can see the guest doing an ARP request for 172.20.1.23 and receives the
>>>>>>>>> response and it does the same after reboot but the answer does not come.
>>>>>>>>
>>>>>>>> So you see the arp packet in guest but not in host?
>>>>>>>
>>>>>>> Yes.
>>>>>>>
>>>>>>>
>>>>>>>> One thing to try is to boot debug kernel - where pr_debug is
>>>>>>>> enabled - then you might see some errors in the kernel log.
>>>>>>>
>>>>>>> Tried and added lot more debug printk myself, not clear at all what is
>>>>>>> happening there.
>>>>>>>
>>>>>>> One more hint - if I boot the guest and the guest does not bring eth0 up
>>>>>>> AND wait more than 200 seconds (and less than 210 seconds), then eth0 will
>>>>>>> not work at all. I.e. this script produces not-working-eth0:
>>>>>>>
>>>>>>>
>>>>>>> ifconfig eth0 172.20.1.2 down
>>>>>>> sleep 210
>>>>>>> ifconfig eth0 172.20.1.2 up
>>>>>>> ping 172.20.1.23
>>>>>>>
>>>>>>> s/210/200/ - and it starts working. No reboot is required to reproduce.
>>>>>>>
>>>>>>> No "vhost" == always works. The only difference I can see here is vhost's
>>>>>>> thread which may get suspended if not used for a while after the start and
>>>>>>> does not wake up but this is almost a blind guess.
>>>>>>
>>>>>>
>>>>>> Yet another clue - this host kernel patch seems to help with the guest
>>>>>> reboot but does not help with the initial 210 seconds delay:
>>>>>>
>>>>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>>>>> index 69068e0..5e67650 100644
>>>>>> --- a/drivers/vhost/vhost.c
>>>>>> +++ b/drivers/vhost/vhost.c
>>>>>> @@ -162,10 +162,10 @@ void vhost_work_queue(struct vhost_dev *dev, struct
>>>>>> vhost_work *work)
>>>>>>                 list_add_tail(&work->node, &dev->work_list);
>>>>>>                 work->queue_seq++;
>>>>>>                 spin_unlock_irqrestore(&dev->work_lock, flags);
>>>>>> -               wake_up_process(dev->worker);
>>>>>>         } else {
>>>>>>                 spin_unlock_irqrestore(&dev->work_lock, flags);
>>>>>>         }
>>>>>> +       wake_up_process(dev->worker);
>>>>>>  }
>>>>>>  EXPORT_SYMBOL_GPL(vhost_work_queue);
>>>>>>
>>>>>>
>>>>>
>>>>> Interesting. Some kind of race? A missing memory barrier somewhere?
>>>>
>>>> I do not see how. I boot the guest and just wait 210 seconds, nothing
>>>> happens to cause races.
>>>>
>>>>
>>>>> Since it's all around startup,
>>>>> you can try kicking the host eventfd in
>>>>> vhost_net_start.
>>>>
>>>>
>>>> How exactly? This did not help. Thanks.
>>>>
>>>> diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
>>>> index 006576d..407ecf2 100644
>>>> --- a/hw/net/vhost_net.c
>>>> +++ b/hw/net/vhost_net.c
>>>> @@ -229,6 +229,17 @@ int vhost_net_start(VirtIODevice *dev, NetClientState
>>>> *ncs,
>>>>          if (r < 0) {
>>>>              goto err;
>>>>          }
>>>> +
>>>> +        VHostNetState *vn = tap_get_vhost_net(ncs[i].peer);
>>>> +        struct vhost_vring_file file = {
>>>> +            .index = i
>>>> +        };
>>>> +        file.fd =
>>>> event_notifier_get_fd(virtio_queue_get_host_notifier(dev->vq));
>>>> +        r = ioctl(vn->dev.control, VHOST_SET_VRING_KICK, &file);
>>>
>>> No, this sets the notifier, it does not kick.
>>> To kick you write 1 there:
>>> 	uint6_t  v = 1;
>>> 	write(fd, &v, sizeof v);
>>
>>
>> Please, be precise. How/where do I get that @fd? Is what I do correct?
> 
> Yes.

Turns out that no. The control device in the host kernel does not implement
write() so it always fails.

This works:

uint64_t v = 1;
int fd = event_notifier_get_fd(&vq->host_notifier);
int r = write(fd, &v, sizeof v);

By "works" I mean it helps to wake the whole thing up and the guest's eth0
starts working after 3 minutes delay.



>> What
>> is uint6_t - uint8_t or uint16_t (neither works)?
> 
> Sorry, should have been uint64_t.
> 
>> May be it is a missing barrier - I rebooted machine several times and now
>> sometime after even 240 seconds (not 210 as before) it works (but most of
>> the time still does not)...
>>
>>
>>>> +        if (r) {
>>>> +            error_report("Error notifiyng host notifier: %d", -r);
>>>> +            goto err;
>>>> +        }
>>>>      }
>>>>
>>>>
>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>>>> If to remove vhost=on, it is all good. If to try Fedora19
>>>>>>>>> (v3.10-something), it all good again - works before and after reboot.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> And there 2 questions:
>>>>>>>>>
>>>>>>>>> 1. does anybody have any clue what might go wrong after reboot?
>>>>>>>>>
>>>>>>>>> 2. Is there any good material to read about what exactly and how vhost
>>>>>>>>> accelerates?
>>>>>>>>>
>>>>>>>>> My understanding is that packets from the guest to the real network are
>>>>>>>>> going as:
>>>>>>>>> 1. guest's virtio-pci-net does ioport(VIRTIO_PCI_QUEUE_NOTIFY)
>>>>>>>>> 2. QEMU's net/virtio-net.c calls qemu_net_queue_deliver()
>>>>>>>>> 3. QEMU's net/tap.c calls tap_write_packet() and this is how the host knows
>>>>>>>>> that there is a new packet.
>>>>>>>
>>>>>>>
>>>>>>> What about the documentation? :) or the idea?
>>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This how I run QEMU:
>>>>>>>>> ./qemu-system-ppc64 \
>>>>>>>>> -enable-kvm \
>>>>>>>>> -m 2048 \
>>>>>>>>> -machine pseries \
>>>>>>>>> -initrd 1.cpio \
>>>>>>>>> -kernel vml312_virtio_net_dbg \
>>>>>>>>> -nographic \
>>>>>>>>> -vga none \
>>>>>>>>> -netdev
>>>>>>>>> tap,id=id3,ifname=tap-id3,script=ifup.sh,downscript=ifdown.sh,vhost=on \
>>>>>>>>> -device virtio-net-pci,id=id4,netdev=id3,mac=C0:41:49:4b:00:00
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That is bridge config:
>>>>>>>>> [aik@dyn232 ~]$ brctl show
>>>>>>>>> bridge name	bridge id		STP enabled	interfaces
>>>>>>>>> brtest		8000.00145e992e88	no	pin	eth4
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The ifup.sh script:
>>>>>>>>> ifconfig $1 hw ether ee:01:02:03:04:05
>>>>>>>>> /sbin/ifconfig $1 up
>>>>>>>>> /usr/sbin/brctl addif brtest $1
>>
>>
>>
>> -- 
>> Alexey


-- 
Alexey