Re: [Qemu-devel] Logging dirty pages from vhost-net in-kernel with vIOMMU

[Jason Wang <jasowang@redhat.com>]

On 2018/12/6 下午8:44, Jason Wang wrote:
>
> On 2018/12/6 下午8:11, Jintack Lim wrote:
>> On Thu, Dec 6, 2018 at 2:33 AM Jason Wang <jasowang@redhat.com> wrote:
>>>
>>> On 2018/12/5 下午10:47, Jintack Lim wrote:
>>>> On Tue, Dec 4, 2018 at 8:30 PM Jason Wang <jasowang@redhat.com> wrote:
>>>>> On 2018/12/5 上午2:37, Jintack Lim wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm wondering how the current implementation works when logging 
>>>>>> dirty
>>>>>> pages during migration from vhost-net (in kernel) when used vIOMMU.
>>>>>>
>>>>>> I understand how vhost-net logs GPAs when not using vIOMMU. But when
>>>>>> we use vhost with vIOMMU, then shouldn't vhost-net need to log the
>>>>>> translated address (GPA) instead of the address written in the
>>>>>> descriptor (IOVA) ? The current implementation looks like vhost-net
>>>>>> just logs IOVA without translation in vhost_get_vq_desc() in
>>>>>> drivers/vhost/net.c. It seems like QEMU doesn't do any further
>>>>>> translation of the dirty log when syncing.
>>>>>>
>>>>>> I might be missing something. Could somebody shed some light on 
>>>>>> this?
>>>>> Good catch. It looks like a bug to me. Want to post a patch for this?
>>>> Thanks for the confirmation.
>>>>
>>>> What would be a good setup to catch this kind of migration bug? I
>>>> tried to observe it in the VM expecting to see network applications
>>>> not getting data correctly on the destination, but it was not
>>>> successful (i.e. the VM on the destination just worked fine.) I didn't
>>>> even see anything going wrong when I disabled the vhost logging
>>>> completely without using vIOMMU.
>>>>
>>>> What I did is I ran multiple network benchmarks (e.g. netperf tcp
>>>> stream and my own one to check correctness of received data) in a VM
>>>> without vhost dirty page logging, and the benchmarks just ran fine in
>>>> the destination. I checked the used ring at the time the VM is stopped
>>>> in the source for migration, and it had multiple descriptors that is
>>>> (probably) not processed in the VM yet. Do you have any insight how it
>>>> could just work and what would be a good setup to catch this?
>>>
>>> According to past experience, it could be reproduced by doing scp from
>>> host to guest during migration.
>>>
>> Thanks. I actually tried that, but didn't see any problem either - I
>> copied a large file during migration from host to guest (the copy
>> continued on the destination), and checked md5 hashes using md5sum,
>> but the copied file had the same checksum as the one in the host.
>>
>> Do you recall what kind of symptom you observed when the dirty pages
>> were not migrated correctly with scp?
>
>
> Yes,  the point is to make the migration converge before the end of 
> scp (e.g set migration speed to a very big value). If scp end before 
> migration, we won't catch the bug. And it's better to do several 
> rounds of migration during scp.
>
> Anyway, let me try to reproduce it tomorrow.
>

Looks like I can reproduce this, scp give the following error to me:

scp /home/file root@192.168.100.4:/home
file                                           63% 1301MB 58.1MB/s   
00:12 ETAReceived disconnect from 192.168.100.4: 2: Packet corrupt
lost connection

FYI, I use the following cli:

numactl --cpunodebind 0 --membind 0 $qemu_path $img_path \
            -netdev tap,id=hn0,vhost=on \
            -device ioh3420,id=root.1,chassis=1 \
            -device 
virtio-net-pci,bus=root.1,netdev=hn0,ats=on,disable-legacy=on,disable-modern=off,iommu_platform=on 
\
            -device intel-iommu,device-iotlb=on \
            -M q35 -m 4G -enable-kvm -cpu host -smp 2 $@

Thanks


> Thanks