[PATCH] kvm: arm/arm64 : fix vm's hanging at startup time

[<peng.hao2@zte.com.cn>]

[-- Attachment #1.1: Type: text/plain, Size: 8572 bytes --]

>> >Hi,
>> >
>> >On Wed, Nov 21, 2018 at 04:56:54PM +0800, peng.hao2@zte.com.cn wrote:
>> >> >On 19/11/2018 09:10, Mark Rutland wrote:
>> >> >> On Sat, Nov 17, 2018 at 10:58:37AM +0800, peng.hao2@zte.com.cn wrote:
>> >> >>>> On 16/11/18 00:23, peng.hao2@zte.com.cn wrote:
>> >> >>>>>> Hi,
>> >> >>>>>>> When virtual machine starts, hang up.
>> >> >>>>>>
>> >> >>>>>> I take it you mean the *guest* hangs? Because it doesn't get a timer
>> >> >>>>>> interrupt?
>> >> >>>>>>
>> >> >>>>>>> The kernel version of guest
>> >> >>>>>>> is 4.16. Host support vgic_v3.
>> >> >>>>>>
>> >> >>>>>> Your host kernel is something recent, I guess?
>> >> >>>>>>
>> >> >>>>>>> It was mainly due to the incorrect vgic_irq's(intid=27) group value
>> >> >>>>>>> during injection interruption. when kvm_vgic_vcpu_init is called,
>> >> >>>>>>> dist is not initialized at this time. Unable to get vgic V3 or V2
>> >> >>>>>>> correctly, so group is not set.
>> >> >>>>>>
>> >> >>>>>> Mmh, that shouldn't happen with (v)GICv3. Do you use QEMU (which
>> >> >>>>>> version?) or some other userland tool?
>> >> >>>>>>
>> >> >>>>>
>> >> >>>>> QEMU emulator version 3.0.50 .
>> >> >>>>>
>> >> >>>>>>> group is setted to 1 when vgic_mmio_write_group is invoked at some
>> >> >>>>>>> time.
>> >> >>>>>>> when irq->group=0 (intid=27), No ICH_LR_GROUP flag was set and
>> >> >>>>>>> interrupt injection failed.
>> >> >>>>>>>
>> >> >>>>>>> Signed-off-by: Peng Hao <peng.hao2@zte.com.cn>
>> >> >>>>>>> ---
>> >> >>>>>>>   virt/kvm/arm/vgic/vgic-v3.c | 2 +-
>> >> >>>>>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>> >> >>>>>>>
>> >> >>>>>>> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
>> >> >>>>>>> index 9c0dd23..d101000 100644
>> >> >>>>>>> --- a/virt/kvm/arm/vgic/vgic-v3.c
>> >> >>>>>>> +++ b/virt/kvm/arm/vgic/vgic-v3.c
>> >> >>>>>>> @@ -198,7 +198,7 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu,
>> >> >>>>>>> struct vgic_irq *irq, int lr) if (vgic_irq_is_mapped_level(irq) &&
>> >> >>>>>>> (val & ICH_LR_PENDING_BIT)) irq->line_level = false;
>> >> >>>>>>>
>> >> >>>>>>> -    if (irq->group)
>> >> >>>>>>> +    if (model == KVM_DEV_TYPE_ARM_VGIC_V3)

[...]

>> >Could you try the following patch:
>> >
>> >diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
>> >index c0c0b88af1d5..6fa858c7a5a6 100644
>> >--- a/virt/kvm/arm/vgic/vgic-init.c
>> >+++ b/virt/kvm/arm/vgic/vgic-init.c
>> >@@ -231,13 +231,6 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
>> >irq->config = VGIC_CONFIG_LEVEL;
>> >}
>> >-        /*
>> >-         * GICv3 can only be created via the KVM_DEVICE_CREATE API and
>> >-         * so we always know the emulation type at this point as it's
>> >-         * either explicitly configured as GICv3, or explicitly
>> >-         * configured as GICv2, or not configured yet which also
>> >-         * implies GICv2.
>> >-         */
>> >if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3)
>> >irq->group = 1;
>> >else
>> >@@ -298,6 +291,16 @@ int vgic_init(struct kvm *kvm)
>> >if (ret)
>> >goto out;
>> >+    /* Initialize groups on CPUs created before the VGIC type was known */
>> >+    kvm_for_each_vcpu(i, vcpu, kvm) {
>> >+        struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> >+
>> >+        for (i = 0; i < VGIC_NR_PRIVATE_IRQS; i++) {
>> >+            struct vgic_irq *irq = &vgic_cpu->private_irqs[i];
>> >+            irq->group = 1;
>> >+        }
>> >+    }
>> >+
>> >if (vgic_has_its(kvm)) {
>> >ret = vgic_v4_init(kvm);
>> >if (ret)
>> >
>> Sorry for the late reply.
>> I test the patch and it works. But I think there is randomness here.
>> 1.vgic_init:  set intid/27 group=1
>> 2.vgic_mmio_write_group:## intid/27 group=0
>> 3.!!! receive a  vtimer irq   ----- it may exist.
>> 4.vgic_mmio_write_group:## intid/27 group=1
>>
>Are these writes from userspace or from the guest?
>(I assume they're from the guest.)
>
>>
>> I made an attempt before:
>>
>>Was this previous experiment using my patch or without my patch?
>>
>> 1. kvm_vgic_vcpu_init
>>      according to if  kvm_vgic_global_state.type == VGIC_V3, set group=1
>> 2.vgic_mmio_write_group:## intid/27 group=0
> >3.!!! receive a  vtimer irq   -----I really receive a vtimer irq, then the guest hangs.
>>
>
>If the guest you're using is flipping the group of the interrupt and yet
>enabling the device and interrupt while flipping the group, then it's
>not expected to work and it's a guest bug.
>
>My hunch here tells me that my patch is what we need for KVM, and either
>
>(1) You no longer have a problem but are tracing some guest
>initialization behavior which is fine, or
>
>(2) You still have a problem because you also have some broken guest
>code (guest UEFI or guest Linux) that also needs fixing.
>
>If you can demonstrate that you still have a problem, with my path
>applied to KVM, and with a recent guest Linux and guest UEFI, then I'll
>be happy to have another look.  In that case, could you please provide
>the exact versions of your guest Linux and UEFI, as well as the QEMU
>version and command line.
>
>
I'm sorry I make a mistake.I updated UEFI but never had time to test it. Yesterday I merged your patch
 to test, but forgot that I had updated UEFI.
Here are the corrected test results :
-----------------------------------------------------------------------------------------
|_new uefi + your patch_____________|______guest start normally__|
|_old uefi + your patch______________|______guest hangs________|
|_bypass uefi _(with/without your patch)|______guest start normally__|
|_new uefi + without your patch ______|______guest start normally_  |
|_old uefi + without your patch_ ______| _____ guest hangs________|

So the problem seems to be in UEFI. new uefi can be sure there is no a irq between
the group=0 and group=1 from  vgic_mmio_write_group.

guest kernel :4.16.0 (commit c18bb396d3d261ebbb4efbc05129c5d354c541e4)
the version of guest old uefi  corresponds to qemu 3.0.50.
the version of new uefi is the latest edk2.
I use libvirt to start guest, and the qemu command line  is like this:
/home/qemu-patch/linshi/qemu/aarch64-softmmu/qemu-system-aarch64 -name guest=centos74-ph-1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-centos74-ph-1/master-key.aes -machine virt-3.1,accel=kvm,usb=off,dump-guest-core=off,gic-version=3 -cpu host -drive file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/centos74-ph-1_VARS.fd,if=pflash,format=raw,unit=1 -m 4096 -realtime mlock=off -smp 4,maxcpus=8,sockets=8,cores=1,threads=1 -uuid ff5ecf4e-1234-425f-948d-8d91135e109c -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-centos74-ph-1/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 -device pcie-root-port,port=0x10,chassis=3,id=pci.3,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x1 -device usb-ehci,id=usb,bus=pci.2,addr=0x1 -device piix3-usb-uhci,id=usb1,bus=pci.2,addr=0x2 -device virtio-scsi-device,id=scsi0 -device virtio-serial-device,id=virtio-serial0 -drive file=/home/centos74-ph/boot.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive file=/home/centos74-ph/data.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-1 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1 -drive file=/home/CentOS-7-aarch64-Everything.iso,format=raw,if=none,id=drive-scsi0-0-0-2,readonly=on -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi0-0-0-2,id=scsi0-0-0-2,bootindex=2 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-device,netdev=hostnet0,id=net0,mac=aa:35:97:5b:33:ac -add-fd set=2,fd=30 -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on -serial chardev:charserial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/centos74.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -device usb-kbd,id=input1,bus=usb.0,port=2 -vnc 0.0.0.0:0 -k en-us -device virtio-gpu-pci,id=video0,max_outputs=1,bus=pci.3,addr=0x0 -device pvpanic-mmio -msg timestamp=on

Thanks.
>Thanks,
>
>Christoffer

[-- Attachment #2: Type: text/plain, Size: 151 bytes --]

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm