[Qemu-devel] [Question] Seabios HANGs when mmio wirte in a nested vmware host wtih virito-scsi controller

[Qemu-devel] [Question] Seabios HANGs when mmio wirte in a nested vmware host wtih virito-scsi controller

[Xulei (Stone)]

Hello all,
Recently, I met a werid question when i run a VM in the following platfrom:

Vmware Vsphere 6.0/6.5
   |-- centos 7.3 nested VM (with qemu 2.8, kmod 4.4.11, seabios 1.10)
        |-- VM (with virtio-scsi controller, modern mode)

VM MUST hang in seabios when try to mmio write during virtio-scsi init.
I  deeply analyze the code, and i found that:

Seabios calltrace:
scsi_drive_setup
   |-- cdb_get_inquiry
        |-- virtio_scsi_process_op
             |-- vring_kick
                  |-- writew  (hang here, mmio write, never return...)

Kmod calltrace:
handle_EPT_MISCONFIG
    |--kvm_io_bus_write
        |-- kvm_iodevice_write (fast mmio through ioeventfd to trigger qemu)
             |-- vcpu_run
                  |-- vcpu_enter_guest

Qemu calltrace:
os_host_main_loop_wait
      |--glib_pollfds_poll
          |-- aio_dispatch
               |-- vritio_scsi_handle_cmd_vq
                   |--virtio_scsi_pop_req
                       |-- virtqueue_pop
                            |--virtio_queue_empty

Then, kmod falls in infinite loop in handle EPT_MISCONFIG.
As far as i know, when kvm enters guest after handling EPT_MISCONFIG, seabios should return
from mmio write and wait for virtio backend(qemu) to handle this mmio writing.
What puzzles me:
1) i can not understand why kvm runs in infinite loop and seabios does not return from writew.
2) kvm nested kvm is ok. But vmware nested kvm is not ok. This problem has anything related
    with vmware?

Looking forward for directions...

Attached Kmod ftrace:
       CPU 0/KVM-9435  [008] .... 241968.479572: kvm_exit: reason EPT_MISCONFIG rip 0xee696 info 0 0
       CPU 0/KVM-9435  [008] .... 241968.479573: kvm_fast_mmio: fast mmio at gpa 0xfe003008
       CPU 0/KVM-9435  [008] d... 241968.479573: kvm_entry: vcpu 0
       CPU 0/KVM-9435  [008] .... 241968.479583: kvm_exit: reason EPT_MISCONFIG rip 0xee696 info 0 0
       CPU 0/KVM-9435  [008] .... 241968.479589: kvm_fast_mmio: fast mmio at gpa 0xfe003008
       CPU 0/KVM-9435  [008] d... 241968.479589: kvm_entry: vcpu 0
       CPU 0/KVM-9435  [008] .... 241968.479599: kvm_exit: reason EPT_MISCONFIG rip 0xee696 info 0 0
       CPU 0/KVM-9435  [008] .... 241968.479600: kvm_fast_mmio: fast mmio at gpa 0xfe003008
       CPU 0/KVM-9435  [008] d... 241968.479601: kvm_entry: vcpu 0
       CPU 0/KVM-9435  [008] .... 241968.479610: kvm_exit: reason EPT_MISCONFIG rip 0xee696 info 0 0
       CPU 0/KVM-9435  [008] .... 241968.479616: kvm_fast_mmio: fast mmio at gpa 0xfe003008
       CPU 0/KVM-9435  [008] d... 241968.479617: kvm_entry: vcpu 0
       CPU 0/KVM-9435  [008] .... 241968.479627: kvm_exit: reason EPT_MISCONFIG rip 0xee696 info 0 0
       CPU 0/KVM-9435  [008] .... 241968.479628: kvm_fast_mmio: fast mmio at gpa 0xfe003008
       CPU 0/KVM-9435  [008] d... 241968.479628: kvm_entry: vcpu 0
       CPU 0/KVM-9435  [008] .... 241968.479638: kvm_exit: reason EPT_MISCONFIG rip 0xee696 info 0 0
       CPU 0/KVM-9435  [008] .... 241968.479644: kvm_fast_mmio: fast mmio at gpa 0xfe003008
       CPU 0/KVM-9435  [008] d... 241968.479644: kvm_entry: vcpu 0
       CPU 0/KVM-9435  [008] .... 241968.479655: kvm_exit: reason EPT_MISCONFIG rip 0xee696 info 0 0
       CPU 0/KVM-9435  [008] .... 241968.479656: kvm_fast_mmio: fast mmio at gpa 0xfe003008
       CPU 0/KVM-9435  [008] d... 241968.479656: kvm_entry: vcpu 0

________________________________

Re: [Qemu-devel] [Question] Seabios HANGs when mmio wirte in a nested vmware host wtih virito-scsi controller

[Xiao Guangrong]

On 07/17/2017 11:13 AM, Xulei (Stone) wrote:

>                              |--virtio_queue_empty
> 
> Then, kmod falls in infinite loop in handle EPT_MISCONFIG.
> As far as i know, when kvm enters guest after handling EPT_MISCONFIG, seabios should return
> from mmio write and wait for virtio backend(qemu) to handle this mmio writing.
> What puzzles me:
> 1) i can not understand why kvm runs in infinite loop and seabios does not return from writew.

 From the trace in L1 guest, it did not hang in KVM, as it went into guest mode normally.

> 2) kvm nested kvm is ok. But vmware nested kvm is not ok. This problem has anything related
>      with vmware?
> 

It is the issue in vmware, i guess the RIP register is not handled correctly.
BTW, does kvm unit tests run well on nested vmware?

Re: [Qemu-devel] [Question] Seabios HANGs when mmio wirte in a nested vmware host wtih virito-scsi controller

[Xulei (Stone)]

On 07/17/2017 11:13 AM, Xulei (Stone) wrote:

>>                              |--virtio_queue_empty
>>
>> Then, kmod falls in infinite loop in handle EPT_MISCONFIG.
>> As far as i know, when kvm enters guest after handling EPT_MISCONFIG, seabios should return
>> from mmio write and wait for virtio backend(qemu) to handle this mmio writing.
>> What puzzles me:
>> 1) i can not understand why kvm runs in infinite loop and seabios does not return from writew.

  >From the trace in L1 guest, it did not hang in KVM, as it went into guest mode normally.

>> 2) kvm nested kvm is ok. But vmware nested kvm is not ok. This problem has anything related
>>     with vmware?
>>

> It is the issue in vmware, i guess the RIP register is not handled correctly.
> BTW, does kvm unit tests run well on nested vmware?


Sadly, kvm unit tests run fail on nested vmware and make the  CentOS 7.3(with kmod 4.4.11) crashed...

Re: [Qemu-devel] [Question] Seabios HANGs when mmio wirte in a nested vmware host wtih virito-scsi controller

[Xiao Guangrong]

On 07/17/2017 03:34 PM, Xulei (Stone) wrote:
> 
>     On 07/17/2017 11:13 AM, Xulei (Stone) wrote:
> 
>      >>                              |--virtio_queue_empty
>      >>
>      >> Then, kmod falls in infinite loop in handle EPT_MISCONFIG.
>      >> As far as i know, when kvm enters guest after handling EPT_MISCONFIG, seabios should return
>      >> from mmio write and wait for virtio backend(qemu) to handle this mmio writing.
>      >> What puzzles me:
>      >> 1) i can not understand why kvm runs in infinite loop and seabios does not return from writew.
> 
>        >From the trace in L1 guest, it did not hang in KVM, as it went into guest mode normally.
> 
>      >> 2) kvm nested kvm is ok. But vmware nested kvm is not ok. This problem has anything related
>      >>     with vmware?
>      >>
> 
>      > It is the issue in vmware, i guess the RIP register is not handled correctly.
>      > BTW, does kvm unit tests run well on nested vmware?
> 
>     Sadly, kvm unit tests run fail on nested vmware and make the  CentOS 7.3(with kmod 4.4.11) crashed...
> 

Well, so that should be reported to vmware. :-)