probable vm exit latency issue?

probable vm exit latency issue?

[Joe Buehler]

I am debugging large latency issues in a realtime KVM instance, trying to make an existing VM (debugged on another type of box) run on an HP proliant now.

cyclictest came up with the following snippet:

kworker/-71      0.....11 13483223us : ata_sff_exec_command <-ata_bmdma_setup
kworker/-71      0.....11 13483228us : ata_sff_sync <-ata_sff_exec_command
kworker/-71      0.....11 13483231us : delay_tsc <-__const_udelay
kworker/-71      0.....11 13483231us*: ata_bmdma_start <-ata_bmdma_qc_issue
kworker/-71      0d....11 13496181us : smp_apic_timer_interrupt <-apic_timer_interrupt
kworker/-71      0d....11 13496181us : irq_enter <-smp_apic_timer_interrupt
kworker/-71      0d....11 13496182us : rcu_irq_enter <-irq_enter

I don't have a lot of experience with this; is this a DMA start for disk I/O, presumably causing an exit from the VM?  For almost 13 milliseconds????

I haven't been able to find much obvious about eliminating this -- any suggestions?  I set caching to writeback without effect.

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/boot.qcow2'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/root.qcow2'/>
        <backingStore/>
      </backingStore>
      <target dev='hdb' bus='ide'/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>

Joe Buehler

Re: probable vm exit latency issue?

[Konrad Rzeszutek Wilk]

On Fri, Dec 15, 2017 at 02:36:56PM -0500, Joe Buehler wrote:
> I am debugging large latency issues in a realtime KVM instance, trying to make an existing VM (debugged on another type of box) run on an HP proliant now.
> 
> cyclictest came up with the following snippet:
> 
> kworker/-71      0.....11 13483223us : ata_sff_exec_command <-ata_bmdma_setup
> kworker/-71      0.....11 13483228us : ata_sff_sync <-ata_sff_exec_command
> kworker/-71      0.....11 13483231us : delay_tsc <-__const_udelay
> kworker/-71      0.....11 13483231us*: ata_bmdma_start <-ata_bmdma_qc_issue
> kworker/-71      0d....11 13496181us : smp_apic_timer_interrupt <-apic_timer_interrupt
> kworker/-71      0d....11 13496181us : irq_enter <-smp_apic_timer_interrupt
> kworker/-71      0d....11 13496182us : rcu_irq_enter <-irq_enter
> 
> I don't have a lot of experience with this; is this a DMA start for disk I/O, presumably causing an exit from the VM?  For almost 13 milliseconds????
> 
> I haven't been able to find much obvious about eliminating this -- any suggestions?  I set caching to writeback without effect.
> 
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='qcow2' cache='writeback'/>
>       <source file='/boot.qcow2'/>
>       <backingStore type='file' index='1'>
>         <format type='raw'/>
>         <source file='/root.qcow2'/>
>         <backingStore/>
>       </backingStore>
>       <target dev='hdb' bus='ide'/>
>       <alias name='ide0-0-1'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='1'/>
>     </disk>

Um. Don't use IDE? Use VirtIO.

> 
> Joe Buehler

Re: probable vm exit latency issue?

[Joe Buehler]

Konrad Rzeszutek Wilk wrote:

> Um. Don't use IDE? Use VirtIO.

I will revert to virtio.  It was virtio originally, I changed it to ide since that is what is used on the box the works.

Perhaps the difference between boxes is not the driver but the fact that the working box has an SSD disk and the new box has rotating disks.

To eliminate disk I/O entirely I am trying Ubuntu's "overlayroot".  However, I am still getting enormous latency from ata_bmdma_start calls, just nowhere near as often.

Joe Buehler

Re: probable vm exit latency issue?

[Joe Buehler]

I switched back to virtio for the disk; it has the same issue.  It apparently exits the VM via vp_notify(); in the trace I am looking at for almost 8 milliseconds.

I had been thinking in terms of disk writes but of course this could be any disk activity so overlayroot will not solve this.  It would seem that I might be looking at an inevitable disk rotation latency, which no host tuning will fix.

Would a second CPU in the VM solve this or does virtio halt the whole VM for disk I/O?

Or perhaps what I really need is to quiesce everything else in the VM to avoid all disk I/O?  I suppose I could use a RAM root filesystem -- ugh.

I find it a little puzzling that I have not seen disk I/O mentioned much in the realtime KVM literature I have found on the web.

Joe Buehler