Re: Timer delays in VM

[Marc Zyngier <maz@kernel.org>]

On Tue, 01 Mar 2022 19:03:33 +0000,
Eugene Huang <eugeneh@nvidia.com> wrote:
> 
> > >       * Does this timer rely on kvm timer irq injection?
> > 
> > Yes. A timer interrupt is always injected in SW. But the timer interrupt can
> > either come from the HW timer itself (the VM was running while the timer
> > expired), or from a SW timer that KVM as setup if the guest was blocked on
> > WFI.
> 
> <EH> Here for arm64, EL1Virtual Timer is used. EL1 Virtual Timer is
> a HW timer, correct?  There is an armvtimer implementation in QEMU
> 6.1+. Does this armvtimer make a difference?

KVM only deals with the EL1 timers (both physical and virtual). I
guess that by 'armvtimer', you mean libvirt's front-end for the stolen
time feature to expose to the guest how wall clock and CPU time
diverge (i.e. it isn't a timer at all, but a dynamic correction for
it).

> > >       * What can be any possible causes for the timer delay? Are there
> > > some locking mechanisms which can cause the delay?
> > 
> > This completely depend on how loaded your host is, the respective priorities
> > of the various processes, and a million of other things.
> > This is no different from the same userspace running on the host.
> > It also depends on the *guest* kernel, by the way.
> 
> <EH> Our guest kernel is 5.4. How is the *guest* kernel involved?
> Can you give an example? Do you have suggestions on the guest kernel
> version as well.

It is the guest kernel that programs the timer, and KVM isn't involved
at all, specially on your HW (direct access to both timers on
VHE-capable systems).

> > >       * What parameters can tune this timer?
> > 
> > None. You may want to check whether the delay is observed when the VM
> > has hit WFI or not.
> 
> <EH> Yes, delay is observed after vm_exit because of WFx (not sure
> WFI or WFE) but only when on a different vCPU in the same VM some
> workload is started.

Let me see if I understand what you mean:

- vcpu-0 is running your timer test, everything is fine
- vcpu-1 starts some other workload, and this affects the timer test
  on the other vcpu

Is that correct? It so, this would tend to indicate that both vcpu
share some physical resources such as a physical CPU. How do you run
your VM?

Also, please work out whether you exit because of a blocking WFI or
WFE, as they are indicative of different guest behaviour.

> Since we pin that workload to its own vCPU, in
> theory, it should not affect the timing of another vCPU.

Why not? a vcpu is just a host thread, and if they share a physical
CPU at some point, there is a knock-on effect.

> > You also don't mention what host kernel version you are running.
> > In general, please try and reproduce the issue using the latest kernel version
> > (5.16 at the moment). Please also indicate what HW you are using.
> 
> <EH> Tried 5.15 and 5.4 kernels. Both have the issue. Do you think
> 5.16 can make a difference? The HW is an Ampere Altra system.

Unlikely. The Altra is a mostly sane system, as long as you make sure
that VMs don't migrate across sockets (at which point it becomes
laughably bad). Nothing to do with KVM though.

Are these kernels compiled from scratch? Or are they whatever the
distro ships? Same question for the guest.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm