Re: Timekeeping on ARM guests/hosts

[Christoffer Dall <christoffer.dall@arm.com>]

On Fri, Nov 02, 2018 at 05:19:20PM -0700, Bijan Mottahedeh wrote:
> On 11/2/2018 7:34 AM, Christoffer Dall wrote:
> >The key is whether the userspace program that controls the KVM VM
> >(kvmtool, QEMU, crosvm) uses the KVM_REG_ARM_TIMER_CNT ioctl to save the
> >VM view of virtual time, and to retore that at a later time.
> >
> >KVM adjusts the CNTVOFF_EL2 for the VM on which the ioctl is executed to
> >represent the value written by userspace to the VM when the VM reads
> >CNTVCT_EL0.
> >
> >>>The Arm ARM doesn't say a great deal about power saving modes so I
> >>>wouldn't be surprised if there's differing behaviour as to whether the
> >>>system clock is stopped during suspend modes. Indeed I wonder if the
> >>>clock can even go backwards during a suspend-to-disk/resume cycle? I
> >>>don't have hardware handy to test this.
> >>The specific hardware I have in front of me is a Samsung Chromebook
> >>Plus (board codename "kevin"), which I believe has an RK3399
> >>processor. (More info at
> >>https://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices)
> >>
> >For the purpose of timekeeping in KVM, we need an architecturally
> >meaningful solution, not something specific to any device.
> 
> I've been working on a QEMU patch to address a problem observed when an
> active guest is paused and resumed after a certain delay with virsh or the
> QEMU monitor.
> 
> A simple test to reproduce the problem executes one or more instances of the
> following command in the guest:
> 
> dd if=/dev/zero of=/dev/null &
> 
> and then pauses and resumes the guest after a certain delay:
> 
> virsh suspend <guest>        # pauses the guest
> sleep 120
> virsh resume <guest>
> 
> After the guest is resumed, there are soft lockup warning messages displayed
> on the console.
> 
> A comparison with x86 shows that hwclock and date values diverge after the
> above pause and resume sequence for x86 but remain the same for Arm.
> 
> The patch accumulates the total guest pause time in QEMU and adjusts the
> virtual offset counter accordingly with the KVM_REG_ARM_TIMER_CNT ioctl
> before the guest is resumed.  With the patch the time behavior is the same
> as x86 and the soft lockup messages go away.
> 
> I've tested the patch on an Ampere eMag server but I'm not sure how
> complete, generic, and backward compatible of a solution the patch is in
> terms of other Arm platforms.

Without having looked at the patch, that sounds to me like it would work
on all Arm implementations (ignoring hardware bugs around the
counter/time, if any should exist).

> 
> Also, I'm not sure if and when this patch would be superseded by the
> proposal from your KVM Forum 2018 presentation:
> 
> Paravirtualized Time for Arm-based Systems
> https://developer.arm.com/docs/den0057/a

I think that patch would be part of the work to support the semantics
in the PV Timer spec.  This is part of what userspace would have to do
to communicate 'Live Physical Time (LPT)' to the guest, by telling KVM
to preserve a previous value of LPT, which it in turns does by adjusting
the offset in CNTVOFF_EL2 (which effectively accounts for 'paused' time
- in PV Timer spec terms).

> 
> Would it make sense to send the patch as an RFC for evaluation at this point
> or do you suggest any other considerations?
> 

Definitely that would make sense.

We need additional work to account for suspended time, either via hooks
in userspace or via hooks directly in the arch timer in KVM, and all of
the PV features come on top.


Thanks,

    Christoffer
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm