From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,INCLUDES_CR_TRAILER, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4FC9C6369E for ; Wed, 2 Dec 2020 22:58:32 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9C593221FD for ; Wed, 2 Dec 2020 22:58:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9C593221FD Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=csgraf.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:49160 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kkb4w-00021t-Am for qemu-devel@archiver.kernel.org; Wed, 02 Dec 2020 17:58:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:58324) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kkb3g-0001S5-0l; Wed, 02 Dec 2020 17:57:13 -0500 Received: from mail.csgraf.de ([188.138.100.120]:39804 helo=zulu616.server4you.de) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kkb3Y-0001nK-Pn; Wed, 02 Dec 2020 17:57:11 -0500 Received: from freeip.amazon.com (ec2-3-122-114-9.eu-central-1.compute.amazonaws.com [3.122.114.9]) by csgraf.de (Postfix) with ESMTPSA id 1E18D390036D; Wed, 2 Dec 2020 23:57:00 +0100 (CET) Subject: Re: [PATCH v1 1/1] hvf: arm: Properly sync guest time on migration To: Frank Yang References: <2dc974cc-abe2-d034-1720-d5a2651a9042@csgraf.de> From: Alexander Graf Message-ID: <658b1719-4635-edab-f3e2-6a9ac0bc01a1@csgraf.de> Date: Wed, 2 Dec 2020 23:56:59 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------8E81D7DC2B5F2A2D3B1F9CAE" Content-Language: en-US Received-SPF: pass client-ip=188.138.100.120; envelope-from=agraf@csgraf.de; helo=zulu616.server4you.de X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Eduardo Habkost , Richard Henderson , qemu-devel , Cameron Esfahani , Roman Bolshakov , qemu-arm , Claudio Fontana , Paolo Bonzini , Peter Collingbourne Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is a multi-part message in MIME format. --------------8E81D7DC2B5F2A2D3B1F9CAE Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 02.12.20 23:46, Frank Yang wrote: > > > On Wed, Dec 2, 2020 at 2:28 PM Alexander Graf > wrote: > > > On 02.12.20 23:19, Frank Yang wrote: >> >> From downstream: >> https://android-review.googlesource.com/c/platform/external/qemu/+/1515002 >> >> >> Based on v3 of Alexander Graf's patches >> >> https://patchew.org/QEMU/20201202190408.2041-1-agraf@csgraf.de >> >> >> We need to adjust CNTVOFF_EL2 so that time doesnt warp.  Even >> though we >> can set separate CNTVOFF_EL2 values per vCPU, it just is not >> worth the >> require effort to do that accurately---with individual values, >> even if >> they are a tiny bit off it can result in a lockup due to inconsistent >> time differences between vCPUs. So just use a global approximate >> value >> for now. >> >> Not tested in upstream yet, but Android emulator snapshots work >> without >> time warp now. >> >> Signed-off-by: Lingfeng Yang > > > > If we just always make CNTV start at the same 0 as > QEMU_CLOCK_VIRTUAL, we should be able to just recover the offset > after migration by looking at QEMU_CLOCK_VIRTUAL to set CNTVOFF, > right? > > That would end up much easier than this patch I hope. > > > > The virtual clock interfaces/implementations in QEMU seem complex to > me relative to the fix needed here and they don't seem to compute > ticks with mach_absolute_time() (which in this case we want since we > want to compute in timer ticks instead of having to mess with ns / > cycle conversions). I do agree this patch does seem more complicated > on the surface though versus "just" setting cntvoff directly to some > value. Maybe we should simplify the QEMU_CLOCK_VIRTUAL implementation > first to maintain CNTVOFF_EL2/CNTV using mach_absolute_time() first? So QEMU_CLOCK_VIRTUAL calls cpu_get_clock() which just adds an offset to gettimeofday(). This offset is already part of the live migration stream[1]. So if you just configure CNTVOFF_EL2 based on QEMU_CLOCK_VIRTUAL adjusted by the clock frequency on vcpu init, you should have everything you need. You can do that on every CPU init even, as the virtual clock will just be 0 on start. The only thing we need to change then is to move the WFI from a direct call to mach_absolute_time() to also check the virtual clock instead. I would hope that gettimeofday() calls mach_absolute_time() in the background too to speed it up. Alex [1] https://git.qemu.org/?p=qemu.git;a=blob;f=softmmu/cpu-timers.c;h=1eb7c675c18bda7773d4a9c549f0157c6e978a83;hb=HEAD#l229 --------------8E81D7DC2B5F2A2D3B1F9CAE Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit


On 02.12.20 23:46, Frank Yang wrote:


On Wed, Dec 2, 2020 at 2:28 PM Alexander Graf <agraf@csgraf.de> wrote:


On 02.12.20 23:19, Frank Yang wrote:

From downstream: https://android-review.googlesource.com/c/platform/external/qemu/+/1515002

Based on v3 of Alexander Graf's patches

https://patchew.org/QEMU/20201202190408.2041-1-agraf@csgraf.de

We need to adjust CNTVOFF_EL2 so that time doesnt warp.  Even though we
can set separate CNTVOFF_EL2 values per vCPU, it just is not worth the
require effort to do that accurately---with individual values, even if
they are a tiny bit off it can result in a lockup due to inconsistent
time differences between vCPUs. So just use a global approximate value
for now.

Not tested in upstream yet, but Android emulator snapshots work without
time warp now.

Signed-off-by: Lingfeng Yang <lfy@google.com>


If we just always make CNTV start at the same 0 as QEMU_CLOCK_VIRTUAL, we should be able to just recover the offset after migration by looking at QEMU_CLOCK_VIRTUAL to set CNTVOFF, right?

That would end up much easier than this patch I hope.



The virtual clock interfaces/implementations in QEMU seem complex to me relative to the fix needed here and they don't seem to compute ticks with mach_absolute_time() (which in this case we want since we want to compute in timer ticks instead of having to mess with ns / cycle conversions). I do agree this patch does seem more complicated on the surface though versus "just" setting cntvoff directly to some value. Maybe we should simplify the QEMU_CLOCK_VIRTUAL implementation first to maintain CNTVOFF_EL2/CNTV using mach_absolute_time() first?


So QEMU_CLOCK_VIRTUAL calls cpu_get_clock() which just adds an offset to gettimeofday(). This offset is already part of the live migration stream[1]. So if you just configure CNTVOFF_EL2 based on QEMU_CLOCK_VIRTUAL adjusted by the clock frequency on vcpu init, you should have everything you need. You can do that on every CPU init even, as the virtual clock will just be 0 on start.

The only thing we need to change then is to move the WFI from a direct call to mach_absolute_time() to also check the virtual clock instead. I would hope that gettimeofday() calls mach_absolute_time() in the background too to speed it up.


Alex


[1] https://git.qemu.org/?p=qemu.git;a=blob;f=softmmu/cpu-timers.c;h=1eb7c675c18bda7773d4a9c549f0157c6e978a83;hb=HEAD#l229

--------------8E81D7DC2B5F2A2D3B1F9CAE--