From mboxrd@z Thu Jan 1 00:00:00 1970 From: Oliver Upton Date: Tue, 4 Jun 2024 23:00:28 +0000 Subject: [PATCH v4 6/7] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn In-Reply-To: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-7-jthoughton@google.com> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Tue, Jun 04, 2024 at 03:20:20PM -0700, James Houghton wrote: > On Fri, May 31, 2024 at 12:18?PM Oliver Upton wrote: > > > > On Fri, May 31, 2024 at 12:11:33PM -0700, Oliver Upton wrote: > > > On Wed, May 29, 2024 at 06:05:09PM +0000, James Houghton wrote: > > > > > > [...] > > > > > > > diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c > > > > index 9e2bbee77491..eabb07c66a07 100644 > > > > --- a/arch/arm64/kvm/hyp/pgtable.c > > > > +++ b/arch/arm64/kvm/hyp/pgtable.c > > > > @@ -1319,10 +1319,8 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, > > > > data->young = true; > > > > > > > > /* > > > > - * stage2_age_walker() is always called while holding the MMU lock for > > > > - * write, so this will always succeed. Nonetheless, this deliberately > > > > - * follows the race detection pattern of the other stage-2 walkers in > > > > - * case the locking mechanics of the MMU notifiers is ever changed. > > > > + * This walk may not be exclusive; the PTE is permitted to change > > > > + * from under us. > > > > */ > > > > if (data->mkold && !stage2_try_set_pte(ctx, new)) > > > > return -EAGAIN; > > > > > > It is probably worth mentioning that if there was a race to update the > > > PTE then the GFN is most likely young, so failing to clear AF probably > > > isn't even consequential. > > Thanks Oliver. > > > > > Oh, and the WARN_ON() in kvm_pgtable_stage2_test_clear_young() is bogus > > now. Maybe demote it to: > > > > r = kvm_pgtable_walk(...); > > WARN_ON_ONCE(r && r != -EAGAIN); > > Oh, indeed, thank you. Just to make sure -- does it make sense to > retry the cmpxchg if it fails? For example, the way I have it now for > x86[1], we retry the cmpxchg if the spte is still a leaf, otherwise we > move on to the next one having done nothing. Does something like that > make sense for arm64? At least for arm64 I do not see a need for retry. The only possible races are: - A stage-2 fault handler establishing / adjusting the mapping for the GFN. If the guest is directly accessing the GFN in question, what's the point of wiping out AF? Even when returning -EAGAIN we've already primed stage2_age_data::young, so we report the correct state back to the primary MMU. - Another kvm_age_gfn() trying to age the same GFN. I haven't even looked to see if this is possible from the primary MMU POV, but in theory one of the calls will win the race and clear AF. Given Yu's concerns about making pending writers wait, we should take every opportunity to bail on the walk. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0004114D432 for ; Tue, 4 Jun 2024 23:00:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.182 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717542042; cv=none; b=iO6v1hgXmVn841L9LMBS6kkswpVNy56kKSb0I4z0pHd3sNYc+L1Xw2T7WHEs18HaiMthTjcOAluaAoHdrjADsLeng6uyiB9Ez3dBv/LnOq71vYI860EgCQ6rl+psrME2lM0NSyqr9nJaNQeqoktsCJJyBvDYcze0t4vJIw4O1vA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717542042; c=relaxed/simple; bh=3aD6TtBD/gMBjuVttDp8+r2L84PgItrVAANYOy++fgs=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=FkO1Cp8GyANryVJzdNAwjyeLSc4Pzumjy+MVHtT6EOtv3O9tYlmegCzoL9qhy44MFRM0UGlTQZjg5/aurfMpkFITjTUPHUnzmNwaY/BjgIc0iKX0yGgXjCaUxb04xRQYXySHGi2k2OhXuYRZrGShwKZMFg9W2Xxs/4rSO4usKN0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Y9kw/6fC; arc=none smtp.client-ip=95.215.58.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Y9kw/6fC" X-Envelope-To: jthoughton@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717542038; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+MGPWWPfczYg0bV6msWVM8wo7iUe4hq2RoxquwpyNXc=; b=Y9kw/6fCw7WeofkUSeTV2BTDMr5jngNUCtFMMUMXtMQwQZ57KMcePca1DBVS/t4qEWis16 iF4e3V2COwO/t2Yct7KRRLkVhD5NRIOHTq7i/FQgEBT9QWzcLz+QujFnkzpcYe5jZDxDJW +mCbFXXZogKs/7ZLafe//WQoP8h37Tw= X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzhao@google.com X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Tue, 4 Jun 2024 23:00:28 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: James Houghton Cc: Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Yu Zhao , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Subject: Re: [PATCH v4 6/7] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-7-jthoughton@google.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT On Tue, Jun 04, 2024 at 03:20:20PM -0700, James Houghton wrote: > On Fri, May 31, 2024 at 12:18 PM Oliver Upton wrote: > > > > On Fri, May 31, 2024 at 12:11:33PM -0700, Oliver Upton wrote: > > > On Wed, May 29, 2024 at 06:05:09PM +0000, James Houghton wrote: > > > > > > [...] > > > > > > > diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c > > > > index 9e2bbee77491..eabb07c66a07 100644 > > > > --- a/arch/arm64/kvm/hyp/pgtable.c > > > > +++ b/arch/arm64/kvm/hyp/pgtable.c > > > > @@ -1319,10 +1319,8 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, > > > > data->young = true; > > > > > > > > /* > > > > - * stage2_age_walker() is always called while holding the MMU lock for > > > > - * write, so this will always succeed. Nonetheless, this deliberately > > > > - * follows the race detection pattern of the other stage-2 walkers in > > > > - * case the locking mechanics of the MMU notifiers is ever changed. > > > > + * This walk may not be exclusive; the PTE is permitted to change > > > > + * from under us. > > > > */ > > > > if (data->mkold && !stage2_try_set_pte(ctx, new)) > > > > return -EAGAIN; > > > > > > It is probably worth mentioning that if there was a race to update the > > > PTE then the GFN is most likely young, so failing to clear AF probably > > > isn't even consequential. > > Thanks Oliver. > > > > > Oh, and the WARN_ON() in kvm_pgtable_stage2_test_clear_young() is bogus > > now. Maybe demote it to: > > > > r = kvm_pgtable_walk(...); > > WARN_ON_ONCE(r && r != -EAGAIN); > > Oh, indeed, thank you. Just to make sure -- does it make sense to > retry the cmpxchg if it fails? For example, the way I have it now for > x86[1], we retry the cmpxchg if the spte is still a leaf, otherwise we > move on to the next one having done nothing. Does something like that > make sense for arm64? At least for arm64 I do not see a need for retry. The only possible races are: - A stage-2 fault handler establishing / adjusting the mapping for the GFN. If the guest is directly accessing the GFN in question, what's the point of wiping out AF? Even when returning -EAGAIN we've already primed stage2_age_data::young, so we report the correct state back to the primary MMU. - Another kvm_age_gfn() trying to age the same GFN. I haven't even looked to see if this is possible from the primary MMU POV, but in theory one of the calls will win the race and clear AF. Given Yu's concerns about making pending writers wait, we should take every opportunity to bail on the walk. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 51D03C27C50 for ; Tue, 4 Jun 2024 23:00:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OMo/D9629xDDOiUhVL8K5tmUAJn9DAX+RBAV389EpoE=; b=T6NDtMVOv+QYAg /NjNgrVJOPpTCHxri0UOnIVfFPxRVCWET34XEZZ/x5ukvQ8w6Hcd1aGQCC0xwMaVtoLpJqAUbe5r1 W0taV1uv2Q5FIN0VlF1cprInwKIzNd4vE/Gy0x+lqT4sGCTWzQBVgIFX5Fr7ZlTHywR3mvzXMiHRN 301iWfigP+KiOrWllXnpLbt4lVpo58JEuEbk7cUcWuyi9utORVTU4RJaPPEwa4Rt/VWvhKKyah5my uKIpZvUEghrritRlt81kL43fZiOH/nBI19CMnolpl/+A+U3qSXjD5axDoZAUO9St4kIHV60prSFP+ WkMyRofYRjmRaCdh7LXg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sEd97-000000047gm-2ZFe; Tue, 04 Jun 2024 23:00:49 +0000 Received: from out-182.mta1.migadu.com ([2001:41d0:203:375::b6]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sEd91-000000047bQ-0Br5; Tue, 04 Jun 2024 23:00:45 +0000 X-Envelope-To: jthoughton@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717542038; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+MGPWWPfczYg0bV6msWVM8wo7iUe4hq2RoxquwpyNXc=; b=Y9kw/6fCw7WeofkUSeTV2BTDMr5jngNUCtFMMUMXtMQwQZ57KMcePca1DBVS/t4qEWis16 iF4e3V2COwO/t2Yct7KRRLkVhD5NRIOHTq7i/FQgEBT9QWzcLz+QujFnkzpcYe5jZDxDJW +mCbFXXZogKs/7ZLafe//WQoP8h37Tw= X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzhao@google.com X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Tue, 4 Jun 2024 23:00:28 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: James Houghton Cc: Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Yu Zhao , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Subject: Re: [PATCH v4 6/7] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-7-jthoughton@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240604_160043_325462_59B5F64C X-CRM114-Status: GOOD ( 29.25 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gVHVlLCBKdW4gMDQsIDIwMjQgYXQgMDM6MjA6MjBQTSAtMDcwMCwgSmFtZXMgSG91Z2h0b24g d3JvdGU6Cj4gT24gRnJpLCBNYXkgMzEsIDIwMjQgYXQgMTI6MTjigK9QTSBPbGl2ZXIgVXB0b24g PG9saXZlci51cHRvbkBsaW51eC5kZXY+IHdyb3RlOgo+ID4KPiA+IE9uIEZyaSwgTWF5IDMxLCAy MDI0IGF0IDEyOjExOjMzUE0gLTA3MDAsIE9saXZlciBVcHRvbiB3cm90ZToKPiA+ID4gT24gV2Vk LCBNYXkgMjksIDIwMjQgYXQgMDY6MDU6MDlQTSArMDAwMCwgSmFtZXMgSG91Z2h0b24gd3JvdGU6 Cj4gPiA+Cj4gPiA+IFsuLi5dCj4gPiA+Cj4gPiA+ID4gZGlmZiAtLWdpdCBhL2FyY2gvYXJtNjQv a3ZtL2h5cC9wZ3RhYmxlLmMgYi9hcmNoL2FybTY0L2t2bS9oeXAvcGd0YWJsZS5jCj4gPiA+ID4g aW5kZXggOWUyYmJlZTc3NDkxLi5lYWJiMDdjNjZhMDcgMTAwNjQ0Cj4gPiA+ID4gLS0tIGEvYXJj aC9hcm02NC9rdm0vaHlwL3BndGFibGUuYwo+ID4gPiA+ICsrKyBiL2FyY2gvYXJtNjQva3ZtL2h5 cC9wZ3RhYmxlLmMKPiA+ID4gPiBAQCAtMTMxOSwxMCArMTMxOSw4IEBAIHN0YXRpYyBpbnQgc3Rh Z2UyX2FnZV93YWxrZXIoY29uc3Qgc3RydWN0IGt2bV9wZ3RhYmxlX3Zpc2l0X2N0eCAqY3R4LAo+ ID4gPiA+ICAgICBkYXRhLT55b3VuZyA9IHRydWU7Cj4gPiA+ID4KPiA+ID4gPiAgICAgLyoKPiA+ ID4gPiAtICAgICogc3RhZ2UyX2FnZV93YWxrZXIoKSBpcyBhbHdheXMgY2FsbGVkIHdoaWxlIGhv bGRpbmcgdGhlIE1NVSBsb2NrIGZvcgo+ID4gPiA+IC0gICAgKiB3cml0ZSwgc28gdGhpcyB3aWxs IGFsd2F5cyBzdWNjZWVkLiBOb25ldGhlbGVzcywgdGhpcyBkZWxpYmVyYXRlbHkKPiA+ID4gPiAt ICAgICogZm9sbG93cyB0aGUgcmFjZSBkZXRlY3Rpb24gcGF0dGVybiBvZiB0aGUgb3RoZXIgc3Rh Z2UtMiB3YWxrZXJzIGluCj4gPiA+ID4gLSAgICAqIGNhc2UgdGhlIGxvY2tpbmcgbWVjaGFuaWNz IG9mIHRoZSBNTVUgbm90aWZpZXJzIGlzIGV2ZXIgY2hhbmdlZC4KPiA+ID4gPiArICAgICogVGhp cyB3YWxrIG1heSBub3QgYmUgZXhjbHVzaXZlOyB0aGUgUFRFIGlzIHBlcm1pdHRlZCB0byBjaGFu Z2UKPiA+ID4gPiArICAgICogZnJvbSB1bmRlciB1cy4KPiA+ID4gPiAgICAgICovCj4gPiA+ID4g ICAgIGlmIChkYXRhLT5ta29sZCAmJiAhc3RhZ2UyX3RyeV9zZXRfcHRlKGN0eCwgbmV3KSkKPiA+ ID4gPiAgICAgICAgICAgICByZXR1cm4gLUVBR0FJTjsKPiA+ID4KPiA+ID4gSXQgaXMgcHJvYmFi bHkgd29ydGggbWVudGlvbmluZyB0aGF0IGlmIHRoZXJlIHdhcyBhIHJhY2UgdG8gdXBkYXRlIHRo ZQo+ID4gPiBQVEUgdGhlbiB0aGUgR0ZOIGlzIG1vc3QgbGlrZWx5IHlvdW5nLCBzbyBmYWlsaW5n IHRvIGNsZWFyIEFGIHByb2JhYmx5Cj4gPiA+IGlzbid0IGV2ZW4gY29uc2VxdWVudGlhbC4KPiAK PiBUaGFua3MgT2xpdmVyLgo+IAo+ID4KPiA+IE9oLCBhbmQgdGhlIFdBUk5fT04oKSBpbiBrdm1f cGd0YWJsZV9zdGFnZTJfdGVzdF9jbGVhcl95b3VuZygpIGlzIGJvZ3VzCj4gPiBub3cuIE1heWJl IGRlbW90ZSBpdCB0bzoKPiA+Cj4gPiAgIHIgPSBrdm1fcGd0YWJsZV93YWxrKC4uLik7Cj4gPiAg IFdBUk5fT05fT05DRShyICYmIHIgIT0gLUVBR0FJTik7Cj4gCj4gT2gsIGluZGVlZCwgdGhhbmsg eW91LiBKdXN0IHRvIG1ha2Ugc3VyZSAtLSBkb2VzIGl0IG1ha2Ugc2Vuc2UgdG8KPiByZXRyeSB0 aGUgY21weGNoZyBpZiBpdCBmYWlscz8gRm9yIGV4YW1wbGUsIHRoZSB3YXkgSSBoYXZlIGl0IG5v dyBmb3IKPiB4ODZbMV0sIHdlIHJldHJ5IHRoZSBjbXB4Y2hnIGlmIHRoZSBzcHRlIGlzIHN0aWxs IGEgbGVhZiwgb3RoZXJ3aXNlIHdlCj4gbW92ZSBvbiB0byB0aGUgbmV4dCBvbmUgaGF2aW5nIGRv bmUgbm90aGluZy4gRG9lcyBzb21ldGhpbmcgbGlrZSB0aGF0Cj4gbWFrZSBzZW5zZSBmb3IgYXJt NjQ/CgpBdCBsZWFzdCBmb3IgYXJtNjQgSSBkbyBub3Qgc2VlIGEgbmVlZCBmb3IgcmV0cnkuIFRo ZSBvbmx5IHBvc3NpYmxlCnJhY2VzIGFyZToKCiAtIEEgc3RhZ2UtMiBmYXVsdCBoYW5kbGVyIGVz dGFibGlzaGluZyAvIGFkanVzdGluZyB0aGUgbWFwcGluZyBmb3IgdGhlCiAgIEdGTi4gSWYgdGhl IGd1ZXN0IGlzIGRpcmVjdGx5IGFjY2Vzc2luZyB0aGUgR0ZOIGluIHF1ZXN0aW9uLCB3aGF0J3MK ICAgdGhlIHBvaW50IG9mIHdpcGluZyBvdXQgQUY/CgogICBFdmVuIHdoZW4gcmV0dXJuaW5nIC1F QUdBSU4gd2UndmUgYWxyZWFkeSBwcmltZWQgc3RhZ2UyX2FnZV9kYXRhOjp5b3VuZywKICAgc28g d2UgcmVwb3J0IHRoZSBjb3JyZWN0IHN0YXRlIGJhY2sgdG8gdGhlIHByaW1hcnkgTU1VLgoKIC0g QW5vdGhlciBrdm1fYWdlX2dmbigpIHRyeWluZyB0byBhZ2UgdGhlIHNhbWUgR0ZOLiBJIGhhdmVu J3QgZXZlbgogICBsb29rZWQgdG8gc2VlIGlmIHRoaXMgaXMgcG9zc2libGUgZnJvbSB0aGUgcHJp bWFyeSBNTVUgUE9WLCBidXQgaW4KICAgdGhlb3J5IG9uZSBvZiB0aGUgY2FsbHMgd2lsbCB3aW4g dGhlIHJhY2UgYW5kIGNsZWFyIEFGLgoKR2l2ZW4gWXUncyBjb25jZXJucyBhYm91dCBtYWtpbmcg cGVuZGluZyB3cml0ZXJzIHdhaXQsIHdlIHNob3VsZCB0YWtlCmV2ZXJ5IG9wcG9ydHVuaXR5IHRv IGJhaWwgb24gdGhlIHdhbGsuCgotLSAKVGhhbmtzLApPbGl2ZXIKCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LXJpc2N2IG1haWxpbmcgbGlzdAps aW51eC1yaXNjdkBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3Jn L21haWxtYW4vbGlzdGluZm8vbGludXgtcmlzY3YK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 42FC6C27C50 for ; Tue, 4 Jun 2024 23:02:07 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=Y9kw/6fC; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Vv5hF1MQhz3dDT for ; Wed, 5 Jun 2024 09:02:05 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.dev (client-ip=2001:41d0:203:375::b3; helo=out-179.mta1.migadu.com; envelope-from=oliver.upton@linux.dev; receiver=lists.ozlabs.org) Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [IPv6:2001:41d0:203:375::b3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Vv5gG3zpWz30Wf for ; Wed, 5 Jun 2024 09:01:06 +1000 (AEST) X-Envelope-To: jthoughton@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717542038; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+MGPWWPfczYg0bV6msWVM8wo7iUe4hq2RoxquwpyNXc=; b=Y9kw/6fCw7WeofkUSeTV2BTDMr5jngNUCtFMMUMXtMQwQZ57KMcePca1DBVS/t4qEWis16 iF4e3V2COwO/t2Yct7KRRLkVhD5NRIOHTq7i/FQgEBT9QWzcLz+QujFnkzpcYe5jZDxDJW +mCbFXXZogKs/7ZLafe//WQoP8h37Tw= X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzhao@google.com X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Tue, 4 Jun 2024 23:00:28 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: James Houghton Subject: Re: [PATCH v4 6/7] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-7-jthoughton@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, linux-doc@vger.kernel.org, Catalin Marinas , Atish Patra , linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-kselftest@vger.kernel.org, Raghavendra Rao Ananta , linux-riscv@lists.infradead.org, Shuah Khan , Yu Zhao , Jonathan Corbet , Anup Patel , Huacai Chen , David Rientjes , Zenghui Yu , Axel Rasmussen , linux-mips@vger.kernel.org, Albert Ou , Ryan Roberts , Will Deacon , Suzuki K Poulose , Shaoqin Huang , Nicholas Piggin , Bibo Mao , loongarch@lists.linux.dev, Paul Walmsley , David Matlack , Palmer Dabbelt , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, Sean Christopherson , Ankit Agrawal , James Morse , kvm-riscv@lists.infradead.org, Marc Zyngier , Paolo Bonzini , Andrew Morton , Tianrui Zhao , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Tue, Jun 04, 2024 at 03:20:20PM -0700, James Houghton wrote: > On Fri, May 31, 2024 at 12:18 PM Oliver Upton wrote: > > > > On Fri, May 31, 2024 at 12:11:33PM -0700, Oliver Upton wrote: > > > On Wed, May 29, 2024 at 06:05:09PM +0000, James Houghton wrote: > > > > > > [...] > > > > > > > diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c > > > > index 9e2bbee77491..eabb07c66a07 100644 > > > > --- a/arch/arm64/kvm/hyp/pgtable.c > > > > +++ b/arch/arm64/kvm/hyp/pgtable.c > > > > @@ -1319,10 +1319,8 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, > > > > data->young = true; > > > > > > > > /* > > > > - * stage2_age_walker() is always called while holding the MMU lock for > > > > - * write, so this will always succeed. Nonetheless, this deliberately > > > > - * follows the race detection pattern of the other stage-2 walkers in > > > > - * case the locking mechanics of the MMU notifiers is ever changed. > > > > + * This walk may not be exclusive; the PTE is permitted to change > > > > + * from under us. > > > > */ > > > > if (data->mkold && !stage2_try_set_pte(ctx, new)) > > > > return -EAGAIN; > > > > > > It is probably worth mentioning that if there was a race to update the > > > PTE then the GFN is most likely young, so failing to clear AF probably > > > isn't even consequential. > > Thanks Oliver. > > > > > Oh, and the WARN_ON() in kvm_pgtable_stage2_test_clear_young() is bogus > > now. Maybe demote it to: > > > > r = kvm_pgtable_walk(...); > > WARN_ON_ONCE(r && r != -EAGAIN); > > Oh, indeed, thank you. Just to make sure -- does it make sense to > retry the cmpxchg if it fails? For example, the way I have it now for > x86[1], we retry the cmpxchg if the spte is still a leaf, otherwise we > move on to the next one having done nothing. Does something like that > make sense for arm64? At least for arm64 I do not see a need for retry. The only possible races are: - A stage-2 fault handler establishing / adjusting the mapping for the GFN. If the guest is directly accessing the GFN in question, what's the point of wiping out AF? Even when returning -EAGAIN we've already primed stage2_age_data::young, so we report the correct state back to the primary MMU. - Another kvm_age_gfn() trying to age the same GFN. I haven't even looked to see if this is possible from the primary MMU POV, but in theory one of the calls will win the race and clear AF. Given Yu's concerns about making pending writers wait, we should take every opportunity to bail on the walk. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BDC56C27C50 for ; Tue, 4 Jun 2024 23:01:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/dvauZDUdHPCRzELnpcsup3f0TgGVmli0vWh1xWf9V8=; b=3CsZmn76VN/yjR g7tw1dzL8fkI5GpmBZmMkBBaRLE7jW2eNLmJfKpsQI8SYBSgjdhiLtMakCsJz69gq0w7TMrP0N30O XRJS9MHatMXmTbNZvLnM9lbUKfc8TZy2TFBAB1x6ePqJTDQ5hgVu1TCTG+tVSH8z13aM92NOhVddw SHdQ6AWDuZlcv9N7VVIlkJltmMOzV6t+jUSyFAmiBT4GI2cTazc211KAyjXo4QDpo0JFuWGjKPwi5 l6/XqKRq3tKRUGGJM+GgE9MWAocMquViknpNylVyzfOzuCny9oexXQbhXLvPgbCq7uRv+gqLrz/J1 UlDSxCE/dj3iQ7SVvKKA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sEd96-000000047gH-1ew9; Tue, 04 Jun 2024 23:00:48 +0000 Received: from out-182.mta1.migadu.com ([2001:41d0:203:375::b6]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sEd91-000000047bQ-0Br5; Tue, 04 Jun 2024 23:00:45 +0000 X-Envelope-To: jthoughton@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717542038; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+MGPWWPfczYg0bV6msWVM8wo7iUe4hq2RoxquwpyNXc=; b=Y9kw/6fCw7WeofkUSeTV2BTDMr5jngNUCtFMMUMXtMQwQZ57KMcePca1DBVS/t4qEWis16 iF4e3V2COwO/t2Yct7KRRLkVhD5NRIOHTq7i/FQgEBT9QWzcLz+QujFnkzpcYe5jZDxDJW +mCbFXXZogKs/7ZLafe//WQoP8h37Tw= X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzhao@google.com X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Tue, 4 Jun 2024 23:00:28 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: James Houghton Cc: Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Yu Zhao , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Subject: Re: [PATCH v4 6/7] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-7-jthoughton@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240604_160043_325462_59B5F64C X-CRM114-Status: GOOD ( 29.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVHVlLCBKdW4gMDQsIDIwMjQgYXQgMDM6MjA6MjBQTSAtMDcwMCwgSmFtZXMgSG91Z2h0b24g d3JvdGU6Cj4gT24gRnJpLCBNYXkgMzEsIDIwMjQgYXQgMTI6MTjigK9QTSBPbGl2ZXIgVXB0b24g PG9saXZlci51cHRvbkBsaW51eC5kZXY+IHdyb3RlOgo+ID4KPiA+IE9uIEZyaSwgTWF5IDMxLCAy MDI0IGF0IDEyOjExOjMzUE0gLTA3MDAsIE9saXZlciBVcHRvbiB3cm90ZToKPiA+ID4gT24gV2Vk LCBNYXkgMjksIDIwMjQgYXQgMDY6MDU6MDlQTSArMDAwMCwgSmFtZXMgSG91Z2h0b24gd3JvdGU6 Cj4gPiA+Cj4gPiA+IFsuLi5dCj4gPiA+Cj4gPiA+ID4gZGlmZiAtLWdpdCBhL2FyY2gvYXJtNjQv a3ZtL2h5cC9wZ3RhYmxlLmMgYi9hcmNoL2FybTY0L2t2bS9oeXAvcGd0YWJsZS5jCj4gPiA+ID4g aW5kZXggOWUyYmJlZTc3NDkxLi5lYWJiMDdjNjZhMDcgMTAwNjQ0Cj4gPiA+ID4gLS0tIGEvYXJj aC9hcm02NC9rdm0vaHlwL3BndGFibGUuYwo+ID4gPiA+ICsrKyBiL2FyY2gvYXJtNjQva3ZtL2h5 cC9wZ3RhYmxlLmMKPiA+ID4gPiBAQCAtMTMxOSwxMCArMTMxOSw4IEBAIHN0YXRpYyBpbnQgc3Rh Z2UyX2FnZV93YWxrZXIoY29uc3Qgc3RydWN0IGt2bV9wZ3RhYmxlX3Zpc2l0X2N0eCAqY3R4LAo+ ID4gPiA+ICAgICBkYXRhLT55b3VuZyA9IHRydWU7Cj4gPiA+ID4KPiA+ID4gPiAgICAgLyoKPiA+ ID4gPiAtICAgICogc3RhZ2UyX2FnZV93YWxrZXIoKSBpcyBhbHdheXMgY2FsbGVkIHdoaWxlIGhv bGRpbmcgdGhlIE1NVSBsb2NrIGZvcgo+ID4gPiA+IC0gICAgKiB3cml0ZSwgc28gdGhpcyB3aWxs IGFsd2F5cyBzdWNjZWVkLiBOb25ldGhlbGVzcywgdGhpcyBkZWxpYmVyYXRlbHkKPiA+ID4gPiAt ICAgICogZm9sbG93cyB0aGUgcmFjZSBkZXRlY3Rpb24gcGF0dGVybiBvZiB0aGUgb3RoZXIgc3Rh Z2UtMiB3YWxrZXJzIGluCj4gPiA+ID4gLSAgICAqIGNhc2UgdGhlIGxvY2tpbmcgbWVjaGFuaWNz IG9mIHRoZSBNTVUgbm90aWZpZXJzIGlzIGV2ZXIgY2hhbmdlZC4KPiA+ID4gPiArICAgICogVGhp cyB3YWxrIG1heSBub3QgYmUgZXhjbHVzaXZlOyB0aGUgUFRFIGlzIHBlcm1pdHRlZCB0byBjaGFu Z2UKPiA+ID4gPiArICAgICogZnJvbSB1bmRlciB1cy4KPiA+ID4gPiAgICAgICovCj4gPiA+ID4g ICAgIGlmIChkYXRhLT5ta29sZCAmJiAhc3RhZ2UyX3RyeV9zZXRfcHRlKGN0eCwgbmV3KSkKPiA+ ID4gPiAgICAgICAgICAgICByZXR1cm4gLUVBR0FJTjsKPiA+ID4KPiA+ID4gSXQgaXMgcHJvYmFi bHkgd29ydGggbWVudGlvbmluZyB0aGF0IGlmIHRoZXJlIHdhcyBhIHJhY2UgdG8gdXBkYXRlIHRo ZQo+ID4gPiBQVEUgdGhlbiB0aGUgR0ZOIGlzIG1vc3QgbGlrZWx5IHlvdW5nLCBzbyBmYWlsaW5n IHRvIGNsZWFyIEFGIHByb2JhYmx5Cj4gPiA+IGlzbid0IGV2ZW4gY29uc2VxdWVudGlhbC4KPiAK PiBUaGFua3MgT2xpdmVyLgo+IAo+ID4KPiA+IE9oLCBhbmQgdGhlIFdBUk5fT04oKSBpbiBrdm1f cGd0YWJsZV9zdGFnZTJfdGVzdF9jbGVhcl95b3VuZygpIGlzIGJvZ3VzCj4gPiBub3cuIE1heWJl IGRlbW90ZSBpdCB0bzoKPiA+Cj4gPiAgIHIgPSBrdm1fcGd0YWJsZV93YWxrKC4uLik7Cj4gPiAg IFdBUk5fT05fT05DRShyICYmIHIgIT0gLUVBR0FJTik7Cj4gCj4gT2gsIGluZGVlZCwgdGhhbmsg eW91LiBKdXN0IHRvIG1ha2Ugc3VyZSAtLSBkb2VzIGl0IG1ha2Ugc2Vuc2UgdG8KPiByZXRyeSB0 aGUgY21weGNoZyBpZiBpdCBmYWlscz8gRm9yIGV4YW1wbGUsIHRoZSB3YXkgSSBoYXZlIGl0IG5v dyBmb3IKPiB4ODZbMV0sIHdlIHJldHJ5IHRoZSBjbXB4Y2hnIGlmIHRoZSBzcHRlIGlzIHN0aWxs IGEgbGVhZiwgb3RoZXJ3aXNlIHdlCj4gbW92ZSBvbiB0byB0aGUgbmV4dCBvbmUgaGF2aW5nIGRv bmUgbm90aGluZy4gRG9lcyBzb21ldGhpbmcgbGlrZSB0aGF0Cj4gbWFrZSBzZW5zZSBmb3IgYXJt NjQ/CgpBdCBsZWFzdCBmb3IgYXJtNjQgSSBkbyBub3Qgc2VlIGEgbmVlZCBmb3IgcmV0cnkuIFRo ZSBvbmx5IHBvc3NpYmxlCnJhY2VzIGFyZToKCiAtIEEgc3RhZ2UtMiBmYXVsdCBoYW5kbGVyIGVz dGFibGlzaGluZyAvIGFkanVzdGluZyB0aGUgbWFwcGluZyBmb3IgdGhlCiAgIEdGTi4gSWYgdGhl IGd1ZXN0IGlzIGRpcmVjdGx5IGFjY2Vzc2luZyB0aGUgR0ZOIGluIHF1ZXN0aW9uLCB3aGF0J3MK ICAgdGhlIHBvaW50IG9mIHdpcGluZyBvdXQgQUY/CgogICBFdmVuIHdoZW4gcmV0dXJuaW5nIC1F QUdBSU4gd2UndmUgYWxyZWFkeSBwcmltZWQgc3RhZ2UyX2FnZV9kYXRhOjp5b3VuZywKICAgc28g d2UgcmVwb3J0IHRoZSBjb3JyZWN0IHN0YXRlIGJhY2sgdG8gdGhlIHByaW1hcnkgTU1VLgoKIC0g QW5vdGhlciBrdm1fYWdlX2dmbigpIHRyeWluZyB0byBhZ2UgdGhlIHNhbWUgR0ZOLiBJIGhhdmVu J3QgZXZlbgogICBsb29rZWQgdG8gc2VlIGlmIHRoaXMgaXMgcG9zc2libGUgZnJvbSB0aGUgcHJp bWFyeSBNTVUgUE9WLCBidXQgaW4KICAgdGhlb3J5IG9uZSBvZiB0aGUgY2FsbHMgd2lsbCB3aW4g dGhlIHJhY2UgYW5kIGNsZWFyIEFGLgoKR2l2ZW4gWXUncyBjb25jZXJucyBhYm91dCBtYWtpbmcg cGVuZGluZyB3cml0ZXJzIHdhaXQsIHdlIHNob3VsZCB0YWtlCmV2ZXJ5IG9wcG9ydHVuaXR5IHRv IGJhaWwgb24gdGhlIHdhbGsuCgotLSAKVGhhbmtzLApPbGl2ZXIKCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFpbGluZyBs aXN0CmxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5m cmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK