From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Tue, 4 Jun 2024 16:36:56 -0700 Subject: [PATCH v4 6/7] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn In-Reply-To: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-7-jthoughton@google.com> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Tue, Jun 04, 2024, Oliver Upton wrote: > On Tue, Jun 04, 2024 at 03:20:20PM -0700, James Houghton wrote: > > On Fri, May 31, 2024 at 12:18?PM Oliver Upton wrote: > > > > > > On Fri, May 31, 2024 at 12:11:33PM -0700, Oliver Upton wrote: > > > > On Wed, May 29, 2024 at 06:05:09PM +0000, James Houghton wrote: > > > Oh, and the WARN_ON() in kvm_pgtable_stage2_test_clear_young() is bogus > > > now. Maybe demote it to: > > > > > > r = kvm_pgtable_walk(...); > > > WARN_ON_ONCE(r && r != -EAGAIN); > > > > Oh, indeed, thank you. Just to make sure -- does it make sense to > > retry the cmpxchg if it fails? For example, the way I have it now for > > x86[1], we retry the cmpxchg if the spte is still a leaf, otherwise we > > move on to the next one having done nothing. Does something like that > > make sense for arm64? > > At least for arm64 I do not see a need for retry. The only possible > races are: > > - A stage-2 fault handler establishing / adjusting the mapping for the > GFN. If the guest is directly accessing the GFN in question, what's > the point of wiping out AF? > > Even when returning -EAGAIN we've already primed stage2_age_data::young, > so we report the correct state back to the primary MMU. > > - Another kvm_age_gfn() trying to age the same GFN. I haven't even > looked to see if this is possible from the primary MMU POV, but in > theory one of the calls will win the race and clear AF. > > Given Yu's concerns about making pending writers wait, we should take > every opportunity to bail on the walk. +1. The x86 path that retries is, for all intents and purposes, limited to Intel CPUs that don't support EPT A/D bits, i.e. to pre-HSW CPUs. I wouldn't make any decisions based on that code. From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B90514D28E for ; Tue, 4 Jun 2024 23:36:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717544220; cv=none; b=lrvvESvWbusfi1p2FrD7jJidXeUf6uGmUk73PAk/OsXE5/vnJh8C+fNzEpSg0J/JsiyRHRDEjlgNH3SkEdJpStDy3dZDJrXkA779un+HIyXxVfK5MVPeHUGxQeyOMy5jiunmci0W8PZvK8azy+NHRzQAgetJ4ww0cih9B8LvHuM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717544220; c=relaxed/simple; bh=qmv+9P9Yb6Ks0sGqgE9BZpW8i9Nv7p16qdK82LX6Qiw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=PaVT1PODo9XaJuCTdPtmw9YYOXZsVgUZzPey1cDi+XlGKmQsCIF7MWs0YB91IDqToOuvt0a6wEwGqwhg7JFb0P4GjQLtUkcBT4gkkEn7+EzouuFdsgr9cg9DePMIeANirdvuT7Vj5G6sGbeBGzYa6+lEA0vzd5DF8Q5D50dpdaM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=x7gl4+44; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="x7gl4+44" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dfa5743655eso10581970276.1 for ; Tue, 04 Jun 2024 16:36:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717544218; x=1718149018; darn=lists.linux.dev; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=p6cRhpzsCNepMdsrfgI93QBKDGHo0p9E5+86VMMwQfk=; b=x7gl4+44v2jLleWAqkb6gtxhxxUGqb+xHBB8V/LJ0OKyv3ACqGUG7w1TRjNwDyBKya ZVlYhmRIY7k9FNBAJy2lNKzLY4OtcRa2+LlSZZ0shx5lcB8XSYB9vme27PuazSpT6uRC 16oKfBCzD0whu4xQEsFwsC75F4Cp4ANXiBFr5iUAxJ4VU2DTr6o2XKqxeuAmEzIUm6sT MRuSeITUQu/R28o03UJFj1LG2IH3SWhLBSHcu79KKESUDVOZoS47501UdOUdfignxBKC Zq69GtfiO9e/r+4Im6LhfoZow3Ie9nte8+Qx67A1KfT2JxEYTIaz5I2XjrpBcQH1pe0z /l7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717544218; x=1718149018; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=p6cRhpzsCNepMdsrfgI93QBKDGHo0p9E5+86VMMwQfk=; b=Ve4+bzQQAlplRwCvuBCHCA3IbbZmKcAlbluWTLMVygDc1lXhG6jss+MFKcRpG/H0Wx J1w9obzofka2QjwLBQy3kVkIBDxtqtiz/rBBpRD3paZxKRa8W7H8vTarZbjP2NP8qJ0Z HHiaZPs+lcG5z4rdh8PTaBjmYUTPcvuCDzNlevdlskYjJElR5EwEXLAh0N8efWjP6MfL BwP7RzG5uQMs/JqRtofeDKkkeKVe+eSWFLjVIT22CMaon4EBp5KCjHfA008McRA25PhF qdZzxFasYBwk7kCCDNEvLFC7VDootJDaqeN4CXxW3BxVustqx9fc1qzB9X5I3BMrD/PU EIow== X-Forwarded-Encrypted: i=1; AJvYcCVZqjdk7bD2mgvOqg5GbmrRyuzw1H5hj4fEjKfdRIEl+fhVVXo++B+KIj0VFCFr5d/D7+yB/YoTyrZTEXLzMBn3Y5XdM6J5 X-Gm-Message-State: AOJu0Yz47uGbYXINptvA1sAPk61aSmgce8KogcX10y9HT2UEvCf6wjs9 qo4SH25sK0HV1uKAr1MDeJB5IABPEHg/TD5/xKMaQ6FvPOIJqkrlLPNdIZdAm427JkTNe5oO7s4 Cuw== X-Google-Smtp-Source: AGHT+IGvDf74cxgWq1gEKNpKx7itTcK7riRR46b0a169PbO8HQdl4CdVTPFkPhLbD8V/0e9PxPNrLdgSGrY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:1142:b0:df7:9ac4:f1b2 with SMTP id 3f1490d57ef6-dfacac478ebmr240926276.5.1717544218030; Tue, 04 Jun 2024 16:36:58 -0700 (PDT) Date: Tue, 4 Jun 2024 16:36:56 -0700 In-Reply-To: Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-7-jthoughton@google.com> Message-ID: Subject: Re: [PATCH v4 6/7] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn From: Sean Christopherson To: Oliver Upton Cc: James Houghton , Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Yu Zhao , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Tue, Jun 04, 2024, Oliver Upton wrote: > On Tue, Jun 04, 2024 at 03:20:20PM -0700, James Houghton wrote: > > On Fri, May 31, 2024 at 12:18=E2=80=AFPM Oliver Upton wrote: > > > > > > On Fri, May 31, 2024 at 12:11:33PM -0700, Oliver Upton wrote: > > > > On Wed, May 29, 2024 at 06:05:09PM +0000, James Houghton wrote: > > > Oh, and the WARN_ON() in kvm_pgtable_stage2_test_clear_young() is bog= us > > > now. Maybe demote it to: > > > > > > r =3D kvm_pgtable_walk(...); > > > WARN_ON_ONCE(r && r !=3D -EAGAIN); > >=20 > > Oh, indeed, thank you. Just to make sure -- does it make sense to > > retry the cmpxchg if it fails? For example, the way I have it now for > > x86[1], we retry the cmpxchg if the spte is still a leaf, otherwise we > > move on to the next one having done nothing. Does something like that > > make sense for arm64? >=20 > At least for arm64 I do not see a need for retry. The only possible > races are: >=20 > - A stage-2 fault handler establishing / adjusting the mapping for the > GFN. If the guest is directly accessing the GFN in question, what's > the point of wiping out AF? >=20 > Even when returning -EAGAIN we've already primed stage2_age_data::youn= g, > so we report the correct state back to the primary MMU. >=20 > - Another kvm_age_gfn() trying to age the same GFN. I haven't even > looked to see if this is possible from the primary MMU POV, but in > theory one of the calls will win the race and clear AF. >=20 > Given Yu's concerns about making pending writers wait, we should take > every opportunity to bail on the walk. +1. The x86 path that retries is, for all intents and purposes, limited to= Intel CPUs that don't support EPT A/D bits, i.e. to pre-HSW CPUs. I wouldn't mak= e any decisions based on that code. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 51B29C25B78 for ; Tue, 4 Jun 2024 23:37:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=mh9a6Gy1LJvSyhHsj3U2waXdI3HDxMjHStD+O3rBsM0=; b=WAuOHrl9QzbC9So9Tvq2xjLSXf XjmnYtWtrIJb3zJf74HzrT+v2C6/sXVpIxo+00rcZ9GhiP6I7RMJFtu03PmF7Dr6tTloQXOVeX/mW vKS0Dcp8SBIf/42CipkWgOB1SYbXWVDG/+1P2Y1YQLhEzMUxaDUIsJilqBcRyU8ODkheP5BD4m/87 iT7G8kfyRXxopqjJcine3uvSiDomoSm6yI5TfjkAs9ZD1OejgTTva0cKtxXuoylTp5nDg/v+3NvGq xMv1xcah4lkyUBZSb+0jy3Vu7eI1XbdYHfORo/q8hMvwc5t02ik25zAx1PScpGTVIeA7jFppAMKiH Pis/8IRQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sEdiC-00000004BWv-44GG; Tue, 04 Jun 2024 23:37:04 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sEdi9-00000004BV2-3PWB for linux-riscv@lists.infradead.org; Tue, 04 Jun 2024 23:37:03 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-62a08436241so104350657b3.3 for ; Tue, 04 Jun 2024 16:36:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717544218; x=1718149018; darn=lists.infradead.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=p6cRhpzsCNepMdsrfgI93QBKDGHo0p9E5+86VMMwQfk=; b=kbO4jETYEKM+LngctUDVQKwmt6urPbPlHOkghB9AWAA6jJ8umcio6hJFY3AjstbRDG uIHLwLs0AfD+TddLmwo73/Mdink52/Xgd66ZsjkBdkiMPBgZQKb+4lxVpr455rNfwLkj 06bGOJxk8ExtqHXBrbufv3PfH4JEM3XjReEQ0Ga053KyuYxmoO4EuIlRcwGQQJWiDE3n 3q8H6poGb9oreGuSsyle/t2O5/s/fzPvX1drZxoV+Ydff/fB050s6+G4stYQ83bChffV qvS11QpwC4HOwIA0nbH7RaetJ48BYmEkv+S1qwcvOj1MlIr4P4alXLyNzREfjWOl3tL0 fXzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717544218; x=1718149018; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=p6cRhpzsCNepMdsrfgI93QBKDGHo0p9E5+86VMMwQfk=; b=HRZ3MepjiFkTT7H2VOjRfim/WL97C0T3cjl7VUKVPwyQ1f5sn08pAr7KCNlnn3ea72 v83ZSS46Q843nzzEmN4RnoBemY8NS0YCNHfe5hyYRaRGgRzY9MIG/NaRx/OyI7g22GGf 4f85SNaJXMa+lTORPxNaLfFazW5lMAC2XungFCPIXgu9UM6qJxneIoy0Vtyh/2dM+kny 3FeeW2KsD65yx7qyj75KlkosMOHd4+Er5yIBDF+xmaP4a7foOGM0BBX5xfYiLUjnxWAW UZ89+xvx5UR5L7qQKwD8+PtGtxWNFhxMGAOZVHO+h3wHU+O/7IBH1aFvhP3vcdYFETfZ AVyw== X-Forwarded-Encrypted: i=1; AJvYcCXE/o1YbWt405GwgywtWznDZArdTT4Hy4U6OpgwGhpUSWKlgEsQNI7vp/G/ouMz/ciksy8CK3wFodB3b3Z5Xceh9GMd2EvK7A+VdO+aEa6v X-Gm-Message-State: AOJu0YxIAh/UJHKDAUkzBbdiOAVsKrtDCaY68DbUQthtd5rQUFfXa3m1 A47lttdPCFEu80eX6g6MSXJstvOiuBH2+IQ85SJqnWZaXW0A6KtCJkSUzs+kXQ3IEvp9N1lPht5 aWg== X-Google-Smtp-Source: AGHT+IGvDf74cxgWq1gEKNpKx7itTcK7riRR46b0a169PbO8HQdl4CdVTPFkPhLbD8V/0e9PxPNrLdgSGrY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:1142:b0:df7:9ac4:f1b2 with SMTP id 3f1490d57ef6-dfacac478ebmr240926276.5.1717544218030; Tue, 04 Jun 2024 16:36:58 -0700 (PDT) Date: Tue, 4 Jun 2024 16:36:56 -0700 In-Reply-To: Mime-Version: 1.0 References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-7-jthoughton@google.com> Message-ID: Subject: Re: [PATCH v4 6/7] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn From: Sean Christopherson To: Oliver Upton Cc: James Houghton , Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Yu Zhao , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240604_163702_049872_1A034754 X-CRM114-Status: GOOD ( 22.37 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gVHVlLCBKdW4gMDQsIDIwMjQsIE9saXZlciBVcHRvbiB3cm90ZToKPiBPbiBUdWUsIEp1biAw NCwgMjAyNCBhdCAwMzoyMDoyMFBNIC0wNzAwLCBKYW1lcyBIb3VnaHRvbiB3cm90ZToKPiA+IE9u IEZyaSwgTWF5IDMxLCAyMDI0IGF0IDEyOjE44oCvUE0gT2xpdmVyIFVwdG9uIDxvbGl2ZXIudXB0 b25AbGludXguZGV2PiB3cm90ZToKPiA+ID4KPiA+ID4gT24gRnJpLCBNYXkgMzEsIDIwMjQgYXQg MTI6MTE6MzNQTSAtMDcwMCwgT2xpdmVyIFVwdG9uIHdyb3RlOgo+ID4gPiA+IE9uIFdlZCwgTWF5 IDI5LCAyMDI0IGF0IDA2OjA1OjA5UE0gKzAwMDAsIEphbWVzIEhvdWdodG9uIHdyb3RlOgo+ID4g PiBPaCwgYW5kIHRoZSBXQVJOX09OKCkgaW4ga3ZtX3BndGFibGVfc3RhZ2UyX3Rlc3RfY2xlYXJf eW91bmcoKSBpcyBib2d1cwo+ID4gPiBub3cuIE1heWJlIGRlbW90ZSBpdCB0bzoKPiA+ID4KPiA+ ID4gICByID0ga3ZtX3BndGFibGVfd2FsayguLi4pOwo+ID4gPiAgIFdBUk5fT05fT05DRShyICYm IHIgIT0gLUVBR0FJTik7Cj4gPiAKPiA+IE9oLCBpbmRlZWQsIHRoYW5rIHlvdS4gSnVzdCB0byBt YWtlIHN1cmUgLS0gZG9lcyBpdCBtYWtlIHNlbnNlIHRvCj4gPiByZXRyeSB0aGUgY21weGNoZyBp ZiBpdCBmYWlscz8gRm9yIGV4YW1wbGUsIHRoZSB3YXkgSSBoYXZlIGl0IG5vdyBmb3IKPiA+IHg4 NlsxXSwgd2UgcmV0cnkgdGhlIGNtcHhjaGcgaWYgdGhlIHNwdGUgaXMgc3RpbGwgYSBsZWFmLCBv dGhlcndpc2Ugd2UKPiA+IG1vdmUgb24gdG8gdGhlIG5leHQgb25lIGhhdmluZyBkb25lIG5vdGhp bmcuIERvZXMgc29tZXRoaW5nIGxpa2UgdGhhdAo+ID4gbWFrZSBzZW5zZSBmb3IgYXJtNjQ/Cj4g Cj4gQXQgbGVhc3QgZm9yIGFybTY0IEkgZG8gbm90IHNlZSBhIG5lZWQgZm9yIHJldHJ5LiBUaGUg b25seSBwb3NzaWJsZQo+IHJhY2VzIGFyZToKPiAKPiAgLSBBIHN0YWdlLTIgZmF1bHQgaGFuZGxl ciBlc3RhYmxpc2hpbmcgLyBhZGp1c3RpbmcgdGhlIG1hcHBpbmcgZm9yIHRoZQo+ICAgIEdGTi4g SWYgdGhlIGd1ZXN0IGlzIGRpcmVjdGx5IGFjY2Vzc2luZyB0aGUgR0ZOIGluIHF1ZXN0aW9uLCB3 aGF0J3MKPiAgICB0aGUgcG9pbnQgb2Ygd2lwaW5nIG91dCBBRj8KPiAKPiAgICBFdmVuIHdoZW4g cmV0dXJuaW5nIC1FQUdBSU4gd2UndmUgYWxyZWFkeSBwcmltZWQgc3RhZ2UyX2FnZV9kYXRhOjp5 b3VuZywKPiAgICBzbyB3ZSByZXBvcnQgdGhlIGNvcnJlY3Qgc3RhdGUgYmFjayB0byB0aGUgcHJp bWFyeSBNTVUuCj4gCj4gIC0gQW5vdGhlciBrdm1fYWdlX2dmbigpIHRyeWluZyB0byBhZ2UgdGhl IHNhbWUgR0ZOLiBJIGhhdmVuJ3QgZXZlbgo+ICAgIGxvb2tlZCB0byBzZWUgaWYgdGhpcyBpcyBw b3NzaWJsZSBmcm9tIHRoZSBwcmltYXJ5IE1NVSBQT1YsIGJ1dCBpbgo+ICAgIHRoZW9yeSBvbmUg b2YgdGhlIGNhbGxzIHdpbGwgd2luIHRoZSByYWNlIGFuZCBjbGVhciBBRi4KPiAKPiBHaXZlbiBZ dSdzIGNvbmNlcm5zIGFib3V0IG1ha2luZyBwZW5kaW5nIHdyaXRlcnMgd2FpdCwgd2Ugc2hvdWxk IHRha2UKPiBldmVyeSBvcHBvcnR1bml0eSB0byBiYWlsIG9uIHRoZSB3YWxrLgoKKzEuICBUaGUg eDg2IHBhdGggdGhhdCByZXRyaWVzIGlzLCBmb3IgYWxsIGludGVudHMgYW5kIHB1cnBvc2VzLCBs aW1pdGVkIHRvIEludGVsCkNQVXMgdGhhdCBkb24ndCBzdXBwb3J0IEVQVCBBL0QgYml0cywgaS5l LiB0byBwcmUtSFNXIENQVXMuICBJIHdvdWxkbid0IG1ha2UgYW55CmRlY2lzaW9ucyBiYXNlZCBv biB0aGF0IGNvZGUuCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXwpsaW51eC1yaXNjdiBtYWlsaW5nIGxpc3QKbGludXgtcmlzY3ZAbGlzdHMuaW5mcmFkZWFk Lm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LXJp c2N2Cg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31F7CC27C50 for ; Tue, 4 Jun 2024 23:37:48 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=bWp/+fMg; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Vv6TP6xH8z3dK9 for ; Wed, 5 Jun 2024 09:37:45 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20230601 header.b=bWp/+fMg; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=flex--seanjc.bounces.google.com (client-ip=2607:f8b0:4864:20::114a; helo=mail-yw1-x114a.google.com; envelope-from=3gqvfzgykdb0l73gc59hh9e7.5hfebgnqii5-67oeblml.hse34l.hk9@flex--seanjc.bounces.google.com; receiver=lists.ozlabs.org) Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Vv6Sb0VwRz3c3D for ; Wed, 5 Jun 2024 09:37:01 +1000 (AEST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-62ca03fc1ceso40677237b3.1 for ; Tue, 04 Jun 2024 16:37:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717544218; x=1718149018; darn=lists.ozlabs.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=p6cRhpzsCNepMdsrfgI93QBKDGHo0p9E5+86VMMwQfk=; b=bWp/+fMgJySedtKqF1D+TC3FfeJYNy0zD8kaIbZv/yR8fjok7r3iwzn3usTB+B8rEH wEzI9Nv0DvOIJGhGBDGlLmh8gv8mLz9t72D6k7A2sNzWVRH6+8dkhgL/VEq0lTRcRpzk yj3Gb//GjxwAtrEiNj6Pe4g+X8Sd9APp2U2MmpX/IuBDcXFfYyLRRpDzM9Q/jy7e/7Ca Wn3G8eUQUSOzxQAmv0A+eyPrJSSj/TEVsQx7UBJGrv9WKw5PGGNYhvZY+M4wr9fsT6sa f5dSVqva4IvPUeGMfkBQpiZDpmKmsHwM2Az1p5oBbdHFXAoI231xe25YoIosBCRWgQjH cm0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717544218; x=1718149018; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=p6cRhpzsCNepMdsrfgI93QBKDGHo0p9E5+86VMMwQfk=; b=CSzXlNc1I9j4okve+2tEsd06VfT8lxJt7AOcVMQbCvqvQrpV1W2atYd2cfR0tVKYPp hCzxjxJjdvakiFrQsDSK/SSNuIlgBIZFDzJ/NrI8eS8/p+xfZqG2p91xxZkDirsTlo/l 1uVwQD070NM4FbYoIRXuRs27AbnORg2spcmCi48/rEx1idliXgxvHitDvPFcM2vvTnhY 1HCLWZbw0QP7Dg5AExeiphXvPFo3BDuH+8l987fUpHxx5nIe293aI0YxDmIq0PH17Kaw hXAOBUU5it7UZmXIOs3t/6qUErFJHXalcsXTMrxWS4qvXCMRYSUInwJUj00Q7+f+JCKm pVGQ== X-Forwarded-Encrypted: i=1; AJvYcCVYHdTDeQSmFnQyxooOMe2MgF3H6TXfR2iHMVWWnoYggGpIlnnVoaYoi9pLEIA2dJCFkR8MwLrT/RVvMNtpOgpC1wA+x+P1DHoVr7luyw== X-Gm-Message-State: AOJu0Yy5Khle5Laq3JhZERWvZgOZ6k++8xVekot7E+5AK1xpAkQqGHPg W4aLY0KO8HjHmcQ6Tyl6S4YMt78qVbOIcgNCYlFZyvPbxuW+2DDkMN6MWZRsC+CZnL+cjZ7vSs+ Giw== X-Google-Smtp-Source: AGHT+IGvDf74cxgWq1gEKNpKx7itTcK7riRR46b0a169PbO8HQdl4CdVTPFkPhLbD8V/0e9PxPNrLdgSGrY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:1142:b0:df7:9ac4:f1b2 with SMTP id 3f1490d57ef6-dfacac478ebmr240926276.5.1717544218030; Tue, 04 Jun 2024 16:36:58 -0700 (PDT) Date: Tue, 4 Jun 2024 16:36:56 -0700 In-Reply-To: Mime-Version: 1.0 References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-7-jthoughton@google.com> Message-ID: Subject: Re: [PATCH v4 6/7] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn From: Sean Christopherson To: Oliver Upton Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , kvm@vger.kernel.org, linux-doc@vger.kernel.org, Catalin Marinas , Atish Patra , linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-kselftest@vger.kernel.org, Raghavendra Rao Ananta , linux-riscv@lists.infradead.org, Shuah Khan , Yu Zhao , Jonathan Corbet , Anup Patel , Huacai Chen , David Rientjes , Zenghui Yu , Axel Rasmussen , linux-mips@vger.kernel.org, Albert Ou , Ryan Roberts , Will Deacon , Suzuki K Poulose , Shaoqin Huang , Nicholas Piggin , Bibo Mao , loongarch@lists.linux.dev, Paul Walmsley , David Matlack , Palmer Dabbelt , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, Ankit Agrawal , James Morse , kvm-riscv@lists.infradead.org, Marc Zyngier , Paolo Bonzini , Andrew Morton , Tianrui Zhao , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Tue, Jun 04, 2024, Oliver Upton wrote: > On Tue, Jun 04, 2024 at 03:20:20PM -0700, James Houghton wrote: > > On Fri, May 31, 2024 at 12:18=E2=80=AFPM Oliver Upton wrote: > > > > > > On Fri, May 31, 2024 at 12:11:33PM -0700, Oliver Upton wrote: > > > > On Wed, May 29, 2024 at 06:05:09PM +0000, James Houghton wrote: > > > Oh, and the WARN_ON() in kvm_pgtable_stage2_test_clear_young() is bog= us > > > now. Maybe demote it to: > > > > > > r =3D kvm_pgtable_walk(...); > > > WARN_ON_ONCE(r && r !=3D -EAGAIN); > >=20 > > Oh, indeed, thank you. Just to make sure -- does it make sense to > > retry the cmpxchg if it fails? For example, the way I have it now for > > x86[1], we retry the cmpxchg if the spte is still a leaf, otherwise we > > move on to the next one having done nothing. Does something like that > > make sense for arm64? >=20 > At least for arm64 I do not see a need for retry. The only possible > races are: >=20 > - A stage-2 fault handler establishing / adjusting the mapping for the > GFN. If the guest is directly accessing the GFN in question, what's > the point of wiping out AF? >=20 > Even when returning -EAGAIN we've already primed stage2_age_data::youn= g, > so we report the correct state back to the primary MMU. >=20 > - Another kvm_age_gfn() trying to age the same GFN. I haven't even > looked to see if this is possible from the primary MMU POV, but in > theory one of the calls will win the race and clear AF. >=20 > Given Yu's concerns about making pending writers wait, we should take > every opportunity to bail on the walk. +1. The x86 path that retries is, for all intents and purposes, limited to= Intel CPUs that don't support EPT A/D bits, i.e. to pre-HSW CPUs. I wouldn't mak= e any decisions based on that code. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D029AC25B78 for ; Tue, 4 Jun 2024 23:37:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Ci+t955NvlC+PxqmQLVR2o/wjrj4YB6LNuG+zfHB2RY=; b=uwoe8d81QYVnFARWjyJc3fk6TE iYXeyZSOV5fAnfIy1UzN21znaC0Nluk14HQYu7Mv8GR+feaOEoAcG4zZiOjDiGy9CGNlt7/BCZW8f NdcOpT0GVq5hu642M1TTaWqquQo6HUysKcq5f3xW0n9UG/6FI4dFrfZOzihhS8LbTox8yk14WvR7T vTLwJVWX0YqlShA7wFUrGyAIXf6uxkfHlDfOLqY2n0AgajvDJ4FbNCRu7ybZcm2U4wfHSxMncMVgV WlqplTasY0eVQ14MC75odrdEIOuM56TWtdsR+jNkY0+DKfEy1tj+gNGi7gk6Xd9tujYaO0uSitkJs isEYZGrg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sEdiD-00000004BXS-3WKP; Tue, 04 Jun 2024 23:37:05 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sEdiB-00000004BV0-0KDh for linux-arm-kernel@lists.infradead.org; Tue, 04 Jun 2024 23:37:04 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-62a08436241so104350597b3.3 for ; Tue, 04 Jun 2024 16:36:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717544218; x=1718149018; darn=lists.infradead.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=p6cRhpzsCNepMdsrfgI93QBKDGHo0p9E5+86VMMwQfk=; b=kbO4jETYEKM+LngctUDVQKwmt6urPbPlHOkghB9AWAA6jJ8umcio6hJFY3AjstbRDG uIHLwLs0AfD+TddLmwo73/Mdink52/Xgd66ZsjkBdkiMPBgZQKb+4lxVpr455rNfwLkj 06bGOJxk8ExtqHXBrbufv3PfH4JEM3XjReEQ0Ga053KyuYxmoO4EuIlRcwGQQJWiDE3n 3q8H6poGb9oreGuSsyle/t2O5/s/fzPvX1drZxoV+Ydff/fB050s6+G4stYQ83bChffV qvS11QpwC4HOwIA0nbH7RaetJ48BYmEkv+S1qwcvOj1MlIr4P4alXLyNzREfjWOl3tL0 fXzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717544218; x=1718149018; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=p6cRhpzsCNepMdsrfgI93QBKDGHo0p9E5+86VMMwQfk=; b=sdRA1Ye6qHHgLU5hXZK4QmfSN5Aziub0AgHNZQs6eEZnbHYwcAqs0I7QwEvqyNobE5 XWG+jFCf8k9irY6RNral9mANhxOf6WjoiFgcG4VgdGGfJCw+1/4kuoz74MR6cGtm7Rqi c9E6T6rkDjfYN2GSJyeaG2MSXBM7mtPCJud5we0YyBOVUxJN/aDgipBGAC/nWFz5nauU DQ3CJKcUg6LZAAzTAsKgQrp5g1xbnfo7DiEANRlyODQ1gRRoU9G7niL/n2D1hHer0Onx WRcbBV8KAXh08NdPyh+CBmafHEDy/gBCOk3bknAaSsMhU041yrRrvdZvn/PP7BvS8STN p8Zg== X-Forwarded-Encrypted: i=1; AJvYcCUnbYLWm6nVozAgM6V1BtXnexsQpeoGUGktLLltWlbJoisnv4YwnOu77ryZ5X1+yjz8V7oFYwgvwL3rPV2dpIUPhMFDsyKBB/k0dq85G4EkhSyYjBY= X-Gm-Message-State: AOJu0YzDfl5wxeGbuhPbnHSgvIhJLDpBXZai4Did7k8pbSfNw7lXjTxO AzGxVYFXpnWBAU25GVUQKartoPz746e3tmTvi/9WlgN/lqOuIjyCrl2cIEAjArjgw8UcfY5Yq2u j5A== X-Google-Smtp-Source: AGHT+IGvDf74cxgWq1gEKNpKx7itTcK7riRR46b0a169PbO8HQdl4CdVTPFkPhLbD8V/0e9PxPNrLdgSGrY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:1142:b0:df7:9ac4:f1b2 with SMTP id 3f1490d57ef6-dfacac478ebmr240926276.5.1717544218030; Tue, 04 Jun 2024 16:36:58 -0700 (PDT) Date: Tue, 4 Jun 2024 16:36:56 -0700 In-Reply-To: Mime-Version: 1.0 References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-7-jthoughton@google.com> Message-ID: Subject: Re: [PATCH v4 6/7] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn From: Sean Christopherson To: Oliver Upton Cc: James Houghton , Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Yu Zhao , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240604_163703_127119_54779A50 X-CRM114-Status: GOOD ( 23.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVHVlLCBKdW4gMDQsIDIwMjQsIE9saXZlciBVcHRvbiB3cm90ZToKPiBPbiBUdWUsIEp1biAw NCwgMjAyNCBhdCAwMzoyMDoyMFBNIC0wNzAwLCBKYW1lcyBIb3VnaHRvbiB3cm90ZToKPiA+IE9u IEZyaSwgTWF5IDMxLCAyMDI0IGF0IDEyOjE44oCvUE0gT2xpdmVyIFVwdG9uIDxvbGl2ZXIudXB0 b25AbGludXguZGV2PiB3cm90ZToKPiA+ID4KPiA+ID4gT24gRnJpLCBNYXkgMzEsIDIwMjQgYXQg MTI6MTE6MzNQTSAtMDcwMCwgT2xpdmVyIFVwdG9uIHdyb3RlOgo+ID4gPiA+IE9uIFdlZCwgTWF5 IDI5LCAyMDI0IGF0IDA2OjA1OjA5UE0gKzAwMDAsIEphbWVzIEhvdWdodG9uIHdyb3RlOgo+ID4g PiBPaCwgYW5kIHRoZSBXQVJOX09OKCkgaW4ga3ZtX3BndGFibGVfc3RhZ2UyX3Rlc3RfY2xlYXJf eW91bmcoKSBpcyBib2d1cwo+ID4gPiBub3cuIE1heWJlIGRlbW90ZSBpdCB0bzoKPiA+ID4KPiA+ ID4gICByID0ga3ZtX3BndGFibGVfd2FsayguLi4pOwo+ID4gPiAgIFdBUk5fT05fT05DRShyICYm IHIgIT0gLUVBR0FJTik7Cj4gPiAKPiA+IE9oLCBpbmRlZWQsIHRoYW5rIHlvdS4gSnVzdCB0byBt YWtlIHN1cmUgLS0gZG9lcyBpdCBtYWtlIHNlbnNlIHRvCj4gPiByZXRyeSB0aGUgY21weGNoZyBp ZiBpdCBmYWlscz8gRm9yIGV4YW1wbGUsIHRoZSB3YXkgSSBoYXZlIGl0IG5vdyBmb3IKPiA+IHg4 NlsxXSwgd2UgcmV0cnkgdGhlIGNtcHhjaGcgaWYgdGhlIHNwdGUgaXMgc3RpbGwgYSBsZWFmLCBv dGhlcndpc2Ugd2UKPiA+IG1vdmUgb24gdG8gdGhlIG5leHQgb25lIGhhdmluZyBkb25lIG5vdGhp bmcuIERvZXMgc29tZXRoaW5nIGxpa2UgdGhhdAo+ID4gbWFrZSBzZW5zZSBmb3IgYXJtNjQ/Cj4g Cj4gQXQgbGVhc3QgZm9yIGFybTY0IEkgZG8gbm90IHNlZSBhIG5lZWQgZm9yIHJldHJ5LiBUaGUg b25seSBwb3NzaWJsZQo+IHJhY2VzIGFyZToKPiAKPiAgLSBBIHN0YWdlLTIgZmF1bHQgaGFuZGxl ciBlc3RhYmxpc2hpbmcgLyBhZGp1c3RpbmcgdGhlIG1hcHBpbmcgZm9yIHRoZQo+ICAgIEdGTi4g SWYgdGhlIGd1ZXN0IGlzIGRpcmVjdGx5IGFjY2Vzc2luZyB0aGUgR0ZOIGluIHF1ZXN0aW9uLCB3 aGF0J3MKPiAgICB0aGUgcG9pbnQgb2Ygd2lwaW5nIG91dCBBRj8KPiAKPiAgICBFdmVuIHdoZW4g cmV0dXJuaW5nIC1FQUdBSU4gd2UndmUgYWxyZWFkeSBwcmltZWQgc3RhZ2UyX2FnZV9kYXRhOjp5 b3VuZywKPiAgICBzbyB3ZSByZXBvcnQgdGhlIGNvcnJlY3Qgc3RhdGUgYmFjayB0byB0aGUgcHJp bWFyeSBNTVUuCj4gCj4gIC0gQW5vdGhlciBrdm1fYWdlX2dmbigpIHRyeWluZyB0byBhZ2UgdGhl IHNhbWUgR0ZOLiBJIGhhdmVuJ3QgZXZlbgo+ICAgIGxvb2tlZCB0byBzZWUgaWYgdGhpcyBpcyBw b3NzaWJsZSBmcm9tIHRoZSBwcmltYXJ5IE1NVSBQT1YsIGJ1dCBpbgo+ICAgIHRoZW9yeSBvbmUg b2YgdGhlIGNhbGxzIHdpbGwgd2luIHRoZSByYWNlIGFuZCBjbGVhciBBRi4KPiAKPiBHaXZlbiBZ dSdzIGNvbmNlcm5zIGFib3V0IG1ha2luZyBwZW5kaW5nIHdyaXRlcnMgd2FpdCwgd2Ugc2hvdWxk IHRha2UKPiBldmVyeSBvcHBvcnR1bml0eSB0byBiYWlsIG9uIHRoZSB3YWxrLgoKKzEuICBUaGUg eDg2IHBhdGggdGhhdCByZXRyaWVzIGlzLCBmb3IgYWxsIGludGVudHMgYW5kIHB1cnBvc2VzLCBs aW1pdGVkIHRvIEludGVsCkNQVXMgdGhhdCBkb24ndCBzdXBwb3J0IEVQVCBBL0QgYml0cywgaS5l LiB0byBwcmUtSFNXIENQVXMuICBJIHdvdWxkbid0IG1ha2UgYW55CmRlY2lzaW9ucyBiYXNlZCBv biB0aGF0IGNvZGUuCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fXwpsaW51eC1hcm0ta2VybmVsIG1haWxpbmcgbGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3Rz LmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5m by9saW51eC1hcm0ta2VybmVsCg==