From mboxrd@z Thu Jan 1 00:00:00 1970 From: Oliver Upton Date: Fri, 31 May 2024 21:18:12 +0000 Subject: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging In-Reply-To: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Fri, May 31, 2024 at 02:31:17PM -0600, Yu Zhao wrote: > On Fri, May 31, 2024 at 1:24?AM Oliver Upton wrote: [...] > > Grabbing the MMU lock for write to scan sucks, no argument there. But > > can you please be specific about the impact of read lock v. RCU in the > > case of arm64? I had asked about this before and you never replied. > > > > My concern remains that adding support for software table walkers > > outside of the MMU lock entirely requires more work than just deferring > > the deallocation to an RCU callback. Walkers that previously assumed > > 'exclusive' access while holding the MMU lock for write must now cope > > with volatile PTEs. > > > > Yes, this problem already exists when hardware sets the AF, but the > > lock-free walker implementation needs to be generic so it can be applied > > for other PTE bits. > > Direct reclaim is multi-threaded and each reclaimer can take the mmu > lock for read (testing the A-bit) or write (unmapping before paging > out) on arm64. The fundamental problem of using the readers-writer > lock in this case is priority inversion: the readers have lower > priority than the writers, so ideally, we don't want the readers to > block the writers at all. So we already have this sort of problem of stage-2 fault handling v. secondary MMU invalidations, which is why I've been doubtful of the perceived issue. In fact, I'd argue that needing to wait for faults is worse than aging participation since those can be trivially influenced by userspace/guest. In any case, we shouldn't ever be starved since younger readers cannot enter the critical section with a pending writer. > As I said earlier, I prefer we drop the arm64 support for now, but I > will not object to taking the mmu lock for read when clearing the > A-bit, as long as we fully understand the problem here and document it > clearly. I'd be convinced of this if there's data that shows read lock acquisition is in fact consequential. Otherwise, I'm not sure the added complexity of RCU table walkers (per my statement above) is worth the effort / maintenance burden. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-187.mta0.migadu.com (out-187.mta0.migadu.com [91.218.175.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4CD341CAA6 for ; Fri, 31 May 2024 21:18:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.187 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717190306; cv=none; b=hGvUDfH3hLyhyf91yEBHUDhZxiO+Q6L3jLkXmqsfYXSrbdQuQrQCSG2fj+xnV2AV7JsXoo5Foke/oGLPbvNE6ALF4F63ghCT/WTPXh4gzlM/MjPyl1+sx+yXWAP9AVQHiSXrUA/hKGkZpCGaGpEYgBdljx8vogX86wPQCgRmzPo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717190306; c=relaxed/simple; bh=3DBErqafiHT5NghS4OQdJ+pZJSKeWI3xFtKVnMKcBR0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=sPHNoft/UCNlSqqu0S65uQKSjvdbRCizZ6lx6Dn1vUeXSkbuE2xjiVxxQ73p7cyLBTNiwUzPG5P90XEu76bLeYMhmwhhZp4jkdycCeJEoU86g3qXRgZgTjxG5HO6E6ED9/ZF3IN7vwY2s3eJ3HtL0JPcn92X4/NiUUtojoUsDMU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Ls3C5/08; arc=none smtp.client-ip=91.218.175.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Ls3C5/08" X-Envelope-To: yuzhao@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717190302; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b4Xmi0701V1mOSvq8yprUubnI972AbFWvRE29wJdk78=; b=Ls3C5/08/280zRJqgHSiEjY525Nc+6AHW21VTOJinn6gvpWo5wGpHRdfxPSCAblARNzCTk Ef8yPQJkMIbUZrU0gLqIxyh5em+aap01i8qUkndBnnlYA3LWYTDaFt+T56XGowNC6n0gMV 1xZHPusSMv7Np2WADZqnpp/2B0dMfIM= X-Envelope-To: jthoughton@google.com X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Fri, 31 May 2024 21:18:12 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Yu Zhao Cc: James Houghton , Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Subject: Re: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT On Fri, May 31, 2024 at 02:31:17PM -0600, Yu Zhao wrote: > On Fri, May 31, 2024 at 1:24 AM Oliver Upton wrote: [...] > > Grabbing the MMU lock for write to scan sucks, no argument there. But > > can you please be specific about the impact of read lock v. RCU in the > > case of arm64? I had asked about this before and you never replied. > > > > My concern remains that adding support for software table walkers > > outside of the MMU lock entirely requires more work than just deferring > > the deallocation to an RCU callback. Walkers that previously assumed > > 'exclusive' access while holding the MMU lock for write must now cope > > with volatile PTEs. > > > > Yes, this problem already exists when hardware sets the AF, but the > > lock-free walker implementation needs to be generic so it can be applied > > for other PTE bits. > > Direct reclaim is multi-threaded and each reclaimer can take the mmu > lock for read (testing the A-bit) or write (unmapping before paging > out) on arm64. The fundamental problem of using the readers-writer > lock in this case is priority inversion: the readers have lower > priority than the writers, so ideally, we don't want the readers to > block the writers at all. So we already have this sort of problem of stage-2 fault handling v. secondary MMU invalidations, which is why I've been doubtful of the perceived issue. In fact, I'd argue that needing to wait for faults is worse than aging participation since those can be trivially influenced by userspace/guest. In any case, we shouldn't ever be starved since younger readers cannot enter the critical section with a pending writer. > As I said earlier, I prefer we drop the arm64 support for now, but I > will not object to taking the mmu lock for read when clearing the > A-bit, as long as we fully understand the problem here and document it > clearly. I'd be convinced of this if there's data that shows read lock acquisition is in fact consequential. Otherwise, I'm not sure the added complexity of RCU table walkers (per my statement above) is worth the effort / maintenance burden. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 415D6C27C4F for ; Fri, 31 May 2024 21:18:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=XeSYqEbntA5Qklf2FwtCdFP51iP63XVPCtbPdBm9mc8=; b=bFc+ZODcLBU2bU oKI4PNp6TOTFGCazrvbCcTqvXyqVZwRQwwO64mptGyN/E1zS48RB/f40A07TSejFmDjHqqJA2Dy/9 a7v271i6wIsNzFCm+ZfNPjFEKMOHSDtNM2Tq+DVWLeo8Hs3SYPf3LKYb2CFUNvgr+sHTUqpGB4zXU TimmxcdkiB9+yHNGyCFhjb5H/Q7esvTAzs3STNqqiiMtMmNWVwcx1ibz0EtkeHzGqvo1FPdTNvHwg 7L7/sp9+smIHgCPM4VzBmMJJvzoruaQft1zLGaZa1fB3rFjqEaa/Tdw/yHhq5szBD9cUBdk6sgkDv a2JtxDdRnbhNTv6m5JdA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sD9e5-0000000BTQz-1EXR; Fri, 31 May 2024 21:18:44 +0000 Received: from out-185.mta0.migadu.com ([91.218.175.185]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sD9e0-0000000BTMS-3z2P for linux-riscv@lists.infradead.org; Fri, 31 May 2024 21:18:39 +0000 X-Envelope-To: yuzhao@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717190302; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b4Xmi0701V1mOSvq8yprUubnI972AbFWvRE29wJdk78=; b=Ls3C5/08/280zRJqgHSiEjY525Nc+6AHW21VTOJinn6gvpWo5wGpHRdfxPSCAblARNzCTk Ef8yPQJkMIbUZrU0gLqIxyh5em+aap01i8qUkndBnnlYA3LWYTDaFt+T56XGowNC6n0gMV 1xZHPusSMv7Np2WADZqnpp/2B0dMfIM= X-Envelope-To: jthoughton@google.com X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Fri, 31 May 2024 21:18:12 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Yu Zhao Cc: James Houghton , Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Subject: Re: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240531_141837_312730_25A01275 X-CRM114-Status: GOOD ( 22.88 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gRnJpLCBNYXkgMzEsIDIwMjQgYXQgMDI6MzE6MTdQTSAtMDYwMCwgWXUgWmhhbyB3cm90ZToK PiBPbiBGcmksIE1heSAzMSwgMjAyNCBhdCAxOjI04oCvQU0gT2xpdmVyIFVwdG9uIDxvbGl2ZXIu dXB0b25AbGludXguZGV2PiB3cm90ZToKClsuLi5dCgo+ID4gR3JhYmJpbmcgdGhlIE1NVSBsb2Nr IGZvciB3cml0ZSB0byBzY2FuIHN1Y2tzLCBubyBhcmd1bWVudCB0aGVyZS4gQnV0Cj4gPiBjYW4g eW91IHBsZWFzZSBiZSBzcGVjaWZpYyBhYm91dCB0aGUgaW1wYWN0IG9mIHJlYWQgbG9jayB2LiBS Q1UgaW4gdGhlCj4gPiBjYXNlIG9mIGFybTY0PyBJIGhhZCBhc2tlZCBhYm91dCB0aGlzIGJlZm9y ZSBhbmQgeW91IG5ldmVyIHJlcGxpZWQuCj4gPgo+ID4gTXkgY29uY2VybiByZW1haW5zIHRoYXQg YWRkaW5nIHN1cHBvcnQgZm9yIHNvZnR3YXJlIHRhYmxlIHdhbGtlcnMKPiA+IG91dHNpZGUgb2Yg dGhlIE1NVSBsb2NrIGVudGlyZWx5IHJlcXVpcmVzIG1vcmUgd29yayB0aGFuIGp1c3QgZGVmZXJy aW5nCj4gPiB0aGUgZGVhbGxvY2F0aW9uIHRvIGFuIFJDVSBjYWxsYmFjay4gV2Fsa2VycyB0aGF0 IHByZXZpb3VzbHkgYXNzdW1lZAo+ID4gJ2V4Y2x1c2l2ZScgYWNjZXNzIHdoaWxlIGhvbGRpbmcg dGhlIE1NVSBsb2NrIGZvciB3cml0ZSBtdXN0IG5vdyBjb3BlCj4gPiB3aXRoIHZvbGF0aWxlIFBU RXMuCj4gPgo+ID4gWWVzLCB0aGlzIHByb2JsZW0gYWxyZWFkeSBleGlzdHMgd2hlbiBoYXJkd2Fy ZSBzZXRzIHRoZSBBRiwgYnV0IHRoZQo+ID4gbG9jay1mcmVlIHdhbGtlciBpbXBsZW1lbnRhdGlv biBuZWVkcyB0byBiZSBnZW5lcmljIHNvIGl0IGNhbiBiZSBhcHBsaWVkCj4gPiBmb3Igb3RoZXIg UFRFIGJpdHMuCj4gCj4gRGlyZWN0IHJlY2xhaW0gaXMgbXVsdGktdGhyZWFkZWQgYW5kIGVhY2gg cmVjbGFpbWVyIGNhbiB0YWtlIHRoZSBtbXUKPiBsb2NrIGZvciByZWFkICh0ZXN0aW5nIHRoZSBB LWJpdCkgb3Igd3JpdGUgKHVubWFwcGluZyBiZWZvcmUgcGFnaW5nCj4gb3V0KSBvbiBhcm02NC4g VGhlIGZ1bmRhbWVudGFsIHByb2JsZW0gb2YgdXNpbmcgdGhlIHJlYWRlcnMtd3JpdGVyCj4gbG9j ayBpbiB0aGlzIGNhc2UgaXMgcHJpb3JpdHkgaW52ZXJzaW9uOiB0aGUgcmVhZGVycyBoYXZlIGxv d2VyCj4gcHJpb3JpdHkgdGhhbiB0aGUgd3JpdGVycywgc28gaWRlYWxseSwgd2UgZG9uJ3Qgd2Fu dCB0aGUgcmVhZGVycyB0bwo+IGJsb2NrIHRoZSB3cml0ZXJzIGF0IGFsbC4KClNvIHdlIGFscmVh ZHkgaGF2ZSB0aGlzIHNvcnQgb2YgcHJvYmxlbSBvZiBzdGFnZS0yIGZhdWx0IGhhbmRsaW5nIHYu CnNlY29uZGFyeSBNTVUgaW52YWxpZGF0aW9ucywgd2hpY2ggaXMgd2h5IEkndmUgYmVlbiBkb3Vi dGZ1bCBvZiB0aGUKcGVyY2VpdmVkIGlzc3VlLiBJbiBmYWN0LCBJJ2QgYXJndWUgdGhhdCBuZWVk aW5nIHRvIHdhaXQgZm9yIGZhdWx0cyBpcwp3b3JzZSB0aGFuIGFnaW5nIHBhcnRpY2lwYXRpb24g c2luY2UgdGhvc2UgY2FuIGJlIHRyaXZpYWxseSBpbmZsdWVuY2VkCmJ5IHVzZXJzcGFjZS9ndWVz dC4KCkluIGFueSBjYXNlLCB3ZSBzaG91bGRuJ3QgZXZlciBiZSBzdGFydmVkIHNpbmNlIHlvdW5n ZXIgcmVhZGVycyBjYW5ub3QKZW50ZXIgdGhlIGNyaXRpY2FsIHNlY3Rpb24gd2l0aCBhIHBlbmRp bmcgd3JpdGVyLgoKPiBBcyBJIHNhaWQgZWFybGllciwgSSBwcmVmZXIgd2UgZHJvcCB0aGUgYXJt NjQgc3VwcG9ydCBmb3Igbm93LCBidXQgSQo+IHdpbGwgbm90IG9iamVjdCB0byB0YWtpbmcgdGhl IG1tdSBsb2NrIGZvciByZWFkIHdoZW4gY2xlYXJpbmcgdGhlCj4gQS1iaXQsIGFzIGxvbmcgYXMg d2UgZnVsbHkgdW5kZXJzdGFuZCB0aGUgcHJvYmxlbSBoZXJlIGFuZCBkb2N1bWVudCBpdAo+IGNs ZWFybHkuCgpJJ2QgYmUgY29udmluY2VkIG9mIHRoaXMgaWYgdGhlcmUncyBkYXRhIHRoYXQgc2hv d3MgcmVhZCBsb2NrCmFjcXVpc2l0aW9uIGlzIGluIGZhY3QgY29uc2VxdWVudGlhbC4gT3RoZXJ3 aXNlLCBJJ20gbm90IHN1cmUgdGhlIGFkZGVkCmNvbXBsZXhpdHkgb2YgUkNVIHRhYmxlIHdhbGtl cnMgKHBlciBteSBzdGF0ZW1lbnQgYWJvdmUpIGlzIHdvcnRoIHRoZQplZmZvcnQgLyBtYWludGVu YW5jZSBidXJkZW4uCgotLSAKVGhhbmtzLApPbGl2ZXIKCl9fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LXJpc2N2IG1haWxpbmcgbGlzdApsaW51eC1y aXNjdkBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxt YW4vbGlzdGluZm8vbGludXgtcmlzY3YK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E3AC9C25B75 for ; Fri, 31 May 2024 21:19:33 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=Ls3C5/08; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Vrbbm0Wlvz3frY for ; Sat, 1 Jun 2024 07:19:32 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=Ls3C5/08; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.dev (client-ip=91.218.175.182; helo=out-182.mta0.migadu.com; envelope-from=oliver.upton@linux.dev; receiver=lists.ozlabs.org) Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4VrbZv0vs6z3dWZ for ; Sat, 1 Jun 2024 07:18:44 +1000 (AEST) X-Envelope-To: yuzhao@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717190302; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b4Xmi0701V1mOSvq8yprUubnI972AbFWvRE29wJdk78=; b=Ls3C5/08/280zRJqgHSiEjY525Nc+6AHW21VTOJinn6gvpWo5wGpHRdfxPSCAblARNzCTk Ef8yPQJkMIbUZrU0gLqIxyh5em+aap01i8qUkndBnnlYA3LWYTDaFt+T56XGowNC6n0gMV 1xZHPusSMv7Np2WADZqnpp/2B0dMfIM= X-Envelope-To: jthoughton@google.com X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Fri, 31 May 2024 21:18:12 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Yu Zhao Subject: Re: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , kvm@vger.kernel.org, linux-doc@vger.kernel.org, Catalin Marinas , Atish Patra , linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-kselftest@vger.kernel.org, Raghavendra Rao Ananta , linux-riscv@lists.infradead.org, Shuah Khan , Jonathan Corbet , Anup Patel , Huacai Chen , David Rientjes , Zenghui Yu , Axel Rasmussen , linux-mips@vger.kernel.org, Albert Ou , Ryan Roberts , Will Deacon , Suzuki K Poulose , Shaoqin Huang , Nicholas Piggin , Bibo Mao , loongarch@lists.linux.dev, Paul Walmsley , David Matlack , Palmer Dabbelt , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, Sean Christopherson , Ankit Agrawal , James Morse , kvm-riscv@lists.infradead.org, Marc Zyngier , Paolo Bonzini , Andrew Morton , Tianrui Zhao , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Fri, May 31, 2024 at 02:31:17PM -0600, Yu Zhao wrote: > On Fri, May 31, 2024 at 1:24 AM Oliver Upton wrote: [...] > > Grabbing the MMU lock for write to scan sucks, no argument there. But > > can you please be specific about the impact of read lock v. RCU in the > > case of arm64? I had asked about this before and you never replied. > > > > My concern remains that adding support for software table walkers > > outside of the MMU lock entirely requires more work than just deferring > > the deallocation to an RCU callback. Walkers that previously assumed > > 'exclusive' access while holding the MMU lock for write must now cope > > with volatile PTEs. > > > > Yes, this problem already exists when hardware sets the AF, but the > > lock-free walker implementation needs to be generic so it can be applied > > for other PTE bits. > > Direct reclaim is multi-threaded and each reclaimer can take the mmu > lock for read (testing the A-bit) or write (unmapping before paging > out) on arm64. The fundamental problem of using the readers-writer > lock in this case is priority inversion: the readers have lower > priority than the writers, so ideally, we don't want the readers to > block the writers at all. So we already have this sort of problem of stage-2 fault handling v. secondary MMU invalidations, which is why I've been doubtful of the perceived issue. In fact, I'd argue that needing to wait for faults is worse than aging participation since those can be trivially influenced by userspace/guest. In any case, we shouldn't ever be starved since younger readers cannot enter the critical section with a pending writer. > As I said earlier, I prefer we drop the arm64 support for now, but I > will not object to taking the mmu lock for read when clearing the > A-bit, as long as we fully understand the problem here and document it > clearly. I'd be convinced of this if there's data that shows read lock acquisition is in fact consequential. Otherwise, I'm not sure the added complexity of RCU table walkers (per my statement above) is worth the effort / maintenance burden. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AFCC0C25B75 for ; Fri, 31 May 2024 21:18:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PxU6AzdKsatV59u/HuE19yu1Tg76jSwg8kib3NTL4GE=; b=wuRjySdS7ltVR8 d/Ady+SxiATyHA4+Gr/WcvLJQ+AOfK8V6wkIhL4Mrr2av0Ux/bVm5Y4klMHCL1PgYwAevQ1+A5xCv 6jtnNUmSnc3QAJAatk3A0PuJCbwOUB5BxHIYe1iGiA5XK4tlARBd9zJWrUj/wnPqtDf8D04ud3Ezz P8JxxXi0UM4KWGhZmtIQeKmb/U9H7hqc4r+w9UrTOcDvRpEStP4YpHkYRN/i43QytQIw2h/qyhEN7 jEB7eH/cA9wJZ31hHONV7B57KXlC5caiHOII4mPJF7jzQwcpMQLQviKQhpSsWNH8AvwG02frN4KxA 78KkD/9m8scqTH3yW3JQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sD9e0-0000000BTO3-3VQy; Fri, 31 May 2024 21:18:36 +0000 Received: from out-187.mta0.migadu.com ([91.218.175.187]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sD9dx-0000000BTMO-0dHy for linux-arm-kernel@lists.infradead.org; Fri, 31 May 2024 21:18:35 +0000 X-Envelope-To: yuzhao@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717190302; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b4Xmi0701V1mOSvq8yprUubnI972AbFWvRE29wJdk78=; b=Ls3C5/08/280zRJqgHSiEjY525Nc+6AHW21VTOJinn6gvpWo5wGpHRdfxPSCAblARNzCTk Ef8yPQJkMIbUZrU0gLqIxyh5em+aap01i8qUkndBnnlYA3LWYTDaFt+T56XGowNC6n0gMV 1xZHPusSMv7Np2WADZqnpp/2B0dMfIM= X-Envelope-To: jthoughton@google.com X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Fri, 31 May 2024 21:18:12 +0000 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Yu Zhao Cc: James Houghton , Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Subject: Re: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240531_141833_508707_0A840D2A X-CRM114-Status: GOOD ( 23.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gRnJpLCBNYXkgMzEsIDIwMjQgYXQgMDI6MzE6MTdQTSAtMDYwMCwgWXUgWmhhbyB3cm90ZToK PiBPbiBGcmksIE1heSAzMSwgMjAyNCBhdCAxOjI04oCvQU0gT2xpdmVyIFVwdG9uIDxvbGl2ZXIu dXB0b25AbGludXguZGV2PiB3cm90ZToKClsuLi5dCgo+ID4gR3JhYmJpbmcgdGhlIE1NVSBsb2Nr IGZvciB3cml0ZSB0byBzY2FuIHN1Y2tzLCBubyBhcmd1bWVudCB0aGVyZS4gQnV0Cj4gPiBjYW4g eW91IHBsZWFzZSBiZSBzcGVjaWZpYyBhYm91dCB0aGUgaW1wYWN0IG9mIHJlYWQgbG9jayB2LiBS Q1UgaW4gdGhlCj4gPiBjYXNlIG9mIGFybTY0PyBJIGhhZCBhc2tlZCBhYm91dCB0aGlzIGJlZm9y ZSBhbmQgeW91IG5ldmVyIHJlcGxpZWQuCj4gPgo+ID4gTXkgY29uY2VybiByZW1haW5zIHRoYXQg YWRkaW5nIHN1cHBvcnQgZm9yIHNvZnR3YXJlIHRhYmxlIHdhbGtlcnMKPiA+IG91dHNpZGUgb2Yg dGhlIE1NVSBsb2NrIGVudGlyZWx5IHJlcXVpcmVzIG1vcmUgd29yayB0aGFuIGp1c3QgZGVmZXJy aW5nCj4gPiB0aGUgZGVhbGxvY2F0aW9uIHRvIGFuIFJDVSBjYWxsYmFjay4gV2Fsa2VycyB0aGF0 IHByZXZpb3VzbHkgYXNzdW1lZAo+ID4gJ2V4Y2x1c2l2ZScgYWNjZXNzIHdoaWxlIGhvbGRpbmcg dGhlIE1NVSBsb2NrIGZvciB3cml0ZSBtdXN0IG5vdyBjb3BlCj4gPiB3aXRoIHZvbGF0aWxlIFBU RXMuCj4gPgo+ID4gWWVzLCB0aGlzIHByb2JsZW0gYWxyZWFkeSBleGlzdHMgd2hlbiBoYXJkd2Fy ZSBzZXRzIHRoZSBBRiwgYnV0IHRoZQo+ID4gbG9jay1mcmVlIHdhbGtlciBpbXBsZW1lbnRhdGlv biBuZWVkcyB0byBiZSBnZW5lcmljIHNvIGl0IGNhbiBiZSBhcHBsaWVkCj4gPiBmb3Igb3RoZXIg UFRFIGJpdHMuCj4gCj4gRGlyZWN0IHJlY2xhaW0gaXMgbXVsdGktdGhyZWFkZWQgYW5kIGVhY2gg cmVjbGFpbWVyIGNhbiB0YWtlIHRoZSBtbXUKPiBsb2NrIGZvciByZWFkICh0ZXN0aW5nIHRoZSBB LWJpdCkgb3Igd3JpdGUgKHVubWFwcGluZyBiZWZvcmUgcGFnaW5nCj4gb3V0KSBvbiBhcm02NC4g VGhlIGZ1bmRhbWVudGFsIHByb2JsZW0gb2YgdXNpbmcgdGhlIHJlYWRlcnMtd3JpdGVyCj4gbG9j ayBpbiB0aGlzIGNhc2UgaXMgcHJpb3JpdHkgaW52ZXJzaW9uOiB0aGUgcmVhZGVycyBoYXZlIGxv d2VyCj4gcHJpb3JpdHkgdGhhbiB0aGUgd3JpdGVycywgc28gaWRlYWxseSwgd2UgZG9uJ3Qgd2Fu dCB0aGUgcmVhZGVycyB0bwo+IGJsb2NrIHRoZSB3cml0ZXJzIGF0IGFsbC4KClNvIHdlIGFscmVh ZHkgaGF2ZSB0aGlzIHNvcnQgb2YgcHJvYmxlbSBvZiBzdGFnZS0yIGZhdWx0IGhhbmRsaW5nIHYu CnNlY29uZGFyeSBNTVUgaW52YWxpZGF0aW9ucywgd2hpY2ggaXMgd2h5IEkndmUgYmVlbiBkb3Vi dGZ1bCBvZiB0aGUKcGVyY2VpdmVkIGlzc3VlLiBJbiBmYWN0LCBJJ2QgYXJndWUgdGhhdCBuZWVk aW5nIHRvIHdhaXQgZm9yIGZhdWx0cyBpcwp3b3JzZSB0aGFuIGFnaW5nIHBhcnRpY2lwYXRpb24g c2luY2UgdGhvc2UgY2FuIGJlIHRyaXZpYWxseSBpbmZsdWVuY2VkCmJ5IHVzZXJzcGFjZS9ndWVz dC4KCkluIGFueSBjYXNlLCB3ZSBzaG91bGRuJ3QgZXZlciBiZSBzdGFydmVkIHNpbmNlIHlvdW5n ZXIgcmVhZGVycyBjYW5ub3QKZW50ZXIgdGhlIGNyaXRpY2FsIHNlY3Rpb24gd2l0aCBhIHBlbmRp bmcgd3JpdGVyLgoKPiBBcyBJIHNhaWQgZWFybGllciwgSSBwcmVmZXIgd2UgZHJvcCB0aGUgYXJt NjQgc3VwcG9ydCBmb3Igbm93LCBidXQgSQo+IHdpbGwgbm90IG9iamVjdCB0byB0YWtpbmcgdGhl IG1tdSBsb2NrIGZvciByZWFkIHdoZW4gY2xlYXJpbmcgdGhlCj4gQS1iaXQsIGFzIGxvbmcgYXMg d2UgZnVsbHkgdW5kZXJzdGFuZCB0aGUgcHJvYmxlbSBoZXJlIGFuZCBkb2N1bWVudCBpdAo+IGNs ZWFybHkuCgpJJ2QgYmUgY29udmluY2VkIG9mIHRoaXMgaWYgdGhlcmUncyBkYXRhIHRoYXQgc2hv d3MgcmVhZCBsb2NrCmFjcXVpc2l0aW9uIGlzIGluIGZhY3QgY29uc2VxdWVudGlhbC4gT3RoZXJ3 aXNlLCBJJ20gbm90IHN1cmUgdGhlIGFkZGVkCmNvbXBsZXhpdHkgb2YgUkNVIHRhYmxlIHdhbGtl cnMgKHBlciBteSBzdGF0ZW1lbnQgYWJvdmUpIGlzIHdvcnRoIHRoZQplZmZvcnQgLyBtYWludGVu YW5jZSBidXJkZW4uCgotLSAKVGhhbmtzLApPbGl2ZXIKCl9fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFpbGluZyBsaXN0Cmxp bnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFk Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK