From mboxrd@z Thu Jan 1 00:00:00 1970 From: Oliver Upton Date: Fri, 31 May 2024 00:24:18 -0700 Subject: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging In-Reply-To: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Wed, May 29, 2024 at 03:03:21PM -0600, Yu Zhao wrote: > On Wed, May 29, 2024 at 12:05?PM James Houghton wrote: > > > > Secondary MMUs are currently consulted for access/age information at > > eviction time, but before then, we don't get accurate age information. > > That is, pages that are mostly accessed through a secondary MMU (like > > guest memory, used by KVM) will always just proceed down to the oldest > > generation, and then at eviction time, if KVM reports the page to be > > young, the page will be activated/promoted back to the youngest > > generation. > > Correct, and as I explained offline, this is the only reasonable > behavior if we can't locklessly walk secondary MMUs. > > Just for the record, the (crude) analogy I used was: > Imagine a large room with many bills ($1, $5, $10, ...) on the floor, > but you are only allowed to pick up 10 of them (and put them in your > pocket). A smart move would be to survey the room *first and then* > pick up the largest ones. But if you are carrying a 500 lbs backpack, > you would just want to pick up whichever that's in front of you rather > than walk the entire room. > > MGLRU should only scan (or lookaround) secondary MMUs if it can be > done lockless. Otherwise, it should just fall back to the existing > approach, which existed in previous versions but is removed in this > version. Grabbing the MMU lock for write to scan sucks, no argument there. But can you please be specific about the impact of read lock v. RCU in the case of arm64? I had asked about this before and you never replied. My concern remains that adding support for software table walkers outside of the MMU lock entirely requires more work than just deferring the deallocation to an RCU callback. Walkers that previously assumed 'exclusive' access while holding the MMU lock for write must now cope with volatile PTEs. Yes, this problem already exists when hardware sets the AF, but the lock-free walker implementation needs to be generic so it can be applied for other PTE bits. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-173.mta0.migadu.com (out-173.mta0.migadu.com [91.218.175.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46E8D1CD11 for ; Fri, 31 May 2024 07:24:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717140277; cv=none; b=m4TWCeLhbd0zbpAcX6zidjl5un0va+LE3A1ejdDlfKKKJ3LKSsNfkD3k3waVNGfgum1VDNNliGWF2VJczybtdYaS/ToNrgXmsjc3pSX9NgFWGZfnkRF31CRkD9yqa9YIdHXXSpr4b5RESSP9c2svV0svmhqbnwi724TSB+i6288= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717140277; c=relaxed/simple; bh=TCC6im0osMHH/ZsvTuf9nvpEoA9dgsRlPGyvUIj8bLY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Ctl6thucfnDx/TKWXEr7wZMvMY2tql0ZfTqjl8v6lkI7MpaB0NXCkbf+RlfN1q5gV4oI8XYfICexUUajf20AfqhcUwEHowIR6V98yn7t4pYgXYhNbPViumSTz+YMFDNwlgRbOixACSJKplrtWU6BDblwhM5zTZANJa0WEnQ2KD4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=r37JI0j7; arc=none smtp.client-ip=91.218.175.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="r37JI0j7" X-Envelope-To: yuzhao@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717140273; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v6pyP4SRO/9BT5GrT52jbiQuxRO/wk0HtBPD4YgPhLk=; b=r37JI0j7taWeZmoSnTE82903GeLstAD8AcPnN/lPbb+eAAkY0defxPmiJPHwGWnd294WLb FjgHxx1Px9O8RnwjA1f9bxRoJnyH+BVooELFS2MSFgtlTifw+nQzg6kbOK9o4nC/oVDgtV trAyzjYJwoWDB3IsG5rivC2BXJjWLQk= X-Envelope-To: jthoughton@google.com X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Fri, 31 May 2024 00:24:18 -0700 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Yu Zhao Cc: James Houghton , Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Subject: Re: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT On Wed, May 29, 2024 at 03:03:21PM -0600, Yu Zhao wrote: > On Wed, May 29, 2024 at 12:05 PM James Houghton wrote: > > > > Secondary MMUs are currently consulted for access/age information at > > eviction time, but before then, we don't get accurate age information. > > That is, pages that are mostly accessed through a secondary MMU (like > > guest memory, used by KVM) will always just proceed down to the oldest > > generation, and then at eviction time, if KVM reports the page to be > > young, the page will be activated/promoted back to the youngest > > generation. > > Correct, and as I explained offline, this is the only reasonable > behavior if we can't locklessly walk secondary MMUs. > > Just for the record, the (crude) analogy I used was: > Imagine a large room with many bills ($1, $5, $10, ...) on the floor, > but you are only allowed to pick up 10 of them (and put them in your > pocket). A smart move would be to survey the room *first and then* > pick up the largest ones. But if you are carrying a 500 lbs backpack, > you would just want to pick up whichever that's in front of you rather > than walk the entire room. > > MGLRU should only scan (or lookaround) secondary MMUs if it can be > done lockless. Otherwise, it should just fall back to the existing > approach, which existed in previous versions but is removed in this > version. Grabbing the MMU lock for write to scan sucks, no argument there. But can you please be specific about the impact of read lock v. RCU in the case of arm64? I had asked about this before and you never replied. My concern remains that adding support for software table walkers outside of the MMU lock entirely requires more work than just deferring the deallocation to an RCU callback. Walkers that previously assumed 'exclusive' access while holding the MMU lock for write must now cope with volatile PTEs. Yes, this problem already exists when hardware sets the AF, but the lock-free walker implementation needs to be generic so it can be applied for other PTE bits. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 51C4FC27C44 for ; Fri, 31 May 2024 07:24:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ftSgDlwNoi0Y4KoOhrY38XYdcipvyOxdmX8qvA06Pmc=; b=2AJLjeajTctfPW phCSGerqhrER7+2ejrb/9GOZ56Ioc4FM+IGU+UXI+MAhWvoHWVBtBFFAUuSn2UITKHKnY+QECNXTT It7JIt7rDNwRLSEwI1K0JVbjvdoxybSs7Uo12VodWMhf7OLoKCdiRpHmpiZErPIZ0Oy02ywYH9ASu tAu2kCYswuRZQeaT++ENVQvKr126K2Nwqt/tafpHcwrsT193auHNkCGkzbb+Dhzi+8b22IPgcz1OF 3rxiHqbHWxQacjdhO55A+kgjeckkcZFfA3f9giN5cRIXxjgyAS48Fj+NPgnqpqC7bpphPyriojl7Z l2iANRG7V1D6qqhwKyRw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sCwd0-00000009UKr-31Jn; Fri, 31 May 2024 07:24:42 +0000 Received: from out-185.mta0.migadu.com ([91.218.175.185]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sCwcw-00000009UIr-2u74 for linux-riscv@lists.infradead.org; Fri, 31 May 2024 07:24:41 +0000 X-Envelope-To: yuzhao@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717140273; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v6pyP4SRO/9BT5GrT52jbiQuxRO/wk0HtBPD4YgPhLk=; b=r37JI0j7taWeZmoSnTE82903GeLstAD8AcPnN/lPbb+eAAkY0defxPmiJPHwGWnd294WLb FjgHxx1Px9O8RnwjA1f9bxRoJnyH+BVooELFS2MSFgtlTifw+nQzg6kbOK9o4nC/oVDgtV trAyzjYJwoWDB3IsG5rivC2BXJjWLQk= X-Envelope-To: jthoughton@google.com X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Fri, 31 May 2024 00:24:18 -0700 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Yu Zhao Cc: James Houghton , Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Subject: Re: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240531_002439_209080_85A579D1 X-CRM114-Status: GOOD ( 19.68 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gV2VkLCBNYXkgMjksIDIwMjQgYXQgMDM6MDM6MjFQTSAtMDYwMCwgWXUgWmhhbyB3cm90ZToK PiBPbiBXZWQsIE1heSAyOSwgMjAyNCBhdCAxMjowNeKAr1BNIEphbWVzIEhvdWdodG9uIDxqdGhv dWdodG9uQGdvb2dsZS5jb20+IHdyb3RlOgo+ID4KPiA+IFNlY29uZGFyeSBNTVVzIGFyZSBjdXJy ZW50bHkgY29uc3VsdGVkIGZvciBhY2Nlc3MvYWdlIGluZm9ybWF0aW9uIGF0Cj4gPiBldmljdGlv biB0aW1lLCBidXQgYmVmb3JlIHRoZW4sIHdlIGRvbid0IGdldCBhY2N1cmF0ZSBhZ2UgaW5mb3Jt YXRpb24uCj4gPiBUaGF0IGlzLCBwYWdlcyB0aGF0IGFyZSBtb3N0bHkgYWNjZXNzZWQgdGhyb3Vn aCBhIHNlY29uZGFyeSBNTVUgKGxpa2UKPiA+IGd1ZXN0IG1lbW9yeSwgdXNlZCBieSBLVk0pIHdp bGwgYWx3YXlzIGp1c3QgcHJvY2VlZCBkb3duIHRvIHRoZSBvbGRlc3QKPiA+IGdlbmVyYXRpb24s IGFuZCB0aGVuIGF0IGV2aWN0aW9uIHRpbWUsIGlmIEtWTSByZXBvcnRzIHRoZSBwYWdlIHRvIGJl Cj4gPiB5b3VuZywgdGhlIHBhZ2Ugd2lsbCBiZSBhY3RpdmF0ZWQvcHJvbW90ZWQgYmFjayB0byB0 aGUgeW91bmdlc3QKPiA+IGdlbmVyYXRpb24uCj4gCj4gQ29ycmVjdCwgYW5kIGFzIEkgZXhwbGFp bmVkIG9mZmxpbmUsIHRoaXMgaXMgdGhlIG9ubHkgcmVhc29uYWJsZQo+IGJlaGF2aW9yIGlmIHdl IGNhbid0IGxvY2tsZXNzbHkgd2FsayBzZWNvbmRhcnkgTU1Vcy4KPiAKPiBKdXN0IGZvciB0aGUg cmVjb3JkLCB0aGUgKGNydWRlKSBhbmFsb2d5IEkgdXNlZCB3YXM6Cj4gSW1hZ2luZSBhIGxhcmdl IHJvb20gd2l0aCBtYW55IGJpbGxzICgkMSwgJDUsICQxMCwgLi4uKSBvbiB0aGUgZmxvb3IsCj4g YnV0IHlvdSBhcmUgb25seSBhbGxvd2VkIHRvIHBpY2sgdXAgMTAgb2YgdGhlbSAoYW5kIHB1dCB0 aGVtIGluIHlvdXIKPiBwb2NrZXQpLiBBIHNtYXJ0IG1vdmUgd291bGQgYmUgdG8gc3VydmV5IHRo ZSByb29tICpmaXJzdCBhbmQgdGhlbioKPiBwaWNrIHVwIHRoZSBsYXJnZXN0IG9uZXMuIEJ1dCBp ZiB5b3UgYXJlIGNhcnJ5aW5nIGEgNTAwIGxicyBiYWNrcGFjaywKPiB5b3Ugd291bGQganVzdCB3 YW50IHRvIHBpY2sgdXAgd2hpY2hldmVyIHRoYXQncyBpbiBmcm9udCBvZiB5b3UgcmF0aGVyCj4g dGhhbiB3YWxrIHRoZSBlbnRpcmUgcm9vbS4KPiAKPiBNR0xSVSBzaG91bGQgb25seSBzY2FuIChv ciBsb29rYXJvdW5kKSBzZWNvbmRhcnkgTU1VcyBpZiBpdCBjYW4gYmUKPiBkb25lIGxvY2tsZXNz LiBPdGhlcndpc2UsIGl0IHNob3VsZCBqdXN0IGZhbGwgYmFjayB0byB0aGUgZXhpc3RpbmcKPiBh cHByb2FjaCwgd2hpY2ggZXhpc3RlZCBpbiBwcmV2aW91cyB2ZXJzaW9ucyBidXQgaXMgcmVtb3Zl ZCBpbiB0aGlzCj4gdmVyc2lvbi4KCkdyYWJiaW5nIHRoZSBNTVUgbG9jayBmb3Igd3JpdGUgdG8g c2NhbiBzdWNrcywgbm8gYXJndW1lbnQgdGhlcmUuIEJ1dApjYW4geW91IHBsZWFzZSBiZSBzcGVj aWZpYyBhYm91dCB0aGUgaW1wYWN0IG9mIHJlYWQgbG9jayB2LiBSQ1UgaW4gdGhlCmNhc2Ugb2Yg YXJtNjQ/IEkgaGFkIGFza2VkIGFib3V0IHRoaXMgYmVmb3JlIGFuZCB5b3UgbmV2ZXIgcmVwbGll ZC4KCk15IGNvbmNlcm4gcmVtYWlucyB0aGF0IGFkZGluZyBzdXBwb3J0IGZvciBzb2Z0d2FyZSB0 YWJsZSB3YWxrZXJzCm91dHNpZGUgb2YgdGhlIE1NVSBsb2NrIGVudGlyZWx5IHJlcXVpcmVzIG1v cmUgd29yayB0aGFuIGp1c3QgZGVmZXJyaW5nCnRoZSBkZWFsbG9jYXRpb24gdG8gYW4gUkNVIGNh bGxiYWNrLiBXYWxrZXJzIHRoYXQgcHJldmlvdXNseSBhc3N1bWVkCidleGNsdXNpdmUnIGFjY2Vz cyB3aGlsZSBob2xkaW5nIHRoZSBNTVUgbG9jayBmb3Igd3JpdGUgbXVzdCBub3cgY29wZQp3aXRo IHZvbGF0aWxlIFBURXMuCgpZZXMsIHRoaXMgcHJvYmxlbSBhbHJlYWR5IGV4aXN0cyB3aGVuIGhh cmR3YXJlIHNldHMgdGhlIEFGLCBidXQgdGhlCmxvY2stZnJlZSB3YWxrZXIgaW1wbGVtZW50YXRp b24gbmVlZHMgdG8gYmUgZ2VuZXJpYyBzbyBpdCBjYW4gYmUgYXBwbGllZApmb3Igb3RoZXIgUFRF IGJpdHMuCgotLSAKVGhhbmtzLApPbGl2ZXIKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fCmxpbnV4LXJpc2N2IG1haWxpbmcgbGlzdApsaW51eC1yaXNjdkBs aXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlz dGluZm8vbGludXgtcmlzY3YK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 44C19C25B75 for ; Fri, 31 May 2024 07:25:45 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=r37JI0j7; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4VrF5g2JKXz2yN3 for ; Fri, 31 May 2024 17:25:43 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=r37JI0j7; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.dev (client-ip=91.218.175.182; helo=out-182.mta0.migadu.com; envelope-from=oliver.upton@linux.dev; receiver=lists.ozlabs.org) Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4VrF4n0CWyz30Ts for ; Fri, 31 May 2024 17:24:55 +1000 (AEST) X-Envelope-To: yuzhao@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717140273; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v6pyP4SRO/9BT5GrT52jbiQuxRO/wk0HtBPD4YgPhLk=; b=r37JI0j7taWeZmoSnTE82903GeLstAD8AcPnN/lPbb+eAAkY0defxPmiJPHwGWnd294WLb FjgHxx1Px9O8RnwjA1f9bxRoJnyH+BVooELFS2MSFgtlTifw+nQzg6kbOK9o4nC/oVDgtV trAyzjYJwoWDB3IsG5rivC2BXJjWLQk= X-Envelope-To: jthoughton@google.com X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Fri, 31 May 2024 00:24:18 -0700 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Yu Zhao Subject: Re: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , kvm@vger.kernel.org, linux-doc@vger.kernel.org, Catalin Marinas , Atish Patra , linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-kselftest@vger.kernel.org, Raghavendra Rao Ananta , linux-riscv@lists.infradead.org, Shuah Khan , Jonathan Corbet , Anup Patel , Huacai Chen , David Rientjes , Zenghui Yu , Axel Rasmussen , linux-mips@vger.kernel.org, Albert Ou , Ryan Roberts , Will Deacon , Suzuki K Poulose , Shaoqin Huang , Nicholas Piggin , Bibo Mao , loongarch@lists.linux.dev, Paul Walmsley , David Matlack , Palmer Dabbelt , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, Sean Christopherson , Ankit Agrawal , James Morse , kvm-riscv@lists.infradead.org, Marc Zyngier , Paolo Bonzini , Andrew Morton , Tianrui Zhao , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Wed, May 29, 2024 at 03:03:21PM -0600, Yu Zhao wrote: > On Wed, May 29, 2024 at 12:05 PM James Houghton wrote: > > > > Secondary MMUs are currently consulted for access/age information at > > eviction time, but before then, we don't get accurate age information. > > That is, pages that are mostly accessed through a secondary MMU (like > > guest memory, used by KVM) will always just proceed down to the oldest > > generation, and then at eviction time, if KVM reports the page to be > > young, the page will be activated/promoted back to the youngest > > generation. > > Correct, and as I explained offline, this is the only reasonable > behavior if we can't locklessly walk secondary MMUs. > > Just for the record, the (crude) analogy I used was: > Imagine a large room with many bills ($1, $5, $10, ...) on the floor, > but you are only allowed to pick up 10 of them (and put them in your > pocket). A smart move would be to survey the room *first and then* > pick up the largest ones. But if you are carrying a 500 lbs backpack, > you would just want to pick up whichever that's in front of you rather > than walk the entire room. > > MGLRU should only scan (or lookaround) secondary MMUs if it can be > done lockless. Otherwise, it should just fall back to the existing > approach, which existed in previous versions but is removed in this > version. Grabbing the MMU lock for write to scan sucks, no argument there. But can you please be specific about the impact of read lock v. RCU in the case of arm64? I had asked about this before and you never replied. My concern remains that adding support for software table walkers outside of the MMU lock entirely requires more work than just deferring the deallocation to an RCU callback. Walkers that previously assumed 'exclusive' access while holding the MMU lock for write must now cope with volatile PTEs. Yes, this problem already exists when hardware sets the AF, but the lock-free walker implementation needs to be generic so it can be applied for other PTE bits. -- Thanks, Oliver From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 462F9C25B75 for ; Fri, 31 May 2024 07:24:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QS84xXVd1fdHUZnAbBt+utyRJnhPBz94kxvUi8UIh4U=; b=bg07di3IAuSAIl ZBFTY3cPmAJahoZRtFu2G6ZwNyDGs98nevgre7Ufn/MS+Zpv27aOhYdckIImRw4gidzOYMcdNedNk 4oFeyLFloBoxO0IxBIEMyaTLq582xOI1+T8DNFvrqUFPyn8+aurnfOJDaAFI8q+ledvMWucq/mk6z 3RgS1J49lb9AnqnPwcxitow48SPyuZAD+Qk04lSvX2IOX4MkcW0QqWtP8YNe+Yr9SlsExTBZbs2Qw NYEn+F/8N6MyLlcgfpEjirkpZEaplu7ZxNlLX7ETjruQ7U9Ed150GecfZWjyhWJP8tNHCF+LAi7eC uSi3SAUM+B4XRkF+UGZg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sCwd1-00000009ULE-3iHi; Fri, 31 May 2024 07:24:43 +0000 Received: from out-171.mta0.migadu.com ([2001:41d0:1004:224b::ab]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sCwcw-00000009UIp-2zWG for linux-arm-kernel@lists.infradead.org; Fri, 31 May 2024 07:24:42 +0000 X-Envelope-To: yuzhao@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717140273; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v6pyP4SRO/9BT5GrT52jbiQuxRO/wk0HtBPD4YgPhLk=; b=r37JI0j7taWeZmoSnTE82903GeLstAD8AcPnN/lPbb+eAAkY0defxPmiJPHwGWnd294WLb FjgHxx1Px9O8RnwjA1f9bxRoJnyH+BVooELFS2MSFgtlTifw+nQzg6kbOK9o4nC/oVDgtV trAyzjYJwoWDB3IsG5rivC2BXJjWLQk= X-Envelope-To: jthoughton@google.com X-Envelope-To: akpm@linux-foundation.org X-Envelope-To: pbonzini@redhat.com X-Envelope-To: aou@eecs.berkeley.edu X-Envelope-To: ankita@nvidia.com X-Envelope-To: anup@brainfault.org X-Envelope-To: atishp@atishpatra.org X-Envelope-To: axelrasmussen@google.com X-Envelope-To: maobibo@loongson.cn X-Envelope-To: catalin.marinas@arm.com X-Envelope-To: dmatlack@google.com X-Envelope-To: rientjes@google.com X-Envelope-To: chenhuacai@kernel.org X-Envelope-To: james.morse@arm.com X-Envelope-To: corbet@lwn.net X-Envelope-To: maz@kernel.org X-Envelope-To: mpe@ellerman.id.au X-Envelope-To: npiggin@gmail.com X-Envelope-To: palmer@dabbelt.com X-Envelope-To: paul.walmsley@sifive.com X-Envelope-To: rananta@google.com X-Envelope-To: ryan.roberts@arm.com X-Envelope-To: seanjc@google.com X-Envelope-To: shahuang@redhat.com X-Envelope-To: shuah@kernel.org X-Envelope-To: suzuki.poulose@arm.com X-Envelope-To: zhaotianrui@loongson.cn X-Envelope-To: will@kernel.org X-Envelope-To: yuzenghui@huawei.com X-Envelope-To: kvm-riscv@lists.infradead.org X-Envelope-To: kvm@vger.kernel.org X-Envelope-To: kvmarm@lists.linux.dev X-Envelope-To: linux-arm-kernel@lists.infradead.org X-Envelope-To: linux-doc@vger.kernel.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linux-kselftest@vger.kernel.org X-Envelope-To: linux-mips@vger.kernel.org X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-riscv@lists.infradead.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: loongarch@lists.linux.dev Date: Fri, 31 May 2024 00:24:18 -0700 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Oliver Upton To: Yu Zhao Cc: James Houghton , Andrew Morton , Paolo Bonzini , Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev Subject: Re: [PATCH v4 2/7] mm: multi-gen LRU: Have secondary MMUs participate in aging Message-ID: References: <20240529180510.2295118-1-jthoughton@google.com> <20240529180510.2295118-3-jthoughton@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240531_002439_226889_1D1F539F X-CRM114-Status: GOOD ( 20.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkLCBNYXkgMjksIDIwMjQgYXQgMDM6MDM6MjFQTSAtMDYwMCwgWXUgWmhhbyB3cm90ZToK PiBPbiBXZWQsIE1heSAyOSwgMjAyNCBhdCAxMjowNeKAr1BNIEphbWVzIEhvdWdodG9uIDxqdGhv dWdodG9uQGdvb2dsZS5jb20+IHdyb3RlOgo+ID4KPiA+IFNlY29uZGFyeSBNTVVzIGFyZSBjdXJy ZW50bHkgY29uc3VsdGVkIGZvciBhY2Nlc3MvYWdlIGluZm9ybWF0aW9uIGF0Cj4gPiBldmljdGlv biB0aW1lLCBidXQgYmVmb3JlIHRoZW4sIHdlIGRvbid0IGdldCBhY2N1cmF0ZSBhZ2UgaW5mb3Jt YXRpb24uCj4gPiBUaGF0IGlzLCBwYWdlcyB0aGF0IGFyZSBtb3N0bHkgYWNjZXNzZWQgdGhyb3Vn aCBhIHNlY29uZGFyeSBNTVUgKGxpa2UKPiA+IGd1ZXN0IG1lbW9yeSwgdXNlZCBieSBLVk0pIHdp bGwgYWx3YXlzIGp1c3QgcHJvY2VlZCBkb3duIHRvIHRoZSBvbGRlc3QKPiA+IGdlbmVyYXRpb24s IGFuZCB0aGVuIGF0IGV2aWN0aW9uIHRpbWUsIGlmIEtWTSByZXBvcnRzIHRoZSBwYWdlIHRvIGJl Cj4gPiB5b3VuZywgdGhlIHBhZ2Ugd2lsbCBiZSBhY3RpdmF0ZWQvcHJvbW90ZWQgYmFjayB0byB0 aGUgeW91bmdlc3QKPiA+IGdlbmVyYXRpb24uCj4gCj4gQ29ycmVjdCwgYW5kIGFzIEkgZXhwbGFp bmVkIG9mZmxpbmUsIHRoaXMgaXMgdGhlIG9ubHkgcmVhc29uYWJsZQo+IGJlaGF2aW9yIGlmIHdl IGNhbid0IGxvY2tsZXNzbHkgd2FsayBzZWNvbmRhcnkgTU1Vcy4KPiAKPiBKdXN0IGZvciB0aGUg cmVjb3JkLCB0aGUgKGNydWRlKSBhbmFsb2d5IEkgdXNlZCB3YXM6Cj4gSW1hZ2luZSBhIGxhcmdl IHJvb20gd2l0aCBtYW55IGJpbGxzICgkMSwgJDUsICQxMCwgLi4uKSBvbiB0aGUgZmxvb3IsCj4g YnV0IHlvdSBhcmUgb25seSBhbGxvd2VkIHRvIHBpY2sgdXAgMTAgb2YgdGhlbSAoYW5kIHB1dCB0 aGVtIGluIHlvdXIKPiBwb2NrZXQpLiBBIHNtYXJ0IG1vdmUgd291bGQgYmUgdG8gc3VydmV5IHRo ZSByb29tICpmaXJzdCBhbmQgdGhlbioKPiBwaWNrIHVwIHRoZSBsYXJnZXN0IG9uZXMuIEJ1dCBp ZiB5b3UgYXJlIGNhcnJ5aW5nIGEgNTAwIGxicyBiYWNrcGFjaywKPiB5b3Ugd291bGQganVzdCB3 YW50IHRvIHBpY2sgdXAgd2hpY2hldmVyIHRoYXQncyBpbiBmcm9udCBvZiB5b3UgcmF0aGVyCj4g dGhhbiB3YWxrIHRoZSBlbnRpcmUgcm9vbS4KPiAKPiBNR0xSVSBzaG91bGQgb25seSBzY2FuIChv ciBsb29rYXJvdW5kKSBzZWNvbmRhcnkgTU1VcyBpZiBpdCBjYW4gYmUKPiBkb25lIGxvY2tsZXNz LiBPdGhlcndpc2UsIGl0IHNob3VsZCBqdXN0IGZhbGwgYmFjayB0byB0aGUgZXhpc3RpbmcKPiBh cHByb2FjaCwgd2hpY2ggZXhpc3RlZCBpbiBwcmV2aW91cyB2ZXJzaW9ucyBidXQgaXMgcmVtb3Zl ZCBpbiB0aGlzCj4gdmVyc2lvbi4KCkdyYWJiaW5nIHRoZSBNTVUgbG9jayBmb3Igd3JpdGUgdG8g c2NhbiBzdWNrcywgbm8gYXJndW1lbnQgdGhlcmUuIEJ1dApjYW4geW91IHBsZWFzZSBiZSBzcGVj aWZpYyBhYm91dCB0aGUgaW1wYWN0IG9mIHJlYWQgbG9jayB2LiBSQ1UgaW4gdGhlCmNhc2Ugb2Yg YXJtNjQ/IEkgaGFkIGFza2VkIGFib3V0IHRoaXMgYmVmb3JlIGFuZCB5b3UgbmV2ZXIgcmVwbGll ZC4KCk15IGNvbmNlcm4gcmVtYWlucyB0aGF0IGFkZGluZyBzdXBwb3J0IGZvciBzb2Z0d2FyZSB0 YWJsZSB3YWxrZXJzCm91dHNpZGUgb2YgdGhlIE1NVSBsb2NrIGVudGlyZWx5IHJlcXVpcmVzIG1v cmUgd29yayB0aGFuIGp1c3QgZGVmZXJyaW5nCnRoZSBkZWFsbG9jYXRpb24gdG8gYW4gUkNVIGNh bGxiYWNrLiBXYWxrZXJzIHRoYXQgcHJldmlvdXNseSBhc3N1bWVkCidleGNsdXNpdmUnIGFjY2Vz cyB3aGlsZSBob2xkaW5nIHRoZSBNTVUgbG9jayBmb3Igd3JpdGUgbXVzdCBub3cgY29wZQp3aXRo IHZvbGF0aWxlIFBURXMuCgpZZXMsIHRoaXMgcHJvYmxlbSBhbHJlYWR5IGV4aXN0cyB3aGVuIGhh cmR3YXJlIHNldHMgdGhlIEFGLCBidXQgdGhlCmxvY2stZnJlZSB3YWxrZXIgaW1wbGVtZW50YXRp b24gbmVlZHMgdG8gYmUgZ2VuZXJpYyBzbyBpdCBjYW4gYmUgYXBwbGllZApmb3Igb3RoZXIgUFRF IGJpdHMuCgotLSAKVGhhbmtzLApPbGl2ZXIKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFy bS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9t YWlsbWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK