From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 54E48EA4; Sun, 25 Jun 2023 17:44:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8AC02C433C8; Sun, 25 Jun 2023 17:44:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687715063; bh=WoUrmf0JEvFfQkljC61rPZoFHAXMId+ovOZ+fz1xXRM=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=YP5riMELjwGif5MetoqunCE/NEQNM05SODCHCM3NP274NT5kLDkCw+lYlFK9UhEOT qDLdn9rWmrAiPScC30o8bBxNq06ndu4LyRdafclIFEHnyh1dl1m6lm8HKVRj0PW8Gd bS1NGRYWfoyju/EEsTN0Hfg++5BGw6JBu3tNKZMoh+4lxQ/InBdA1cnWhewfz/DpH3 NQvBcro2ROnD4F+sChNda583y6v06vsiiNksZdiHaJOziCCpsjvIYI5dguVCw3W8jl /7gz0Aq4l2D8Ub/ROd+l5Cg2ih7cqqiikyhGIYUqney6omj2WYc5s+RTv4euyPzP/E NLShZkWcrywjw== Date: Sun, 25 Jun 2023 20:42:57 +0300 From: Mike Rapoport To: Andy Lutomirski Cc: Mark Rutland , Kees Cook , Linux Kernel Mailing List , Andrew Morton , Catalin Marinas , Christophe Leroy , "David S. Miller" , Dinh Nguyen , Heiko Carstens , Helge Deller , Huacai Chen , Kent Overstreet , Luis Chamberlain , Michael Ellerman , Nadav Amit , "Naveen N. Rao" , Palmer Dabbelt , Puranjay Mohan , Rick P Edgecombe , "Russell King (Oracle)" , Song Liu , Steven Rostedt , Thomas Bogendoerfer , Thomas Gleixner , Will Deacon , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev, netdev@vger.kernel.org, sparclinux@vger.kernel.org, the arch/x86 maintainers Subject: Re: [PATCH v2 02/12] mm: introduce execmem_text_alloc() and jit_text_alloc() Message-ID: <20230625174257.GL52412@kernel.org> References: <20230616085038.4121892-1-rppt@kernel.org> <20230616085038.4121892-3-rppt@kernel.org> <20230618080027.GA52412@kernel.org> <20230625161417.GK52412@kernel.org> <90161ac9-3ca0-4c72-b1c4-ab1293e55445@app.fastmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <90161ac9-3ca0-4c72-b1c4-ab1293e55445@app.fastmail.com> On Sun, Jun 25, 2023 at 09:59:34AM -0700, Andy Lutomirski wrote: > > > On Sun, Jun 25, 2023, at 9:14 AM, Mike Rapoport wrote: > > On Mon, Jun 19, 2023 at 10:09:02AM -0700, Andy Lutomirski wrote: > >> > >> On Sun, Jun 18, 2023, at 1:00 AM, Mike Rapoport wrote: > >> > On Sat, Jun 17, 2023 at 01:38:29PM -0700, Andy Lutomirski wrote: > >> >> On Fri, Jun 16, 2023, at 1:50 AM, Mike Rapoport wrote: > >> >> > From: "Mike Rapoport (IBM)" > >> >> > > >> >> > module_alloc() is used everywhere as a mean to allocate memory for code. > >> >> > > >> >> > Beside being semantically wrong, this unnecessarily ties all subsystems > >> >> > that need to allocate code, such as ftrace, kprobes and BPF to modules > >> >> > and puts the burden of code allocation to the modules code. > >> >> > > >> >> > Several architectures override module_alloc() because of various > >> >> > constraints where the executable memory can be located and this causes > >> >> > additional obstacles for improvements of code allocation. > >> >> > > >> >> > Start splitting code allocation from modules by introducing > >> >> > execmem_text_alloc(), execmem_free(), jit_text_alloc(), jit_free() APIs. > >> >> > > >> >> > Initially, execmem_text_alloc() and jit_text_alloc() are wrappers for > >> >> > module_alloc() and execmem_free() and jit_free() are replacements of > >> >> > module_memfree() to allow updating all call sites to use the new APIs. > >> >> > > >> >> > The intention semantics for new allocation APIs: > >> >> > > >> >> > * execmem_text_alloc() should be used to allocate memory that must reside > >> >> > close to the kernel image, like loadable kernel modules and generated > >> >> > code that is restricted by relative addressing. > >> >> > > >> >> > * jit_text_alloc() should be used to allocate memory for generated code > >> >> > when there are no restrictions for the code placement. For > >> >> > architectures that require that any code is within certain distance > >> >> > from the kernel image, jit_text_alloc() will be essentially aliased to > >> >> > execmem_text_alloc(). > >> >> > > >> >> > >> >> Is there anything in this series to help users do the appropriate > >> >> synchronization when the actually populate the allocated memory with > >> >> code? See here, for example: > >> > > >> > This series only factors out the executable allocations from modules and > >> > puts them in a central place. > >> > Anything else would go on top after this lands. > >> > >> Hmm. > >> > >> On the one hand, there's nothing wrong with factoring out common code. On > >> the other hand, this is probably the right time to at least start > >> thinking about synchronization, at least to the extent that it might make > >> us want to change this API. (I'm not at all saying that this series > >> should require changes -- I'm just saying that this is a good time to > >> think about how this should work.) > >> > >> The current APIs, *and* the proposed jit_text_alloc() API, don't actually > >> look like the one think in the Linux ecosystem that actually > >> intelligently and efficiently maps new text into an address space: > >> mmap(). > >> > >> On x86, you can mmap() an existing file full of executable code PROT_EXEC > >> and jump to it with minimal synchronization (just the standard implicit > >> ordering in the kernel that populates the pages before setting up the > >> PTEs and whatever user synchronization is needed to avoid jumping into > >> the mapping before mmap() finishes). It works across CPUs, and the only > >> possible way userspace can screw it up (for a read-only mapping of > >> read-only text, anyway) is to jump to the mapping too early, in which > >> case userspace gets a page fault. Incoherence is impossible, and no one > >> needs to "serialize" (in the SDM sense). > >> > >> I think the same sequence (from userspace's perspective) works on other > >> architectures, too, although I think more cache management is needed on > >> the kernel's end. As far as I know, no Linux SMP architecture needs an > >> IPI to map executable text into usermode, but I could easily be wrong. > >> (IIRC RISC-V has very developer-unfriendly icache management, but I don't > >> remember the details.) > >> > >> Of course, using ptrace or any other FOLL_FORCE to modify text on x86 is > >> rather fraught, and I bet many things do it wrong when userspace is > >> multithreaded. But not in production because it's mostly not used in > >> production.) > >> > >> But jit_text_alloc() can't do this, because the order of operations > >> doesn't match. With jit_text_alloc(), the executable mapping shows up > >> before the text is populated, so there is no atomic change from not-there > >> to populated-and-executable. Which means that there is an opportunity > >> for CPUs, speculatively or otherwise, to start filling various caches > >> with intermediate states of the text, which means that various > >> architectures (even x86!) may need serialization. > >> > >> For eBPF- and module- like use cases, where JITting/code gen is quite > >> coarse-grained, perhaps something vaguely like: > >> > >> jit_text_alloc() -> returns a handle and an executable virtual address, > >> but does *not* map it there > >> jit_text_write() -> write to that handle > >> jit_text_map() -> map it and synchronize if needed (no sync needed on > >> x86, I think) > >> > >> could be more efficient and/or safer. > >> > >> (Modules could use this too. Getting alternatives right might take some > >> fiddling, because off the top of my head, this doesn't match how it works > >> now.) > >> > >> To make alternatives easier, this could work, maybe (haven't fully > >> thought it through): > >> > >> jit_text_alloc() > >> jit_text_map_rw_inplace() -> map at the target address, but RW, !X > >> > >> write the text and apply alternatives > >> > >> jit_text_finalize() -> change from RW to RX *and synchronize* > >> > >> jit_text_finalize() would either need to wait for RCU (possibly extra > >> heavy weight RCU to get "serialization") or send an IPI. > > > > This essentially how modules work now. The memory is allocated RW, written > > and updated with alternatives and then made ROX in the end with set_memory > > APIs. > > > > The issue with not having the memory mapped X when it's written is that we > > cannot use large pages to map it. One of the goals is to have executable > > memory mapped with large pages and make code allocator able to divide that > > page among several callers. > > > > So the idea was that jit_text_alloc() will have a cache of large pages > > mapped ROX, will allocate memory from those caches and there will be > > jit_update() that uses text poking for writing to that memory. > > > > Upon allocation of a large page to increase the cache, that large page will > > be "invalidated" by filling it with breakpoint instructions (e.g int3 on > > x86) > > Is this actually valid? In between int3 and real code, there’s a > potential torn read of real code mixed up with 0xcc. You mean while doing text poking? > > To improve the performance of this process, we can write to !X copy and > > then text_poke it to the actual address in one go. This will require some > > changes to get the alternatives right. > > > > -- > > Sincerely yours, > > Mike. -- Sincerely yours, Mike. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6B1FEB64DC for ; Sun, 25 Jun 2023 17:44:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IKWcpX/gKKXsxTuPSmBOe4I0RrIvNR4xKW9xuqGleXI=; b=el6cs+ubNr5Ylf kPc+o7Itdjtd77aFEj6kbRN5ktSNru1x3K7baxphQLOx0awJFhi23CAzW4mY48MdN93J8Q2QGQW/t 6whgoFPdO/E8KbF7arZIbnWI8SmY+TOQMzfIMnteDGj5HylGRHfH+aF7h/x4LzC+niV6qJnyRMfYx lipIZYuigXDYb7ea6oKtYmnxbrvLah/ceLyEw8tMxMRXcM9K9qtLDvBvMeJhepEOxMu0Gc+uD0O1s 6T6HtksB2JPvIdGXn2HxYiwzLSZlp1mC6aOGmQrDtEjeezEJqVlBNZnhRDX4GklaxO9oN/JjO6iSp KRd5en8t1gm+FKUb3+Mg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qDTmq-008OeF-1c; Sun, 25 Jun 2023 17:44:32 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qDTmj-008Obd-0z; Sun, 25 Jun 2023 17:44:27 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5A3C1602F9; Sun, 25 Jun 2023 17:44:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8AC02C433C8; Sun, 25 Jun 2023 17:44:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687715063; bh=WoUrmf0JEvFfQkljC61rPZoFHAXMId+ovOZ+fz1xXRM=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=YP5riMELjwGif5MetoqunCE/NEQNM05SODCHCM3NP274NT5kLDkCw+lYlFK9UhEOT qDLdn9rWmrAiPScC30o8bBxNq06ndu4LyRdafclIFEHnyh1dl1m6lm8HKVRj0PW8Gd bS1NGRYWfoyju/EEsTN0Hfg++5BGw6JBu3tNKZMoh+4lxQ/InBdA1cnWhewfz/DpH3 NQvBcro2ROnD4F+sChNda583y6v06vsiiNksZdiHaJOziCCpsjvIYI5dguVCw3W8jl /7gz0Aq4l2D8Ub/ROd+l5Cg2ih7cqqiikyhGIYUqney6omj2WYc5s+RTv4euyPzP/E NLShZkWcrywjw== Date: Sun, 25 Jun 2023 20:42:57 +0300 From: Mike Rapoport To: Andy Lutomirski Cc: Mark Rutland , Kees Cook , Linux Kernel Mailing List , Andrew Morton , Catalin Marinas , Christophe Leroy , "David S. Miller" , Dinh Nguyen , Heiko Carstens , Helge Deller , Huacai Chen , Kent Overstreet , Luis Chamberlain , Michael Ellerman , Nadav Amit , "Naveen N. Rao" , Palmer Dabbelt , Puranjay Mohan , Rick P Edgecombe , "Russell King (Oracle)" , Song Liu , Steven Rostedt , Thomas Bogendoerfer , Thomas Gleixner , Will Deacon , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev, netdev@vger.kernel.org, sparclinux@vger.kernel.org, the arch/x86 maintainers Subject: Re: [PATCH v2 02/12] mm: introduce execmem_text_alloc() and jit_text_alloc() Message-ID: <20230625174257.GL52412@kernel.org> References: <20230616085038.4121892-1-rppt@kernel.org> <20230616085038.4121892-3-rppt@kernel.org> <20230618080027.GA52412@kernel.org> <20230625161417.GK52412@kernel.org> <90161ac9-3ca0-4c72-b1c4-ab1293e55445@app.fastmail.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <90161ac9-3ca0-4c72-b1c4-ab1293e55445@app.fastmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230625_104425_438099_D9B5D343 X-CRM114-Status: GOOD ( 60.80 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gU3VuLCBKdW4gMjUsIDIwMjMgYXQgMDk6NTk6MzRBTSAtMDcwMCwgQW5keSBMdXRvbWlyc2tp IHdyb3RlOgo+IAo+IAo+IE9uIFN1biwgSnVuIDI1LCAyMDIzLCBhdCA5OjE0IEFNLCBNaWtlIFJh cG9wb3J0IHdyb3RlOgo+ID4gT24gTW9uLCBKdW4gMTksIDIwMjMgYXQgMTA6MDk6MDJBTSAtMDcw MCwgQW5keSBMdXRvbWlyc2tpIHdyb3RlOgo+ID4+IAo+ID4+IE9uIFN1biwgSnVuIDE4LCAyMDIz LCBhdCAxOjAwIEFNLCBNaWtlIFJhcG9wb3J0IHdyb3RlOgo+ID4+ID4gT24gU2F0LCBKdW4gMTcs IDIwMjMgYXQgMDE6Mzg6MjlQTSAtMDcwMCwgQW5keSBMdXRvbWlyc2tpIHdyb3RlOgo+ID4+ID4+ IE9uIEZyaSwgSnVuIDE2LCAyMDIzLCBhdCAxOjUwIEFNLCBNaWtlIFJhcG9wb3J0IHdyb3RlOgo+ ID4+ID4+ID4gRnJvbTogIk1pa2UgUmFwb3BvcnQgKElCTSkiIDxycHB0QGtlcm5lbC5vcmc+Cj4g Pj4gPj4gPgo+ID4+ID4+ID4gbW9kdWxlX2FsbG9jKCkgaXMgdXNlZCBldmVyeXdoZXJlIGFzIGEg bWVhbiB0byBhbGxvY2F0ZSBtZW1vcnkgZm9yIGNvZGUuCj4gPj4gPj4gPgo+ID4+ID4+ID4gQmVz aWRlIGJlaW5nIHNlbWFudGljYWxseSB3cm9uZywgdGhpcyB1bm5lY2Vzc2FyaWx5IHRpZXMgYWxs IHN1YnN5c3RlbXMKPiA+PiA+PiA+IHRoYXQgbmVlZCB0byBhbGxvY2F0ZSBjb2RlLCBzdWNoIGFz IGZ0cmFjZSwga3Byb2JlcyBhbmQgQlBGIHRvIG1vZHVsZXMKPiA+PiA+PiA+IGFuZCBwdXRzIHRo ZSBidXJkZW4gb2YgY29kZSBhbGxvY2F0aW9uIHRvIHRoZSBtb2R1bGVzIGNvZGUuCj4gPj4gPj4g Pgo+ID4+ID4+ID4gU2V2ZXJhbCBhcmNoaXRlY3R1cmVzIG92ZXJyaWRlIG1vZHVsZV9hbGxvYygp IGJlY2F1c2Ugb2YgdmFyaW91cwo+ID4+ID4+ID4gY29uc3RyYWludHMgd2hlcmUgdGhlIGV4ZWN1 dGFibGUgbWVtb3J5IGNhbiBiZSBsb2NhdGVkIGFuZCB0aGlzIGNhdXNlcwo+ID4+ID4+ID4gYWRk aXRpb25hbCBvYnN0YWNsZXMgZm9yIGltcHJvdmVtZW50cyBvZiBjb2RlIGFsbG9jYXRpb24uCj4g Pj4gPj4gPgo+ID4+ID4+ID4gU3RhcnQgc3BsaXR0aW5nIGNvZGUgYWxsb2NhdGlvbiBmcm9tIG1v ZHVsZXMgYnkgaW50cm9kdWNpbmcKPiA+PiA+PiA+IGV4ZWNtZW1fdGV4dF9hbGxvYygpLCBleGVj bWVtX2ZyZWUoKSwgaml0X3RleHRfYWxsb2MoKSwgaml0X2ZyZWUoKSBBUElzLgo+ID4+ID4+ID4K PiA+PiA+PiA+IEluaXRpYWxseSwgZXhlY21lbV90ZXh0X2FsbG9jKCkgYW5kIGppdF90ZXh0X2Fs bG9jKCkgYXJlIHdyYXBwZXJzIGZvcgo+ID4+ID4+ID4gbW9kdWxlX2FsbG9jKCkgYW5kIGV4ZWNt ZW1fZnJlZSgpIGFuZCBqaXRfZnJlZSgpIGFyZSByZXBsYWNlbWVudHMgb2YKPiA+PiA+PiA+IG1v ZHVsZV9tZW1mcmVlKCkgdG8gYWxsb3cgdXBkYXRpbmcgYWxsIGNhbGwgc2l0ZXMgdG8gdXNlIHRo ZSBuZXcgQVBJcy4KPiA+PiA+PiA+Cj4gPj4gPj4gPiBUaGUgaW50ZW50aW9uIHNlbWFudGljcyBm b3IgbmV3IGFsbG9jYXRpb24gQVBJczoKPiA+PiA+PiA+Cj4gPj4gPj4gPiAqIGV4ZWNtZW1fdGV4 dF9hbGxvYygpIHNob3VsZCBiZSB1c2VkIHRvIGFsbG9jYXRlIG1lbW9yeSB0aGF0IG11c3QgcmVz aWRlCj4gPj4gPj4gPiAgIGNsb3NlIHRvIHRoZSBrZXJuZWwgaW1hZ2UsIGxpa2UgbG9hZGFibGUg a2VybmVsIG1vZHVsZXMgYW5kIGdlbmVyYXRlZAo+ID4+ID4+ID4gICBjb2RlIHRoYXQgaXMgcmVz dHJpY3RlZCBieSByZWxhdGl2ZSBhZGRyZXNzaW5nLgo+ID4+ID4+ID4KPiA+PiA+PiA+ICogaml0 X3RleHRfYWxsb2MoKSBzaG91bGQgYmUgdXNlZCB0byBhbGxvY2F0ZSBtZW1vcnkgZm9yIGdlbmVy YXRlZCBjb2RlCj4gPj4gPj4gPiAgIHdoZW4gdGhlcmUgYXJlIG5vIHJlc3RyaWN0aW9ucyBmb3Ig dGhlIGNvZGUgcGxhY2VtZW50LiBGb3IKPiA+PiA+PiA+ICAgYXJjaGl0ZWN0dXJlcyB0aGF0IHJl cXVpcmUgdGhhdCBhbnkgY29kZSBpcyB3aXRoaW4gY2VydGFpbiBkaXN0YW5jZQo+ID4+ID4+ID4g ICBmcm9tIHRoZSBrZXJuZWwgaW1hZ2UsIGppdF90ZXh0X2FsbG9jKCkgd2lsbCBiZSBlc3NlbnRp YWxseSBhbGlhc2VkIHRvCj4gPj4gPj4gPiAgIGV4ZWNtZW1fdGV4dF9hbGxvYygpLgo+ID4+ID4+ ID4KPiA+PiA+PiAKPiA+PiA+PiBJcyB0aGVyZSBhbnl0aGluZyBpbiB0aGlzIHNlcmllcyB0byBo ZWxwIHVzZXJzIGRvIHRoZSBhcHByb3ByaWF0ZQo+ID4+ID4+IHN5bmNocm9uaXphdGlvbiB3aGVu IHRoZSBhY3R1YWxseSBwb3B1bGF0ZSB0aGUgYWxsb2NhdGVkIG1lbW9yeSB3aXRoCj4gPj4gPj4g Y29kZT8gIFNlZSBoZXJlLCBmb3IgZXhhbXBsZToKPiA+PiA+Cj4gPj4gPiBUaGlzIHNlcmllcyBv bmx5IGZhY3RvcnMgb3V0IHRoZSBleGVjdXRhYmxlIGFsbG9jYXRpb25zIGZyb20gbW9kdWxlcyBh bmQKPiA+PiA+IHB1dHMgdGhlbSBpbiBhIGNlbnRyYWwgcGxhY2UuCj4gPj4gPiBBbnl0aGluZyBl bHNlIHdvdWxkIGdvIG9uIHRvcCBhZnRlciB0aGlzIGxhbmRzLgo+ID4+IAo+ID4+IEhtbS4KPiA+ PiAKPiA+PiBPbiB0aGUgb25lIGhhbmQsIHRoZXJlJ3Mgbm90aGluZyB3cm9uZyB3aXRoIGZhY3Rv cmluZyBvdXQgY29tbW9uIGNvZGUuIE9uCj4gPj4gdGhlIG90aGVyIGhhbmQsIHRoaXMgaXMgcHJv YmFibHkgdGhlIHJpZ2h0IHRpbWUgdG8gYXQgbGVhc3Qgc3RhcnQKPiA+PiB0aGlua2luZyBhYm91 dCBzeW5jaHJvbml6YXRpb24sIGF0IGxlYXN0IHRvIHRoZSBleHRlbnQgdGhhdCBpdCBtaWdodCBt YWtlCj4gPj4gdXMgd2FudCB0byBjaGFuZ2UgdGhpcyBBUEkuICAoSSdtIG5vdCBhdCBhbGwgc2F5 aW5nIHRoYXQgdGhpcyBzZXJpZXMKPiA+PiBzaG91bGQgcmVxdWlyZSBjaGFuZ2VzIC0tIEknbSBq dXN0IHNheWluZyB0aGF0IHRoaXMgaXMgYSBnb29kIHRpbWUgdG8KPiA+PiB0aGluayBhYm91dCBo b3cgdGhpcyBzaG91bGQgd29yay4pCj4gPj4gCj4gPj4gVGhlIGN1cnJlbnQgQVBJcywgKmFuZCog dGhlIHByb3Bvc2VkIGppdF90ZXh0X2FsbG9jKCkgQVBJLCBkb24ndCBhY3R1YWxseQo+ID4+IGxv b2sgbGlrZSB0aGUgb25lIHRoaW5rIGluIHRoZSBMaW51eCBlY29zeXN0ZW0gdGhhdCBhY3R1YWxs eQo+ID4+IGludGVsbGlnZW50bHkgYW5kIGVmZmljaWVudGx5IG1hcHMgbmV3IHRleHQgaW50byBh biBhZGRyZXNzIHNwYWNlOgo+ID4+IG1tYXAoKS4KPiA+PiAKPiA+PiBPbiB4ODYsIHlvdSBjYW4g bW1hcCgpIGFuIGV4aXN0aW5nIGZpbGUgZnVsbCBvZiBleGVjdXRhYmxlIGNvZGUgUFJPVF9FWEVD Cj4gPj4gYW5kIGp1bXAgdG8gaXQgd2l0aCBtaW5pbWFsIHN5bmNocm9uaXphdGlvbiAoanVzdCB0 aGUgc3RhbmRhcmQgaW1wbGljaXQKPiA+PiBvcmRlcmluZyBpbiB0aGUga2VybmVsIHRoYXQgcG9w dWxhdGVzIHRoZSBwYWdlcyBiZWZvcmUgc2V0dGluZyB1cCB0aGUKPiA+PiBQVEVzIGFuZCB3aGF0 ZXZlciB1c2VyIHN5bmNocm9uaXphdGlvbiBpcyBuZWVkZWQgdG8gYXZvaWQganVtcGluZyBpbnRv Cj4gPj4gdGhlIG1hcHBpbmcgYmVmb3JlIG1tYXAoKSBmaW5pc2hlcykuICBJdCB3b3JrcyBhY3Jv c3MgQ1BVcywgYW5kIHRoZSBvbmx5Cj4gPj4gcG9zc2libGUgd2F5IHVzZXJzcGFjZSBjYW4gc2Ny ZXcgaXQgdXAgKGZvciBhIHJlYWQtb25seSBtYXBwaW5nIG9mCj4gPj4gcmVhZC1vbmx5IHRleHQs IGFueXdheSkgaXMgdG8ganVtcCB0byB0aGUgbWFwcGluZyB0b28gZWFybHksIGluIHdoaWNoCj4g Pj4gY2FzZSB1c2Vyc3BhY2UgZ2V0cyBhIHBhZ2UgZmF1bHQuICBJbmNvaGVyZW5jZSBpcyBpbXBv c3NpYmxlLCBhbmQgbm8gb25lCj4gPj4gbmVlZHMgdG8gInNlcmlhbGl6ZSIgKGluIHRoZSBTRE0g c2Vuc2UpLgo+ID4+IAo+ID4+IEkgdGhpbmsgdGhlIHNhbWUgc2VxdWVuY2UgKGZyb20gdXNlcnNw YWNlJ3MgcGVyc3BlY3RpdmUpIHdvcmtzIG9uIG90aGVyCj4gPj4gYXJjaGl0ZWN0dXJlcywgdG9v LCBhbHRob3VnaCBJIHRoaW5rIG1vcmUgY2FjaGUgbWFuYWdlbWVudCBpcyBuZWVkZWQgb24KPiA+ PiB0aGUga2VybmVsJ3MgZW5kLiAgQXMgZmFyIGFzIEkga25vdywgbm8gTGludXggU01QIGFyY2hp dGVjdHVyZSBuZWVkcyBhbgo+ID4+IElQSSB0byBtYXAgZXhlY3V0YWJsZSB0ZXh0IGludG8gdXNl cm1vZGUsIGJ1dCBJIGNvdWxkIGVhc2lseSBiZSB3cm9uZy4KPiA+PiAoSUlSQyBSSVNDLVYgaGFz IHZlcnkgZGV2ZWxvcGVyLXVuZnJpZW5kbHkgaWNhY2hlIG1hbmFnZW1lbnQsIGJ1dCBJIGRvbid0 Cj4gPj4gcmVtZW1iZXIgdGhlIGRldGFpbHMuKQo+ID4+IAo+ID4+IE9mIGNvdXJzZSwgdXNpbmcg cHRyYWNlIG9yIGFueSBvdGhlciBGT0xMX0ZPUkNFIHRvIG1vZGlmeSB0ZXh0IG9uIHg4NiBpcwo+ ID4+IHJhdGhlciBmcmF1Z2h0LCBhbmQgSSBiZXQgbWFueSB0aGluZ3MgZG8gaXQgd3Jvbmcgd2hl biB1c2Vyc3BhY2UgaXMKPiA+PiBtdWx0aXRocmVhZGVkLiAgQnV0IG5vdCBpbiBwcm9kdWN0aW9u IGJlY2F1c2UgaXQncyBtb3N0bHkgbm90IHVzZWQgaW4KPiA+PiBwcm9kdWN0aW9uLikKPiA+PiAK PiA+PiBCdXQgaml0X3RleHRfYWxsb2MoKSBjYW4ndCBkbyB0aGlzLCBiZWNhdXNlIHRoZSBvcmRl ciBvZiBvcGVyYXRpb25zCj4gPj4gZG9lc24ndCBtYXRjaC4gIFdpdGggaml0X3RleHRfYWxsb2Mo KSwgdGhlIGV4ZWN1dGFibGUgbWFwcGluZyBzaG93cyB1cAo+ID4+IGJlZm9yZSB0aGUgdGV4dCBp cyBwb3B1bGF0ZWQsIHNvIHRoZXJlIGlzIG5vIGF0b21pYyBjaGFuZ2UgZnJvbSBub3QtdGhlcmUK PiA+PiB0byBwb3B1bGF0ZWQtYW5kLWV4ZWN1dGFibGUuICBXaGljaCBtZWFucyB0aGF0IHRoZXJl IGlzIGFuIG9wcG9ydHVuaXR5Cj4gPj4gZm9yIENQVXMsIHNwZWN1bGF0aXZlbHkgb3Igb3RoZXJ3 aXNlLCB0byBzdGFydCBmaWxsaW5nIHZhcmlvdXMgY2FjaGVzCj4gPj4gd2l0aCBpbnRlcm1lZGlh dGUgc3RhdGVzIG9mIHRoZSB0ZXh0LCB3aGljaCBtZWFucyB0aGF0IHZhcmlvdXMKPiA+PiBhcmNo aXRlY3R1cmVzIChldmVuIHg4NiEpIG1heSBuZWVkIHNlcmlhbGl6YXRpb24uCj4gPj4gCj4gPj4g Rm9yIGVCUEYtIGFuZCBtb2R1bGUtIGxpa2UgdXNlIGNhc2VzLCB3aGVyZSBKSVR0aW5nL2NvZGUg Z2VuIGlzIHF1aXRlCj4gPj4gY29hcnNlLWdyYWluZWQsIHBlcmhhcHMgc29tZXRoaW5nIHZhZ3Vl bHkgbGlrZToKPiA+PiAKPiA+PiBqaXRfdGV4dF9hbGxvYygpIC0+IHJldHVybnMgYSBoYW5kbGUg YW5kIGFuIGV4ZWN1dGFibGUgdmlydHVhbCBhZGRyZXNzLAo+ID4+IGJ1dCBkb2VzICpub3QqIG1h cCBpdCB0aGVyZQo+ID4+IGppdF90ZXh0X3dyaXRlKCkgLT4gd3JpdGUgdG8gdGhhdCBoYW5kbGUK PiA+PiBqaXRfdGV4dF9tYXAoKSAtPiBtYXAgaXQgYW5kIHN5bmNocm9uaXplIGlmIG5lZWRlZCAo bm8gc3luYyBuZWVkZWQgb24KPiA+PiB4ODYsIEkgdGhpbmspCj4gPj4gCj4gPj4gY291bGQgYmUg bW9yZSBlZmZpY2llbnQgYW5kL29yIHNhZmVyLgo+ID4+IAo+ID4+IChNb2R1bGVzIGNvdWxkIHVz ZSB0aGlzIHRvby4gIEdldHRpbmcgYWx0ZXJuYXRpdmVzIHJpZ2h0IG1pZ2h0IHRha2Ugc29tZQo+ ID4+IGZpZGRsaW5nLCBiZWNhdXNlIG9mZiB0aGUgdG9wIG9mIG15IGhlYWQsIHRoaXMgZG9lc24n dCBtYXRjaCBob3cgaXQgd29ya3MKPiA+PiBub3cuKQo+ID4+IAo+ID4+IFRvIG1ha2UgYWx0ZXJu YXRpdmVzIGVhc2llciwgdGhpcyBjb3VsZCB3b3JrLCBtYXliZSAoaGF2ZW4ndCBmdWxseQo+ID4+ IHRob3VnaHQgaXQgdGhyb3VnaCk6Cj4gPj4gCj4gPj4gaml0X3RleHRfYWxsb2MoKQo+ID4+IGpp dF90ZXh0X21hcF9yd19pbnBsYWNlKCkgLT4gbWFwIGF0IHRoZSB0YXJnZXQgYWRkcmVzcywgYnV0 IFJXLCAhWAo+ID4+IAo+ID4+IHdyaXRlIHRoZSB0ZXh0IGFuZCBhcHBseSBhbHRlcm5hdGl2ZXMK PiA+PiAKPiA+PiBqaXRfdGV4dF9maW5hbGl6ZSgpIC0+IGNoYW5nZSBmcm9tIFJXIHRvIFJYICph bmQgc3luY2hyb25pemUqCj4gPj4gCj4gPj4gaml0X3RleHRfZmluYWxpemUoKSB3b3VsZCBlaXRo ZXIgbmVlZCB0byB3YWl0IGZvciBSQ1UgKHBvc3NpYmx5IGV4dHJhCj4gPj4gaGVhdnkgd2VpZ2h0 IFJDVSB0byBnZXQgInNlcmlhbGl6YXRpb24iKSBvciBzZW5kIGFuIElQSS4KPiA+Cj4gPiBUaGlz IGVzc2VudGlhbGx5IGhvdyBtb2R1bGVzIHdvcmsgbm93LiBUaGUgbWVtb3J5IGlzIGFsbG9jYXRl ZCBSVywgd3JpdHRlbgo+ID4gYW5kIHVwZGF0ZWQgd2l0aCBhbHRlcm5hdGl2ZXMgYW5kIHRoZW4g bWFkZSBST1ggaW4gdGhlIGVuZCB3aXRoIHNldF9tZW1vcnkKPiA+IEFQSXMuCj4gPgo+ID4gVGhl IGlzc3VlIHdpdGggbm90IGhhdmluZyB0aGUgbWVtb3J5IG1hcHBlZCBYIHdoZW4gaXQncyB3cml0 dGVuIGlzIHRoYXQgd2UKPiA+IGNhbm5vdCB1c2UgbGFyZ2UgcGFnZXMgdG8gbWFwIGl0LiBPbmUg b2YgdGhlIGdvYWxzIGlzIHRvIGhhdmUgZXhlY3V0YWJsZQo+ID4gbWVtb3J5IG1hcHBlZCB3aXRo IGxhcmdlIHBhZ2VzIGFuZCBtYWtlIGNvZGUgYWxsb2NhdG9yIGFibGUgdG8gZGl2aWRlIHRoYXQK PiA+IHBhZ2UgYW1vbmcgc2V2ZXJhbCBjYWxsZXJzLgo+ID4KPiA+IFNvIHRoZSBpZGVhIHdhcyB0 aGF0IGppdF90ZXh0X2FsbG9jKCkgd2lsbCBoYXZlIGEgY2FjaGUgb2YgbGFyZ2UgcGFnZXMKPiA+ IG1hcHBlZCBST1gsIHdpbGwgYWxsb2NhdGUgbWVtb3J5IGZyb20gdGhvc2UgY2FjaGVzIGFuZCB0 aGVyZSB3aWxsIGJlCj4gPiBqaXRfdXBkYXRlKCkgdGhhdCB1c2VzIHRleHQgcG9raW5nIGZvciB3 cml0aW5nIHRvIHRoYXQgbWVtb3J5Lgo+ID4KPiA+IFVwb24gYWxsb2NhdGlvbiBvZiBhIGxhcmdl IHBhZ2UgdG8gaW5jcmVhc2UgdGhlIGNhY2hlLCB0aGF0IGxhcmdlIHBhZ2Ugd2lsbAo+ID4gYmUg ImludmFsaWRhdGVkIiBieSBmaWxsaW5nIGl0IHdpdGggYnJlYWtwb2ludCBpbnN0cnVjdGlvbnMg KGUuZyBpbnQzIG9uCj4gPiB4ODYpCj4gCj4gSXMgdGhpcyBhY3R1YWxseSB2YWxpZD8gIEluIGJl dHdlZW4gaW50MyBhbmQgcmVhbCBjb2RlLCB0aGVyZeKAmXMgYQo+IHBvdGVudGlhbCB0b3JuIHJl YWQgb2YgcmVhbCBjb2RlIG1peGVkIHVwIHdpdGggMHhjYy4KIApZb3UgbWVhbiB3aGlsZSBkb2lu ZyB0ZXh0IHBva2luZz8KCj4gPiBUbyBpbXByb3ZlIHRoZSBwZXJmb3JtYW5jZSBvZiB0aGlzIHBy b2Nlc3MsIHdlIGNhbiB3cml0ZSB0byAhWCBjb3B5IGFuZAo+ID4gdGhlbiB0ZXh0X3Bva2UgaXQg dG8gdGhlIGFjdHVhbCBhZGRyZXNzIGluIG9uZSBnby4gVGhpcyB3aWxsIHJlcXVpcmUgc29tZQo+ ID4gY2hhbmdlcyB0byBnZXQgdGhlIGFsdGVybmF0aXZlcyByaWdodC4KPiA+Cj4gPiAtLSAKPiA+ IFNpbmNlcmVseSB5b3VycywKPiA+IE1pa2UuCgotLSAKU2luY2VyZWx5IHlvdXJzLApNaWtlLgoK X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtcmlz Y3YgbWFpbGluZyBsaXN0CmxpbnV4LXJpc2N2QGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xp c3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1yaXNjdgo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E7786EB64DC for ; Sun, 25 Jun 2023 17:45:25 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=YP5riMEL; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Qpz041h7nz30fj for ; Mon, 26 Jun 2023 03:45:24 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=YP5riMEL; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=139.178.84.217; helo=dfw.source.kernel.org; envelope-from=rppt@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Qpyz01TLxz2yV5 for ; Mon, 26 Jun 2023 03:44:28 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5A3C1602F9; Sun, 25 Jun 2023 17:44:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8AC02C433C8; Sun, 25 Jun 2023 17:44:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687715063; bh=WoUrmf0JEvFfQkljC61rPZoFHAXMId+ovOZ+fz1xXRM=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=YP5riMELjwGif5MetoqunCE/NEQNM05SODCHCM3NP274NT5kLDkCw+lYlFK9UhEOT qDLdn9rWmrAiPScC30o8bBxNq06ndu4LyRdafclIFEHnyh1dl1m6lm8HKVRj0PW8Gd bS1NGRYWfoyju/EEsTN0Hfg++5BGw6JBu3tNKZMoh+4lxQ/InBdA1cnWhewfz/DpH3 NQvBcro2ROnD4F+sChNda583y6v06vsiiNksZdiHaJOziCCpsjvIYI5dguVCw3W8jl /7gz0Aq4l2D8Ub/ROd+l5Cg2ih7cqqiikyhGIYUqney6omj2WYc5s+RTv4euyPzP/E NLShZkWcrywjw== Date: Sun, 25 Jun 2023 20:42:57 +0300 From: Mike Rapoport To: Andy Lutomirski Subject: Re: [PATCH v2 02/12] mm: introduce execmem_text_alloc() and jit_text_alloc() Message-ID: <20230625174257.GL52412@kernel.org> References: <20230616085038.4121892-1-rppt@kernel.org> <20230616085038.4121892-3-rppt@kernel.org> <20230618080027.GA52412@kernel.org> <20230625161417.GK52412@kernel.org> <90161ac9-3ca0-4c72-b1c4-ab1293e55445@app.fastmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <90161ac9-3ca0-4c72-b1c4-ab1293e55445@app.fastmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , the arch/x86 maintainers , Catalin Marinas , linux-mips@vger.kernel.org, Song Liu , sparclinux@vger.kernel.org, linux-riscv@lists.infradead.org, Nadav Amit , linux-s390@vger.kernel.org, Helge Deller , Huacai Chen , "Russell King \(Oracle\)" , "Naveen N. Rao" , linux-trace-kernel@vger.kernel.org, Kees Cook , Will Deacon , Heiko Carstens , Steven Rostedt , loongarch@lists.linux.dev, Thomas Gleixner , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Thomas Bogendoerfer , linux-parisc@vger.kernel.org, Puranjay Mohan , linux-mm@kvack.org, netdev@vger.kernel.org, Kent Overstreet , Linux Kernel Mailing List , Dinh Nguyen , Luis Chamberlain , Palmer Dabbelt , Andrew Morton , Rick P Edgecombe , linuxppc-dev@lists.ozlabs.org, "David S. Miller" , linux-modules@vger.kernel.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Sun, Jun 25, 2023 at 09:59:34AM -0700, Andy Lutomirski wrote: > > > On Sun, Jun 25, 2023, at 9:14 AM, Mike Rapoport wrote: > > On Mon, Jun 19, 2023 at 10:09:02AM -0700, Andy Lutomirski wrote: > >> > >> On Sun, Jun 18, 2023, at 1:00 AM, Mike Rapoport wrote: > >> > On Sat, Jun 17, 2023 at 01:38:29PM -0700, Andy Lutomirski wrote: > >> >> On Fri, Jun 16, 2023, at 1:50 AM, Mike Rapoport wrote: > >> >> > From: "Mike Rapoport (IBM)" > >> >> > > >> >> > module_alloc() is used everywhere as a mean to allocate memory for code. > >> >> > > >> >> > Beside being semantically wrong, this unnecessarily ties all subsystems > >> >> > that need to allocate code, such as ftrace, kprobes and BPF to modules > >> >> > and puts the burden of code allocation to the modules code. > >> >> > > >> >> > Several architectures override module_alloc() because of various > >> >> > constraints where the executable memory can be located and this causes > >> >> > additional obstacles for improvements of code allocation. > >> >> > > >> >> > Start splitting code allocation from modules by introducing > >> >> > execmem_text_alloc(), execmem_free(), jit_text_alloc(), jit_free() APIs. > >> >> > > >> >> > Initially, execmem_text_alloc() and jit_text_alloc() are wrappers for > >> >> > module_alloc() and execmem_free() and jit_free() are replacements of > >> >> > module_memfree() to allow updating all call sites to use the new APIs. > >> >> > > >> >> > The intention semantics for new allocation APIs: > >> >> > > >> >> > * execmem_text_alloc() should be used to allocate memory that must reside > >> >> > close to the kernel image, like loadable kernel modules and generated > >> >> > code that is restricted by relative addressing. > >> >> > > >> >> > * jit_text_alloc() should be used to allocate memory for generated code > >> >> > when there are no restrictions for the code placement. For > >> >> > architectures that require that any code is within certain distance > >> >> > from the kernel image, jit_text_alloc() will be essentially aliased to > >> >> > execmem_text_alloc(). > >> >> > > >> >> > >> >> Is there anything in this series to help users do the appropriate > >> >> synchronization when the actually populate the allocated memory with > >> >> code? See here, for example: > >> > > >> > This series only factors out the executable allocations from modules and > >> > puts them in a central place. > >> > Anything else would go on top after this lands. > >> > >> Hmm. > >> > >> On the one hand, there's nothing wrong with factoring out common code. On > >> the other hand, this is probably the right time to at least start > >> thinking about synchronization, at least to the extent that it might make > >> us want to change this API. (I'm not at all saying that this series > >> should require changes -- I'm just saying that this is a good time to > >> think about how this should work.) > >> > >> The current APIs, *and* the proposed jit_text_alloc() API, don't actually > >> look like the one think in the Linux ecosystem that actually > >> intelligently and efficiently maps new text into an address space: > >> mmap(). > >> > >> On x86, you can mmap() an existing file full of executable code PROT_EXEC > >> and jump to it with minimal synchronization (just the standard implicit > >> ordering in the kernel that populates the pages before setting up the > >> PTEs and whatever user synchronization is needed to avoid jumping into > >> the mapping before mmap() finishes). It works across CPUs, and the only > >> possible way userspace can screw it up (for a read-only mapping of > >> read-only text, anyway) is to jump to the mapping too early, in which > >> case userspace gets a page fault. Incoherence is impossible, and no one > >> needs to "serialize" (in the SDM sense). > >> > >> I think the same sequence (from userspace's perspective) works on other > >> architectures, too, although I think more cache management is needed on > >> the kernel's end. As far as I know, no Linux SMP architecture needs an > >> IPI to map executable text into usermode, but I could easily be wrong. > >> (IIRC RISC-V has very developer-unfriendly icache management, but I don't > >> remember the details.) > >> > >> Of course, using ptrace or any other FOLL_FORCE to modify text on x86 is > >> rather fraught, and I bet many things do it wrong when userspace is > >> multithreaded. But not in production because it's mostly not used in > >> production.) > >> > >> But jit_text_alloc() can't do this, because the order of operations > >> doesn't match. With jit_text_alloc(), the executable mapping shows up > >> before the text is populated, so there is no atomic change from not-there > >> to populated-and-executable. Which means that there is an opportunity > >> for CPUs, speculatively or otherwise, to start filling various caches > >> with intermediate states of the text, which means that various > >> architectures (even x86!) may need serialization. > >> > >> For eBPF- and module- like use cases, where JITting/code gen is quite > >> coarse-grained, perhaps something vaguely like: > >> > >> jit_text_alloc() -> returns a handle and an executable virtual address, > >> but does *not* map it there > >> jit_text_write() -> write to that handle > >> jit_text_map() -> map it and synchronize if needed (no sync needed on > >> x86, I think) > >> > >> could be more efficient and/or safer. > >> > >> (Modules could use this too. Getting alternatives right might take some > >> fiddling, because off the top of my head, this doesn't match how it works > >> now.) > >> > >> To make alternatives easier, this could work, maybe (haven't fully > >> thought it through): > >> > >> jit_text_alloc() > >> jit_text_map_rw_inplace() -> map at the target address, but RW, !X > >> > >> write the text and apply alternatives > >> > >> jit_text_finalize() -> change from RW to RX *and synchronize* > >> > >> jit_text_finalize() would either need to wait for RCU (possibly extra > >> heavy weight RCU to get "serialization") or send an IPI. > > > > This essentially how modules work now. The memory is allocated RW, written > > and updated with alternatives and then made ROX in the end with set_memory > > APIs. > > > > The issue with not having the memory mapped X when it's written is that we > > cannot use large pages to map it. One of the goals is to have executable > > memory mapped with large pages and make code allocator able to divide that > > page among several callers. > > > > So the idea was that jit_text_alloc() will have a cache of large pages > > mapped ROX, will allocate memory from those caches and there will be > > jit_update() that uses text poking for writing to that memory. > > > > Upon allocation of a large page to increase the cache, that large page will > > be "invalidated" by filling it with breakpoint instructions (e.g int3 on > > x86) > > Is this actually valid? In between int3 and real code, there’s a > potential torn read of real code mixed up with 0xcc. You mean while doing text poking? > > To improve the performance of this process, we can write to !X copy and > > then text_poke it to the actual address in one go. This will require some > > changes to get the alternatives right. > > > > -- > > Sincerely yours, > > Mike. -- Sincerely yours, Mike. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 03AD4C001B0 for ; Sun, 25 Jun 2023 17:45:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Jv/nRlceLpvHu/AgZ+ZkwkGJcCsznlHQ5GzC96cl1to=; b=pD/Rn/Fc74goX9 8z9Ux0xm/hortrFYOq89fo2iTLMwh8RjneUwo6kzXXAenECL/MeV92FovgJ7GBDVP0FCdiuu8Fx1I Eehm2k/V2zAU4X/Y8F/E+mYd2kUnGCXGaLlqfPVs4t/MgHePDvhkPdJ2AbPplp0WaryiN7PHqJMo5 mDRPsmKsEH1lE6+GeKMMdRbnfDp3vnl46KKbhC0QI1bIci4GDpAaIQb0X4tsDFcsXNr5MkQtIsfn5 esTWdUF8kR10R5FvHYLPivWLW3vk7qItJWFzo7c2pdtTdY8lnzq30OLSHXTBgFlFDnF+biGQ7oiuu Pf8eA4FRCEkz4M3momPA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qDTmp-008Od3-31; Sun, 25 Jun 2023 17:44:31 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qDTmj-008Obd-0z; Sun, 25 Jun 2023 17:44:27 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5A3C1602F9; Sun, 25 Jun 2023 17:44:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8AC02C433C8; Sun, 25 Jun 2023 17:44:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687715063; bh=WoUrmf0JEvFfQkljC61rPZoFHAXMId+ovOZ+fz1xXRM=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=YP5riMELjwGif5MetoqunCE/NEQNM05SODCHCM3NP274NT5kLDkCw+lYlFK9UhEOT qDLdn9rWmrAiPScC30o8bBxNq06ndu4LyRdafclIFEHnyh1dl1m6lm8HKVRj0PW8Gd bS1NGRYWfoyju/EEsTN0Hfg++5BGw6JBu3tNKZMoh+4lxQ/InBdA1cnWhewfz/DpH3 NQvBcro2ROnD4F+sChNda583y6v06vsiiNksZdiHaJOziCCpsjvIYI5dguVCw3W8jl /7gz0Aq4l2D8Ub/ROd+l5Cg2ih7cqqiikyhGIYUqney6omj2WYc5s+RTv4euyPzP/E NLShZkWcrywjw== Date: Sun, 25 Jun 2023 20:42:57 +0300 From: Mike Rapoport To: Andy Lutomirski Cc: Mark Rutland , Kees Cook , Linux Kernel Mailing List , Andrew Morton , Catalin Marinas , Christophe Leroy , "David S. Miller" , Dinh Nguyen , Heiko Carstens , Helge Deller , Huacai Chen , Kent Overstreet , Luis Chamberlain , Michael Ellerman , Nadav Amit , "Naveen N. Rao" , Palmer Dabbelt , Puranjay Mohan , Rick P Edgecombe , "Russell King (Oracle)" , Song Liu , Steven Rostedt , Thomas Bogendoerfer , Thomas Gleixner , Will Deacon , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev, netdev@vger.kernel.org, sparclinux@vger.kernel.org, the arch/x86 maintainers Subject: Re: [PATCH v2 02/12] mm: introduce execmem_text_alloc() and jit_text_alloc() Message-ID: <20230625174257.GL52412@kernel.org> References: <20230616085038.4121892-1-rppt@kernel.org> <20230616085038.4121892-3-rppt@kernel.org> <20230618080027.GA52412@kernel.org> <20230625161417.GK52412@kernel.org> <90161ac9-3ca0-4c72-b1c4-ab1293e55445@app.fastmail.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <90161ac9-3ca0-4c72-b1c4-ab1293e55445@app.fastmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230625_104425_438099_D9B5D343 X-CRM114-Status: GOOD ( 60.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gU3VuLCBKdW4gMjUsIDIwMjMgYXQgMDk6NTk6MzRBTSAtMDcwMCwgQW5keSBMdXRvbWlyc2tp IHdyb3RlOgo+IAo+IAo+IE9uIFN1biwgSnVuIDI1LCAyMDIzLCBhdCA5OjE0IEFNLCBNaWtlIFJh cG9wb3J0IHdyb3RlOgo+ID4gT24gTW9uLCBKdW4gMTksIDIwMjMgYXQgMTA6MDk6MDJBTSAtMDcw MCwgQW5keSBMdXRvbWlyc2tpIHdyb3RlOgo+ID4+IAo+ID4+IE9uIFN1biwgSnVuIDE4LCAyMDIz LCBhdCAxOjAwIEFNLCBNaWtlIFJhcG9wb3J0IHdyb3RlOgo+ID4+ID4gT24gU2F0LCBKdW4gMTcs IDIwMjMgYXQgMDE6Mzg6MjlQTSAtMDcwMCwgQW5keSBMdXRvbWlyc2tpIHdyb3RlOgo+ID4+ID4+ IE9uIEZyaSwgSnVuIDE2LCAyMDIzLCBhdCAxOjUwIEFNLCBNaWtlIFJhcG9wb3J0IHdyb3RlOgo+ ID4+ID4+ID4gRnJvbTogIk1pa2UgUmFwb3BvcnQgKElCTSkiIDxycHB0QGtlcm5lbC5vcmc+Cj4g Pj4gPj4gPgo+ID4+ID4+ID4gbW9kdWxlX2FsbG9jKCkgaXMgdXNlZCBldmVyeXdoZXJlIGFzIGEg bWVhbiB0byBhbGxvY2F0ZSBtZW1vcnkgZm9yIGNvZGUuCj4gPj4gPj4gPgo+ID4+ID4+ID4gQmVz aWRlIGJlaW5nIHNlbWFudGljYWxseSB3cm9uZywgdGhpcyB1bm5lY2Vzc2FyaWx5IHRpZXMgYWxs IHN1YnN5c3RlbXMKPiA+PiA+PiA+IHRoYXQgbmVlZCB0byBhbGxvY2F0ZSBjb2RlLCBzdWNoIGFz IGZ0cmFjZSwga3Byb2JlcyBhbmQgQlBGIHRvIG1vZHVsZXMKPiA+PiA+PiA+IGFuZCBwdXRzIHRo ZSBidXJkZW4gb2YgY29kZSBhbGxvY2F0aW9uIHRvIHRoZSBtb2R1bGVzIGNvZGUuCj4gPj4gPj4g Pgo+ID4+ID4+ID4gU2V2ZXJhbCBhcmNoaXRlY3R1cmVzIG92ZXJyaWRlIG1vZHVsZV9hbGxvYygp IGJlY2F1c2Ugb2YgdmFyaW91cwo+ID4+ID4+ID4gY29uc3RyYWludHMgd2hlcmUgdGhlIGV4ZWN1 dGFibGUgbWVtb3J5IGNhbiBiZSBsb2NhdGVkIGFuZCB0aGlzIGNhdXNlcwo+ID4+ID4+ID4gYWRk aXRpb25hbCBvYnN0YWNsZXMgZm9yIGltcHJvdmVtZW50cyBvZiBjb2RlIGFsbG9jYXRpb24uCj4g Pj4gPj4gPgo+ID4+ID4+ID4gU3RhcnQgc3BsaXR0aW5nIGNvZGUgYWxsb2NhdGlvbiBmcm9tIG1v ZHVsZXMgYnkgaW50cm9kdWNpbmcKPiA+PiA+PiA+IGV4ZWNtZW1fdGV4dF9hbGxvYygpLCBleGVj bWVtX2ZyZWUoKSwgaml0X3RleHRfYWxsb2MoKSwgaml0X2ZyZWUoKSBBUElzLgo+ID4+ID4+ID4K PiA+PiA+PiA+IEluaXRpYWxseSwgZXhlY21lbV90ZXh0X2FsbG9jKCkgYW5kIGppdF90ZXh0X2Fs bG9jKCkgYXJlIHdyYXBwZXJzIGZvcgo+ID4+ID4+ID4gbW9kdWxlX2FsbG9jKCkgYW5kIGV4ZWNt ZW1fZnJlZSgpIGFuZCBqaXRfZnJlZSgpIGFyZSByZXBsYWNlbWVudHMgb2YKPiA+PiA+PiA+IG1v ZHVsZV9tZW1mcmVlKCkgdG8gYWxsb3cgdXBkYXRpbmcgYWxsIGNhbGwgc2l0ZXMgdG8gdXNlIHRo ZSBuZXcgQVBJcy4KPiA+PiA+PiA+Cj4gPj4gPj4gPiBUaGUgaW50ZW50aW9uIHNlbWFudGljcyBm b3IgbmV3IGFsbG9jYXRpb24gQVBJczoKPiA+PiA+PiA+Cj4gPj4gPj4gPiAqIGV4ZWNtZW1fdGV4 dF9hbGxvYygpIHNob3VsZCBiZSB1c2VkIHRvIGFsbG9jYXRlIG1lbW9yeSB0aGF0IG11c3QgcmVz aWRlCj4gPj4gPj4gPiAgIGNsb3NlIHRvIHRoZSBrZXJuZWwgaW1hZ2UsIGxpa2UgbG9hZGFibGUg a2VybmVsIG1vZHVsZXMgYW5kIGdlbmVyYXRlZAo+ID4+ID4+ID4gICBjb2RlIHRoYXQgaXMgcmVz dHJpY3RlZCBieSByZWxhdGl2ZSBhZGRyZXNzaW5nLgo+ID4+ID4+ID4KPiA+PiA+PiA+ICogaml0 X3RleHRfYWxsb2MoKSBzaG91bGQgYmUgdXNlZCB0byBhbGxvY2F0ZSBtZW1vcnkgZm9yIGdlbmVy YXRlZCBjb2RlCj4gPj4gPj4gPiAgIHdoZW4gdGhlcmUgYXJlIG5vIHJlc3RyaWN0aW9ucyBmb3Ig dGhlIGNvZGUgcGxhY2VtZW50LiBGb3IKPiA+PiA+PiA+ICAgYXJjaGl0ZWN0dXJlcyB0aGF0IHJl cXVpcmUgdGhhdCBhbnkgY29kZSBpcyB3aXRoaW4gY2VydGFpbiBkaXN0YW5jZQo+ID4+ID4+ID4g ICBmcm9tIHRoZSBrZXJuZWwgaW1hZ2UsIGppdF90ZXh0X2FsbG9jKCkgd2lsbCBiZSBlc3NlbnRp YWxseSBhbGlhc2VkIHRvCj4gPj4gPj4gPiAgIGV4ZWNtZW1fdGV4dF9hbGxvYygpLgo+ID4+ID4+ ID4KPiA+PiA+PiAKPiA+PiA+PiBJcyB0aGVyZSBhbnl0aGluZyBpbiB0aGlzIHNlcmllcyB0byBo ZWxwIHVzZXJzIGRvIHRoZSBhcHByb3ByaWF0ZQo+ID4+ID4+IHN5bmNocm9uaXphdGlvbiB3aGVu IHRoZSBhY3R1YWxseSBwb3B1bGF0ZSB0aGUgYWxsb2NhdGVkIG1lbW9yeSB3aXRoCj4gPj4gPj4g Y29kZT8gIFNlZSBoZXJlLCBmb3IgZXhhbXBsZToKPiA+PiA+Cj4gPj4gPiBUaGlzIHNlcmllcyBv bmx5IGZhY3RvcnMgb3V0IHRoZSBleGVjdXRhYmxlIGFsbG9jYXRpb25zIGZyb20gbW9kdWxlcyBh bmQKPiA+PiA+IHB1dHMgdGhlbSBpbiBhIGNlbnRyYWwgcGxhY2UuCj4gPj4gPiBBbnl0aGluZyBl bHNlIHdvdWxkIGdvIG9uIHRvcCBhZnRlciB0aGlzIGxhbmRzLgo+ID4+IAo+ID4+IEhtbS4KPiA+ PiAKPiA+PiBPbiB0aGUgb25lIGhhbmQsIHRoZXJlJ3Mgbm90aGluZyB3cm9uZyB3aXRoIGZhY3Rv cmluZyBvdXQgY29tbW9uIGNvZGUuIE9uCj4gPj4gdGhlIG90aGVyIGhhbmQsIHRoaXMgaXMgcHJv YmFibHkgdGhlIHJpZ2h0IHRpbWUgdG8gYXQgbGVhc3Qgc3RhcnQKPiA+PiB0aGlua2luZyBhYm91 dCBzeW5jaHJvbml6YXRpb24sIGF0IGxlYXN0IHRvIHRoZSBleHRlbnQgdGhhdCBpdCBtaWdodCBt YWtlCj4gPj4gdXMgd2FudCB0byBjaGFuZ2UgdGhpcyBBUEkuICAoSSdtIG5vdCBhdCBhbGwgc2F5 aW5nIHRoYXQgdGhpcyBzZXJpZXMKPiA+PiBzaG91bGQgcmVxdWlyZSBjaGFuZ2VzIC0tIEknbSBq dXN0IHNheWluZyB0aGF0IHRoaXMgaXMgYSBnb29kIHRpbWUgdG8KPiA+PiB0aGluayBhYm91dCBo b3cgdGhpcyBzaG91bGQgd29yay4pCj4gPj4gCj4gPj4gVGhlIGN1cnJlbnQgQVBJcywgKmFuZCog dGhlIHByb3Bvc2VkIGppdF90ZXh0X2FsbG9jKCkgQVBJLCBkb24ndCBhY3R1YWxseQo+ID4+IGxv b2sgbGlrZSB0aGUgb25lIHRoaW5rIGluIHRoZSBMaW51eCBlY29zeXN0ZW0gdGhhdCBhY3R1YWxs eQo+ID4+IGludGVsbGlnZW50bHkgYW5kIGVmZmljaWVudGx5IG1hcHMgbmV3IHRleHQgaW50byBh biBhZGRyZXNzIHNwYWNlOgo+ID4+IG1tYXAoKS4KPiA+PiAKPiA+PiBPbiB4ODYsIHlvdSBjYW4g bW1hcCgpIGFuIGV4aXN0aW5nIGZpbGUgZnVsbCBvZiBleGVjdXRhYmxlIGNvZGUgUFJPVF9FWEVD Cj4gPj4gYW5kIGp1bXAgdG8gaXQgd2l0aCBtaW5pbWFsIHN5bmNocm9uaXphdGlvbiAoanVzdCB0 aGUgc3RhbmRhcmQgaW1wbGljaXQKPiA+PiBvcmRlcmluZyBpbiB0aGUga2VybmVsIHRoYXQgcG9w dWxhdGVzIHRoZSBwYWdlcyBiZWZvcmUgc2V0dGluZyB1cCB0aGUKPiA+PiBQVEVzIGFuZCB3aGF0 ZXZlciB1c2VyIHN5bmNocm9uaXphdGlvbiBpcyBuZWVkZWQgdG8gYXZvaWQganVtcGluZyBpbnRv Cj4gPj4gdGhlIG1hcHBpbmcgYmVmb3JlIG1tYXAoKSBmaW5pc2hlcykuICBJdCB3b3JrcyBhY3Jv c3MgQ1BVcywgYW5kIHRoZSBvbmx5Cj4gPj4gcG9zc2libGUgd2F5IHVzZXJzcGFjZSBjYW4gc2Ny ZXcgaXQgdXAgKGZvciBhIHJlYWQtb25seSBtYXBwaW5nIG9mCj4gPj4gcmVhZC1vbmx5IHRleHQs IGFueXdheSkgaXMgdG8ganVtcCB0byB0aGUgbWFwcGluZyB0b28gZWFybHksIGluIHdoaWNoCj4g Pj4gY2FzZSB1c2Vyc3BhY2UgZ2V0cyBhIHBhZ2UgZmF1bHQuICBJbmNvaGVyZW5jZSBpcyBpbXBv c3NpYmxlLCBhbmQgbm8gb25lCj4gPj4gbmVlZHMgdG8gInNlcmlhbGl6ZSIgKGluIHRoZSBTRE0g c2Vuc2UpLgo+ID4+IAo+ID4+IEkgdGhpbmsgdGhlIHNhbWUgc2VxdWVuY2UgKGZyb20gdXNlcnNw YWNlJ3MgcGVyc3BlY3RpdmUpIHdvcmtzIG9uIG90aGVyCj4gPj4gYXJjaGl0ZWN0dXJlcywgdG9v LCBhbHRob3VnaCBJIHRoaW5rIG1vcmUgY2FjaGUgbWFuYWdlbWVudCBpcyBuZWVkZWQgb24KPiA+ PiB0aGUga2VybmVsJ3MgZW5kLiAgQXMgZmFyIGFzIEkga25vdywgbm8gTGludXggU01QIGFyY2hp dGVjdHVyZSBuZWVkcyBhbgo+ID4+IElQSSB0byBtYXAgZXhlY3V0YWJsZSB0ZXh0IGludG8gdXNl cm1vZGUsIGJ1dCBJIGNvdWxkIGVhc2lseSBiZSB3cm9uZy4KPiA+PiAoSUlSQyBSSVNDLVYgaGFz IHZlcnkgZGV2ZWxvcGVyLXVuZnJpZW5kbHkgaWNhY2hlIG1hbmFnZW1lbnQsIGJ1dCBJIGRvbid0 Cj4gPj4gcmVtZW1iZXIgdGhlIGRldGFpbHMuKQo+ID4+IAo+ID4+IE9mIGNvdXJzZSwgdXNpbmcg cHRyYWNlIG9yIGFueSBvdGhlciBGT0xMX0ZPUkNFIHRvIG1vZGlmeSB0ZXh0IG9uIHg4NiBpcwo+ ID4+IHJhdGhlciBmcmF1Z2h0LCBhbmQgSSBiZXQgbWFueSB0aGluZ3MgZG8gaXQgd3Jvbmcgd2hl biB1c2Vyc3BhY2UgaXMKPiA+PiBtdWx0aXRocmVhZGVkLiAgQnV0IG5vdCBpbiBwcm9kdWN0aW9u IGJlY2F1c2UgaXQncyBtb3N0bHkgbm90IHVzZWQgaW4KPiA+PiBwcm9kdWN0aW9uLikKPiA+PiAK PiA+PiBCdXQgaml0X3RleHRfYWxsb2MoKSBjYW4ndCBkbyB0aGlzLCBiZWNhdXNlIHRoZSBvcmRl ciBvZiBvcGVyYXRpb25zCj4gPj4gZG9lc24ndCBtYXRjaC4gIFdpdGggaml0X3RleHRfYWxsb2Mo KSwgdGhlIGV4ZWN1dGFibGUgbWFwcGluZyBzaG93cyB1cAo+ID4+IGJlZm9yZSB0aGUgdGV4dCBp cyBwb3B1bGF0ZWQsIHNvIHRoZXJlIGlzIG5vIGF0b21pYyBjaGFuZ2UgZnJvbSBub3QtdGhlcmUK PiA+PiB0byBwb3B1bGF0ZWQtYW5kLWV4ZWN1dGFibGUuICBXaGljaCBtZWFucyB0aGF0IHRoZXJl IGlzIGFuIG9wcG9ydHVuaXR5Cj4gPj4gZm9yIENQVXMsIHNwZWN1bGF0aXZlbHkgb3Igb3RoZXJ3 aXNlLCB0byBzdGFydCBmaWxsaW5nIHZhcmlvdXMgY2FjaGVzCj4gPj4gd2l0aCBpbnRlcm1lZGlh dGUgc3RhdGVzIG9mIHRoZSB0ZXh0LCB3aGljaCBtZWFucyB0aGF0IHZhcmlvdXMKPiA+PiBhcmNo aXRlY3R1cmVzIChldmVuIHg4NiEpIG1heSBuZWVkIHNlcmlhbGl6YXRpb24uCj4gPj4gCj4gPj4g Rm9yIGVCUEYtIGFuZCBtb2R1bGUtIGxpa2UgdXNlIGNhc2VzLCB3aGVyZSBKSVR0aW5nL2NvZGUg Z2VuIGlzIHF1aXRlCj4gPj4gY29hcnNlLWdyYWluZWQsIHBlcmhhcHMgc29tZXRoaW5nIHZhZ3Vl bHkgbGlrZToKPiA+PiAKPiA+PiBqaXRfdGV4dF9hbGxvYygpIC0+IHJldHVybnMgYSBoYW5kbGUg YW5kIGFuIGV4ZWN1dGFibGUgdmlydHVhbCBhZGRyZXNzLAo+ID4+IGJ1dCBkb2VzICpub3QqIG1h cCBpdCB0aGVyZQo+ID4+IGppdF90ZXh0X3dyaXRlKCkgLT4gd3JpdGUgdG8gdGhhdCBoYW5kbGUK PiA+PiBqaXRfdGV4dF9tYXAoKSAtPiBtYXAgaXQgYW5kIHN5bmNocm9uaXplIGlmIG5lZWRlZCAo bm8gc3luYyBuZWVkZWQgb24KPiA+PiB4ODYsIEkgdGhpbmspCj4gPj4gCj4gPj4gY291bGQgYmUg bW9yZSBlZmZpY2llbnQgYW5kL29yIHNhZmVyLgo+ID4+IAo+ID4+IChNb2R1bGVzIGNvdWxkIHVz ZSB0aGlzIHRvby4gIEdldHRpbmcgYWx0ZXJuYXRpdmVzIHJpZ2h0IG1pZ2h0IHRha2Ugc29tZQo+ ID4+IGZpZGRsaW5nLCBiZWNhdXNlIG9mZiB0aGUgdG9wIG9mIG15IGhlYWQsIHRoaXMgZG9lc24n dCBtYXRjaCBob3cgaXQgd29ya3MKPiA+PiBub3cuKQo+ID4+IAo+ID4+IFRvIG1ha2UgYWx0ZXJu YXRpdmVzIGVhc2llciwgdGhpcyBjb3VsZCB3b3JrLCBtYXliZSAoaGF2ZW4ndCBmdWxseQo+ID4+ IHRob3VnaHQgaXQgdGhyb3VnaCk6Cj4gPj4gCj4gPj4gaml0X3RleHRfYWxsb2MoKQo+ID4+IGpp dF90ZXh0X21hcF9yd19pbnBsYWNlKCkgLT4gbWFwIGF0IHRoZSB0YXJnZXQgYWRkcmVzcywgYnV0 IFJXLCAhWAo+ID4+IAo+ID4+IHdyaXRlIHRoZSB0ZXh0IGFuZCBhcHBseSBhbHRlcm5hdGl2ZXMK PiA+PiAKPiA+PiBqaXRfdGV4dF9maW5hbGl6ZSgpIC0+IGNoYW5nZSBmcm9tIFJXIHRvIFJYICph bmQgc3luY2hyb25pemUqCj4gPj4gCj4gPj4gaml0X3RleHRfZmluYWxpemUoKSB3b3VsZCBlaXRo ZXIgbmVlZCB0byB3YWl0IGZvciBSQ1UgKHBvc3NpYmx5IGV4dHJhCj4gPj4gaGVhdnkgd2VpZ2h0 IFJDVSB0byBnZXQgInNlcmlhbGl6YXRpb24iKSBvciBzZW5kIGFuIElQSS4KPiA+Cj4gPiBUaGlz IGVzc2VudGlhbGx5IGhvdyBtb2R1bGVzIHdvcmsgbm93LiBUaGUgbWVtb3J5IGlzIGFsbG9jYXRl ZCBSVywgd3JpdHRlbgo+ID4gYW5kIHVwZGF0ZWQgd2l0aCBhbHRlcm5hdGl2ZXMgYW5kIHRoZW4g bWFkZSBST1ggaW4gdGhlIGVuZCB3aXRoIHNldF9tZW1vcnkKPiA+IEFQSXMuCj4gPgo+ID4gVGhl IGlzc3VlIHdpdGggbm90IGhhdmluZyB0aGUgbWVtb3J5IG1hcHBlZCBYIHdoZW4gaXQncyB3cml0 dGVuIGlzIHRoYXQgd2UKPiA+IGNhbm5vdCB1c2UgbGFyZ2UgcGFnZXMgdG8gbWFwIGl0LiBPbmUg b2YgdGhlIGdvYWxzIGlzIHRvIGhhdmUgZXhlY3V0YWJsZQo+ID4gbWVtb3J5IG1hcHBlZCB3aXRo IGxhcmdlIHBhZ2VzIGFuZCBtYWtlIGNvZGUgYWxsb2NhdG9yIGFibGUgdG8gZGl2aWRlIHRoYXQK PiA+IHBhZ2UgYW1vbmcgc2V2ZXJhbCBjYWxsZXJzLgo+ID4KPiA+IFNvIHRoZSBpZGVhIHdhcyB0 aGF0IGppdF90ZXh0X2FsbG9jKCkgd2lsbCBoYXZlIGEgY2FjaGUgb2YgbGFyZ2UgcGFnZXMKPiA+ IG1hcHBlZCBST1gsIHdpbGwgYWxsb2NhdGUgbWVtb3J5IGZyb20gdGhvc2UgY2FjaGVzIGFuZCB0 aGVyZSB3aWxsIGJlCj4gPiBqaXRfdXBkYXRlKCkgdGhhdCB1c2VzIHRleHQgcG9raW5nIGZvciB3 cml0aW5nIHRvIHRoYXQgbWVtb3J5Lgo+ID4KPiA+IFVwb24gYWxsb2NhdGlvbiBvZiBhIGxhcmdl IHBhZ2UgdG8gaW5jcmVhc2UgdGhlIGNhY2hlLCB0aGF0IGxhcmdlIHBhZ2Ugd2lsbAo+ID4gYmUg ImludmFsaWRhdGVkIiBieSBmaWxsaW5nIGl0IHdpdGggYnJlYWtwb2ludCBpbnN0cnVjdGlvbnMg KGUuZyBpbnQzIG9uCj4gPiB4ODYpCj4gCj4gSXMgdGhpcyBhY3R1YWxseSB2YWxpZD8gIEluIGJl dHdlZW4gaW50MyBhbmQgcmVhbCBjb2RlLCB0aGVyZeKAmXMgYQo+IHBvdGVudGlhbCB0b3JuIHJl YWQgb2YgcmVhbCBjb2RlIG1peGVkIHVwIHdpdGggMHhjYy4KIApZb3UgbWVhbiB3aGlsZSBkb2lu ZyB0ZXh0IHBva2luZz8KCj4gPiBUbyBpbXByb3ZlIHRoZSBwZXJmb3JtYW5jZSBvZiB0aGlzIHBy b2Nlc3MsIHdlIGNhbiB3cml0ZSB0byAhWCBjb3B5IGFuZAo+ID4gdGhlbiB0ZXh0X3Bva2UgaXQg dG8gdGhlIGFjdHVhbCBhZGRyZXNzIGluIG9uZSBnby4gVGhpcyB3aWxsIHJlcXVpcmUgc29tZQo+ ID4gY2hhbmdlcyB0byBnZXQgdGhlIGFsdGVybmF0aXZlcyByaWdodC4KPiA+Cj4gPiAtLSAKPiA+ IFNpbmNlcmVseSB5b3VycywKPiA+IE1pa2UuCgotLSAKU2luY2VyZWx5IHlvdXJzLApNaWtlLgoK X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJt LWtlcm5lbCBtYWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3Jn Cmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtl cm5lbAo=