From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="u2cVkyxL" Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92E89FF; Mon, 4 Dec 2023 10:34:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=yRxkmI4dXPVCTApSsXOOM346LAa1g76vwgSBu1bA/08=; b=u2cVkyxLqoUOqz1TA/dDurEImX yd+RKQfGNzDw1nNzm7F9pDtqQt51c1WaOKmTKmAbAROOiek+hm8k6eZisRZ/QWEyDAbQFc+SykZth yo1Jqw2hGSDZx0/kwV/QhUbR1CaOvc39CcdS6VXzjuqXt4HYSSGFkZCVv/chpwHXClxU600842aLI poUrNAgUfXs1GTMxHRcsbo3DI/Uyx2B0XaXImtz+wS5a8oBto19jcxToog2JblpiSQQodC47SWxtq uoUKbAOA8OTlQYtBciApJtmEJWfW4deFx0DxiGRyt/1iUxUN98T1XdrCmGbwL4cixtGF6odl8Z3oi wU5Z7Stg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1rADlT-000x6k-F8; Mon, 04 Dec 2023 18:33:56 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id 5462830057C; Mon, 4 Dec 2023 19:33:54 +0100 (CET) Date: Mon, 4 Dec 2023 19:33:54 +0100 From: Peter Zijlstra To: Jiri Olsa Cc: Alexei Starovoitov , Song Liu , Song Liu , Paul Walmsley , Palmer Dabbelt , Albert Ou , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , X86 ML , "H. Peter Anvin" , "David S. Miller" , David Ahern , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Arnd Bergmann , Sami Tolvanen , Kees Cook , Nathan Chancellor , Nick Desaulniers , linux-riscv , LKML , Network Development , bpf , linux-arch , clang-built-linux , Josh Poimboeuf , Joao Moreira , Mark Rutland Subject: Re: [PATCH v2 2/2] x86/cfi,bpf: Fix BPF JIT call Message-ID: <20231204183354.GC7299@noisy.programming.kicks-ass.net> References: <20231130133630.192490507@infradead.org> <20231130134204.136058029@infradead.org> <20231204091334.GM3818@noisy.programming.kicks-ass.net> <20231204111128.GV8262@noisy.programming.kicks-ass.net> <20231204125239.GA1319@noisy.programming.kicks-ass.net> <20231204181614.GA7299@noisy.programming.kicks-ass.net> Precedence: bulk X-Mailing-List: linux-arch@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231204181614.GA7299@noisy.programming.kicks-ass.net> On Mon, Dec 04, 2023 at 07:16:14PM +0100, Peter Zijlstra wrote: > On Mon, Dec 04, 2023 at 06:25:34PM +0100, Jiri Olsa wrote: > > > that boots properly for me but gives crash below when running bpf tests > > OK, more funnies.. > > > [ 482.145182][ T699] RIP: 0010:bpf_for_each_array_elem+0xbb/0x120 > > [ 482.145672][ T699] Code: 4c 01 f5 89 5c 24 04 4c 89 e7 48 8d 74 24 04 48 89 ea 4c 89 fd 4c 89 f9 45 31 c0 4d 89 eb 41 ba ef 86 cd 67 45 03 53 f1 74 02 <0f> 0b 41 ff d3 0f 1f 00 48 85 c0 75 0e 48 8d 43 01 41 8b 4c 24 24 > > [ 482.147221][ T699] RSP: 0018:ffffc900017e3e88 EFLAGS: 00010217 > > [ 482.147702][ T699] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffc900017e3ed8 > > [ 482.152162][ T699] RDX: ffff888152eb0210 RSI: ffffc900017e3e8c RDI: ffff888152eb0000 > > [ 482.152770][ T699] RBP: ffffc900017e3ed8 R08: 0000000000000000 R09: 0000000000000000 > > [ 482.153350][ T699] R10: 000000004704ef28 R11: ffffffffa0012774 R12: ffff888152eb0000 > > [ 482.153951][ T699] R13: ffffffffa0012774 R14: ffff888152eb0210 R15: ffffc900017e3ed8 > > [ 482.154554][ T699] FS: 00007fa60d4fdd00(0000) GS:ffff88846d200000(0000) knlGS:0000000000000000 > > [ 482.155138][ T699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 482.155564][ T699] CR2: 00007fa60d7d8000 CR3: 00000001502a2005 CR4: 0000000000770ef0 > > [ 482.156095][ T699] PKRU: 55555554 > > [ 482.156349][ T699] Call Trace: > > [ 482.156596][ T699] > > [ 482.156816][ T699] ? __die_body+0x68/0xb0 > > [ 482.157138][ T699] ? die+0xba/0xe0 > > [ 482.157456][ T699] ? do_trap+0xa5/0x180 > > [ 482.157826][ T699] ? bpf_for_each_array_elem+0xbb/0x120 > > [ 482.158277][ T699] ? bpf_for_each_array_elem+0xbb/0x120 > > [ 482.158711][ T699] ? do_error_trap+0xc4/0x140 > > [ 482.159052][ T699] ? bpf_for_each_array_elem+0xbb/0x120 > > [ 482.159506][ T699] ? handle_invalid_op+0x2c/0x40 > > [ 482.159906][ T699] ? bpf_for_each_array_elem+0xbb/0x120 > > [ 482.160990][ T699] ? exc_invalid_op+0x38/0x60 > > [ 482.161375][ T699] ? asm_exc_invalid_op+0x1a/0x20 > > [ 482.161788][ T699] ? 0xffffffffa0012774 > > [ 482.162149][ T699] ? 0xffffffffa0012774 > > [ 482.162513][ T699] ? bpf_for_each_array_elem+0xbb/0x120 > > [ 482.162905][ T699] bpf_prog_ca45ea7f9cb8ac1a_inner_map+0x94/0x98 > > [ 482.163471][ T699] bpf_trampoline_6442549234+0x47/0x1000 > > Looks like this trips an #UD, I'll go try and figure out what this > bpf_for_each_array_elem() does to cause this. Looks like it has an > indirect call, could be the callback_fn thing has a CFI mis-match. So afaict this is used through bpf_for_each_map_elem(), where the argument still is properly callback_fn. However, in the desriptor bpf_for_each_map_elem_proto the argument gets described as: ARG_PTR_TO_FUNC, which in turn has a comment like: ARG_PTR_TO_FUNC, /* pointer to a bpf program function */ Which to me sounds like there is definite type punning involved. The call in bpf_for_each_array_elem() is a regular C indirect call, which gets adorned with the kCFI magic. But I doubt the BPF function that gets used gets the correct matching bits on. TL;DR, I think this is a pre-existing problem with kCFI + eBPF and not caused by my patches. Could any of you bpf knowledgeable folks please explain me exactly what gets used as the function pointer in this case? -- I'm not sure I can follow along well enough to begin looking for a solution at this point :/