From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB4EA2F12AE for ; Tue, 16 Jun 2026 20:47:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781642848; cv=none; b=MBGq9CYvJ9WwdbxNvUiWLZ7o9mswJ/cdI3J6Nf66Ymb05CzL1MfmdKzqMObEQag6++o7RHgVrb3UQANXDwA690+oeN4STI5YEI9y4GjECunsGCFC2pknus2fYcpvSRHmj+sW0ZeUw7161rN7ffhAsRgdmzkID0iZ9riFoai2FoI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781642848; c=relaxed/simple; bh=mL+wOZQiL28w7puau+GqyfDifkEa0FODm8ebrzMby+U=; h=Date:From:To:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Ux/7BL8ODlIVsRRiTbHCahrfHtuZic8RQV/a7/ojYF6rNtS9n5EkhF0BcdhemfMQ2IjqoDpJECgUEaAXmwkDaXgYqtfh+0SUlzn8hGyAvubZoSq1PtPenQp6q3+Nz/cZZVkgH7XsZJlTHScBX6JOj5TGYpAPmfzvQb5mT+A+sSY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OIQZv++e; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OIQZv++e" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-490b211ee6aso37980535e9.3 for ; Tue, 16 Jun 2026 13:47:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1781642845; x=1782247645; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=3FUMdL5eh/Y17TG5wIomYQ9CYDQCgPi8iVfZQyjBW8c=; b=OIQZv++e0Gf6tJkvCMY7WxnQh+letbFHhVoUT48BYOzVDDhM2N3cKR+OePmPfFpxwu Pj9WIxvJrPpILfo7m4F6gUM59SLeId0i/vLod3trIw8Wpo7CyCwh5aerpo3yn10l1g49 OEPygGqZKKYOz7KhJ4JMu9Jzkfwe/Bse5/5o/9qkxFGWizdnRK079IncxFxPrPYH4TRg BCDwIazVyKdD29CD58tAbHcupSMetLXUdcZlu9157yoBvYaChjGQBgSJq8l63kpJa2Vl 9zMgXhPDjhqDyYGIHP/NbJyxc83SA8rYnd1kkZqjHkjhZXZj3zF7bHFzFka7aiB9tSzh IFgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781642845; x=1782247645; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=3FUMdL5eh/Y17TG5wIomYQ9CYDQCgPi8iVfZQyjBW8c=; b=s2fgMYnZVCelCzl54JbqRHRIikhZLubYKwYRMgqhWvP+JwbgYpm+9X3JQJIJt+P1ET 8rZ3KZnzA+dS7lYoP9ipf2hpw50LIoXl5iYma4sCV4EqCgd6vRg5/HoIM86JVNTJHICq 5Nn6IzyjxtOe2glJHLGVoubvtAPzLLKqygmAX7AbBEI6aYQESSHj6V0yQBaQhyDDiMlW ileB16Yn2AuyfaFor3wWHikE9e3w9Re+mt+wU9qbF0O3MYS6gHoQ/keStLkIcYApggAP 2nQTtK33YVc/taiD1l332+uuLzuL4d5V+rBzZ2b8SVgLTupQb4HOluesRby5/nNzH+oO 61oQ== X-Forwarded-Encrypted: i=1; AFNElJ+68XtzOjhV51b2walZrHhepc6mu9gjfcatRayi7P5LTVwhhS96MpbSSz1fWJ7ZOdDZWbQP/p1l+1PWa/0=@vger.kernel.org X-Gm-Message-State: AOJu0Yw3T8TdWGELfm6NvngnLuqXPIuCnzWEJpEDg3hgf85ey2o88xBp 7OtDlhgOCxl4azPMevPMgWV4aFpKWx8kvf12fFRtT1kx0BEr/S/Nsc0O X-Gm-Gg: Acq92OHgo+WTJ6T7YqJYIS+iVofaGZF9Xz0t3vhNg1//lUMImEoAXh3g5ydCvDkmj+D UZcru5RK7QVBi7RwMWrAVNgbi155sXmzRxc+h2tnCbaOj0gJ7sPNKy9mqBTd0sViZS7KLSMYvv5 jphXVwhfUxxXNO+KI48POMv3xWgebiyKnVUpMNQYd+HX/UBnRF6zfIFIVGPG6Lx+QnJcA1m2nPQ OqJB8XlsxaOZmbsa/ojB9joE9PPisfRbXnneT+qoIxDUIbVT3kvIZS/4mzDcY1wQaGIEA3Vm7li rV8rgCV4lLM7LGq8rvfLcnwimMZ5eADpnFeOJzXkKcHB0s2qqFoxmFM1eJFjBk7ydPpnMUYgZfl ekQjsxczFhTDOz0ns6TWkT2SgKKfCPlzfPydt1Xyzwg3rhco9I8PamlNTdTNMiaUykJM2AUi/Pt ktSRBB9c/jbUobfHak3jQ12fxOCCMLTm6Vm9beafxrbbfnA3ONYRYAgx9anrRf X-Received: by 2002:a05:600c:4e04:b0:492:2fa4:2563 with SMTP id 5b1f17b1804b1-492333eb324mr16744255e9.25.1781642844922; Tue, 16 Jun 2026 13:47:24 -0700 (PDT) Received: from pumpkin (82-69-66-36.dsl.in-addr.zen.co.uk. [82.69.66.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-4606f2dbfb1sm50531509f8f.35.2026.06.16.13.47.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jun 2026 13:47:24 -0700 (PDT) Date: Tue, 16 Jun 2026 21:47:22 +0100 From: David Laight To: Peter Zijlstra Cc: x86@kernel.org, linux-kernel@vger.kernel.org, hpa@zystor.com, samitolvanen@google.com, kees@kernel.org, nathan@kernel.org, scott.d.constable@intel.com Subject: Re: [PATCH] x86/kcfi: Optimize call sequence Message-ID: <20260616214722.7742e394@pumpkin> In-Reply-To: <20260612071506.GQ187714@noisy.programming.kicks-ass.net> References: <20260612071506.GQ187714@noisy.programming.kicks-ass.net> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.38; arm-unknown-linux-gnueabihf) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Fri, 12 Jun 2026 09:15:06 +0200 Peter Zijlstra wrote: > As noted in commit 85a2d4a890dc ("x86,ibt: Use UDB instead of 0xEA") Jcc should > be assumed not-taken, however the normal kCFI (ABI) emits the following sequence: > > movl $(-hash), %r10d > addl -15(%r11), %r10d > je 1f > ud2 > 1: cs call __x86_indirect_thunk_r11 > > (when used in conjunction with -mretpoline-external-thunk). > > Notably, the Jcc here is always taken, resulting in lower throughput than would > be ideal. Replace it with the following sequence on boot: > > movl $(-hash), %r10d > addl -15(%r11), %r10d > jne . + 3 > test $0xd6, %al > cs call __x86_indirect_thunk_r11 > > This jumps to the UDB instruction used as an immediate byte in the test > instruction. The test instruction will clobber eflags, but that is immaterial, > eflags is already changed by the preceding addl. > > Intel recommends the FineIBT sequence on platforms that support IBT; older > platforms are still widely used and would benefit from this. > > An earlier PoC was benchmarked by Scott: > > Indirect branch miss rate (br_misp_retired.indirect:k / br_inst_retired.indirect:k) > > BHI_DIS_S=1 > > Benchmark Baseline IBT kCFI kCFI-opt > ----------------------------------------------------------------------------- > iperf3 UDP 0.103764 0.103180 0.104311 0.102945 > hackbench 0.000885 0.000876 0.001996 0.000826 > lmbench syscall 0.005089 0.004486 0.016990 0.005852 > lmbench fork+exit 0.018454 0.019176 0.031085 0.015153 > lmbench fork+exec 0.017147 0.021613 0.029129 0.016337 > redis 0.032220 0.032655 0.045540 0.027946 > nginx+wrk 0.109033 0.112765 0.132557 0.102417 > fio randread 0.009704 0.009620 0.008548 0.000962 > fio seqwrite 0.006927 0.006707 0.019372 0.004590 > kbuild 0.056748 0.057324 0.064640 0.048136 > > BHI_DIS_S=0 > > Benchmark Baseline IBT kCFI kCFI-opt > ----------------------------------------------------------------------------- > iperf3 UDP 0.000077 0.000106 0.000186 0.000073 > hackbench 0.000123 0.000132 0.000367 0.000097 > lmbench syscall 0.023259 0.018319 0.040903 0.012772 > lmbench fork+exit 0.011494 0.011887 0.029079 0.016415 > lmbench fork+exec 0.037782 0.038994 0.055378 0.026381 > redis 0.002481 0.003152 0.017073 0.000184 > nginx+wrk 0.015478 0.016266 0.033637 0.000268 > fio randread 0.009836 0.007949 0.007096 0.000143 > fio seqwrite 0.014587 0.014165 0.041792 0.002157 > kbuild 0.055774 0.055249 0.062590 0.046546 > > Cc: Sami Tolvanen > Cc: Kees Cook > Cc: Nathan Chancellor > Cc: hpa@zystor.com > Suggested-by: Scott D Constable > Signed-off-by: Peter Zijlstra (Intel) > --- > arch/x86/kernel/alternative.c | 11 ++++++++++- > arch/x86/kernel/cfi.c | 6 ++++++ > 2 files changed, 16 insertions(+), 1 deletion(-) > > --- a/arch/x86/kernel/alternative.c > +++ b/arch/x86/kernel/alternative.c > @@ -1356,6 +1356,10 @@ early_param("cfi", cfi_parse_cmdline); > * "Make conditional jumps most often not taken: The efficiency and throughput > * for not-taken branches is better than for taken branches on most > * processors. Therefore, it is good to place the most frequent branch first" > + * > + * NOTE: Update the kCFI caller sequence to make use of this observation. > + * Replace the "je 1f; ud2" sequence with "jne +1; test $0xd6, %al". This > + * clobbers flags, but those are clobbered by the hash test anyway. I think it would be better to give the byte sequences for both pairs of instructions - it takes a bit of sleuthing to check they are the same size. I think it would also be better it the code doing the patching checked what it was overwriting. Also, what actually generates the list of cfi locations in the first place? If it is objtool, then maybe it could do the rewrite instead. David > */ > > /* > @@ -1518,9 +1522,10 @@ static int cfi_disable_callers(s32 *star > static int cfi_enable_callers(s32 *start, s32 *end) > { > /* > - * Re-enable kCFI, undo what cfi_disable_callers() did. > + * Re-enable (and update) kCFI, undo what cfi_disable_callers() did. > */ > const u8 mov[] = { 0x41, 0xba }; > + const u8 udne[] = { 0x75, 0x01, 0xa8, 0xd6 }; > s32 *s; > > for (s = start; s < end; s++) { > @@ -1532,6 +1537,10 @@ static int cfi_enable_callers(s32 *start > if (!hash) /* nocfi callers */ > continue; > > + /* > + * See the kCFI/FineIBT comment above -- update note. > + */ > + text_poke_early(addr + 10, udne, 4); > text_poke_early(addr, mov, 2); > } > > --- a/arch/x86/kernel/cfi.c > +++ b/arch/x86/kernel/cfi.c > @@ -72,6 +72,12 @@ enum bug_trap_type handle_cfi_failure(st > > switch (cfi_mode) { > case CFI_KCFI: > + /* > + * The updated kCFI sequence has "test $0xd6, %al" instead of > + * "ud2", adjust the offset. > + */ > + addr -= 1; > + > if (!is_cfi_trap(addr)) > return BUG_TRAP_TYPE_NONE; > >