From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FD243750DB for ; Sat, 13 Jun 2026 08:59:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781341176; cv=none; b=Avei0pgRnN05MIzKclN32NU4q1WpdWS2b3O3x7uyUVdXXAo75o631SLXIhkdhZCFx9jLYMtLcH0I6iRXa0XJ5lqSoAJovOmMuv4ymUWJNTesr5THKm3eAHgyhfVhX6p9VBI7Q87aaKWqWxElMCJ8dWzY3EUBbeaZ2oo80UXloD4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781341176; c=relaxed/simple; bh=I2Cki194xus5u9PRaFnpS2f4l/V98qJeWxZH1ImtzZI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=UrhzIcB4Brj3B6RGpKNQ+k7kYDKuxrAd62WD2FjjsF68a3Kcy0tPVqcVm+SfAaef5Zks2Xi6mCmxm6rR1VizgX2Hu7KwL6XBoxEIkzHlodsq/fCBmQ5CCa8KGiOB60fTBAqOm7n93sqgrjoA1TtsotinhJBCa1joD18ce0ZadoU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=PmzchtVG; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="PmzchtVG" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=mPQH4A5axVGcf+ZnAnbkcWN+MIuur3wZpLelQwJZCeM=; b=PmzchtVG/WmsCx/Pga5qI9z3jM QK+VoaOks/apYLQpcvjLVZ9HBHijs0xV1iIoMOed41Vt5aIO3aF4II3v1E2FcNBoVk+udnwZhLJG+ mH9eEd2Ga5YJBtu9BB+eSVSGFx1ZByi9UENqB46GzESwyWBds6eq4jx7zQMlEGpLPqL36eT6bkn7z JysU63GNos6oeKbHo4DfE7Mq9p6g+aHv1YY16hMgtfVksI77dmGKXuR5Gh2ADgqmEsjaSFcNF48pm HU4CbFfQwsWXbRgJTP3+BZZKSfMOGDnExZ3x1iaXnX0719ci3SltyOqgIGOvi9GFFWXIbinjZQ7F0 u38E4uQg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.99.2 #2 (Red Hat Linux)) id 1wYKD3-00000007M2e-1Nhi; Sat, 13 Jun 2026 08:59:22 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id C340430036F; Sat, 13 Jun 2026 10:59:19 +0200 (CEST) Date: Sat, 13 Jun 2026 10:59:19 +0200 From: Peter Zijlstra To: "\"H. Peter Anvin\" (Intel)" Cc: tglx@kernel.org, mingo@redhat.com, bp@alien8.de, Nathan Chancellor , Calvin Owens , Dave Hansen , torvalds@linux-foundation.org, x86-ML , LKML Subject: Re: 8aeb879baf12 - significant system call latency regression, bisected Message-ID: <20260613085919.GF42921@noisy.programming.kicks-ass.net> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Fri, Jun 12, 2026 at 06:45:06PM -0700, "H. Peter Anvin" (Intel) wrote: > So I was trying to figure out a significant -- about 13% -- increase > in system call latency between v7.0 and the current master, and it > bisects down to: > > 8aeb879baf12 x86/kvm/vmx: Fix x86_64 CFI build > > This is on Panther Lake (Core Ultra X7 358H) with FRED enabled. This > is a bare metal boot, no KVM. > > I'm personally extremely puzzled how this could possibly be related, > and I will be investigating the possibility that this is a false > bisect, but it is not a Heisenbug in any way; it has been extremely > reproducible, and the difference is statistically valid by close to 10 > sigma. Futhermore, the bisection at least gave the appearance of > stability. > > Given how late in the cycle this is I wanted to send an alert sooner > rather than later; I will update as I get more data. Uhm, massive WTF indeed. I don't immediately see how this could possibly affect a FRED host either, except perhaps in code layout. I don't actually have a FRED capable machine, but have you tried running one of those top-down perf things on it, to see where its hurting?