From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1720A1F95C for ; Sun, 22 Dec 2024 04:07:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734840473; cv=none; b=eZOARvIEAMnHmKu7Yvla56/Jnsy7ZppNsvvo8pgT0xtrqG651hMd2k9v39iSxKlt6CnBUTsaGbHQS2YA0wTsGAMiwoQlvc21dy4j8xKD44DUhu2IAZXPdPYJJHZ9vSOLJ6/QDCqrP0yeGadIpwWgqn0aZyt1xmlslkP3x2kHaRA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734840473; c=relaxed/simple; bh=nydV89CuZQ9tFRN+HcOno0c+hhwB/e2LSPzHMlmrbws=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=jQMpXRkA6YlqCtPDIG739jIRl10URAwXDqPF14XwCOhR3fu1j+m49fN2hu/dK0fRxnAotUcvXxiwAMaIivzJ9alpxuZuE65QNn1+O2G7JTysc/z711v514t9TvOCWjmdGR/vUvqUihCPlU+SQ4pJ4AxIzg8RfcX1+Ab4Qnw+Puo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=shelob.surriel.com; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shelob.surriel.com Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tPDFS-000000000V0-3to9; Sat, 21 Dec 2024 23:07:22 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, akpm@linux-foundation.org Subject: [RFC PATCH 00/10] AMD broadcast TLB invalidation Date: Sat, 21 Dec 2024 23:06:32 -0500 Message-ID: <20241222040717.3096835-1-riel@surriel.com> X-Mailer: git-send-email 2.47.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: riel@surriel.com Add support for broadcast TLB invalidation using AMD's INVLPGB instruction. This allows the kernel to invalidate TLB entries on remote CPUs without needing to send IPIs, without having to wait for remote CPUs to handle those interrupts, and with less interruption to what was running on those CPUs. Because x86 PCID space is limited, and there are some very large systems out there, broadcast TLB invalidation is only used for processes that are active on 3 or more CPUs, with the threshold being gradually increased the more the PCID space gets exhausted. Combined with the removal of unnecessary lru_add_drain calls (see https://lkml.org/lkml/2024/12/19/1388) this results in a nice performance boost for the will-it-scale tlb_flush2_threads test on an AMD Milan system with 36 cores: - vanilla kernel: 527k loops/second - lru_add_drain removal: 731k loops/second - only INVLPGB: 527k loops/second - lru_add_drain + INVLPGB: 1157k loops/second Profiling with only the INVLPGB changes showed while TLB invalidation went down from 40% of the total CPU time to only around 4% of CPU time, the contention simply moved to the LRU lock. Fixing both at the same time about doubles the number of iterations per second from this case.