From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A82841A4B42 for ; Wed, 31 Jul 2024 07:31:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722411080; cv=none; b=QBuQB7SUL0ScW+JJ1aIvkqvwgLnwAa8mh80z+OMjPOfIVYeUNdyW4CUrRbd8ggyt2b5bDrtqwc0LFW3YanJkWteu+ww2JvOFiLqF7unvoR2M7NJooTbiJrVU0YHUNIbm3ZL5WlOsnuS1IilMGDJqkYCngiv+7ANTC6joX3hEnyY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722411080; c=relaxed/simple; bh=BVlRaG9oeJyaltVO2VfM7RNLo0pxUx9YDfxmMHqs3GI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iNBsEdwGDgRts462adVV8a2WE4io3WdldLfioDhk+oYKq38BJ1sd+lxm8fh6StIuz2lpCyafC9txQRbx0kg1lPkWGSf0fSKRpWwjJ71C7hLDrUyS/9JlNtozgAF7J6bXNcPEcu6ly7AKcHJZZ32X34bGPkCRf4F8tlnUy3UHipg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=3QavZXbU; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="3QavZXbU" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-428085a3ad1so35021485e9.1 for ; Wed, 31 Jul 2024 00:31:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722411077; x=1723015877; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BHoHYllmH7EYuYJy+Jv8B241H4q5/CnBJ4HnToKFT0c=; b=3QavZXbU6dp4/R6igZ+eopofuf9EhloOCsSGGSswHlTKDwTfYImHTyePbg5uq1ePxS lceCEQLiv7wbirFvlORY9RzcdA99BCs/QcthiT2BL46ZqSxRLqUbNS6kuhlwZSu6P9Mn HkvlottjQlVhkThOVzhITsn6PcvCxm6Lj000ZakiY05pQhs4Eywep8empxaBXm602anK wZaqvDSL+hqAefafL95sdXuTnE2uFAiCkSH4YBQsk8IEnMBRKOkXtLCGVWSsuz30otDk WT11uqkjfPXbiLu/tuBgZVCghF3flYVb5JodCejJ8GDRywqlB/YLW/5gQKlJkQrmuIuk 172Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722411077; x=1723015877; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BHoHYllmH7EYuYJy+Jv8B241H4q5/CnBJ4HnToKFT0c=; b=iyiXTJTLyKa15Qj1u92tWPNs/poWlPx6eThMrONEx5YXu10I/6Y9+T/NsvDyED704u HJaGymKwke3L1mn7iTD4vfrK6hPtO/09I3a+o/INQkCGuiV+vQ4AAM6LclzphOwAoQLC xj2KN2OWb9hcmZmeMuqqT5Nx+zX26RI7iqfG5aQdAQ/LnA/habCJQgR27R/pBKJEyXFy rIUAECec521FCvlX08Jr4uRRj8McVNwyOygKpnJHAQkrMLpdsQ7Urf42m70KrVBasiQp lvbXyADvaqA0R7VFvG9dlfgNE/1hjzVFXVC79bbdLgtyw3dbChgDhMknIpxl7VNPlyh8 BD6g== X-Forwarded-Encrypted: i=1; AJvYcCWzt778+I6HZLRN8FVp7Z3dOi5WXqQQNeiSrzLiIeRf5i35Hx70vHKsjYCgekHTc0rG7fswt2tAZmDC8BEgghvuYXKzQJDO0hvRAQ== X-Gm-Message-State: AOJu0YzonGOStog4q5i/fWOJWRzwq2GhE7TdZuMGhpUnxQ47p5/CyLIM b+EIwpfWHZZ+gejU3SsrvlHY+lS6OHewIynaYP4eawEAb3GTsG90aEhPgzg+Z1o= X-Google-Smtp-Source: AGHT+IFWKmJ8e4ndadH473CTdPU3AlRXobOboTCXTtcOkKGrZDLPp8Y0XiEkCzxGJJj4GbuZsEuDpQ== X-Received: by 2002:a05:600c:4447:b0:426:5c34:b19b with SMTP id 5b1f17b1804b1-42811d9af49mr92305945e9.20.1722411076723; Wed, 31 Jul 2024 00:31:16 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4282bac8069sm10520875e9.28.2024.07.31.00.31.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:31:16 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 07/13] riscv: Implement arch_cmpxchg128() using Zacas Date: Wed, 31 Jul 2024 09:23:59 +0200 Message-Id: <20240731072405.197046-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: devicetree@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Now that Zacas is supported in the kernel, let's use the double word atomic version of amocas to improve the SLUB allocator. Note that we have to select fixed registers, otherwise gcc fails to pick even registers and then produces a reserved encoding which fails to assemble. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cmpxchg.h | 38 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 212ec2aab389..ef55ab94027e 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -111,6 +111,7 @@ config RISCV select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 391730367213..ce9613516bbb 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -276,6 +276,44 @@ arch_cmpxchg_release((ptr), (o), (n)); \ }) +#if defined(CONFIG_64BIT) && defined(CONFIG_RISCV_ISA_ZACAS) + +#define system_has_cmpxchg128() riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS) + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(p, o, n, cas_sfx) \ +({ \ + __typeof__(*(p)) __o = (o); \ + union __u128_halves __hn = { .full = (n) }; \ + union __u128_halves __ho = { .full = (__o) }; \ + register unsigned long t1 asm ("t1") = __hn.low; \ + register unsigned long t2 asm ("t2") = __hn.high; \ + register unsigned long t3 asm ("t3") = __ho.low; \ + register unsigned long t4 asm ("t4") = __ho.high; \ + \ + __asm__ __volatile__ ( \ + " amocas.q" cas_sfx " %0, %z3, %2" \ + : "+&r" (t3), "+&r" (t4), "+A" (*(p)) \ + : "rJ" (t1), "rJ" (t2) \ + : "memory"); \ + \ + ((u128)t4 << 64) | t3; \ +}) + +#define arch_cmpxchg128(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), ".aqrl") + +#define arch_cmpxchg128_local(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "") + +#endif /* CONFIG_64BIT && CONFIG_RISCV_ISA_ZACAS */ + #ifdef CONFIG_RISCV_ISA_ZAWRS /* * Despite wrs.nto being "WRS-with-no-timeout", in the absence of changes to -- 2.39.2