From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C81E7D2126D for ; Thu, 17 Oct 2024 09:59:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:Mime-Version:Date:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=NL1g8QiMiw+GMLkaabWajcTuwbPu8SeRvj7pvi19l7E=; b=5At+MlA2P/objivuU01h3yyknu I+Y3lR+S69oF/b25f87YBQwTCItGWhlgZg6HDsxgCJaAffNQ5KytvrITQq3TwcCdm8SrJqzobZYN6 f3fFQNbzaP3bJ8MPHdo57MoJYo7bqSKehU+ps1JvILaCyuLwU2E9c+Mim9mnCGhiSKD77wPbBauzf mncYN4t1dLlOXOOJGtNhNaHS9U2X155HKsfj/LTA2E+gb0sP3RyjPS+yZJ18drQ+02ICYuGrExSmL PSNuni4Tc/OLEVhL8ffdlMZ2rqN6Hg4QPafVtITtCTASpn4691wTGylT3lbKI4wIhi7snVy2w8yfi n+947wxg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t1NIH-0000000ERxN-45eS; Thu, 17 Oct 2024 09:59:45 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t1N0n-0000000ENMT-0Cbk for linux-arm-kernel@lists.infradead.org; Thu, 17 Oct 2024 09:41:42 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-6e3529d80e5so12930297b3.0 for ; Thu, 17 Oct 2024 02:41:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729158099; x=1729762899; darn=lists.infradead.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=NL1g8QiMiw+GMLkaabWajcTuwbPu8SeRvj7pvi19l7E=; b=4DlNiQC+m7xThcPzoHRd7OOPeAXly1P0PweIuwkNtdI9cbOxWrmeEB6G3KLSGB3nmV enMLLW/CQ9wUEs4Qm388D9PqfU3OC6l2UWsn66OeAv8G+c99T7IhElXd+07LOGh0m7Zy up0ExJA/wG4j5jfyQBR0R2DV7rS4AGtfmwIHNaTYsnDcIPBlvyxYdYYxSo6dfM+27VsO K8u/SS42wGcbum/GPGx9yOxK4QHrLrBemRCiU9+M/1E0ipq6i0GyKsyAdTKHDQO18af5 6lhtWLXMxmDD8CyAkMK8pJZ91hcN/W/xxpfhsqgj4txjZu4i5Ks6l77tWgjgUouuhd1I mdaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729158099; x=1729762899; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=NL1g8QiMiw+GMLkaabWajcTuwbPu8SeRvj7pvi19l7E=; b=H0aqeXZNfOyFkXU/PUNOI7LxHAA8tf10O8jb3wFpFf44JDbO6I565TwQcvyDjiWm0p m7xpsHaw5kWWmKzAURnV/6f9/fsc3VUZhJHKuKlZbPBX4x7yM6qJJ5ifHum+q73HeDNg NEmtcOT8EJg4CMrRGyhnjEZFwxipcu0gWDGM3xk4nHhW4qZJYcbeMne318pDgR1G02n6 XlFSxFR0eGk7SdvnegdOMQrdIps6oZ9NiglY7Ol0pse9U1g10HtUgnae5DoEcmIaRQka pDKZYICVj6ptoOuzUM1BCLlxTBkXXVVzRKBc2gVCi+zaMaYtxGGPdL0shnz7yfhb1D4Q Taog== X-Gm-Message-State: AOJu0Yx6ud0+4zyG/SYHS+DT1Y+WZm+ypDpy5xFt2+ngN68nEEzZknbb 6iRfathoaPIEe+8uG5qDRtFhB9Y/lGxqX5xWqW6ACvmsok9kmCwqkshFpF59cwXhNcIKJYXtVN/ fWuV7GS5GQDoPQJKy7mo1weDKNCDBtbGwyQGHOkWKx7G05nUIBPqMM2dE8UTvPrmg+cTfGOTzYA WHFhTCvkRPQ7hDjsO3a1YoBqAzZICUArIZT8thASkr X-Google-Smtp-Source: AGHT+IGjgTrn0OdrhHX3rWorpZYzZht2bXo+DDqn8QkiQ+7uwl+ZSKCW9wNflH8e1sSt3+MKmVzadN5H X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:7b:198d:ac11:8138]) (user=ardb job=sendgmr) by 2002:a05:690c:6703:b0:6dd:bf69:7e06 with SMTP id 00721157ae682-6e3d41e7a44mr1259657b3.7.1729158098641; Thu, 17 Oct 2024 02:41:38 -0700 (PDT) Date: Thu, 17 Oct 2024 11:41:33 +0200 Mime-Version: 1.0 X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1748; i=ardb@kernel.org; h=from:subject; bh=A2qb4yG344KAaBkQYaztyqpHjkQf1nkWoa4ObJtOQZo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIV3g9lnPuYlevqv5zz9+lbff6fRSr3zNzayiJ40XqJQZy Rv8M/7YUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACZy0YfhfyqHwLsTiTP493Ex Tzi9b5/49XDvbRGPgvp/Hbr7u/x0OyPDP7XrDJH+8Y8ZP014WDnzUmzswtbdZ1STtmSrdtg77ng SzwMA X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241017094132.2482168-4-ardb+git@google.com> Subject: [PATCH v3 0/2] arm64: Speed up CRC-32 using PMULL instructions From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, will@kernel.org, catalin.marinas@arm.com, Ard Biesheuvel , Eric Biggers , Kees Cook Content-Type: text/plain; charset="UTF-8" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241017_024141_121618_F6ECEB1C X-CRM114-Status: GOOD ( 12.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel The CRC-32 code is library code, and is not part of the crypto subsystem. This means that callers may not generally be aware of the kind of implementation that backs it, and so we've refrained from using FP/SIMD code in the past, as it disables preemption, and this may incur scheduling latencies that the caller did not anticipate. This was solved a while ago, and on arm64, kernel mode FP/SIMD no longer disables preemption. This means we can happily use PMULL instructions in the CRC-32 library code, which permits an optimization to be implemented that results in a speedup of 2 - 2.8x for inputs >1k in size (on Apple M2) Patch #1 implements some prepwork to handle the scalar CRC-32 alternatives patching in C code. Changes since v2: - drop alternatives.h #include (#1) - drop unneeded branch (#2) - fix comment max -> min (#2) - add Eric's Rb Changes since v1: - rename crc32-pmull.S to crc32-4way.S and avoid pmull in the function names to avoid confusion about the nature of the implementation; - polish the asm a bit, and add some comments - don't return via the scalar code if len dropped to 0 after calling the 4-way code. Cc: Eric Biggers Cc: Kees Cook Ard Biesheuvel (2): arm64/lib: Handle CRC-32 alternative in C code arm64/crc32: Implement 4-way interleave using PMULL arch/arm64/lib/Makefile | 2 +- arch/arm64/lib/crc32-4way.S | 242 ++++++++++++++++++++ arch/arm64/lib/crc32-glue.c | 82 +++++++ arch/arm64/lib/crc32.S | 22 +- 4 files changed, 331 insertions(+), 17 deletions(-) create mode 100644 arch/arm64/lib/crc32-4way.S create mode 100644 arch/arm64/lib/crc32-glue.c -- 2.47.0.rc1.288.g06298d1525-goog