From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7F9EBC61D97 for ; Fri, 24 Nov 2023 16:00:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Date:Cc:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yidagCP8UyPvp6nkblfXgZ3m20Ijm5tsp2zMtrgje5U=; b=nMcGg69fIoB8eR NiyFWGfh2bdEbU7qY2OdPA2Sn/Ev3FOWLxEDIUgzPsO0jzQuck4L6rLYeVKv8fAKvkJkx1RlAvgdL SGCQM5T6ZMs5O0mL97YK9BGnfPEqkfClH1yiB19ifMzXmvikhvgGvNGA689EeAlD7m+0EMlVip/Tt ufVinUH/psOI80pOs6PIdXSnn1FewBLejlvxcRwunJUr3+8J0PmXzSAP19WIBHwYuMwzDfQcApmgn 7C6RrB0A+suCp7CLeWdodGASuQKYs0P1RkEOkA8MrRGkDxjVK7z1MzDfbBX5MaGYnLW/R88XFF/MP pYzc+UwFwe0CmOLLlEmA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r6Yax-007b9R-0w; Fri, 24 Nov 2023 15:59:55 +0000 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r6Yau-007b91-2Y for linux-arm-kernel@lists.infradead.org; Fri, 24 Nov 2023 15:59:54 +0000 Received: from pps.filterd (m0353728.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3AOFv2lX015477; Fri, 24 Nov 2023 15:59:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : subject : from : to : cc : date : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=pp1; bh=rnorhOgJqTAhJ2ff91jA+Eihqw7bxMBbiU6K3KlQoiY=; b=FvHJjHxZO1InjBWtTGTI30is4uOBHWOyOp6bV551FD0+YTRF+W+/tLYyvGuLpvqgB0xX 7BYq345nQ/2pLZKLIvMr73mFM14zZQFXoblTfWVP/yCHnJhHZy3mKzrn9NSNnnkDeCh+ DEzK63fhOQPAHJMAXrCGndH695gHWu0s5gAEPbagASHalAsBWIEqJmdnM7L8leeWaKdE wwXeKbCR7TTQ5LoAiihSw7Y8xGk3vkDBoMZvnV9Si6eo0tDKqJ3A0jVm0zC/k9uF7WwE JjCI/H7d1949eBtjGYVVi3fNrJ4qi/AHPt47APPNgfmAbLQLKR1830aMqgp56fpWk9kX yA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3ujxx101t4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Nov 2023 15:59:42 +0000 Received: from m0353728.ppops.net (m0353728.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3AOFxEqk021637; Fri, 24 Nov 2023 15:59:42 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3ujxx101sa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Nov 2023 15:59:42 +0000 Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3AOF5xbU015327; Fri, 24 Nov 2023 15:59:40 GMT Received: from smtprelay05.fra02v.mail.ibm.com ([9.218.2.225]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3uf8006tda-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Nov 2023 15:59:40 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay05.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3AOFxcnA18088654 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Nov 2023 15:59:38 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8E8BF20043; Fri, 24 Nov 2023 15:59:38 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4EAB420040; Fri, 24 Nov 2023 15:59:38 +0000 (GMT) Received: from [9.152.212.236] (unknown [9.152.212.236]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Nov 2023 15:59:38 +0000 (GMT) Message-ID: Subject: Re: [PATCH rdma-next 1/2] arm64/io: add memcpy_toio_64 From: Niklas Schnelle To: Jason Gunthorpe Cc: Leon Romanovsky , Arnd Bergmann , Catalin Marinas , linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-rdma@vger.kernel.org, llvm@lists.linux.dev, Michael Guralnik , Nathan Chancellor , Nick Desaulniers , Will Deacon Date: Fri, 24 Nov 2023 16:59:38 +0100 In-Reply-To: <20231124145529.GG436702@nvidia.com> References: <20231124142049.GF436702@nvidia.com> <14103e31e0c47c0594e7479126ce7fe34f2de467.camel@linux.ibm.com> <20231124145529.GG436702@nvidia.com> User-Agent: Evolution 3.48.4 (3.48.4-1.fc38) MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 3KltwUJsWxyHpdOewx-yp-U8XnIdt2m9 X-Proofpoint-GUID: Bk8fkUCCN1I4BSgtnI337b9eqIR7URmK X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.987,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-11-24_02,2023-11-22_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 lowpriorityscore=0 spamscore=0 phishscore=0 suspectscore=0 malwarescore=0 priorityscore=1501 mlxscore=0 bulkscore=0 clxscore=1015 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311060000 definitions=main-2311240125 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231124_075953_049774_019B36AC X-CRM114-Status: GOOD ( 32.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, 2023-11-24 at 10:55 -0400, Jason Gunthorpe wrote: > On Fri, Nov 24, 2023 at 03:48:22PM +0100, Niklas Schnelle wrote: > > On Fri, 2023-11-24 at 10:20 -0400, Jason Gunthorpe wrote: > > > On Fri, Nov 24, 2023 at 03:10:29PM +0100, Niklas Schnelle wrote: > > > > > > > What's the reasoning behind not using the existing memcpy_toio() > > > > here? > > > > > > Going forward CPUs are implementing an instruction to do a 64 byte > > > aligned store, this is a wrapper for exactly that operation. > > > > > > memcpy_toio() is much more general, it allows unaligned buffers and > > > non-multiples of 64. Adapting the general version to generate the > > > optimized version in the cases it can is complex and has a codegen > > > penalty.. > > > > I think you misunderstood me. I understand why you want a separate > > memcpy_toio_64(). I just wonder if its generic implementation shouldn't > > just be a define or inline wrapper for memcpy_toio(addr, buffer, 64). > > Oh, yes, I totally did. > > I'm worried that x86 will less reliably generate write combining with > it's memcpy_toio implemention. It codegens byte copies for that > function :( Oh ok I see what you mean. > > > Also seeing the second patch of course that would no longer really test > > for write combining for us which we can also do but I think that's okay > > and you're probably going to use memcpy_toio_64() in more places and > > there we really want the PCI store block. > > Right now we don't have in-kernel performance use cases for write > combining for mlx5. Is the code in patch 2 performance critical? > > Userspace uses the WC and we already have the special 390 instructions > for batching in rdma-core already, IIRC. Yes, I added that support to rdma-core :-) > > So it would be appropriate for s390 to use a consistent path. > > Jason This should be as easy as adding #define memcpy_toio_64(to, from) zpci_memcpy_toio(to, from, 64) to arch/s390/include/asm/io.h. I'm wondering if we should do that as part of this series. It's not as good as a special case but probably better than the existing loop. I don't think we have any existing in-kernel users of memcpy_toio() on s390 so far though so I'd like to give this some extra testing. Could you share instructions on how to exercise the code path of patch 2 on a ConnectX-5 or 6? Is this exercised e.g. when using NVMe-oF RDMA? Thanks, Niklas _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel