From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55093C433DB for ; Thu, 11 Feb 2021 21:11:39 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1B1C964E3C for ; Thu, 11 Feb 2021 21:11:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1B1C964E3C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=+fgt8wu9JzRkVo7HmQ+dlG2mpWxI+YJgl1+gFmSi+BI=; b=jz/HZjdtWa8ay+DjZul68kru5 ARzlefLyp+6q2R2qrXRcMbhXUrJ1VX7I6zjJVkXUn98ULvwL5zypfr5rgg+ZiNvkBxRWCow9uDx8a p9Ji5dPv6ngE7fXAQBtzZCERCvXTE9Cfi8OiLmI9nGDJcUOlbk7MuYRLn2/MQG+P9vSZ0F+XimcVh HX576rR2XUVWBZi2yUWAyYp49+dAzxI0xaQZ/+3pBV15Q6GuPQQYbkZ+zseM6i+D66iN+no+8i6Wo saWy8xxJwuAqk3U5nMohE0YDXV9DGjfxYH3dECI7fKeFLfgKH9kOt9o5ATweVw+O0WIO9tbcOCHDE DPSaf9Gjg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1lAJFM-000455-Qp; Thu, 11 Feb 2021 21:11:32 +0000 Received: from hqnvemgate26.nvidia.com ([216.228.121.65]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1lAJF7-00040Z-VV for linux-nvme@lists.infradead.org; Thu, 11 Feb 2021 21:11:21 +0000 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Thu, 11 Feb 2021 13:11:17 -0800 Received: from HQMAIL109.nvidia.com (172.20.187.15) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 11 Feb 2021 21:11:08 +0000 Received: from vdi.nvidia.com (172.20.145.6) by mail.nvidia.com (172.20.187.15) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 11 Feb 2021 21:11:03 +0000 From: Boris Pismenny To: , , , , , , , , , , Subject: [PATCH v4 net-next 03/21] iov_iter: DDP copy to iter/pages Date: Thu, 11 Feb 2021 23:10:26 +0200 Message-ID: <20210211211044.32701-4-borisp@mellanox.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20210211211044.32701-1-borisp@mellanox.com> References: <20210211211044.32701-1-borisp@mellanox.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210211_161118_448361_F9F9461A X-CRM114-Status: GOOD ( 12.97 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Yoray Zack , Boris Pismenny , yorayz@nvidia.com, boris.pismenny@gmail.com, Ben Ben-Ishay , benishay@nvidia.com, linux-nvme@lists.infradead.org, netdev@vger.kernel.org, Or Gerlitz , ogerlitz@nvidia.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org When using direct data placement (DDP) the NIC writes some of the payload directly to the destination buffer, and constructs SKBs such that they point to this data. To skip copies when SKB data already resides in the destination we use the newly introduced routines in this commit, which check if (src == dst), and skip the copy when that's true. As the current user for these routines is in the block layer (nvme-tcp), then we only apply the change for bio_vec. Other routines use the normal methods for copying. Signed-off-by: Boris Pismenny Signed-off-by: Ben Ben-Ishay Signed-off-by: Or Gerlitz Signed-off-by: Yoray Zack --- include/linux/uio.h | 17 +++++++++++++++ lib/iov_iter.c | 53 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/include/linux/uio.h b/include/linux/uio.h index 72d88566694e..8438e6d5ac25 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -123,6 +123,9 @@ size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i); size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i); +#ifdef CONFIG_TCP_DDP +size_t _ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i); +#endif size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i); bool _copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i); size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i); @@ -137,6 +140,16 @@ size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) return _copy_to_iter(addr, bytes, i); } +#ifdef CONFIG_TCP_DDP +static __always_inline __must_check +size_t ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +{ + if (unlikely(!check_copy_size(addr, bytes, true))) + return 0; + return _ddp_copy_to_iter(addr, bytes, i); +} +#endif + static __always_inline __must_check size_t copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) { @@ -265,6 +278,10 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct io bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i); size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i); +#ifdef CONFIG_TCP_DDP +size_t ddp_hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, + struct iov_iter *i); +#endif struct iovec *iovec_from_user(const struct iovec __user *uvector, unsigned long nr_segs, unsigned long fast_segs, diff --git a/lib/iov_iter.c b/lib/iov_iter.c index a21e6a5792c5..c5990275e583 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -473,6 +473,18 @@ static void memcpy_from_page(char *to, struct page *page, size_t offset, size_t kunmap_atomic(from); } +#ifdef CONFIG_TCP_DDP +static void ddp_memcpy_to_page(struct page *page, size_t offset, const char *from, size_t len) +{ + char *to = kmap_atomic(page); + + if (to + offset != from) + memcpy(to + offset, from, len); + + kunmap_atomic(to); +} +#endif + static void memcpy_to_page(struct page *page, size_t offset, const char *from, size_t len) { char *to = kmap_atomic(page); @@ -625,6 +637,26 @@ static size_t csum_and_copy_to_pipe_iter(const void *addr, size_t bytes, return bytes; } +#ifdef CONFIG_TCP_DDP +size_t _ddp_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +{ + const char *from = addr; + if (unlikely(iov_iter_is_pipe(i))) + return copy_pipe_to_iter(addr, bytes, i); + if (iter_is_iovec(i)) + might_fault(); + iterate_and_advance(i, bytes, v, + copyout(v.iov_base, (from += v.iov_len) - v.iov_len, v.iov_len), + ddp_memcpy_to_page(v.bv_page, v.bv_offset, + (from += v.bv_len) - v.bv_len, v.bv_len), + memcpy(v.iov_base, (from += v.iov_len) - v.iov_len, v.iov_len) + ) + + return bytes; +} +EXPORT_SYMBOL(_ddp_copy_to_iter); +#endif + size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { const char *from = addr; @@ -1566,6 +1598,27 @@ size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, } EXPORT_SYMBOL(csum_and_copy_to_iter); +#ifdef CONFIG_TCP_DDP +size_t ddp_hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, + struct iov_iter *i) +{ +#ifdef CONFIG_CRYPTO_HASH + struct ahash_request *hash = hashp; + struct scatterlist sg; + size_t copied; + + copied = ddp_copy_to_iter(addr, bytes, i); + sg_init_one(&sg, addr, copied); + ahash_request_set_crypt(hash, &sg, NULL, copied); + crypto_ahash_update(hash); + return copied; +#else + return 0; +#endif +} +EXPORT_SYMBOL(ddp_hash_and_copy_to_iter); +#endif + size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i) { -- 2.24.1 _______________________________________________ Linux-nvme mailing list Linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme