From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C398C4332F for ; Fri, 15 Dec 2023 03:31:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AA028D0107; Thu, 14 Dec 2023 22:31:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 25A0F8D0103; Thu, 14 Dec 2023 22:31:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 121F58D0107; Thu, 14 Dec 2023 22:31:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0272C8D0103 for ; Thu, 14 Dec 2023 22:31:04 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C490E403D5 for ; Fri, 15 Dec 2023 03:31:03 +0000 (UTC) X-FDA: 81567626406.26.5BD6FF1 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by imf04.hostedemail.com (Postfix) with ESMTP id 056D34000B for ; Fri, 15 Dec 2023 03:31:01 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dBxcmljA; spf=pass (imf04.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.167.176 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702611062; a=rsa-sha256; cv=none; b=DVgxgiasvrOJXdXYoPCwwzprp6KF/XCECY+WLDsAeo9kTOQSxuqiJT3B0zRtfcRqdthK8a ooNKqti39ilyMk1XxVbQkOxPidEh1oXsPYbbgyaOuRozinPyZNNlZcZ0towWwdfzQ4Xmg+ yt2LYXy2S+xjU4FOVHCj57KL1b/G+js= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dBxcmljA; spf=pass (imf04.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.167.176 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702611062; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wH8JE7RX2AG09Ee8T7TBD98rU2IlRL47FNJ/XRL4ggY=; b=Lrxz1ynWvd5NGbUcgz+5IBKAEHTXd0pAzmJg8BPGhBIuh3P9vU0tvWdntefN09WeXyMDDx AtZOR9UkrVjgMDWUHgdocLnrWNopNTghHen6C3QrHybRPQ/5ZFSo0LJifg8WT/ieI/MZyQ 3eafWE4jSuwi2ywl5SJHTGwwqiYmEFQ= Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3b9f8c9307dso253199b6e.0 for ; Thu, 14 Dec 2023 19:31:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702611061; x=1703215861; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wH8JE7RX2AG09Ee8T7TBD98rU2IlRL47FNJ/XRL4ggY=; b=dBxcmljA4DwF/1I+akK8Scp7PuxvMfHKS3CtFpRvNp51gmZc0o3Coo8k65xeqriZC8 xYeGx46BmNqtGpWUdS2SfF9HSxywz6V4xPEiNxKnveQHykK0xwPmuYZ10fhdT55GQG1F omukUgQ3Mlust39E0h340liHzSuHCHgL3NTKqZNKQ7p9T82efxyIZp8yVoMgyPvbRHAq 663yy7o1p8LsXHbui3VnRdlArrerYiRwGS+s+z+cY3WXmJZflyiWC5K7T246i9jq0ggp +Nk/PHZpbnKKxrHMFVx5ZwFmcgVqNc3e3Cpmk66vpxffBrjDL9RWo0WCZmPFTvm1QHJc 8KDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702611061; x=1703215861; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wH8JE7RX2AG09Ee8T7TBD98rU2IlRL47FNJ/XRL4ggY=; b=hNtWG81/8Gu1LeHzAxxldG4cS8HzsbqzLO3DeFaf3nFNB9aVH7J+h9d+aqdnFekHdq 0Pi1xLjpQO82nzaxHpyxg4GE68NiED6b738eZsfgCuldmhl/wF0n01GCqGw/ZlEXUqQ+ Sx9bSXsk7g45fCXcvWYusAOUzB62QbqGh0WAycyKTVaRvDWVrkMXuku0rQ6jRUzklo37 D8fE5W4lR4BCoi06HSwu3/JOZUAaYocgoIVxMIeLvJhTl1ddtgXFG4ncoLlFCF4g4fjL XE/NpmL/dKZKA9vWt+xMeZIs7pR4XYALohSRkvDOaN7kHG3qlFeEvKyOgxR0AnoNJU9K djBw== X-Gm-Message-State: AOJu0YynRbJH9jneFq9HYhre9AK7sxYHKnKEHmG0VjuBpB6l6WNrbFXd rzjPWZhalivcdY3DQRNzdvc= X-Google-Smtp-Source: AGHT+IESZfpT6HwFqjKqBUdBuBfxsgq6TarG/TO3M212JYVWWelKU8fvA5JObVa8HMPeiL8XmcFBeA== X-Received: by 2002:a05:6808:1b1f:b0:3b8:b73b:d991 with SMTP id bx31-20020a0568081b1f00b003b8b73bd991mr14586506oib.52.1702611061135; Thu, 14 Dec 2023 19:31:01 -0800 (PST) Received: from localhost.localdomain ([23.104.209.6]) by smtp.gmail.com with ESMTPSA id v15-20020aa7850f000000b006ce467a2475sm3702775pfn.181.2023.12.14.19.30.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Dec 2023 19:31:00 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, jasowang@redhat.com, almasrymina@google.com, liangchen.linux@gmail.com Subject: [PATCH net-next v11 3/3] skbuff: Optimization of SKB coalescing for page pool Date: Fri, 15 Dec 2023 11:30:11 +0800 Message-Id: <20231215033011.12107-4-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231215033011.12107-1-liangchen.linux@gmail.com> References: <20231215033011.12107-1-liangchen.linux@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 056D34000B X-Stat-Signature: ainmpzbfop9ntdmj8nzrsxzysw31iswp X-Rspam-User: X-HE-Tag: 1702611061-791724 X-HE-Meta: U2FsdGVkX1/v08J+bru/Je+nFL24xxIrYaq+BulvkJ9vtZrHi4VjV58BR+63p4+ZkD2QFHek1xpIwtelOtTSQZq/xIIKSBxpjYR0rH1YPc1HoGs5GuYxZIXNDGVfngXnFuMqL20/hz1f7uEOj+aeEhnOvFaQoP10kgdZvKo2F7KoQ7sVd4Ix6bJnUXiGXDVULR17FjUb/6Cf+40Zv47+DS/AifgAlG7ro7g1QRwEf3/wV8tj/MkRDJDmTylIGRoNjqXC1d1BVQV9k+whQ08i/zOrA+4VkIp58GE1VeydpGH7C8ep6m2Gc8OD/JG32phKa0u3Lkh1NTWd447iJuD8MDZWi3KGoMMy9W3gDmlBiN02xSWFSL+RETWsz8/aHTB4VJ6XnGqsDvlMzHO5wXy4HoZ6HKcUIO65jvqhc22M3alMF5A0hb2I88srH7y7Y3pzsMO063SJU60ra1E4UcjS+8a3K5mBPtpM24gYDDIEe2UOKkhFAI0hJZcYrQkhOP7reF3SEqkC/zMcq4mkQIHVeYFsrSi/roJ3fP1kp5M8nuyFI+FH43i1iy4yPRstxgOmSTMksu41tNvOZ5szgmRSakIvIHXnjHJI3fh3YEL3LAkI/lCtOScIOPXYOLCP9V3psJMXKrkJ0cQo27PgsSTjbYBwsaYjiUyB2WjfjOYdBCb1V00WmhVBKcxwTisYe9A70/5froYC44q7FKbyWEQwHliuFXVF55dBEWfQkUjz9oJYUKvINvJgliAqQg9/69hxpLF8W3P0HySzIBdMQwkaKsY3AxwmtqSOSt1fWgwI4v66pRXsoWcObK7b/QRizKyJqeM8zYfALxVWBrKyN0n/CpaLUp1UMQIV+H5ljBOTJFUe4KVfbe54JSMohYa9ogaf61Jc6qS6KIKXFCrpvGAeThLbJXWiSYIJoqbxxmbpwqSFfi9f5WjiGiuDuMPmTIkHBN+BQBHRTiDgrY+Mg7l 9865ySRj OMresPOC0873F+eeu11DfML3v+rLAiOJN+2KDLEXuZdCJHBFBWl/GjrEXGd2+E4zM3UUaknajsLPPHAEj3ALQLlwAF81cdvngmJ8E0lpzYiVMC37CDgpcaHkSXFoRWGoY4eXXiu2FdMqvjVohZ2fzbVqncMFu2bMVI2BbUBQcGbp8fmTn2vDzlnywvlfX8+7YdXca5Zjx7ngEbHNAHAOMstAOF44gj5oRo/bKDm9AdITm98mWp/p/rmaLg/VkufMwKsoPibIWMttlNlZ8eSQhRhe51rhhZHAIsmr3KWSNYVn9ohKad1MFw+WWUJhyac+gC5HpEq7F6EjGjz3DmYCojCfoFdDKuP2tCUE3GnHcks/NiZo87LIqhmxXFvetT6h31ZykB2Mhp5t/vbJg10qP6wIR3kjQoR0mv5Q2kYJpFxnok7EumpvyQwJh2TQXxqCvQJ6YBxifrWVoYSUlhSFV3r/Ncn0dWBGFsEG9Mea6f4CMrvsWyhkp/N5b/xBTK0X3LY1LbwL4uqOhaELWcIHjgZtxp+m3D2vqAyQGblGqAvoiuRPDnxhkAbUtW5pUa6iN7VVWKzqsgICXyoajvbzMlDRzXc+Lj8hZUYpj0HGhIqpQ80XhXSIAsxSTvw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to address the issues encountered with commit 1effe8ca4e34 ("skbuff: fix coalescing for page_pool fragment recycling"), the combination of the following condition was excluded from skb coalescing: from->pp_recycle = 1 from->cloned = 1 to->pp_recycle = 1 However, with page pool environments, the aforementioned combination can be quite common(ex. NetworkMananger may lead to the additional packet_type being registered, thus the cloning). In scenarios with a higher number of small packets, it can significantly affect the success rate of coalescing. For example, considering packets of 256 bytes size, our comparison of coalescing success rate is as follows: Without page pool: 70% With page pool: 13% Consequently, this has an impact on performance: Without page pool: 2.57 Gbits/sec With page pool: 2.26 Gbits/sec Therefore, it seems worthwhile to optimize this scenario and enable coalescing of this particular combination. To achieve this, we need to ensure the correct increment of the "from" SKB page's page pool reference count (pp_ref_count). Following this optimization, the success rate of coalescing measured in our environment has improved as follows: With page pool: 60% This success rate is approaching the rate achieved without using page pool, and the performance has also been improved: With page pool: 2.52 Gbits/sec Below is the performance comparison for small packets before and after this optimization. We observe no impact to packets larger than 4K. packet size before after improved (bytes) (Gbits/sec) (Gbits/sec) 128 1.19 1.27 7.13% 256 2.26 2.52 11.75% 512 4.13 4.81 16.50% 1024 6.17 6.73 9.05% 2048 14.54 15.47 6.45% 4096 25.44 27.87 9.52% Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin Suggested-by: Jason Wang Reviewed-by: Mina Almasry --- include/net/page_pool/helpers.h | 5 ++++ net/core/skbuff.c | 52 +++++++++++++++++++++++++-------- 2 files changed, 45 insertions(+), 12 deletions(-) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index d0c5e7e6857a..0dc8fab43bef 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -281,6 +281,11 @@ static inline long page_pool_unref_page(struct page *page, long nr) return ret; } +static inline void page_pool_ref_page(struct page *page) +{ + atomic_long_inc(&page->pp_ref_count); +} + static inline bool page_pool_is_last_ref(struct page *page) { /* If page_pool_unref_page() returns 0, we were the last user */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7e26b56cda38..cebdeafe691b 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -947,6 +947,37 @@ static bool skb_pp_recycle(struct sk_buff *skb, void *data, bool napi_safe) return napi_pp_put_page(virt_to_page(data), napi_safe); } +/** + * skb_pp_frag_ref() - Increase fragment references of a page pool aware skb + * @skb: page pool aware skb + * + * Increase the fragment reference count (pp_ref_count) of a skb. This is + * intended to gain fragment references only for page pool aware skbs, + * i.e. when skb->pp_recycle is true, and not for fragments in a + * non-pp-recycling skb. It has a fallback to increase references on normal + * pages, as page pool aware skbs may also have normal page fragments. + */ +static int skb_pp_frag_ref(struct sk_buff *skb) +{ + struct skb_shared_info *shinfo; + struct page *head_page; + int i; + + if (!skb->pp_recycle) + return -EINVAL; + + shinfo = skb_shinfo(skb); + + for (i = 0; i < shinfo->nr_frags; i++) { + head_page = compound_head(skb_frag_page(&shinfo->frags[i])); + if (likely(is_pp_page(head_page))) + page_pool_ref_page(head_page); + else + page_ref_inc(head_page); + } + return 0; +} + static void skb_kfree_head(void *head, unsigned int end_offset) { if (end_offset == SKB_SMALL_HEAD_HEADROOM) @@ -5769,17 +5800,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, return false; /* In general, avoid mixing page_pool and non-page_pool allocated - * pages within the same SKB. Additionally avoid dealing with clones - * with page_pool pages, in case the SKB is using page_pool fragment - * references (page_pool_alloc_frag()). Since we only take full page - * references for cloned SKBs at the moment that would result in - * inconsistent reference counts. - * In theory we could take full references if @from is cloned and - * !@to->pp_recycle but its tricky (due to potential race with - * the clone disappearing) and rare, so not worth dealing with. + * pages within the same SKB. In theory we could take full + * references if @from is cloned and !@to->pp_recycle but its + * tricky (due to potential race with the clone disappearing) and + * rare, so not worth dealing with. */ - if (to->pp_recycle != from->pp_recycle || - (from->pp_recycle && skb_cloned(from))) + if (to->pp_recycle != from->pp_recycle) return false; if (len <= skb_tailroom(to)) { @@ -5836,8 +5862,10 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, /* if the skb is not cloned this does nothing * since we set nr_frags to 0. */ - for (i = 0; i < from_shinfo->nr_frags; i++) - __skb_frag_ref(&from_shinfo->frags[i]); + if (skb_pp_frag_ref(from)) { + for (i = 0; i < from_shinfo->nr_frags; i++) + __skb_frag_ref(&from_shinfo->frags[i]); + } to->truesize += delta; to->len += len; -- 2.31.1