From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f69.google.com (mail-ej1-f69.google.com [209.85.218.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2403848F02E for ; Sat, 3 Oct 2026 21:23:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.69 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791062590; cv=none; b=iTzlrTp0HU7oieyeZV8GHtqatptovOTG0lWUxNSfNU8SQWMK9YlKsnhvbEzuFvJOhec3XdcP6yrzECW1qolIvUr904v+h7QAviCMK38HLRWjnHJ1sizpQE9NcW9lRdFszkrzq/0ZSJIHKxkJ54a4m6gcB79AmycbiuNWdnI0JIo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791062590; c=relaxed/simple; bh=7/+K5UbPoLVagSVVzP6ds9SnbBN5AQfhr70LFIIqoBg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=GlY9TnXYqePbYhi8fm/AekFx3qVUlRyQiDDVvf1f9O6xBVp9mO139x0tpcA+UEPnbF/ettx9PufJe+wP2cT4H6leWuG3g1k8CByAd9TenjGJa5LZmPsk/wxp19yrzzjfKUXnwyI0/7p2tHIL0FyQTYvi/R1fSd0usFfCVAKsAKs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=fGrZLVIF; arc=none smtp.client-ip=209.85.218.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fGrZLVIF" Received: by mail-ej1-f69.google.com with SMTP id a640c23a62f3a-c2dd399f761so82638766b.3 for ; Sat, 03 Oct 2026 14:23:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1791062583; x=1791667383; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=9Y6eccMU3oNOC6nPRXFDlu2gRIyDfLt0ekbGijFBkOg=; b=fGrZLVIFCBViVUFrMfdnmp0BRQfbj2ypV08ZfOadC6iiQsZBU+XKMDlVyaVk+nvzz0 fySaTe85YkJBXuxQ5+DAgOvEvPkgrHrtqDKfxaERDdaNMDWwLEg3jGw03kHN9njIFfCH 7oI37+VbWKORjbaGKNgIBmjc88WTo1JagMR176Wg62kw/iVnEe9lAFK3/ZZ14bXuNnMz H0+yEMX/G/aZg2bvc2c65/nBoni6Ly08Az0oc9doElsG6WtZ4heeY12GwFztvXmzf7Rg MGSJ5ieTJgM1hqoFyafNCRrQG11rtYvf7Td573CqaZ629mA/uWynmwmnFvBjWluuCi4f 0SKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791062583; x=1791667383; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9Y6eccMU3oNOC6nPRXFDlu2gRIyDfLt0ekbGijFBkOg=; b=pWTWBMdNXG65PxyZOGSOGOt9TPcmx78Onbeh92Qk2vg3RGwskyVKL0+43nQ0nS1DhC dsc5fUBBVF/L5xIzDWUSbRUgowXmziUX15+mZKUmq2C74g5oOfjA4RdXOyk8PlYCREFN /8lzyLzcmLwflTrdFm3LHHVVpr3suYJXgCziTAWyBpStmx15lDLsVjyeyw/r1P/73zDd R+aVipX668/bSKIYrijnOKnfcQyK9H9PUvRFsTRSZrZx4eigwuFx+uJ09w1GSon/ZdX3 hod/X/uwisy/CSsgPFo3i3rAJA+z2BAx7O260zgcJoF96AhjR85gYGYd2t7VO0zDxeT9 sjCg== X-Forwarded-Encrypted: i=1; AKwUvBxTvg/ofSQ2ljc2cxswhondFXzCGdrVrZlriaBsJ1impJNflOHzMOFrdRUaUirkyhEf/wvGLYAsN0w=@vger.kernel.org X-Gm-Message-State: AFuF++nxxw2oVZxMM+TrueDa+2yUZjbv+GXyOIyj+oV4lNEb9fOJ4P4O pX5VKRUNVX3UmoqaTcXc3E1FioCVVwUFRqaIc44XAbN3oLjFWbDW2r6U+lRegalvxxdD2z0d4a6 gNIwqyQ== X-Received: from ejbs7.prod.google.com ([2002:a17:906:607:b0:c2e:1f26:6d02]) (user=lrizzo job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:72cf:b0:c2d:cbc9:3529 with SMTP id a640c23a62f3a-c2e4af393e2mr567647366b.20.1791062582284; Sat, 03 Oct 2026 14:23:02 -0700 (PDT) Date: Sat, 3 Oct 2026 21:22:31 +0000 In-Reply-To: <20261003212241.3432303-1-lrizzo@google.com> Precedence: bulk X-Mailing-List: linux-doc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261003212241.3432303-1-lrizzo@google.com> X-Mailer: git-send-email 2.56.0.rc1.315.gc6ed9934b7-goog Message-ID: <20261003212241.3432303-13-lrizzo@google.com> Subject: [RFC: DMA_PMD 12/22] net/core: Use per-CPU DMA_PMD pools for skb_page_frag_refill() From: Luigi Rizzo To: Luigi Rizzo , Joerg Roedel , Will Deacon , Robin Murphy , Christoph Hellwig , Marek Szyprowski , Andrew Morton , Vlastimil Babka , David Hildenbrand , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Greg Kroah-Hartman , "Rafael J . Wysocki" , Danilo Krummrich , Jonathan Corbet , Jesper Dangaard Brouer , Ilias Apalodimas , Willem de Bruijn , Kuniyuki Iwashima , Joshua Washington , Harshitha Ramamurthy , Saeed Mahameed , Tariq Toukan , Tony Nguyen , Przemek Kitszel , Alexander Lobakin , Michael Chan , Pavan Chebbi , iommu@lists.linux.dev, netdev@vger.kernel.org, linux-mm@kvack.org, driver-core@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Luigi Rizzo Content-Type: text/plain; charset="UTF-8" Add sysctl net.core.tx_enable_dma_pmd to back skb_page_frag_refill() with per-CPU order-3 (32KB block) and order-0 (4KB block) DMA_PMD page pools. When enabled, TX socket buffer page fragment allocations draw fragments from DMA_PMD physical pages registered with per-CPU dma_pmd_pool. The first allocation maps the entire DMA_PMD page, subsequent allocations reuse the cached PMD_SIZE IOVA mapping locklessly without per-packet unmap or IOTLB flush. Signed-off-by: Luigi Rizzo --- include/net/sock.h | 3 ++ net/core/sock.c | 86 +++++++++++++++++++++++++++++++++++++- net/core/sysctl_net_core.c | 7 ++++ 3 files changed, 94 insertions(+), 2 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 60ea55dc18854..a06b53e9ff225 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -3092,6 +3092,9 @@ extern __u32 sysctl_rmem_default; #define SKB_FRAG_PAGE_ORDER get_order(32768) DECLARE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); +DECLARE_STATIC_KEY_FALSE(net_tx_enable_dma_pmd_key); +int net_tx_dma_pmd_sysctl(const struct ctl_table *table, int write, + void *buffer, size_t *lenp, loff_t *ppos); static inline int sk_get_wmem0(const struct sock *sk, const struct proto *proto) { diff --git a/net/core/sock.c b/net/core/sock.c index e8551df8330ff..1fd338a547ce1 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -115,6 +115,7 @@ #include #include #include +#include #include #include #include @@ -3173,6 +3174,87 @@ static void sk_leave_memory_pressure(struct sock *sk) } DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); +DEFINE_STATIC_KEY_FALSE(net_tx_enable_dma_pmd_key); + +static DEFINE_PER_CPU(struct dma_pmd_pool *, tx_pmd_pool_high); +static DEFINE_PER_CPU(struct dma_pmd_pool *, tx_pmd_pool_order0); + +/* + * Lazily initialize the per-CPU DMA_PMD page pools on the first write to sysctl + * net.core.tx_enable_dma_pmd. + * + * Once initialized, the struct dma_pmd_pool descriptors remain allocated for + * the lifetime of the kernel so that lockless raw_cpu_read() in + * __alloc_pmd() is always safe against concurrent sysctl + * toggles. If pool creation fails partway through, all pools created so far are + * destroyed and per-CPU pointers are reset to NULL. + */ +static int net_tx_dma_pmd_init(void) +{ + static DEFINE_MUTEX(mutex); + struct dma_pmd_pool *pool; + static bool initialized; + int cpu; + + if (!IS_ENABLED(CONFIG_DMA_PMD)) + return -EOPNOTSUPP; + + mutex_lock(&mutex); + if (initialized) { + mutex_unlock(&mutex); + return 0; + } + + for_each_possible_cpu(cpu) { + if (SKB_FRAG_PAGE_ORDER) { + pool = dma_pmd_pool_create(SKB_FRAG_PAGE_ORDER, 16); + if (!pool) + goto err_cleanup; + per_cpu(tx_pmd_pool_high, cpu) = pool; + } + + pool = dma_pmd_pool_create(0, 16); + if (!pool) + goto err_cleanup; + per_cpu(tx_pmd_pool_order0, cpu) = pool; + } + + initialized = true; + mutex_unlock(&mutex); + return 0; + +err_cleanup: + for_each_possible_cpu(cpu) { + per_cpu(tx_pmd_pool_high, cpu) = + dma_pmd_pool_destroy(per_cpu(tx_pmd_pool_high, cpu)); + per_cpu(tx_pmd_pool_order0, cpu) = + dma_pmd_pool_destroy(per_cpu(tx_pmd_pool_order0, cpu)); + } + mutex_unlock(&mutex); + return -ENOMEM; +} + +int net_tx_dma_pmd_sysctl(const struct ctl_table *table, int write, + void *buffer, size_t *lenp, loff_t *ppos) +{ + if (write) { + int ret = net_tx_dma_pmd_init(); + + if (ret) + return ret; + } + + return proc_do_static_key(table, write, buffer, lenp, ppos); +} + +static struct page *__alloc_pmd(gfp_t gfp, unsigned int order) +{ + if (!static_branch_unlikely(&net_tx_enable_dma_pmd_key)) + return alloc_pages(gfp, order); + if (order == SKB_FRAG_PAGE_ORDER) + return dma_pmd_pool_alloc(raw_cpu_read(tx_pmd_pool_high), gfp); + return dma_pmd_pool_alloc(raw_cpu_read(tx_pmd_pool_order0), gfp) ?: alloc_page(gfp); +} /** * skb_page_frag_refill - check that a page_frag contains enough room @@ -3200,7 +3282,7 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp) if (SKB_FRAG_PAGE_ORDER && !static_branch_unlikely(&net_high_order_alloc_disable_key)) { /* Avoid direct reclaim but allow kswapd to wake */ - pfrag->page = alloc_pages((gfp & ~__GFP_DIRECT_RECLAIM) | + pfrag->page = __alloc_pmd((gfp & ~__GFP_DIRECT_RECLAIM) | __GFP_COMP | __GFP_NOWARN | __GFP_NORETRY, SKB_FRAG_PAGE_ORDER); @@ -3209,7 +3291,7 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp) return true; } } - pfrag->page = alloc_page(gfp); + pfrag->page = __alloc_pmd(gfp, 0); if (likely(pfrag->page)) { pfrag->size = PAGE_SIZE; return true; diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index eb35da3556f4a..d9a7d9cb349e0 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -651,6 +651,13 @@ static struct ctl_table net_core_table[] = { .mode = 0644, .proc_handler = proc_do_static_key, }, + { + .procname = "tx_enable_dma_pmd", + .data = &net_tx_enable_dma_pmd_key.key, + .maxlen = sizeof(net_tx_enable_dma_pmd_key), + .mode = 0644, + .proc_handler = net_tx_dma_pmd_sysctl, + }, { .procname = "gro_normal_batch", .data = &net_hotdata.gro_normal_batch, -- 2.56.0.rc1.315.gc6ed9934b7-goog