* [PATCH net-next v2 00/15] First try to replace page_frag with page_frag_cache
@ 2024-04-15 13:19 Yunsheng Lin
2024-04-15 13:19 ` [PATCH net-next v2 07/15] mm: page_frag: add '_va' suffix to page_frag API Yunsheng Lin
0 siblings, 1 reply; 4+ messages in thread
From: Yunsheng Lin @ 2024-04-15 13:19 UTC (permalink / raw)
To: davem, kuba, pabeni
Cc: netdev, linux-kernel, Yunsheng Lin, Matthias Brugger,
AngeloGioacchino Del Regno, Alexei Starovoitov, Daniel Borkmann,
Jesper Dangaard Brouer, John Fastabend, linux-arm-kernel,
linux-mediatek, bpf
After [1], there are still two implementations for page frag:
1. mm/page_alloc.c: net stack seems to be using it in the
rx part with 'struct page_frag_cache' and the main API
being page_frag_alloc_align().
2. net/core/sock.c: net stack seems to be using it in the
tx part with 'struct page_frag' and the main API being
skb_page_frag_refill().
This patchset tries to unfiy the page frag implementation
by replacing page_frag with page_frag_cache for sk_page_frag()
first. net_high_order_alloc_disable_key for the implementation
in net/core/sock.c doesn't seems matter that much now have
have pcp support for high-order pages in commit 44042b449872
("mm/page_alloc: allow high-order pages to be stored on the
per-cpu lists").
As the related change is mostly related to networking, so
targeting the net-next. And will try to replace the rest
of page_frag in the follow patchset.
After this patchset:
1. Unify the page frag implementation by taking the best out of
two the existing implementations: we are able to save some space
for the 'page_frag_cache' API user, and avoid 'get_page()' for
the old 'page_frag' API user.
2. Future bugfix and performance can be done in one place, hence
improving maintainability of page_frag's implementation.
Performance validation:
1. Using micro-benchmark ko added in patch 1, we have about 3.2%
performance boot for the 'page_frag_cache' after this patchset.
2. Use the below netcat test case, we have about 1.9% performance
boot for repalcing 'page_frag' with 'page_frag_cache' after this
patchset.
server: nc -l -k 1234 > /dev/null
client: perf stat -r 30 -- head -c 51200000000 /dev/zero | nc -N 127.0.0.1 1234
In order to avoid performance noise as much as possible, the testing
is done in system without any other laod and have enough iterations to
prove the data is stable enogh, complete log for testing is below:
*After* this patchset:
insmod ./page_frag_test.ko nr_test=99999999
Performance counter stats for 'insmod ./page_frag_test.ko nr_test=99999999' (30 runs):
40.09 msec task-clock # 0.001 CPUs utilized ( +- 4.60% )
5 context-switches # 124.722 /sec ( +- 3.45% )
1 cpu-migrations # 24.944 /sec ( +- 12.62% )
197 page-faults # 4.914 K/sec ( +- 0.11% )
10221721 cycles # 0.255 GHz ( +- 9.05% ) (27.73%)
2459009 stalled-cycles-frontend # 24.06% frontend cycles idle ( +- 10.80% ) (29.05%)
5148423 stalled-cycles-backend # 50.37% backend cycles idle ( +- 7.30% ) (82.47%)
5889929 instructions # 0.58 insn per cycle
# 0.87 stalled cycles per insn ( +- 11.85% ) (87.75%)
1276667 branches # 31.846 M/sec ( +- 11.48% ) (89.80%)
50631 branch-misses # 3.97% of all branches ( +- 8.72% ) (83.20%)
29.341 +- 0.300 seconds time elapsed ( +- 1.02% )
perf stat -r 30 -- head -c 51200000000 /dev/zero | nc -N 127.0.0.1 1234
Performance counter stats for 'head -c 51200000000 /dev/zero' (30 runs):
107793.53 msec task-clock # 0.881 CPUs utilized ( +- 0.36% )
380421 context-switches # 3.529 K/sec ( +- 0.32% )
374 cpu-migrations # 3.470 /sec ( +- 1.31% )
74 page-faults # 0.686 /sec ( +- 0.28% )
92758718093 cycles # 0.861 GHz ( +- 0.48% ) (69.47%)
7035559641 stalled-cycles-frontend # 7.58% frontend cycles idle ( +- 1.19% ) (69.65%)
33668082825 stalled-cycles-backend # 36.30% backend cycles idle ( +- 0.84% ) (70.18%)
52424770535 instructions # 0.57 insn per cycle
# 0.64 stalled cycles per insn ( +- 0.26% ) (61.93%)
13240874953 branches # 122.836 M/sec ( +- 0.40% ) (60.36%)
208178019 branch-misses # 1.57% of all branches ( +- 0.65% ) (68.42%)
122.294 +- 0.402 seconds time elapsed ( +- 0.33% )
*Before* this patchset:
insmod ./page_frag_test.ko nr_test=99999999
Performance counter stats for 'insmod ./page_frag_test.ko nr_test=99999999' (30 runs):
39.12 msec task-clock # 0.001 CPUs utilized ( +- 4.51% )
5 context-switches # 127.805 /sec ( +- 3.76% )
1 cpu-migrations # 25.561 /sec ( +- 15.52% )
197 page-faults # 5.035 K/sec ( +- 0.10% )
10689913 cycles # 0.273 GHz ( +- 9.46% ) (72.72%)
2821237 stalled-cycles-frontend # 26.39% frontend cycles idle ( +- 12.04% ) (76.23%)
5035549 stalled-cycles-backend # 47.11% backend cycles idle ( +- 9.69% ) (49.40%)
5439395 instructions # 0.51 insn per cycle
# 0.93 stalled cycles per insn ( +- 11.58% ) (51.45%)
1274419 branches # 32.575 M/sec ( +- 12.69% ) (77.88%)
49562 branch-misses # 3.89% of all branches ( +- 9.91% ) (72.32%)
30.309 +- 0.305 seconds time elapsed ( +- 1.01% )
perf stat -r 30 -- head -c 51200000000 /dev/zero | nc -N 127.0.0.1 1234
Performance counter stats for 'head -c 51200000000 /dev/zero' (30 runs):
110198.93 msec task-clock # 0.884 CPUs utilized ( +- 0.83% )
387680 context-switches # 3.518 K/sec ( +- 0.85% )
366 cpu-migrations # 3.321 /sec ( +- 11.38% )
74 page-faults # 0.672 /sec ( +- 0.27% )
92978008685 cycles # 0.844 GHz ( +- 0.49% ) (64.93%)
7339938950 stalled-cycles-frontend # 7.89% frontend cycles idle ( +- 1.48% ) (67.15%)
34783792329 stalled-cycles-backend # 37.41% backend cycles idle ( +- 1.52% ) (68.96%)
51704527141 instructions # 0.56 insn per cycle
# 0.67 stalled cycles per insn ( +- 0.37% ) (68.28%)
12865503633 branches # 116.748 M/sec ( +- 0.88% ) (66.11%)
212414695 branch-misses # 1.65% of all branches ( +- 0.45% ) (64.57%)
124.664 +- 0.990 seconds time elapsed ( +- 0.79% )
Note, ipv4-udp, ipv6-tcp and ipv6-udp is also tested with the below script:
nc -u -l -k 1234 > /dev/null
perf stat -r 4 -- head -c 51200000000 /dev/zero | nc -N -u 127.0.0.1 1234
nc -l6 -k 1234 > /dev/null
perf stat -r 4 -- head -c 51200000000 /dev/zero | nc -N ::1 1234
nc -l6 -k -u 1234 > /dev/null
perf stat -r 4 -- head -c 51200000000 /dev/zero | nc -u -N ::1 1234
Change log:
V2:
1. reorder test module to patch 1.
2. split doc and maintainer updating to two patches.
3. refactor the page_frag before moving.
4. fix a type and 'static' warning in test module.
5. add a patch for xtensa arch to enable using get_order() in
BUILD_BUG_ON().
6. Add test case and performance data for the socket code.
Yunsheng Lin (15):
mm: page_frag: add a test module for page_frag
xtensa: remove the get_order() implementation
mm: page_frag: use free_unref_page() to free page fragment
mm: move the page fragment allocator from page_alloc into its own file
mm: page_frag: use initial zero offset for page_frag_alloc_align()
mm: page_frag: change page_frag_alloc_* API to accept align param
mm: page_frag: add '_va' suffix to page_frag API
mm: page_frag: add two inline helper for page_frag API
mm: page_frag: reuse MSB of 'size' field for pfmemalloc
mm: page_frag: reuse existing bit field of 'va' for pagecnt_bias
net: introduce the skb_copy_to_va_nocache() helper
mm: page_frag: introduce prepare/commit API for page_frag
net: replace page_frag with page_frag_cache
mm: page_frag: update documentation for page_frag
mm: page_frag: add a entry in MAINTAINERS for page_frag
Documentation/mm/page_frags.rst | 148 ++++++-
MAINTAINERS | 11 +
arch/xtensa/include/asm/page.h | 18 -
.../chelsio/inline_crypto/chtls/chtls.h | 3 -
.../chelsio/inline_crypto/chtls/chtls_io.c | 101 ++---
.../chelsio/inline_crypto/chtls/chtls_main.c | 3 -
drivers/net/ethernet/google/gve/gve_rx.c | 4 +-
drivers/net/ethernet/intel/ice/ice_txrx.c | 2 +-
drivers/net/ethernet/intel/ice/ice_txrx.h | 2 +-
drivers/net/ethernet/intel/ice/ice_txrx_lib.c | 2 +-
.../net/ethernet/intel/ixgbevf/ixgbevf_main.c | 4 +-
.../marvell/octeontx2/nic/otx2_common.c | 2 +-
drivers/net/ethernet/mediatek/mtk_wed_wo.c | 4 +-
drivers/net/tun.c | 34 +-
drivers/nvme/host/tcp.c | 8 +-
drivers/nvme/target/tcp.c | 22 +-
drivers/vhost/net.c | 6 +-
include/linux/gfp.h | 22 --
include/linux/mm_types.h | 18 -
include/linux/page_frag_cache.h | 344 +++++++++++++++++
include/linux/sched.h | 4 +-
include/linux/skbuff.h | 15 +-
include/net/sock.h | 29 +-
kernel/bpf/cpumap.c | 2 +-
kernel/exit.c | 3 +-
kernel/fork.c | 2 +-
mm/Kconfig.debug | 8 +
mm/Makefile | 2 +
mm/page_alloc.c | 136 -------
mm/page_frag_cache.c | 154 ++++++++
mm/page_frag_test.c | 365 ++++++++++++++++++
net/core/skbuff.c | 57 +--
net/core/skmsg.c | 22 +-
net/core/sock.c | 46 ++-
net/core/xdp.c | 2 +-
net/ipv4/ip_output.c | 35 +-
net/ipv4/tcp.c | 35 +-
net/ipv4/tcp_output.c | 28 +-
net/ipv6/ip6_output.c | 35 +-
net/kcm/kcmsock.c | 30 +-
net/mptcp/protocol.c | 74 ++--
net/rxrpc/txbuf.c | 16 +-
net/sunrpc/svcsock.c | 6 +-
net/tls/tls_device.c | 139 ++++---
44 files changed, 1450 insertions(+), 553 deletions(-)
create mode 100644 include/linux/page_frag_cache.h
create mode 100644 mm/page_frag_cache.c
create mode 100644 mm/page_frag_test.c
--
2.33.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH net-next v2 07/15] mm: page_frag: add '_va' suffix to page_frag API
2024-04-15 13:19 [PATCH net-next v2 00/15] First try to replace page_frag with page_frag_cache Yunsheng Lin
@ 2024-04-15 13:19 ` Yunsheng Lin
2024-04-16 16:12 ` Alexander H Duyck
0 siblings, 1 reply; 4+ messages in thread
From: Yunsheng Lin @ 2024-04-15 13:19 UTC (permalink / raw)
To: davem, kuba, pabeni
Cc: netdev, linux-kernel, Yunsheng Lin, Jeroen de Borst,
Praveen Kaligineedi, Shailend Chand, Eric Dumazet,
Jesse Brandeburg, Tony Nguyen, Sunil Goutham, Geetha sowjanya,
Subbaraya Sundeep, hariprasad, Felix Fietkau, Sean Wang, Mark Lee,
Lorenzo Bianconi, Matthias Brugger, AngeloGioacchino Del Regno,
Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg,
Chaitanya Kulkarni, Michael S. Tsirkin, Jason Wang,
Alexander Duyck, Andrew Morton, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Song Liu,
Yonghong Song, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David Howells, Marc Dionne, Chuck Lever, Jeff Layton, Neil Brown,
Olga Kornievskaia, Dai Ngo, Tom Talpey, Trond Myklebust,
Anna Schumaker, intel-wired-lan, linux-arm-kernel, linux-mediatek,
linux-nvme, kvm, virtualization, linux-mm, bpf, linux-afs,
linux-nfs
Currently most of the API for page_frag API is returning
'virtual address' as output or expecting 'virtual address'
as input, in order to differentiate the API handling between
'virtual address' and 'struct page', add '_va' suffix to the
corresponding API mirroring the page_pool_alloc_va() API of
the page_pool.
Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
---
drivers/net/ethernet/google/gve/gve_rx.c | 4 ++--
drivers/net/ethernet/intel/ice/ice_txrx.c | 2 +-
drivers/net/ethernet/intel/ice/ice_txrx.h | 2 +-
drivers/net/ethernet/intel/ice/ice_txrx_lib.c | 2 +-
.../net/ethernet/intel/ixgbevf/ixgbevf_main.c | 4 ++--
.../marvell/octeontx2/nic/otx2_common.c | 2 +-
drivers/net/ethernet/mediatek/mtk_wed_wo.c | 4 ++--
drivers/nvme/host/tcp.c | 8 +++----
drivers/nvme/target/tcp.c | 22 ++++++++---------
drivers/vhost/net.c | 6 ++---
include/linux/page_frag_cache.h | 24 ++++++++++---------
include/linux/skbuff.h | 2 +-
kernel/bpf/cpumap.c | 2 +-
mm/page_frag_cache.c | 10 ++++----
mm/page_frag_test.c | 6 ++---
net/core/skbuff.c | 15 ++++++------
net/core/xdp.c | 2 +-
net/rxrpc/txbuf.c | 15 ++++++------
net/sunrpc/svcsock.c | 6 ++---
19 files changed, 71 insertions(+), 67 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c
index cd727e55ae0f..820874c1c570 100644
--- a/drivers/net/ethernet/google/gve/gve_rx.c
+++ b/drivers/net/ethernet/google/gve/gve_rx.c
@@ -687,7 +687,7 @@ static int gve_xdp_redirect(struct net_device *dev, struct gve_rx_ring *rx,
total_len = headroom + SKB_DATA_ALIGN(len) +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
- frame = page_frag_alloc(&rx->page_cache, total_len, GFP_ATOMIC);
+ frame = page_frag_alloc_va(&rx->page_cache, total_len, GFP_ATOMIC);
if (!frame) {
u64_stats_update_begin(&rx->statss);
rx->xdp_alloc_fails++;
@@ -700,7 +700,7 @@ static int gve_xdp_redirect(struct net_device *dev, struct gve_rx_ring *rx,
err = xdp_do_redirect(dev, &new, xdp_prog);
if (err)
- page_frag_free(frame);
+ page_frag_free_va(frame);
return err;
}
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c
index 8bb743f78fcb..399b317c509d 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.c
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.c
@@ -126,7 +126,7 @@ ice_unmap_and_free_tx_buf(struct ice_tx_ring *ring, struct ice_tx_buf *tx_buf)
dev_kfree_skb_any(tx_buf->skb);
break;
case ICE_TX_BUF_XDP_TX:
- page_frag_free(tx_buf->raw_buf);
+ page_frag_free_va(tx_buf->raw_buf);
break;
case ICE_TX_BUF_XDP_XMIT:
xdp_return_frame(tx_buf->xdpf);
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.h b/drivers/net/ethernet/intel/ice/ice_txrx.h
index feba314a3fe4..6379f57d8228 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.h
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.h
@@ -148,7 +148,7 @@ static inline int ice_skb_pad(void)
* @ICE_TX_BUF_DUMMY: dummy Flow Director packet, unmap and kfree()
* @ICE_TX_BUF_FRAG: mapped skb OR &xdp_buff frag, only unmap DMA
* @ICE_TX_BUF_SKB: &sk_buff, unmap and consume_skb(), update stats
- * @ICE_TX_BUF_XDP_TX: &xdp_buff, unmap and page_frag_free(), stats
+ * @ICE_TX_BUF_XDP_TX: &xdp_buff, unmap and page_frag_free_va(), stats
* @ICE_TX_BUF_XDP_XMIT: &xdp_frame, unmap and xdp_return_frame(), stats
* @ICE_TX_BUF_XSK_TX: &xdp_buff on XSk queue, xsk_buff_free(), stats
*/
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx_lib.c b/drivers/net/ethernet/intel/ice/ice_txrx_lib.c
index df072ce767b1..c34cc02ad578 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx_lib.c
+++ b/drivers/net/ethernet/intel/ice/ice_txrx_lib.c
@@ -288,7 +288,7 @@ ice_clean_xdp_tx_buf(struct device *dev, struct ice_tx_buf *tx_buf,
switch (tx_buf->type) {
case ICE_TX_BUF_XDP_TX:
- page_frag_free(tx_buf->raw_buf);
+ page_frag_free_va(tx_buf->raw_buf);
break;
case ICE_TX_BUF_XDP_XMIT:
xdp_return_frame_bulk(tx_buf->xdpf, bq);
diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
index 3161a13079fe..c35b8f675b48 100644
--- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
+++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
@@ -303,7 +303,7 @@ static bool ixgbevf_clean_tx_irq(struct ixgbevf_q_vector *q_vector,
/* free the skb */
if (ring_is_xdp(tx_ring))
- page_frag_free(tx_buffer->data);
+ page_frag_free_va(tx_buffer->data);
else
napi_consume_skb(tx_buffer->skb, napi_budget);
@@ -2413,7 +2413,7 @@ static void ixgbevf_clean_tx_ring(struct ixgbevf_ring *tx_ring)
/* Free all the Tx ring sk_buffs */
if (ring_is_xdp(tx_ring))
- page_frag_free(tx_buffer->data);
+ page_frag_free_va(tx_buffer->data);
else
dev_kfree_skb_any(tx_buffer->skb);
diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c
index a85ac039d779..8eb5820b8a70 100644
--- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c
+++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c
@@ -553,7 +553,7 @@ static int __otx2_alloc_rbuf(struct otx2_nic *pfvf, struct otx2_pool *pool,
*dma = dma_map_single_attrs(pfvf->dev, buf, pool->rbsize,
DMA_FROM_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);
if (unlikely(dma_mapping_error(pfvf->dev, *dma))) {
- page_frag_free(buf);
+ page_frag_free_va(buf);
return -ENOMEM;
}
diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c
index 7063c78bd35f..c4228719f8a4 100644
--- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c
+++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c
@@ -142,8 +142,8 @@ mtk_wed_wo_queue_refill(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q,
dma_addr_t addr;
void *buf;
- buf = page_frag_alloc(&q->cache, q->buf_size,
- GFP_ATOMIC | GFP_DMA32);
+ buf = page_frag_alloc_va(&q->cache, q->buf_size,
+ GFP_ATOMIC | GFP_DMA32);
if (!buf)
break;
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index fdbcdcedcee9..79eddd74bfbb 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -500,7 +500,7 @@ static void nvme_tcp_exit_request(struct blk_mq_tag_set *set,
{
struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
- page_frag_free(req->pdu);
+ page_frag_free_va(req->pdu);
}
static int nvme_tcp_init_request(struct blk_mq_tag_set *set,
@@ -514,7 +514,7 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set,
struct nvme_tcp_queue *queue = &ctrl->queues[queue_idx];
u8 hdgst = nvme_tcp_hdgst_len(queue);
- req->pdu = page_frag_alloc(&queue->pf_cache,
+ req->pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(struct nvme_tcp_cmd_pdu) + hdgst,
GFP_KERNEL | __GFP_ZERO);
if (!req->pdu)
@@ -1331,7 +1331,7 @@ static void nvme_tcp_free_async_req(struct nvme_tcp_ctrl *ctrl)
{
struct nvme_tcp_request *async = &ctrl->async_req;
- page_frag_free(async->pdu);
+ page_frag_free_va(async->pdu);
}
static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)
@@ -1340,7 +1340,7 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)
struct nvme_tcp_request *async = &ctrl->async_req;
u8 hdgst = nvme_tcp_hdgst_len(queue);
- async->pdu = page_frag_alloc(&queue->pf_cache,
+ async->pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(struct nvme_tcp_cmd_pdu) + hdgst,
GFP_KERNEL | __GFP_ZERO);
if (!async->pdu)
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index a5422e2c979a..ea356ce22672 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1462,24 +1462,24 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,
c->queue = queue;
c->req.port = queue->port->nport;
- c->cmd_pdu = page_frag_alloc(&queue->pf_cache,
+ c->cmd_pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(*c->cmd_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO);
if (!c->cmd_pdu)
return -ENOMEM;
c->req.cmd = &c->cmd_pdu->cmd;
- c->rsp_pdu = page_frag_alloc(&queue->pf_cache,
+ c->rsp_pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(*c->rsp_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO);
if (!c->rsp_pdu)
goto out_free_cmd;
c->req.cqe = &c->rsp_pdu->cqe;
- c->data_pdu = page_frag_alloc(&queue->pf_cache,
+ c->data_pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(*c->data_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO);
if (!c->data_pdu)
goto out_free_rsp;
- c->r2t_pdu = page_frag_alloc(&queue->pf_cache,
+ c->r2t_pdu = page_frag_alloc_va(&queue->pf_cache,
sizeof(*c->r2t_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO);
if (!c->r2t_pdu)
goto out_free_data;
@@ -1494,20 +1494,20 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,
return 0;
out_free_data:
- page_frag_free(c->data_pdu);
+ page_frag_free_va(c->data_pdu);
out_free_rsp:
- page_frag_free(c->rsp_pdu);
+ page_frag_free_va(c->rsp_pdu);
out_free_cmd:
- page_frag_free(c->cmd_pdu);
+ page_frag_free_va(c->cmd_pdu);
return -ENOMEM;
}
static void nvmet_tcp_free_cmd(struct nvmet_tcp_cmd *c)
{
- page_frag_free(c->r2t_pdu);
- page_frag_free(c->data_pdu);
- page_frag_free(c->rsp_pdu);
- page_frag_free(c->cmd_pdu);
+ page_frag_free_va(c->r2t_pdu);
+ page_frag_free_va(c->data_pdu);
+ page_frag_free_va(c->rsp_pdu);
+ page_frag_free_va(c->cmd_pdu);
}
static int nvmet_tcp_alloc_cmds(struct nvmet_tcp_queue *queue)
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index c64ded183f8d..96d5ca299552 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -682,8 +682,8 @@ static int vhost_net_build_xdp(struct vhost_net_virtqueue *nvq,
return -ENOSPC;
buflen += SKB_DATA_ALIGN(len + pad);
- buf = page_frag_alloc_align(&net->pf_cache, buflen, GFP_KERNEL,
- SMP_CACHE_BYTES);
+ buf = page_frag_alloc_va_align(&net->pf_cache, buflen, GFP_KERNEL,
+ SMP_CACHE_BYTES);
if (unlikely(!buf))
return -ENOMEM;
@@ -730,7 +730,7 @@ static int vhost_net_build_xdp(struct vhost_net_virtqueue *nvq,
return 0;
err:
- page_frag_free(buf);
+ page_frag_free_va(buf);
return ret;
}
diff --git a/include/linux/page_frag_cache.h b/include/linux/page_frag_cache.h
index cc0ede0912f3..9d5d86b2d3ab 100644
--- a/include/linux/page_frag_cache.h
+++ b/include/linux/page_frag_cache.h
@@ -25,27 +25,29 @@ struct page_frag_cache {
void page_frag_cache_drain(struct page_frag_cache *nc);
void __page_frag_cache_drain(struct page *page, unsigned int count);
-void *page_frag_alloc(struct page_frag_cache *nc, unsigned int fragsz,
- gfp_t gfp_mask);
+void *page_frag_alloc_va(struct page_frag_cache *nc, unsigned int fragsz,
+ gfp_t gfp_mask);
-static inline void *__page_frag_alloc_align(struct page_frag_cache *nc,
- unsigned int fragsz, gfp_t gfp_mask,
- unsigned int align)
+static inline void *__page_frag_alloc_va_align(struct page_frag_cache *nc,
+ unsigned int fragsz,
+ gfp_t gfp_mask,
+ unsigned int align)
{
nc->offset = ALIGN(nc->offset, align);
- return page_frag_alloc(nc, fragsz, gfp_mask);
+ return page_frag_alloc_va(nc, fragsz, gfp_mask);
}
-static inline void *page_frag_alloc_align(struct page_frag_cache *nc,
- unsigned int fragsz, gfp_t gfp_mask,
- unsigned int align)
+static inline void *page_frag_alloc_va_align(struct page_frag_cache *nc,
+ unsigned int fragsz,
+ gfp_t gfp_mask,
+ unsigned int align)
{
WARN_ON_ONCE(!is_power_of_2(align));
- return __page_frag_alloc_align(nc, fragsz, gfp_mask, align);
+ return __page_frag_alloc_va_align(nc, fragsz, gfp_mask, align);
}
-void page_frag_free(void *addr);
+void page_frag_free_va(void *addr);
#endif
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 43c704589deb..cc80600dcedf 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -3343,7 +3343,7 @@ static inline struct sk_buff *netdev_alloc_skb_ip_align(struct net_device *dev,
static inline void skb_free_frag(void *addr)
{
- page_frag_free(addr);
+ page_frag_free_va(addr);
}
void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align);
diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c
index a8e34416e960..3a6a237e7dd3 100644
--- a/kernel/bpf/cpumap.c
+++ b/kernel/bpf/cpumap.c
@@ -322,7 +322,7 @@ static int cpu_map_kthread_run(void *data)
/* Bring struct page memory area to curr CPU. Read by
* build_skb_around via page_is_pfmemalloc(), and when
- * freed written by page_frag_free call.
+ * freed written by page_frag_free_va call.
*/
prefetchw(page);
}
diff --git a/mm/page_frag_cache.c b/mm/page_frag_cache.c
index b4408187e1ab..50511d8522d0 100644
--- a/mm/page_frag_cache.c
+++ b/mm/page_frag_cache.c
@@ -61,8 +61,8 @@ void __page_frag_cache_drain(struct page *page, unsigned int count)
}
EXPORT_SYMBOL(__page_frag_cache_drain);
-void *page_frag_alloc(struct page_frag_cache *nc, unsigned int fragsz,
- gfp_t gfp_mask)
+void *page_frag_alloc_va(struct page_frag_cache *nc, unsigned int fragsz,
+ gfp_t gfp_mask)
{
unsigned int size, offset;
struct page *page;
@@ -128,16 +128,16 @@ void *page_frag_alloc(struct page_frag_cache *nc, unsigned int fragsz,
return nc->va + offset;
}
-EXPORT_SYMBOL(page_frag_alloc);
+EXPORT_SYMBOL(page_frag_alloc_va);
/*
* Frees a page fragment allocated out of either a compound or order 0 page.
*/
-void page_frag_free(void *addr)
+void page_frag_free_va(void *addr)
{
struct page *page = virt_to_head_page(addr);
if (unlikely(put_page_testzero(page)))
free_unref_page(page, compound_order(page));
}
-EXPORT_SYMBOL(page_frag_free);
+EXPORT_SYMBOL(page_frag_free_va);
diff --git a/mm/page_frag_test.c b/mm/page_frag_test.c
index ebfd1c3dae8f..cab05b8a2e77 100644
--- a/mm/page_frag_test.c
+++ b/mm/page_frag_test.c
@@ -260,7 +260,7 @@ static int page_frag_pop_thread(void *arg)
if (obj) {
nr--;
- page_frag_free(obj);
+ page_frag_free_va(obj);
} else {
cond_resched();
}
@@ -289,13 +289,13 @@ static int page_frag_push_thread(void *arg)
int ret;
size = clamp(size, 4U, 4096U);
- va = page_frag_alloc(&test_frag, size, GFP_KERNEL);
+ va = page_frag_alloc_va(&test_frag, size, GFP_KERNEL);
if (!va)
continue;
ret = objpool_push(va, pool);
if (ret) {
- page_frag_free(va);
+ page_frag_free_va(va);
cond_resched();
} else {
nr--;
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 676e2d857f02..139a193853cc 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -312,7 +312,7 @@ void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align)
fragsz = SKB_DATA_ALIGN(fragsz);
- return __page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align);
+ return __page_frag_alloc_va_align(&nc->page, fragsz, GFP_ATOMIC, align);
}
EXPORT_SYMBOL(__napi_alloc_frag_align);
@@ -324,14 +324,15 @@ void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align)
if (in_hardirq() || irqs_disabled()) {
struct page_frag_cache *nc = this_cpu_ptr(&netdev_alloc_cache);
- data = __page_frag_alloc_align(nc, fragsz, GFP_ATOMIC, align);
+ data = __page_frag_alloc_va_align(nc, fragsz, GFP_ATOMIC,
+ align);
} else {
struct napi_alloc_cache *nc;
local_bh_disable();
nc = this_cpu_ptr(&napi_alloc_cache);
- data = __page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC,
- align);
+ data = __page_frag_alloc_va_align(&nc->page, fragsz, GFP_ATOMIC,
+ align);
local_bh_enable();
}
return data;
@@ -741,12 +742,12 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len,
if (in_hardirq() || irqs_disabled()) {
nc = this_cpu_ptr(&netdev_alloc_cache);
- data = page_frag_alloc(nc, len, gfp_mask);
+ data = page_frag_alloc_va(nc, len, gfp_mask);
pfmemalloc = nc->pfmemalloc;
} else {
local_bh_disable();
nc = this_cpu_ptr(&napi_alloc_cache.page);
- data = page_frag_alloc(nc, len, gfp_mask);
+ data = page_frag_alloc_va(nc, len, gfp_mask);
pfmemalloc = nc->pfmemalloc;
local_bh_enable();
}
@@ -834,7 +835,7 @@ struct sk_buff *napi_alloc_skb(struct napi_struct *napi, unsigned int len)
} else {
len = SKB_HEAD_ALIGN(len);
- data = page_frag_alloc(&nc->page, len, gfp_mask);
+ data = page_frag_alloc_va(&nc->page, len, gfp_mask);
pfmemalloc = nc->page.pfmemalloc;
}
diff --git a/net/core/xdp.c b/net/core/xdp.c
index 41693154e426..245a2d011aeb 100644
--- a/net/core/xdp.c
+++ b/net/core/xdp.c
@@ -391,7 +391,7 @@ void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct,
page_pool_put_full_page(page->pp, page, napi_direct);
break;
case MEM_TYPE_PAGE_SHARED:
- page_frag_free(data);
+ page_frag_free_va(data);
break;
case MEM_TYPE_PAGE_ORDER0:
page = virt_to_page(data); /* Assumes order0 page*/
diff --git a/net/rxrpc/txbuf.c b/net/rxrpc/txbuf.c
index eb640875bf07..f2fa98360789 100644
--- a/net/rxrpc/txbuf.c
+++ b/net/rxrpc/txbuf.c
@@ -34,8 +34,8 @@ struct rxrpc_txbuf *rxrpc_alloc_data_txbuf(struct rxrpc_call *call, size_t data_
data_align = max_t(size_t, data_align, L1_CACHE_BYTES);
mutex_lock(&call->conn->tx_data_alloc_lock);
- buf = page_frag_alloc_align(&call->conn->tx_data_alloc, total, gfp,
- data_align);
+ buf = page_frag_alloc_va_align(&call->conn->tx_data_alloc, total, gfp,
+ data_align);
mutex_unlock(&call->conn->tx_data_alloc_lock);
if (!buf) {
kfree(txb);
@@ -97,17 +97,18 @@ struct rxrpc_txbuf *rxrpc_alloc_ack_txbuf(struct rxrpc_call *call, size_t sack_s
if (!txb)
return NULL;
- buf = page_frag_alloc(&call->local->tx_alloc,
- sizeof(*whdr) + sizeof(*ack) + 1 + 3 + sizeof(*trailer), gfp);
+ buf = page_frag_alloc_va(&call->local->tx_alloc,
+ sizeof(*whdr) + sizeof(*ack) + 1 + 3 + sizeof(*trailer), gfp);
if (!buf) {
kfree(txb);
return NULL;
}
if (sack_size) {
- buf2 = page_frag_alloc(&call->local->tx_alloc, sack_size, gfp);
+ buf2 = page_frag_alloc_va(&call->local->tx_alloc, sack_size,
+ gfp);
if (!buf2) {
- page_frag_free(buf);
+ page_frag_free_va(buf);
kfree(txb);
return NULL;
}
@@ -181,7 +182,7 @@ static void rxrpc_free_txbuf(struct rxrpc_txbuf *txb)
rxrpc_txbuf_free);
for (i = 0; i < txb->nr_kvec; i++)
if (txb->kvec[i].iov_base)
- page_frag_free(txb->kvec[i].iov_base);
+ page_frag_free_va(txb->kvec[i].iov_base);
kfree(txb);
atomic_dec(&rxrpc_nr_txbuf);
}
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index 6b3f01beb294..42d20412c1c3 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -1222,8 +1222,8 @@ static int svc_tcp_sendmsg(struct svc_sock *svsk, struct svc_rqst *rqstp,
/* The stream record marker is copied into a temporary page
* fragment buffer so that it can be included in rq_bvec.
*/
- buf = page_frag_alloc(&svsk->sk_frag_cache, sizeof(marker),
- GFP_KERNEL);
+ buf = page_frag_alloc_va(&svsk->sk_frag_cache, sizeof(marker),
+ GFP_KERNEL);
if (!buf)
return -ENOMEM;
memcpy(buf, &marker, sizeof(marker));
@@ -1235,7 +1235,7 @@ static int svc_tcp_sendmsg(struct svc_sock *svsk, struct svc_rqst *rqstp,
iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, rqstp->rq_bvec,
1 + count, sizeof(marker) + rqstp->rq_res.len);
ret = sock_sendmsg(svsk->sk_sock, &msg);
- page_frag_free(buf);
+ page_frag_free_va(buf);
if (ret < 0)
return ret;
*sentp += ret;
--
2.33.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH net-next v2 07/15] mm: page_frag: add '_va' suffix to page_frag API
2024-04-15 13:19 ` [PATCH net-next v2 07/15] mm: page_frag: add '_va' suffix to page_frag API Yunsheng Lin
@ 2024-04-16 16:12 ` Alexander H Duyck
2024-04-17 13:18 ` Yunsheng Lin
0 siblings, 1 reply; 4+ messages in thread
From: Alexander H Duyck @ 2024-04-16 16:12 UTC (permalink / raw)
To: Yunsheng Lin, davem, kuba, pabeni
Cc: netdev, linux-kernel, Jeroen de Borst, Praveen Kaligineedi,
Shailend Chand, Eric Dumazet, Jesse Brandeburg, Tony Nguyen,
Sunil Goutham, Geetha sowjanya, Subbaraya Sundeep, hariprasad,
Felix Fietkau, Sean Wang, Mark Lee, Lorenzo Bianconi,
Matthias Brugger, AngeloGioacchino Del Regno, Keith Busch,
Jens Axboe, Christoph Hellwig, Sagi Grimberg, Chaitanya Kulkarni,
Michael S. Tsirkin, Jason Wang, Andrew Morton, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Song Liu,
Yonghong Song, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David Howells, Marc Dionne, Chuck Lever, Jeff Layton, Neil Brown,
Olga Kornievskaia, Dai Ngo, Tom Talpey, Trond Myklebust,
Anna Schumaker, intel-wired-lan, linux-arm-kernel, linux-mediatek,
linux-nvme, kvm, virtualization, linux-mm, bpf, linux-afs,
linux-nfs
On Mon, 2024-04-15 at 21:19 +0800, Yunsheng Lin wrote:
> Currently most of the API for page_frag API is returning
> 'virtual address' as output or expecting 'virtual address'
> as input, in order to differentiate the API handling between
> 'virtual address' and 'struct page', add '_va' suffix to the
> corresponding API mirroring the page_pool_alloc_va() API of
> the page_pool.
>
> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
This patch is a total waste of time. By that logic we should be
renaming __get_free_pages since it essentially does the same thing.
This just seems like more code changes for the sake of adding code
changes rather than fixing anything. In my opinion it should be dropped
from the set.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH net-next v2 07/15] mm: page_frag: add '_va' suffix to page_frag API
2024-04-16 16:12 ` Alexander H Duyck
@ 2024-04-17 13:18 ` Yunsheng Lin
0 siblings, 0 replies; 4+ messages in thread
From: Yunsheng Lin @ 2024-04-17 13:18 UTC (permalink / raw)
To: Alexander H Duyck, davem, kuba, pabeni
Cc: netdev, linux-kernel, Jeroen de Borst, Praveen Kaligineedi,
Shailend Chand, Eric Dumazet, Jesse Brandeburg, Tony Nguyen,
Sunil Goutham, Geetha sowjanya, Subbaraya Sundeep, hariprasad,
Felix Fietkau, Sean Wang, Mark Lee, Lorenzo Bianconi,
Matthias Brugger, AngeloGioacchino Del Regno, Keith Busch,
Jens Axboe, Christoph Hellwig, Sagi Grimberg, Chaitanya Kulkarni,
Michael S. Tsirkin, Jason Wang, Andrew Morton, Alexei Starovoitov,
Daniel Borkmann, Jesper Dangaard Brouer, John Fastabend,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Song Liu,
Yonghong Song, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David Howells, Marc Dionne, Chuck Lever, Jeff Layton, Neil Brown,
Olga Kornievskaia, Dai Ngo, Tom Talpey, Trond Myklebust,
Anna Schumaker, intel-wired-lan, linux-arm-kernel, linux-mediatek,
linux-nvme, kvm, virtualization, linux-mm, bpf, linux-afs,
linux-nfs
On 2024/4/17 0:12, Alexander H Duyck wrote:
> On Mon, 2024-04-15 at 21:19 +0800, Yunsheng Lin wrote:
>> Currently most of the API for page_frag API is returning
>> 'virtual address' as output or expecting 'virtual address'
>> as input, in order to differentiate the API handling between
>> 'virtual address' and 'struct page', add '_va' suffix to the
>> corresponding API mirroring the page_pool_alloc_va() API of
>> the page_pool.
>>
>> Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
>
> This patch is a total waste of time. By that logic we should be
> renaming __get_free_pages since it essentially does the same thing.
>
> This just seems like more code changes for the sake of adding code
> changes rather than fixing anything. In my opinion it should be dropped
> from the set.
The rename is to support different use case as mentioned below in patch
14:
"Depending on different use cases, callers expecting to deal with va, page or
both va and page for them may call page_frag_alloc_va*, page_frag_alloc_pg*,
or page_frag_alloc* API accordingly."
Naming is hard anyway, I am open to better API naming for the above use cases.
>
> .
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-04-17 13:18 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-04-15 13:19 [PATCH net-next v2 00/15] First try to replace page_frag with page_frag_cache Yunsheng Lin
2024-04-15 13:19 ` [PATCH net-next v2 07/15] mm: page_frag: add '_va' suffix to page_frag API Yunsheng Lin
2024-04-16 16:12 ` Alexander H Duyck
2024-04-17 13:18 ` Yunsheng Lin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox