* [RFC net 0/2] af_packet: asynchronous TPACKET_V3 block retirement
@ 2026-10-28 7:05 Jingxuan Xie
2026-10-28 7:05 ` [RFC net 1/2] af_packet: retire TPACKET_V3 blocks without waiting for copiers Jingxuan Xie
2026-10-28 7:05 ` [RFC net 2/2] af_packet: add KUnit coverage for asynchronous V3 retirement Jingxuan Xie
0 siblings, 2 replies; 4+ messages in thread
From: Jingxuan Xie @ 2026-10-28 7:05 UTC (permalink / raw)
To: netdev; +Cc: willemdebruijn.kernel, davem, edumazet, kuba, pabeni, horms
TPACKET_V3 currently serializes block retirement against payload copies
using a ring-wide read/write lock. The retiring path also holds the
receive queue lock. An unfinished copier can therefore delay unrelated
packet work and the timer, particularly on PREEMPT_RT.
This RFC separates sealing from publishing. At most one block is ACTIVE;
several may be SEALED while their copiers finish. Copy completion retains
its original block, and publication advances strictly in ring order. A
producer only tries the next block and drops capture copies when that
block cannot be reused. The mmap ABI remains unchanged.
The first patch implements this state machine. The second adds KUnit
tests of the actual private helpers. Test-only delay instrumentation is
excluded from both patches.
This is offered for design review, especially:
* Whether maintaining ordered publication of sealed blocks is acceptable.
* Whether eight-block batches plus system_dfl_wq are appropriate for
a completed backlog. Each batch releases the receive queue lock.
* Whether the private metadata cost (24 bytes/block on ARM64) and the
extra completion lock acquisition are acceptable.
Discovery and validation:
* Investigation started with an approximately 355 ms retirement wait in
a real PREEMPT_RT deployment. The trace alone does not establish the
identity or full duration of a task preempting the copier.
* A test-only hook in QEMU delays copier A after its frame reservation.
A second CPU sends B, and another RT task adds load on A's CPU. The
baseline blocks B until A is released. With this series, B returns
while A remains delayed, for both full-block and timer retirement.
Checks of ordered sequence numbers, packet lengths and payload pass;
neither A nor B is published before A completes.
* An uninstrumented QEMU ARM64 virt guest with four CPUs, PREEMPT_RT,
lockdep and virtio-net passes all 17 packet-prb KUnit cases, plus
psock_tpacket and psock_fanout. The injected comparison uses loopback
with an outgoing-only capture filter to avoid a shared NIC TX lock
confounding packet-socket synchronization.
* W=1 af_packet.o builds pass with ARM64 allyesconfig and allmodconfig.
Complete builds fail in unchanged ARM64 syscall-table initialization;
allyesconfig also fails in lockdep formatting with GCC 13 and
CONFIG_WERROR. The errors also
reproduce on the same net baseline; full builds are not claimed as
passing.
Hardware integration, performance overhead, KASAN and KCSAN validation
are outstanding. QEMU results describe generic packet-socket/RT tests,
not hardware validation. Finite capture capacity remains a limitation:
this design avoids waiting for a stalled copier, not capture loss.
An LLM assisted with analysis, implementation, tests and writing. The
technical request was to replace synchronous retirement with per-block
copy accounting, preserve the UAPI and publish completed blocks in order.
Base: f75062536e2d45ee182828f0e47e0124be2fdc8d (net)
jingxuanxie (2):
af_packet: retire TPACKET_V3 blocks without waiting for copiers
af_packet: add KUnit coverage for asynchronous V3 retirement
net/packet/.kunitconfig | 10 +
net/packet/Kconfig | 9 +
net/packet/af_packet.c | 323 +++++++++++-------
net/packet/af_packet_test.c | 649 ++++++++++++++++++++++++++++++++++++
net/packet/internal.h | 27 +-
5 files changed, 893 insertions(+), 125 deletions(-)
create mode 100644 net/packet/.kunitconfig
create mode 100644 net/packet/af_packet_test.c
base-commit: f75062536e2d45ee182828f0e47e0124be2fdc8d
--
2.25.1
________________________________
本电子邮件及转载的任何文件都属绝密文件，并且仅供标明地址的个人或团体使用。如果您收到的电子邮件存在错误，请通知本公司，删除全部原始信息（包括任何原始信息的备份），并且禁止散布或分发全部或部分原始信息。本公司不担保本电子邮件中信息的准确性、适当性或完整性，并且对此产生的任何错误或疏忽不承担任何责任。在正式协议签订以前，这里提到的任何参考术语、与主题相关的情形以及随后的电子邮件中的谈判，都将严格“遵守合同”。接收方应在接收电子邮件或任何附件时检查有无病毒，本公司对由于转载本电子邮件而引发病毒产生的任何损坏不承担任何责任。
Reproduced in this e-mail and any files are considered top-secret documents, and only indicate the address of individuals or groups. If you receive an e-mail there is an error, please notify the Company, delete all the original information (including any backup of the original information), and the prohibition of the dissemination or distribution of all or part of the original information. The Company does not guarantee the accuracy of the information in this e-mail, adequacy, or completeness, and this produces any error or omission does not bear any responsibility. The formal agreement was signed prior to any reference to the term mentioned here, and the circumstances relevant to the subject and subsequent e-mail in the negotiations will be strictly "comply with the contract." The receiver should be receiving an e-mail or any attachment, check for viruses. The company reproduced this e-mail because of the virus caused any damage resulting from not bear any responsibility.
^ permalink raw reply [flat|nested] 4+ messages in thread
* [RFC net 1/2] af_packet: retire TPACKET_V3 blocks without waiting for copiers
2026-10-28 7:05 [RFC net 0/2] af_packet: asynchronous TPACKET_V3 block retirement Jingxuan Xie
@ 2026-10-28 7:05 ` Jingxuan Xie
2026-10-28 21:42 ` Willem de Bruijn
2026-10-28 7:05 ` [RFC net 2/2] af_packet: add KUnit coverage for asynchronous V3 retirement Jingxuan Xie
1 sibling, 1 reply; 4+ messages in thread
From: Jingxuan Xie @ 2026-10-28 7:05 UTC (permalink / raw)
To: netdev; +Cc: willemdebruijn.kernel, davem, edumazet, kuba, pabeni, horms
From: jingxuanxie <jingxuanxie@deeproute.ai>
TPACKET_V3 waits for a ring-wide read lock before retiring a block,
while holding sk_receive_queue.lock. A copier keeps the read lock
after reserving a frame and dropping the queue lock. On PREEMPT_RT,
preempting that copier can therefore delay another packet path or
the retirement timer until the copier runs again.
This was investigated after an approximately 355 ms block-retirement
wait in a real PREEMPT_RT deployment. A QEMU ARM64 SMP PREEMPT_RT
reproducer uses test-only instrumentation to delay one outgoing
copier after reservation, with a second sender forcing retirement.
The unmodified retirement logic blocks the second sender. The
instrumentation is not part of this series.
Track FREE, ACTIVE, SEALED and USER state per block in private
metadata. Reserve and complete copies under the receive queue lock,
retaining the original block reference across the unlocked copy.
Seal a full or expired block and try the immediately following block
without waiting. Publish sealed blocks in reservation order once
their copies have completed. Do not reuse a sealed block merely
because its mmap status still reads TP_STATUS_KERNEL.
Limit each locked publication batch to eight blocks and continue
ready backlogs on an unbound worker. Stop the timer and worker before
freeing metadata. Check readiness using the last published block,
and validate virtio headers before reserving a frame so an invalid
header cannot strand an unfinished reservation.
Tested against that reproducer with both full-block and timer
retirement: B completes while A remains delayed, blocks are not
published early, and sequence numbers, lengths and payload match.
An uninstrumented QEMU guest also passes psock_tpacket and
psock_fanout. The accompanying KUnit suite covers the state helpers.
Keep the mmap layout and userspace status protocol unchanged. Ring
exhaustion can still drop capture copies; a stalled copier no longer
has to block retirement or copying into another available block.
The block producer index must cover the full unsigned block count,
matching the independent publication index across large rings.
Assisted-by: LLM
Fixes: 632ca50f2cbd ("af_packet: TPACKET_V3: replace busy-wait loop")
Signed-off-by: Jingxuan Xie <jingxuanxie@deeproute.ai>
---
net/packet/af_packet.c | 319 +++++++++++++++++++++++++----------------
net/packet/internal.h | 27 +++-
2 files changed, 221 insertions(+), 125 deletions(-)
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 823e0911c910..b48982059a76 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -197,7 +197,6 @@ static void *packet_previous_frame(struct packet_sock *po,
struct packet_ring_buffer *rb,
int status);
static void packet_increment_head(struct packet_ring_buffer *buff);
-static int prb_curr_blk_in_use(struct tpacket_block_desc *);
static void *prb_dispatch_next_block(struct tpacket_kbdq_core *,
struct packet_sock *);
static void prb_retire_current_block(struct tpacket_kbdq_core *,
@@ -205,6 +204,8 @@ static void prb_retire_current_block(struct tpacket_kbdq_core *,
static int prb_queue_frozen(struct tpacket_kbdq_core *);
static void prb_open_block(struct tpacket_kbdq_core *,
struct tpacket_block_desc *);
+static void prb_publish_ready(struct tpacket_kbdq_core *, struct packet_sock *);
+static void prb_retire_work(struct work_struct *);
static enum hrtimer_restart prb_retire_rx_blk_timer_expired(struct hrtimer *);
static void prb_fill_rxhash(struct tpacket_kbdq_core *, struct tpacket3_hdr *);
static void prb_clear_rxhash(struct tpacket_kbdq_core *,
@@ -241,6 +242,26 @@ struct packet_skb_cb {
(((x)->kactive_blk_num < ((x)->knum_blocks-1)) ? \
((x)->kactive_blk_num+1) : 0)
+#define PRB_PUBLISH_BUDGET 8
+
+static unsigned int prb_next_block(const struct tpacket_kbdq_core *pkc,
+ unsigned int index)
+{
+ return index + 1 == pkc->knum_blocks ? 0 : index + 1;
+}
+
+static bool prb_block_reusable(const struct tpacket_kbdq_core *pkc,
+ unsigned int index)
+{
+ const struct prb_block_state *block = &pkc->blocks[index];
+ struct tpacket_block_desc *pbd = GET_PBLOCK_DESC(pkc, index);
+ enum prb_block_phase phase = READ_ONCE(block->phase);
+
+ return phase == PRB_BLOCK_FREE ||
+ (phase == PRB_BLOCK_USER &&
+ !(READ_ONCE(BLOCK_STATUS(pbd)) & TP_STATUS_USER));
+}
+
static void __fanout_unlink(struct sock *sk, struct packet_sock *po);
static void __fanout_link(struct sock *sk, struct packet_sock *po);
@@ -595,7 +616,11 @@ static void prb_shutdown_retire_blk_timer(struct packet_sock *po,
struct tpacket_kbdq_core *pkc;
pkc = GET_PBDQC_FROM_RB(&po->rx_ring);
+ spin_lock_bh(&rb_queue->lock);
+ pkc->retire_shutdown = true;
+ spin_unlock_bh(&rb_queue->lock);
hrtimer_cancel(&pkc->retire_blk_timer);
+ cancel_work_sync(&pkc->retire_work);
}
static int prb_calc_retire_blk_tmo(struct packet_sock *po,
@@ -641,15 +666,22 @@ static void prb_init_ft_ops(struct tpacket_kbdq_core *p1,
p1->feature_req_word = req_u->req3.tp_feature_req_word;
}
-static void init_prb_bdqc(struct packet_sock *po,
- struct packet_ring_buffer *rb,
- struct pgv *pg_vec,
- union tpacket_req_u *req_u)
+static int init_prb_bdqc(struct packet_sock *po,
+ struct packet_ring_buffer *rb,
+ struct pgv *pg_vec,
+ union tpacket_req_u *req_u)
{
struct tpacket_kbdq_core *p1 = GET_PBDQC_FROM_RB(rb);
struct tpacket_block_desc *pbd;
+ struct prb_block_state *blocks;
+
+ blocks = kvcalloc(req_u->req3.tp_block_nr, sizeof(*blocks), GFP_KERNEL);
+ if (!blocks)
+ return -ENOMEM;
memset(p1, 0x0, sizeof(*p1));
+ p1->blocks = blocks;
+ INIT_WORK(&p1->retire_work, prb_retire_work);
p1->knxt_seq_num = 1;
p1->pkbdq = pg_vec;
@@ -666,15 +698,15 @@ static void init_prb_bdqc(struct packet_sock *po,
p1->interval_ktime = ms_to_ktime(prb_calc_retire_blk_tmo(po,
req_u->req3.tp_block_size));
p1->blk_sizeof_priv = req_u->req3.tp_sizeof_priv;
- rwlock_init(&p1->blk_fill_in_prog_lock);
p1->max_frame_len = p1->kblk_size - BLK_PLUS_PRIV(p1->blk_sizeof_priv);
prb_init_ft_ops(p1, req_u);
hrtimer_setup(&p1->retire_blk_timer, prb_retire_rx_blk_timer_expired,
CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
+ prb_open_block(p1, pbd);
hrtimer_start(&p1->retire_blk_timer, p1->interval_ktime,
HRTIMER_MODE_REL_SOFT);
- prb_open_block(p1, pbd);
+ return 0;
}
/*
@@ -697,36 +729,24 @@ static enum hrtimer_restart prb_retire_rx_blk_timer_expired(struct hrtimer *t)
struct tpacket_block_desc *pbd;
spin_lock(&po->sk.sk_receive_queue.lock);
+ if (pkc->retire_shutdown) {
+ spin_unlock(&po->sk.sk_receive_queue.lock);
+ return HRTIMER_NORESTART;
+ }
frozen = prb_queue_frozen(pkc);
pbd = GET_CURR_PBLOCK_DESC_FROM_CORE(pkc);
- /* We only need to plug the race when the block is partially filled.
- * tpacket_rcv:
- * lock(); increment BLOCK_NUM_PKTS; unlock()
- * copy_bits() is in progress ...
- * timer fires on other cpu:
- * we can't retire the current block because copy_bits
- * is in progress.
- *
- */
- if (BLOCK_NUM_PKTS(pbd)) {
- /* Waiting for skb_copy_bits to finish... */
- write_lock(&pkc->blk_fill_in_prog_lock);
- write_unlock(&pkc->blk_fill_in_prog_lock);
- }
-
if (!frozen) {
- if (BLOCK_NUM_PKTS(pbd)) {
+ if (pkc->blocks[pkc->kactive_blk_num].phase == PRB_BLOCK_ACTIVE &&
+ BLOCK_NUM_PKTS(pbd)) {
/* Not an empty block. Need retire the block. */
prb_retire_current_block(pkc, po, TP_STATUS_BLK_TMO);
prb_dispatch_next_block(pkc, po);
}
} else {
- /* Case 1. Queue was frozen because user-space was
- * lagging behind.
- */
- if (!prb_curr_blk_in_use(pbd)) {
+ /* A USER or SEALED block can block the write cursor. */
+ if (prb_block_reusable(pkc, pkc->kactive_blk_num)) {
/* Case 2. queue was frozen,user-space caught up,
* now the link went idle && the timer fired.
* We don't have a block to close.So we open this
@@ -756,7 +776,7 @@ static void prb_flush_block(struct tpacket_kbdq_core *pkc1,
/* Skip the block header(we know header WILL fit in 4K) */
start += PAGE_SIZE;
- end = (u8 *)PAGE_ALIGN((unsigned long)pkc1->pkblk_end);
+ end = (u8 *)PAGE_ALIGN((unsigned long)pbd1 + pkc1->kblk_size);
for (; start < end; start += PAGE_SIZE)
flush_dcache_page(pgv_to_page(start));
@@ -777,15 +797,7 @@ static void prb_flush_block(struct tpacket_kbdq_core *pkc1,
#endif
}
-/*
- * Side effect:
- *
- * 1) flush the block
- * 2) Increment active_blk_num
- *
- * Note:We DONT refresh the timer on purpose.
- * Because almost always the next block will be opened.
- */
+/* The write cursor has moved on; publish using this block's saved metadata. */
static void prb_close_block(struct tpacket_kbdq_core *pkc1,
struct tpacket_block_desc *pbd1,
struct packet_sock *po, unsigned int stat)
@@ -799,7 +811,7 @@ static void prb_close_block(struct tpacket_kbdq_core *pkc1,
if (atomic_read(&po->tp_drops))
status |= TP_STATUS_LOSING;
- last_pkt = (struct tpacket3_hdr *)pkc1->prev;
+ last_pkt = pkc1->blocks[pkc1->publish_blk_num].last_pkt;
last_pkt->tp_next_offset = 0;
/* Get the ts of the last pkt */
@@ -824,8 +836,61 @@ static void prb_close_block(struct tpacket_kbdq_core *pkc1,
prb_flush_block(pkc1, pbd1, status);
sk->sk_data_ready(sk);
+}
+
+static bool prb_publishable(const struct tpacket_kbdq_core *pkc)
+{
+ const struct prb_block_state *block;
+
+ if (!pkc->sealed_blocks || pkc->retire_shutdown)
+ return false;
+ block = &pkc->blocks[pkc->publish_blk_num];
+ return block->phase == PRB_BLOCK_SEALED && !block->in_flight;
+}
- pkc1->kactive_blk_num = GET_NEXT_PRB_BLK_NUM(pkc1);
+/* Caller holds the receive-queue lock; never wait for an unfinished copy. */
+static bool prb_publish_batch(struct tpacket_kbdq_core *pkc,
+ struct packet_sock *po)
+{
+ unsigned int budget = PRB_PUBLISH_BUDGET;
+
+ lockdep_assert_held(&po->sk.sk_receive_queue.lock);
+ while (budget-- && prb_publishable(pkc)) {
+ unsigned int index = pkc->publish_blk_num;
+ struct prb_block_state *block = &pkc->blocks[index];
+ struct tpacket_block_desc *pbd = GET_PBLOCK_DESC(pkc, index);
+
+ prb_close_block(pkc, pbd, po, block->retire_status);
+ WRITE_ONCE(block->phase, PRB_BLOCK_USER);
+ pkc->last_published = index;
+ pkc->has_published = true;
+ pkc->publish_blk_num = prb_next_block(pkc, index);
+ pkc->sealed_blocks--;
+ }
+ return prb_publishable(pkc);
+}
+
+static void prb_publish_ready(struct tpacket_kbdq_core *pkc,
+ struct packet_sock *po)
+{
+ if (prb_publish_batch(pkc, po))
+ queue_work(system_dfl_wq, &pkc->retire_work);
+}
+
+static void prb_retire_work(struct work_struct *work)
+{
+ struct tpacket_kbdq_core *pkc =
+ container_of(work, struct tpacket_kbdq_core, retire_work);
+ struct packet_sock *po =
+ container_of(pkc, struct packet_sock, rx_ring.prb_bdqc);
+ bool more;
+
+ do {
+ spin_lock_bh(&po->sk.sk_receive_queue.lock);
+ more = prb_publish_batch(pkc, po);
+ spin_unlock_bh(&po->sk.sk_receive_queue.lock);
+ cond_resched();
+ } while (more);
}
static void prb_thaw_queue(struct tpacket_kbdq_core *pkc)
@@ -852,6 +917,7 @@ static void prb_open_block(struct tpacket_kbdq_core *pkc1,
{
struct timespec64 ts;
struct tpacket_hdr_v1 *h1 = &pbd1->hdr.bh1;
+ struct prb_block_state *block = &pkc1->blocks[pkc1->kactive_blk_num];
smp_rmb();
@@ -877,6 +943,10 @@ static void prb_open_block(struct tpacket_kbdq_core *pkc1,
pbd1->version = pkc1->version;
pkc1->prev = pkc1->nxt_offset;
pkc1->pkblk_end = pkc1->pkblk_start + pkc1->kblk_size;
+ block->in_flight = 0;
+ block->last_pkt = NULL;
+ block->retire_status = 0;
+ WRITE_ONCE(block->phase, PRB_BLOCK_ACTIVE);
prb_thaw_queue(pkc1);
@@ -932,7 +1002,7 @@ static void *prb_dispatch_next_block(struct tpacket_kbdq_core *pkc,
pbd = GET_CURR_PBLOCK_DESC_FROM_CORE(pkc);
/* 2. If this block is currently in_use then freeze the queue */
- if (TP_STATUS_USER & BLOCK_STATUS(pbd)) {
+ if (!prb_block_reusable(pkc, pkc->kactive_blk_num)) {
prb_freeze_queue(pkc, po);
return NULL;
}
@@ -950,31 +1020,19 @@ static void prb_retire_current_block(struct tpacket_kbdq_core *pkc,
struct packet_sock *po, unsigned int status)
{
struct tpacket_block_desc *pbd = GET_CURR_PBLOCK_DESC_FROM_CORE(pkc);
+ struct prb_block_state *block = &pkc->blocks[pkc->kactive_blk_num];
- /* retire/close the current block */
- if (likely(TP_STATUS_KERNEL == BLOCK_STATUS(pbd))) {
- /*
- * Plug the case where copy_bits() is in progress on
- * cpu-0 and tpacket_rcv() got invoked on cpu-1, didn't
- * have space to copy the pkt in the current block and
- * called prb_retire_current_block()
- *
- * We don't need to worry about the TMO case because
- * the timer-handler already handled this case.
- */
- if (!(status & TP_STATUS_BLK_TMO)) {
- /* Waiting for skb_copy_bits to finish... */
- write_lock(&pkc->blk_fill_in_prog_lock);
- write_unlock(&pkc->blk_fill_in_prog_lock);
- }
- prb_close_block(pkc, pbd, po, status);
+ lockdep_assert_held(&po->sk.sk_receive_queue.lock);
+ if (block->phase != PRB_BLOCK_ACTIVE || !BLOCK_NUM_PKTS(pbd))
return;
- }
-}
-
-static int prb_curr_blk_in_use(struct tpacket_block_desc *pbd)
-{
- return TP_STATUS_USER & BLOCK_STATUS(pbd);
+ block->last_pkt = (struct tpacket3_hdr *)pkc->prev;
+ block->retire_status = status;
+ WRITE_ONCE(block->phase, PRB_BLOCK_SEALED);
+ if (!pkc->sealed_blocks)
+ pkc->publish_blk_num = pkc->kactive_blk_num;
+ pkc->sealed_blocks++;
+ pkc->kactive_blk_num = GET_NEXT_PRB_BLK_NUM(pkc);
+ prb_publish_ready(pkc, po);
}
static int prb_queue_frozen(struct tpacket_kbdq_core *pkc)
@@ -982,12 +1040,17 @@ static int prb_queue_frozen(struct tpacket_kbdq_core *pkc)
return pkc->reset_pending_on_curr_blk;
}
-static void prb_clear_blk_fill_status(struct packet_ring_buffer *rb)
- __releases(&pkc->blk_fill_in_prog_lock)
+static void prb_complete_copy(struct packet_sock *po,
+ struct prb_block_state *block)
{
- struct tpacket_kbdq_core *pkc = GET_PBDQC_FROM_RB(rb);
+ struct tpacket_kbdq_core *pkc = GET_PBDQC_FROM_RB(&po->rx_ring);
- read_unlock(&pkc->blk_fill_in_prog_lock);
+ lockdep_assert_held(&po->sk.sk_receive_queue.lock);
+ if (WARN_ON_ONCE(!block->in_flight))
+ return;
+ block->in_flight--;
+ if (!block->in_flight)
+ prb_publish_ready(pkc, po);
}
static void prb_fill_rxhash(struct tpacket_kbdq_core *pkc,
@@ -1038,7 +1101,6 @@ static void prb_fill_curr_block(char *curr,
struct tpacket_kbdq_core *pkc,
struct tpacket_block_desc *pbd,
unsigned int len)
- __acquires(&pkc->blk_fill_in_prog_lock)
{
struct tpacket3_hdr *ppd;
@@ -1048,7 +1110,7 @@ static void prb_fill_curr_block(char *curr,
pkc->nxt_offset += TOTAL_PKT_LEN_INCL_ALIGN(len);
BLOCK_LEN(pbd) += TOTAL_PKT_LEN_INCL_ALIGN(len);
BLOCK_NUM_PKTS(pbd) += 1;
- read_lock(&pkc->blk_fill_in_prog_lock);
+ pkc->blocks[pkc->kactive_blk_num].in_flight++;
prb_run_all_ft_ops(pkc, ppd);
}
@@ -1065,13 +1127,13 @@ static void *__packet_lookup_frame_in_block(struct packet_sock *po,
pkc = GET_PBDQC_FROM_RB(&po->rx_ring);
pbd = GET_CURR_PBLOCK_DESC_FROM_CORE(pkc);
- /* Queue is frozen when user space is lagging behind */
+ /* The next block may belong to userspace or an unfinished copier. */
if (prb_queue_frozen(pkc)) {
/*
* Check if that last block which caused the queue to freeze,
- * is still in_use by user-space.
+ * is still owned by userspace or has pending publication.
*/
- if (prb_curr_blk_in_use(pbd)) {
+ if (!prb_block_reusable(pkc, pkc->kactive_blk_num)) {
/* Can't record this packet */
return NULL;
} else {
@@ -1091,7 +1153,7 @@ static void *__packet_lookup_frame_in_block(struct packet_sock *po,
end = (char *)pbd + pkc->kblk_size;
/* first try the current block */
- if (curr+TOTAL_PKT_LEN_INCL_ALIGN(len) < end) {
+ if (curr + TOTAL_PKT_LEN_INCL_ALIGN(len) <= end) {
prb_fill_curr_block(curr, pkc, pbd, len);
return (void *)curr;
}
@@ -1134,36 +1196,20 @@ static void *packet_current_rx_frame(struct packet_sock *po,
}
}
-static void *prb_lookup_block(const struct packet_sock *po,
- const struct packet_ring_buffer *rb,
- unsigned int idx,
- int status)
-{
- struct tpacket_kbdq_core *pkc = GET_PBDQC_FROM_RB(rb);
- struct tpacket_block_desc *pbd = GET_PBLOCK_DESC(pkc, idx);
-
- if (status != BLOCK_STATUS(pbd))
- return NULL;
- return pbd;
-}
-
-static int prb_previous_blk_num(struct packet_ring_buffer *rb)
-{
- unsigned int prev;
- if (rb->prb_bdqc.kactive_blk_num)
- prev = rb->prb_bdqc.kactive_blk_num-1;
- else
- prev = rb->prb_bdqc.knum_blocks-1;
- return prev;
-}
-
/* Assumes caller has held the rx_queue.lock */
static void *__prb_previous_block(struct packet_sock *po,
struct packet_ring_buffer *rb,
int status)
{
- unsigned int previous = prb_previous_blk_num(rb);
- return prb_lookup_block(po, rb, previous, status);
+ struct tpacket_kbdq_core *pkc = &rb->prb_bdqc;
+ struct tpacket_block_desc *pbd;
+
+ if (!pkc->has_published)
+ return NULL;
+ pbd = GET_PBLOCK_DESC(pkc, pkc->last_published);
+ if ((READ_ONCE(BLOCK_STATUS(pbd)) & TP_STATUS_USER) != status)
+ return NULL;
+ return pbd;
}
static void *packet_previous_rx_frame(struct packet_sock *po,
@@ -1265,15 +1311,20 @@ static bool __tpacket_has_room(const struct packet_sock *po, int pow_off)
static bool __tpacket_v3_has_room(const struct packet_sock *po, int pow_off)
{
+ const struct tpacket_kbdq_core *pkc = &po->rx_ring.prb_bdqc;
int idx, len;
- len = READ_ONCE(po->rx_ring.prb_bdqc.knum_blocks);
- idx = READ_ONCE(po->rx_ring.prb_bdqc.kactive_blk_num);
+ len = READ_ONCE(pkc->knum_blocks);
+ idx = READ_ONCE(pkc->kactive_blk_num);
+ if (READ_ONCE(pkc->blocks[idx].phase) != PRB_BLOCK_ACTIVE &&
+ !prb_block_reusable(pkc, idx))
+ return false;
if (pow_off)
idx += len >> pow_off;
if (idx >= len)
idx -= len;
- return prb_lookup_block(po, &po->rx_ring, idx, TP_STATUS_KERNEL);
+ return READ_ONCE(pkc->blocks[idx].phase) == PRB_BLOCK_ACTIVE ||
+ prb_block_reusable(pkc, idx);
}
static int __packet_rcv_has_room(const struct packet_sock *po,
@@ -2262,6 +2313,8 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
struct packet_sock *po;
struct sockaddr_ll *sll;
union tpacket_uhdr h;
+ struct prb_block_state *copy_block = NULL;
+ struct virtio_net_hdr vnet_hdr;
u8 *skb_head = skb->data;
int skb_len = skb->len;
unsigned int snaplen, res;
@@ -2375,11 +2428,23 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
vnet_hdr_sz = 0;
}
}
+ if (vnet_hdr_sz &&
+ virtio_net_hdr_from_skb(skb, &vnet_hdr, vio_le(), true, 0)) {
+ atomic_inc(&po->tp_drops);
+ drop_reason = SKB_DROP_REASON_PACKET_SOCK_ERROR;
+ sk_skb_reason_drop(sk, copy_skb, drop_reason);
+ goto drop_n_restore;
+ }
spin_lock(&sk->sk_receive_queue.lock);
h.raw = packet_current_rx_frame(po, skb,
TP_STATUS_KERNEL, (macoff+snaplen));
if (!h.raw)
goto drop_n_account;
+ if (po->tp_version == TPACKET_V3) {
+ struct tpacket_kbdq_core *pkc = &po->rx_ring.prb_bdqc;
+
+ copy_block = &pkc->blocks[pkc->kactive_blk_num];
+ }
if (po->tp_version <= TPACKET_V2) {
slot_id = po->rx_ring.head;
@@ -2388,16 +2453,9 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
__set_bit(slot_id, po->rx_ring.rx_owner_map);
}
- if (vnet_hdr_sz &&
- virtio_net_hdr_from_skb(skb, h.raw + macoff -
- sizeof(struct virtio_net_hdr),
- vio_le(), true, 0)) {
- if (po->tp_version <= TPACKET_V2)
- __clear_bit(slot_id, po->rx_ring.rx_owner_map);
- else
- prb_clear_blk_fill_status(&po->rx_ring);
- goto drop_n_account;
- }
+ if (vnet_hdr_sz)
+ memcpy(h.raw + macoff - sizeof(vnet_hdr), &vnet_hdr,
+ sizeof(vnet_hdr));
if (po->tp_version <= TPACKET_V2) {
packet_increment_rx_head(po, &po->rx_ring);
@@ -2516,7 +2574,9 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
spin_unlock(&sk->sk_receive_queue.lock);
sk->sk_data_ready(sk);
} else if (po->tp_version == TPACKET_V3) {
- prb_clear_blk_fill_status(&po->rx_ring);
+ spin_lock(&sk->sk_receive_queue.lock);
+ prb_complete_copy(po, copy_block);
+ spin_unlock(&sk->sk_receive_queue.lock);
}
drop_n_restore:
@@ -4374,8 +4434,9 @@ static __poll_t packet_poll(struct file *file, struct socket *sock,
spin_lock_bh(&sk->sk_receive_queue.lock);
if (po->rx_ring.pg_vec) {
- if (!packet_previous_rx_frame(po, &po->rx_ring,
- TP_STATUS_KERNEL))
+ if (po->tp_version == TPACKET_V3 ?
+ !!packet_previous_rx_frame(po, &po->rx_ring, TP_STATUS_USER) :
+ !packet_previous_rx_frame(po, &po->rx_ring, TP_STATUS_KERNEL))
mask |= EPOLLIN | EPOLLRDNORM;
}
__packet_rcv_try_clear_pressure(po);
@@ -4564,6 +4625,8 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
int was_running, order = 0;
struct packet_ring_buffer *rb;
struct sk_buff_head *rb_queue;
+ struct prb_block_state *old_blocks = NULL;
+ bool prb_created = false;
__be16 num;
int err;
/* Added to avoid minimal code churn */
@@ -4633,7 +4696,10 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
case TPACKET_V3:
/* Block transmit is not supported yet */
if (!tx_ring) {
- init_prb_bdqc(po, rb, pg_vec, req_u);
+ err = init_prb_bdqc(po, rb, pg_vec, req_u);
+ if (err)
+ goto out_free_pg_vec;
+ prb_created = true;
} else {
struct tpacket_req3 *req3 = &req_u->req3;
@@ -4682,16 +4748,23 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
goto out_unlock;
err = 0;
+ if (!tx_ring && po->tp_version == TPACKET_V3 && rb->pg_vec) {
+ prb_shutdown_retire_blk_timer(po, rb_queue);
+ old_blocks = rb->prb_bdqc.blocks;
+ }
spin_lock_bh(&rb_queue->lock);
swap(rb->pg_vec, pg_vec);
+ if (old_blocks)
+ rb->prb_bdqc.blocks = NULL;
if (po->tp_version <= TPACKET_V2)
swap(rb->rx_owner_map, rx_owner_map);
rb->frame_max = (req->tp_frame_nr - 1);
rb->head = 0;
rb->frame_size = req->tp_frame_size;
- po->prot_hook.func = (po->rx_ring.pg_vec) ?
- tpacket_rcv : packet_rcv;
+ if (!tx_ring)
+ po->prot_hook.func = rb->pg_vec ? tpacket_rcv : packet_rcv;
spin_unlock_bh(&rb_queue->lock);
+ prb_created = false;
swap(rb->pg_vec_order, order);
swap(rb->pg_vec_len, req->tp_block_nr);
@@ -4715,13 +4788,13 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
register_prot_hook(sk);
spin_unlock(&po->bind_lock);
- if (pg_vec && (po->tp_version > TPACKET_V2)) {
- /* Because we don't support block-based V3 on tx-ring */
- if (!tx_ring)
- prb_shutdown_retire_blk_timer(po, rb_queue);
- }
-
out_free_pg_vec:
+ if (prb_created) {
+ prb_shutdown_retire_blk_timer(po, rb_queue);
+ kvfree(rb->prb_bdqc.blocks);
+ rb->prb_bdqc.blocks = NULL;
+ }
+ kvfree(old_blocks);
if (pg_vec) {
bitmap_free(rx_owner_map);
if (tx_ring && closing)
diff --git a/net/packet/internal.h b/net/packet/internal.h
index f5c8cd0eed2c..0761cf6de6d4 100644
--- a/net/packet/internal.h
+++ b/net/packet/internal.h
@@ -3,6 +3,7 @@
#define __PACKET_INTERNAL_H__
#include <linux/refcount.h>
+#include <linux/workqueue.h>
struct packet_mclist {
struct packet_mclist *next;
@@ -14,13 +15,28 @@ struct packet_mclist {
struct list_head remove_list;
};
+enum prb_block_phase {
+ PRB_BLOCK_FREE,
+ PRB_BLOCK_ACTIVE,
+ PRB_BLOCK_SEALED,
+ PRB_BLOCK_USER,
+};
+
+/* Private metadata; never exposed through the packet mmap mapping. */
+struct prb_block_state {
+ enum prb_block_phase phase;
+ unsigned int in_flight;
+ struct tpacket3_hdr *last_pkt;
+ unsigned int retire_status;
+};
+
/* kbdq - kernel block descriptor queue */
struct tpacket_kbdq_core {
struct pgv *pkbdq;
unsigned int feature_req_word;
unsigned int hdrlen;
unsigned char reset_pending_on_curr_blk;
- unsigned short kactive_blk_num;
+ unsigned int kactive_blk_num;
unsigned int blk_sizeof_priv;
unsigned short version;
@@ -36,7 +52,14 @@ struct tpacket_kbdq_core {
struct sk_buff *skb;
- rwlock_t blk_fill_in_prog_lock;
+ /* State changes require sk_receive_queue.lock; copy payload outside it. */
+ struct prb_block_state *blocks;
+ unsigned int publish_blk_num;
+ unsigned int sealed_blocks;
+ unsigned int last_published;
+ bool has_published;
+ bool retire_shutdown;
+ struct work_struct retire_work;
/* Default is set to 8ms */
#define DEFAULT_PRB_RETIRE_TOV (8)
--
2.25.1
________________________________
本电子邮件及转载的任何文件都属绝密文件，并且仅供标明地址的个人或团体使用。如果您收到的电子邮件存在错误，请通知本公司，删除全部原始信息（包括任何原始信息的备份），并且禁止散布或分发全部或部分原始信息。本公司不担保本电子邮件中信息的准确性、适当性或完整性，并且对此产生的任何错误或疏忽不承担任何责任。在正式协议签订以前，这里提到的任何参考术语、与主题相关的情形以及随后的电子邮件中的谈判，都将严格“遵守合同”。接收方应在接收电子邮件或任何附件时检查有无病毒，本公司对由于转载本电子邮件而引发病毒产生的任何损坏不承担任何责任。
Reproduced in this e-mail and any files are considered top-secret documents, and only indicate the address of individuals or groups. If you receive an e-mail there is an error, please notify the Company, delete all the original information (including any backup of the original information), and the prohibition of the dissemination or distribution of all or part of the original information. The Company does not guarantee the accuracy of the information in this e-mail, adequacy, or completeness, and this produces any error or omission does not bear any responsibility. The formal agreement was signed prior to any reference to the term mentioned here, and the circumstances relevant to the subject and subsequent e-mail in the negotiations will be strictly "comply with the contract." The receiver should be receiving an e-mail or any attachment, check for viruses. The company reproduced this e-mail because of the virus caused any damage resulting from not bear any responsibility.
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [RFC net 2/2] af_packet: add KUnit coverage for asynchronous V3 retirement
2026-10-28 7:05 [RFC net 0/2] af_packet: asynchronous TPACKET_V3 block retirement Jingxuan Xie
2026-10-28 7:05 ` [RFC net 1/2] af_packet: retire TPACKET_V3 blocks without waiting for copiers Jingxuan Xie
@ 2026-10-28 7:05 ` Jingxuan Xie
1 sibling, 0 replies; 4+ messages in thread
From: Jingxuan Xie @ 2026-10-28 7:05 UTC (permalink / raw)
To: netdev; +Cc: willemdebruijn.kernel, davem, edumazet, kuba, pabeni, horms
From: jingxuanxie <jingxuanxie@deeproute.ai>
Exercise the actual private reservation, completion, retirement,
timer, poll and ring lifecycle helpers from af_packet.c. Tests are
built only with PACKET_KUNIT_TEST and do not export production test
interfaces or require external packet traffic.
Cover unfinished A with writable B, ordered publication after B
finishes first, completion without subsequent traffic, both seal/copy
orderings, multiple copiers, empty timeouts, single/two-block rings,
exact fits, frozen USER/SEALED blocks, reuse after userspace returns
a block, bounded publication and worker shutdown. Also cover invalid
and valid virtio headers, V2 fallback skb cleanup, concurrent polling
during ring creation/teardown and 65536/65537-block cursor wraparound.
The packet-prb suite passes all 17 cases on QEMU ARM64 with four CPUs,
PREEMPT_RT and lockdep. The same guest passes psock_tpacket and
psock_fanout.
Assisted-by: LLM
Signed-off-by: Jingxuan Xie <jingxuanxie@deeproute.ai>
---
net/packet/.kunitconfig | 10 +
net/packet/Kconfig | 9 +
net/packet/af_packet.c | 4 +
net/packet/af_packet_test.c | 649 ++++++++++++++++++++++++++++++++++++
4 files changed, 672 insertions(+)
create mode 100644 net/packet/.kunitconfig
create mode 100644 net/packet/af_packet_test.c
diff --git a/net/packet/.kunitconfig b/net/packet/.kunitconfig
new file mode 100644
index 000000000000..baed7c14b1d7
--- /dev/null
+++ b/net/packet/.kunitconfig
@@ -0,0 +1,10 @@
+CONFIG_KUNIT=y
+CONFIG_KUNIT_AUTORUN_ENABLED=y
+CONFIG_NET=y
+CONFIG_INET=y
+CONFIG_PACKET=y
+CONFIG_PACKET_KUNIT_TEST=y
+CONFIG_DEBUG_KERNEL=y
+CONFIG_DEBUG_SPINLOCK=y
+CONFIG_DEBUG_LOCK_ALLOC=y
+CONFIG_PROVE_LOCKING=y
diff --git a/net/packet/Kconfig b/net/packet/Kconfig
index 2997382d597c..9f7190ff31df 100644
--- a/net/packet/Kconfig
+++ b/net/packet/Kconfig
@@ -23,3 +23,12 @@ config PACKET_DIAG
help
Support for PF_PACKET sockets monitoring interface used by the ss tool.
If unsure, say Y.
+
+config PACKET_KUNIT_TEST
+ bool "KUnit tests for TPACKET_V3 block retirement" if !KUNIT_ALL_TESTS
+ depends on KUNIT=y && PACKET=y
+ default KUNIT_ALL_TESTS
+ help
+ Test packet ring reservation, block retirement and ordered publication.
+ These tests do not register a network device or require external traffic.
+ If unsure, say N.
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index b48982059a76..b8cb477eb754 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -5037,6 +5037,10 @@ static int __init packet_init(void)
module_init(packet_init);
module_exit(packet_exit);
+
+#if IS_ENABLED(CONFIG_PACKET_KUNIT_TEST)
+#include "af_packet_test.c"
+#endif
MODULE_DESCRIPTION("Packet socket support (AF_PACKET)");
MODULE_LICENSE("GPL");
MODULE_ALIAS_NETPROTO(PF_PACKET);
diff --git a/net/packet/af_packet_test.c b/net/packet/af_packet_test.c
new file mode 100644
index 000000000000..ecb758e11ff6
--- /dev/null
+++ b/net/packet/af_packet_test.c
@@ -0,0 +1,649 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Included by af_packet.c so tests exercise the actual private helpers. */
+#include <kunit/test.h>
+#include <linux/etherdevice.h>
+#include <linux/kthread.h>
+
+struct prb_test_context {
+ struct packet_sock *po;
+ struct socket socket;
+ struct net_device *dev;
+ struct pgv *blocks;
+ unsigned int notifications;
+};
+
+static void prb_test_data_ready(struct sock *sk)
+{
+ struct prb_test_context *ctx = sk->sk_user_data;
+
+ ctx->notifications++;
+}
+
+static int prb_test_init(struct kunit *test)
+{
+ struct prb_test_context *ctx;
+ union tpacket_req_u req = {};
+ unsigned int i;
+ int err;
+
+ ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return -ENOMEM;
+ test->priv = ctx;
+ ctx->po = kunit_kzalloc(test, sizeof(*ctx->po), GFP_KERNEL);
+ ctx->blocks = kunit_kcalloc(test, 16, sizeof(*ctx->blocks), GFP_KERNEL);
+ if (!ctx->po || !ctx->blocks)
+ return -ENOMEM;
+ for (i = 0; i < 16; i++) {
+ ctx->blocks[i].buffer = kunit_kzalloc(test, PAGE_SIZE, GFP_KERNEL);
+ if (!ctx->blocks[i].buffer)
+ return -ENOMEM;
+ }
+ ctx->dev = alloc_netdev(0, "prb%d", NET_NAME_UNKNOWN, ether_setup);
+ if (!ctx->dev)
+ return -ENOMEM;
+ dev_net_set(ctx->dev, &init_net);
+ sock_net_set(&ctx->po->sk, &init_net);
+ ctx->socket.type = SOCK_RAW;
+ init_waitqueue_head(&ctx->socket.wq.wait);
+ sock_init_data_uid(&ctx->socket, &ctx->po->sk, GLOBAL_ROOT_UID);
+ ctx->po->sk.sk_type = SOCK_RAW;
+ ctx->po->sk.sk_user_data = ctx;
+ ctx->po->sk.sk_data_ready = prb_test_data_ready;
+ ctx->po->prot_hook.func = tpacket_rcv;
+ ctx->po->prot_hook.af_packet_priv = &ctx->po->sk;
+ ctx->po->tp_version = TPACKET_V3;
+ ctx->po->tp_hdrlen = TPACKET3_HDRLEN;
+ ctx->po->rx_ring.pg_vec = ctx->blocks;
+ req.req3.tp_block_size = PAGE_SIZE;
+ req.req3.tp_block_nr = 16;
+ req.req3.tp_retire_blk_tov = 60000;
+ err = init_prb_bdqc(ctx->po, &ctx->po->rx_ring, ctx->blocks, &req);
+ if (err) {
+ free_netdev(ctx->dev);
+ return err;
+ }
+ hrtimer_cancel(&ctx->po->rx_ring.prb_bdqc.retire_blk_timer);
+ return 0;
+}
+
+static void prb_test_exit(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+
+ prb_shutdown_retire_blk_timer(ctx->po, &ctx->po->sk.sk_receive_queue);
+ kvfree(ctx->po->rx_ring.prb_bdqc.blocks);
+ free_netdev(ctx->dev);
+}
+
+static void prb_test_invalid_vnet_does_not_reserve(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_block_desc *pbd = (void *)ctx->blocks[0].buffer;
+ struct sk_buff *skb = alloc_skb(128, GFP_KERNEL);
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, skb);
+ skb_put_zero(skb, 64);
+ skb_reset_mac_header(skb);
+ skb_reset_network_header(skb);
+ skb->dev = ctx->dev;
+ skb->protocol = htons(ETH_P_IP);
+ skb_shinfo(skb)->gso_size = 32;
+ skb_shinfo(skb)->gso_type = SKB_GSO_DODGY;
+ ctx->po->vnet_hdr_sz = sizeof(struct virtio_net_hdr);
+ tpacket_rcv(skb, ctx->dev, &ctx->po->prot_hook, ctx->dev);
+ KUNIT_EXPECT_EQ(test, BLOCK_NUM_PKTS(pbd), 0U);
+ KUNIT_EXPECT_EQ(test, BLOCK_LEN(pbd), (u32)BLK_PLUS_PRIV(0));
+ KUNIT_EXPECT_EQ(test, atomic_read(&ctx->po->tp_drops), 1);
+}
+
+struct prb_test_copy {
+ struct prb_block_state *block;
+ struct tpacket3_hdr *hdr;
+};
+
+static struct prb_test_copy prb_test_reserve(struct prb_test_context *ctx,
+ unsigned int length)
+{
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ struct prb_test_copy copy = {};
+ struct sk_buff skb = {};
+
+ spin_lock_bh(&ctx->po->sk.sk_receive_queue.lock);
+ copy.hdr = __packet_lookup_frame_in_block(ctx->po, &skb, length);
+ if (copy.hdr)
+ copy.block = &pkc->blocks[pkc->kactive_blk_num];
+ spin_unlock_bh(&ctx->po->sk.sk_receive_queue.lock);
+ return copy;
+}
+
+static void prb_test_complete(struct prb_test_context *ctx,
+ struct prb_test_copy copy, unsigned int stamp)
+{
+ copy.hdr->tp_len = 64;
+ copy.hdr->tp_snaplen = 64;
+ copy.hdr->tp_sec = stamp;
+ copy.hdr->tp_nsec = stamp;
+ spin_lock_bh(&ctx->po->sk.sk_receive_queue.lock);
+ prb_complete_copy(ctx->po, copy.block);
+ spin_unlock_bh(&ctx->po->sk.sk_receive_queue.lock);
+}
+
+static void prb_test_seal(struct prb_test_context *ctx, unsigned int status)
+{
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+
+ spin_lock_bh(&ctx->po->sk.sk_receive_queue.lock);
+ prb_retire_current_block(pkc, ctx->po, status);
+ prb_dispatch_next_block(pkc, ctx->po);
+ spin_unlock_bh(&ctx->po->sk.sk_receive_queue.lock);
+}
+
+static struct tpacket_block_desc *prb_test_block(struct prb_test_context *ctx,
+ unsigned int index)
+{
+ return (void *)ctx->blocks[index].buffer;
+}
+
+static void prb_test_busy_a_allows_b(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ unsigned int length = pkc->max_frame_len - V3_ALIGNMENT;
+ struct prb_test_copy a = prb_test_reserve(ctx, length);
+ struct prb_test_copy b;
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, a.hdr);
+ b = prb_test_reserve(ctx, length);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, b.hdr);
+ KUNIT_EXPECT_EQ(test, pkc->kactive_blk_num, 1U);
+ KUNIT_EXPECT_EQ(test, a.block->phase, PRB_BLOCK_SEALED);
+ KUNIT_EXPECT_EQ(test, b.block->phase, PRB_BLOCK_ACTIVE);
+ KUNIT_EXPECT_EQ(test, a.block->in_flight, 1U);
+ KUNIT_EXPECT_EQ(test, BLOCK_STATUS(prb_test_block(ctx, 0)), 0U);
+ KUNIT_EXPECT_FALSE(test, !!(packet_poll(NULL, &ctx->socket, NULL) & EPOLLIN));
+ prb_test_complete(ctx, b, 22);
+ prb_test_seal(ctx, 0);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 0U);
+ KUNIT_EXPECT_EQ(test, BLOCK_STATUS(prb_test_block(ctx, 1)), 0U);
+ prb_test_complete(ctx, a, 11);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 2U);
+ KUNIT_EXPECT_EQ(test, pkc->sealed_blocks, 0U);
+ KUNIT_EXPECT_EQ(test, BLOCK_STATUS(prb_test_block(ctx, 0)), TP_STATUS_USER);
+ KUNIT_EXPECT_EQ(test, BLOCK_STATUS(prb_test_block(ctx, 1)), TP_STATUS_USER);
+ KUNIT_EXPECT_EQ(test, prb_test_block(ctx, 0)->hdr.bh1.ts_last_pkt.ts_sec, 11U);
+ KUNIT_EXPECT_EQ(test, prb_test_block(ctx, 1)->hdr.bh1.ts_last_pkt.ts_sec, 22U);
+ KUNIT_EXPECT_EQ(test, BLOCK_SNUM(prb_test_block(ctx, 0)), 1ULL);
+ KUNIT_EXPECT_EQ(test, BLOCK_SNUM(prb_test_block(ctx, 1)), 2ULL);
+ KUNIT_EXPECT_EQ(test, a.hdr->tp_next_offset, 0U);
+ KUNIT_EXPECT_EQ(test, b.hdr->tp_next_offset, 0U);
+ KUNIT_EXPECT_TRUE(test, !!(packet_poll(NULL, &ctx->socket, NULL) & EPOLLIN));
+}
+
+static void prb_test_complete_before_seal(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct prb_test_copy copy = prb_test_reserve(ctx, 128);
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copy.hdr);
+ prb_test_complete(ctx, copy, 7);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 0U);
+ prb_test_seal(ctx, 0);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 1U);
+ KUNIT_EXPECT_EQ(test, BLOCK_STATUS(prb_test_block(ctx, 0)), TP_STATUS_USER);
+}
+
+static void prb_test_multiple_copiers(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct prb_test_copy first = prb_test_reserve(ctx, 128);
+ struct prb_test_copy last = prb_test_reserve(ctx, 128);
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, first.hdr);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, last.hdr);
+ KUNIT_EXPECT_PTR_EQ(test, first.block, last.block);
+ prb_test_seal(ctx, 0);
+ prb_test_complete(ctx, last, 20);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 0U);
+ prb_test_complete(ctx, first, 10);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 1U);
+ KUNIT_EXPECT_EQ(test, prb_test_block(ctx, 0)->hdr.bh1.ts_last_pkt.ts_sec, 20U);
+}
+
+static void prb_test_timeout_busy_and_empty(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ struct prb_test_copy copy;
+
+ prb_retire_rx_blk_timer_expired(&pkc->retire_blk_timer);
+ KUNIT_EXPECT_EQ(test, pkc->kactive_blk_num, 0U);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 0U);
+ copy = prb_test_reserve(ctx, 128);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copy.hdr);
+ prb_retire_rx_blk_timer_expired(&pkc->retire_blk_timer);
+ KUNIT_EXPECT_EQ(test, pkc->kactive_blk_num, 1U);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 0U);
+ prb_test_complete(ctx, copy, 9);
+ KUNIT_EXPECT_EQ(test, BLOCK_STATUS(prb_test_block(ctx, 0)),
+ (u32)(TP_STATUS_USER | TP_STATUS_BLK_TMO));
+ KUNIT_EXPECT_TRUE(test, !!(packet_poll(NULL, &ctx->socket, NULL) & EPOLLIN));
+}
+
+static void prb_test_single_block_wrap(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ struct prb_test_copy copy, next;
+
+ pkc->knum_blocks = 1;
+ copy = prb_test_reserve(ctx, pkc->max_frame_len - V3_ALIGNMENT);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copy.hdr);
+ next = prb_test_reserve(ctx, 128);
+ KUNIT_EXPECT_PTR_EQ(test, next.hdr, NULL);
+ KUNIT_EXPECT_EQ(test, pkc->sealed_blocks, 1U);
+ KUNIT_EXPECT_EQ(test, __packet_rcv_has_room(ctx->po, NULL), ROOM_NONE);
+ prb_test_complete(ctx, copy, 3);
+ next = prb_test_reserve(ctx, 128);
+ KUNIT_EXPECT_PTR_EQ(test, next.hdr, NULL);
+ WRITE_ONCE(BLOCK_STATUS(prb_test_block(ctx, 0)), TP_STATUS_KERNEL);
+ next = prb_test_reserve(ctx, 128);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, next.hdr);
+ KUNIT_EXPECT_EQ(test, BLOCK_SNUM(prb_test_block(ctx, 0)), 2ULL);
+ KUNIT_EXPECT_EQ(test, next.block->in_flight, 1U);
+ prb_test_complete(ctx, next, 4);
+ prb_test_seal(ctx, 0);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 2U);
+}
+
+static void prb_test_exact_fit(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ struct prb_test_copy copy = prb_test_reserve(ctx, pkc->max_frame_len);
+ struct prb_test_copy next;
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copy.hdr);
+ KUNIT_EXPECT_EQ(test, BLOCK_LEN(prb_test_block(ctx, 0)), pkc->kblk_size);
+ next = prb_test_reserve(ctx, 128);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, next.hdr);
+ KUNIT_EXPECT_PTR_EQ(test, next.block, &pkc->blocks[1]);
+ prb_test_complete(ctx, next, 2);
+ prb_test_seal(ctx, 0);
+ prb_test_complete(ctx, copy, 1);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 2U);
+}
+
+static void prb_test_no_scan_past_user(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ struct prb_test_copy copy = prb_test_reserve(ctx, 128);
+ struct prb_test_copy next;
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copy.hdr);
+ pkc->blocks[1].phase = PRB_BLOCK_USER;
+ BLOCK_STATUS(prb_test_block(ctx, 1)) = TP_STATUS_USER;
+ prb_test_seal(ctx, 0);
+ next = prb_test_reserve(ctx, 128);
+ KUNIT_EXPECT_PTR_EQ(test, next.hdr, NULL);
+ KUNIT_EXPECT_EQ(test, pkc->blocks[2].phase, PRB_BLOCK_FREE);
+ prb_test_complete(ctx, copy, 1);
+ WRITE_ONCE(BLOCK_STATUS(prb_test_block(ctx, 1)), TP_STATUS_KERNEL);
+ next = prb_test_reserve(ctx, 128);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, next.hdr);
+ KUNIT_EXPECT_PTR_EQ(test, next.block, &pkc->blocks[1]);
+ prb_test_complete(ctx, next, 2);
+}
+
+static void prb_test_no_reuse_of_sealed(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ struct prb_test_copy a, b, next;
+ unsigned int length = pkc->max_frame_len - V3_ALIGNMENT;
+
+ pkc->knum_blocks = 2;
+ a = prb_test_reserve(ctx, length);
+ b = prb_test_reserve(ctx, length);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, a.hdr);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, b.hdr);
+ next = prb_test_reserve(ctx, 128);
+ KUNIT_EXPECT_PTR_EQ(test, next.hdr, NULL);
+ KUNIT_EXPECT_EQ(test, pkc->sealed_blocks, 2U);
+ KUNIT_EXPECT_EQ(test, a.block->in_flight, 1U);
+ KUNIT_EXPECT_EQ(test, __packet_rcv_has_room(ctx->po, NULL), ROOM_NONE);
+ prb_test_complete(ctx, b, 2);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 0U);
+ prb_test_complete(ctx, a, 1);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 2U);
+ WRITE_ONCE(BLOCK_STATUS(prb_test_block(ctx, 0)), TP_STATUS_KERNEL);
+ next = prb_test_reserve(ctx, 128);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, next.hdr);
+ prb_test_complete(ctx, next, 3);
+}
+
+static void prb_test_large_ring_wrap(struct kunit *test, unsigned int count)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ struct prb_block_state *states;
+ struct prb_test_copy copies[4];
+ struct pgv *pages;
+ unsigned int start = 65534, i;
+
+ pages = kunit_kcalloc(test, count, sizeof(*pages), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pages);
+ states = kvcalloc(count, sizeof(*states), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, states);
+ /* Only blocks around the wrap point need backing storage. */
+ for (i = 0; i <= ARRAY_SIZE(copies); i++)
+ pages[(start + i) % count].buffer = ctx->blocks[i].buffer;
+ kvfree(pkc->blocks);
+ ctx->blocks = pages;
+ ctx->po->rx_ring.pg_vec = pages;
+ pkc->pkbdq = pages;
+ pkc->blocks = states;
+ pkc->knum_blocks = count;
+ pkc->kactive_blk_num = start;
+ prb_open_block(pkc, prb_test_block(ctx, start));
+ for (i = 0; i < ARRAY_SIZE(copies); i++) {
+ copies[i] = prb_test_reserve(ctx, pkc->max_frame_len);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copies[i].hdr);
+ KUNIT_EXPECT_EQ(test, pkc->kactive_blk_num, (start + i) % count);
+ }
+ prb_test_seal(ctx, 0);
+ for (i = 1; i < ARRAY_SIZE(copies); i++)
+ prb_test_complete(ctx, copies[i], i + 1);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 0U);
+ prb_test_complete(ctx, copies[0], 1);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 4U);
+ KUNIT_EXPECT_EQ(test, pkc->sealed_blocks, 0U);
+ KUNIT_EXPECT_EQ(test, pkc->publish_blk_num, (start + 4) % count);
+ for (i = 0; i < ARRAY_SIZE(copies); i++) {
+ struct tpacket_block_desc *pbd = prb_test_block(ctx, (start + i) % count);
+
+ KUNIT_EXPECT_EQ(test, BLOCK_STATUS(pbd), TP_STATUS_USER);
+ KUNIT_EXPECT_EQ(test, BLOCK_SNUM(pbd), (u64)i + 2);
+ }
+}
+
+static void prb_test_65536_blocks(struct kunit *test)
+{
+ prb_test_large_ring_wrap(test, 65536);
+}
+
+static void prb_test_65537_blocks(struct kunit *test)
+{
+ prb_test_large_ring_wrap(test, 65537);
+}
+
+static void prb_test_bounded_publish_and_worker(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ struct prb_test_copy copies[12];
+ unsigned int i, notifications, pending;
+
+ for (i = 0; i < ARRAY_SIZE(copies); i++) {
+ copies[i] = prb_test_reserve(ctx, pkc->max_frame_len - V3_ALIGNMENT);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copies[i].hdr);
+ }
+ prb_test_seal(ctx, 0);
+ for (i = 1; i < ARRAY_SIZE(copies); i++)
+ prb_test_complete(ctx, copies[i], i + 1);
+ copies[0].hdr->tp_sec = 1;
+ spin_lock_bh(&ctx->po->sk.sk_receive_queue.lock);
+ prb_complete_copy(ctx->po, copies[0].block);
+ notifications = ctx->notifications;
+ pending = pkc->sealed_blocks;
+ spin_unlock_bh(&ctx->po->sk.sk_receive_queue.lock);
+ KUNIT_EXPECT_EQ(test, notifications, (unsigned int)PRB_PUBLISH_BUDGET);
+ KUNIT_EXPECT_EQ(test, pending, 4U);
+ flush_work(&pkc->retire_work);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 12U);
+ KUNIT_EXPECT_EQ(test, pkc->sealed_blocks, 0U);
+ for (i = 0; i < ARRAY_SIZE(copies); i++) {
+ KUNIT_EXPECT_EQ(test, BLOCK_STATUS(prb_test_block(ctx, i)), TP_STATUS_USER);
+ KUNIT_EXPECT_EQ(test, BLOCK_SNUM(prb_test_block(ctx, i)), (u64)i + 1);
+ }
+}
+
+static void prb_test_shutdown_stops_publication(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ struct prb_test_copy copy = prb_test_reserve(ctx, 128);
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, copy.hdr);
+ prb_test_seal(ctx, 0);
+ spin_lock_bh(&ctx->po->sk.sk_receive_queue.lock);
+ queue_work(system_dfl_wq, &pkc->retire_work);
+ pkc->retire_shutdown = true;
+ spin_unlock_bh(&ctx->po->sk.sk_receive_queue.lock);
+ prb_shutdown_retire_blk_timer(ctx->po, &ctx->po->sk.sk_receive_queue);
+ prb_test_complete(ctx, copy, 1);
+ KUNIT_EXPECT_EQ(test, ctx->notifications, 0U);
+ KUNIT_EXPECT_FALSE(test, work_pending(&pkc->retire_work));
+ KUNIT_EXPECT_FALSE(test, hrtimer_active(&pkc->retire_blk_timer));
+}
+
+static void prb_test_payload_and_full_ring_drop(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_kbdq_core *pkc = &ctx->po->rx_ring.prb_bdqc;
+ struct tpacket_block_desc *pbd = prb_test_block(ctx, 0);
+ struct tpacket3_hdr *hdr;
+ struct sk_buff *skb;
+ unsigned int i;
+
+ pkc->knum_blocks = 1;
+ for (i = 0; i < 2; i++) {
+ skb = alloc_skb(128, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, skb);
+ memset(skb_put(skb, 64), 0x5a + i, 64);
+ skb_reset_mac_header(skb);
+ skb_reset_network_header(skb);
+ skb->dev = ctx->dev;
+ skb->protocol = htons(ETH_P_IP);
+ local_bh_disable();
+ tpacket_rcv(skb, ctx->dev, &ctx->po->prot_hook, ctx->dev);
+ local_bh_enable();
+ if (!i)
+ prb_test_seal(ctx, 0);
+ }
+ KUNIT_EXPECT_EQ(test, atomic_read(&ctx->po->tp_drops), 1);
+ KUNIT_EXPECT_EQ(test, pkc->blocks[0].in_flight, 0U);
+ KUNIT_EXPECT_EQ(test, BLOCK_NUM_PKTS(pbd), 1U);
+ hdr = (void *)((char *)pbd + pbd->hdr.bh1.offset_to_first_pkt);
+ KUNIT_EXPECT_EQ(test, hdr->tp_len, 64U);
+ KUNIT_EXPECT_EQ(test, hdr->tp_snaplen, 64U);
+ KUNIT_EXPECT_EQ(test, hdr->tp_next_offset, 0U);
+ for (i = 0; i < 64; i++)
+ KUNIT_EXPECT_EQ(test, *((u8 *)hdr + hdr->tp_mac + i), (u8)0x5a);
+}
+
+static void prb_test_valid_vnet_header(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct tpacket_block_desc *pbd = prb_test_block(ctx, 0);
+ struct tpacket3_hdr *hdr;
+ struct virtio_net_hdr *vnet;
+ struct sk_buff *skb = alloc_skb(128, GFP_KERNEL);
+ unsigned int i;
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, skb);
+ memset(skb_put(skb, 64), 0x5a, 64);
+ skb_reset_mac_header(skb);
+ skb_reset_network_header(skb);
+ skb->dev = ctx->dev;
+ skb->protocol = htons(ETH_P_IP);
+ skb_shinfo(skb)->gso_size = 32;
+ skb_shinfo(skb)->gso_type = SKB_GSO_TCPV4 | SKB_GSO_TCP_ECN;
+ skb->ip_summed = CHECKSUM_PARTIAL;
+ skb->csum_start = skb_headroom(skb) + 20;
+ skb->csum_offset = 16;
+ ctx->po->vnet_hdr_sz = sizeof(struct virtio_net_hdr);
+ local_bh_disable();
+ tpacket_rcv(skb, ctx->dev, &ctx->po->prot_hook, ctx->dev);
+ local_bh_enable();
+ prb_test_seal(ctx, 0);
+ KUNIT_EXPECT_EQ(test, atomic_read(&ctx->po->tp_drops), 0);
+ KUNIT_EXPECT_EQ(test, BLOCK_NUM_PKTS(pbd), 1U);
+ hdr = (void *)((char *)pbd + pbd->hdr.bh1.offset_to_first_pkt);
+ vnet = (void *)((char *)hdr + hdr->tp_mac - sizeof(*vnet));
+ KUNIT_EXPECT_EQ(test, vnet->gso_type,
+ (u8)(VIRTIO_NET_HDR_GSO_TCPV4 | VIRTIO_NET_HDR_GSO_ECN));
+ KUNIT_EXPECT_EQ(test, vnet->flags, (u8)VIRTIO_NET_HDR_F_NEEDS_CSUM);
+ KUNIT_EXPECT_EQ(test, __virtio16_to_cpu(vio_le(), vnet->hdr_len), (u16)64);
+ KUNIT_EXPECT_EQ(test, __virtio16_to_cpu(vio_le(), vnet->gso_size), (u16)32);
+ KUNIT_EXPECT_EQ(test, __virtio16_to_cpu(vio_le(), vnet->csum_start), (u16)20);
+ KUNIT_EXPECT_EQ(test, __virtio16_to_cpu(vio_le(), vnet->csum_offset), (u16)16);
+ KUNIT_EXPECT_EQ(test, hdr->tp_snaplen, 64U);
+ for (i = 0; i < 64; i++)
+ KUNIT_EXPECT_EQ(test, *((u8 *)hdr + hdr->tp_mac + i), (u8)0x5a);
+}
+
+static void prb_test_invalid_vnet_releases_copy(struct kunit *test)
+{
+ struct prb_test_context *ctx = test->priv;
+ struct sk_buff *skb = alloc_skb(512, GFP_KERNEL);
+ struct packet_sock *po = ctx->po;
+
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, skb);
+ skb_put_zero(skb, 256);
+ skb_reset_mac_header(skb);
+ skb_reset_network_header(skb);
+ skb->dev = ctx->dev;
+ skb->protocol = htons(ETH_P_IP);
+ skb_shinfo(skb)->gso_size = 32;
+ skb_shinfo(skb)->gso_type = SKB_GSO_DODGY;
+ po->tp_version = TPACKET_V2;
+ po->sk.sk_prot = &packet_proto;
+ po->tp_hdrlen = TPACKET2_HDRLEN;
+ po->rx_ring.frame_size = 128;
+ po->rx_ring.frames_per_block = PAGE_SIZE / 128;
+ WRITE_ONCE(((struct tpacket2_hdr *)ctx->blocks[0].buffer)->tp_status,
+ TP_STATUS_KERNEL);
+ po->vnet_hdr_sz = sizeof(struct virtio_net_hdr);
+ po->copy_thresh = 1;
+ local_bh_disable();
+ tpacket_rcv(skb, ctx->dev, &po->prot_hook, ctx->dev);
+ local_bh_enable();
+ KUNIT_EXPECT_EQ(test, atomic_read(&po->sk.sk_rmem_alloc), 0);
+ KUNIT_EXPECT_EQ(test, atomic_read(&po->tp_drops), 1);
+ KUNIT_EXPECT_TRUE(test, skb_queue_empty(&po->sk.sk_receive_queue));
+}
+
+struct prb_test_poller {
+ struct socket socket;
+ atomic_t polls;
+ struct completion started;
+};
+
+static int prb_test_poll_thread(void *data)
+{
+ struct prb_test_poller *poller = data;
+ struct packet_sock *po = pkt_sk(poller->socket.sk);
+
+ while (!kthread_should_stop()) {
+ packet_sock_flag_set(po, PACKET_SOCK_PRESSURE, true);
+ packet_poll(NULL, &poller->socket, NULL);
+ if (atomic_inc_return(&poller->polls) == 1)
+ complete(&poller->started);
+ cond_resched();
+ }
+ return 0;
+}
+
+static void prb_test_ring_lifecycle(struct kunit *test)
+{
+ struct packet_sock *po;
+ struct prb_test_poller *poller;
+ struct task_struct *task;
+ union tpacket_req_u req = {}, empty = {};
+ unsigned int i;
+ int result;
+
+ po = kunit_kzalloc(test, sizeof(*po), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, po);
+ poller = kunit_kzalloc(test, sizeof(*poller), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, poller);
+ sock_net_set(&po->sk, &init_net);
+ init_waitqueue_head(&poller->socket.wq.wait);
+ sock_init_data_uid(&poller->socket, &po->sk, GLOBAL_ROOT_UID);
+ mutex_init(&po->pg_vec_lock);
+ spin_lock_init(&po->bind_lock);
+ po->tp_version = TPACKET_V3;
+ po->sk.sk_type = SOCK_RAW;
+ po->prot_hook.func = packet_rcv;
+ init_completion(&poller->started);
+ task = kthread_run(prb_test_poll_thread, poller, "prb-poll-test");
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, task);
+ KUNIT_EXPECT_NE(test, wait_for_completion_timeout(&poller->started, HZ), 0UL);
+ for (i = 0; i < 16; i++) {
+ memset(&req, 0, sizeof(req));
+ memset(&empty, 0, sizeof(empty));
+ req.req3.tp_block_size = PAGE_SIZE;
+ req.req3.tp_block_nr = 2;
+ req.req3.tp_frame_size = 256;
+ req.req3.tp_frame_nr = 2 * PAGE_SIZE / 256;
+ req.req3.tp_retire_blk_tov = 1000;
+ result = packet_set_ring(&po->sk, &req, 0, 0);
+ KUNIT_EXPECT_EQ(test, result, 0);
+ if (result)
+ break;
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, po->rx_ring.pg_vec);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, po->rx_ring.prb_bdqc.blocks);
+ queue_work(system_dfl_wq, &po->rx_ring.prb_bdqc.retire_work);
+ atomic_long_set(&po->mapped, 1);
+ KUNIT_EXPECT_EQ(test, packet_set_ring(&po->sk, &empty, 0, 0), -EBUSY);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, po->rx_ring.pg_vec);
+ atomic_long_set(&po->mapped, 0);
+ result = packet_set_ring(&po->sk, &empty, 1, 0);
+ KUNIT_EXPECT_EQ(test, result, 0);
+ if (result)
+ break;
+ KUNIT_EXPECT_PTR_EQ(test, po->rx_ring.pg_vec, NULL);
+ KUNIT_EXPECT_PTR_EQ(test, po->rx_ring.prb_bdqc.blocks, NULL);
+ KUNIT_EXPECT_FALSE(test, hrtimer_active(&po->rx_ring.prb_bdqc.retire_blk_timer));
+ KUNIT_EXPECT_FALSE(test, work_pending(&po->rx_ring.prb_bdqc.retire_work));
+ schedule_timeout_uninterruptible(1);
+ }
+ kthread_stop(task);
+ KUNIT_EXPECT_GT(test, atomic_read(&poller->polls), 0);
+ if (po->rx_ring.pg_vec) {
+ memset(&empty, 0, sizeof(empty));
+ packet_set_ring(&po->sk, &empty, 1, 0);
+ }
+}
+
+static struct kunit_case prb_test_cases[] = {
+ KUNIT_CASE(prb_test_invalid_vnet_does_not_reserve),
+ KUNIT_CASE(prb_test_busy_a_allows_b),
+ KUNIT_CASE(prb_test_complete_before_seal),
+ KUNIT_CASE(prb_test_multiple_copiers),
+ KUNIT_CASE(prb_test_timeout_busy_and_empty),
+ KUNIT_CASE(prb_test_single_block_wrap),
+ KUNIT_CASE(prb_test_exact_fit),
+ KUNIT_CASE(prb_test_no_scan_past_user),
+ KUNIT_CASE(prb_test_no_reuse_of_sealed),
+ KUNIT_CASE(prb_test_65536_blocks),
+ KUNIT_CASE(prb_test_65537_blocks),
+ KUNIT_CASE(prb_test_bounded_publish_and_worker),
+ KUNIT_CASE(prb_test_shutdown_stops_publication),
+ KUNIT_CASE(prb_test_payload_and_full_ring_drop),
+ KUNIT_CASE(prb_test_valid_vnet_header),
+ KUNIT_CASE(prb_test_invalid_vnet_releases_copy),
+ KUNIT_CASE(prb_test_ring_lifecycle),
+ {}
+};
+
+static struct kunit_suite prb_test_suite = {
+ .name = "packet-prb",
+ .init = prb_test_init,
+ .exit = prb_test_exit,
+ .test_cases = prb_test_cases,
+};
+
+kunit_test_suite(prb_test_suite);
--
2.25.1
________________________________
本电子邮件及转载的任何文件都属绝密文件，并且仅供标明地址的个人或团体使用。如果您收到的电子邮件存在错误，请通知本公司，删除全部原始信息（包括任何原始信息的备份），并且禁止散布或分发全部或部分原始信息。本公司不担保本电子邮件中信息的准确性、适当性或完整性，并且对此产生的任何错误或疏忽不承担任何责任。在正式协议签订以前，这里提到的任何参考术语、与主题相关的情形以及随后的电子邮件中的谈判，都将严格“遵守合同”。接收方应在接收电子邮件或任何附件时检查有无病毒，本公司对由于转载本电子邮件而引发病毒产生的任何损坏不承担任何责任。
Reproduced in this e-mail and any files are considered top-secret documents, and only indicate the address of individuals or groups. If you receive an e-mail there is an error, please notify the Company, delete all the original information (including any backup of the original information), and the prohibition of the dissemination or distribution of all or part of the original information. The Company does not guarantee the accuracy of the information in this e-mail, adequacy, or completeness, and this produces any error or omission does not bear any responsibility. The formal agreement was signed prior to any reference to the term mentioned here, and the circumstances relevant to the subject and subsequent e-mail in the negotiations will be strictly "comply with the contract." The receiver should be receiving an e-mail or any attachment, check for viruses. The company reproduced this e-mail because of the virus caused any damage resulting from not bear any responsibility.
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [RFC net 1/2] af_packet: retire TPACKET_V3 blocks without waiting for copiers
2026-10-28 7:05 ` [RFC net 1/2] af_packet: retire TPACKET_V3 blocks without waiting for copiers Jingxuan Xie
@ 2026-10-28 21:42 ` Willem de Bruijn
0 siblings, 0 replies; 4+ messages in thread
From: Willem de Bruijn @ 2026-10-28 21:42 UTC (permalink / raw)
To: Jingxuan Xie, netdev
Cc: willemdebruijn.kernel, davem, edumazet, kuba, pabeni, horms
Jingxuan Xie wrote:
> From: jingxuanxie <jingxuanxie@deeproute.ai>
>
> TPACKET_V3 waits for a ring-wide read lock before retiring a block,
> while holding sk_receive_queue.lock. A copier keeps the read lock
> after reserving a frame and dropping the queue lock. On PREEMPT_RT,
> preempting that copier can therefore delay another packet path or
> the retirement timer until the copier runs again.
>
> This was investigated after an approximately 355 ms block-retirement
> wait in a real PREEMPT_RT deployment. A QEMU ARM64 SMP PREEMPT_RT
> reproducer uses test-only instrumentation to delay one outgoing
> copier after reservation, with a second sender forcing retirement.
> The unmodified retirement logic blocks the second sender. The
> instrumentation is not part of this series.
>
> Track FREE, ACTIVE, SEALED and USER state per block in private
> metadata. Reserve and complete copies under the receive queue lock,
> retaining the original block reference across the unlocked copy.
> Seal a full or expired block and try the immediately following block
> without waiting. Publish sealed blocks in reservation order once
> their copies have completed. Do not reuse a sealed block merely
> because its mmap status still reads TP_STATUS_KERNEL.
>
> Limit each locked publication batch to eight blocks and continue
> ready backlogs on an unbound worker. Stop the timer and worker before
> freeing metadata. Check readiness using the last published block,
> and validate virtio headers before reserving a frame so an invalid
> header cannot strand an unfinished reservation.
>
> Tested against that reproducer with both full-block and timer
> retirement: B completes while A remains delayed, blocks are not
> published early, and sequence numbers, lengths and payload match.
> An uninstrumented QEMU guest also passes psock_tpacket and
> psock_fanout. The accompanying KUnit suite covers the state helpers.
>
> Keep the mmap layout and userspace status protocol unchanged. Ring
> exhaustion can still drop capture copies; a stalled copier no longer
> has to block retirement or copying into another available block.
>
> The block producer index must cover the full unsigned block count,
> matching the independent publication index across large rings.
>
> Assisted-by: LLM
> Fixes: 632ca50f2cbd ("af_packet: TPACKET_V3: replace busy-wait loop")
> Signed-off-by: Jingxuan Xie <jingxuanxie@deeproute.ai>
> ---
> net/packet/af_packet.c | 319 +++++++++++++++++++++++++----------------
> net/packet/internal.h | 27 +++-
This is a huge change. Try to solve the problem with a minimal fix.
The patch is doing more in one patch than needed. Such as moving when
virtio_net_hdr_to_skb is called. Probably for good reason but then that is
a separate change.
The main purpose can hopefully be achieved with less code churn too.
Try to avoid re-acquiring sk_receive_queue.lock, a regression. Batching
adds complexity and is probably not needed. system_dfl_wq runs as
SCHED_NORMAL, so this may introduce a new priority inversion.
Also: whitespace corruption, a confidential ("top secret") footer.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-10-28 21:42 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-28 7:05 [RFC net 0/2] af_packet: asynchronous TPACKET_V3 block retirement Jingxuan Xie
2026-10-28 7:05 ` [RFC net 1/2] af_packet: retire TPACKET_V3 blocks without waiting for copiers Jingxuan Xie
2026-10-28 21:42 ` Willem de Bruijn
2026-10-28 7:05 ` [RFC net 2/2] af_packet: add KUnit coverage for asynchronous V3 retirement Jingxuan Xie
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox