Fw: [Bug 220774] New: netem is broken in 6.18

Fw: [Bug 220774] New: netem is broken in 6.18

[Stephen Hemminger]

Regression caused by:

commit ec8e0e3d7adef940cdf9475e2352c0680189d14e
Author: William Liu <will@willsroot.io>
Date:   Tue Jul 8 16:43:26 2025 +0000

    net/sched: Restrict conditions for adding duplicating netems to qdisc tree
    
    netem_enqueue's duplication prevention logic breaks when a netem
    resides in a qdisc tree with other netems - this can lead to a
    soft lockup and OOM loop in netem_dequeue, as seen in [1].
    Ensure that a duplicating netem cannot exist in a tree with other
    netems.
    
    Previous approaches suggested in discussions in chronological order:
    
    1) Track duplication status or ttl in the sk_buff struct. Considered
    too specific a use case to extend such a struct, though this would
    be a resilient fix and address other previous and potential future
    DOS bugs like the one described in loopy fun [2].
    
    2) Restrict netem_enqueue recursion depth like in act_mirred with a
    per cpu variable. However, netem_dequeue can call enqueue on its
    child, and the depth restriction could be bypassed if the child is a
    netem.
    
    3) Use the same approach as in 2, but add metadata in netem_skb_cb
    to handle the netem_dequeue case and track a packet's involvement
    in duplication. This is an overly complex approach, and Jamal
    notes that the skb cb can be overwritten to circumvent this
    safeguard.
    
    4) Prevent the addition of a netem to a qdisc tree if its ancestral
    path contains a netem. However, filters and actions can cause a
    packet to change paths when re-enqueued to the root from netem
    duplication, leading us to the current solution: prevent a
    duplicating netem from inhabiting the same tree as other netems.
    
    [1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
    [2] https://lwn.net/Articles/719297/
    
    Fixes: 0afb51e72855 ("[PKT_SCHED]: netem: reinsert for duplication")
    Reported-by: William Liu <will@willsroot.io>
    Reported-by: Savino Dicanosa <savy@syst3mfailure.io>
    Signed-off-by: William Liu <will@willsroot.io>
    Signed-off-by: Savino Dicanosa <savy@syst3mfailure.io>
    Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
    Link: https://patch.msgid.link/20250708164141.875402-1-will@willsroot.io
    Signed-off-by: Jakub Kicinski <kuba@kernel.org>


Begin forwarded message:

Date: Mon, 10 Nov 2025 19:13:57 +0000
From: bugzilla-daemon@kernel.org
To: stephen@networkplumber.org
Subject: [Bug 220774] New: netem is broken in 6.18


https://bugzilla.kernel.org/show_bug.cgi?id=220774

            Bug ID: 220774
           Summary: netem is broken in 6.18
           Product: Networking
           Version: 2.5
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: high
          Priority: P3
         Component: Other
          Assignee: stephen@networkplumber.org
          Reporter: jschung2@proton.me
        Regression: No

[jschung@localhost ~]$ cat test.sh 
#!/bin/bash

DEV="eth0"
NUM_QUEUES=32
DUPLICATE_PERCENT="5%"

tc qdisc del dev $DEV root > /dev/null 2>&1
tc qdisc add dev $DEV root handle 1: mq

for i in $(seq 1 $NUM_QUEUES); do
    HANDLE_ID=$((i * 10))
    PARENT_ID="1:$i"
    tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem
duplicate $DUPLICATE_PERCENT
done

[jschung@localhost ~]$ sudo ./test.sh 
[  2976.073299] netem: change failed
Error: netem: cannot mix duplicating netems with other netems in tree.

[jschung@localhost ~]$ uname -r
6.18.0-rc4

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

Re: Fw: [Bug 220774] New: netem is broken in 6.18

[Cong Wang]

Hi Will, Jamal and Jakub,

I already warned you many times before you applied it. Now we have users
complaining, please let me know if you still respect users.

Also, Jamal, if I remember correctly, you said you will work on a long
term solution, now after 4 months, please let us know what your plan is.

Regards,
Cong


On Mon, Nov 10, 2025 at 12:38:07PM -0800, Stephen Hemminger wrote:
> Regression caused by:
> 
> commit ec8e0e3d7adef940cdf9475e2352c0680189d14e
> Author: William Liu <will@willsroot.io>
> Date:   Tue Jul 8 16:43:26 2025 +0000
> 
>     net/sched: Restrict conditions for adding duplicating netems to qdisc tree
>     
>     netem_enqueue's duplication prevention logic breaks when a netem
>     resides in a qdisc tree with other netems - this can lead to a
>     soft lockup and OOM loop in netem_dequeue, as seen in [1].
>     Ensure that a duplicating netem cannot exist in a tree with other
>     netems.
>     
>     Previous approaches suggested in discussions in chronological order:
>     
>     1) Track duplication status or ttl in the sk_buff struct. Considered
>     too specific a use case to extend such a struct, though this would
>     be a resilient fix and address other previous and potential future
>     DOS bugs like the one described in loopy fun [2].
>     
>     2) Restrict netem_enqueue recursion depth like in act_mirred with a
>     per cpu variable. However, netem_dequeue can call enqueue on its
>     child, and the depth restriction could be bypassed if the child is a
>     netem.
>     
>     3) Use the same approach as in 2, but add metadata in netem_skb_cb
>     to handle the netem_dequeue case and track a packet's involvement
>     in duplication. This is an overly complex approach, and Jamal
>     notes that the skb cb can be overwritten to circumvent this
>     safeguard.
>     
>     4) Prevent the addition of a netem to a qdisc tree if its ancestral
>     path contains a netem. However, filters and actions can cause a
>     packet to change paths when re-enqueued to the root from netem
>     duplication, leading us to the current solution: prevent a
>     duplicating netem from inhabiting the same tree as other netems.
>     
>     [1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
>     [2] https://lwn.net/Articles/719297/
>     
>     Fixes: 0afb51e72855 ("[PKT_SCHED]: netem: reinsert for duplication")
>     Reported-by: William Liu <will@willsroot.io>
>     Reported-by: Savino Dicanosa <savy@syst3mfailure.io>
>     Signed-off-by: William Liu <will@willsroot.io>
>     Signed-off-by: Savino Dicanosa <savy@syst3mfailure.io>
>     Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
>     Link: https://patch.msgid.link/20250708164141.875402-1-will@willsroot.io
>     Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> 
> 
> Begin forwarded message:
> 
> Date: Mon, 10 Nov 2025 19:13:57 +0000
> From: bugzilla-daemon@kernel.org
> To: stephen@networkplumber.org
> Subject: [Bug 220774] New: netem is broken in 6.18
> 
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=220774
> 
>             Bug ID: 220774
>            Summary: netem is broken in 6.18
>            Product: Networking
>            Version: 2.5
>           Hardware: All
>                 OS: Linux
>             Status: NEW
>           Severity: high
>           Priority: P3
>          Component: Other
>           Assignee: stephen@networkplumber.org
>           Reporter: jschung2@proton.me
>         Regression: No
> 
> [jschung@localhost ~]$ cat test.sh 
> #!/bin/bash
> 
> DEV="eth0"
> NUM_QUEUES=32
> DUPLICATE_PERCENT="5%"
> 
> tc qdisc del dev $DEV root > /dev/null 2>&1
> tc qdisc add dev $DEV root handle 1: mq
> 
> for i in $(seq 1 $NUM_QUEUES); do
>     HANDLE_ID=$((i * 10))
>     PARENT_ID="1:$i"
>     tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem
> duplicate $DUPLICATE_PERCENT
> done
> 
> [jschung@localhost ~]$ sudo ./test.sh 
> [  2976.073299] netem: change failed
> Error: netem: cannot mix duplicating netems with other netems in tree.
> 
> [jschung@localhost ~]$ uname -r
> 6.18.0-rc4
> 
> -- 
> You may reply to this email to add a comment.
> 
> You are receiving this mail because:
> You are the assignee for the bug.

Re: Fw: [Bug 220774] New: netem is broken in 6.18

[Jamal Hadi Salim]

On Thu, Nov 20, 2025 at 11:29 PM Cong Wang <xiyou.wangcong@gmail.com> wrote:
>
> Hi Will, Jamal and Jakub,
>
> I already warned you many times before you applied it. Now we have users
> complaining, please let me know if you still respect users.
>
> Also, Jamal, if I remember correctly, you said you will work on a long
> term solution, now after 4 months, please let us know what your plan is.
>
> Regards,
> Cong
>
>
> On Mon, Nov 10, 2025 at 12:38:07PM -0800, Stephen Hemminger wrote:
> > Regression caused by:
> >
> > commit ec8e0e3d7adef940cdf9475e2352c0680189d14e
> > Author: William Liu <will@willsroot.io>
> > Date:   Tue Jul 8 16:43:26 2025 +0000
> >
> >     net/sched: Restrict conditions for adding duplicating netems to qdisc tree
> >
> >     netem_enqueue's duplication prevention logic breaks when a netem
> >     resides in a qdisc tree with other netems - this can lead to a
> >     soft lockup and OOM loop in netem_dequeue, as seen in [1].
> >     Ensure that a duplicating netem cannot exist in a tree with other
> >     netems.
> >
> >     Previous approaches suggested in discussions in chronological order:
> >
> >     1) Track duplication status or ttl in the sk_buff struct. Considered
> >     too specific a use case to extend such a struct, though this would
> >     be a resilient fix and address other previous and potential future
> >     DOS bugs like the one described in loopy fun [2].
> >
> >     2) Restrict netem_enqueue recursion depth like in act_mirred with a
> >     per cpu variable. However, netem_dequeue can call enqueue on its
> >     child, and the depth restriction could be bypassed if the child is a
> >     netem.
> >
> >     3) Use the same approach as in 2, but add metadata in netem_skb_cb
> >     to handle the netem_dequeue case and track a packet's involvement
> >     in duplication. This is an overly complex approach, and Jamal
> >     notes that the skb cb can be overwritten to circumvent this
> >     safeguard.
> >
> >     4) Prevent the addition of a netem to a qdisc tree if its ancestral
> >     path contains a netem. However, filters and actions can cause a
> >     packet to change paths when re-enqueued to the root from netem
> >     duplication, leading us to the current solution: prevent a
> >     duplicating netem from inhabiting the same tree as other netems.
> >
> >     [1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
> >     [2] https://lwn.net/Articles/719297/
> >
> >     Fixes: 0afb51e72855 ("[PKT_SCHED]: netem: reinsert for duplication")
> >     Reported-by: William Liu <will@willsroot.io>
> >     Reported-by: Savino Dicanosa <savy@syst3mfailure.io>
> >     Signed-off-by: William Liu <will@willsroot.io>
> >     Signed-off-by: Savino Dicanosa <savy@syst3mfailure.io>
> >     Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
> >     Link: https://patch.msgid.link/20250708164141.875402-1-will@willsroot.io
> >     Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> >
> >
> > Begin forwarded message:
> >
> > Date: Mon, 10 Nov 2025 19:13:57 +0000
> > From: bugzilla-daemon@kernel.org
> > To: stephen@networkplumber.org
> > Subject: [Bug 220774] New: netem is broken in 6.18
> >
> >
> > https://bugzilla.kernel.org/show_bug.cgi?id=220774
> >
> >             Bug ID: 220774
> >            Summary: netem is broken in 6.18
> >            Product: Networking
> >            Version: 2.5
> >           Hardware: All
> >                 OS: Linux
> >             Status: NEW
> >           Severity: high
> >           Priority: P3
> >          Component: Other
> >           Assignee: stephen@networkplumber.org
> >           Reporter: jschung2@proton.me
> >         Regression: No
> >
> > [jschung@localhost ~]$ cat test.sh
> > #!/bin/bash
> >
> > DEV="eth0"
> > NUM_QUEUES=32
> > DUPLICATE_PERCENT="5%"
> >
> > tc qdisc del dev $DEV root > /dev/null 2>&1
> > tc qdisc add dev $DEV root handle 1: mq
> >
> > for i in $(seq 1 $NUM_QUEUES); do
> >     HANDLE_ID=$((i * 10))
> >     PARENT_ID="1:$i"
> >     tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem
> > duplicate $DUPLICATE_PERCENT
> > done
> >

jschung2@proton.me: Can you please provide more details about what you
are trying to do so we can see if a different approach can be
prescribed?

cheers,
jamal

> > [jschung@localhost ~]$ sudo ./test.sh
> > [  2976.073299] netem: change failed
> > Error: netem: cannot mix duplicating netems with other netems in tree.
> >
> > [jschung@localhost ~]$ uname -r
> > 6.18.0-rc4
> >
> > --
> > You may reply to this email to add a comment.
> >
> > You are receiving this mail because:
> > You are the assignee for the bug.

Re: Fw: [Bug 220774] New: netem is broken in 6.18

[Cong Wang]

On Fri, Nov 21, 2025 at 07:52:37AM -0500, Jamal Hadi Salim wrote:
 
> jschung2@proton.me: Can you please provide more details about what you
> are trying to do so we can see if a different approach can be
> prescribed?
> 

An alternative approach is to use eBPF qdisc to replace netem, but:
1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
2) I doubt everyone wants to write eBPF code when they already have a
working cmdline.

BTW, Jamal, if your plan is to solve them one by one, even if it could work,
it wouldn't scale. There are still many users don't get hit by this
regression yet (not until hitting LTS or major distro).

Regards,
Cong

Re: [Bug 220774] New: netem is broken in 6.18

[Stephen Hemminger]

On Fri, 21 Nov 2025 13:45:06 -0800
Cong Wang <xiyou.wangcong@gmail.com> wrote:

> On Fri, Nov 21, 2025 at 07:52:37AM -0500, Jamal Hadi Salim wrote:
>  
> > jschung2@proton.me: Can you please provide more details about what you
> > are trying to do so we can see if a different approach can be
> > prescribed?
> >   
> 
> An alternative approach is to use eBPF qdisc to replace netem, but:
> 1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
> 2) I doubt everyone wants to write eBPF code when they already have a
> working cmdline.
> 
> BTW, Jamal, if your plan is to solve them one by one, even if it could work,
> it wouldn't scale. There are still many users don't get hit by this
> regression yet (not until hitting LTS or major distro).
> 
> Regards,
> Cong

The bug still needs to be fixed.
eBPF would still have the same kind of issues.

Re: [Bug 220774] New: netem is broken in 6.18

[Jakub Kicinski]

On Fri, 21 Nov 2025 16:13:22 -0800 Stephen Hemminger wrote:
> On Fri, 21 Nov 2025 13:45:06 -0800
> Cong Wang <xiyou.wangcong@gmail.com> wrote:
> 
> > On Fri, Nov 21, 2025 at 07:52:37AM -0500, Jamal Hadi Salim wrote:
> >    
> > > jschung2@proton.me: Can you please provide more details about what you
> > > are trying to do so we can see if a different approach can be
> > > prescribed?
> > >     
> > 
> > An alternative approach is to use eBPF qdisc to replace netem, but:
> > 1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
> > 2) I doubt everyone wants to write eBPF code when they already have a
> > working cmdline.
> > 
> > BTW, Jamal, if your plan is to solve them one by one, even if it could work,
> > it wouldn't scale. There are still many users don't get hit by this
> > regression yet (not until hitting LTS or major distro).
> 
> The bug still needs to be fixed.
> eBPF would still have the same kind of issues.

I guess we forgot about mq.. IIRC mq doesn't come into play in
duplication, we should be able to just adjust the check to allow 
the mq+netem hierarchy?

Re: [Bug 220774] New: netem is broken in 6.18

[Jamal Hadi Salim]

On Fri, Nov 21, 2025 at 8:55 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Fri, 21 Nov 2025 16:13:22 -0800 Stephen Hemminger wrote:
> > On Fri, 21 Nov 2025 13:45:06 -0800
> > Cong Wang <xiyou.wangcong@gmail.com> wrote:
> >
> > > On Fri, Nov 21, 2025 at 07:52:37AM -0500, Jamal Hadi Salim wrote:
> > >
> > > > jschung2@proton.me: Can you please provide more details about what you
> > > > are trying to do so we can see if a different approach can be
> > > > prescribed?
> > > >
> > >
> > > An alternative approach is to use eBPF qdisc to replace netem, but:
> > > 1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
> > > 2) I doubt everyone wants to write eBPF code when they already have a
> > > working cmdline.
> > >
> > > BTW, Jamal, if your plan is to solve them one by one, even if it could work,
> > > it wouldn't scale. There are still many users don't get hit by this
> > > regression yet (not until hitting LTS or major distro).
> >
> > The bug still needs to be fixed.
> > eBPF would still have the same kind of issues.
>
> I guess we forgot about mq.. IIRC mq doesn't come into play in
> duplication, we should be able to just adjust the check to allow
> the mq+netem hierarchy?

Yes, something like that - but imo it would be fugly to do that check
in current code just for mq.
I proposed to add another qdisc ops which checks for cross-qdisc
semantics. This ops will be invoked by a qdisc's change/init - sort of
what pci quirks ops does. For example in this case it will disallow
setting up netem when it would cause loops but have the quirck checker
allow multiple children such as with mq with as long as none is
looping. I believe some sort of "feature bits" checking should work
here.

The majority of the bugs we are dealing with follow the formula of:
a) create a nonsense hierarchy of qdiscs which has no pratical value,
b) start sending packets
c) netlink cmds to change hierarchy some more; It's more fun if you
can get packets stuck - the formula in this case includes non-working
conserving qdsiscs somewhere in the hierarchy
d) send more packets
e) profit
Current init/change are not really designed to catch things that deal
with children or parents, so catching #a or #c is a challenge. We do
"make it work" eventually but it keeps adding these "checks" for
nonsense setups adds more unnecessary code.

Perhaps its time to start this effort and use this specific scenario
as a use case.

cheers,
jamal

Re: [Bug 220774] New: netem is broken in 6.18

[Cong Wang]

On Fri, Nov 21, 2025 at 05:55:56PM -0800, Jakub Kicinski wrote:
> On Fri, 21 Nov 2025 16:13:22 -0800 Stephen Hemminger wrote:
> > On Fri, 21 Nov 2025 13:45:06 -0800
> > Cong Wang <xiyou.wangcong@gmail.com> wrote:
> > 
> > > On Fri, Nov 21, 2025 at 07:52:37AM -0500, Jamal Hadi Salim wrote:
> > >    
> > > > jschung2@proton.me: Can you please provide more details about what you
> > > > are trying to do so we can see if a different approach can be
> > > > prescribed?
> > > >     
> > > 
> > > An alternative approach is to use eBPF qdisc to replace netem, but:
> > > 1) I am not sure if we could duplicate and re-inject a packet in eBPF Qdisc
> > > 2) I doubt everyone wants to write eBPF code when they already have a
> > > working cmdline.
> > > 
> > > BTW, Jamal, if your plan is to solve them one by one, even if it could work,
> > > it wouldn't scale. There are still many users don't get hit by this
> > > regression yet (not until hitting LTS or major distro).
> > 
> > The bug still needs to be fixed.
> > eBPF would still have the same kind of issues.
> 
> I guess we forgot about mq.. IIRC mq doesn't come into play in
> duplication, we should be able to just adjust the check to allow 

This is not true, I warned you and Jamal with precisely the mq+netem
combination before applying the patch, both of you chose to ignore.

> the mq+netem hierarchy?

This would make the code even uglier, it is already ugly enough to
hard-code and single out this case in the code.

Not to mention there could be other combinations we don't know yet.

We need to revert it and fix the original issue with changing the
problematic duplication behavior.

Regards,
Cong

Re: [Bug 220774] New: netem is broken in 6.18

[Jakub Kicinski]

On Sat, 22 Nov 2025 10:14:50 -0800 Cong Wang wrote:
> > I guess we forgot about mq.. IIRC mq doesn't come into play in
> > duplication, we should be able to just adjust the check to allow   
> 
> This is not true, I warned you and Jamal with precisely the mq+netem
> combination before applying the patch, both of you chose to ignore.

I'm curious why we did.. Link?

Re: [Bug 220774] New: netem is broken in 6.18

[William Liu]

Part of the original issue was that alternative proposed fixes did not work and still allowed for local DOSes [1] [2].

An easy fix that was suggested was to not enqueue from the root [3][4]. However, this changes behavior that has existed since the beginning, immediately caused a new bug, and Hemminger mentioned that the original behavior was necessary for "trying to keep proper semantics and accounting especially when doing rate limits." An argument can be made that Cong's change doesn't violate manpage semantics, but that may still break user setups, just like the current scenario.

Jamal had a good idea on using tc_skb_ext extensions, but I pointed out a few implementation issues we would have to deal with [5]. From what I understood though, that solution would preserve all the existing behaviors and avoid the DOS bug. It's been a while so let me know if that is not the case. Otherwise, I can try to implement it - I am quite busy though so it might take a while.

On Tuesday, November 25th, 2025 at 3:16 AM, Jakub Kicinski <kuba@kernel.org> wrote:

> 
> 
> On Sat, 22 Nov 2025 10:14:50 -0800 Cong Wang wrote:
> 
> > > I guess we forgot about mq.. IIRC mq doesn't come into play in
> > > duplication, we should be able to just adjust the check to allow
> > 
> > This is not true, I warned you and Jamal with precisely the mq+netem
> > combination before applying the patch, both of you chose to ignore.
> 
> 
> I'm curious why we did.. Link?

The issue was due to filters redirecting packets iirc [6]. We decided to move with it and come back to it if someone did rely on this rare setup [7].

If anyone wants a quick refresher on the bug, please take a read at [6] as it contains all the details and issues.

[1] https://lore.kernel.org/netdev/20250701231306.376762-1-xiyou.wangcong@gmail.com/T/#u
[2] https://lore.kernel.org/netdev/20250707195015.823492-1-xiyou.wangcong@gmail.com/T/#u
[3] https://lore.kernel.org/netdev/20250713214748.1377876-1-xiyou.wangcong@gmail.com/T/#u
[4] https://lore.kernel.org/netdev/CAM0EoMmTZon=nFmLsDPKhDEzHruw701iV9=mq92At9oKo0LGpA@mail.gmail.com/T/#u
[5] https://lore.kernel.org/netdev/pGE9OHWRSf4oJwC4gS0oPonBy8_0WsDthxgLzBYGBtMVeT_EDc-HAz8NbhJxcWe0NEUrf_a7Fyq2op5FVFujfc2KyO-I38Yx_HlQhFwB0Cs=@willsroot.io/
[6] https://lore.kernel.org/netdev/CAM0EoMnd0nZxJW3zpEuBGWTwB3AnJSnj242f9hMpcLdBWdcbfQ@mail.gmail.com/
[7] https://lore.kernel.org/netdev/20250707142617.10849b9e@kernel.org/

Re: [Bug 220774] New: netem is broken in 6.18

[Cong Wang]

On Mon, Nov 24, 2025 at 07:16:25PM -0800, Jakub Kicinski wrote:
> On Sat, 22 Nov 2025 10:14:50 -0800 Cong Wang wrote:
> > > I guess we forgot about mq.. IIRC mq doesn't come into play in
> > > duplication, we should be able to just adjust the check to allow   
> > 
> > This is not true, I warned you and Jamal with precisely the mq+netem
> > combination before applying the patch, both of you chose to ignore.
> 
> I'm curious why we did.. Link?

https://lore.kernel.org/all/aG10rqwjX6elG1Gx@pop-os.localdomain/#t

Jamal just denied the use case and let users complain.

This strategy does not work, since majority users would need to wait
until LTS gets hit by this regression. (IMHO, it is also unethical to
knowingly break valid use cases, regardless of purpose.)

Regards,
Cong Wang

Re: Fw: [Bug 220774] New: netem is broken in 6.18

[정지수]

#!/bin/bash

set -euo pipefail

DEV="wlo0"
QUEUE="1"
RTP_DST_PORT="5004"
DUP_PCT="25%"
CORR_PCT="60%"
DELAY="1ms"
VERIFY_SECONDS=15

usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P UDP_PORT]"; exit 1; }
while [[ $# -gt 0 ]]; do
  case "$1" in
    -d) DEV="$2"; shift 2;;
    -q) QUEUE="$2"; shift 2;;
    -P) RTP_DST_PORT="$2"; shift 2;;
    *) usage;;
  endac
done || true

[[ -d /sys/class/net/$DEV ]] || { echo "No such dev $DEV"; exit 1; }


if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then
  echo "Setting root qdisc to mq.."
  tc qdisc replace dev "$DEV" root handle 1: mq
fi


echo "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..."
ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || {
  echo "some flows will still land on :$QUEUE"
}


echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.."
tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
  netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"

tc qdisc show dev "$DEV"

echo
echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true


if command -v tshark >/dev/null 2>&1; then
  echo "tshark live RTP view :"
  timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true
fi

echo
echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):"
tc -s qdisc show dev "$DEV" | sed -n '1,200p'




Sent with Proton Mail secure email.

On Friday, November 21st, 2025 at 12:52, Jamal Hadi Salim <jhs@mojatatu.com> wrote:

> On Thu, Nov 20, 2025 at 11:29 PM Cong Wang xiyou.wangcong@gmail.com wrote:
> 
> > Hi Will, Jamal and Jakub,
> > 
> > I already warned you many times before you applied it. Now we have users
> > complaining, please let me know if you still respect users.
> > 
> > Also, Jamal, if I remember correctly, you said you will work on a long
> > term solution, now after 4 months, please let us know what your plan is.
> > 
> > Regards,
> > Cong
> > 
> > On Mon, Nov 10, 2025 at 12:38:07PM -0800, Stephen Hemminger wrote:
> > 
> > > Regression caused by:
> > > 
> > > commit ec8e0e3d7adef940cdf9475e2352c0680189d14e
> > > Author: William Liu will@willsroot.io
> > > Date: Tue Jul 8 16:43:26 2025 +0000
> > > 
> > > net/sched: Restrict conditions for adding duplicating netems to qdisc tree
> > > 
> > > netem_enqueue's duplication prevention logic breaks when a netem
> > > resides in a qdisc tree with other netems - this can lead to a
> > > soft lockup and OOM loop in netem_dequeue, as seen in [1].
> > > Ensure that a duplicating netem cannot exist in a tree with other
> > > netems.
> > > 
> > > Previous approaches suggested in discussions in chronological order:
> > > 
> > > 1) Track duplication status or ttl in the sk_buff struct. Considered
> > > too specific a use case to extend such a struct, though this would
> > > be a resilient fix and address other previous and potential future
> > > DOS bugs like the one described in loopy fun [2].
> > > 
> > > 2) Restrict netem_enqueue recursion depth like in act_mirred with a
> > > per cpu variable. However, netem_dequeue can call enqueue on its
> > > child, and the depth restriction could be bypassed if the child is a
> > > netem.
> > > 
> > > 3) Use the same approach as in 2, but add metadata in netem_skb_cb
> > > to handle the netem_dequeue case and track a packet's involvement
> > > in duplication. This is an overly complex approach, and Jamal
> > > notes that the skb cb can be overwritten to circumvent this
> > > safeguard.
> > > 
> > > 4) Prevent the addition of a netem to a qdisc tree if its ancestral
> > > path contains a netem. However, filters and actions can cause a
> > > packet to change paths when re-enqueued to the root from netem
> > > duplication, leading us to the current solution: prevent a
> > > duplicating netem from inhabiting the same tree as other netems.
> > > 
> > > [1] https://lore.kernel.org/netdev/8DuRWwfqjoRDLDmBMlIfbrsZg9Gx50DHJc1ilxsEBNe2D6NMoigR_eIRIG0LOjMc3r10nUUZtArXx4oZBIdUfZQrwjcQhdinnMis_0G7VEk=@willsroot.io/
> > > [2] https://lwn.net/Articles/719297/
> > > 
> > > Fixes: 0afb51e72855 ("[PKT_SCHED]: netem: reinsert for duplication")
> > > Reported-by: William Liu will@willsroot.io
> > > Reported-by: Savino Dicanosa savy@syst3mfailure.io
> > > Signed-off-by: William Liu will@willsroot.io
> > > Signed-off-by: Savino Dicanosa savy@syst3mfailure.io
> > > Acked-by: Jamal Hadi Salim jhs@mojatatu.com
> > > Link: https://patch.msgid.link/20250708164141.875402-1-will@willsroot.io
> > > Signed-off-by: Jakub Kicinski kuba@kernel.org
> > > 
> > > Begin forwarded message:
> > > 
> > > Date: Mon, 10 Nov 2025 19:13:57 +0000
> > > From: bugzilla-daemon@kernel.org
> > > To: stephen@networkplumber.org
> > > Subject: [Bug 220774] New: netem is broken in 6.18
> > > 
> > > https://bugzilla.kernel.org/show_bug.cgi?id=220774
> > > 
> > > Bug ID: 220774
> > > Summary: netem is broken in 6.18
> > > Product: Networking
> > > Version: 2.5
> > > Hardware: All
> > > OS: Linux
> > > Status: NEW
> > > Severity: high
> > > Priority: P3
> > > Component: Other
> > > Assignee: stephen@networkplumber.org
> > > Reporter: jschung2@proton.me
> > > Regression: No
> > > 
> > > [jschung@localhost ~]$ cat test.sh
> > > #!/bin/bash
> > > 
> > > DEV="eth0"
> > > NUM_QUEUES=32
> > > DUPLICATE_PERCENT="5%"
> > > 
> > > tc qdisc del dev $DEV root > /dev/null 2>&1
> > > tc qdisc add dev $DEV root handle 1: mq
> > > 
> > > for i in $(seq 1 $NUM_QUEUES); do
> > > HANDLE_ID=$((i * 10))
> > > PARENT_ID="1:$i"
> > > tc qdisc add dev $DEV parent $PARENT_ID handle ${HANDLE_ID}: netem
> > > duplicate $DUPLICATE_PERCENT
> > > done
> 
> 
> jschung2@proton.me: Can you please provide more details about what you
> are trying to do so we can see if a different approach can be
> prescribed?
> 
> cheers,
> jamal
> 
> > > [jschung@localhost ~]$ sudo ./test.sh
> > > [ 2976.073299] netem: change failed
> > > Error: netem: cannot mix duplicating netems with other netems in tree.
> > > 
> > > [jschung@localhost ~]$ uname -r
> > > 6.18.0-rc4
> > > 
> > > --
> > > You may reply to this email to add a comment.
> > > 
> > > You are receiving this mail because:
> > > You are the assignee for the bug.

Re: Fw: [Bug 220774] New: netem is broken in 6.18

[Jamal Hadi Salim]

Hi 정지수,

On Sat, Nov 22, 2025 at 1:56 AM 정지수 <jschung2@proton.me> wrote:
>
>
> #!/bin/bash
>
> set -euo pipefail
>
> DEV="wlo0"
> QUEUE="1"
> RTP_DST_PORT="5004"
> DUP_PCT="25%"
> CORR_PCT="60%"
> DELAY="1ms"
> VERIFY_SECONDS=15
>
> usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P UDP_PORT]"; exit 1; }
> while [[ $# -gt 0 ]]; do
>   case "$1" in
>     -d) DEV="$2"; shift 2;;
>     -q) QUEUE="$2"; shift 2;;
>     -P) RTP_DST_PORT="$2"; shift 2;;
>     *) usage;;
>   endac
> done || true
>
> [[ -d /sys/class/net/$DEV ]] || { echo "No such dev $DEV"; exit 1; }
>
>
> if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then
>   echo "Setting root qdisc to mq.."
>   tc qdisc replace dev "$DEV" root handle 1: mq
> fi
>
>
> echo "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..."
> ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || {
>   echo "some flows will still land on :$QUEUE"
> }
>
>
> echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.."
> tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
>   netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"
>
> tc qdisc show dev "$DEV"
>
> echo
> echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
> echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
> timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true
>
>
> if command -v tshark >/dev/null 2>&1; then
>   echo "tshark live RTP view :"
>   timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true
> fi
>
> echo
> echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):"
> tc -s qdisc show dev "$DEV" | sed -n '1,200p'
>

Thanks for the config.

If you can talk about it: I was more interested in what your end goal is.
From the dev name it seems $DEV is a wireless device? Are you
replicating these RTP packets across different ssids mapped to
different hw queues? Are you forwarding these packets? The ethtool
config indicates the RX direction but the netem replication is on the
tx.
And in the short term if a tc action could achieve what you are trying
to achieve - would that work for you?

cheers,
jamal

Re: Fw: [Bug 220774] New: netem is broken in 6.18

[Cong Wang]

On Sat, Nov 22, 2025 at 12:24:43PM -0500, Jamal Hadi Salim wrote:
> If you can talk about it: I was more interested in what your end goal is.
> From the dev name it seems $DEV is a wireless device? Are you
> replicating these RTP packets across different ssids mapped to
> different hw queues? Are you forwarding these packets? The ethtool
> config indicates the RX direction but the netem replication is on the
> tx.
> And in the short term if a tc action could achieve what you are trying
> to achieve - would that work for you?

I am not speaking for Ji-Soo, but which tc action are you talking about
here?

Personally, I am not aware of any tc action could achieve the same netem
duplication. gact offers some randomness, mirred offers duplication, but
it is not trivial to just combine them to be same as netem duplication.

Regards,
Cong Wang

Re: Fw: [Bug 220774] New: netem is broken in 6.18

[정지수]

Mr Salim,

I do not understand what your saying. You provide a bash script, and I can test

- Ji-Soo

에 2025년 11월 22일 토요일 17:24 Jamal Hadi Salim <jhs@mojatatu.com> 님이 작성함:

> 
> 
> Hi 정지수,
> 
> On Sat, Nov 22, 2025 at 1:56 AM 정지수 jschung2@proton.me wrote:
> 
> > #!/bin/bash
> > 
> > set -euo pipefail
> > 
> > DEV="wlo0"
> > QUEUE="1"
> > RTP_DST_PORT="5004"
> > DUP_PCT="25%"
> > CORR_PCT="60%"
> > DELAY="1ms"
> > VERIFY_SECONDS=15
> > 
> > usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P UDP_PORT]"; exit 1; }
> > while [[ $# -gt 0 ]]; do
> > case "$1" in
> > -d) DEV="$2"; shift 2;;
> > -q) QUEUE="$2"; shift 2;;
> > -P) RTP_DST_PORT="$2"; shift 2;;
> > *) usage;;
> > endac
> > done || true
> > 
> > [[ -d /sys/class/net/$DEV ]] || { echo "No such dev $DEV"; exit 1; }
> > 
> > if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then
> > echo "Setting root qdisc to mq.."
> > tc qdisc replace dev "$DEV" root handle 1: mq
> > fi
> > 
> > echo "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..."
> > ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || {
> > echo "some flows will still land on :$QUEUE"
> > }
> > 
> > echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.."
> > tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
> > netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"
> > 
> > tc qdisc show dev "$DEV"
> > 
> > echo
> > echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
> > echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
> > timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true
> > 
> > if command -v tshark >/dev/null 2>&1; then
> > echo "tshark live RTP view :"
> > timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true
> > fi
> > 
> > echo
> > echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):"
> > tc -s qdisc show dev "$DEV" | sed -n '1,200p'
> 
> 
> Thanks for the config.
> 
> If you can talk about it: I was more interested in what your end goal is.
> From the dev name it seems $DEV is a wireless device? Are you
> replicating these RTP packets across different ssids mapped to
> different hw queues? Are you forwarding these packets? The ethtool
> config indicates the RX direction but the netem replication is on the
> tx.
> And in the short term if a tc action could achieve what you are trying
> to achieve - would that work for you?
> 
> cheers,
> jamal

Re: Fw: [Bug 220774] New: netem is broken in 6.18

[Jamal Hadi Salim]

Hi,

On Thu, Nov 27, 2025 at 1:08 AM 정지수 <jschung2@proton.me> wrote:
>
> Mr Salim,
>
> I do not understand what your saying. You provide a bash script, and I can test
>

Will you be willing to go on a zoom/gmeet call? I think it will be
easier to communicate.

cheers,
jamal
> - Ji-Soo
>
> 에 2025년 11월 22일 토요일 17:24 Jamal Hadi Salim <jhs@mojatatu.com> 님이 작성함:
>
> >
> >
> > Hi 정지수,
> >
> > On Sat, Nov 22, 2025 at 1:56 AM 정지수 jschung2@proton.me wrote:
> >
> > > #!/bin/bash
> > >
> > > set -euo pipefail
> > >
> > > DEV="wlo0"
> > > QUEUE="1"
> > > RTP_DST_PORT="5004"
> > > DUP_PCT="25%"
> > > CORR_PCT="60%"
> > > DELAY="1ms"
> > > VERIFY_SECONDS=15
> > >
> > > usage(){ echo "Usage: sudo $0 [-d DEV] [-q QUEUE] [-P UDP_PORT]"; exit 1; }
> > > while [[ $# -gt 0 ]]; do
> > > case "$1" in
> > > -d) DEV="$2"; shift 2;;
> > > -q) QUEUE="$2"; shift 2;;
> > > -P) RTP_DST_PORT="$2"; shift 2;;
> > > *) usage;;
> > > endac
> > > done || true
> > >
> > > [[ -d /sys/class/net/$DEV ]] || { echo "No such dev $DEV"; exit 1; }
> > >
> > > if ! tc qdisc show dev "$DEV" | grep -q ' qdisc mq '; then
> > > echo "Setting root qdisc to mq.."
> > > tc qdisc replace dev "$DEV" root handle 1: mq
> > > fi
> > >
> > > echo "Adding ntuple to steer UDP dport $RTP_DST_PORT -> tx-queue $QUEUE..."
> > > ethtool -N "$DEV" flow-type udp4 dst-port $RTP_DST_PORT action $QUEUE || {
> > > echo "some flows will still land on :$QUEUE"
> > > }
> > >
> > > echo "Attaching netem duplicate=$DUP_PCT corr=$CORR_PCT delay=$DELAY on parent :$QUEUE.."
> > > tc qdisc replace dev "$DEV" parent :$QUEUE handle ${QUEUE}00: \
> > > netem duplicate "$DUP_PCT" "$CORR_PCT" delay "$DELAY"
> > >
> > > tc qdisc show dev "$DEV"
> > >
> > > echo
> > > echo "Start an RTP/WebRTC/SFU downlink to a test client on UDP port $RTP_DST_PORT."
> > > echo "Capturing for $VERIFY_SECONDS s to observe duplicates by RTP seqno.."
> > > timeout "$VERIFY_SECONDS" tcpdump -ni "$DEV" "udp and dst port $RTP_DST_PORT" -vv -c 0 2>/dev/null | head -n 3 || true
> > >
> > > if command -v tshark >/dev/null 2>&1; then
> > > echo "tshark live RTP view :"
> > > timeout "$VERIFY_SECONDS" tshark -i "$DEV" -f "udp dst port $RTP_DST_PORT" -q -z rtp,streams || true
> > > fi
> > >
> > > echo
> > > echo "Netem stats, see duplicated counters under handle ${QUEUE}00:):"
> > > tc -s qdisc show dev "$DEV" | sed -n '1,200p'
> >
> >
> > Thanks for the config.
> >
> > If you can talk about it: I was more interested in what your end goal is.
> > From the dev name it seems $DEV is a wireless device? Are you
> > replicating these RTP packets across different ssids mapped to
> > different hw queues? Are you forwarding these packets? The ethtool
> > config indicates the RX direction but the netem replication is on the
> > tx.
> > And in the short term if a tc action could achieve what you are trying
> > to achieve - would that work for you?
> >
> > cheers,
> > jamal