knob to disable locally-originating qdisc optimisation?

knob to disable locally-originating qdisc optimisation?

[Thorsten Glaser]

Hi,

when traffic (e.g. iperf) is originating locally (as opposed to
forward traffic), the Linux kernel seems to apply some optimisations
probably to reduce overall bufferbloat: when the qdisc is “full” or
(and especially) when its dequeue often returns NULL (because packets
are delayed), the sender traffic rate is reduced by as much as ⅓ with
40 ms extra latency (30 → 20 Mbit/s).

This is probably good in general but not so good for L4S where we
actually want the packets to queue up in the qdisc so they get ECN
marking appropriately (I guess there probably are some socket ioctls
or something with which the sending application could detect this
state; if so, we’d be interested in knowing about them as well).

This is especially bad in a testbed for writing L4S-aware applications,
so if there’s a knob (sysctl or something) to disable this optimisation
please do tell (I guess probably not, but asking doesn’t hurt).

Thanks,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: knob to disable locally-originating qdisc optimisation?

[Jakub Kicinski]

On Wed, 26 Apr 2023 14:54:30 +0200 (CEST) Thorsten Glaser wrote:
> when traffic (e.g. iperf) is originating locally (as opposed to
> forward traffic), the Linux kernel seems to apply some optimisations
> probably to reduce overall bufferbloat: when the qdisc is “full” or
> (and especially) when its dequeue often returns NULL (because packets
> are delayed), the sender traffic rate is reduced by as much as ⅓ with
> 40 ms extra latency (30 → 20 Mbit/s).

Doesn't ring a bell, what's your setup?

> This is probably good in general but not so good for L4S where we
> actually want the packets to queue up in the qdisc so they get ECN
> marking appropriately (I guess there probably are some socket ioctls
> or something with which the sending application could detect this
> state; if so, we’d be interested in knowing about them as well).
> 
> This is especially bad in a testbed for writing L4S-aware applications,
> so if there’s a knob (sysctl or something) to disable this optimisation
> please do tell (I guess probably not, but asking doesn’t hurt).

Re: knob to disable locally-originating qdisc optimisation?

[Stephen Hemminger]

On Thu, 27 Apr 2023 13:21:26 -0700
Jakub Kicinski <kuba@kernel.org> wrote:

> On Wed, 26 Apr 2023 14:54:30 +0200 (CEST) Thorsten Glaser wrote:
> > when traffic (e.g. iperf) is originating locally (as opposed to
> > forward traffic), the Linux kernel seems to apply some optimisations
> > probably to reduce overall bufferbloat: when the qdisc is “full” or
> > (and especially) when its dequeue often returns NULL (because packets
> > are delayed), the sender traffic rate is reduced by as much as ⅓ with
> > 40 ms extra latency (30 → 20 Mbit/s).  
> 
> Doesn't ring a bell, what's your setup?
> 
> > This is probably good in general but not so good for L4S where we
> > actually want the packets to queue up in the qdisc so they get ECN
> > marking appropriately (I guess there probably are some socket ioctls
> > or something with which the sending application could detect this
> > state; if so, we’d be interested in knowing about them as well).
> > 
> > This is especially bad in a testbed for writing L4S-aware applications,
> > so if there’s a knob (sysctl or something) to disable this optimisation
> > please do tell (I guess probably not, but asking doesn’t hurt).  

It might be BQL trying to limit outstanding packets locally.

Re: knob to disable locally-originating qdisc optimisation?

[Thorsten Glaser]

On Thu, 27 Apr 2023, Stephen Hemminger wrote:

>On Thu, 27 Apr 2023 13:21:26 -0700
>Jakub Kicinski <kuba@kernel.org> wrote:

>> Doesn't ring a bell, what's your setup?

Intel NUC with Debian bullseye on it and a custom qdisc that
limits and delays outgoing traffic, therefore occasionally
returning NULL from .dequeue even if the qdisc is not empty.

iperf3 sending from the same NUC to a device on the network
behind that qdisc where the corresponding iperf server runs.

>It might be BQL trying to limit outstanding packets locally.

Possibly?

Thanks,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: knob to disable locally-originating qdisc optimisation?

[Toke Høiland-Jørgensen]

Thorsten Glaser <t.glaser@tarent.de> writes:

> On Thu, 27 Apr 2023, Stephen Hemminger wrote:
>
>>On Thu, 27 Apr 2023 13:21:26 -0700
>>Jakub Kicinski <kuba@kernel.org> wrote:
>
>>> Doesn't ring a bell, what's your setup?
>
> Intel NUC with Debian bullseye on it and a custom qdisc that
> limits and delays outgoing traffic, therefore occasionally
> returning NULL from .dequeue even if the qdisc is not empty.
>
> iperf3 sending from the same NUC to a device on the network
> behind that qdisc where the corresponding iperf server runs.
>
>>It might be BQL trying to limit outstanding packets locally.
>
> Possibly?

Sounds like it's TSQ kicking in? The objective of that is to provide the
maximum backpressure against the application socket buffer, precisely so
the application can better react to congestion. Turning that off isn't
going to help your E2E latency, quite the contrary. Pushing stuff into a
qdisc so it can be ECN-marked is also nonsensical for locally generated
traffic; you don't need the ECN roundtrip, you can just directly tell
the local TCP sender to slow down (which is exactly what TSQ does).

-Toke

Re: knob to disable locally-originating qdisc optimisation?

[Thorsten Glaser]

On Tue, 16 May 2023, Toke Høiland-Jørgensen wrote:

>Pushing stuff into a
>qdisc so it can be ECN-marked is also nonsensical for locally generated
>traffic; you don't need the ECN roundtrip, you can just directly tell
>the local TCP sender to slow down (which is exactly what TSQ does).

Yes, but the point of this exercise is to develop algorithms which
react to ECN marking; in production, the RAN BTS will do the marking
so the sender will not be at the place where congestion happens, so
adding that kind of insight is not needed.

Some people have asked for the ability to make Linux behave as if
the sender was remote to ease the test setup (i.e. require one less
machine), nothing more.

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: knob to disable locally-originating qdisc optimisation?

[Toke Høiland-Jørgensen]

Thorsten Glaser <t.glaser@tarent.de> writes:

> On Tue, 16 May 2023, Toke Høiland-Jørgensen wrote:
>
>>Pushing stuff into a
>>qdisc so it can be ECN-marked is also nonsensical for locally generated
>>traffic; you don't need the ECN roundtrip, you can just directly tell
>>the local TCP sender to slow down (which is exactly what TSQ does).
>
> Yes, but the point of this exercise is to develop algorithms which
> react to ECN marking; in production, the RAN BTS will do the marking
> so the sender will not be at the place where congestion happens, so
> adding that kind of insight is not needed.
>
> Some people have asked for the ability to make Linux behave as if
> the sender was remote to ease the test setup (i.e. require one less
> machine), nothing more.

Well, if it's a custom qdisc you could just call skb_orphan() on the
skbs when enqueueing them?

-Toke

Re: knob to disable locally-originating qdisc optimisation?

[Stephen Hemminger]

On Wed, 17 May 2023 00:11:02 +0200
Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Thorsten Glaser <t.glaser@tarent.de> writes:
> 
> > On Tue, 16 May 2023, Toke Høiland-Jørgensen wrote:
> >  
> >>Pushing stuff into a
> >>qdisc so it can be ECN-marked is also nonsensical for locally generated
> >>traffic; you don't need the ECN roundtrip, you can just directly tell
> >>the local TCP sender to slow down (which is exactly what TSQ does).  
> >
> > Yes, but the point of this exercise is to develop algorithms which
> > react to ECN marking; in production, the RAN BTS will do the marking
> > so the sender will not be at the place where congestion happens, so
> > adding that kind of insight is not needed.
> >
> > Some people have asked for the ability to make Linux behave as if
> > the sender was remote to ease the test setup (i.e. require one less
> > machine), nothing more.  
> 
> Well, if it's a custom qdisc you could just call skb_orphan() on the
> skbs when enqueueing them?
> 
> -Toke

If your qdisc was upstream, others could collaborate to resolve the issue.
As it stands, the kernel development process is intentionally hostile
to out of tree developer changes like this.

Re: knob to disable locally-originating qdisc optimisation?

[Thorsten Glaser]

On Tue, 16 May 2023, Stephen Hemminger wrote:

>If your qdisc was upstream, others could collaborate to resolve the issue.
>As it stands, the kernel development process is intentionally hostile
>to out of tree developer changes like this.

The qdisc is a simulation for the network connection of one
5G UE. It’s not suitable as general router qdisc. I doubt
it would even be considered for acceptance. The kernel
upstreaming process is not the least hostile either…

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: knob to disable locally-originating qdisc optimisation?

[Thorsten Glaser]

On Wed, 17 May 2023, Toke Høiland-Jørgensen wrote:

>Well, if it's a custom qdisc you could just call skb_orphan() on the
>skbs when enqueueing them?

What does that even do? (Yes, I found the comment in skbuff.h that
passes for documentation. It isn’t comprehensible to an aspiring
qdisc writer though.)

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: knob to disable locally-originating qdisc optimisation?

[Stephen Hemminger]

On Wed, 17 May 2023 01:40:23 +0200 (CEST)
Thorsten Glaser <t.glaser@tarent.de> wrote:

> On Tue, 16 May 2023, Stephen Hemminger wrote:
> 
> >If your qdisc was upstream, others could collaborate to resolve the issue.
> >As it stands, the kernel development process is intentionally hostile
> >to out of tree developer changes like this.  
> 
> The qdisc is a simulation for the network connection of one
> 5G UE. It’s not suitable as general router qdisc. I doubt
> it would even be considered for acceptance. The kernel
> upstreaming process is not the least hostile either…


Sounds like it could be extension or part of existing netem.

Re: knob to disable locally-originating qdisc optimisation?

[Thorsten Glaser]

On Tue, 16 May 2023, Stephen Hemminger wrote:

>Sounds like it could be extension or part of existing netem.

No, it does so much more… very extensive reporting to userspace
using debugfs/relayfs, for example, and very specific extra
behaviour. We did look at netem, but it caused trouble.

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: knob to disable locally-originating qdisc optimisation?

[Stephen Hemminger]

On Wed, 17 May 2023 01:47:10 +0200 (CEST)
Thorsten Glaser <t.glaser@tarent.de> wrote:

> On Tue, 16 May 2023, Stephen Hemminger wrote:
> 
> >Sounds like it could be extension or part of existing netem.  
> 
> No, it does so much more… very extensive reporting to userspace
> using debugfs/relayfs, for example, and very specific extra
> behaviour. We did look at netem, but it caused trouble.
> 
> bye,
> //mirabilos

The bottom line is the upstream kernel is very unlikely to accept
an knob that is only useful with a non upstream kernel component.

Re: knob to disable locally-originating qdisc optimisation?

[Toke Høiland-Jørgensen]

Thorsten Glaser <t.glaser@tarent.de> writes:

> On Wed, 17 May 2023, Toke Høiland-Jørgensen wrote:
>
>>Well, if it's a custom qdisc you could just call skb_orphan() on the
>>skbs when enqueueing them?
>
> What does that even do? (Yes, I found the comment in skbuff.h that
> passes for documentation. It isn’t comprehensible to an aspiring
> qdisc writer though.)

It detaches the skb from the socket it came from; so from the TCP stack
PoV it will look like the packet left the machine, which should
short-circuit the backpressure thing...

-Toke

Re: knob to disable locally-originating qdisc optimisation?

[Thorsten Glaser]

On Tue, 16 May 2023, Stephen Hemminger wrote:

>The bottom line is the upstream kernel is very unlikely to accept
>an knob that is only useful with a non upstream kernel component.

I wasn’t asking for a new knob, I was asking whether one existed
already and if not then so be it.

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************