inter-qdisc communication?

inter-qdisc communication?

[Thorsten Glaser]

Hi,

is it possible for qdiscs to communicate with each other?

For example, if I have a normal egress qdisc and an ingress
qdisc, how could I “share” configuration values so that e.g.
a “tc change” can affect both in one run?

Use case: we have a normal egress qdisc that does a lot of
things. Now we add artificial latency, and we need to do that
on ingress as well, for symmetry, obviously, so I’ll write a
small qdisc that does just that for ingress. But we’re already
firing a “tc change” to the egress qdisc every few ms or so,
and I don’t want to double that overhead.

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: inter-qdisc communication?

[Jakub Kicinski]

On Fri, 26 Aug 2022 21:34:47 +0200 (CEST) Thorsten Glaser wrote:
> Hi,
> 
> is it possible for qdiscs to communicate with each other?
> 
> For example, if I have a normal egress qdisc and an ingress
> qdisc, how could I “share” configuration values so that e.g.
> a “tc change” can affect both in one run?
> 
> Use case: we have a normal egress qdisc that does a lot of
> things. Now we add artificial latency, and we need to do that
> on ingress as well, for symmetry, obviously, so I’ll write a
> small qdisc that does just that for ingress. But we’re already
> firing a “tc change” to the egress qdisc every few ms or so,
> and I don’t want to double that overhead.

How do you add latency on ingress? 🤔 
The ingress qdisc is just a stub to hook classifiers/actions. 

Re: inter-qdisc communication?

[Thorsten Glaser]

On Fri, 26 Aug 2022, Jakub Kicinski wrote:

> How do you add latency on ingress? 🤔 
> The ingress qdisc is just a stub to hook classifiers/actions. 

Oh, damn. Then, I guess, I’ll have to do that on egress on
the other interface, which makes it at least symmetric for
passing traffic but catch not the set of traffic it should.
Especially not the traffic terminating locally. Meh.

The question remains the same, just the use case magically
mutated under me.

(Maybe if there were documentation like an intro to qdisc
writing like I asked for already, I’d have known that.)

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: inter-qdisc communication?

[Jakub Kicinski]

On Sat, 27 Aug 2022 02:45:02 +0200 (CEST) Thorsten Glaser wrote:
> > How do you add latency on ingress? 🤔 
> > The ingress qdisc is just a stub to hook classifiers/actions.   
> 
> Oh, damn. Then, I guess, I’ll have to do that on egress on
> the other interface, which makes it at least symmetric for
> passing traffic but catch not the set of traffic it should.
> Especially not the traffic terminating locally. Meh.
> 
> The question remains the same, just the use case magically
> mutated under me.
> 
> (Maybe if there were documentation like an intro to qdisc
> writing like I asked for already, I’d have known that.)

These days the recommendation for adding workload specific sauce 
at the qdisc layer is to use the fq qdisc (mq + fq if you have 
a multi queue device) and add a BPF program on top (cls-bpf)
which sets transmission times for each packet.

Obviously you can still write a qdisc if you wish or your needs 
are sufficiently hardcore.

On the docs, nothing official AFAIK, if it doesn't pop up in 
the first two pages of Google results it probably doesn't exist :(

Re: inter-qdisc communication?

[Thorsten Glaser]

On Fri, 26 Aug 2022, Jakub Kicinski wrote:

> These days the recommendation for adding workload specific sauce 
> at the qdisc layer is to use the fq qdisc (mq + fq if you have 
> a multi queue device) and add a BPF program on top (cls-bpf)
> which sets transmission times for each packet.

We started with fq_codel layered on htb (though tbf probably would
have sufficed) for bw limiting, but this turned out to both do too
much and be too hard to extend. I also added deep packet inspection
and relayfs/debugfs reporting so it’s quite a bit…

… plus this also wouldn’t help me on the ingress side.

> Obviously you can still write a qdisc if you wish or your needs 
> are sufficiently hardcore.

After months of working with the above scenario, I finally think
I understand enough to do so, and for our testbed a basic FIFO
setup as backing suffices, though I did a three-FIFO setup similar
to the default qdisc. The added juice is where my headaches lie…

(The customer is not only ok with this being developed as FOSS but
even requested this so I could link if there’s actual interest, or
desire to critique ;)

> On the docs, nothing official AFAIK, if it doesn't pop up in 
> the first two pages of Google results it probably doesn't exist :(

Yeah :/

Worse, many of these things that *do* show up no longer exist.
I find references to documentation that used to be either in the
Linux tree or in manpages all the time :(

At this point I’d be glad for something on the API docs level,
what functions I need to provide, what can I call, what can I
expect, what is expected of me. These things. I figured out the
watchdog thing can be used for packet pacing which I used for
bw limiting in the new qdisc even.

But now… shared state between different LKMs even? Also, I’ve
never programmed in a multithreaded environment before save for
a tiny network demo in Java™… not needed it in my decades (my
main personal experience lays in an old BSD).

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: inter-qdisc communication?

[Thorsten Glaser]

On Fri, 26 Aug 2022, Dave Taht wrote:

> This has some useful info on using netem properly, inc latency
> https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/

It mostly has that netem cannot be used together with other qdiscs,
and my colleague found that out, the hard way, as well. So I need to
implement that part myself as well. I have an idea for egress, but
nothing past that yet.

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: inter-qdisc communication?

[Thorsten Glaser]

(note the communication aspect is still open, see near the end)

On Fri, 26 Aug 2022, Jakub Kicinski wrote:

> How do you add latency on ingress? 🤔

I’ve found something that might do it.

https://serverfault.com/a/386791/189656 and
https://wiki.linuxfoundation.org/networking/netem#how_can_i_use_netem_on_incoming_traffic

So, basically:

$ extif=eth0  # or eth1 or wlan0 or…
$ sudo modprobe ifb  # once or via config
$ sudo ip link set dev ifb0 up
$ sudo tc qdisc add dev $extif handle FFFF: ingress
$ sudo tc filter add dev $extif parent FFFF: [… (see below) …]
$ sudo tc qdisc add dev ifb0 root myqdiscname opts…

All references to ifb seem to cargo-cult the following filter…

	protocol ip u32 match u32 0 0 flowid 1:1
	 action mirred egress redirect dev ifb0

… without explaining any of it. (I’ve concerned myself more with
the implementing of the qdisc than with the configuring, which my
coworkers did before, and I found the info quite… not easily
comprehensible.)

I’ve found that mirred means mirror or redirect, so the action
part’s probably fine. I’m very unsure of the protocol/match
part.

I require any and all traffic of all protocols to be redirected.
Not just IPv4, and not just traffic that matches anything. Can I
do that with the filter, and will this “trick” get me the effect
I want to have?

(I could just use netem but there’s still the issue of inter-
qdisc communication which I’d *very* much like to have, not just
for this but also for features that come later… and, perhaps, one
that’s already there — RAN “handover” emulation, i.e. stopping
all traffic for a few dozen ms or so.)

Thanks in advance,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: inter-qdisc communication?

[Thorsten Glaser]

On Mon, 29 Aug 2022, Thorsten Glaser wrote:

> All references to ifb seem to cargo-cult the following filter…
> 
> 	protocol ip u32 match u32 0 0 flowid 1:1
> 	 action mirred egress redirect dev ifb0

> I require any and all traffic of all protocols to be redirected.
> Not just IPv4, and not just traffic that matches anything. Can I
> do that with the filter, and will this “trick” get me the effect
> I want to have?

https://wiki.gentoo.org/wiki/Traffic_shaping#Traffic_Shaping_Script
has “protocol all” but there is still a filter required? Looking at
net/sched/cls_u32.c this is _quite_ an amount of code involved; is
there really no pass-all? Maybe it’s time to write one…

bye,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: inter-qdisc communication?

[Thorsten Glaser]

On Mon, 29 Aug 2022, Dave Taht wrote:

> had means to do this, so did my debloat script.

OK, this basically does the same but with protocol all.

I don’t need ready-made scripts, I need building blocks
and some amount of understanding.

> > (I could just use netem but there’s still the issue of inter-
> > qdisc communication which I’d *very* much like to have, not just

This is still open… nothing? No inter-module communication possible?

Ideally, I’d configure one of the qdiscs to know the handle and device¹
of another, and then its .change op would call the .change op (or a
special function) from the other, but I lack sufficient Linux kernel
experience (even locking for SMP is something I always fear to miss).

① it really would help if just the handle is enough; they are practically
  global (in my setup) anyway because they determine the debugfs filename

Alternatively, the ability to run multiple tc commands without having to
do multiple fork/exec of the binary would help. Maybe even a “continuous”
mode, in which the binary would REPL commands from stdin, running them as
they’re read…

Thanks in advance,
//mirabilos
-- 
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/⁀\ The UTF-8 Ribbon
╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
 ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
╱ ╲ header encryption!
                        ****************************************************

Re: inter-qdisc communication?

[Dave Taht]

On Mon, Aug 29, 2022 at 9:08 AM Thorsten Glaser <t.glaser@tarent.de> wrote:
>
> On Mon, 29 Aug 2022, Thorsten Glaser wrote:
>
> > All references to ifb seem to cargo-cult the following filter…
> >
> >       protocol ip u32 match u32 0 0 flowid 1:1
> >        action mirred egress redirect dev ifb0
>
> > I require any and all traffic of all protocols to be redirected.
> > Not just IPv4, and not just traffic that matches anything. Can I
> > do that with the filter, and will this “trick” get me the effect
> > I want to have?
>
> https://wiki.gentoo.org/wiki/Traffic_shaping#Traffic_Shaping_Script
> has “protocol all” but there is still a filter required? Looking at
> net/sched/cls_u32.c this is _quite_ an amount of code involved; is
> there really no pass-all? Maybe it’s time to write one…

In general I would really like to simplify the ingress path within the
kernel for common cases. My dream has been, of course, to be able to
do this:

tc qdisc add dev eth0 ingress cake bandwidth 40Mbit.

without needing a filter, mirred, and especially, have it remain multicore.


> bye,
> //mirabilos
> --
> Infrastrukturexperte • tarent solutions GmbH
> Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
> Telephon +49 228 54881-393 • Fax: +49 228 54881-235
> HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
> Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg
>
>                         ****************************************************
> /⁀\ The UTF-8 Ribbon
> ╲ ╱ Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
>  ╳  HTML eMail! Also,     https://www.tarent.de/newsletter
> ╱ ╲ header encryption!
>                         ****************************************************



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC