taprio vs. wireless/mac80211

taprio vs. wireless/mac80211

[Johannes Berg]

Hi,

We're exploring the use of taprio with wireless/mac80211, and in
mac80211 today (**) we don't have multiple queues (any more) since all
the queuing actually happens in FQ/Codel inside mac80211. We also set
IFF_NO_QUEUE, but that of course only really affects the default qdisc,
not the fact that you could use it or not.

In mac80211 thus we never back-pressure against the qdiscs, which is why
we basically selected a single queue. Also, there's nothing else we do
about the queue other than immediately pull a packet from it if
available, so it'd basically pure overhead to have real queues there.

In a sense, given that we cannot back-pressure against the queues, it
feels a bit wrong to even have multiple queues. We also don't benefit in
any way from splitting data structures onto multiple CPUs or something
since we put it all into the same FQ/Codel anyway.


Now, in order to use taprio, you're more or less assuming that you have
multiple (equivalent) queues, as far as I can tell.


Obviously we can trivially expose multiple equivalent queues from
mac80211, but it feels somewhat wrong if that's just to make taprio be
able to do something?

Also how many should we have, there's more code to run so in many cases
you probably don't want more than one, but if you want to use taprio you
need at least two, but who says that's good enough, you might want more
classes:

        /* taprio imposes that traffic classes map 1:n to tx queues */
        if (qopt->num_tc > dev->num_tx_queues) {
                NL_SET_ERR_MSG(extack, "Number of traffic classes is
greater than number of HW queues");
                return -EINVAL;
        }


The way taprio is done almost feels like maybe it shouldn't even care
about the number of queues since taprio_dequeue_soft() anyway only
queries the sub-qdiscs? I mean, it's scheduling traffic, if you over-
subscribe and send more than the link can handle, you've already lost
anyway, no?

(But then Avi pointed out that the sub qdiscs are initialized per HW
queue, so this doesn't really hold either ...)


Anyone have recommendations what we should do?


Thanks,
johannes


(**) Assuming internal TXQ usage, but let's do that, we even have a
first patch somewhere that converts everything to use it; otherwise we
used to have the queues mapped to the ACs with ndo_select_queue, which
is also quite wrong from this perspective.

Re: taprio vs. wireless/mac80211

[Vinicius Costa Gomes]

Hi Johannes,

Johannes Berg <johannes@sipsolutions.net> writes:

> Hi,
>
> We're exploring the use of taprio with wireless/mac80211, and in
> mac80211 today (**) we don't have multiple queues (any more) since all
> the queuing actually happens in FQ/Codel inside mac80211. We also set
> IFF_NO_QUEUE, but that of course only really affects the default qdisc,
> not the fact that you could use it or not.
>
> In mac80211 thus we never back-pressure against the qdiscs, which is why
> we basically selected a single queue. Also, there's nothing else we do
> about the queue other than immediately pull a packet from it if
> available, so it'd basically pure overhead to have real queues there.
>
> In a sense, given that we cannot back-pressure against the queues, it
> feels a bit wrong to even have multiple queues. We also don't benefit in
> any way from splitting data structures onto multiple CPUs or something
> since we put it all into the same FQ/Codel anyway.
>
>
> Now, in order to use taprio, you're more or less assuming that you have
> multiple (equivalent) queues, as far as I can tell.
>
>
> Obviously we can trivially expose multiple equivalent queues from
> mac80211, but it feels somewhat wrong if that's just to make taprio be
> able to do something?
>
> Also how many should we have, there's more code to run so in many cases
> you probably don't want more than one, but if you want to use taprio you
> need at least two, but who says that's good enough, you might want more
> classes:
>
>         /* taprio imposes that traffic classes map 1:n to tx queues */
>         if (qopt->num_tc > dev->num_tx_queues) {
>                 NL_SET_ERR_MSG(extack, "Number of traffic classes is
> greater than number of HW queues");
>                 return -EINVAL;
>         }
>
>
> The way taprio is done almost feels like maybe it shouldn't even care
> about the number of queues since taprio_dequeue_soft() anyway only
> queries the sub-qdiscs? I mean, it's scheduling traffic, if you over-
> subscribe and send more than the link can handle, you've already lost
> anyway, no?
>
> (But then Avi pointed out that the sub qdiscs are initialized per HW
> queue, so this doesn't really hold either ...)
>
>
> Anyone have recommendations what we should do?

I will need to sleep on this, but at first glance, it seems you are
showing a limitation of taprio.

Removing that limitation seems possible, but it would add a bit of
complexity (but not much it seems) to the code, let me write down what I
am thinking:

 0. right now I can trust that there are more queues than traffic
 classes, and using the netdev prio->tc->queue map, I can do the
 scheduling almost entirely on queues. I have to remove this assumption.

 1. with that assumption removed, it means that I can have more traffic
 classes than queues, and so I have to be able to handle multiple
 traffic classes mapped to a single queue, i.e. one child qdisc per TC
 vs. one child per queue that we have today. Enqueueing each packet to
 the right child qdisc is easy. Dequeueing also is also very similar to
 what taprio does right now.

 2. it would be great if I knew the context in which each ->dequeue() is
 called, specifically which queue the ->dequeue() is for, it would
 reduce the number of children that I would have to check.

After writing this, I got the impression that it's feasible. Anyway,
will think a bit more about it.

(2) I don't think is possible right now, but I think we can go on
without it, and leave it as a future optimization.

Does it make sense? Did I understand the problem you are having right?


Cheers,
-- 
Vinicius

Re: taprio vs. wireless/mac80211

[Dave Taht]

On Wed, Aug 24, 2022 at 4:36 PM Vinicius Costa Gomes
<vinicius.gomes@intel.com> wrote:
>
> Hi Johannes,
>
> Johannes Berg <johannes@sipsolutions.net> writes:
>
> > Hi,
> >
> > We're exploring the use of taprio with wireless/mac80211, and in
> > mac80211 today (**) we don't have multiple queues (any more) since all
> > the queuing actually happens in FQ/Codel inside mac80211. We also set
> > IFF_NO_QUEUE, but that of course only really affects the default qdisc,
> > not the fact that you could use it or not.

It would be good if more people understood the packet aggregation
problem deeply,
and lost 8 minutes of their life to this.

https://www.youtube.com/watch?v=Rb-UnHDw02o&t=1745s

Theoretically, in wifi 7, something like single packet taprio-like
scheduling across "DU"s
might become feasible.

> > In mac80211 thus we never back-pressure against the qdiscs, which is why
> > we basically selected a single queue. Also, there's nothing else we do
> > about the queue other than immediately pull a packet from it if
> > available, so it'd basically pure overhead to have real queues there.
> >
> > In a sense, given that we cannot back-pressure against the queues, it
> > feels a bit wrong to even have multiple queues. We also don't benefit in
> > any way from splitting data structures onto multiple CPUs or something
> > since we put it all into the same FQ/Codel anyway.
> >
> >
> > Now, in order to use taprio, you're more or less assuming that you have
> > multiple (equivalent) queues, as far as I can tell.

802.11e's notion of four hardware queues could possibly be utilized
more effectively.

Or you could program those hw queues differently from the default.

> >
> > Obviously we can trivially expose multiple equivalent queues from
> > mac80211, but it feels somewhat wrong if that's just to make taprio be
> > able to do something?
> >
> > Also how many should we have, there's more code to run so in many cases
> > you probably don't want more than one, but if you want to use taprio you
> > need at least two, but who says that's good enough, you might want more
> > classes:
> >
> >         /* taprio imposes that traffic classes map 1:n to tx queues */
> >         if (qopt->num_tc > dev->num_tx_queues) {
> >                 NL_SET_ERR_MSG(extack, "Number of traffic classes is
> > greater than number of HW queues");
> >                 return -EINVAL;
> >         }
> >
> >
> > The way taprio is done almost feels like maybe it shouldn't even care
> > about the number of queues since taprio_dequeue_soft() anyway only
> > queries the sub-qdiscs? I mean, it's scheduling traffic, if you over-
> > subscribe and send more than the link can handle, you've already lost
> > anyway, no?
> >
> > (But then Avi pointed out that the sub qdiscs are initialized per HW
> > queue, so this doesn't really hold either ...)
> >
> >
> > Anyone have recommendations what we should do?
>
> I will need to sleep on this, but at first glance, it seems you are
> showing a limitation of taprio.
>
> Removing that limitation seems possible, but it would add a bit of
> complexity (but not much it seems) to the code, let me write down what I
> am thinking:
>
>  0. right now I can trust that there are more queues than traffic
>  classes, and using the netdev prio->tc->queue map, I can do the
>  scheduling almost entirely on queues. I have to remove this assumption.
>
>  1. with that assumption removed, it means that I can have more traffic
>  classes than queues, and so I have to be able to handle multiple
>  traffic classes mapped to a single queue, i.e. one child qdisc per TC
>  vs. one child per queue that we have today. Enqueueing each packet to
>  the right child qdisc is easy. Dequeueing also is also very similar to
>  what taprio does right now.
>
>  2. it would be great if I knew the context in which each ->dequeue() is
>  called, specifically which queue the ->dequeue() is for, it would
>  reduce the number of children that I would have to check.
>
> After writing this, I got the impression that it's feasible. Anyway,
> will think a bit more about it.
>
> (2) I don't think is possible right now, but I think we can go on
> without it, and leave it as a future optimization.
>
> Does it make sense? Did I understand the problem you are having right?
>
>
> Cheers,
> --
> Vinicius



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: taprio vs. wireless/mac80211

[Jakub Kicinski]

On Wed, 24 Aug 2022 23:50:18 +0200 Johannes Berg wrote:
> Anyone have recommendations what we should do?

Likely lack of sleep or intelligence on my side but I could not grok
from the email what the stacking is, and what the goal is.

Are you putting taprio inside mac80211, or leaving it at the netdev
layer but taking the fq/codel out?

Re: taprio vs. wireless/mac80211

[Vinicius Costa Gomes]

Jakub Kicinski <kuba@kernel.org> writes:

> On Wed, 24 Aug 2022 23:50:18 +0200 Johannes Berg wrote:
>> Anyone have recommendations what we should do?
>
> Likely lack of sleep or intelligence on my side but I could not grok
> from the email what the stacking is, and what the goal is.
>
> Are you putting taprio inside mac80211, or leaving it at the netdev
> layer but taking the fq/codel out?

My read was that they want to do something with taprio with wireless
devices and were hit by the current limitation that taprio only supports
multiqueue interfaces.

The fq/codel part is that, as far as I know, there's already a fq/codel
implementation inside mac80211.

The stacking seems to be that packets would be scheduled by taprio and
then by the scheduler inside mac80211 (fq/codel based?).


Cheers,
-- 
Vinicius

Re: taprio vs. wireless/mac80211

[Jakub Kicinski]

On Fri, 26 Aug 2022 15:10:36 -0700 Vinicius Costa Gomes wrote:
> Jakub Kicinski <kuba@kernel.org> writes:
> 
> > On Wed, 24 Aug 2022 23:50:18 +0200 Johannes Berg wrote:  
> >> Anyone have recommendations what we should do?  
> >
> > Likely lack of sleep or intelligence on my side but I could not grok
> > from the email what the stacking is, and what the goal is.
> >
> > Are you putting taprio inside mac80211, or leaving it at the netdev
> > layer but taking the fq/codel out?  
> 
> My read was that they want to do something with taprio with wireless
> devices and were hit by the current limitation that taprio only supports
> multiqueue interfaces.
> 
> The fq/codel part is that, as far as I know, there's already a fq/codel
> implementation inside mac80211.
> 
> The stacking seems to be that packets would be scheduled by taprio and
> then by the scheduler inside mac80211 (fq/codel based?).

Doesn't adding another layer of non-time-aware queuing after taprio
completely defeat its purpose?  Perhaps I'm revealing my lack of
understanding too much..

Re: taprio vs. wireless/mac80211

[Dave Taht]

On Fri, Aug 26, 2022 at 5:05 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Fri, 26 Aug 2022 15:10:36 -0700 Vinicius Costa Gomes wrote:
> > Jakub Kicinski <kuba@kernel.org> writes:
> >
> > > On Wed, 24 Aug 2022 23:50:18 +0200 Johannes Berg wrote:
> > >> Anyone have recommendations what we should do?
> > >
> > > Likely lack of sleep or intelligence on my side but I could not grok
> > > from the email what the stacking is, and what the goal is.
> > >
> > > Are you putting taprio inside mac80211, or leaving it at the netdev
> > > layer but taking the fq/codel out?
> >
> > My read was that they want to do something with taprio with wireless
> > devices and were hit by the current limitation that taprio only supports
> > multiqueue interfaces.
> >
> > The fq/codel part is that, as far as I know, there's already a fq/codel
> > implementation inside mac80211.
> >
> > The stacking seems to be that packets would be scheduled by taprio and
> > then by the scheduler inside mac80211 (fq/codel based?).
>
> Doesn't adding another layer of non-time-aware queuing after taprio
> completely defeat its purpose?  Perhaps I'm revealing my lack of
> understanding too much..
>

A pre wifi-7 mac itself is incapable of doing time aware queuing. It
has to listen before talk, and
wait til the media is free. Even if wifi-7 bears out, it's still dicy.

So even if you only had one taprio'd packet in the sysem and no other
packets, submitting that packet only starts an election to gain the
media.

-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: taprio vs. wireless/mac80211

[Toke Høiland-Jørgensen]

Johannes Berg <johannes@sipsolutions.net> writes:

> Anyone have recommendations what we should do?

Given that (presumably?) the use case taprio enables needs special
handling all the way down to the hardware to be able to work at all on
WiFi, why not just make the driver (or mac80211) expose a separate (set
of) netdev queue(s) tied to that handling when prompted to do so?

-Toke