lantiq_xrx200: Ethernet MAC with multiple TX queues

lantiq_xrx200: Ethernet MAC with multiple TX queues

[Martin Blumenstingl]

Hello,

the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
support for multiple (TX) queues.
This MAC is connected to the SoC's built-in switch IP (called GSWIP).

Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
The vendor driver (which mixes DSA/switch and MAC functionality in one
driver) uses the following approach:
- eth0 ("lan") uses the first TX queue
- eth1 ("wan") uses the second TX queue

With the current (mainline) lantiq_xrx200 driver some users are able
to fill up the first (and only) queue.
This is why I am thinking about adding support for the second queue to
the lantiq_xrx200 driver.

My main question is: how do I do it properly?
Initializing the second TX queue seems simple (calling
netif_tx_napi_add for a second time).
But how do I choose the "right" TX queue in xrx200_start_xmit then?

If my description is too vague then please let me know about any
specific questions you have.
Also if there's an existing driver that "does things right" I am happy
to look at that one.


Thank you and best regards,
Martin

Re: lantiq_xrx200: Ethernet MAC with multiple TX queues

[Vladimir Oltean]

Hi Martin,

On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
> Hello,
> 
> the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
> support for multiple (TX) queues.
> This MAC is connected to the SoC's built-in switch IP (called GSWIP).
> 
> Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
> The vendor driver (which mixes DSA/switch and MAC functionality in one
> driver) uses the following approach:
> - eth0 ("lan") uses the first TX queue
> - eth1 ("wan") uses the second TX queue
> 
> With the current (mainline) lantiq_xrx200 driver some users are able
> to fill up the first (and only) queue.
> This is why I am thinking about adding support for the second queue to
> the lantiq_xrx200 driver.
> 
> My main question is: how do I do it properly?
> Initializing the second TX queue seems simple (calling
> netif_tx_napi_add for a second time).
> But how do I choose the "right" TX queue in xrx200_start_xmit then?
> 
> If my description is too vague then please let me know about any
> specific questions you have.
> Also if there's an existing driver that "does things right" I am happy
> to look at that one.

Is this question specific in any way to DSA?
Many Ethernet drivers are multi-queue. Some map one queue per CPU, some
use mqprio to map one or more queues per priority.

Re: lantiq_xrx200: Ethernet MAC with multiple TX queues

[Florian Fainelli]

On 3/24/2021 1:13 PM, Vladimir Oltean wrote:
> Hi Martin,
> 
> On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
>> Hello,
>>
>> the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
>> support for multiple (TX) queues.
>> This MAC is connected to the SoC's built-in switch IP (called GSWIP).
>>
>> Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
>> The vendor driver (which mixes DSA/switch and MAC functionality in one
>> driver) uses the following approach:
>> - eth0 ("lan") uses the first TX queue
>> - eth1 ("wan") uses the second TX queue
>>
>> With the current (mainline) lantiq_xrx200 driver some users are able
>> to fill up the first (and only) queue.
>> This is why I am thinking about adding support for the second queue to
>> the lantiq_xrx200 driver.
>>
>> My main question is: how do I do it properly?
>> Initializing the second TX queue seems simple (calling
>> netif_tx_napi_add for a second time).
>> But how do I choose the "right" TX queue in xrx200_start_xmit then?

If you use DSA you will have a DSA slave network device which will be
calling into dev_queue_xmit() into the DSA master which will be the
xrx200 driver, so it's fairly simple for you to implement a queue
selection within the xrx200 tagger for instance.

You can take a look at how net/dsa/tag_brcm.c and
drivers/net/ethernet/broadcom/bcmsysport.c work as far as mapping queues
from the DSA slave network device queue/port number into a queue number
for the DSA master.
-- 
Florian

Re: lantiq_xrx200: Ethernet MAC with multiple TX queues

[Vladimir Oltean]

Hi Florian,

On Wed, Mar 24, 2021 at 02:09:02PM -0700, Florian Fainelli wrote:
> 
> 
> On 3/24/2021 1:13 PM, Vladimir Oltean wrote:
> > Hi Martin,
> > 
> > On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
> >> Hello,
> >>
> >> the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
> >> support for multiple (TX) queues.
> >> This MAC is connected to the SoC's built-in switch IP (called GSWIP).
> >>
> >> Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
> >> The vendor driver (which mixes DSA/switch and MAC functionality in one
> >> driver) uses the following approach:
> >> - eth0 ("lan") uses the first TX queue
> >> - eth1 ("wan") uses the second TX queue
> >>
> >> With the current (mainline) lantiq_xrx200 driver some users are able
> >> to fill up the first (and only) queue.
> >> This is why I am thinking about adding support for the second queue to
> >> the lantiq_xrx200 driver.
> >>
> >> My main question is: how do I do it properly?
> >> Initializing the second TX queue seems simple (calling
> >> netif_tx_napi_add for a second time).
> >> But how do I choose the "right" TX queue in xrx200_start_xmit then?
> 
> If you use DSA you will have a DSA slave network device which will be
> calling into dev_queue_xmit() into the DSA master which will be the
> xrx200 driver, so it's fairly simple for you to implement a queue
> selection within the xrx200 tagger for instance.
> 
> You can take a look at how net/dsa/tag_brcm.c and
> drivers/net/ethernet/broadcom/bcmsysport.c work as far as mapping queues
> from the DSA slave network device queue/port number into a queue number
> for the DSA master.

What are the benefits of mapping packets to TX queues of the DSA master
from the DSA layer?

Re: lantiq_xrx200: Ethernet MAC with multiple TX queues

[Florian Fainelli]

On 3/24/2021 3:21 PM, Vladimir Oltean wrote:
> Hi Florian,
> 
> On Wed, Mar 24, 2021 at 02:09:02PM -0700, Florian Fainelli wrote:
>>
>>
>> On 3/24/2021 1:13 PM, Vladimir Oltean wrote:
>>> Hi Martin,
>>>
>>> On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
>>>> Hello,
>>>>
>>>> the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
>>>> support for multiple (TX) queues.
>>>> This MAC is connected to the SoC's built-in switch IP (called GSWIP).
>>>>
>>>> Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
>>>> The vendor driver (which mixes DSA/switch and MAC functionality in one
>>>> driver) uses the following approach:
>>>> - eth0 ("lan") uses the first TX queue
>>>> - eth1 ("wan") uses the second TX queue
>>>>
>>>> With the current (mainline) lantiq_xrx200 driver some users are able
>>>> to fill up the first (and only) queue.
>>>> This is why I am thinking about adding support for the second queue to
>>>> the lantiq_xrx200 driver.
>>>>
>>>> My main question is: how do I do it properly?
>>>> Initializing the second TX queue seems simple (calling
>>>> netif_tx_napi_add for a second time).
>>>> But how do I choose the "right" TX queue in xrx200_start_xmit then?
>>
>> If you use DSA you will have a DSA slave network device which will be
>> calling into dev_queue_xmit() into the DSA master which will be the
>> xrx200 driver, so it's fairly simple for you to implement a queue
>> selection within the xrx200 tagger for instance.
>>
>> You can take a look at how net/dsa/tag_brcm.c and
>> drivers/net/ethernet/broadcom/bcmsysport.c work as far as mapping queues
>> from the DSA slave network device queue/port number into a queue number
>> for the DSA master.
> 
> What are the benefits of mapping packets to TX queues of the DSA master
> from the DSA layer?

For systemport and bcm_sf2 this was explained in this commit:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d156576362c07e954dc36e07b0d7b0733a010f7d

in a nutshell, the switch hardware can return the queue status back to
the systemport's transmit DMA such that it can automatically pace the TX
completion interrupts. To do that we need to establish a mapping between
the DSA slave and master that is comprised of the switch port number and
TX queue number, and tell the HW to inspect the congestion status of
that particular port and queue.

What this is meant to address is a "lossless" (within the SoC at least)
behavior when you have user ports that are connected at a speed lower
than that of your internal connection to the switch typically Gigabit or
more. If you send 1Gbits/sec worth of traffic down to a port that is
connected at 100Mbits/sec there will be roughly 90% packet loss unless
you have a way to pace the Ethernet controller's transmit DMA, which
then ultimately limits the TX completion of the socket buffers so things
work nicely. I believe that per queue flow control was evaluated before
and an out of band mechanism was preferred but I do not remember the
details of that decision to use ACB.
-- 
Florian

Re: lantiq_xrx200: Ethernet MAC with multiple TX queues

[Vladimir Oltean]

On Wed, Mar 24, 2021 at 04:07:47PM -0700, Florian Fainelli wrote:
> > What are the benefits of mapping packets to TX queues of the DSA master
> > from the DSA layer?
> 
> For systemport and bcm_sf2 this was explained in this commit:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d156576362c07e954dc36e07b0d7b0733a010f7d
> 
> in a nutshell, the switch hardware can return the queue status back to
> the systemport's transmit DMA such that it can automatically pace the TX
> completion interrupts. To do that we need to establish a mapping between
> the DSA slave and master that is comprised of the switch port number and
> TX queue number, and tell the HW to inspect the congestion status of
> that particular port and queue.
> 
> What this is meant to address is a "lossless" (within the SoC at least)
> behavior when you have user ports that are connected at a speed lower
> than that of your internal connection to the switch typically Gigabit or
> more. If you send 1Gbits/sec worth of traffic down to a port that is
> connected at 100Mbits/sec there will be roughly 90% packet loss unless
> you have a way to pace the Ethernet controller's transmit DMA, which
> then ultimately limits the TX completion of the socket buffers so things
> work nicely. I believe that per queue flow control was evaluated before
> and an out of band mechanism was preferred but I do not remember the
> details of that decision to use ACB.

Interesting system design.

Just to clarify, this port to queue mapping is completely optional, right?
You can send packets to a certain switch port through any TX queue of
the systemport?

Re: lantiq_xrx200: Ethernet MAC with multiple TX queues

[Florian Fainelli]

On 3/24/2021 6:18 PM, Vladimir Oltean wrote:
> On Wed, Mar 24, 2021 at 04:07:47PM -0700, Florian Fainelli wrote:
>>> What are the benefits of mapping packets to TX queues of the DSA master
>>> from the DSA layer?
>>
>> For systemport and bcm_sf2 this was explained in this commit:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d156576362c07e954dc36e07b0d7b0733a010f7d
>>
>> in a nutshell, the switch hardware can return the queue status back to
>> the systemport's transmit DMA such that it can automatically pace the TX
>> completion interrupts. To do that we need to establish a mapping between
>> the DSA slave and master that is comprised of the switch port number and
>> TX queue number, and tell the HW to inspect the congestion status of
>> that particular port and queue.
>>
>> What this is meant to address is a "lossless" (within the SoC at least)
>> behavior when you have user ports that are connected at a speed lower
>> than that of your internal connection to the switch typically Gigabit or
>> more. If you send 1Gbits/sec worth of traffic down to a port that is
>> connected at 100Mbits/sec there will be roughly 90% packet loss unless
>> you have a way to pace the Ethernet controller's transmit DMA, which
>> then ultimately limits the TX completion of the socket buffers so things
>> work nicely. I believe that per queue flow control was evaluated before
>> and an out of band mechanism was preferred but I do not remember the
>> details of that decision to use ACB.
> 
> Interesting system design.
> 
> Just to clarify, this port to queue mapping is completely optional, right?
> You can send packets to a certain switch port through any TX queue of
> the systemport?

Absolutely, the Broadcom tags allow you to steer traffic towards any TX
queue of any switch port. If ring inspection is disabled then there is
no flow control applied whatsoever.
-- 
Florian

Re: lantiq_xrx200: Ethernet MAC with multiple TX queues

[Martin Blumenstingl]

Hello Florian, Vladimir, Hauke,

first of all: thank you very much for this very informative discussion!

On Thu, Mar 25, 2021 at 4:08 AM Florian Fainelli <f.fainelli@gmail.com> wrote:
[...]
> > Just to clarify, this port to queue mapping is completely optional, right?
> > You can send packets to a certain switch port through any TX queue of
> > the systemport?
>
> Absolutely, the Broadcom tags allow you to steer traffic towards any TX
> queue of any switch port. If ring inspection is disabled then there is
> no flow control applied whatsoever.
I think with the Lantiq PMAC and GSWIP the situation is very similar

skb_set_queue_mapping from tag_brcm.c seems very interesting.
a per-CPU queue is also very interesting since these Lantiq SoCs have
two MIPS cores


Many thanks to all of you again!

Best regards,
Martin

Re: lantiq_xrx200: Ethernet MAC with multiple TX queues

[Hauke Mehrtens]

On 3/24/21 10:09 PM, Florian Fainelli wrote:
> 
> 
> On 3/24/2021 1:13 PM, Vladimir Oltean wrote:
>> Hi Martin,
>>
>> On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
>>> Hello,
>>>
>>> the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
>>> support for multiple (TX) queues.
>>> This MAC is connected to the SoC's built-in switch IP (called GSWIP).
>>>
>>> Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
>>> The vendor driver (which mixes DSA/switch and MAC functionality in one
>>> driver) uses the following approach:
>>> - eth0 ("lan") uses the first TX queue
>>> - eth1 ("wan") uses the second TX queue
>>>
>>> With the current (mainline) lantiq_xrx200 driver some users are able
>>> to fill up the first (and only) queue.
>>> This is why I am thinking about adding support for the second queue to
>>> the lantiq_xrx200 driver.
>>>
>>> My main question is: how do I do it properly?
>>> Initializing the second TX queue seems simple (calling
>>> netif_tx_napi_add for a second time).
>>> But how do I choose the "right" TX queue in xrx200_start_xmit then?
> 
> If you use DSA you will have a DSA slave network device which will be
> calling into dev_queue_xmit() into the DSA master which will be the
> xrx200 driver, so it's fairly simple for you to implement a queue
> selection within the xrx200 tagger for instance.
> 
> You can take a look at how net/dsa/tag_brcm.c and
> drivers/net/ethernet/broadcom/bcmsysport.c work as far as mapping queues
> from the DSA slave network device queue/port number into a queue number
> for the DSA master.
> 

Hi,

The PMAC in the xrx200 has 4 TX queues and 8 RX queues. We can not map 
one queue to each port as there are more ports than queues. I am also 
unsure if the DSL part which is using an out of tree driver uses some of 
these DMA resources.

Is it possible to configure a mapping between a DSA bridge and a queue 
on the mater device with tc from user space? We could expose these 4 TX 
queues on the mac driver to Linux and then Linux configure somehow a 
mapping between ports or bridges and queues.

Hauke

Re: lantiq_xrx200: Ethernet MAC with multiple TX queues

[Florian Fainelli]

On 3/24/2021 3:50 PM, Hauke Mehrtens wrote:
> On 3/24/21 10:09 PM, Florian Fainelli wrote:
>>
>>
>> On 3/24/2021 1:13 PM, Vladimir Oltean wrote:
>>> Hi Martin,
>>>
>>> On Wed, Mar 24, 2021 at 09:04:16PM +0100, Martin Blumenstingl wrote:
>>>> Hello,
>>>>
>>>> the PMAC (Ethernet MAC) IP built into the Lantiq xRX200 SoCs has
>>>> support for multiple (TX) queues.
>>>> This MAC is connected to the SoC's built-in switch IP (called GSWIP).
>>>>
>>>> Right now the lantiq_xrx200 driver only uses one TX and one RX queue.
>>>> The vendor driver (which mixes DSA/switch and MAC functionality in one
>>>> driver) uses the following approach:
>>>> - eth0 ("lan") uses the first TX queue
>>>> - eth1 ("wan") uses the second TX queue
>>>>
>>>> With the current (mainline) lantiq_xrx200 driver some users are able
>>>> to fill up the first (and only) queue.
>>>> This is why I am thinking about adding support for the second queue to
>>>> the lantiq_xrx200 driver.
>>>>
>>>> My main question is: how do I do it properly?
>>>> Initializing the second TX queue seems simple (calling
>>>> netif_tx_napi_add for a second time).
>>>> But how do I choose the "right" TX queue in xrx200_start_xmit then?
>>
>> If you use DSA you will have a DSA slave network device which will be
>> calling into dev_queue_xmit() into the DSA master which will be the
>> xrx200 driver, so it's fairly simple for you to implement a queue
>> selection within the xrx200 tagger for instance.
>>
>> You can take a look at how net/dsa/tag_brcm.c and
>> drivers/net/ethernet/broadcom/bcmsysport.c work as far as mapping queues
>> from the DSA slave network device queue/port number into a queue number
>> for the DSA master.
>>
> 
> Hi,
> 
> The PMAC in the xrx200 has 4 TX queues and 8 RX queues. We can not map
> one queue to each port as there are more ports than queues. I am also
> unsure if the DSL part which is using an out of tree driver uses some of
> these DMA resources.

bcmsysport.c for the "Lite" version has the same kind of limitation, so
what it does is limit the number of DSA slave queues to continue to
support a 1:1 mapping.

> 
> Is it possible to configure a mapping between a DSA bridge and a queue
> on the mater device with tc from user space? We could expose these 4 TX
> queues on the mac driver to Linux and then Linux configure somehow a
> mapping between ports or bridges and queues.

There is not a way to establish a mapping from user-space between the
DSA slave queues and the DSA master queues, I am not too sure what would
be the right way to do that but it would likely require changes to the
'tc' tool and ndo_setup_tc() most likely.
-- 
Florian