Re: [RFC net-next v2 2/2] tg3: Link queues to NAPIs

[Joe Damato <jdamato@fastly.com>]

On Fri, Sep 27, 2024 at 09:33:51AM +0530, Pavan Chebbi wrote:
> On Fri, Sep 27, 2024 at 4:47 AM Joe Damato <jdamato@fastly.com> wrote:
> >
> > On Wed, Sep 25, 2024 at 04:20:48PM +0000, Joe Damato wrote:
> > > Link queues to NAPIs using the netdev-genl API so this information is
> > > queryable.
> > >
> > > $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \
> > >                          --dump queue-get --json='{"ifindex": 2}'
> > >
> > > [{'id': 0, 'ifindex': 2, 'type': 'rx'},
> > >  {'id': 1, 'ifindex': 2, 'napi-id': 146, 'type': 'rx'},
> > >  {'id': 2, 'ifindex': 2, 'napi-id': 147, 'type': 'rx'},
> > >  {'id': 3, 'ifindex': 2, 'napi-id': 148, 'type': 'rx'},
> > >  {'id': 0, 'ifindex': 2, 'napi-id': 145, 'type': 'tx'}]
> > >
> > > Signed-off-by: Joe Damato <jdamato@fastly.com>
> > > ---
> > >  drivers/net/ethernet/broadcom/tg3.c | 24 ++++++++++++++++++++----
> > >  1 file changed, 20 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/broadcom/tg3.c
> > > index ddf0bb65c929..f78d7e8c40b2 100644
> > > --- a/drivers/net/ethernet/broadcom/tg3.c
> > > +++ b/drivers/net/ethernet/broadcom/tg3.c
> > > @@ -7395,18 +7395,34 @@ static int tg3_poll(struct napi_struct *napi, int budget)
> > >
> > >  static void tg3_napi_disable(struct tg3 *tp)
> > >  {
> > > +     struct tg3_napi *tnapi;
> > >       int i;
> > >
> > > -     for (i = tp->irq_cnt - 1; i >= 0; i--)
> > > -             napi_disable(&tp->napi[i].napi);
> > > +     ASSERT_RTNL();
> > > +     for (i = tp->irq_cnt - 1; i >= 0; i--) {
> > > +             tnapi = &tp->napi[i];
> > > +             if (tnapi->tx_buffers)
> > > +                     netif_queue_set_napi(tp->dev, i, NETDEV_QUEUE_TYPE_TX, NULL);
> >
> > It looks like the ASSERT_RTNL is unnecessary; netif_queue_set_napi
> > will call it internally, so I'll remove it before sending this to
> > the list (barring any other feedback).
> 
> Thanks LGTM. You can use Reviewed-by: Pavan Chebbi <pavan.chebbi@broadcom.com>

I noticed there's a misnumbering issue in the code.

Note the output from the first patch:

[{'id': 149, 'ifindex': 2, 'irq': 335},
 {'id': 148, 'ifindex': 2, 'irq': 334},
 {'id': 147, 'ifindex': 2, 'irq': 333},
 {'id': 146, 'ifindex': 2, 'irq': 332},
 {'id': 145, 'ifindex': 2, 'irq': 331}]

Note the output in the commit message above:

 [{'id': 0, 'ifindex': 2, 'type': 'rx'},
  {'id': 1, 'ifindex': 2, 'napi-id': 146, 'type': 'rx'},
  {'id': 2, 'ifindex': 2, 'napi-id': 147, 'type': 'rx'},
  {'id': 3, 'ifindex': 2, 'napi-id': 148, 'type': 'rx'},
  {'id': 0, 'ifindex': 2, 'napi-id': 145, 'type': 'tx'}]

Note that id 0 type: 'rx' has no napi-id associated with it, and in
the second block, NAPI ID 149 is nowhere to be found.

This is happening because the code in the driver does this:

  for (i = 0; i < tp->irq_cnt; i++) {
          tnapi = &tp->napi[i];
          napi_enable(&tnapi->napi);
          if (tnapi->tx_buffers)
                netif_queue_set_napi(tp->dev, i, NETDEV_QUEUE_TYPE_TX,
                                     &tnapi->napi);

The code I added assumed that i is the txq or rxq index, but it's
not - it's the index into the array of struct tg3_napi.

Corrected, the code looks like something like this:

  int txq_idx = 0, rxq_idx = 0;
  [...]

  for (i = 0; i < tp->irq_cnt; i++) {
          tnapi = &tp->napi[i];
          napi_enable(&tnapi->napi);
          if (tnapi->tx_buffers) {
                netif_queue_set_napi(tp->dev, txq_idx, NETDEV_QUEUE_TYPE_TX,
                                     &tnapi->napi);
                txq_idx++ 
          } else if (tnapi->rx_rcb) {
                 netif_queue_set_napi(tp->dev, rxq_idx, NETDEV_QUEUE_TYPE_RX,
                                      &tnapi->napi);
                 rxq_idx++;
          [...]

I tested that and the output looks correct to me. However, what to
do about tg3_napi_disable ?

Probably something like this (txq only for brevity):

  int txq_idx = tp->txq_cnt - 1;
  [...]

  for (i = tp->irq_cnt - 1; i >= 0; i--) {
    [...]
    if (tnapi->tx_buffers) {
        netif_queue_set_napi(tp->dev, txq_idx, NETDEV_QUEUE_TYPE_TX,
                             NULL);
        txq_idx--;
    }
    [...]

Does that seem correct to you? I wanted to ask before sending
another revision, since I am not a tg3 expert.

I will of course remove your Reviewed-by from this patch (but leave
it on patch 1 which is unmodified) when I resend it.