From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Edward Cree <ecree.xilinx@gmail.com>
Cc: brouer@redhat.com, Ivan Babrou <ivan@cloudflare.com>,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
bpf@vger.kernel.org, kernel-team@cloudflare.com,
Martin Habets <habetsm.xilinx@gmail.com>,
"David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Jesper Dangaard Brouer <hawk@kernel.org>,
John Fastabend <john.fastabend@gmail.com>
Subject: Re: [PATCH net-next] sfc: reduce the number of requested xdp ev queues
Date: Wed, 16 Dec 2020 09:45:24 +0100 [thread overview]
Message-ID: <20201216094524.0c6e521c@carbon> (raw)
In-Reply-To: <205ba636-f180-3003-a41c-828e1fe1a13b@gmail.com>
On Tue, 15 Dec 2020 18:49:55 +0000
Edward Cree <ecree.xilinx@gmail.com> wrote:
> On 15/12/2020 09:43, Jesper Dangaard Brouer wrote:
> > On Mon, 14 Dec 2020 17:29:06 -0800
> > Ivan Babrou <ivan@cloudflare.com> wrote:
> >
> >> Without this change the driver tries to allocate too many queues,
> >> breaching the number of available msi-x interrupts on machines
> >> with many logical cpus and default adapter settings:
> >>
> >> Insufficient resources for 12 XDP event queues (24 other channels, max 32)
> >>
> >> Which in turn triggers EINVAL on XDP processing:
> >>
> >> sfc 0000:86:00.0 ext0: XDP TX failed (-22)
> >
> > I have a similar QA report with XDP_REDIRECT:
> > sfc 0000:05:00.0 ens1f0np0: XDP redirect failed (-22)
> >
> > Here we are back to the issue we discussed with ixgbe, that NIC / msi-x
> > interrupts hardware resources are not enough on machines with many
> > logical cpus.
> >
> > After this fix, what will happen if (cpu >= efx->xdp_tx_queue_count) ?
>
> Same as happened before: the "failed -22". But this fix will make that
> less likely to happen, because it ties more TXQs to each EVQ, and it's
> the EVQs that are in short supply.
>
So, what I hear is that this fix is just pampering over the real issue.
I suggest that you/we detect the situation, and have a code path that
will take a lock (per 16 packets bulk) and solve the issue.
If you care about maximum performance you can implement this via
changing the ndo_xdp_xmit pointer to the fallback function when needed,
to avoid having a to check for the fallback mode in the fast-path.
>
> (Strictly speaking, I believe the limitation is a software one, that
> comes from the driver's channel structures having been designed a
> decade ago when 32 cpus ought to be enough for anybody... AFAIR the
> hardware is capable of giving us something like 1024 evqs if we ask
> for them, it just might not have that many msi-x vectors for us.)
> Anyway, the patch looks correct, so
> Acked-by: Edward Cree <ecree.xilinx@gmail.com>
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
next prev parent reply other threads:[~2020-12-16 8:47 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-15 1:29 [PATCH net-next] sfc: reduce the number of requested xdp ev queues Ivan Babrou
2020-12-15 9:43 ` Jesper Dangaard Brouer
2020-12-15 18:49 ` Edward Cree
2020-12-16 8:45 ` Jesper Dangaard Brouer [this message]
2020-12-16 23:12 ` Edward Cree
2020-12-16 8:18 ` Martin Habets
2020-12-17 18:14 ` Jakub Kicinski
2021-01-19 23:43 ` Ivan Babrou
2021-01-20 0:31 ` Jakub Kicinski
-- strict thread matches above, loose matches on Subject: below --
2021-01-20 21:27 Ivan Babrou
2021-01-21 16:10 ` Edward Cree
2021-01-21 17:11 ` Jesper Dangaard Brouer
2021-01-21 17:14 ` Maciej Fijalkowski
2021-01-23 3:30 ` patchwork-bot+netdevbpf
2021-02-06 10:43 ` Martin Habets
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201216094524.0c6e521c@carbon \
--to=brouer@redhat.com \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=ecree.xilinx@gmail.com \
--cc=habetsm.xilinx@gmail.com \
--cc=hawk@kernel.org \
--cc=ivan@cloudflare.com \
--cc=john.fastabend@gmail.com \
--cc=kernel-team@cloudflare.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.