Intel-Wired-Lan Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Jesse Brandeburg <jbrandeburg@cloudflare.com>
To: Michal Kubiak <michal.kubiak@intel.com>,
	intel-wired-lan@lists.osuosl.org
Cc: maciej.fijalkowski@intel.com, aleksander.lobakin@intel.com,
	przemyslaw.kitszel@intel.com, dawid.osuchowski@linux.intel.com,
	jacob.e.keller@intel.com, netdev@vger.kernel.org,
	kernel-team@cloudflare.com
Subject: Re: [Intel-wired-lan] [PATCH iwl-net 0/3] Fix XDP loading on machines with many CPUs
Date: Tue, 6 May 2025 22:31:59 -0700	[thread overview]
Message-ID: <b36a7cb6-582b-422d-82ce-98dc8985fd0d@cloudflare.com> (raw)
In-Reply-To: <20250422153659.284868-1-michal.kubiak@intel.com>

On 4/22/25 8:36 AM, Michal Kubiak wrote:
> Hi,
>
> Some of our customers have reported a crash problem when trying to load
> the XDP program on machines with a large number of CPU cores. After
> extensive debugging, it became clear that the root cause of the problem
> lies in the Tx scheduler implementation, which does not seem to be able
> to handle the creation of a large number of Tx queues (even though this
> number does not exceed the number of available queues reported by the
> FW).
> This series addresses this problem.


Hi Michal,

Unfortunately this version of the series seems to reintroduce the 
original problem error: -22.

I double checked the patches, they looked like they were applied in our 
test version 2025.5.8 build which contained a 6.12.26 kernel with this 
series applied (all 3)

Our setup is saying max 252 combined queues, but running 384 CPUs by 
default, loads an XDP program, then reduces the number of queues using 
ethtool, to 192. After that we get the error -22 and link is down.

Sorry to bring some bad news, and I know it took a while, it is a bit of 
a process to test this in our lab.

The original version you had sent us was working fine when we tested it, 
so the problem seems to be between those two versions. I suppose it 
could be possible (but unlikely because I used git to apply the patches) 
that there was something wrong with the source code, but I sincerely 
doubt it as the patches had applied cleanly.

We are only able to test 6.12.y or 6.6.y stable variants of the kernel 
if you want to make a test version of a fixed series for us to try.

Thanks,

Jesse


some dmesg follows:

sudo dmesg | grep -E "ice 0000:c1:00.0|ice:"

[  20.932638] ice: Intel(R) Ethernet Connection E800 Series Linux Driver

[  20.932642] ice: Copyright (c) 2018, Intel Corporation.

[  21.259332] ice 0000:c1:00.0: DDP package does not support Tx 
scheduling layers switching feature - please update to the latest DDP 
package and try again

[  21.552597] ice 0000:c1:00.0: The DDP package was successfully loaded: 
ICE COMMS Package version 1.3.51.0

[  21.610275] ice 0000:c1:00.0: 252.048 Gb/s available PCIe bandwidth 
(16.0 GT/s PCIe x16 link)

[  21.623960] ice 0000:c1:00.0: RDMA is not supported on this device

[  21.672421] ice 0000:c1:00.0: DCB is enabled in the hardware, max 
number of TCs supported on this port are 8

[  21.705729] ice 0000:c1:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.

[  21.722873] ice 0000:c1:00.0: Commit DCB Configuration to the hardware

[  22.086346] ice 0000:c1:00.1: DDP package already present on device: 
ICE COMMS Package version 1.3.51.0

[  22.289956] ice 0000:c1:00.0 ext0: renamed from eth0

[  23.137538] ice 0000:c1:00.0 ext0: NIC Link is up 25 Gbps Full Duplex, 
Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: On, 
Autoneg Negotiated: False, Flow Control: None

*[ 499.643936] ice 0000:c1:00.0: Failed to set LAN Tx queue context, 
error: -22*

*
*


  parent reply	other threads:[~2025-05-07  5:32 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-04-22 15:36 [Intel-wired-lan] [PATCH iwl-net 0/3] Fix XDP loading on machines with many CPUs Michal Kubiak
2025-04-22 15:36 ` [Intel-wired-lan] [PATCH iwl-net 1/3] ice: fix Tx scheduler error handling in XDP callback Michal Kubiak
2025-04-22 17:02   ` Loktionov, Aleksandr
2025-04-22 15:36 ` [Intel-wired-lan] [PATCH iwl-net 2/3] ice: create new Tx scheduler nodes for new queues only Michal Kubiak
2025-04-22 15:36 ` [Intel-wired-lan] [PATCH iwl-net 3/3] ice: fix rebuilding the Tx scheduler tree for large queue counts Michal Kubiak
2025-05-07  5:31 ` Jesse Brandeburg [this message]
2025-05-07  8:00   ` [Intel-wired-lan] [PATCH iwl-net 0/3] Fix XDP loading on machines with many CPUs Michal Kubiak
2025-05-08  5:51     ` Jesse Brandeburg
2025-05-08 14:29       ` Michal Kubiak
2025-05-09 10:07         ` Michal Kubiak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=b36a7cb6-582b-422d-82ce-98dc8985fd0d@cloudflare.com \
    --to=jbrandeburg@cloudflare.com \
    --cc=aleksander.lobakin@intel.com \
    --cc=dawid.osuchowski@linux.intel.com \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=jacob.e.keller@intel.com \
    --cc=kernel-team@cloudflare.com \
    --cc=maciej.fijalkowski@intel.com \
    --cc=michal.kubiak@intel.com \
    --cc=netdev@vger.kernel.org \
    --cc=przemyslaw.kitszel@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox