All of lore.kernel.org
 help / color / mirror / Atom feed
From: Vinicius Costa Gomes <vinicius.gomes@intel.com>
To: Ferenc Fejes <ferenc.fejes@ericsson.com>,
	"jesse.brandeburg@intel.com" <jesse.brandeburg@intel.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"sasha.neftin@intel.com" <sasha.neftin@intel.com>,
	"intel-wired-lan@lists.osuosl.org"
	<intel-wired-lan@lists.osuosl.org>,
	"anthony.l.nguyen@intel.com" <anthony.l.nguyen@intel.com>
Cc: "hawk@kernel.org" <hawk@kernel.org>
Subject: Re: [Intel-wired-lan] BUG(?): igc link up and XDP program init fails
Date: Fri, 25 Aug 2023 15:56:01 -0700	[thread overview]
Message-ID: <87ttsmohoe.fsf@intel.com> (raw)
In-Reply-To: <0caf33cf6adb3a5bf137eeaa20e89b167c9986d5.camel@ericsson.com>

Hi Ferenc,

Ferenc Fejes <ferenc.fejes@ericsson.com> writes:

> Dear igc Maintainers!
>
> I noticed that ip link set dev up fails with igc (Intel i225) driver
> when XDP program is attached to it. More precisely, only when we have
> incoming traffic and the incoming packet rate is too fast (like 100
> packets per-sec).
>
> I don't have a very smart reproducer, so 4 i225 cards are needed to
> trigger it. My setup (enp3s0 and enp4s0 directly connected with a
> cable, similarly enp6s0 and enp7s0).
>
> veth0 ----> veth1 --redir---> enp3s0 ~~~~~~~ enp4s0
> 			  |
>                           +-> enp6s0 ~~~~~~~ enp7s0
>
> ip link add dev type veth
> ip nei change 1.2.3.4 lladdr aa:aa:aa:aa:aa:aa dev veth0
> xdp-bench redirect-multi veth1 enp3s0 enp6s0	#in terminal 1
> xdpdump -i enp4s0				#in terminal 2
> ping -I veth0 1.2.3.4 -i 0.5 #slow packet rate  #in terminal 3
>

I was just able to reproduce this issue, with a different setup: 

|             System A                 |   System B   | 
veth0 ----> veth1 --redir---> "enp3s0" ~~~~~~~ "enp4s0"

And running xdp-bench like this:

$ xdp-bench redirect-multi veth1 enp3s0

Also I am running a different traffic generator.

> Now in a separate terminal do a "ip link set dev enp4s0 down" and "ip
> link set dev enp4s0 up". After a while, xdpdump will see the incoming
> packets.
>

It seems that anything that triggers a reset of the adapter would
trigger the bug: I am able to trigger the bug when I run 'xdp-bench'
last (after "ping"/traffic generator), no need for 'link down/link up'.

> Now in terminal 3, change the ping to a faster rate:
> ping -I veth0 1.2.3.4 -i 0.01
>
> And do the ip link down/up again. In my setup, I no longer see incoming
> packets. With bpftrace I see the driver keep trying to initialize
> itself in an endless loop.
>
> Now stop the ping, wait about 4-5 seconds, and start the ping again.
> This is enough time for the driver to initialize properly, and packets
> are visible in xdpdump again.
>
> If anyone has an idea what is wrong with my setup I would be happy to
> hear it, and I can help with testing fixes if this is indeed a bug.
> I have replicated the setup with veths and it looks fine.
>

I don't think there's anything wrong with your setup.

I am considering this a bug, I don't have any patches from the top of my
head for you to try, but taking a look.

Anyway, thanks for the report.


Cheers,
-- 
Vinicius
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

WARNING: multiple messages have this Message-ID (diff)
From: Vinicius Costa Gomes <vinicius.gomes@intel.com>
To: Ferenc Fejes <ferenc.fejes@ericsson.com>,
	"jesse.brandeburg@intel.com" <jesse.brandeburg@intel.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"sasha.neftin@intel.com" <sasha.neftin@intel.com>,
	"intel-wired-lan@lists.osuosl.org"
	<intel-wired-lan@lists.osuosl.org>,
	"anthony.l.nguyen@intel.com" <anthony.l.nguyen@intel.com>
Cc: "hawk@kernel.org" <hawk@kernel.org>
Subject: Re: BUG(?): igc link up and XDP program init fails
Date: Fri, 25 Aug 2023 15:56:01 -0700	[thread overview]
Message-ID: <87ttsmohoe.fsf@intel.com> (raw)
In-Reply-To: <0caf33cf6adb3a5bf137eeaa20e89b167c9986d5.camel@ericsson.com>

Hi Ferenc,

Ferenc Fejes <ferenc.fejes@ericsson.com> writes:

> Dear igc Maintainers!
>
> I noticed that ip link set dev up fails with igc (Intel i225) driver
> when XDP program is attached to it. More precisely, only when we have
> incoming traffic and the incoming packet rate is too fast (like 100
> packets per-sec).
>
> I don't have a very smart reproducer, so 4 i225 cards are needed to
> trigger it. My setup (enp3s0 and enp4s0 directly connected with a
> cable, similarly enp6s0 and enp7s0).
>
> veth0 ----> veth1 --redir---> enp3s0 ~~~~~~~ enp4s0
> 			  |
>                           +-> enp6s0 ~~~~~~~ enp7s0
>
> ip link add dev type veth
> ip nei change 1.2.3.4 lladdr aa:aa:aa:aa:aa:aa dev veth0
> xdp-bench redirect-multi veth1 enp3s0 enp6s0	#in terminal 1
> xdpdump -i enp4s0				#in terminal 2
> ping -I veth0 1.2.3.4 -i 0.5 #slow packet rate  #in terminal 3
>

I was just able to reproduce this issue, with a different setup: 

|             System A                 |   System B   | 
veth0 ----> veth1 --redir---> "enp3s0" ~~~~~~~ "enp4s0"

And running xdp-bench like this:

$ xdp-bench redirect-multi veth1 enp3s0

Also I am running a different traffic generator.

> Now in a separate terminal do a "ip link set dev enp4s0 down" and "ip
> link set dev enp4s0 up". After a while, xdpdump will see the incoming
> packets.
>

It seems that anything that triggers a reset of the adapter would
trigger the bug: I am able to trigger the bug when I run 'xdp-bench'
last (after "ping"/traffic generator), no need for 'link down/link up'.

> Now in terminal 3, change the ping to a faster rate:
> ping -I veth0 1.2.3.4 -i 0.01
>
> And do the ip link down/up again. In my setup, I no longer see incoming
> packets. With bpftrace I see the driver keep trying to initialize
> itself in an endless loop.
>
> Now stop the ping, wait about 4-5 seconds, and start the ping again.
> This is enough time for the driver to initialize properly, and packets
> are visible in xdpdump again.
>
> If anyone has an idea what is wrong with my setup I would be happy to
> hear it, and I can help with testing fixes if this is indeed a bug.
> I have replicated the setup with veths and it looks fine.
>

I don't think there's anything wrong with your setup.

I am considering this a bug, I don't have any patches from the top of my
head for you to try, but taking a look.

Anyway, thanks for the report.


Cheers,
-- 
Vinicius

  reply	other threads:[~2023-08-25 22:56 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-08-24 13:43 [Intel-wired-lan] BUG(?): igc link up and XDP program init fails Ferenc Fejes
2023-08-24 13:43 ` Ferenc Fejes
2023-08-25 22:56 ` Vinicius Costa Gomes [this message]
2023-08-25 22:56   ` Vinicius Costa Gomes
2023-08-25 23:51   ` [Intel-wired-lan] " Vinicius Costa Gomes
2023-08-25 23:51     ` Vinicius Costa Gomes
2023-08-26 10:35     ` [Intel-wired-lan] " Ferenc Fejes
2023-09-04 12:29   ` Ferenc Fejes
2023-09-04 12:29     ` Ferenc Fejes

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87ttsmohoe.fsf@intel.com \
    --to=vinicius.gomes@intel.com \
    --cc=anthony.l.nguyen@intel.com \
    --cc=ferenc.fejes@ericsson.com \
    --cc=hawk@kernel.org \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=jesse.brandeburg@intel.com \
    --cc=netdev@vger.kernel.org \
    --cc=sasha.neftin@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.