[Intel-wired-lan] BUG(?): igc link up and XDP program init fails

[Intel-wired-lan] BUG(?): igc link up and XDP program init fails

[Ferenc Fejes]

Dear igc Maintainers!

I noticed that ip link set dev up fails with igc (Intel i225) driver
when XDP program is attached to it. More precisely, only when we have
incoming traffic and the incoming packet rate is too fast (like 100
packets per-sec).

I don't have a very smart reproducer, so 4 i225 cards are needed to
trigger it. My setup (enp3s0 and enp4s0 directly connected with a
cable, similarly enp6s0 and enp7s0).

veth0 ----> veth1 --redir---> enp3s0 ~~~~~~~ enp4s0
			  |
                          +-> enp6s0 ~~~~~~~ enp7s0

ip link add dev type veth
ip nei change 1.2.3.4 lladdr aa:aa:aa:aa:aa:aa dev veth0
xdp-bench redirect-multi veth1 enp3s0 enp6s0	#in terminal 1
xdpdump -i enp4s0				#in terminal 2
ping -I veth0 1.2.3.4 -i 0.5 #slow packet rate  #in terminal 3

Now in a separate terminal do a "ip link set dev enp4s0 down" and "ip
link set dev enp4s0 up". After a while, xdpdump will see the incoming
packets.

Now in terminal 3, change the ping to a faster rate:
ping -I veth0 1.2.3.4 -i 0.01

And do the ip link down/up again. In my setup, I no longer see incoming
packets. With bpftrace I see the driver keep trying to initialize
itself in an endless loop.

Now stop the ping, wait about 4-5 seconds, and start the ping again.
This is enough time for the driver to initialize properly, and packets
are visible in xdpdump again.

If anyone has an idea what is wrong with my setup I would be happy to
hear it, and I can help with testing fixes if this is indeed a bug.
I have replicated the setup with veths and it looks fine.

Thanks in advance!

Best,
Ferenc

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

BUG(?): igc link up and XDP program init fails

[Ferenc Fejes]

Dear igc Maintainers!

I noticed that ip link set dev up fails with igc (Intel i225) driver
when XDP program is attached to it. More precisely, only when we have
incoming traffic and the incoming packet rate is too fast (like 100
packets per-sec).

I don't have a very smart reproducer, so 4 i225 cards are needed to
trigger it. My setup (enp3s0 and enp4s0 directly connected with a
cable, similarly enp6s0 and enp7s0).

veth0 ----> veth1 --redir---> enp3s0 ~~~~~~~ enp4s0
			  |
                          +-> enp6s0 ~~~~~~~ enp7s0

ip link add dev type veth
ip nei change 1.2.3.4 lladdr aa:aa:aa:aa:aa:aa dev veth0
xdp-bench redirect-multi veth1 enp3s0 enp6s0	#in terminal 1
xdpdump -i enp4s0				#in terminal 2
ping -I veth0 1.2.3.4 -i 0.5 #slow packet rate  #in terminal 3

Now in a separate terminal do a "ip link set dev enp4s0 down" and "ip
link set dev enp4s0 up". After a while, xdpdump will see the incoming
packets.

Now in terminal 3, change the ping to a faster rate:
ping -I veth0 1.2.3.4 -i 0.01

And do the ip link down/up again. In my setup, I no longer see incoming
packets. With bpftrace I see the driver keep trying to initialize
itself in an endless loop.

Now stop the ping, wait about 4-5 seconds, and start the ping again.
This is enough time for the driver to initialize properly, and packets
are visible in xdpdump again.

If anyone has an idea what is wrong with my setup I would be happy to
hear it, and I can help with testing fixes if this is indeed a bug.
I have replicated the setup with veths and it looks fine.

Thanks in advance!

Best,
Ferenc

Re: [Intel-wired-lan] BUG(?): igc link up and XDP program init fails

[Vinicius Costa Gomes]

Hi Ferenc,

Ferenc Fejes <ferenc.fejes@ericsson.com> writes:

> Dear igc Maintainers!
>
> I noticed that ip link set dev up fails with igc (Intel i225) driver
> when XDP program is attached to it. More precisely, only when we have
> incoming traffic and the incoming packet rate is too fast (like 100
> packets per-sec).
>
> I don't have a very smart reproducer, so 4 i225 cards are needed to
> trigger it. My setup (enp3s0 and enp4s0 directly connected with a
> cable, similarly enp6s0 and enp7s0).
>
> veth0 ----> veth1 --redir---> enp3s0 ~~~~~~~ enp4s0
> 			  |
>                           +-> enp6s0 ~~~~~~~ enp7s0
>
> ip link add dev type veth
> ip nei change 1.2.3.4 lladdr aa:aa:aa:aa:aa:aa dev veth0
> xdp-bench redirect-multi veth1 enp3s0 enp6s0	#in terminal 1
> xdpdump -i enp4s0				#in terminal 2
> ping -I veth0 1.2.3.4 -i 0.5 #slow packet rate  #in terminal 3
>

I was just able to reproduce this issue, with a different setup: 

|             System A                 |   System B   | 
veth0 ----> veth1 --redir---> "enp3s0" ~~~~~~~ "enp4s0"

And running xdp-bench like this:

$ xdp-bench redirect-multi veth1 enp3s0

Also I am running a different traffic generator.

> Now in a separate terminal do a "ip link set dev enp4s0 down" and "ip
> link set dev enp4s0 up". After a while, xdpdump will see the incoming
> packets.
>

It seems that anything that triggers a reset of the adapter would
trigger the bug: I am able to trigger the bug when I run 'xdp-bench'
last (after "ping"/traffic generator), no need for 'link down/link up'.

> Now in terminal 3, change the ping to a faster rate:
> ping -I veth0 1.2.3.4 -i 0.01
>
> And do the ip link down/up again. In my setup, I no longer see incoming
> packets. With bpftrace I see the driver keep trying to initialize
> itself in an endless loop.
>
> Now stop the ping, wait about 4-5 seconds, and start the ping again.
> This is enough time for the driver to initialize properly, and packets
> are visible in xdpdump again.
>
> If anyone has an idea what is wrong with my setup I would be happy to
> hear it, and I can help with testing fixes if this is indeed a bug.
> I have replicated the setup with veths and it looks fine.
>

I don't think there's anything wrong with your setup.

I am considering this a bug, I don't have any patches from the top of my
head for you to try, but taking a look.

Anyway, thanks for the report.


Cheers,
-- 
Vinicius
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: BUG(?): igc link up and XDP program init fails

[Vinicius Costa Gomes]

Hi Ferenc,

Ferenc Fejes <ferenc.fejes@ericsson.com> writes:

> Dear igc Maintainers!
>
> I noticed that ip link set dev up fails with igc (Intel i225) driver
> when XDP program is attached to it. More precisely, only when we have
> incoming traffic and the incoming packet rate is too fast (like 100
> packets per-sec).
>
> I don't have a very smart reproducer, so 4 i225 cards are needed to
> trigger it. My setup (enp3s0 and enp4s0 directly connected with a
> cable, similarly enp6s0 and enp7s0).
>
> veth0 ----> veth1 --redir---> enp3s0 ~~~~~~~ enp4s0
> 			  |
>                           +-> enp6s0 ~~~~~~~ enp7s0
>
> ip link add dev type veth
> ip nei change 1.2.3.4 lladdr aa:aa:aa:aa:aa:aa dev veth0
> xdp-bench redirect-multi veth1 enp3s0 enp6s0	#in terminal 1
> xdpdump -i enp4s0				#in terminal 2
> ping -I veth0 1.2.3.4 -i 0.5 #slow packet rate  #in terminal 3
>

I was just able to reproduce this issue, with a different setup: 

|             System A                 |   System B   | 
veth0 ----> veth1 --redir---> "enp3s0" ~~~~~~~ "enp4s0"

And running xdp-bench like this:

$ xdp-bench redirect-multi veth1 enp3s0

Also I am running a different traffic generator.

> Now in a separate terminal do a "ip link set dev enp4s0 down" and "ip
> link set dev enp4s0 up". After a while, xdpdump will see the incoming
> packets.
>

It seems that anything that triggers a reset of the adapter would
trigger the bug: I am able to trigger the bug when I run 'xdp-bench'
last (after "ping"/traffic generator), no need for 'link down/link up'.

> Now in terminal 3, change the ping to a faster rate:
> ping -I veth0 1.2.3.4 -i 0.01
>
> And do the ip link down/up again. In my setup, I no longer see incoming
> packets. With bpftrace I see the driver keep trying to initialize
> itself in an endless loop.
>
> Now stop the ping, wait about 4-5 seconds, and start the ping again.
> This is enough time for the driver to initialize properly, and packets
> are visible in xdpdump again.
>
> If anyone has an idea what is wrong with my setup I would be happy to
> hear it, and I can help with testing fixes if this is indeed a bug.
> I have replicated the setup with veths and it looks fine.
>

I don't think there's anything wrong with your setup.

I am considering this a bug, I don't have any patches from the top of my
head for you to try, but taking a look.

Anyway, thanks for the report.


Cheers,
-- 
Vinicius

Re: [Intel-wired-lan] BUG(?): igc link up and XDP program init fails

[Vinicius Costa Gomes]

Hi Ferenc,

Vinicius Costa Gomes <vinicius.gomes@intel.com> writes:

> Hi Ferenc,
>
> Ferenc Fejes <ferenc.fejes@ericsson.com> writes:
>

[...]

> I don't think there's anything wrong with your setup.
>
> I am considering this a bug, I don't have any patches from the top of my
> head for you to try, but taking a look.
>

See if the following patch works. Doesn't look too bad, but I have to
think a bit more about it.

--8<---------------cut here---------------start------------->8---
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index e7701866d8b4..d1b3c897c3ac 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -6462,7 +6462,7 @@ static int igc_xdp_xmit(struct net_device *dev, int num_frames,
 	struct igc_ring *ring;
 	int i, drops;
 
-	if (unlikely(test_bit(__IGC_DOWN, &adapter->state)))
+	if (unlikely(!netif_carrier_ok(dev)))
 		return -ENETDOWN;
 
 	if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
--8<---------------cut here---------------end--------------->8---

Cheers,
-- 
Vinicius
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: BUG(?): igc link up and XDP program init fails

[Vinicius Costa Gomes]

Hi Ferenc,

Vinicius Costa Gomes <vinicius.gomes@intel.com> writes:

> Hi Ferenc,
>
> Ferenc Fejes <ferenc.fejes@ericsson.com> writes:
>

[...]

> I don't think there's anything wrong with your setup.
>
> I am considering this a bug, I don't have any patches from the top of my
> head for you to try, but taking a look.
>

See if the following patch works. Doesn't look too bad, but I have to
think a bit more about it.

--8<---------------cut here---------------start------------->8---
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index e7701866d8b4..d1b3c897c3ac 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -6462,7 +6462,7 @@ static int igc_xdp_xmit(struct net_device *dev, int num_frames,
 	struct igc_ring *ring;
 	int i, drops;
 
-	if (unlikely(test_bit(__IGC_DOWN, &adapter->state)))
+	if (unlikely(!netif_carrier_ok(dev)))
 		return -ENETDOWN;
 
 	if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
--8<---------------cut here---------------end--------------->8---

Cheers,
-- 
Vinicius

Re: [Intel-wired-lan] BUG(?): igc link up and XDP program init fails

[Ferenc Fejes]

[-- Attachment #1.1: Type: text/plain, Size: 1899 bytes --]

Hi Vinicius!

Thank you for the prompt reply! I'm OoO right now, but I'll try it in a few days when I'm back.

Thanks,
Ferenc

________________________________
Feladó: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Elküldve: 2023. augusztus 26., szombat 1:51
Címzett: Ferenc Fejes <ferenc.fejes@ericsson.com>; jesse.brandeburg@intel.com <jesse.brandeburg@intel.com>; netdev@vger.kernel.org <netdev@vger.kernel.org>; sasha.neftin@intel.com <sasha.neftin@intel.com>; intel-wired-lan@lists.osuosl.org <intel-wired-lan@lists.osuosl.org>; anthony.l.nguyen@intel.com <anthony.l.nguyen@intel.com>
Másolatot kap: hawk@kernel.org <hawk@kernel.org>
Tárgy: Re: BUG(?): igc link up and XDP program init fails

Hi Ferenc,

Vinicius Costa Gomes <vinicius.gomes@intel.com> writes:

> Hi Ferenc,
>
> Ferenc Fejes <ferenc.fejes@ericsson.com> writes:
>

[...]

> I don't think there's anything wrong with your setup.
>
> I am considering this a bug, I don't have any patches from the top of my
> head for you to try, but taking a look.
>

See if the following patch works. Doesn't look too bad, but I have to
think a bit more about it.

--8<---------------cut here---------------start------------->8---
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index e7701866d8b4..d1b3c897c3ac 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -6462,7 +6462,7 @@ static int igc_xdp_xmit(struct net_device *dev, int num_frames,
         struct igc_ring *ring;
         int i, drops;

-       if (unlikely(test_bit(__IGC_DOWN, &adapter->state)))
+       if (unlikely(!netif_carrier_ok(dev)))
                 return -ENETDOWN;

         if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
--8<---------------cut here---------------end--------------->8---

Cheers,
--
Vinicius

[-- Attachment #1.2: Type: text/html, Size: 2999 bytes --]

[-- Attachment #2: Type: text/plain, Size: 162 bytes --]

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] BUG(?): igc link up and XDP program init fails

[Ferenc Fejes]

Hi Vinicius!

Sorry for the delay.

On Fri, 2023-08-25 at 15:56 -0700, Vinicius Costa Gomes wrote:
> Hi Ferenc,
> 
> Ferenc Fejes <ferenc.fejes@ericsson.com> writes:
> 
> > Dear igc Maintainers!
> > 
> > I noticed that ip link set dev up fails with igc (Intel i225)
> > driver
> > when XDP program is attached to it. More precisely, only when we
> > have
> > incoming traffic and the incoming packet rate is too fast (like 100
> > packets per-sec).
> > 
> > I don't have a very smart reproducer, so 4 i225 cards are needed to
> > trigger it. My setup (enp3s0 and enp4s0 directly connected with a
> > cable, similarly enp6s0 and enp7s0).
> > 
> > veth0 ----> veth1 --redir---> enp3s0 ~~~~~~~ enp4s0
> > 			  |
> >                           +-> enp6s0 ~~~~~~~ enp7s0
> > 
> > ip link add dev type veth
> > ip nei change 1.2.3.4 lladdr aa:aa:aa:aa:aa:aa dev veth0
> > xdp-bench redirect-multi veth1 enp3s0 enp6s0	#in terminal 1
> > xdpdump -i enp4s0				#in terminal 2
> > ping -I veth0 1.2.3.4 -i 0.5 #slow packet rate  #in terminal 3
> > 
> 
> I was just able to reproduce this issue, with a different setup: 
> 
> >             System A                 |   System B   | 
> veth0 ----> veth1 --redir---> "enp3s0" ~~~~~~~ "enp4s0"
> 
> And running xdp-bench like this:
> 
> $ xdp-bench redirect-multi veth1 enp3s0
> 
> Also I am running a different traffic generator.

Thank you for the simplistic reproducer and the patch with the fix!

I confirm your patch posted in this thread solve the issue.

I tested with your reproducer, my reproducer and my original prototype
code which triggered the bug first time - all of them works. I tested
with 100k pps and still OK.


> 
> > Now in a separate terminal do a "ip link set dev enp4s0 down" and
> > "ip
> > link set dev enp4s0 up". After a while, xdpdump will see the
> > incoming
> > packets.
> > 
> 
> It seems that anything that triggers a reset of the adapter would
> trigger the bug: I am able to trigger the bug when I run 'xdp-bench'
> last (after "ping"/traffic generator), no need for 'link down/link
> up'.
> 
> > Now in terminal 3, change the ping to a faster rate:
> > ping -I veth0 1.2.3.4 -i 0.01
> > 
> > And do the ip link down/up again. In my setup, I no longer see
> > incoming
> > packets. With bpftrace I see the driver keep trying to initialize
> > itself in an endless loop.
> > 
> > Now stop the ping, wait about 4-5 seconds, and start the ping
> > again.
> > This is enough time for the driver to initialize properly, and
> > packets
> > are visible in xdpdump again.
> > 
> > If anyone has an idea what is wrong with my setup I would be happy
> > to
> > hear it, and I can help with testing fixes if this is indeed a bug.
> > I have replicated the setup with veths and it looks fine.
> > 
> 
> I don't think there's anything wrong with your setup.
> 
> I am considering this a bug, I don't have any patches from the top of
> my
> head for you to try, but taking a look.
> 
> Anyway, thanks for the report.

Feel free apply Tested-by: Ferenc Fejes <ferenc.fejes@ericsson.com>
in case of submitting it to net.

> 
> 
> Cheers,

Best,
Ferenc

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: BUG(?): igc link up and XDP program init fails

[Ferenc Fejes]

Hi Vinicius!

Sorry for the delay.

On Fri, 2023-08-25 at 15:56 -0700, Vinicius Costa Gomes wrote:
> Hi Ferenc,
> 
> Ferenc Fejes <ferenc.fejes@ericsson.com> writes:
> 
> > Dear igc Maintainers!
> > 
> > I noticed that ip link set dev up fails with igc (Intel i225)
> > driver
> > when XDP program is attached to it. More precisely, only when we
> > have
> > incoming traffic and the incoming packet rate is too fast (like 100
> > packets per-sec).
> > 
> > I don't have a very smart reproducer, so 4 i225 cards are needed to
> > trigger it. My setup (enp3s0 and enp4s0 directly connected with a
> > cable, similarly enp6s0 and enp7s0).
> > 
> > veth0 ----> veth1 --redir---> enp3s0 ~~~~~~~ enp4s0
> > 			  |
> >                           +-> enp6s0 ~~~~~~~ enp7s0
> > 
> > ip link add dev type veth
> > ip nei change 1.2.3.4 lladdr aa:aa:aa:aa:aa:aa dev veth0
> > xdp-bench redirect-multi veth1 enp3s0 enp6s0	#in terminal 1
> > xdpdump -i enp4s0				#in terminal 2
> > ping -I veth0 1.2.3.4 -i 0.5 #slow packet rate  #in terminal 3
> > 
> 
> I was just able to reproduce this issue, with a different setup: 
> 
> >             System A                 |   System B   | 
> veth0 ----> veth1 --redir---> "enp3s0" ~~~~~~~ "enp4s0"
> 
> And running xdp-bench like this:
> 
> $ xdp-bench redirect-multi veth1 enp3s0
> 
> Also I am running a different traffic generator.

Thank you for the simplistic reproducer and the patch with the fix!

I confirm your patch posted in this thread solve the issue.

I tested with your reproducer, my reproducer and my original prototype
code which triggered the bug first time - all of them works. I tested
with 100k pps and still OK.


> 
> > Now in a separate terminal do a "ip link set dev enp4s0 down" and
> > "ip
> > link set dev enp4s0 up". After a while, xdpdump will see the
> > incoming
> > packets.
> > 
> 
> It seems that anything that triggers a reset of the adapter would
> trigger the bug: I am able to trigger the bug when I run 'xdp-bench'
> last (after "ping"/traffic generator), no need for 'link down/link
> up'.
> 
> > Now in terminal 3, change the ping to a faster rate:
> > ping -I veth0 1.2.3.4 -i 0.01
> > 
> > And do the ip link down/up again. In my setup, I no longer see
> > incoming
> > packets. With bpftrace I see the driver keep trying to initialize
> > itself in an endless loop.
> > 
> > Now stop the ping, wait about 4-5 seconds, and start the ping
> > again.
> > This is enough time for the driver to initialize properly, and
> > packets
> > are visible in xdpdump again.
> > 
> > If anyone has an idea what is wrong with my setup I would be happy
> > to
> > hear it, and I can help with testing fixes if this is indeed a bug.
> > I have replicated the setup with veths and it looks fine.
> > 
> 
> I don't think there's anything wrong with your setup.
> 
> I am considering this a bug, I don't have any patches from the top of
> my
> head for you to try, but taking a look.
> 
> Anyway, thanks for the report.

Feel free apply Tested-by: Ferenc Fejes <ferenc.fejes@ericsson.com>
in case of submitting it to net.

> 
> 
> Cheers,

Best,
Ferenc