[PATCH net v3] selftests: net: local_termination: annotate the expected failures

[PATCH net v3] selftests: net: local_termination: annotate the expected failures

[Jakub Kicinski]

Vladimir said when adding this test:

  The bridge driver fares particularly badly [...] mainly because
  it does not implement IFF_UNICAST_FLT.

See commit 90b9566aa5cd ("selftests: forwarding: add a test for
local_termination.sh").

We don't want to hide the known gaps, but having a test which
always fails prevents us from catching regressions. Report
the cases we know may fail as XFAIL.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
CC: liuhangbin@gmail.com
CC: shuah@kernel.org
CC: linux-kselftest@vger.kernel.org
CC: Petr Machata <petrm@nvidia.com>
CC: vladimir.oltean@nxp.com

v3:
 - use xfail_on_veth correctly as a "prefix" call
 - dropping Vladimir's tags since the code is quite different now
v2: https://lore.kernel.org/r/20240509235553.5740-1-kuba@kernel.org/
 - remove duplicated log_test_xfail
v1: https://lore.kernel.org/all/20240509235553.5740-1-kuba@kernel.org/
---
 .../net/forwarding/local_termination.sh       | 30 +++++++++++--------
 1 file changed, 18 insertions(+), 12 deletions(-)

diff --git a/tools/testing/selftests/net/forwarding/local_termination.sh b/tools/testing/selftests/net/forwarding/local_termination.sh
index c5b0cbc85b3e..4b364cdf3ef0 100755
--- a/tools/testing/selftests/net/forwarding/local_termination.sh
+++ b/tools/testing/selftests/net/forwarding/local_termination.sh
@@ -155,25 +155,30 @@ run_test()
 		"$smac > $MACVLAN_ADDR, ethertype IPv4 (0x0800)" \
 		true
 
-	check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address" \
-		"$smac > $UNKNOWN_UC_ADDR1, ethertype IPv4 (0x0800)" \
-		false
+	xfail_on_veth $h1 \
+		check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address" \
+			"$smac > $UNKNOWN_UC_ADDR1, ethertype IPv4 (0x0800)" \
+			false
 
 	check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address, promisc" \
 		"$smac > $UNKNOWN_UC_ADDR2, ethertype IPv4 (0x0800)" \
 		true
 
-	check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address, allmulti" \
-		"$smac > $UNKNOWN_UC_ADDR3, ethertype IPv4 (0x0800)" \
-		false
+	xfail_on_veth $h1 \
+		check_rcv $rcv_if_name \
+			"Unicast IPv4 to unknown MAC address, allmulti" \
+			"$smac > $UNKNOWN_UC_ADDR3, ethertype IPv4 (0x0800)" \
+			false
 
 	check_rcv $rcv_if_name "Multicast IPv4 to joined group" \
 		"$smac > $JOINED_MACV4_MC_ADDR, ethertype IPv4 (0x0800)" \
 		true
 
-	check_rcv $rcv_if_name "Multicast IPv4 to unknown group" \
-		"$smac > $UNKNOWN_MACV4_MC_ADDR1, ethertype IPv4 (0x0800)" \
-		false
+	xfail_on_veth $h1 \
+		check_rcv $rcv_if_name \
+			"Multicast IPv4 to unknown group" \
+			"$smac > $UNKNOWN_MACV4_MC_ADDR1, ethertype IPv4 (0x0800)" \
+			false
 
 	check_rcv $rcv_if_name "Multicast IPv4 to unknown group, promisc" \
 		"$smac > $UNKNOWN_MACV4_MC_ADDR2, ethertype IPv4 (0x0800)" \
@@ -187,9 +192,10 @@ run_test()
 		"$smac > $JOINED_MACV6_MC_ADDR, ethertype IPv6 (0x86dd)" \
 		true
 
-	check_rcv $rcv_if_name "Multicast IPv6 to unknown group" \
-		"$smac > $UNKNOWN_MACV6_MC_ADDR1, ethertype IPv6 (0x86dd)" \
-		false
+	xfail_on_veth $h1 \
+		check_rcv $rcv_if_name "Multicast IPv6 to unknown group" \
+			"$smac > $UNKNOWN_MACV6_MC_ADDR1, ethertype IPv6 (0x86dd)" \
+			false
 
 	check_rcv $rcv_if_name "Multicast IPv6 to unknown group, promisc" \
 		"$smac > $UNKNOWN_MACV6_MC_ADDR2, ethertype IPv6 (0x86dd)" \
-- 
2.45.0

Re: [PATCH net v3] selftests: net: local_termination: annotate the expected failures

[Simon Horman]

On Thu, May 16, 2024 at 08:25:13AM -0700, Jakub Kicinski wrote:
> Vladimir said when adding this test:
> 
>   The bridge driver fares particularly badly [...] mainly because
>   it does not implement IFF_UNICAST_FLT.
> 
> See commit 90b9566aa5cd ("selftests: forwarding: add a test for
> local_termination.sh").
> 
> We don't want to hide the known gaps, but having a test which
> always fails prevents us from catching regressions. Report
> the cases we know may fail as XFAIL.
> 
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>

I agree that XFAIL is appropriate for cases like this - the test
is expected to fail and this can act like a TODO.

I also looked over how xfail_on_veth works and this usage seems
correct to me.

Reviewed-by: Simon Horman <horms@kernel.org>

Re: [PATCH net v3] selftests: net: local_termination: annotate the expected failures

[Hangbin Liu]

On Thu, May 16, 2024 at 08:25:13AM -0700, Jakub Kicinski wrote:
> Vladimir said when adding this test:
> 
>   The bridge driver fares particularly badly [...] mainly because
>   it does not implement IFF_UNICAST_FLT.
> 
> See commit 90b9566aa5cd ("selftests: forwarding: add a test for
> local_termination.sh").
> 
> We don't want to hide the known gaps, but having a test which
> always fails prevents us from catching regressions. Report
> the cases we know may fail as XFAIL.
> 
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
> CC: liuhangbin@gmail.com
> CC: shuah@kernel.org
> CC: linux-kselftest@vger.kernel.org
> CC: Petr Machata <petrm@nvidia.com>
> CC: vladimir.oltean@nxp.com
> 
> v3:
>  - use xfail_on_veth correctly as a "prefix" call
>  - dropping Vladimir's tags since the code is quite different now
> v2: https://lore.kernel.org/r/20240509235553.5740-1-kuba@kernel.org/
>  - remove duplicated log_test_xfail
> v1: https://lore.kernel.org/all/20240509235553.5740-1-kuba@kernel.org/
> ---
>  .../net/forwarding/local_termination.sh       | 30 +++++++++++--------
>  1 file changed, 18 insertions(+), 12 deletions(-)
> 
> diff --git a/tools/testing/selftests/net/forwarding/local_termination.sh b/tools/testing/selftests/net/forwarding/local_termination.sh
> index c5b0cbc85b3e..4b364cdf3ef0 100755
> --- a/tools/testing/selftests/net/forwarding/local_termination.sh
> +++ b/tools/testing/selftests/net/forwarding/local_termination.sh
> @@ -155,25 +155,30 @@ run_test()
>  		"$smac > $MACVLAN_ADDR, ethertype IPv4 (0x0800)" \
>  		true
>  
> -	check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address" \
> -		"$smac > $UNKNOWN_UC_ADDR1, ethertype IPv4 (0x0800)" \
> -		false
> +	xfail_on_veth $h1 \
> +		check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address" \
> +			"$smac > $UNKNOWN_UC_ADDR1, ethertype IPv4 (0x0800)" \
> +			false
>  
>  	check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address, promisc" \
>  		"$smac > $UNKNOWN_UC_ADDR2, ethertype IPv4 (0x0800)" \
>  		true
>  
> -	check_rcv $rcv_if_name "Unicast IPv4 to unknown MAC address, allmulti" \
> -		"$smac > $UNKNOWN_UC_ADDR3, ethertype IPv4 (0x0800)" \
> -		false
> +	xfail_on_veth $h1 \
> +		check_rcv $rcv_if_name \
> +			"Unicast IPv4 to unknown MAC address, allmulti" \
> +			"$smac > $UNKNOWN_UC_ADDR3, ethertype IPv4 (0x0800)" \
> +			false
>  
>  	check_rcv $rcv_if_name "Multicast IPv4 to joined group" \
>  		"$smac > $JOINED_MACV4_MC_ADDR, ethertype IPv4 (0x0800)" \
>  		true
>  
> -	check_rcv $rcv_if_name "Multicast IPv4 to unknown group" \
> -		"$smac > $UNKNOWN_MACV4_MC_ADDR1, ethertype IPv4 (0x0800)" \
> -		false
> +	xfail_on_veth $h1 \
> +		check_rcv $rcv_if_name \
> +			"Multicast IPv4 to unknown group" \
> +			"$smac > $UNKNOWN_MACV4_MC_ADDR1, ethertype IPv4 (0x0800)" \
> +			false
>  
>  	check_rcv $rcv_if_name "Multicast IPv4 to unknown group, promisc" \
>  		"$smac > $UNKNOWN_MACV4_MC_ADDR2, ethertype IPv4 (0x0800)" \
> @@ -187,9 +192,10 @@ run_test()
>  		"$smac > $JOINED_MACV6_MC_ADDR, ethertype IPv6 (0x86dd)" \
>  		true
>  
> -	check_rcv $rcv_if_name "Multicast IPv6 to unknown group" \
> -		"$smac > $UNKNOWN_MACV6_MC_ADDR1, ethertype IPv6 (0x86dd)" \
> -		false
> +	xfail_on_veth $h1 \
> +		check_rcv $rcv_if_name "Multicast IPv6 to unknown group" \
> +			"$smac > $UNKNOWN_MACV6_MC_ADDR1, ethertype IPv6 (0x86dd)" \
> +			false
>  
>  	check_rcv $rcv_if_name "Multicast IPv6 to unknown group, promisc" \
>  		"$smac > $UNKNOWN_MACV6_MC_ADDR2, ethertype IPv6 (0x86dd)" \
> -- 
> 2.45.0
> 

Reviewed-by: Hangbin Liu <liuhangbin@gmail.com>

Re: [PATCH net v3] selftests: net: local_termination: annotate the expected failures

[Petr Machata]

Hangbin Liu <liuhangbin@gmail.com> writes:

> On Thu, May 16, 2024 at 08:25:13AM -0700, Jakub Kicinski wrote:
>> Vladimir said when adding this test:
>> 
>>   The bridge driver fares particularly badly [...] mainly because
>>   it does not implement IFF_UNICAST_FLT.
>> 
>> See commit 90b9566aa5cd ("selftests: forwarding: add a test for
>> local_termination.sh").
>> 
>> We don't want to hide the known gaps, but having a test which
>> always fails prevents us from catching regressions. Report
>> the cases we know may fail as XFAIL.
>> 
>> Signed-off-by: Jakub Kicinski <kuba@kernel.org>

I'm still confused that the failure was shown for $rcv_if_name==bridge,
yet we are testing $h1==veth. But mechanically the code is correct.

Reviewed-by: Petr Machata <petrm@nvidia.com>

Re: [PATCH net v3] selftests: net: local_termination: annotate the expected failures

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Thu, 16 May 2024 08:25:13 -0700 you wrote:
> Vladimir said when adding this test:
> 
>   The bridge driver fares particularly badly [...] mainly because
>   it does not implement IFF_UNICAST_FLT.
> 
> See commit 90b9566aa5cd ("selftests: forwarding: add a test for
> local_termination.sh").
> 
> [...]

Here is the summary with links:
  - [net,v3] selftests: net: local_termination: annotate the expected failures
    https://git.kernel.org/netdev/net/c/fe56d6e4a99a

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html