VRF Routing Rule Matching Issue: oif Rules Not Working After Commit 40867d74c374

VRF Routing Rule Matching Issue: oif Rules Not Working After Commit 40867d74c374

[hanhuihui]

Dear Kernel Community and Network Maintainers,
I am analyzing the issue, and I am very happy for any replies.
After the application committed 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.

Problem Description:
When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule.
Here are our configuration steps:
ip address add 11.47.3.130/16 dev enp4s0
ip address add 11.47.3.130/16 dev enp5s0

ip link add name vrf-srv-1 type vrf table 10
ip link set dev vrf-srv-1 up
ip link set dev enp4s0 master vrf-srv-1

ip link add name vrf-srv type vrf table 20
ip link set dev vrf-srv up
ip link set dev enp5s0 master vrf-srv

ip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0
ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0
ip rule add from 11.47.3.130 table 20 prio 997


In this configuration, when the following commands are executed:
ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.

Is this the expected behavior?
The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?

Re: VRF Routing Rule Matching Issue: oif Rules Not Working After Commit 40867d74c374

[Ido Schimmel]

On Thu, Apr 03, 2025 at 01:58:46AM +0000, hanhuihui wrote:
> Dear Kernel Community and Network Maintainers,
> I am analyzing the issue, and I am very happy for any replies.
> After the application committed 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.
> 
> Problem Description:
> When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule.
> Here are our configuration steps:
> ip address add 11.47.3.130/16 dev enp4s0
> ip address add 11.47.3.130/16 dev enp5s0
> 
> ip link add name vrf-srv-1 type vrf table 10
> ip link set dev vrf-srv-1 up
> ip link set dev enp4s0 master vrf-srv-1
> 
> ip link add name vrf-srv type vrf table 20
> ip link set dev vrf-srv up
> ip link set dev enp5s0 master vrf-srv
> 
> ip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0
> ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0
> ip rule add from 11.47.3.130 table 20 prio 997
> 
> 
> In this configuration, when the following commands are executed:
> ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
> Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
> Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.
> 
> Is this the expected behavior?
> The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
> If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?

Can you try replacing the first two rules with:

ip rule add from 11.47.3.130 l3mdev prio 0

And see if it helps?

It's not exactly equivalent to your two rules, but it says "if source
address is 11.47.3.130 and flow is associated with a L3 master device,
then direct the FIB lookup to the table associated with the L3 master
device"

The commit you referenced added the index of the L3 master device to the
flow structure, but I don't believe we have an explicit way to match on
it using FIB rules. It would be useful to add a new keyword (e.g.,
l3mdevif) and then your rules can become:

ip rule add from 11.47.3.130 l3mdevif vrf-srv-1 table 10 prio 0
ip rule add from 11.47.3.130 table 20 prio 997

But it requires kernel changes.

Re: VRF Routing Rule Matching Issue: oif Rules Not Working After Commit 40867d74c374

[hanhuihui]

On Mon, 7 Apr 2025 11:29:02 +0300, Ido Schimmel wrote:
>On Thu, Apr 03, 2025 at 01:58:46AM +0000, hanhuihui wrote:
>> Dear Kernel Community and Network Maintainers,
>> I am analyzing the issue, and I am very happy for any replies.
>> After the application committed 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.
>> 
>> Problem Description:
>> When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule.
>> Here are our configuration steps:
>> ip address add 11.47.3.130/16 dev enp4s0
>> ip address add 11.47.3.130/16 dev enp5s0
>> 
>> ip link add name vrf-srv-1 type vrf table 10
>> ip link set dev vrf-srv-1 up
>> ip link set dev enp4s0 master vrf-srv-1
>> 
>> ip link add name vrf-srv type vrf table 20
>> ip link set dev vrf-srv up
>> ip link set dev enp5s0 master vrf-srv
>> 
>> ip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0
>> ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0
>> ip rule add from 11.47.3.130 table 20 prio 997
>> 
>> 
>> In this configuration, when the following commands are executed:
>> ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
>> Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
>> Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.
>> 
>> Is this the expected behavior?
>> The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
>> If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?
>
>Can you try replacing the first two rules with:
>
>ip rule add from 11.47.3.130 l3mdev prio 0
>

This does not work in scenarios where the routing table specified by the oif/iif is not in the l3mdev-table.

>And see if it helps?
>
>It's not exactly equivalent to your two rules, but it says "if source
>address is 11.47.3.130 and flow is associated with a L3 master device,
>then direct the FIB lookup to the table associated with the L3 master
>device"
>
>The commit you referenced added the index of the L3 master device to the
>flow structure, but I don't believe we have an explicit way to match on
>it using FIB rules. It would be useful to add a new keyword (e.g.,
>l3mdevif) and then your rules can become:
>
>ip rule add from 11.47.3.130 l3mdevif vrf-srv-1 table 10 prio 0
>ip rule add from 11.47.3.130 table 20 prio 997
>
>But it requires kernel changes.

Before the patch is installed, oif/iif rules can be configured for traffic from the VRF and traffic can be forwarded normally. 
However, in this patch, traffic from the VRF resets fl->flowi_oif in l3mdev_update_flow. As a result, the 'rule->oifindex != fl->flowi_oif' 
condition in fib_rule_match cannot be met, the oif rule cannot be matched. The patch also mentions "oif set to L3mdev directs lookup to its table; 
reset to avoid oif match in fib_lookup" in the modification, which seems to be intentional. I'm rather confused about this. Does the modification 
ignore the scenario where the oif/iif rule is configured on the VRF, or is the usage of the oif/iif rule no longer supported by the community after 
the patch is installed, or is the usage of the oif/iif rule incorrectly used?
Any reply would be greatly appreciated.

Thanks!

Re: VRF Routing Rule Matching Issue: oif Rules Not Working After Commit 40867d74c374

[David Ahern]

On 4/8/25 10:17 AM, hanhuihui wrote:
> Before the patch is installed, oif/iif rules can be configured for traffic from the VRF and traffic can be forwarded normally. 
> However, in this patch, traffic from the VRF resets fl->flowi_oif in l3mdev_update_flow. As a result, the 'rule->oifindex != fl->flowi_oif' 
> condition in fib_rule_match cannot be met, the oif rule cannot be matched. The patch also mentions "oif set to L3mdev directs lookup to its table; 
> reset to avoid oif match in fib_lookup" in the modification, which seems to be intentional. I'm rather confused about this. Does the modification 
> ignore the scenario where the oif/iif rule is configured on the VRF, or is the usage of the oif/iif rule no longer supported by the community after 
> the patch is installed, or is the usage of the oif/iif rule incorrectly used?
> Any reply would be greatly appreciated.
> 

oif/iif rules per VRF does not scale; the l3mdev rule was added to
address that problem.

Re: VRF Routing Rule Matching Issue: oif Rules Not Working After Commit 40867d74c374

[Ido Schimmel]

On Wed, Apr 09, 2025 at 12:17:56AM +0800, hanhuihui wrote:
> On Mon, 7 Apr 2025 11:29:02 +0300, Ido Schimmel wrote:
> >On Thu, Apr 03, 2025 at 01:58:46AM +0000, hanhuihui wrote:
> >> Dear Kernel Community and Network Maintainers,
> >> I am analyzing the issue, and I am very happy for any replies.
> >> After the application committed 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices"), we noticed an unexpected change in VRF routing rule matching behavior. We hereby submit a problem report to confirm whether this is the expected behavior.
> >> 
> >> Problem Description:
> >> When interfaces bound to multiple VRFs share the same IP address, the OIF (output interface) routing rule is no longer matched after being committed. As a result, traffic incorrectly matches the low-priority rule.
> >> Here are our configuration steps:
> >> ip address add 11.47.3.130/16 dev enp4s0
> >> ip address add 11.47.3.130/16 dev enp5s0
> >> 
> >> ip link add name vrf-srv-1 type vrf table 10
> >> ip link set dev vrf-srv-1 up
> >> ip link set dev enp4s0 master vrf-srv-1
> >> 
> >> ip link add name vrf-srv type vrf table 20
> >> ip link set dev vrf-srv up
> >> ip link set dev enp5s0 master vrf-srv
> >> 
> >> ip rule add from 11.47.3.130 oif vrf-srv-1 table 10 prio 0
> >> ip rule add from 11.47.3.130 iif vrf-srv-1 table 10 prio 0
> >> ip rule add from 11.47.3.130 table 20 prio 997
> >> 
> >> 
> >> In this configuration, when the following commands are executed:
> >> ip vrf exec vrf-srv-1 ping "11.47.9.250" -I 11.47.3.130
> >> Expected behavior: The traffic should match the oif vrf-srv-1 rule of prio 0. Table 10 is used.
> >> Actual behavior: The traffic skips the oif rule and matches the default rule of prio 997 (Table 20), causing the ping to fail.
> >> 
> >> Is this the expected behavior?
> >> The submission description mentions "avoid oif reset of port devices". Does this change the matching logic of oif in VRF scenarios?
> >> If this change is intentional, how should the VRF configuration be adjusted to ensure that oif rules are matched first? Is it necessary to introduce a new mechanism?
> >
> >Can you try replacing the first two rules with:
> >
> >ip rule add from 11.47.3.130 l3mdev prio 0
> >
> 
> This does not work in scenarios where the routing table specified by the oif/iif is not in the l3mdev-table.

You mean when "oif" / "iif" points to a VRF and "table" specifies a
table different than the one associated with the VRF? Then, yes, it
won't work, but in your example "vrf-srv-1" is associated with table
"10".

> 
> >And see if it helps?
> >
> >It's not exactly equivalent to your two rules, but it says "if source
> >address is 11.47.3.130 and flow is associated with a L3 master device,
> >then direct the FIB lookup to the table associated with the L3 master
> >device"
> >
> >The commit you referenced added the index of the L3 master device to the
> >flow structure, but I don't believe we have an explicit way to match on
> >it using FIB rules. It would be useful to add a new keyword (e.g.,
> >l3mdevif) and then your rules can become:
> >
> >ip rule add from 11.47.3.130 l3mdevif vrf-srv-1 table 10 prio 0
> >ip rule add from 11.47.3.130 table 20 prio 997
> >
> >But it requires kernel changes.
> 
> Before the patch is installed, oif/iif rules can be configured for traffic from the VRF and traffic can be forwarded normally. 
> However, in this patch, traffic from the VRF resets fl->flowi_oif in l3mdev_update_flow. As a result, the 'rule->oifindex != fl->flowi_oif' 
> condition in fib_rule_match cannot be met, the oif rule cannot be matched. The patch also mentions "oif set to L3mdev directs lookup to its table; 
> reset to avoid oif match in fib_lookup" in the modification, which seems to be intentional. I'm rather confused about this. Does the modification 
> ignore the scenario where the oif/iif rule is configured on the VRF, or is the usage of the oif/iif rule no longer supported by the community after 
> the patch is installed, or is the usage of the oif/iif rule incorrectly used?
> Any reply would be greatly appreciated.

Before 40867d74c374 you couldn't match with FIB rules on oif/iif being a
VRF slave because these fields in the flow structure were reset to the
index of the VRF device in l3mdev_update_flow().

I will try to check if we can interpret FIB rules with "oif" / "iif"
pointing to a VRF as matching on "fl->flowi_l3mdev" rather than
"fl->flowi_oif" / "fl->flowi_iif".

Re: VRF Routing Rule Matching Issue: oif Rules Not Working After Commit 40867d74c374

[hanhuihui]

On Tue, 8 Apr 2025 22:54:33 +0300 Ido Schimmel wrote:
>Before 40867d74c374 you couldn't match with FIB rules on oif/iif being a
>VRF slave because these fields in the flow structure were reset to the
>index of the VRF device in l3mdev_update_flow().
>
>I will try to check if we can interpret FIB rules with "oif" / "iif"
>pointing to a VRF as matching on "fl->flowi_l3mdev" rather than
>"fl->flowi_oif" / "fl->flowi_iif".
Thank you for your reply. Followed by a simple patch to restore the 
previous behavior, hopefully enlightening you.

[PATCH] resume oif rule match l3mdev in fib_lookup

[hanhuihui]

flowi_oif will be reset if flowi_oif set to a l3mdev (e.g., VRF) device.
This causes the oif rule to fail to match the l3mdev device in fib_lookup.
Let's get back to previous behavior.

Fixes: 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")
Signed-off-by: hanhuihui hanhuihui5@huawei.com
---
 net/core/fib_rules.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/core/fib_rules.c b/net/core/fib_rules.c
index 4bc64d9..3c2a2db 100644
--- a/net/core/fib_rules.c
+++ b/net/core/fib_rules.c
@@ -268,7 +268,7 @@ static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops,
 		goto out;
 
 	oifindex = READ_ONCE(rule->oifindex);
-	if (oifindex && (oifindex != fl->flowi_oif))
+	if (oifindex && (oifindex != (fl->flowi_l3mdev ? : fl->flowi_oif)))
 		goto out;
 
 	if ((rule->mark ^ fl->flowi_mark) & rule->mark_mask)
-- 
2.27.0

Re: [PATCH] resume oif rule match l3mdev in fib_lookup

[Jakub Kicinski]

On Sat, 12 Apr 2025 21:19:10 +0800 hanhuihui wrote:
> flowi_oif will be reset if flowi_oif set to a l3mdev (e.g., VRF) device.
> This causes the oif rule to fail to match the l3mdev device in fib_lookup.
> Let's get back to previous behavior.
> 
> Fixes: 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")
> Signed-off-by: hanhuihui hanhuihui5@huawei.com

If you'd like to go ahead with this approach you need to at the very
least adjust the fib_rule_tests.sh selftest..

Re: [PATCH] resume oif rule match l3mdev in fib_lookup

[Ido Schimmel]

On Sat, Apr 12, 2025 at 09:19:10PM +0800, hanhuihui wrote:
> flowi_oif will be reset if flowi_oif set to a l3mdev (e.g., VRF) device.
> This causes the oif rule to fail to match the l3mdev device in fib_lookup.
> Let's get back to previous behavior.
> 
> Fixes: 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")
> Signed-off-by: hanhuihui hanhuihui5@huawei.com
> ---
>  net/core/fib_rules.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/net/core/fib_rules.c b/net/core/fib_rules.c
> index 4bc64d9..3c2a2db 100644
> --- a/net/core/fib_rules.c
> +++ b/net/core/fib_rules.c
> @@ -268,7 +268,7 @@ static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops,
>  		goto out;
>  
>  	oifindex = READ_ONCE(rule->oifindex);
> -	if (oifindex && (oifindex != fl->flowi_oif))
> +	if (oifindex && (oifindex != (fl->flowi_l3mdev ? : fl->flowi_oif)))

This will prevent us from matching on the output device when the device
is enslaved to a VRF. We should try to match on L3 domain only if the
FIB rule matches on a VRF device. I will try to send a fix today (wasn't
feeling well in the last few days).

>  		goto out;
>  
>  	if ((rule->mark ^ fl->flowi_mark) & rule->mark_mask)
> -- 
> 2.27.0
> 

Re: [PATCH] resume oif rule match l3mdev in fib_lookup

[Ido Schimmel]

On Mon, Apr 14, 2025 at 10:17:35AM +0300, Ido Schimmel wrote:
> On Sat, Apr 12, 2025 at 09:19:10PM +0800, hanhuihui wrote:
> > flowi_oif will be reset if flowi_oif set to a l3mdev (e.g., VRF) device.
> > This causes the oif rule to fail to match the l3mdev device in fib_lookup.
> > Let's get back to previous behavior.
> > 
> > Fixes: 40867d74c374 ("net: Add l3mdev index to flow struct and avoid oif reset for port devices")
> > Signed-off-by: hanhuihui hanhuihui5@huawei.com
> > ---
> >  net/core/fib_rules.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/net/core/fib_rules.c b/net/core/fib_rules.c
> > index 4bc64d9..3c2a2db 100644
> > --- a/net/core/fib_rules.c
> > +++ b/net/core/fib_rules.c
> > @@ -268,7 +268,7 @@ static int fib_rule_match(struct fib_rule *rule, struct fib_rules_ops *ops,
> >  		goto out;
> >  
> >  	oifindex = READ_ONCE(rule->oifindex);
> > -	if (oifindex && (oifindex != fl->flowi_oif))
> > +	if (oifindex && (oifindex != (fl->flowi_l3mdev ? : fl->flowi_oif)))
> 
> This will prevent us from matching on the output device when the device
> is enslaved to a VRF. We should try to match on L3 domain only if the
> FIB rule matches on a VRF device. I will try to send a fix today (wasn't
> feeling well in the last few days).

Posted a fix:
https://lore.kernel.org/netdev/20250414172022.242991-2-idosch@nvidia.com/

Can you please test it and tag it if it fixes your issue?

Thanks

Re: [PATCH] resume oif rule match l3mdev in fib_lookup

[hanhuihui]

On Mon, 14 Apr 2025 20:24:11 +0300, Ido Schimmel wrote:
>On Mon, Apr 14, 2025 at 10:17:35AM +0300, Ido Schimmel wrote:
..
>> This will prevent us from matching on the output device when the device
>> is enslaved to a VRF. We should try to match on L3 domain only if the
>> FIB rule matches on a VRF device. I will try to send a fix today (wasn't
>> feeling well in the last few days).
>
>Posted a fix:
>https://lore.kernel.org/netdev/20250414172022.242991-2-idosch@nvidia.com/
>
>Can you please test it and tag it if it fixes your issue?

Tested-by: hanhuihui hanhuihui5@huawei.com

The patch successfully resolves the VRF oif rule matching issue. The FIB lookup now correctly identifies 
the L3 domain when using VRFs. Thank you for the quick fix.