Add network RoutingPolicyRules at OpenBMC Networkd

Add network RoutingPolicyRules at OpenBMC Networkd

[raviteja bailapudi]

[-- Attachment #1: Type: text/plain, Size: 1467 bytes --]

Hi Team

We have noticed network routing issues when the same subnet IP addresses
configured on both eth0 and eth1 ethernet interfaces, this issue applies to
all types of addresses like static, DHCP and LinkLocal address
configuration.

Currently IPv4 LinkLocal addressing is enabled on both interfaces, so both
interfaces come up with the same subnet Link local IP addresses
(169.254.x.y), but only one link local address will be reachable due to
these same subnet routes on both interfaces.

Here is the systemd issue https://github.com/systemd/systemd/issues/28814
I have discussed in the systemd community and explored systemd's
RoutingPolicyRules configuration as suggested by the systemd community and
it works.

To solve this problem we are proposing to make changes in phosphor-networkd
to configure/populate systemd-networkd RoutingPolicyRule for each IP
address configured on each interface, there is no user intervention or user
interface changes needed. phosphor-networkd internally takes care of
updating the systemd-networkd configuration as required

Here is the example of additional systemd configuration required for each
IP address configured on the interface.
Example:
[Route]
PreferredSource=169.254.202.113
Destination=169.254.202.113/16
Table=11
[Route]
Gateway=169.254.0.0
Table=11
[RoutingPolicyRule]
Table=11
To=169.254.202.113/16
[RoutingPolicyRule]
Table=11
From=169.254.202.113/16

Please share your views on the same.

Regards,
Raviteja

[-- Attachment #2: Type: text/html, Size: 1792 bytes --]

RE: Add network RoutingPolicyRules at OpenBMC Networkd

[Alexander Amelkin]

[-- Attachment #1: Type: text/plain, Size: 3549 bytes --]

There is waaaay too many problems that we have with “multiple interfaces using link-local addresses” per se.
They are outlined in https://datatracker.ietf.org/doc/html/rfc3927#section-3 but no solutions are actually provided there.
When, in addition to that, you connect those multiple interfaces to the same physical network segment, you add another
bunch of problems on top that are discussed here:
https://serverfault.com/questions/415304/multiple-physical-interfaces-with-ips-on-the-same-subnet

With your proposal, as I understand it, you may be solving one part of the “scoped addresses” problem here, but the rest remains.

I may be missing some use-cases, but I gather link-local addresses are primarily needed for discovery of newly installed OpenBMC
machines before they are properly integrated into a DHCP-based or statically configure network. Taking that into account, in my
humble opinion, the best way would be to have link-local addressing enabled for just eth0.

I also believe that in production environments the BMC shall not be connected to the same network segment using multiple interfaces.
If that is needed for failover, then we should think about adding the bridging and bonding support instead.

WBR, Alexander Amelkin

P.S. I also believe that it is VERY wrong that we still allow setting per-interface gateways as BMC is not a router device and doesn’t (and shouldn’t) allow for configuring policy routing or any routing whatsoever.

From: openbmc <openbmc-bounces+a.amelkin=yadro.com@lists.ozlabs.org> On Behalf Of raviteja bailapudi
Sent: Wednesday, October 11, 2023 11:27 AM
To: OpenBMC Maillist <openbmc@lists.ozlabs.org>; Ratan Gupta <ratankgupta31@gmail.com>; wak@google.com; Sunitha Harish <sunithaharish04@gmail.com>; johnathanx.mantey@intel.com
Subject: Add network RoutingPolicyRules at OpenBMC Networkd


«Внимание! Данное письмо от внешнего адресата!»
Hi Team

We have noticed network routing issues when the same subnet IP addresses configured on both eth0 and eth1 ethernet interfaces, this issue applies to all types of addresses like static, DHCP and LinkLocal address configuration.

Currently IPv4 LinkLocal addressing is enabled on both interfaces, so both interfaces come up with the same subnet Link local IP addresses (169.254.x.y), but only one link local address will be reachable due to these same subnet routes on both interfaces.

Here is the systemd issue https://github.com/systemd/systemd/issues/28814
I have discussed in the systemd community and explored systemd's RoutingPolicyRules configuration as suggested by the systemd community and it works.

To solve this problem we are proposing to make changes in phosphor-networkd to configure/populate systemd-networkd RoutingPolicyRule for each IP address configured on each interface, there is no user intervention or user interface changes needed. phosphor-networkd internally takes care of updating the systemd-networkd configuration as required

Here is the example of additional systemd configuration required for each IP address configured on the interface.
Example:
[Route]
PreferredSource=169.254.202.113
Destination=169.254.202.113/16<http://169.254.202.113/16>
Table=11
[Route]
Gateway=169.254.0.0
Table=11
[RoutingPolicyRule]
Table=11
To=169.254.202.113/16<http://169.254.202.113/16>
[RoutingPolicyRule]
Table=11
From=169.254.202.113/16<http://169.254.202.113/16>

Please share your views on the same.

Regards,
Raviteja

[-- Attachment #2: Type: text/html, Size: 8356 bytes --]

Re: Add network RoutingPolicyRules at OpenBMC Networkd

[Paul Fertser]

Hello,

A short clarification inline.

On Fri, Oct 13, 2023 at 11:18:33AM +0000, Alexander Amelkin wrote:
> I may be missing some use-cases, but I gather link-local addresses are primarily
> needed for discovery of newly installed OpenBMC
> 
> machines before they are properly integrated into a DHCP-based or statically
> configure network. Taking that into account, in my
> 
> humble opinion, the best way would be to have link-local addressing enabled for
> just eth0.

To avoid potential of confusion it's important to note that the
reasoning applies to legacy IP only, for IPv6 every interface normally
has a link-local address at all times.

-- 
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:fercerpav@gmail.com

Re: Add network RoutingPolicyRules at OpenBMC Networkd

[Sunitha Harish]

[-- Attachment #1: Type: text/plain, Size: 4549 bytes --]

Thank you Alexander for responding. As Paul mentioned rightly in another 
reply, this problem is seen with IPv4 only.

But when a system with two physical interfaces is shipped, i think it is 
expected that they should come up with link-local IP by default. So we 
should find a best way to solve this problem.

On 13-10-2023 16:48, Alexander Amelkin wrote:
>
> There is waaaay too many problems that we have with “multiple 
> interfaces using link-local addresses” per se.
>
Yes actually.
>
> They are outlined in 
> https://datatracker.ietf.org/doc/html/rfc3927#section-3 but no 
> solutions are actually provided there.
>
> When, in addition to that, you connect those multiple interfaces to 
> the same physical network segment, you add another
>
> bunch of problems on top that are discussed here:
>
> https://serverfault.com/questions/415304/multiple-physical-interfaces-with-ips-on-the-same-subnet
>
We tried these settings and this did not cover all the usecases as we 
expected. ( Static & DHCP and combination of both at the 2 interfaces)
>
>
> With your proposal, as I understand it, you may be solving one part of 
> the “scoped addresses” problem here, but the rest remains.
>
> I may be missing some use-cases, but I gather link-local addresses are 
> primarily needed for discovery of newly installed OpenBMC
>
> machines before they are properly integrated into a DHCP-based or 
> statically configure network. Taking that into account, in my
>
> humble opinion, the best way would be to have link-local addressing 
> enabled for just eth0.
>
Yes, currently the problem workaround is to enable the link-local only 
on eth0.

> I also believe that in production environments the BMC shall not be 
> connected to the same network segment using multiple interfaces.
>
> If that is needed for failover, then we should think about adding the 
> bridging and bonding support instead.
>
Do you have more details on this please?
>
> WBR, Alexander Amelkin
>
> P.S. I also believe that it is VERY wrong that we still allow setting 
> per-interface gateways as BMC is not a router device and doesn’t (and 
> shouldn’t) allow for configuring policy routing or any routing whatsoever.
>
I believe this was done purposefully some years ago. @Ravi please share 
the details if you have the history.

Thanks &regards,
Sunitha
>
> *From:*openbmc <openbmc-bounces+a.amelkin=yadro.com@lists.ozlabs.org> 
> *On Behalf Of *raviteja bailapudi
> *Sent:* Wednesday, October 11, 2023 11:27 AM
> *To:* OpenBMC Maillist <openbmc@lists.ozlabs.org>; Ratan Gupta 
> <ratankgupta31@gmail.com>; wak@google.com; Sunitha Harish 
> <sunithaharish04@gmail.com>; johnathanx.mantey@intel.com
> *Subject:* Add network RoutingPolicyRules at OpenBMC Networkd
>
> «Внимание! Данное письмо от внешнего адресата!»
>
> Hi Team
>
> We have noticed network routing issues when the same subnet IP 
> addresses configured on both eth0 and eth1 ethernet interfaces, this 
> issue applies to all types of addresses like static, DHCP and 
> LinkLocal address configuration.
>
> Currently IPv4 LinkLocal addressing is enabled on both interfaces, so 
> both interfaces come up with the same subnet Link local IP addresses 
> (169.254.x.y), but only one link local address will be reachable due 
> to these same subnet routes on both interfaces.
>
> Here is the systemd issue 
> https://github.com/systemd/systemd/issues/28814 
> <https://github.com/systemd/systemd/issues/28814>
> I have discussed in the systemd community and explored systemd's 
> RoutingPolicyRules configuration as suggested by the systemd community 
> and it works.
>
> To solve this problem we are proposing to make changes in 
> phosphor-networkd to configure/populate systemd-networkd 
> RoutingPolicyRule for each IP address configured on each interface, 
> there is no user intervention or user interface changes needed. 
> phosphor-networkd internally takes care of updating the 
> systemd-networkd configuration as required
>
> Here is the example of additional systemd configuration required for 
> each IP address configured on the interface.
> Example:
> [Route]
> PreferredSource=169.254.202.113
> Destination=169.254.202.113/16 <http://169.254.202.113/16>
> Table=11
> [Route]
> Gateway=169.254.0.0
> Table=11
> [RoutingPolicyRule]
> Table=11
> To=169.254.202.113/16 <http://169.254.202.113/16>
> [RoutingPolicyRule]
> Table=11
> From=169.254.202.113/16 <http://169.254.202.113/16>
>
> Please share your views on the same.
>
> Regards,
>
> Raviteja
>

[-- Attachment #2: Type: text/html, Size: 12739 bytes --]

RE: Add network RoutingPolicyRules at OpenBMC Networkd

[Alexander Amelkin]

[-- Attachment #1: Type: text/plain, Size: 1227 bytes --]

Hi Sunitha!

>> I also believe that in production environments the BMC shall not be connected to the same network segment using multiple interfaces.
>> If that is needed for failover, then we should think about adding the bridging and bonding support instead
> Do you have more details on this please?

Sure. Honestly, I haven’t dealt with either technique for quite many years now, and when I had, my experience was far from being vast, but I still remember those things do exist.
So here are some resources that I found that look relevant:

https://developers.redhat.com/articles/2022/04/06/introduction-linux-bridging-commands-and-features
https://www.learnitguide.net/2015/07/what-is-bonding-how-to-configure.html

>> P.S. I also believe that it is VERY wrong that we still allow setting per-interface gateways as BMC is not a router device and doesn’t (and shouldn’t) allow for configuring policy routing or any routing whatsoever.

> I believe this was done purposefully some years ago. @Ravi please share the details if you have the history.
I you can provide any rationale for that, I’d be grateful. We’ve been having all sorts of trouble due to that for quite a long time.

WBR, Alexander.

[-- Attachment #2: Type: text/html, Size: 4214 bytes --]

Re: Add network RoutingPolicyRules at OpenBMC Networkd

[raviteja bailapudi]

[-- Attachment #1: Type: text/plain, Size: 1673 bytes --]

Hi Alexander

Only one issue we have noticed by supporting per-interface specific gateway
is when same subnet IP addresses configured on both
eth0 and eth1 ethernet interfaces, otherwise don't see any BMC
configuration issues by supporting per-interface specific gateway.

I also agree that in production environments the BMC may not be connected
to the same network segment using multiple interfaces
but when a system with two physical interfaces is shipped, each interface
expected to have link-local IP by default to work independently.
instead of having workaround of disabling linkLocalAddressing on eth1
interface.

This proposal of configuring RoutePolicyRule going to fix the same subnet
IPaddrese configuration issue
for all types of addresses like static, DHCP and LinkLocal address
configuration and this is systemd community suggested solution.

 >With your proposal, as I understand it, you may be solving one part of
the “scoped addresses” problem here, but the rest remains.
can you please mention specific BMC configuration problems which
remains here?

I believe adding per-interface specific needed for BMC

> > I believe this was done purposefully some years ago. @Ravi please share
> the details if you have the history.
>
> I you can provide any rationale for that, I’d be grateful. We’ve been
> having all sorts of trouble due to that for quite a long time.
>

Please find discussions  on supporting per-interface specific gateway here
https://github.com/openbmc/openbmc/issues/2671
https://github.com/openbmc/phosphor-networkd/issues/26

Thank you so much for your response

Thanks & Regards
Raviteja

[-- Attachment #2: Type: text/html, Size: 2783 bytes --]