Default Gateway for a system v/s Default gateway per Interface

Default Gateway for a system v/s Default gateway per Interface

[Ratan Gupta]

Hi All,

Currently, OpenBMC stack allows a single default gateway for the system. 
Latest kernel allows to configure multiple default gateways.

Eg: In a system with two interfaces eth0 and eth1,

eth0 configured with static address and having gateway(192.168.2.1)
eth1 configured with DHCP and gets Gateway from DHCP server (10.10.10.1)
~~~~~~~~~~~~~
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt 
Iface
0.0.0.0         19.168.2.1      0.0.0.0         UG        0 0          0 
eth0
0.0.0.0         10.10.10.1      0.0.0.0         UG        0 0          0 
eth1
~~~~~~~~~~~~~~

Kernel will first try using the default gateway having higher metric 
value and then fall back to the lower.

More references: 
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_the_default_gateway

I'm proposing to make this change in the openBMC D-bus interfaces to tie 
the gateway property with the Ethernet interface schema instead of 
System configuration.

Ethernet Interface Schema =>

https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/EthernetInterface.interface.yaml

System Configuration Schema =>

https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/SystemConfiguration.interface.yaml


Please let me know your suggestions.

Regards
Ratan Gupta

Re: Default Gateway for a system v/s Default gateway per Interface

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
> ~~~~~~~~~~~~~
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags   MSS Window  irtt 
> Iface
> 0.0.0.0         19.168.2.1      0.0.0.0         UG        0 0          0 
> eth0
> 0.0.0.0         10.10.10.1      0.0.0.0         UG        0 0          0 
> eth1
> ~~~~~~~~~~~~~~
> 
> Kernel will first try using the default gateway having higher metric 
> value and then fall back to the lower.

I'm not seeing us with an interface to adjust the metric for an
interface.  I think we need to add that at the same time?  Otherwise, I
don't think we have a way to specify which interface outside-the-subnet
should go (vs today we can set the default-gateway to the desired
interface's gateway).
-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[William Kennington]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

Sounds good to me, we have needed this for a long time because the current
gateway configuration breaks our v6 stack with multiple NICs.

On Fri, Apr 24, 2020 at 8:21 AM Patrick Williams <patrick@stwcx.xyz> wrote:

> On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
> > ~~~~~~~~~~~~~
> > Kernel IP routing table
> > Destination     Gateway         Genmask         Flags   MSS Window  irtt
> > Iface
> > 0.0.0.0         19.168.2.1      0.0.0.0         UG        0 0          0
> > eth0
> > 0.0.0.0         10.10.10.1      0.0.0.0         UG        0 0          0
> > eth1
> > ~~~~~~~~~~~~~~
> >
> > Kernel will first try using the default gateway having higher metric
> > value and then fall back to the lower.
>
> I'm not seeing us with an interface to adjust the metric for an
> interface.  I think we need to add that at the same time?  Otherwise, I
> don't think we have a way to specify which interface outside-the-subnet
> should go (vs today we can set the default-gateway to the desired
> interface's gateway).
> --
> Patrick Williams
>

[-- Attachment #2: Type: text/html, Size: 1552 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Ratan Gupta]

[-- Attachment #1: Type: text/plain, Size: 1572 bytes --]

Thanks Patrick, William for sharing the feedbacks

I will start working on the changes.


On 4/24/20 11:15 PM, William Kennington wrote:
> Sounds good to me, we have needed this for a long time because the 
> current gateway configuration breaks our v6 stack with multiple NICs.
>
> On Fri, Apr 24, 2020 at 8:21 AM Patrick Williams <patrick@stwcx.xyz 
> <mailto:patrick@stwcx.xyz>> wrote:
>
>     On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
>     > ~~~~~~~~~~~~~
>     > Kernel IP routing table
>     > Destination     Gateway         Genmask         Flags MSS
>     Window  irtt
>     > Iface
>     > 0.0.0.0         19.168.2.1      0.0.0.0         UG 0 0          0
>     > eth0
>     > 0.0.0.0         10.10.10.1      0.0.0.0         UG 0 0          0
>     > eth1
>     > ~~~~~~~~~~~~~~
>     >
>     > Kernel will first try using the default gateway having higher
>     metric
>     > value and then fall back to the lower.
>
>     I'm not seeing us with an interface to adjust the metric for an
>     interface.  I think we need to add that at the same time?
>
Not now, As per my testing if metric value is not defined and both the 
routes

having same metric then kernel tries one after other. We can bring the 
metric

later.

>     Otherwise, I
>     don't think we have a way to specify which interface
>     outside-the-subnet
>     should go (vs today we can set the default-gateway to the desired
>     interface's gateway).
>     -- 
>     Patrick Williams
>

[-- Attachment #2: Type: text/html, Size: 3061 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Johnathan Mantey]

[-- Attachment #1.1.1: Type: text/plain, Size: 3006 bytes --]

Ratan,

Can you provide more info about how you plan to implement this?

The Redfish schemas already have a per IP addr Gateway entry, which OBMC
is ignoring. Is it better to modify the existing D-Bus schema (i.e.
versioning discussion), or just use what is presently in existence?

That said, doing as you propose melds better with how IPMI (i.e Set LAN
Configuration Parameter 12) works.

As Patrick pointed out, there's no IPMI or Redfish control over the
routing metric parameter. What are your thoughts on how to reconcile
applying a metric value?
I've long been desirous of getting Williams submission merged:
https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/20799
as it can be used to improve the collection of netlink stats.

Redfish exacerbates this whole problem by allowing a collection of
static IP addresses (and a DHCP addr) per NIC, unlike IPMI which was
very one to one with IP addr/NIC.

Thanks for investigating this and working to implement a solution.

On 4/27/20 4:11 AM, Ratan Gupta wrote:
>
> Thanks Patrick, William for sharing the feedbacks
>
> I will start working on the changes.
>
>
> On 4/24/20 11:15 PM, William Kennington wrote:
>> Sounds good to me, we have needed this for a long time because the
>> current gateway configuration breaks our v6 stack with multiple NICs.
>>
>> On Fri, Apr 24, 2020 at 8:21 AM Patrick Williams <patrick@stwcx.xyz
>> <mailto:patrick@stwcx.xyz>> wrote:
>>
>>     On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
>>     > ~~~~~~~~~~~~~
>>     > Kernel IP routing table
>>     > Destination     Gateway         Genmask         Flags   MSS
>>     Window  irtt
>>     > Iface
>>     > 0.0.0.0         19.168.2.1      0.0.0.0         UG        0
>>     0          0
>>     > eth0
>>     > 0.0.0.0         10.10.10.1      0.0.0.0         UG        0
>>     0          0
>>     > eth1
>>     > ~~~~~~~~~~~~~~
>>     >
>>     > Kernel will first try using the default gateway having higher
>>     metric
>>     > value and then fall back to the lower.
>>
>>     I'm not seeing us with an interface to adjust the metric for an
>>     interface.  I think we need to add that at the same time?
>>
> Not now, As per my testing if metric value is not defined and both the
> routes
>
> having same metric then kernel tries one after other. We can bring the
> metric
>
> later.
>
>>       Otherwise, I
>>     don't think we have a way to specify which interface
>>     outside-the-subnet
>>     should go (vs today we can set the default-gateway to the desired
>>     interface's gateway).
>>     -- 
>>     Patrick Williams
>>

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>


[-- Attachment #1.1.2: Type: text/html, Size: 5602 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Ratan Gupta]

[-- Attachment #1: Type: text/plain, Size: 3696 bytes --]

Hi Jonathan,

On 4/28/20 1:00 AM, Johnathan Mantey wrote:
> Ratan,
>
> Can you provide more info about how you plan to implement this?
I am not getting, Do you mean to say how to get the per interface 
gateway from system?  that would be through netlink, and during 
configuration we would use the systemd-networkd file configuration.
>
> The Redfish schemas already have a per IP addr Gateway entry, which 
> OBMC is ignoring. Is it better to modify the existing D-Bus schema 
> (i.e. versioning discussion), or just use what is presently in existence?
Redfish schema are getting changed to accommodate this through the 
following issue https://github.com/DMTF/Redfish/issues/3609.

The above issue(gateway/interface) exist only for the IPv4 redfish 
schema, IPv6 redfish is already having default gateway per interface

http://redfish.dmtf.org/schemas/v1/EthernetInterface.v1_5_1.json#/definitions/EthernetInterface

>
> That said, doing as you propose melds better with how IPMI (i.e Set 
> LAN Configuration Parameter 12) works.
>
> As Patrick pointed out, there's no IPMI or Redfish control over the 
> routing metric parameter. What are your thoughts on how to reconcile 
> applying a metric value?
We will go with default metric value as today as underlying stack tries 
for fallback gateway.
> I've long been desirous of getting Williams submission merged: 
> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/20799
> as it can be used to improve the collection of netlink stats.
>
> Redfish exacerbates this whole problem by allowing a collection of 
> static IP addresses (and a DHCP addr) per NIC, unlike IPMI which was 
> very one to one with IP addr/NIC.
>
> Thanks for investigating this and working to implement a solution.
>
> On 4/27/20 4:11 AM, Ratan Gupta wrote:
>>
>> Thanks Patrick, William for sharing the feedbacks
>>
>> I will start working on the changes.
>>
>>
>> On 4/24/20 11:15 PM, William Kennington wrote:
>>> Sounds good to me, we have needed this for a long time because the 
>>> current gateway configuration breaks our v6 stack with multiple NICs.
>>>
>>> On Fri, Apr 24, 2020 at 8:21 AM Patrick Williams <patrick@stwcx.xyz 
>>> <mailto:patrick@stwcx.xyz>> wrote:
>>>
>>>     On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
>>>     > ~~~~~~~~~~~~~
>>>     > Kernel IP routing table
>>>     > Destination     Gateway         Genmask Flags   MSS Window  irtt
>>>     > Iface
>>>     > 0.0.0.0         19.168.2.1      0.0.0.0 UG        0 0          0
>>>     > eth0
>>>     > 0.0.0.0         10.10.10.1      0.0.0.0 UG        0 0          0
>>>     > eth1
>>>     > ~~~~~~~~~~~~~~
>>>     >
>>>     > Kernel will first try using the default gateway having higher
>>>     metric
>>>     > value and then fall back to the lower.
>>>
>>>     I'm not seeing us with an interface to adjust the metric for an
>>>     interface.  I think we need to add that at the same time?
>>>
>> Not now, As per my testing if metric value is not defined and both 
>> the routes
>>
>> having same metric then kernel tries one after other. We can bring 
>> the metric
>>
>> later.
>>
>>>       Otherwise, I
>>>     don't think we have a way to specify which interface
>>>     outside-the-subnet
>>>     should go (vs today we can set the default-gateway to the desired
>>>     interface's gateway).
>>>     -- 
>>>     Patrick Williams
>>>
>
> -- 
> Johnathan Mantey
> Senior Software Engineer
> *azad te**chnology partners*
> Contributing to Technology Innovation since 1992
> Phone: (503) 712-6764
> Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>

Ratan


[-- Attachment #2: Type: text/html, Size: 7518 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Johnathan Mantey]

[-- Attachment #1.1.1: Type: text/plain, Size: 4333 bytes --]

I think I understand most of your plan.

What does it mean to "use the default metric"?

On 4/28/20 8:16 AM, Ratan Gupta wrote:
>
> Hi Jonathan,
>
> On 4/28/20 1:00 AM, Johnathan Mantey wrote:
>> Ratan,
>>
>> Can you provide more info about how you plan to implement this?
> I am not getting, Do you mean to say how to get the per interface
> gateway from system?  that would be through netlink, and during
> configuration we would use the systemd-networkd file configuration.
>>
>> The Redfish schemas already have a per IP addr Gateway entry, which
>> OBMC is ignoring. Is it better to modify the existing D-Bus schema
>> (i.e. versioning discussion), or just use what is presently in existence?
> Redfish schema are getting changed to accommodate this through the
> following issue https://github.com/DMTF/Redfish/issues/3609.
>
> The above issue(gateway/interface) exist only for the IPv4 redfish
> schema, IPv6 redfish is already having default gateway per interface
>
> http://redfish.dmtf.org/schemas/v1/EthernetInterface.v1_5_1.json#/definitions/EthernetInterface
>
>>
>> That said, doing as you propose melds better with how IPMI (i.e Set
>> LAN Configuration Parameter 12) works.
>>
>> As Patrick pointed out, there's no IPMI or Redfish control over the
>> routing metric parameter. What are your thoughts on how to reconcile
>> applying a metric value?
> We will go with default metric value as today as underlying stack
> tries for fallback gateway.
>> I've long been desirous of getting Williams submission merged:
>> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/20799
>> as it can be used to improve the collection of netlink stats.
>>
>> Redfish exacerbates this whole problem by allowing a collection of
>> static IP addresses (and a DHCP addr) per NIC, unlike IPMI which was
>> very one to one with IP addr/NIC.
>>
>> Thanks for investigating this and working to implement a solution.
>>
>> On 4/27/20 4:11 AM, Ratan Gupta wrote:
>>>
>>> Thanks Patrick, William for sharing the feedbacks
>>>
>>> I will start working on the changes.
>>>
>>>
>>> On 4/24/20 11:15 PM, William Kennington wrote:
>>>> Sounds good to me, we have needed this for a long time because the
>>>> current gateway configuration breaks our v6 stack with multiple NICs.
>>>>
>>>> On Fri, Apr 24, 2020 at 8:21 AM Patrick Williams <patrick@stwcx.xyz
>>>> <mailto:patrick@stwcx.xyz>> wrote:
>>>>
>>>>     On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
>>>>     > ~~~~~~~~~~~~~
>>>>     > Kernel IP routing table
>>>>     > Destination     Gateway         Genmask         Flags   MSS
>>>>     Window  irtt
>>>>     > Iface
>>>>     > 0.0.0.0         19.168.2.1      0.0.0.0         UG        0
>>>>     0          0
>>>>     > eth0
>>>>     > 0.0.0.0         10.10.10.1      0.0.0.0         UG        0
>>>>     0          0
>>>>     > eth1
>>>>     > ~~~~~~~~~~~~~~
>>>>     >
>>>>     > Kernel will first try using the default gateway having higher
>>>>     metric
>>>>     > value and then fall back to the lower.
>>>>
>>>>     I'm not seeing us with an interface to adjust the metric for an
>>>>     interface.  I think we need to add that at the same time?
>>>>
>>> Not now, As per my testing if metric value is not defined and both
>>> the routes
>>>
>>> having same metric then kernel tries one after other. We can bring
>>> the metric
>>>
>>> later.
>>>
>>>>       Otherwise, I
>>>>     don't think we have a way to specify which interface
>>>>     outside-the-subnet
>>>>     should go (vs today we can set the default-gateway to the desired
>>>>     interface's gateway).
>>>>     -- 
>>>>     Patrick Williams
>>>>
>>
>> -- 
>> Johnathan Mantey
>> Senior Software Engineer
>> *azad te**chnology partners*
>> Contributing to Technology Innovation since 1992
>> Phone: (503) 712-6764
>> Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>
>
> Ratan
>

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>


[-- Attachment #1.1.2: Type: text/html, Size: 9046 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Ratan Gupta]

[-- Attachment #1: Type: text/plain, Size: 4870 bytes --]


On 4/29/20 12:19 AM, Johnathan Mantey wrote:
> I think I understand most of your plan.
>
> What does it mean to "use the default metric"?

https://support.microsoft.com/en-in/help/299540/an-explanation-of-the-automatic-metric-feature-for-ipv4-routes

Instead of user level application assigns the metric value, Let the 
kernel decide which gateway should be used.

I understand that there should be a way through which we should be able 
to change the metric value  but that can be implemented later with some 
thing like route management where user should be able to add 
HostRoutes/Network routes etc, chnaging the values of the routes etc.

>
> On 4/28/20 8:16 AM, Ratan Gupta wrote:
>>
>> Hi Jonathan,
>>
>> On 4/28/20 1:00 AM, Johnathan Mantey wrote:
>>> Ratan,
>>>
>>> Can you provide more info about how you plan to implement this?
>> I am not getting, Do you mean to say how to get the per interface 
>> gateway from system?  that would be through netlink, and during 
>> configuration we would use the systemd-networkd file configuration.
>>>
>>> The Redfish schemas already have a per IP addr Gateway entry, which 
>>> OBMC is ignoring. Is it better to modify the existing D-Bus schema 
>>> (i.e. versioning discussion), or just use what is presently in 
>>> existence?
>> Redfish schema are getting changed to accommodate this through the 
>> following issue https://github.com/DMTF/Redfish/issues/3609.
>>
>> The above issue(gateway/interface) exist only for the IPv4 redfish 
>> schema, IPv6 redfish is already having default gateway per interface
>>
>> http://redfish.dmtf.org/schemas/v1/EthernetInterface.v1_5_1.json#/definitions/EthernetInterface
>>
>>>
>>> That said, doing as you propose melds better with how IPMI (i.e Set 
>>> LAN Configuration Parameter 12) works.
>>>
>>> As Patrick pointed out, there's no IPMI or Redfish control over the 
>>> routing metric parameter. What are your thoughts on how to reconcile 
>>> applying a metric value?
>> We will go with default metric value as today as underlying stack 
>> tries for fallback gateway.
>>> I've long been desirous of getting Williams submission merged: 
>>> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/20799
>>> as it can be used to improve the collection of netlink stats.
>>>
>>> Redfish exacerbates this whole problem by allowing a collection of 
>>> static IP addresses (and a DHCP addr) per NIC, unlike IPMI which was 
>>> very one to one with IP addr/NIC.
>>>
>>> Thanks for investigating this and working to implement a solution.
>>>
>>> On 4/27/20 4:11 AM, Ratan Gupta wrote:
>>>>
>>>> Thanks Patrick, William for sharing the feedbacks
>>>>
>>>> I will start working on the changes.
>>>>
>>>>
>>>> On 4/24/20 11:15 PM, William Kennington wrote:
>>>>> Sounds good to me, we have needed this for a long time because the 
>>>>> current gateway configuration breaks our v6 stack with multiple NICs.
>>>>>
>>>>> On Fri, Apr 24, 2020 at 8:21 AM Patrick Williams 
>>>>> <patrick@stwcx.xyz <mailto:patrick@stwcx.xyz>> wrote:
>>>>>
>>>>>     On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
>>>>>     > ~~~~~~~~~~~~~
>>>>>     > Kernel IP routing table
>>>>>     > Destination     Gateway         Genmask Flags   MSS Window 
>>>>>     irtt
>>>>>     > Iface
>>>>>     > 0.0.0.0         19.168.2.1      0.0.0.0 UG        0
>>>>>     0          0
>>>>>     > eth0
>>>>>     > 0.0.0.0         10.10.10.1      0.0.0.0 UG        0
>>>>>     0          0
>>>>>     > eth1
>>>>>     > ~~~~~~~~~~~~~~
>>>>>     >
>>>>>     > Kernel will first try using the default gateway having
>>>>>     higher metric
>>>>>     > value and then fall back to the lower.
>>>>>
>>>>>     I'm not seeing us with an interface to adjust the metric for an
>>>>>     interface.  I think we need to add that at the same time?
>>>>>
>>>> Not now, As per my testing if metric value is not defined and both 
>>>> the routes
>>>>
>>>> having same metric then kernel tries one after other. We can bring 
>>>> the metric
>>>>
>>>> later.
>>>>
>>>>>       Otherwise, I
>>>>>     don't think we have a way to specify which interface
>>>>>     outside-the-subnet
>>>>>     should go (vs today we can set the default-gateway to the desired
>>>>>     interface's gateway).
>>>>>     -- 
>>>>>     Patrick Williams
>>>>>
>>>
>>> -- 
>>> Johnathan Mantey
>>> Senior Software Engineer
>>> *azad te**chnology partners*
>>> Contributing to Technology Innovation since 1992
>>> Phone: (503) 712-6764
>>> Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>
>>
>> Ratan
>>
>
> -- 
> Johnathan Mantey
> Senior Software Engineer
> *azad te**chnology partners*
> Contributing to Technology Innovation since 1992
> Phone: (503) 712-6764
> Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>
>

[-- Attachment #2: Type: text/html, Size: 10484 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Johnathan Mantey]

[-- Attachment #1.1.1: Type: text/plain, Size: 6054 bytes --]

Ratan,

We're not using Windows. What is the Linux kernel/systemd going to do
for the Metric? So far the only value that has been applied to the
Metric is "0" (assigned by ??). As the Microsoft article points out,
using a low value Metric for "private" nets is to be avoided.
Our DHCP servers are supplying a Metric value of 1024. I'm sure other
network managers use different values.
I don't know how to solve this issue. Especially for the IPMI use case
of assigning a static address. IPMI doesn't have the ability to assign
metric values. At least with Redfish we can have that feature added.

On 5/4/20 6:38 AM, Ratan Gupta wrote:
>
>
> On 4/29/20 12:19 AM, Johnathan Mantey wrote:
>> I think I understand most of your plan.
>>
>> What does it mean to "use the default metric"?
>
> https://support.microsoft.com/en-in/help/299540/an-explanation-of-the-automatic-metric-feature-for-ipv4-routes
>
> Instead of user level application assigns the metric value, Let the
> kernel decide which gateway should be used.
>
> I understand that there should be a way through which we should be
> able to change the metric value  but that can be implemented later
> with some thing like route management where user should be able to add
> HostRoutes/Network routes etc, chnaging the values of the routes etc.
>
>>
>> On 4/28/20 8:16 AM, Ratan Gupta wrote:
>>>
>>> Hi Jonathan,
>>>
>>> On 4/28/20 1:00 AM, Johnathan Mantey wrote:
>>>> Ratan,
>>>>
>>>> Can you provide more info about how you plan to implement this?
>>> I am not getting, Do you mean to say how to get the per interface
>>> gateway from system?  that would be through netlink, and during
>>> configuration we would use the systemd-networkd file configuration.
>>>>
>>>> The Redfish schemas already have a per IP addr Gateway entry, which
>>>> OBMC is ignoring. Is it better to modify the existing D-Bus schema
>>>> (i.e. versioning discussion), or just use what is presently in
>>>> existence?
>>> Redfish schema are getting changed to accommodate this through the
>>> following issue https://github.com/DMTF/Redfish/issues/3609.
>>>
>>> The above issue(gateway/interface) exist only for the IPv4 redfish
>>> schema, IPv6 redfish is already having default gateway per interface
>>>
>>> http://redfish.dmtf.org/schemas/v1/EthernetInterface.v1_5_1.json#/definitions/EthernetInterface
>>>
>>>>
>>>> That said, doing as you propose melds better with how IPMI (i.e Set
>>>> LAN Configuration Parameter 12) works.
>>>>
>>>> As Patrick pointed out, there's no IPMI or Redfish control over the
>>>> routing metric parameter. What are your thoughts on how to
>>>> reconcile applying a metric value?
>>> We will go with default metric value as today as underlying stack
>>> tries for fallback gateway.
>>>> I've long been desirous of getting Williams submission merged:
>>>> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/20799
>>>> as it can be used to improve the collection of netlink stats.
>>>>
>>>> Redfish exacerbates this whole problem by allowing a collection of
>>>> static IP addresses (and a DHCP addr) per NIC, unlike IPMI which
>>>> was very one to one with IP addr/NIC.
>>>>
>>>> Thanks for investigating this and working to implement a solution.
>>>>
>>>> On 4/27/20 4:11 AM, Ratan Gupta wrote:
>>>>>
>>>>> Thanks Patrick, William for sharing the feedbacks
>>>>>
>>>>> I will start working on the changes.
>>>>>
>>>>>
>>>>> On 4/24/20 11:15 PM, William Kennington wrote:
>>>>>> Sounds good to me, we have needed this for a long time because
>>>>>> the current gateway configuration breaks our v6 stack with
>>>>>> multiple NICs.
>>>>>>
>>>>>> On Fri, Apr 24, 2020 at 8:21 AM Patrick Williams
>>>>>> <patrick@stwcx.xyz <mailto:patrick@stwcx.xyz>> wrote:
>>>>>>
>>>>>>     On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
>>>>>>     > ~~~~~~~~~~~~~
>>>>>>     > Kernel IP routing table
>>>>>>     > Destination     Gateway         Genmask         Flags   MSS
>>>>>>     Window  irtt
>>>>>>     > Iface
>>>>>>     > 0.0.0.0         19.168.2.1      0.0.0.0         UG        0
>>>>>>     0          0
>>>>>>     > eth0
>>>>>>     > 0.0.0.0         10.10.10.1      0.0.0.0         UG        0
>>>>>>     0          0
>>>>>>     > eth1
>>>>>>     > ~~~~~~~~~~~~~~
>>>>>>     >
>>>>>>     > Kernel will first try using the default gateway having
>>>>>>     higher metric
>>>>>>     > value and then fall back to the lower.
>>>>>>
>>>>>>     I'm not seeing us with an interface to adjust the metric for an
>>>>>>     interface.  I think we need to add that at the same time?
>>>>>>
>>>>> Not now, As per my testing if metric value is not defined and both
>>>>> the routes
>>>>>
>>>>> having same metric then kernel tries one after other. We can bring
>>>>> the metric
>>>>>
>>>>> later.
>>>>>
>>>>>>       Otherwise, I
>>>>>>     don't think we have a way to specify which interface
>>>>>>     outside-the-subnet
>>>>>>     should go (vs today we can set the default-gateway to the desired
>>>>>>     interface's gateway).
>>>>>>     -- 
>>>>>>     Patrick Williams
>>>>>>
>>>>
>>>> -- 
>>>> Johnathan Mantey
>>>> Senior Software Engineer
>>>> *azad te**chnology partners*
>>>> Contributing to Technology Innovation since 1992
>>>> Phone: (503) 712-6764
>>>> Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>
>>>
>>> Ratan
>>>
>>
>> -- 
>> Johnathan Mantey
>> Senior Software Engineer
>> *azad te**chnology partners*
>> Contributing to Technology Innovation since 1992
>> Phone: (503) 712-6764
>> Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>
>>

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>


[-- Attachment #1.1.2: Type: text/html, Size: 12672 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

On Mon, May 04, 2020 at 09:14:05AM -0700, Johnathan Mantey wrote:
> Ratan,
> 
> We're not using Windows. What is the Linux kernel/systemd going to do
> for the Metric? So far the only value that has been applied to the
> Metric is "0" (assigned by ??). As the Microsoft article points out,
> using a low value Metric for "private" nets is to be avoided.
> Our DHCP servers are supplying a Metric value of 1024. I'm sure other
> network managers use different values.
> I don't know how to solve this issue. Especially for the IPMI use case
> of assigning a static address. IPMI doesn't have the ability to assign
> metric values. At least with Redfish we can have that feature added.

Lower metric values have priority (0 is most preferred).  By default
systemd-networkd sets DHCP routes to 1024 per [1].  You can set an 
alternative RouteMetric in a systemd-networkd network file to set up
a different preference.

I do this to automatically tether my phone to my computer when I
plug it in directly and prefer the network routes provided by the phone
by setting RouteMetric to 512:

```
$ cat /etc/systemd/network/30-iphone.network
[Match]
Name=iphone0

[Network]
DHCP=yes

[DHCPv4]
RouteMetric=512
```

[1]: https://www.freedesktop.org/software/systemd/man/systemd.network.html#UseRoutes=
[2]: https://www.freedesktop.org/software/systemd/man/systemd.network.html#RouteMetric=

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Ratan Gupta]

[-- Attachment #1: Type: text/plain, Size: 7047 bytes --]

Jonathan,


On 5/4/20 9:44 PM, Johnathan Mantey wrote:
> Ratan,
>
> We're not using Windows. What is the Linux kernel/systemd going to do 
> for the Metric? So far the only value that has been applied to the 
> Metric is "0" (assigned by ??).
I was trying to explain your query "What does it mean to "use the 
default metric"?" I am hoping the Idea would be same to understand the 
behavior of default metric value.
In OpenBmc systemd assigns the default metric value for static 
routes/DHCP routes(if the DHCP server is not providing the metric value)

Default Metric means what systemd-networkd assigns if the metric is not 
given by the user(in case of static)/DHCP server. Static Routes gets 
added as default metric value as 0 by systemd-networkd.
> As the Microsoft article points out, using a low value Metric for 
> "private" nets is to be avoided.
> Our DHCP servers are supplying a Metric value of 1024. I'm sure other 
> network managers use different values.

I am hoping that this behavior in your case is coming through 
systemd-netwokd, I also observed this.
https://github.com/systemd/systemd/blob/c51c6f2f574e8c107994057c13a6506d93966a23/src/network/networkd-dhcp-common.h#L8

> I don't know how to solve this issue. Especially for the IPMI use case 
> of assigning a static address. IPMI doesn't have the ability to assign 
> metric values. At least with Redfish we can have that feature added.
As I mentioned before that we can come up with route management feature 
where user can add/delete/update route(i.e) user can change the metric 
value also for a route.
>
> On 5/4/20 6:38 AM, Ratan Gupta wrote:
>>
>>
>> On 4/29/20 12:19 AM, Johnathan Mantey wrote:
>>> I think I understand most of your plan.
>>>
>>> What does it mean to "use the default metric"?
>>
>> https://support.microsoft.com/en-in/help/299540/an-explanation-of-the-automatic-metric-feature-for-ipv4-routes
>>
>> Instead of user level application assigns the metric value, Let the 
>> kernel decide which gateway should be used.
>>
>> I understand that there should be a way through which we should be 
>> able to change the metric value  but that can be implemented later 
>> with some thing like route management where user should be able to 
>> add HostRoutes/Network routes etc, chnaging the values of the routes 
>> etc.
>>
>>>
>>> On 4/28/20 8:16 AM, Ratan Gupta wrote:
>>>>
>>>> Hi Jonathan,
>>>>
>>>> On 4/28/20 1:00 AM, Johnathan Mantey wrote:
>>>>> Ratan,
>>>>>
>>>>> Can you provide more info about how you plan to implement this?
>>>> I am not getting, Do you mean to say how to get the per interface 
>>>> gateway from system?  that would be through netlink, and during 
>>>> configuration we would use the systemd-networkd file configuration.
>>>>>
>>>>> The Redfish schemas already have a per IP addr Gateway entry, 
>>>>> which OBMC is ignoring. Is it better to modify the existing D-Bus 
>>>>> schema (i.e. versioning discussion), or just use what is presently 
>>>>> in existence?
>>>> Redfish schema are getting changed to accommodate this through the 
>>>> following issue https://github.com/DMTF/Redfish/issues/3609.
>>>>
>>>> The above issue(gateway/interface) exist only for the IPv4 redfish 
>>>> schema, IPv6 redfish is already having default gateway per interface
>>>>
>>>> http://redfish.dmtf.org/schemas/v1/EthernetInterface.v1_5_1.json#/definitions/EthernetInterface
>>>>
>>>>>
>>>>> That said, doing as you propose melds better with how IPMI (i.e 
>>>>> Set LAN Configuration Parameter 12) works.
>>>>>
>>>>> As Patrick pointed out, there's no IPMI or Redfish control over 
>>>>> the routing metric parameter. What are your thoughts on how to 
>>>>> reconcile applying a metric value?
>>>> We will go with default metric value as today as underlying stack 
>>>> tries for fallback gateway.
>>>>> I've long been desirous of getting Williams submission merged: 
>>>>> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/20799
>>>>> as it can be used to improve the collection of netlink stats.
>>>>>
>>>>> Redfish exacerbates this whole problem by allowing a collection of 
>>>>> static IP addresses (and a DHCP addr) per NIC, unlike IPMI which 
>>>>> was very one to one with IP addr/NIC.
>>>>>
>>>>> Thanks for investigating this and working to implement a solution.
>>>>>
>>>>> On 4/27/20 4:11 AM, Ratan Gupta wrote:
>>>>>>
>>>>>> Thanks Patrick, William for sharing the feedbacks
>>>>>>
>>>>>> I will start working on the changes.
>>>>>>
>>>>>>
>>>>>> On 4/24/20 11:15 PM, William Kennington wrote:
>>>>>>> Sounds good to me, we have needed this for a long time because 
>>>>>>> the current gateway configuration breaks our v6 stack with 
>>>>>>> multiple NICs.
>>>>>>>
>>>>>>> On Fri, Apr 24, 2020 at 8:21 AM Patrick Williams 
>>>>>>> <patrick@stwcx.xyz <mailto:patrick@stwcx.xyz>> wrote:
>>>>>>>
>>>>>>>     On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
>>>>>>>     > ~~~~~~~~~~~~~
>>>>>>>     > Kernel IP routing table
>>>>>>>     > Destination     Gateway Genmask         Flags   MSS
>>>>>>>     Window  irtt
>>>>>>>     > Iface
>>>>>>>     > 0.0.0.0         19.168.2.1 0.0.0.0         UG        0
>>>>>>>     0          0
>>>>>>>     > eth0
>>>>>>>     > 0.0.0.0         10.10.10.1 0.0.0.0         UG        0
>>>>>>>     0          0
>>>>>>>     > eth1
>>>>>>>     > ~~~~~~~~~~~~~~
>>>>>>>     >
>>>>>>>     > Kernel will first try using the default gateway having
>>>>>>>     higher metric
>>>>>>>     > value and then fall back to the lower.
>>>>>>>
>>>>>>>     I'm not seeing us with an interface to adjust the metric for an
>>>>>>>     interface.  I think we need to add that at the same time?
>>>>>>>
>>>>>> Not now, As per my testing if metric value is not defined and 
>>>>>> both the routes
>>>>>>
>>>>>> having same metric then kernel tries one after other. We can 
>>>>>> bring the metric
>>>>>>
>>>>>> later.
>>>>>>
>>>>>>>       Otherwise, I
>>>>>>>     don't think we have a way to specify which interface
>>>>>>>     outside-the-subnet
>>>>>>>     should go (vs today we can set the default-gateway to the
>>>>>>>     desired
>>>>>>>     interface's gateway).
>>>>>>>     -- 
>>>>>>>     Patrick Williams
>>>>>>>
>>>>>
>>>>> -- 
>>>>> Johnathan Mantey
>>>>> Senior Software Engineer
>>>>> *azad te**chnology partners*
>>>>> Contributing to Technology Innovation since 1992
>>>>> Phone: (503) 712-6764
>>>>> Email: johnathanx.mantey@intel.com 
>>>>> <mailto:johnathanx.mantey@intel.com>
>>>>
>>>> Ratan
>>>>
>>>
>>> -- 
>>> Johnathan Mantey
>>> Senior Software Engineer
>>> *azad te**chnology partners*
>>> Contributing to Technology Innovation since 1992
>>> Phone: (503) 712-6764
>>> Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>
>>>
>
> -- 
> Johnathan Mantey
> Senior Software Engineer
> *azad te**chnology partners*
> Contributing to Technology Innovation since 1992
> Phone: (503) 712-6764
> Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>
>

[-- Attachment #2: Type: text/html, Size: 14877 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Johnathan Mantey]

[-- Attachment #1.1.1: Type: text/plain, Size: 7730 bytes --]

Ratan,
Thanks for clarifying.

On 5/7/20 11:35 PM, Ratan Gupta wrote:
>
> Jonathan,
>
>
> On 5/4/20 9:44 PM, Johnathan Mantey wrote:
>> Ratan,
>>
>> We're not using Windows. What is the Linux kernel/systemd going to do
>> for the Metric? So far the only value that has been applied to the
>> Metric is "0" (assigned by ??).
> I was trying to explain your query "What does it mean to "use the
> default metric"?" I am hoping the Idea would be same to understand the
> behavior of default metric value.
> In OpenBmc systemd assigns the default metric value for static
> routes/DHCP routes(if the DHCP server is not providing the metric value)
>
> Default Metric means what systemd-networkd assigns if the metric is
> not given by the user(in case of static)/DHCP server. Static Routes
> gets added as default metric value as 0 by systemd-networkd.
>> As the Microsoft article points out, using a low value Metric for
>> "private" nets is to be avoided.
>> Our DHCP servers are supplying a Metric value of 1024. I'm sure other
>> network managers use different values.
>
> I am hoping that this behavior in your case is coming through
> systemd-netwokd, I also observed this.
> https://github.com/systemd/systemd/blob/c51c6f2f574e8c107994057c13a6506d93966a23/src/network/networkd-dhcp-common.h#L8
>
>> I don't know how to solve this issue. Especially for the IPMI use
>> case of assigning a static address. IPMI doesn't have the ability to
>> assign metric values. At least with Redfish we can have that feature
>> added.
> As I mentioned before that we can come up with route management
> feature where user can add/delete/update route(i.e) user can change
> the metric value also for a route.
>>
>> On 5/4/20 6:38 AM, Ratan Gupta wrote:
>>>
>>>
>>> On 4/29/20 12:19 AM, Johnathan Mantey wrote:
>>>> I think I understand most of your plan.
>>>>
>>>> What does it mean to "use the default metric"?
>>>
>>> https://support.microsoft.com/en-in/help/299540/an-explanation-of-the-automatic-metric-feature-for-ipv4-routes
>>>
>>> Instead of user level application assigns the metric value, Let the
>>> kernel decide which gateway should be used.
>>>
>>> I understand that there should be a way through which we should be
>>> able to change the metric value  but that can be implemented later
>>> with some thing like route management where user should be able to
>>> add HostRoutes/Network routes etc, chnaging the values of the routes
>>> etc.
>>>
>>>>
>>>> On 4/28/20 8:16 AM, Ratan Gupta wrote:
>>>>>
>>>>> Hi Jonathan,
>>>>>
>>>>> On 4/28/20 1:00 AM, Johnathan Mantey wrote:
>>>>>> Ratan,
>>>>>>
>>>>>> Can you provide more info about how you plan to implement this?
>>>>> I am not getting, Do you mean to say how to get the per interface
>>>>> gateway from system?  that would be through netlink, and during
>>>>> configuration we would use the systemd-networkd file configuration.
>>>>>>
>>>>>> The Redfish schemas already have a per IP addr Gateway entry,
>>>>>> which OBMC is ignoring. Is it better to modify the existing D-Bus
>>>>>> schema (i.e. versioning discussion), or just use what is
>>>>>> presently in existence?
>>>>> Redfish schema are getting changed to accommodate this through the
>>>>> following issue https://github.com/DMTF/Redfish/issues/3609.
>>>>>
>>>>> The above issue(gateway/interface) exist only for the IPv4 redfish
>>>>> schema, IPv6 redfish is already having default gateway per interface
>>>>>
>>>>> http://redfish.dmtf.org/schemas/v1/EthernetInterface.v1_5_1.json#/definitions/EthernetInterface
>>>>>
>>>>>>
>>>>>> That said, doing as you propose melds better with how IPMI (i.e
>>>>>> Set LAN Configuration Parameter 12) works.
>>>>>>
>>>>>> As Patrick pointed out, there's no IPMI or Redfish control over
>>>>>> the routing metric parameter. What are your thoughts on how to
>>>>>> reconcile applying a metric value?
>>>>> We will go with default metric value as today as underlying stack
>>>>> tries for fallback gateway.
>>>>>> I've long been desirous of getting Williams submission merged:
>>>>>> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-networkd/+/20799
>>>>>> as it can be used to improve the collection of netlink stats.
>>>>>>
>>>>>> Redfish exacerbates this whole problem by allowing a collection
>>>>>> of static IP addresses (and a DHCP addr) per NIC, unlike IPMI
>>>>>> which was very one to one with IP addr/NIC.
>>>>>>
>>>>>> Thanks for investigating this and working to implement a solution.
>>>>>>
>>>>>> On 4/27/20 4:11 AM, Ratan Gupta wrote:
>>>>>>>
>>>>>>> Thanks Patrick, William for sharing the feedbacks
>>>>>>>
>>>>>>> I will start working on the changes.
>>>>>>>
>>>>>>>
>>>>>>> On 4/24/20 11:15 PM, William Kennington wrote:
>>>>>>>> Sounds good to me, we have needed this for a long time because
>>>>>>>> the current gateway configuration breaks our v6 stack with
>>>>>>>> multiple NICs.
>>>>>>>>
>>>>>>>> On Fri, Apr 24, 2020 at 8:21 AM Patrick Williams
>>>>>>>> <patrick@stwcx.xyz <mailto:patrick@stwcx.xyz>> wrote:
>>>>>>>>
>>>>>>>>     On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
>>>>>>>>     > ~~~~~~~~~~~~~
>>>>>>>>     > Kernel IP routing table
>>>>>>>>     > Destination     Gateway         Genmask         Flags  
>>>>>>>>     MSS Window  irtt
>>>>>>>>     > Iface
>>>>>>>>     > 0.0.0.0         19.168.2.1      0.0.0.0         UG       
>>>>>>>>     0 0          0
>>>>>>>>     > eth0
>>>>>>>>     > 0.0.0.0         10.10.10.1      0.0.0.0         UG       
>>>>>>>>     0 0          0
>>>>>>>>     > eth1
>>>>>>>>     > ~~~~~~~~~~~~~~
>>>>>>>>     >
>>>>>>>>     > Kernel will first try using the default gateway having
>>>>>>>>     higher metric
>>>>>>>>     > value and then fall back to the lower.
>>>>>>>>
>>>>>>>>     I'm not seeing us with an interface to adjust the metric for an
>>>>>>>>     interface.  I think we need to add that at the same time?
>>>>>>>>
>>>>>>> Not now, As per my testing if metric value is not defined and
>>>>>>> both the routes
>>>>>>>
>>>>>>> having same metric then kernel tries one after other. We can
>>>>>>> bring the metric
>>>>>>>
>>>>>>> later.
>>>>>>>
>>>>>>>>       Otherwise, I
>>>>>>>>     don't think we have a way to specify which interface
>>>>>>>>     outside-the-subnet
>>>>>>>>     should go (vs today we can set the default-gateway to the
>>>>>>>>     desired
>>>>>>>>     interface's gateway).
>>>>>>>>     -- 
>>>>>>>>     Patrick Williams
>>>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Johnathan Mantey
>>>>>> Senior Software Engineer
>>>>>> *azad te**chnology partners*
>>>>>> Contributing to Technology Innovation since 1992
>>>>>> Phone: (503) 712-6764
>>>>>> Email: johnathanx.mantey@intel.com
>>>>>> <mailto:johnathanx.mantey@intel.com>
>>>>>
>>>>> Ratan
>>>>>
>>>>
>>>> -- 
>>>> Johnathan Mantey
>>>> Senior Software Engineer
>>>> *azad te**chnology partners*
>>>> Contributing to Technology Innovation since 1992
>>>> Phone: (503) 712-6764
>>>> Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>
>>>>
>>
>> -- 
>> Johnathan Mantey
>> Senior Software Engineer
>> *azad te**chnology partners*
>> Contributing to Technology Innovation since 1992
>> Phone: (503) 712-6764
>> Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>
>>

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>


[-- Attachment #1.1.2: Type: text/html, Size: 16724 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

On Mon, Apr 27, 2020 at 04:41:15PM +0530, Ratan Gupta wrote:
> >     I'm not seeing us with an interface to adjust the metric for an
> >     interface.  I think we need to add that at the same time?
> >
> Not now, As per my testing if metric value is not defined and both the 
> routes
> 
> having same metric then kernel tries one after other. We can bring the 
> metric
> 
> later.
> 
> >     Otherwise, I
> >     don't think we have a way to specify which interface
> >     outside-the-subnet
> >     should go (vs today we can set the default-gateway to the desired
> >     interface's gateway).
> >     -- 

Hi Ratan,

I just want to reiterate that by not doing metric I think you end up with
a slight regression in functionality.  I'm not sure if anyone cares though.

Today, if I have two NICs attached, I can set the gateway to NIC1's
gateway and my traffic will go out NIC1 (if it isn't on NIC0's subnet).
Without the metric, we have no way to specify this behavior and the
kernel will likely default to NIC0.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Andrew Jeffery]

On Mon, 27 Apr 2020, at 20:41, Ratan Gupta wrote:
> Thanks Patrick, William for sharing the feedbacks
> 
> I will start working on the changes.
> 
> 
> 
> On 4/24/20 11:15 PM, William Kennington wrote:
> > Sounds good to me, we have needed this for a long time because the current gateway configuration breaks our v6 stack with multiple NICs.
> > 
> > On Fri, Apr 24, 2020 at 8:21 AM Patrick Williams <patrick@stwcx.xyz> wrote:
> >> On Fri, Apr 24, 2020 at 08:36:26PM +0530, Ratan Gupta wrote:
> >>  > ~~~~~~~~~~~~~
> >>  > Kernel IP routing table
> >>  > Destination Gateway Genmask Flags MSS Window irtt 
> >>  > Iface
> >>  > 0.0.0.0 19.168.2.1 0.0.0.0 UG 0 0 0 
> >>  > eth0
> >>  > 0.0.0.0 10.10.10.1 0.0.0.0 UG 0 0 0 
> >>  > eth1
> >>  > ~~~~~~~~~~~~~~
> >>  > 
> >>  > Kernel will first try using the default gateway having higher metric 
> >>  > value and then fall back to the lower.
> >> 
> >>  I'm not seeing us with an interface to adjust the metric for an
> >>  interface. I think we need to add that at the same time? 
> Not now, As per my testing if metric value is not defined and both the routes
> 
> having same metric then kernel tries one after other. We can bring the metric
> 
> later.

This sounds like implementation-defined behaviour. Is that the case? If so it's 
not something we should be relying on.

Andrew

Re: Default Gateway for a system v/s Default gateway per Interface

[Johnathan Mantey]

[-- Attachment #1.1.1: Type: text/plain, Size: 2094 bytes --]

Ratan,

In the email trail for this topic on 27 April you indicated you were
going to start work on the per NIC gateway.
What is the current state of implementing this feature?

On 4/24/20 8:06 AM, Ratan Gupta wrote:
> Hi All,
>
> Currently, OpenBMC stack allows a single default gateway for the
> system. Latest kernel allows to configure multiple default gateways.
>
> Eg: In a system with two interfaces eth0 and eth1,
>
> eth0 configured with static address and having gateway(192.168.2.1)
> eth1 configured with DHCP and gets Gateway from DHCP server (10.10.10.1)
> ~~~~~~~~~~~~~
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags   MSS Window 
> irtt Iface
> 0.0.0.0         19.168.2.1      0.0.0.0         UG        0 0         
> 0 eth0
> 0.0.0.0         10.10.10.1      0.0.0.0         UG        0 0         
> 0 eth1
> ~~~~~~~~~~~~~~
>
> Kernel will first try using the default gateway having higher metric
> value and then fall back to the lower.
>
> More references:
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_the_default_gateway
>
> I'm proposing to make this change in the openBMC D-bus interfaces to
> tie the gateway property with the Ethernet interface schema instead of
> System configuration.
>
> Ethernet Interface Schema =>
>
> https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/EthernetInterface.interface.yaml
>
>
> System Configuration Schema =>
>
> https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/SystemConfiguration.interface.yaml
>
>
>
> Please let me know your suggestions.
>
> Regards
> Ratan Gupta
>

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>


[-- Attachment #1.1.2: Type: text/html, Size: 4030 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Default Gateway for a system v/s Default gateway per Interface

[Alexander Amelkin]

Please, Ratan, if you implement those per-interface gateways again, 
could you leave an option to have a single system-wide default gateway 
only?

I honestly don't see any use for multiple gateways without policy 
routing being implemented, and then I don't see any use for policy 
routing in a BMC (as opposed to a router). So I would like to keep us 
from unneeded questions from the customers, and so I would prefer to 
have only a single gateway, as well as single sets of DNS and NTP 
servers without having to patch across the whole OpenBMC codebase.

If there was a compile-time option (distro/machine feature?) to use 
either per-interface or system-wide gateway/DNS/NTP, that would be very 
nice.

Thank you.
Alexander.

P.S. Sorry for sending this twice, forgot to include the list the first 
time.

24.04.2020 18:06, Ratan Gupta пишет:
> Hi All,
>
> Currently, OpenBMC stack allows a single default gateway for the 
> system. Latest kernel allows to configure multiple default gateways.
>
> Eg: In a system with two interfaces eth0 and eth1,
>
> eth0 configured with static address and having gateway(192.168.2.1)
> eth1 configured with DHCP and gets Gateway from DHCP server (10.10.10.1)
> ~~~~~~~~~~~~~
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags   MSS Window  
> irtt Iface
> 0.0.0.0         19.168.2.1      0.0.0.0         UG        0 0          
> 0 eth0
> 0.0.0.0         10.10.10.1      0.0.0.0         UG        0 0          
> 0 eth1
> ~~~~~~~~~~~~~~
>
> Kernel will first try using the default gateway having higher metric 
> value and then fall back to the lower.
>
> More references: 
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_the_default_gateway
>
> I'm proposing to make this change in the openBMC D-bus interfaces to 
> tie the gateway property with the Ethernet interface schema instead of 
> System configuration.
>
> Ethernet Interface Schema =>
>
> https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/EthernetInterface.interface.yaml 
>
>
> System Configuration Schema =>
>
> https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/SystemConfiguration.interface.yaml 
>
>
>
> Please let me know your suggestions.
>
> Regards
> Ratan Gupta
>

Re: Default Gateway for a system v/s Default gateway per Interface

[Ratan Gupta]

Hi Alexender,

I am not sure how it can be achieved through some compile time option(as 
the existing dbus interface requires changes. Might be version Dbus 
Interfaces which is currently not supported.

Phosphor-networkd uses systemd-networkd as backend which supports 
multiple default gateway as fall back gateway without implementing 
policy based routing(Both gateways are on the main routing table).

Ratan

On 6/20/20 4:10 AM, Alexander Amelkin wrote:
> Please, Ratan, if you implement those per-interface gateways again, 
> could you leave an option to have a single system-wide default gateway 
> only?
>
> I honestly don't see any use for multiple gateways without policy 
> routing being implemented, and then I don't see any use for policy 
> routing in a BMC (as opposed to a router). So I would like to keep us 
> from unneeded questions from the customers, and so I would prefer to 
> have only a single gateway, as well as single sets of DNS and NTP 
> servers without having to patch across the whole OpenBMC codebase.
>
> If there was a compile-time option (distro/machine feature?) to use 
> either per-interface or system-wide gateway/DNS/NTP, that would be 
> very nice.
>
> Thank you.
> Alexander.
>
> P.S. Sorry for sending this twice, forgot to include the list the 
> first time.
>
> 24.04.2020 18:06, Ratan Gupta пишет:
>> Hi All,
>>
>> Currently, OpenBMC stack allows a single default gateway for the 
>> system. Latest kernel allows to configure multiple default gateways.
>>
>> Eg: In a system with two interfaces eth0 and eth1,
>>
>> eth0 configured with static address and having gateway(192.168.2.1)
>> eth1 configured with DHCP and gets Gateway from DHCP server (10.10.10.1)
>> ~~~~~~~~~~~~~
>> Kernel IP routing table
>> Destination     Gateway         Genmask         Flags   MSS Window  
>> irtt Iface
>> 0.0.0.0         19.168.2.1      0.0.0.0         UG        0 
>> 0          0 eth0
>> 0.0.0.0         10.10.10.1      0.0.0.0         UG        0 
>> 0          0 eth1
>> ~~~~~~~~~~~~~~
>>
>> Kernel will first try using the default gateway having higher metric 
>> value and then fall back to the lower.
>>
>> More references: 
>> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_the_default_gateway
>>
>> I'm proposing to make this change in the openBMC D-bus interfaces to 
>> tie the gateway property with the Ethernet interface schema instead 
>> of System configuration.
>>
>> Ethernet Interface Schema =>
>>
>> https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/EthernetInterface.interface.yaml 
>>
>>
>> System Configuration Schema =>
>>
>> https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Network/SystemConfiguration.interface.yaml 
>>
>>
>>
>> Please let me know your suggestions.
>>
>> Regards
>> Ratan Gupta
>>

Re: Default Gateway for a system v/s Default gateway per Interface

[郁雷]

On Tue, Jun 23, 2020 at 5:11 PM Ratan Gupta <ratagupt@linux.vnet.ibm.com> wrote:
>
> Hi Alexender,
>
> I am not sure how it can be achieved through some compile time option(as
> the existing dbus interface requires changes. Might be version Dbus
> Interfaces which is currently not supported.
>
> Phosphor-networkd uses systemd-networkd as backend which supports
> multiple default gateway as fall back gateway without implementing
> policy based routing(Both gateways are on the main routing table).

What if we do NOT set DefaultGateway (in SystemConfiguration
interface), and set Gateway in each IP interface?
Will it work as interface-specific gateway per interface?

-- 
BRs,
Lei YU

Re: Default Gateway for a system v/s Default gateway per Interface

[Ratan Gupta]

On 6/23/20 4:26 PM, 郁雷 wrote:
> On Tue, Jun 23, 2020 at 5:11 PM Ratan Gupta <ratagupt@linux.vnet.ibm.com> wrote:
>> Hi Alexender,
>>
>> I am not sure how it can be achieved through some compile time option(as
>> the existing dbus interface requires changes. Might be version Dbus
>> Interfaces which is currently not supported.
>>
>> Phosphor-networkd uses systemd-networkd as backend which supports
>> multiple default gateway as fall back gateway without implementing
>> policy based routing(Both gateways are on the main routing table).
> What if we do NOT set DefaultGateway (in SystemConfiguration
> interface), and set Gateway in each IP interface?

Yes that is the plan, Default gateway property will be moved from

SystemConfiguration interface to EthernetInterface. Gateway on IP

interface is of no use today as we are not using it at all in the 
phosphor-networkd

we would be removing the gateway property from the IP address interface.

> Will it work as interface-specific gateway per interface?
>