PROBLEM: Issue with setting veth MAC address being unreliable.

PROBLEM: Issue with setting veth MAC address being unreliable.

[Jason Zhou]

[1.] One line summary of the problem:

Issue with setting veth address being unreliable.

[2.] Full description of the problem/report:

Hello!

We have been investigating a strange behavior within Apache Mesos
where after setting the MAC address on a veth device to the same
address as our eth0 MAC address, the change is sometimes not reflected
appropriately despite the ioctl call succeeding (~4% of the time in
our testing). Note that we also tried using libnl to set the MAC
address but the issue still persists.

Included below is the github link to the section where we set the veth
address, to clarify what we were trying to do. We first create the
veth pair [1] using a libnl function [2], then we set the veth device
MAC addresses to that of our host public interface (eth0) [3] using a
function called setMAC. Inside the setMAC [4] is where we are
observing the aforementioned issue with unreliable setting of veth
addresses..

This behavior was observed when re-fetching the MAC address on said
veth device after we made the function call to set its MAC address. We
have observed this issue on CentOS 9 only, but not on CentOS 7. We
have tried Linux kernels 5.15.147, 5.15.160 & 5.15.161 for CentOS 9,
CentOS 7 was using 5.10, but we also tried upgrading the Centos 7 host
to 5.15.160 but could not reproduce the bug.

We were re-fetching the addresses via the ioctl SIOCGIFHWADDR syscall
as well as via getifaddr (which appears to use netlink under the
covers), and, in problematic cases, both functions reported
discrepancies from the target MAC address we were initially setting
to. We also performed a fetch before we set the MAC addresses and
found that there are instances where getifaddr and ioctl results do
not match for our veth device *even before we perform any setting of
the MAC address*. It's also worth noting that after setting the MAC
address: there are no cases where ioctl or getifaddr come back with
the same MAC address as before we set the address. So, the set
operation always seems to have an effect.

Observed scenarios with incorrectly assigned MAC addresses:

(1) After setting the mac address: ioctl returns the correct MAC
address, but the results from getifaddr, returns an incorrect MAC
address (different from the original value before setting as well!)

(2) After setting the MAC address: both ioctl and getifaddr return the
same MAC address, but are both wrong (and different from the original
one!)

(3) There is a possibility that the MAC address we set ends up
overwritten by a garbage value *after* we have already updated the MAC
address, and checked that the MAC address was set correctly. Since
this error happens after this function has finished, we cannot log nor
detect it in the function where we set the MAC address because we have
not yet studied at what point this late overwriting of MAC address
occurs. It’s worth noting that this is the rarest scenario that we
have encountered, and we were only able to reproduce it in our testing
cluster machine, not in any of the production cluster machines.

[3.] Keywords:

networking, veth, kernel, MAC, netlink

[X.] Other notes, patches, fixes, workarounds:

Notes:

More specific kernel and environment information will be available on
request for security reasons, please let us know if you are interested
and we will be happy to provide you with the necessary information.

We have observed this behavior only on CentOS 9 systems at the moment,
CentOS 7 systems under various kernels do not seem to have the issue
(which is quite strange if this was purely a kernel bug).

We have tried kernels 5.15.147, 5.15.160, 5.15.161, all of these have
this issue on CentOS 9.

We have also tried rewriting our function for setting MAC address to
use libnl rather than ioctl to perform the MAC address setting, but it
did not eliminate the issue.

To work around this bug, we checked that the MAC address is set
correctly after the ioctl set call, and retry the address setting if
necessary. In our testing, this workaround appears to remedy scenarios
(1) and (2) above, but it does not address scenario (3).  You can see
it here:

https://github.com/apache/mesos/commit/8b202bbebdc89429ad82c6983aa1c514eb1b8d95

We would greatly appreciate any insights or guidance on this matter.
Please let me know if you need further information or if there are any
specific tests we should run to assist in diagnosing the issue. Again,
specific details for the production machines on which we encountered
this error can be provided upon request, so please let us know if
there is anything we can provide to help.

Thank you for your time and assistance.

Best regards,
Jason Zhou
Software Engineering Intern
jasonzhou@x.com

embedded links:
[1] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/slave/containerizer/mesos/isolators/network/port_mapping.cpp#L3599
[2] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/linux/routing/link/veth.cpp#L45
[3] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/slave/containerizer/mesos/isolators/network/port_mapping.cpp#L3628
[4] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/linux/routing/link/link.cpp#L283

Re: PROBLEM: Issue with setting veth MAC address being unreliable.

[Ido Schimmel]

On Wed, Jul 10, 2024 at 04:45:55PM -0400, Jason Zhou wrote:
> [1.] One line summary of the problem:
> 
> Issue with setting veth address being unreliable.
> 
> [2.] Full description of the problem/report:
> 
> Hello!
> 
> We have been investigating a strange behavior within Apache Mesos
> where after setting the MAC address on a veth device to the same
> address as our eth0 MAC address, the change is sometimes not reflected
> appropriately despite the ioctl call succeeding (~4% of the time in
> our testing). Note that we also tried using libnl to set the MAC
> address but the issue still persists.
> 
> Included below is the github link to the section where we set the veth
> address, to clarify what we were trying to do. We first create the
> veth pair [1] using a libnl function [2], then we set the veth device
> MAC addresses to that of our host public interface (eth0) [3] using a
> function called setMAC. Inside the setMAC [4] is where we are
> observing the aforementioned issue with unreliable setting of veth
> addresses..
> 
> This behavior was observed when re-fetching the MAC address on said
> veth device after we made the function call to set its MAC address. We
> have observed this issue on CentOS 9 only, but not on CentOS 7. We
> have tried Linux kernels 5.15.147, 5.15.160 & 5.15.161 for CentOS 9,
> CentOS 7 was using 5.10, but we also tried upgrading the Centos 7 host
> to 5.15.160 but could not reproduce the bug.

This suggests that the change in behavior is due to a change in user
space and an obvious suspect is systemd / udev. AFAICT, CentOS 7 is
using systemd 219 whereas CentOS 9 is using systemd 252. In version 242
systemd started setting a persistent MAC address on virtual interfaces.
See:

https://github.com/Mellanox/mlxsw/wiki/Persistent-Configuration#required-changes-to-macaddresspolicy

If that is the issue, then you can either change MACAddressPolicy or
modify your code to create the veth pair with the correct MAC address
from the start.

As I understand it, a possible explanation for the race is that your
program is racing with udev. Udev receives an event about a new device
and reads "addr_assign_type" from sysfs. If your program changed the MAC
address before udev read the file, then udev will read 3 (NET_ADDR_SET)
and will not try to change the MAC address. Otherwise, both your program
and udev will try to change the MAC address.

If it is not udev that is changing the MAC address, then you can run the
following bpftrace script in a different terminal and check which
processes try to change the address:

bpftrace -e 'k:dev_set_mac_address_user { @[comm] = count(); }'

It is only a theory. Actual issue might be entirely different.

> 
> We were re-fetching the addresses via the ioctl SIOCGIFHWADDR syscall
> as well as via getifaddr (which appears to use netlink under the
> covers), and, in problematic cases, both functions reported
> discrepancies from the target MAC address we were initially setting
> to. We also performed a fetch before we set the MAC addresses and
> found that there are instances where getifaddr and ioctl results do
> not match for our veth device *even before we perform any setting of
> the MAC address*. It's also worth noting that after setting the MAC
> address: there are no cases where ioctl or getifaddr come back with
> the same MAC address as before we set the address. So, the set
> operation always seems to have an effect.
> 
> Observed scenarios with incorrectly assigned MAC addresses:
> 
> (1) After setting the mac address: ioctl returns the correct MAC
> address, but the results from getifaddr, returns an incorrect MAC
> address (different from the original value before setting as well!)
> 
> (2) After setting the MAC address: both ioctl and getifaddr return the
> same MAC address, but are both wrong (and different from the original
> one!)
> 
> (3) There is a possibility that the MAC address we set ends up
> overwritten by a garbage value *after* we have already updated the MAC
> address, and checked that the MAC address was set correctly. Since
> this error happens after this function has finished, we cannot log nor
> detect it in the function where we set the MAC address because we have
> not yet studied at what point this late overwriting of MAC address
> occurs. It’s worth noting that this is the rarest scenario that we
> have encountered, and we were only able to reproduce it in our testing
> cluster machine, not in any of the production cluster machines.
> 
> [3.] Keywords:
> 
> networking, veth, kernel, MAC, netlink
> 
> [X.] Other notes, patches, fixes, workarounds:
> 
> Notes:
> 
> More specific kernel and environment information will be available on
> request for security reasons, please let us know if you are interested
> and we will be happy to provide you with the necessary information.
> 
> We have observed this behavior only on CentOS 9 systems at the moment,
> CentOS 7 systems under various kernels do not seem to have the issue
> (which is quite strange if this was purely a kernel bug).
> 
> We have tried kernels 5.15.147, 5.15.160, 5.15.161, all of these have
> this issue on CentOS 9.
> 
> We have also tried rewriting our function for setting MAC address to
> use libnl rather than ioctl to perform the MAC address setting, but it
> did not eliminate the issue.
> 
> To work around this bug, we checked that the MAC address is set
> correctly after the ioctl set call, and retry the address setting if
> necessary. In our testing, this workaround appears to remedy scenarios
> (1) and (2) above, but it does not address scenario (3).  You can see
> it here:
> 
> https://github.com/apache/mesos/commit/8b202bbebdc89429ad82c6983aa1c514eb1b8d95
> 
> We would greatly appreciate any insights or guidance on this matter.
> Please let me know if you need further information or if there are any
> specific tests we should run to assist in diagnosing the issue. Again,
> specific details for the production machines on which we encountered
> this error can be provided upon request, so please let us know if
> there is anything we can provide to help.
> 
> Thank you for your time and assistance.
> 
> Best regards,
> Jason Zhou
> Software Engineering Intern
> jasonzhou@x.com
> 
> embedded links:
> [1] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/slave/containerizer/mesos/isolators/network/port_mapping.cpp#L3599
> [2] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/linux/routing/link/veth.cpp#L45
> [3] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/slave/containerizer/mesos/isolators/network/port_mapping.cpp#L3628
> [4] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/linux/routing/link/link.cpp#L283
> 

Re: PROBLEM: Issue with setting veth MAC address being unreliable.

[Jason Zhou]

Apologies for the late reply,

> When you say 5.15.161, do you mean true 5.15.161, or is this CentOS
> version of 5.15.161 with 1000s of patches on top? Did you download the
> sources from www.kernel.org and build it yourself?

For Andrew's question, yes, we do use the mainline 5.15 LTS kernel
which we built ourselves from the www.kernel.org sources.

On Thu, Jul 11, 2024 at 9:18 AM Ido Schimmel <idosch@idosch.org> wrote:
>
> On Wed, Jul 10, 2024 at 04:45:55PM -0400, Jason Zhou wrote:
> > [1.] One line summary of the problem:
> >
> > Issue with setting veth address being unreliable.
> >
> > [2.] Full description of the problem/report:
> >
> > Hello!
> >
> > We have been investigating a strange behavior within Apache Mesos
> > where after setting the MAC address on a veth device to the same
> > address as our eth0 MAC address, the change is sometimes not reflected
> > appropriately despite the ioctl call succeeding (~4% of the time in
> > our testing). Note that we also tried using libnl to set the MAC
> > address but the issue still persists.
> >
> > Included below is the github link to the section where we set the veth
> > address, to clarify what we were trying to do. We first create the
> > veth pair [1] using a libnl function [2], then we set the veth device
> > MAC addresses to that of our host public interface (eth0) [3] using a
> > function called setMAC. Inside the setMAC [4] is where we are
> > observing the aforementioned issue with unreliable setting of veth
> > addresses..
> >
> > This behavior was observed when re-fetching the MAC address on said
> > veth device after we made the function call to set its MAC address. We
> > have observed this issue on CentOS 9 only, but not on CentOS 7. We
> > have tried Linux kernels 5.15.147, 5.15.160 & 5.15.161 for CentOS 9,
> > CentOS 7 was using 5.10, but we also tried upgrading the Centos 7 host
> > to 5.15.160 but could not reproduce the bug.
>
> This suggests that the change in behavior is due to a change in user
> space and an obvious suspect is systemd / udev. AFAICT, CentOS 7 is
> using systemd 219 whereas CentOS 9 is using systemd 252. In version 242
> systemd started setting a persistent MAC address on virtual interfaces.
> See:
>
> https://github.com/Mellanox/mlxsw/wiki/Persistent-Configuration#required-changes-to-macaddresspolicy
>
> If that is the issue, then you can either change MACAddressPolicy or
> modify your code to create the veth pair with the correct MAC address
> from the start.
>
> As I understand it, a possible explanation for the race is that your
> program is racing with udev. Udev receives an event about a new device
> and reads "addr_assign_type" from sysfs. If your program changed the MAC
> address before udev read the file, then udev will read 3 (NET_ADDR_SET)
> and will not try to change the MAC address. Otherwise, both your program
> and udev will try to change the MAC address.
>
> If it is not udev that is changing the MAC address, then you can run the
> following bpftrace script in a different terminal and check which
> processes try to change the address:
>
> bpftrace -e 'k:dev_set_mac_address_user { @[comm] = count(); }'
>
> It is only a theory. Actual issue might be entirely different.
>

Thank you Ido, we will test whether the systemd change is the true
culprit behind this, and will follow up with what we find :)

Appreciate all the help so far!

Best regards,
Jason Zhou
Software Engineering Intern
jasonzhou@x.com


> >
> > We were re-fetching the addresses via the ioctl SIOCGIFHWADDR syscall
> > as well as via getifaddr (which appears to use netlink under the
> > covers), and, in problematic cases, both functions reported
> > discrepancies from the target MAC address we were initially setting
> > to. We also performed a fetch before we set the MAC addresses and
> > found that there are instances where getifaddr and ioctl results do
> > not match for our veth device *even before we perform any setting of
> > the MAC address*. It's also worth noting that after setting the MAC
> > address: there are no cases where ioctl or getifaddr come back with
> > the same MAC address as before we set the address. So, the set
> > operation always seems to have an effect.
> >
> > Observed scenarios with incorrectly assigned MAC addresses:
> >
> > (1) After setting the mac address: ioctl returns the correct MAC
> > address, but the results from getifaddr, returns an incorrect MAC
> > address (different from the original value before setting as well!)
> >
> > (2) After setting the MAC address: both ioctl and getifaddr return the
> > same MAC address, but are both wrong (and different from the original
> > one!)
> >
> > (3) There is a possibility that the MAC address we set ends up
> > overwritten by a garbage value *after* we have already updated the MAC
> > address, and checked that the MAC address was set correctly. Since
> > this error happens after this function has finished, we cannot log nor
> > detect it in the function where we set the MAC address because we have
> > not yet studied at what point this late overwriting of MAC address
> > occurs. It’s worth noting that this is the rarest scenario that we
> > have encountered, and we were only able to reproduce it in our testing
> > cluster machine, not in any of the production cluster machines.
> >
> > [3.] Keywords:
> >
> > networking, veth, kernel, MAC, netlink
> >
> > [X.] Other notes, patches, fixes, workarounds:
> >
> > Notes:
> >
> > More specific kernel and environment information will be available on
> > request for security reasons, please let us know if you are interested
> > and we will be happy to provide you with the necessary information.
> >
> > We have observed this behavior only on CentOS 9 systems at the moment,
> > CentOS 7 systems under various kernels do not seem to have the issue
> > (which is quite strange if this was purely a kernel bug).
> >
> > We have tried kernels 5.15.147, 5.15.160, 5.15.161, all of these have
> > this issue on CentOS 9.
> >
> > We have also tried rewriting our function for setting MAC address to
> > use libnl rather than ioctl to perform the MAC address setting, but it
> > did not eliminate the issue.
> >
> > To work around this bug, we checked that the MAC address is set
> > correctly after the ioctl set call, and retry the address setting if
> > necessary. In our testing, this workaround appears to remedy scenarios
> > (1) and (2) above, but it does not address scenario (3).  You can see
> > it here:
> >
> > https://github.com/apache/mesos/commit/8b202bbebdc89429ad82c6983aa1c514eb1b8d95
> >
> > We would greatly appreciate any insights or guidance on this matter.
> > Please let me know if you need further information or if there are any
> > specific tests we should run to assist in diagnosing the issue. Again,
> > specific details for the production machines on which we encountered
> > this error can be provided upon request, so please let us know if
> > there is anything we can provide to help.
> >
> > Thank you for your time and assistance.
> >
> > Best regards,
> > Jason Zhou
> > Software Engineering Intern
> > jasonzhou@x.com
> >
> > embedded links:
> > [1] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/slave/containerizer/mesos/isolators/network/port_mapping.cpp#L3599
> > [2] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/linux/routing/link/veth.cpp#L45
> > [3] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/slave/containerizer/mesos/isolators/network/port_mapping.cpp#L3628
> > [4] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/linux/routing/link/link.cpp#L283
> >

Re: PROBLEM: Issue with setting veth MAC address being unreliable.

[Jason Zhou]

Hi all,

We have performed our own testing with the MacAddressPolicy set to
none rather than persistent on CentOS 9, and it fixed the problem we
were seeing with the MAC address mismatches before and after us trying
to manually set it.
So we're pretty confident that the cause is what Ido stated, and that
we were racing against udev as we did not set a MAC address when
creating our veth device pair, making udev think it should give out a
new MAC address.
We will release a patch on Apache Mesos to mitigate this potential
race condition on systems with systemd version > 242.
Thank you so much for the help!

For documenting this issue, I believe that this race condition would
also be present for the peer veth interface?
We create the peer along with veth and move the peer to another
namespace, but udev would be notified of its creation, so it will try
to also overwrite the peer's MAC address.
However this is not an issue for the loopback interface that comes
with every namespace creation, as they will not be affected by
NetworkManager and hence udev will not try to modify them.
Please correct me if I'm wrong!

Once again, thanks for all the help!
Best regards,
Jason Zhou
Software Engineering Intern
jasonzhou@x.com


On Thu, Jul 11, 2024 at 10:07 AM Jason Zhou <jasonzhou@x.com> wrote:
>
> Apologies for the late reply,
>
> > When you say 5.15.161, do you mean true 5.15.161, or is this CentOS
> > version of 5.15.161 with 1000s of patches on top? Did you download the
> > sources from www.kernel.org and build it yourself?
>
> For Andrew's question, yes, we do use the mainline 5.15 LTS kernel
> which we built ourselves from the www.kernel.org sources.
>
> On Thu, Jul 11, 2024 at 9:18 AM Ido Schimmel <idosch@idosch.org> wrote:
> >
> > On Wed, Jul 10, 2024 at 04:45:55PM -0400, Jason Zhou wrote:
> > > [1.] One line summary of the problem:
> > >
> > > Issue with setting veth address being unreliable.
> > >
> > > [2.] Full description of the problem/report:
> > >
> > > Hello!
> > >
> > > We have been investigating a strange behavior within Apache Mesos
> > > where after setting the MAC address on a veth device to the same
> > > address as our eth0 MAC address, the change is sometimes not reflected
> > > appropriately despite the ioctl call succeeding (~4% of the time in
> > > our testing). Note that we also tried using libnl to set the MAC
> > > address but the issue still persists.
> > >
> > > Included below is the github link to the section where we set the veth
> > > address, to clarify what we were trying to do. We first create the
> > > veth pair [1] using a libnl function [2], then we set the veth device
> > > MAC addresses to that of our host public interface (eth0) [3] using a
> > > function called setMAC. Inside the setMAC [4] is where we are
> > > observing the aforementioned issue with unreliable setting of veth
> > > addresses..
> > >
> > > This behavior was observed when re-fetching the MAC address on said
> > > veth device after we made the function call to set its MAC address. We
> > > have observed this issue on CentOS 9 only, but not on CentOS 7. We
> > > have tried Linux kernels 5.15.147, 5.15.160 & 5.15.161 for CentOS 9,
> > > CentOS 7 was using 5.10, but we also tried upgrading the Centos 7 host
> > > to 5.15.160 but could not reproduce the bug.
> >
> > This suggests that the change in behavior is due to a change in user
> > space and an obvious suspect is systemd / udev. AFAICT, CentOS 7 is
> > using systemd 219 whereas CentOS 9 is using systemd 252. In version 242
> > systemd started setting a persistent MAC address on virtual interfaces.
> > See:
> >
> > https://github.com/Mellanox/mlxsw/wiki/Persistent-Configuration#required-changes-to-macaddresspolicy
> >
> > If that is the issue, then you can either change MACAddressPolicy or
> > modify your code to create the veth pair with the correct MAC address
> > from the start.
> >
> > As I understand it, a possible explanation for the race is that your
> > program is racing with udev. Udev receives an event about a new device
> > and reads "addr_assign_type" from sysfs. If your program changed the MAC
> > address before udev read the file, then udev will read 3 (NET_ADDR_SET)
> > and will not try to change the MAC address. Otherwise, both your program
> > and udev will try to change the MAC address.
> >
> > If it is not udev that is changing the MAC address, then you can run the
> > following bpftrace script in a different terminal and check which
> > processes try to change the address:
> >
> > bpftrace -e 'k:dev_set_mac_address_user { @[comm] = count(); }'
> >
> > It is only a theory. Actual issue might be entirely different.
> >
>
> Thank you Ido, we will test whether the systemd change is the true
> culprit behind this, and will follow up with what we find :)
>
> Appreciate all the help so far!
>
> Best regards,
> Jason Zhou
> Software Engineering Intern
> jasonzhou@x.com
>
>
> > >
> > > We were re-fetching the addresses via the ioctl SIOCGIFHWADDR syscall
> > > as well as via getifaddr (which appears to use netlink under the
> > > covers), and, in problematic cases, both functions reported
> > > discrepancies from the target MAC address we were initially setting
> > > to. We also performed a fetch before we set the MAC addresses and
> > > found that there are instances where getifaddr and ioctl results do
> > > not match for our veth device *even before we perform any setting of
> > > the MAC address*. It's also worth noting that after setting the MAC
> > > address: there are no cases where ioctl or getifaddr come back with
> > > the same MAC address as before we set the address. So, the set
> > > operation always seems to have an effect.
> > >
> > > Observed scenarios with incorrectly assigned MAC addresses:
> > >
> > > (1) After setting the mac address: ioctl returns the correct MAC
> > > address, but the results from getifaddr, returns an incorrect MAC
> > > address (different from the original value before setting as well!)
> > >
> > > (2) After setting the MAC address: both ioctl and getifaddr return the
> > > same MAC address, but are both wrong (and different from the original
> > > one!)
> > >
> > > (3) There is a possibility that the MAC address we set ends up
> > > overwritten by a garbage value *after* we have already updated the MAC
> > > address, and checked that the MAC address was set correctly. Since
> > > this error happens after this function has finished, we cannot log nor
> > > detect it in the function where we set the MAC address because we have
> > > not yet studied at what point this late overwriting of MAC address
> > > occurs. It’s worth noting that this is the rarest scenario that we
> > > have encountered, and we were only able to reproduce it in our testing
> > > cluster machine, not in any of the production cluster machines.
> > >
> > > [3.] Keywords:
> > >
> > > networking, veth, kernel, MAC, netlink
> > >
> > > [X.] Other notes, patches, fixes, workarounds:
> > >
> > > Notes:
> > >
> > > More specific kernel and environment information will be available on
> > > request for security reasons, please let us know if you are interested
> > > and we will be happy to provide you with the necessary information.
> > >
> > > We have observed this behavior only on CentOS 9 systems at the moment,
> > > CentOS 7 systems under various kernels do not seem to have the issue
> > > (which is quite strange if this was purely a kernel bug).
> > >
> > > We have tried kernels 5.15.147, 5.15.160, 5.15.161, all of these have
> > > this issue on CentOS 9.
> > >
> > > We have also tried rewriting our function for setting MAC address to
> > > use libnl rather than ioctl to perform the MAC address setting, but it
> > > did not eliminate the issue.
> > >
> > > To work around this bug, we checked that the MAC address is set
> > > correctly after the ioctl set call, and retry the address setting if
> > > necessary. In our testing, this workaround appears to remedy scenarios
> > > (1) and (2) above, but it does not address scenario (3).  You can see
> > > it here:
> > >
> > > https://github.com/apache/mesos/commit/8b202bbebdc89429ad82c6983aa1c514eb1b8d95
> > >
> > > We would greatly appreciate any insights or guidance on this matter.
> > > Please let me know if you need further information or if there are any
> > > specific tests we should run to assist in diagnosing the issue. Again,
> > > specific details for the production machines on which we encountered
> > > this error can be provided upon request, so please let us know if
> > > there is anything we can provide to help.
> > >
> > > Thank you for your time and assistance.
> > >
> > > Best regards,
> > > Jason Zhou
> > > Software Engineering Intern
> > > jasonzhou@x.com
> > >
> > > embedded links:
> > > [1] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/slave/containerizer/mesos/isolators/network/port_mapping.cpp#L3599
> > > [2] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/linux/routing/link/veth.cpp#L45
> > > [3] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/slave/containerizer/mesos/isolators/network/port_mapping.cpp#L3628
> > > [4] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/linux/routing/link/link.cpp#L283
> > >

Re: PROBLEM: Issue with setting veth MAC address being unreliable.

[Ido Schimmel]

On Fri, Jul 12, 2024 at 05:02:13PM -0400, Jason Zhou wrote:
> Hi all,
> 
> We have performed our own testing with the MacAddressPolicy set to
> none rather than persistent on CentOS 9, and it fixed the problem we
> were seeing with the MAC address mismatches before and after us trying
> to manually set it.
> So we're pretty confident that the cause is what Ido stated, and that
> we were racing against udev as we did not set a MAC address when
> creating our veth device pair, making udev think it should give out a
> new MAC address.
> We will release a patch on Apache Mesos to mitigate this potential
> race condition on systems with systemd version > 242.
> Thank you so much for the help!
> 
> For documenting this issue, I believe that this race condition would
> also be present for the peer veth interface?

Yes, but when creating the veth pair you can set the addresses of both
devices:

# ip link add name bla1 address 00:11:22:33:44:55 type veth peer name bla2 address 00:aa:bb:cc:dd:ee
# ip link show dev bla1
11: bla1@bla2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
# ip link show dev bla2
10: bla2@bla1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:aa:bb:cc:dd:ee brd ff:ff:ff:ff:ff:ff

> We create the peer along with veth and move the peer to another
> namespace, but udev would be notified of its creation, so it will try
> to also overwrite the peer's MAC address.

The peer can be created in the desired namespace with the desired
address:

# ip netns add ns1
# ip link add name bla1 address 00:11:22:33:44:55 type veth peer name bla2 address 00:aa:bb:cc:dd:ee netns ns1
# ip link show dev bla1
10: bla1@if8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff link-netns ns1
bash-5.2# ip -n ns1 link show dev bla2
8: bla2@if10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:aa:bb:cc:dd:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0

> However this is not an issue for the loopback interface that comes
> with every namespace creation, as they will not be affected by
> NetworkManager and hence udev will not try to modify them.
> Please correct me if I'm wrong!

AFAICT udev ignores the loopback devices because they have a type of
ARPHRD_LOOPBACK, unlike the veth devices that have a type of
ARPHRD_ETHER.

Re: PROBLEM: Issue with setting veth MAC address being unreliable.

[Jason Zhou]

Got it, thank you so much for the help!

On Sun, Jul 14, 2024 at 8:45 AM Ido Schimmel <idosch@idosch.org> wrote:
>
> On Fri, Jul 12, 2024 at 05:02:13PM -0400, Jason Zhou wrote:
> > Hi all,
> >
> > We have performed our own testing with the MacAddressPolicy set to
> > none rather than persistent on CentOS 9, and it fixed the problem we
> > were seeing with the MAC address mismatches before and after us trying
> > to manually set it.
> > So we're pretty confident that the cause is what Ido stated, and that
> > we were racing against udev as we did not set a MAC address when
> > creating our veth device pair, making udev think it should give out a
> > new MAC address.
> > We will release a patch on Apache Mesos to mitigate this potential
> > race condition on systems with systemd version > 242.
> > Thank you so much for the help!
> >
> > For documenting this issue, I believe that this race condition would
> > also be present for the peer veth interface?
>
> Yes, but when creating the veth pair you can set the addresses of both
> devices:
>
> # ip link add name bla1 address 00:11:22:33:44:55 type veth peer name bla2 address 00:aa:bb:cc:dd:ee
> # ip link show dev bla1
> 11: bla1@bla2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
>     link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
> # ip link show dev bla2
> 10: bla2@bla1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
>     link/ether 00:aa:bb:cc:dd:ee brd ff:ff:ff:ff:ff:ff
>
> > We create the peer along with veth and move the peer to another
> > namespace, but udev would be notified of its creation, so it will try
> > to also overwrite the peer's MAC address.
>
> The peer can be created in the desired namespace with the desired
> address:
>
> # ip netns add ns1
> # ip link add name bla1 address 00:11:22:33:44:55 type veth peer name bla2 address 00:aa:bb:cc:dd:ee netns ns1
> # ip link show dev bla1
> 10: bla1@if8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
>     link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff link-netns ns1
> bash-5.2# ip -n ns1 link show dev bla2
> 8: bla2@if10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
>     link/ether 00:aa:bb:cc:dd:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
>
> > However this is not an issue for the loopback interface that comes
> > with every namespace creation, as they will not be affected by
> > NetworkManager and hence udev will not try to modify them.
> > Please correct me if I'm wrong!
>
> AFAICT udev ignores the loopback devices because they have a type of
> ARPHRD_LOOPBACK, unlike the veth devices that have a type of
> ARPHRD_ETHER.