VDPA MAC address problem

VDPA MAC address problem

[Konstantin Shkolnyy]

I’m observing a problem while testing VDPA with Nvidia ConnectX-6 (mlx5) 
on s390.

Upon start, virtio_net_device_realize() tries to set a new MAC address 
by VHOST_VDPA_SET_CONFIG which doesn’t do anything.

Later, the VM gets started and learns about the old address from 
virtio_net_get_config() which returns whatever VHOST_VDPA_GET_CONFIG 
returns, unless it's "6 zero bytes", in which case it instead returns 
the desired new address (and the problem is avoided).

Then QEMU again tries to set the new address from vhost_net_start(), now 
by calling vhost_vdpa_net_load_cmd(...,VIRTIO_NET_CTRL_MAC, 
VIRTIO_NET_CTRL_MAC_ADDR_SET, ...). This time the new address is 
successfully programmed into the NIC, but the VM doesn't know about it.

As the result, the VM now sends packets with a source address on which 
the NIC doesn’t listen.

Upon reading this forum, I see that VHOST_VDPA_SET_CONFIG is 
“deprecated”, and so VIRTIO_NET_CTRL_MAC_ADDR_SET must be the right 
method, but it’s apparently called too late. Or maybe 
virtio_net_get_config() needs to always return the desired new address 
and not the old one from VHOST_VDPA_GET_CONFIG?

I’m looking for an opinion/direction from someone who knows this code.

As it is, the only VDPA scenario that's working for me is:
1) Avoid specifying the MAC address in the "vdpa dev add" command (which 
will create the "6 zero bytes" condition on the first launch).
2) Keep using the same MAC address for every subsequent VM launch on the 
same NIC "virtual function" (so that the old and new addresses are the 
same).

Re: VDPA MAC address problem

[Jason Wang]

Adding Cindy and Eugenio

On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
>
> I’m observing a problem while testing VDPA with Nvidia ConnectX-6 (mlx5)
> on s390.
>
> Upon start, virtio_net_device_realize() tries to set a new MAC address
> by VHOST_VDPA_SET_CONFIG which doesn’t do anything.
>
> Later, the VM gets started and learns about the old address from
> virtio_net_get_config() which returns whatever VHOST_VDPA_GET_CONFIG
> returns, unless it's "6 zero bytes", in which case it instead returns
> the desired new address (and the problem is avoided).
>
> Then QEMU again tries to set the new address from vhost_net_start(), now
> by calling vhost_vdpa_net_load_cmd(...,VIRTIO_NET_CTRL_MAC,
> VIRTIO_NET_CTRL_MAC_ADDR_SET, ...). This time the new address is
> successfully programmed into the NIC, but the VM doesn't know about it.

Have you enabled shadow virtqueue? If yes, does it work if you don't do that?

>
> As the result, the VM now sends packets with a source address on which
> the NIC doesn’t listen.
>
> Upon reading this forum, I see that VHOST_VDPA_SET_CONFIG is
> “deprecated”, and so VIRTIO_NET_CTRL_MAC_ADDR_SET must be the right
> method, but it’s apparently called too late.

VHOST_VDPA_SET_CONFIG requires the vDPA parent support which is not
necessarily there.
VIRTIO_NET_CTRL_MAC_ADDR_SET requires the shadow virtqueue as well as
the CTRL_MAC_ADDR support.

Neither of them seems robust.

> Or maybe
> virtio_net_get_config() needs to always return the desired new address
> and not the old one from VHOST_VDPA_GET_CONFIG?
>
> I’m looking for an opinion/direction from someone who knows this code.
>
> As it is, the only VDPA scenario that's working for me is:
> 1) Avoid specifying the MAC address in the "vdpa dev add" command (which
> will create the "6 zero bytes" condition on the first launch).
> 2) Keep using the same MAC address for every subsequent VM launch on the
> same NIC "virtual function" (so that the old and new addresses are the
> same).

This is the way we currently use it. Is there any limitation of this?

Thanks

>

Re: VDPA MAC address problem

[Eugenio Perez Martin]

On Thu, Mar 20, 2025 at 1:59 AM Jason Wang <jasowang@redhat.com> wrote:
>
> Adding Cindy and Eugenio
>
> On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
> >
> > I’m observing a problem while testing VDPA with Nvidia ConnectX-6 (mlx5)
> > on s390.
> >
> > Upon start, virtio_net_device_realize() tries to set a new MAC address
> > by VHOST_VDPA_SET_CONFIG which doesn’t do anything.
> >
> > Later, the VM gets started and learns about the old address from
> > virtio_net_get_config() which returns whatever VHOST_VDPA_GET_CONFIG
> > returns, unless it's "6 zero bytes", in which case it instead returns
> > the desired new address (and the problem is avoided).
> >
> > Then QEMU again tries to set the new address from vhost_net_start(), now
> > by calling vhost_vdpa_net_load_cmd(...,VIRTIO_NET_CTRL_MAC,
> > VIRTIO_NET_CTRL_MAC_ADDR_SET, ...). This time the new address is
> > successfully programmed into the NIC, but the VM doesn't know about it.
>
> Have you enabled shadow virtqueue? If yes, does it work if you don't do that?
>

Either you're using SVQ or not, is cmdline nic mac address the same as
the provided with the vdpa command?

> >
> > As the result, the VM now sends packets with a source address on which
> > the NIC doesn’t listen.
> >
> > Upon reading this forum, I see that VHOST_VDPA_SET_CONFIG is
> > “deprecated”, and so VIRTIO_NET_CTRL_MAC_ADDR_SET must be the right
> > method, but it’s apparently called too late.
>
> VHOST_VDPA_SET_CONFIG requires the vDPA parent support which is not
> necessarily there.
> VIRTIO_NET_CTRL_MAC_ADDR_SET requires the shadow virtqueue as well as
> the CTRL_MAC_ADDR support.
>

Nit, VIRTIO_NET_CTRL_MAC_ADDR_SET does not require SVQ. You are not
able to migrate if SVQ cannot be enabled, for example, because it does
not support a given feature.

> Neither of them seems robust.
>
> > Or maybe
> > virtio_net_get_config() needs to always return the desired new address
> > and not the old one from VHOST_VDPA_GET_CONFIG?
> >
> > I’m looking for an opinion/direction from someone who knows this code.
> >
> > As it is, the only VDPA scenario that's working for me is:
> > 1) Avoid specifying the MAC address in the "vdpa dev add" command (which
> > will create the "6 zero bytes" condition on the first launch).
> > 2) Keep using the same MAC address for every subsequent VM launch on the
> > same NIC "virtual function" (so that the old and new addresses are the
> > same).
>
> This is the way we currently use it. Is there any limitation of this?
>
> Thanks
>
> >
>

Re: VDPA MAC address problem

[Cindy Lu]

Hi Konstantin
yes, we have found this kind of issue in this upstream

So here is a fix for this problem in kernel
https://lore.kernel.org/kvm/CACLfguUYny6-1cYABsGS+qtdzO+MKp3O09t_gt-bMM4JgdpZqA@mail.gmail.com/T/
vdpa: support set mac address from vdpa tool

This tool allows the user to change the MAC address before QEMU loads.
The user should ensure that the VDPA device's MAC address matches the
MAC address in the QEMU command line.

On the QEMU side, we plan to add a new parameter to check the MAC
address. If the MAC address in the QEMU command line does not match
the device's MAC address, the system will fail to boot.
https://patchew.org/QEMU/20241109063241.1039433-1-lulu@redhat.com/

but this patch is not merged in upstream yet, I will rebase and send
it to upstream again
Thanks
Cindy

On Thu, Mar 20, 2025 at 8:59 AM Jason Wang <jasowang@redhat.com> wrote:
>
> Adding Cindy and Eugenio
>
> On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
> >
> > I’m observing a problem while testing VDPA with Nvidia ConnectX-6 (mlx5)
> > on s390.
> >
> > Upon start, virtio_net_device_realize() tries to set a new MAC address
> > by VHOST_VDPA_SET_CONFIG which doesn’t do anything.
> >
> > Later, the VM gets started and learns about the old address from
> > virtio_net_get_config() which returns whatever VHOST_VDPA_GET_CONFIG
> > returns, unless it's "6 zero bytes", in which case it instead returns
> > the desired new address (and the problem is avoided).
> >
> > Then QEMU again tries to set the new address from vhost_net_start(), now
> > by calling vhost_vdpa_net_load_cmd(...,VIRTIO_NET_CTRL_MAC,
> > VIRTIO_NET_CTRL_MAC_ADDR_SET, ...). This time the new address is
> > successfully programmed into the NIC, but the VM doesn't know about it.
>
> Have you enabled shadow virtqueue? If yes, does it work if you don't do that?
>
> >
> > As the result, the VM now sends packets with a source address on which
> > the NIC doesn’t listen.
> >
> > Upon reading this forum, I see that VHOST_VDPA_SET_CONFIG is
> > “deprecated”, and so VIRTIO_NET_CTRL_MAC_ADDR_SET must be the right
> > method, but it’s apparently called too late.
>
> VHOST_VDPA_SET_CONFIG requires the vDPA parent support which is not
> necessarily there.
> VIRTIO_NET_CTRL_MAC_ADDR_SET requires the shadow virtqueue as well as
> the CTRL_MAC_ADDR support.
>
> Neither of them seems robust.
>
> > Or maybe
> > virtio_net_get_config() needs to always return the desired new address
> > and not the old one from VHOST_VDPA_GET_CONFIG?
> >
> > I’m looking for an opinion/direction from someone who knows this code.
> >
> > As it is, the only VDPA scenario that's working for me is:
> > 1) Avoid specifying the MAC address in the "vdpa dev add" command (which
> > will create the "6 zero bytes" condition on the first launch).
> > 2) Keep using the same MAC address for every subsequent VM launch on the
> > same NIC "virtual function" (so that the old and new addresses are the
> > same).
>
> This is the way we currently use it. Is there any limitation of this?
>
> Thanks
>
> >
>

Re: VDPA MAC address problem

[Konstantin Shkolnyy]

On 3/20/2025 01:54, Eugenio Perez Martin wrote:
> On Thu, Mar 20, 2025 at 1:59 AM Jason Wang <jasowang@redhat.com> wrote:
>>
>> Adding Cindy and Eugenio
>>
>> On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
>>>
>>> I’m observing a problem while testing VDPA with Nvidia ConnectX-6 (mlx5)
>>> on s390.
>>>
>>> Upon start, virtio_net_device_realize() tries to set a new MAC address
>>> by VHOST_VDPA_SET_CONFIG which doesn’t do anything.
>>>
>>> Later, the VM gets started and learns about the old address from
>>> virtio_net_get_config() which returns whatever VHOST_VDPA_GET_CONFIG
>>> returns, unless it's "6 zero bytes", in which case it instead returns
>>> the desired new address (and the problem is avoided).
>>>
>>> Then QEMU again tries to set the new address from vhost_net_start(), now
>>> by calling vhost_vdpa_net_load_cmd(...,VIRTIO_NET_CTRL_MAC,
>>> VIRTIO_NET_CTRL_MAC_ADDR_SET, ...). This time the new address is
>>> successfully programmed into the NIC, but the VM doesn't know about it.
>>
>> Have you enabled shadow virtqueue? If yes, does it work if you don't do that?
>>
> 
> Either you're using SVQ or not, is cmdline nic mac address the same as
> the provided with the vdpa command?

The problem happens when the cmdline address (new) is different from the 
current one in the VDPA device (old), whether the old one has been set 
by "vdpa dev add" or by an earlier VM run.

In other words, it's generally not possible to run a VM with an 
arbitrary address - it'll have to inherit whatever the VDPA device 
already has. (Except for the very first run and _only_ if "vdpa dev add" 
didn't specify the address (leaving it "6 zero bytes"), or specified the 
same one as the cmdline.)

Therefore, it's not possible to omit the address in the libvirt XML and 
let libvirt generate it.

Re: VDPA MAC address problem

[Konstantin Shkolnyy]

On 3/19/2025 19:58, Jason Wang wrote:
> Adding Cindy and Eugenio
> 
> On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
>>
>> I’m observing a problem while testing VDPA with Nvidia ConnectX-6 (mlx5)
>> on s390.
>>
>> Upon start, virtio_net_device_realize() tries to set a new MAC address
>> by VHOST_VDPA_SET_CONFIG which doesn’t do anything.
>>
>> Later, the VM gets started and learns about the old address from
>> virtio_net_get_config() which returns whatever VHOST_VDPA_GET_CONFIG
>> returns, unless it's "6 zero bytes", in which case it instead returns
>> the desired new address (and the problem is avoided).
>>
>> Then QEMU again tries to set the new address from vhost_net_start(), now
>> by calling vhost_vdpa_net_load_cmd(...,VIRTIO_NET_CTRL_MAC,
>> VIRTIO_NET_CTRL_MAC_ADDR_SET, ...). This time the new address is
>> successfully programmed into the NIC, but the VM doesn't know about it.
> 
> Have you enabled shadow virtqueue? If yes, does it work if you don't do that?

I only run it in the default configuration, which has SVQ enabled.

>> As it is, the only VDPA scenario that's working for me is:
>> 1) Avoid specifying the MAC address in the "vdpa dev add" command (which
>> will create the "6 zero bytes" condition on the first launch).
>> 2) Keep using the same MAC address for every subsequent VM launch on the
>> same NIC "virtual function" (so that the old and new addresses are the
>> same).
> 
> This is the way we currently use it. Is there any limitation of this?

To be honest, I don't know how severe this limitation is, because I'm 
not a sysadmin. The problem is that I was searching www for info on how 
to setup VDPA and ended up hitting this bug and investigating it. 
Because the found info sources are variable and some show the address 
specified in "vdpa dev add".

Re: VDPA MAC address problem

[Konstantin Shkolnyy]

On 3/19/2025 19:58, Jason Wang wrote:
> On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
>> Upon reading this forum, I see that VHOST_VDPA_SET_CONFIG is
>> “deprecated”, and so VIRTIO_NET_CTRL_MAC_ADDR_SET must be the right
>> method, but it’s apparently called too late.
> 
> VHOST_VDPA_SET_CONFIG requires the vDPA parent support which is not
> necessarily there.

The mlx5 driver doesn't do anything for VHOST_VDPA_SET_CONFIG. Intel's 
driver, however, apparently stores the configuration. So, it appears, 
Intel will avoid the problem... Perhaps mlx5 could do the same so that 
QEMU can set the address before it starts the VM (QEMU doesn't have to 
later let the VM change the config). Conceptually, setting the address 
by QEMU cmdline doesn't look different from setting it by "vdpa dev add".

Re: VDPA MAC address problem

[Jason Wang]

On Fri, Mar 21, 2025 at 12:45 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
>
> On 3/19/2025 19:58, Jason Wang wrote:
> > On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
> >> Upon reading this forum, I see that VHOST_VDPA_SET_CONFIG is
> >> “deprecated”, and so VIRTIO_NET_CTRL_MAC_ADDR_SET must be the right
> >> method, but it’s apparently called too late.
> >
> > VHOST_VDPA_SET_CONFIG requires the vDPA parent support which is not
> > necessarily there.
>
> The mlx5 driver doesn't do anything for VHOST_VDPA_SET_CONFIG. Intel's
> driver, however, apparently stores the configuration. So, it appears,
> Intel will avoid the problem... Perhaps mlx5 could do the same so that
> QEMU can set the address before it starts the VM (QEMU doesn't have to
> later let the VM change the config).

The problem is that config space is not allowed to be written, that is
why mlx5 doesn't implement the write method.

> Conceptually, setting the address
> by QEMU cmdline doesn't look different from setting it by "vdpa dev add".

It's kind of different.

E.g the device may not even have VIRTIO_NET_F_MAC feature etc.

Thanks

>

Re: VDPA MAC address problem

[Jason Wang]

On Thu, Mar 20, 2025 at 10:19 PM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
>
> On 3/19/2025 19:58, Jason Wang wrote:
> > Adding Cindy and Eugenio
> >
> > On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
> >>
> >> I’m observing a problem while testing VDPA with Nvidia ConnectX-6 (mlx5)
> >> on s390.
> >>
> >> Upon start, virtio_net_device_realize() tries to set a new MAC address
> >> by VHOST_VDPA_SET_CONFIG which doesn’t do anything.
> >>
> >> Later, the VM gets started and learns about the old address from
> >> virtio_net_get_config() which returns whatever VHOST_VDPA_GET_CONFIG
> >> returns, unless it's "6 zero bytes", in which case it instead returns
> >> the desired new address (and the problem is avoided).
> >>
> >> Then QEMU again tries to set the new address from vhost_net_start(), now
> >> by calling vhost_vdpa_net_load_cmd(...,VIRTIO_NET_CTRL_MAC,
> >> VIRTIO_NET_CTRL_MAC_ADDR_SET, ...). This time the new address is
> >> successfully programmed into the NIC, but the VM doesn't know about it.
> >
> > Have you enabled shadow virtqueue? If yes, does it work if you don't do that?
>
> I only run it in the default configuration, which has SVQ enabled.
>
> >> As it is, the only VDPA scenario that's working for me is:
> >> 1) Avoid specifying the MAC address in the "vdpa dev add" command (which
> >> will create the "6 zero bytes" condition on the first launch).
> >> 2) Keep using the same MAC address for every subsequent VM launch on the
> >> same NIC "virtual function" (so that the old and new addresses are the
> >> same).
> >
> > This is the way we currently use it. Is there any limitation of this?
>
> To be honest, I don't know how severe this limitation is, because I'm
> not a sysadmin. The problem is that I was searching www for info on how
> to setup VDPA and ended up hitting this bug and investigating it.

Exactly, I think we need to improve the documentation anyhow.

> Because the found info sources are variable and some show the address
> specified in "vdpa dev add".
>

Thanks

Re: VDPA MAC address problem

[Konstantin Shkolnyy]

On 3/20/2025 20:22, Jason Wang wrote:
> On Fri, Mar 21, 2025 at 12:45 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
>>
>> On 3/19/2025 19:58, Jason Wang wrote:
>>> On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
>>>> Upon reading this forum, I see that VHOST_VDPA_SET_CONFIG is
>>>> “deprecated”, and so VIRTIO_NET_CTRL_MAC_ADDR_SET must be the right
>>>> method, but it’s apparently called too late.
>>>
>>> VHOST_VDPA_SET_CONFIG requires the vDPA parent support which is not
>>> necessarily there.
>>
>> The mlx5 driver doesn't do anything for VHOST_VDPA_SET_CONFIG. Intel's
>> driver, however, apparently stores the configuration. So, it appears,
>> Intel will avoid the problem... Perhaps mlx5 could do the same so that
>> QEMU can set the address before it starts the VM (QEMU doesn't have to
>> later let the VM change the config).
> 
> The problem is that config space is not allowed to be written, that is
> why mlx5 doesn't implement the write method.
> 
>> Conceptually, setting the address
>> by QEMU cmdline doesn't look different from setting it by "vdpa dev add".
> 
> It's kind of different.
> 
> E.g the device may not even have VIRTIO_NET_F_MAC feature etc.

I'm not sure I understand... This is the MAC address returned by 
VHOST_VDPA_GET_CONFIG. If mlx5 allows it to be set by "vdpa dev add", 
shouldn't it also allow it to be set by VHOST_VDPA_SET_CONFIG called by 
virtio_net_device_realize(), before the VM exists.

When VM starts running, it queries this MAC address and QEMU fetches it 
by VHOST_VDPA_GET_CONFIG. But, because VHOST_VDPA_SET_CONFIG hasn't 
stored it, it fetches a wrong stale address.

Re: VDPA MAC address problem

[Cindy Lu]

On Fri, Mar 21, 2025 at 12:29 PM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
>
> On 3/20/2025 20:22, Jason Wang wrote:
> > On Fri, Mar 21, 2025 at 12:45 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
> >>
> >> On 3/19/2025 19:58, Jason Wang wrote:
> >>> On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
> >>>> Upon reading this forum, I see that VHOST_VDPA_SET_CONFIG is
> >>>> “deprecated”, and so VIRTIO_NET_CTRL_MAC_ADDR_SET must be the right
> >>>> method, but it’s apparently called too late.
> >>>
> >>> VHOST_VDPA_SET_CONFIG requires the vDPA parent support which is not
> >>> necessarily there.
> >>
> >> The mlx5 driver doesn't do anything for VHOST_VDPA_SET_CONFIG. Intel's
> >> driver, however, apparently stores the configuration. So, it appears,
> >> Intel will avoid the problem... Perhaps mlx5 could do the same so that
> >> QEMU can set the address before it starts the VM (QEMU doesn't have to
> >> later let the VM change the config).
> >
> > The problem is that config space is not allowed to be written, that is
> > why mlx5 doesn't implement the write method.
> >
> >> Conceptually, setting the address
> >> by QEMU cmdline doesn't look different from setting it by "vdpa dev add".
> >
> > It's kind of different.
> >
> > E.g the device may not even have VIRTIO_NET_F_MAC feature etc.
>
> I'm not sure I understand... This is the MAC address returned by
> VHOST_VDPA_GET_CONFIG. If mlx5 allows it to be set by "vdpa dev add",
> shouldn't it also allow it to be set by VHOST_VDPA_SET_CONFIG called by
> virtio_net_device_realize(), before the VM exists.
>
> When VM starts running, it queries this MAC address and QEMU fetches it
> by VHOST_VDPA_GET_CONFIG. But, because VHOST_VDPA_SET_CONFIG hasn't
> stored it, it fetches a wrong stale address.
>
Hi Konstantin
This buggy process is functioning in the MLX driver as follow steps

0: kernel creates VDPA device [MLX: if the mac was not set  while
creating, the mac is 0 in hardware]

1. Get Mac from the cmd line.if the cmdline is 0, qemu will create a
random mac address, save it in VirtIONet->mac and
VirtIONet->nic_conf.macaddr

3. qemu called virtio_net_device_realize()
in this function, the mac address in VirtIONet->mac will set to kernel
while calling vhost_net_set_config() [ MLX not supported  ]

4. qemu boot called vhost_net_get_config, get the hardware mac address
and saved in VirtIONet->nic_conf.macaddr
this will read the mac from the kernel and change the mac address to
kernel's mac [ for MLX, if the hardware mac is 0, using the cmdline
mac(or random mac )  ]
NOTE the information in VirtIONet->mac was not change

5, the qemu will call vhost_vdpa_net_cvq_load(). Even if the cvq/svq
not enabled in qemu cmdline
this function will set the mac in VirtIONet->mac to the kernel.[mlx support]

This will result in QEMU's MAC address differing from the hardware's
MAC address.
So our fix is
1. Add a tool for users to set the MAC address before QEMU loads.
2. Include a parameter in QEMU to check the MAC address before
booting; QEMU will fail to load the system if the MAC addresses do not
match.

Thanks

Cindy

Re: VDPA MAC address problem

[Jason Wang]

On Fri, Mar 21, 2025 at 12:29 PM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
>
> On 3/20/2025 20:22, Jason Wang wrote:
> > On Fri, Mar 21, 2025 at 12:45 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
> >>
> >> On 3/19/2025 19:58, Jason Wang wrote:
> >>> On Thu, Mar 20, 2025 at 12:34 AM Konstantin Shkolnyy <kshk@linux.ibm.com> wrote:
> >>>> Upon reading this forum, I see that VHOST_VDPA_SET_CONFIG is
> >>>> “deprecated”, and so VIRTIO_NET_CTRL_MAC_ADDR_SET must be the right
> >>>> method, but it’s apparently called too late.
> >>>
> >>> VHOST_VDPA_SET_CONFIG requires the vDPA parent support which is not
> >>> necessarily there.
> >>
> >> The mlx5 driver doesn't do anything for VHOST_VDPA_SET_CONFIG. Intel's
> >> driver, however, apparently stores the configuration. So, it appears,
> >> Intel will avoid the problem... Perhaps mlx5 could do the same so that
> >> QEMU can set the address before it starts the VM (QEMU doesn't have to
> >> later let the VM change the config).
> >
> > The problem is that config space is not allowed to be written, that is
> > why mlx5 doesn't implement the write method.
> >
> >> Conceptually, setting the address
> >> by QEMU cmdline doesn't look different from setting it by "vdpa dev add".
> >
> > It's kind of different.
> >
> > E.g the device may not even have VIRTIO_NET_F_MAC feature etc.
>
> I'm not sure I understand... This is the MAC address returned by
> VHOST_VDPA_GET_CONFIG. If mlx5 allows it to be set by "vdpa dev add",

This requires privileges where userspace(VM) doesn't necessarily have.

> shouldn't it also allow it to be set by VHOST_VDPA_SET_CONFIG called by
> virtio_net_device_realize(), before the VM exists.
>
> When VM starts running, it queries this MAC address and QEMU fetches it
> by VHOST_VDPA_GET_CONFIG. But, because VHOST_VDPA_SET_CONFIG hasn't
> stored it, it fetches a wrong stale address.

For example, for some reason management wants to forbid mac address
changes from the guest.

Thanks