VDPA MAC address problem

[Konstantin Shkolnyy <kshk@linux.ibm.com>]

I’m observing a problem while testing VDPA with Nvidia ConnectX-6 (mlx5) 
on s390.

Upon start, virtio_net_device_realize() tries to set a new MAC address 
by VHOST_VDPA_SET_CONFIG which doesn’t do anything.

Later, the VM gets started and learns about the old address from 
virtio_net_get_config() which returns whatever VHOST_VDPA_GET_CONFIG 
returns, unless it's "6 zero bytes", in which case it instead returns 
the desired new address (and the problem is avoided).

Then QEMU again tries to set the new address from vhost_net_start(), now 
by calling vhost_vdpa_net_load_cmd(...,VIRTIO_NET_CTRL_MAC, 
VIRTIO_NET_CTRL_MAC_ADDR_SET, ...). This time the new address is 
successfully programmed into the NIC, but the VM doesn't know about it.

As the result, the VM now sends packets with a source address on which 
the NIC doesn’t listen.

Upon reading this forum, I see that VHOST_VDPA_SET_CONFIG is 
“deprecated”, and so VIRTIO_NET_CTRL_MAC_ADDR_SET must be the right 
method, but it’s apparently called too late. Or maybe 
virtio_net_get_config() needs to always return the desired new address 
and not the old one from VHOST_VDPA_GET_CONFIG?

I’m looking for an opinion/direction from someone who knows this code.

As it is, the only VDPA scenario that's working for me is:
1) Avoid specifying the MAC address in the "vdpa dev add" command (which 
will create the "6 zero bytes" condition on the first launch).
2) Keep using the same MAC address for every subsequent VM launch on the 
same NIC "virtual function" (so that the old and new addresses are the 
same).