Re: pci passthrough - VF reset at boot is dropping assigned MAC

[Alex Williamson <alex.williamson@redhat.com>]

On Mon, 2011-04-25 at 10:41 -0600, David Ahern wrote:
> 
> On 04/25/11 10:37, Alex Williamson wrote:
> > On Mon, 2011-04-25 at 10:28 -0600, David Ahern wrote:
> >> Running qemu-kvm.git as of today (ffce28f, April 18, 2011) the virtual
> >> function passed to the VM is losing its assigned mac address. That is,
> >> prior to launching qemu-kvm, the following command is run to set the MAC
> >> address:
> >>
> >> ip link set dev eth2 vf 0 mac 02:12:34:56:79:20
> >>
> >> Yet, when the VM boots the MAC address is random which is what happens
> >> when the VF is reset. Looking through the commit logs between 0.13.0 --
> >> the version in Fedora 14 -- and latest git I found the following:
> >>
> >> commit d9488459ff2ab113293586c1c36b1679bb15deee
> >> Author: Alex Williamson <alex.williamson@redhat.com>
> >> Date:   Thu Mar 17 15:24:31 2011 -0600
> >>
> >>     device-assignment: Reset device on system reset
> >>
> >>     On system reset, we currently try to quiesce DMA by clearing the
> >>     command register.  This assumes that nothing re-enables bus master
> >>     support without first de-programming the device.  Use a bigger
> >>     hammer to help the guest not shoot itself by issuing a function
> >>     reset via sysfs on each system reset.
> >>
> >>     Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> >>     Acked-by: Chris Wright <chrisw@redhat.com>
> >>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> >>
> >>
> >> Is this the cause of the MAC address reset and is this behavior intended?
> > 
> > Ugh, I hope not, it's certainly not an intended side effect.  Can you
> > see if the problem still happens if you revert this patch?  If it does,
> 
> I commented out the write() in the reset function and indeed the mac
> address was not reset on VM boot.

Ok, here's what I see on my system:

# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
# modprobe -r igbvf
# echo 1 > /sys/bus/pci/devices/0000:01:11.5/reset
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c

So yes, it does change.  However, if I set the VF mac instead of using a
randomly generated one, I get:

# modprobe -r igbvf
# ip link set eth2 vf 6 mac 02:00:10:91:73:01
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
# modprobe -r igbvf
# echo 1 > /sys/bus/pci/devices/0000:01:11.5/reset
# modprobe igbvf
# dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
igbvf 0000:01:11.5: Address: 02:00:10:91:73:01

So now it sticks.  You're going to get random mac addresses on the VFs
every time you reload the igb driver (ie. ever boot) anyway (at least
with these sr-iov cards), so if you need consistent macs, they probably
need to be set before launching the VM anyway.  Thanks,

Alex