Re: pci passthrough - VF reset at boot is dropping assigned MAC

[Alex Williamson <alex.williamson@redhat.com>]

On Mon, 2011-04-25 at 13:12 -0600, David Ahern wrote:
> 
> On 04/25/11 12:36, Alex Williamson wrote:
> > On Mon, 2011-04-25 at 12:04 -0600, David Ahern wrote:
> >>
> >> On 04/25/11 11:30, Alex Williamson wrote:
> >>> # modprobe -r igbvf
> >>> # ip link set eth2 vf 6 mac 02:00:10:91:73:01
> >>> # modprobe igbvf
> >>> # dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
> >>> igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
> >>> igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
> >>> igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
> >>> # modprobe -r igbvf
> >>> # echo 1 > /sys/bus/pci/devices/0000:01:11.5/reset
> >>> # modprobe igbvf
> >>> # dmesg | grep "igbvf 0000\:01\:11.5\: Address\:"
> >>> igbvf 0000:01:11.5: Address: d2:c8:17:d6:97:f7
> >>> igbvf 0000:01:11.5: Address: 4e:ee:2a:d8:12:7c
> >>> igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
> >>> igbvf 0000:01:11.5: Address: 02:00:10:91:73:01
> >>>
> >>> So now it sticks.  You're going to get random mac addresses on the VFs
> >>> every time you reload the igb driver (ie. ever boot) anyway (at least
> >>> with these sr-iov cards), so if you need consistent macs, they probably
> >>> need to be set before launching the VM anyway.  Thanks,
> >>>
> >>> Alex
> >>>
> >>
> >> Ok, I was able to repeat the above commands from the host command line.
> >>
> >> However, when qemu-kvm starts the MAC is reset.
> >>
> >> # ip link show | less
> >>
> >> 2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
> >> qlen 1000
> >>     link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff
> >>     vf 0 MAC 02:12:34:56:80:20
> >>
> >> --> that's the MAC address I set
> >>
> >> I start qemu-kvm (unpatched version) and the host side sees the address
> >> changed:
> >>
> >> # ip link show | less
> >>
> >> 2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
> >> qlen 1000
> >>     link/ether 00:1b:21:98:b7:10 brd ff:ff:ff:ff:ff:ff
> >>     vf 0 MAC 7a:17:3f:98:0f:db
> >>
> >>
> >> Can you try that aspect on your end - seeing if the MAC address
> >> maintains after starting qemu-kvm?
> > 
> > I don't see this happening on my system, once manually set the mac never
> > changes.  I can restart and reset the VM and the host and guest both
> > continue seeing the set mac address.  I tested it with both a recent
> > rhel6.1 host kernel as well as upstream 2.6.39-rc4.  If I switch to a VF
> > with an unset mac, those will change on each VM reset or restart.
> 
> Blacklist igbvf in the host and you will. That must be the difference: I
> was preventing the vf driver from loading in the host -- it's not needed
> there, so why load it?

I already have it blacklisted.  It's not needed if you're using the VFs
they way we are, but there are other uses.

> I rebooted for a fresh run. Loaded the igbvf driver before starting the
> VM using my tools. With the igbvf driver loaded in the host the MAC
> address for the VF was not reset.
> 
> As for why I blacklisted it -- udev. What a PITA with VFs. I saw the
> feature for Fedora 15 which should address this.

Yes, my VM is up to renaming the VFs eth1340 since the mac changes every
boot.  I'm still confused though as I did a whole round of testing after
a reboot where igbvf was never loaded and the set mac address stuck
across VM restarts and resets.

Alex