Different behavior with vfio-pci between 6.4.8->6.5.5

Different behavior with vfio-pci between 6.4.8->6.5.5

[Juhani Rautiainen]

Hi!

I noticed a change in my home server which breaks some of KVM VM's
with newer kernels. I have two Intel I350 cards: one with two ports
and another with four ports. I have been using the card with two ports
in a firewall VM with vfio-pci. Other ports have been given to other
VM's as host interface devices in KVM. When I upgraded to 6.6 I
noticed that the four port card is now using vfio-pci driver and not
igb as with 6.4.8 did and those VM's using host interfaces didn't
start. I had earlier built 6.5.5 so I tried that and it works same way
as the 6.6 kernel does, so if something has changed it is probably in
6.5 series. I have this in /etc/modprope.d/vfio.conf:

options vfio_pci ids=8086:1521

With  6.4.8 lspci -vv shows this:
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network
Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 67
        IOMMU group: 11
....
        Kernel driver in use: igb
        Kernel modules: igb

And with 6.5.5 I get:
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network
Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 255
        IOMMU group: 11
....
        Kernel driver in use: vfio-pci
        Kernel modules: igb

Have I been just lucky previously with my config or did something
change? I tried to figure out the change from 6.5 release notes but
could not. My home server is running on AMD Ryzen 5700g and Alma Linux
8.8 (I just compile newer kernels out of habit).

Thanks,
Juhani
-- 
Juhani Rautiainen                                   jrauti@iki.fi

Re: Different behavior with vfio-pci between 6.4.8->6.5.5

[Alex Williamson]

On Tue, 31 Oct 2023 17:33:50 +0200
Juhani Rautiainen <jrauti@iki.fi> wrote:

> Hi!
> 
> I noticed a change in my home server which breaks some of KVM VM's
> with newer kernels. I have two Intel I350 cards: one with two ports
> and another with four ports. I have been using the card with two ports
> in a firewall VM with vfio-pci. Other ports have been given to other
> VM's as host interface devices in KVM. When I upgraded to 6.6 I
> noticed that the four port card is now using vfio-pci driver and not
> igb as with 6.4.8 did and those VM's using host interfaces didn't
> start. I had earlier built 6.5.5 so I tried that and it works same way
> as the 6.6 kernel does, so if something has changed it is probably in
> 6.5 series. I have this in /etc/modprope.d/vfio.conf:
> 
> options vfio_pci ids=8086:1521
> 
> With  6.4.8 lspci -vv shows this:
> 01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network
> Connection (rev 01)
>         Subsystem: Intel Corporation Ethernet Server Adapter I350-T4
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 67
>         IOMMU group: 11
> ....
>         Kernel driver in use: igb
>         Kernel modules: igb
> 
> And with 6.5.5 I get:
> 01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network
> Connection (rev 01)
>         Subsystem: Intel Corporation Ethernet Server Adapter I350-T4
>         Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Interrupt: pin A routed to IRQ 255
>         IOMMU group: 11
> ....
>         Kernel driver in use: vfio-pci
>         Kernel modules: igb
> 
> Have I been just lucky previously with my config or did something
> change? I tried to figure out the change from 6.5 release notes but
> could not. My home server is running on AMD Ryzen 5700g and Alma Linux
> 8.8 (I just compile newer kernels out of habit).

The more curious part to me is how your configuration managed to have
some NICs attached to igb and some attached to vfio-pci.  With the
modprobe.d directive, vfio-pci will try to bind to all matching devices
that aren't already bound to a driver.  If igb loads first, all the
devices would bind to igb.  If vfio-pci loads first, all the devices
bind to vfio-pci (do some have a different device ID?).  The vfio-pci
module wouldn't get loaded without something somewhere else requesting
it, so typically igb would claim everything.

Do you launch your VMs with libvirt, which might have automatically
bound the devices to vfio-pci and now there's something loading the
vfio-pci module before igb?

The driverctl tool might be useful for you to specify a specific
driver for specific devices.  Otherwise I'm not sure what kernel change
might have triggered this behavioral change without knowing more about
how and when the vfio-pci module is loaded relative to the igb module.
Thanks,

Alex

Re: Different behavior with vfio-pci between 6.4.8->6.5.5

[Juhani Rautiainen]

 The 2 port card was added during the summer as replacement for the
older Intel card and I probably was lucky with timing as I just fixed
the ID's to modprobe.conf. I checked the dmesg logs and you are
correct about reserving all the interfaces to either one. With 6.4.8
igb gets all the interfaces and with 6.5 onwards vfio-pci wins. So
libvirt with 6.4.8 KVM can reclaim interface from the igb. So maybe I
don't need the modprobe config at all... The base config for libvirt
qemu FW started probably around 2014 when I started using KVM and it
has survived couple of HW and OS upgrades :D. I will try without
modprobe and check that driverctl so I can figure this out.

Thanks for the help,
-Juhani


On Tue, Oct 31, 2023 at 6:19 PM Alex Williamson
<alex.williamson@redhat.com> wrote:
>


> Do you launch your VMs with libvirt, which might have automatically
> bound the devices to vfio-pci and now there's something loading the
> vfio-pci module before igb?

Libvirt is running the show. What I forgot to mention is that 2 port
version i recent replacement in 6.4 the era (old card was intel card
but using e1000 driver). I probably was lucky with timing as I just
fixed the ID's to modprobe.conf and everything continued to work. I
checked the dmesg logs and you are correct about reserving all the
interfaces to either driver. With 6.4.8 igb gets all the interfaces
and with 6.5 onwards vfio-pci wins. Libvirt with 6.4.8 KVM can reclaim
interface from the igb. So maybe I don't need the modprobe config at
all. The base config for the libvirt qemu FW config started around
2014 when I started using KVM and it has survived couple of HW and OS
upgrades.

>
> The driverctl tool might be useful for you to specify a specific
> driver for specific devices.  Otherwise I'm not sure what kernel change
> might have triggered this behavioral change without knowing more about
> how and when the vfio-pci module is loaded relative to the igb module.

I will try without the modprobe and check that driverctl so I can
figure this out.

> Thanks,
>
> Alex
>
Thanks for the help. I half suspected that I messed up something :D.
-Juhani
-- 
Juhani Rautiainen                                   jrauti@iki.fi