[Intel-wired-lan] [bug report]iavf: deadlock on detach/attach vf to VM with net_failover

[Intel-wired-lan] [bug report]iavf: deadlock on detach/attach vf to VM with net_failover

[Faicker Mo]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]

Hello, we met a problem with iavf driver, the ubuntu22.04 kernel 5.15.0-119 and 6.11rc4 both have the same problem.

When the X710 VF is detached and attached to the VM with net_failover, the kworker thread in VM hang. The backtrace is,
[<0>] iavf_open+0x48/0x230 [iavf]
[<0>] __dev_open+0xe5/0x1c0
[<0>] dev_open+0x52/0xa0
[<0>] net_failover_slave_register+0x131/0x190 [net_failover]
[<0>] failover_slave_register.part.0+0xda/0x180 [failover]
[<0>] failover_event+0x104/0x1f0 [failover]
[<0>] raw_notifier_call_chain+0x46/0x60
[<0>] call_netdevice_notifiers_info+0x52/0xa0
[<0>] register_netdevice+0x4d3/0x650
[<0>] iavf_finish_config+0xc5/0xf0 [iavf]
[<0>] process_one_work+0x228/0x3d0
[<0>] worker_thread+0x53/0x420
[<0>] kthread+0x127/0x150
[<0>] ret_from_fork+0x1f/0x30

It looks like a dead lock of adapter->crit_lock.
A similar problem, https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20220808175845.484968-1-ivecera@redhat.com/#2947762

[-- Attachment #2: Type: text/html, Size: 4075 bytes --]

Re: [Intel-wired-lan] [bug report]iavf: deadlock on detach/attach vf to VM with net_failover

[Glaza, Jan]

On 8/26/2024 5:19 AM, Faicker Mo wrote:
> Hello, we met a problem with iavf driver, the ubuntu22.04 kernel 5.15.0-119 and 6.11rc4 both have the same problem.
> 
>  
> 
> When the X710 VF is detached and attached to the VM with net_failover, the kworker thread in VM hang. The backtrace is
...

> It looks like a dead lock of adapter->crit_lock.
> A similar problem, https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20220808175845.484968-1-ivecera@redhat.com/#2947762 <https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20220808175845.484968-1-ivecera@redhat.com/#2947762>
> 

Hello, we are attempting local reproduction and will work to analyze and 
fix the issue. If you are able to, could you please provide us with the 
reproduction commands or script to facilitate easier investigation and 
rootcausing?

Best regards, 
Jan

Re: [Intel-wired-lan] [EXTERNAL] Re: [bug report]iavf: deadlock on detach/attach vf to VM with net_failover

[Faicker Mo]

[-- Attachment #1: Type: text/plain, Size: 1233 bytes --]

>Hello, we are attempting local reproduction and will work to analyze and
>fix the issue. If you are able to, could you please provide us with the
>reproduction commands or script to facilitate easier investigation and
>rootcausing?

1.Enable SRIOV/iommu on the host.
2.Install the libvirt and qemu on the host.
3.Create VFs on host.
4.Download a ubuntu22.04 cloud image(maybe need change password with virt-customize) and Define a xml and start the VM,
the net_failover interface part on this link, https://docs.kernel.org/networking/net_failover.html
The mainline kernel download from this link, https://kernel.ubuntu.com/mainline/?C=N;O=D
5.A New xml file vf.xml with this content,
<interface type='hostdev' managed='yes'>
  <mac address='52:54:00:00:12:53'/>
  <source>
    <address type='pci' domain='0x0000' bus='0x42' slot='0x02' function='0x5'/>
  </source>
  <teaming type='transient' persistent='ua-backup0'/>
</interface>
6.Execute virsh console vm0 to login to the VM on a new terminal
Detach the VF and attach the VF
virsh detach-device vm0 a.xml
virsh attach-device vm0 a.xml
7.In the VM, the kworker hangs.
Get the process id, ps -ef | grep 'kworker.*iavf'
Get the stack, cat /proc/275/stack

[-- Attachment #2: Type: text/html, Size: 4655 bytes --]

Re: [Intel-wired-lan] [EXTERNAL] Re: [bug report]iavf: deadlock on detach/attach vf to VM with net_failover

[Glaza, Jan]

We were unable to reproduce it using the following configuration and steps:
HOST:
  PF Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572]
  OS: Ubuntu 22.04.4 LTS 6.11.0-061100rc4-generic
  FW firmware-version: 9.50 0x8000f177 1.3597.0
  PF version: 6.11.0-061100rc4-generic
VM:
  PF Intel Corporation Ethernet Virtual Function 700 Series [8086:154c]
  OS: Ubuntu 22.04.2 LTS 6.11.0-061100rc4-generic
  PF version: 6.11.0-061100rc4-generic

Steps:
1. Spawn 1 VF
2. Use script in order to get xml of the vf, it should spawn in /tmp/temp_vf_config/
#!/bin/bash
temp_dir="/tmp/temp_vf_config"

if [ -z "$temp_dir" ];
then
    echo "temp_dir variable is empty, aborting to avoid destroying filesystem"
    exit 1
else
    rm -rf $temp_dir/*
    mkdir -p $temp_dir >/dev/null 2>&1
fi
for VF in $(lspci | grep -i "virtual function" | cut -f 1 -d \ );
do
    busid=$(echo $VF | cut -f 1 -d \:);
    slotid=$(echo $VF | cut -f 2 -d \: | cut -f 1 -d \.);
    functionid=$(echo $VF | cut -f 2 -d \: | cut -f 2 -d \.);
    echo "BUS: $busid, SLOT: $slotid, FUNCTION: $functionid"

echo -e "<hostdev mode='subsystem' type='pci' managed='yes'>\n<source>\n<address domain='0x0000' bus='0x$busid' slot='0x$slotid' function='0x$functionid'/>\n</source>\n</hostdev>" > $temp_dir/$busid-$slotid-$functionid

done

3. Create VM
4. Attach VF to VM while VM is disabled, then enable VM
5. Hot detach VF
6. Hot attach VF

Expected result:
VF detaches and attaches successfully

Actual result:
VF does not detach/attach at all.
Possible to set down/up VF when it's detached.
No error in dmesg, actually nothing appear in dmesg.

We had to replace the XML file provided as it would not produce a working configuration. Can you advise us on any setup inconsistencies or changes that would be required to reproduce it?


Best regards, 
Jan

Re: [Intel-wired-lan] [EXTERNAL] Re: [EXTERNAL] Re: [bug report]iavf: deadlock on detach/attach vf to VM with net_failover

[Faicker Mo]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

Should setup a net failover device.
The xml like this,
<interface type='network'>
  <mac address='52:54:00:00:12:53'/>
  <source network=’virbr0’ />
  <model type='virtio'/>
  <driver name='vhost' queues='4'/>
  <link state='down'/>
  <teaming type='persistent'/>
  <alias name='ua-backup0'/>
</interface>
<interface type='hostdev' managed='yes'>
  <mac address='52:54:00:00:12:53'/>
  <source>
    <address type='pci' domain='0x0000' bus='0x42' slot='0x02' function='0x5'/>
  </source>
  <teaming type='transient' persistent='ua-backup0'/>
</interface>

After the VM booted, detach the VF and attach the VF.
The vf.xml like this,
<interface type='hostdev' managed='yes'>
  <mac address='52:54:00:00:12:53'/>
  <source>
    <address type='pci' domain='0x0000' bus='0x42' slot='0x02' function='0x5'/>
  </source>
  <teaming type='transient' persistent='ua-backup0'/>
</interface>

Command like this, virsh detach-device $domain vf.xml ; , virsh attach-device $domain vf.xml
The VM hangs.

--
Best Regards,
Faicker


[-- Attachment #2: Type: text/html, Size: 5545 bytes --]