Re: The issue about adding multipath device's targets into qemu-pr-helper's namespace

[Lin Ma <lma@suse.de>]

On 2020-07-14 13:30, Lin Ma wrote:
> Hi all,
> 
> I have a namespace question about passthrough disk(multipath device).
> In case of enabling namespace and cgroups in qemu.conf, The target(s) 
> of the
> multipath device won't be added into qemu-pr-helper's namespace under 
> certain
> situation, It causes the PERSISTENT RESERVE command failure in guest.
> 
> While user starts a vm,
> To build namespace, The qemuDomainSetupDisk() will be invoked via 
> threadA(this
> thread id will be the qemu's pid),
> To build cgroup, The qemuSetupImagePathCgroup() will be invoked via 
> threadB.
> 
> Both of the functions invoke the virDevMapperGetTargets() trying to 
> parse a
> multipath device to target paths string, Then fill the targetPaths[].
> 
> The issue I experienced is:
> After libvirtd started, Everything works well for the first booted vm 
> which has
> the passthrough multipath device.
> But If I shut it down & start it again, OR keep it running & start 
> another vm
> which has other passthrough multipath device, Then the target(s) of the 
> fresh
> started vm won't be added into the related qemu-pr-helper's namespace 
> and it
> causes PERSISTENT RESERVE command failure in the corresponding guest.
> I digged into code, In this situation, The targetPaths[] in
> qemuDomainSetupDisk()
> won't be filled, it keeps NULL after virDevMapperGetTargets() returns.
> The virDevMapperGetTargets doesn't fill targetPaths[] because the 
> dm_task_run()
> of libdevmapper returns 0 with errno 9(Bad file descriptor).
> So far, I don't understand why the dm_task_run() return 0 in this 
> situation.
> BTW, The virDevMapperGetTargets() can always successfully fill the 
> targetPaths[]
> in qemuSetupImagePathCgroup().
> 
> Please refer to the following 2 tests:
> The multipath configuration on host:
> host:~ # multipath -l
> vm1-data (3600140582d9024bc13f4b8db5ff12de0) dm-11 FreeNAS,lv68
> size=6.0G features='0' hwhandler='1 alua' wp=rw
> `-+- policy='service-time 0' prio=0 status=active
>   `- 2:0:0:2 sdd 8:48 active undef running
> vm2-data (36001405fc5f29ace3ec4fb8acd32aae5) dm-8 FreeNAS,lv46
> size=4.0G features='0' hwhandler='1 alua' wp=rw
> `-+- policy='service-time 0' prio=0 status=active
>   `- 2:0:0:1 sde 8:64 active undef running
> 
> ===================================================================
> Test A:
> host:~ # systemctl restart libvirtd
> host:~ # virsh list
>  Id   Name   State
> --------------------
> 
> host:~ #
> host:~ # virsh domblklist vm1
>  Target   Source
> ------------------------------------------
>  vda      /opt/vms/vm1/disk0.qcow2
>  sda      /dev/mapper/vm1-data
> 
> host:~ #
> host:~ # virsh start vm1
> Domain vm1 started
> 
> host:~ # virsh list
>  Id   Name        State
> ---------------------------
>  1    vm1        running
> 
> host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
> host:~ # ls -l /dev/sd*
> brw-rw---- 1 root disk 8, 48 Jul 14 16:30 /dev/sdd
> host:~ # exit
> exit
> host:~ #
> 
> vm1:~ # lsscsi
> [0:0:0:0]    disk    FreeNAS  lv68             0123   /dev/sda
> vm1:~ #
> vm1:~ # sg_persist --in -k /dev/sda
>   FreeNAS   lv68              0123
>   Peripheral device type: disk
>   PR generation=0x0, there are NO registered reservation keys
> vm1:~ #
> 
> host:~ # virsh shutdown vm1
> Domain vm1 is being shutdown
> 
> host:~ # virsh list
>  Id   Name   State
> --------------------
> 
> host:~ #
> host:~ # virsh start vm1
> Domain vm1 started
> 
> host:~ # virsh list
>  Id   Name        State
> ---------------------------
>  2    vm1        running
> 
> host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
> host:~ # ls -l /dev/sd*
> ls: cannot access '/dev/sd*': No such file or directory
> host:~ # exit
> exit
> host:~ #
> 
> vm1:~ # sg_persist --in -k /dev/sda
>   FreeNAS   lv68              0123
>   Peripheral device type: disk
> PR in (Read keys): Aborted command
> Aborted command
> vm1:~ #
> ===================================================================
> Test B:
> host:~ # systemctl restart libvirtd
> host:~ # virsh list
>  Id   Name   State
> --------------------
> 
> host:~ #
> host:~ # virsh domblklist vm1
>  Target   Source
> ------------------------------------------
>  vda      /opt/vms/vm1/disk0.qcow2
>  sda      /dev/mapper/vm1-data
> 
> host:~ #
> host:~ # virsh start vm1
> Domain vm1 started
> 
> host:~ # virsh list
>  Id   Name        State
> ---------------------------
>  1    vm1        running
> 
> host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
> host:~ # ls -l /dev/sd*
> brw-rw---- 1 root disk 8, 48 Jul 14 17:28 /dev/sdd
> host:~ # exit
> exit
> host:~ #
> 
> vm1:~ # lsscsi
> [2:0:0:0]    disk    FreeNAS  lv68             0123   /dev/sda
> vm1:~ #
> vm1:~ # sg_persist --in -k /dev/sda
>   FreeNAS   lv68              0123
>   Peripheral device type: disk
>   PR generation=0x0, there are NO registered reservation keys
> vm1:~ #
> 
> host:~ # virsh list
>  Id   Name        State
> ---------------------------
>  1    vm1        running
> 
> host:~ #
> host:~ # virsh domblklist vm2
>  Target   Source
> ------------------------------------------
>  vda      /opt/vms/vm2/disk0.qcow2
>  sda      /dev/mapper/vm2-data
> 
> host:~ #
> host:~ # virsh start vm2
> Domain vm2 started
> 
> host:~ # virsh list
>  Id   Name        State
> ---------------------------
>  1    vm1        running
>  2    vm2        running
> 
> host:~ # nsenter -t $(qemu-pr-helper pid of vm2) -a bash
> host:~ # ls -l /dev/sd*
> ls: cannot access '/dev/sd*': No such file or directory
> host:~ # exit
> exit
> host:~ #
> 
> vm2:~ # lsscsi
> [0:0:0:0]    disk    FreeNAS  lv46             0123   /dev/sda
> vm2:~ #
> vm2:~ # sg_persist --in -k /dev/sda
>   FreeNAS   lv46              0123
>   Peripheral device type: disk
> PR in (Read keys): Aborted command
> Aborted command
> vm2:~ #
> ===================================================================
> 
> Any comments will be much appreciated.
> 
> Thanks in advance,
> Lin

Oops, It shouldn't be sent to qemu ml, It should be sent to libvirt ml, 
Sorry about it.

Lin