* The issue about adding multipath device's targets into qemu-pr-helper's namespace
@ 2020-07-14 13:30 Lin Ma
2020-07-14 13:38 ` Lin Ma
0 siblings, 1 reply; 2+ messages in thread
From: Lin Ma @ 2020-07-14 13:30 UTC (permalink / raw)
To: qemu-devel; +Cc: mprivozn
Hi all,
I have a namespace question about passthrough disk(multipath device).
In case of enabling namespace and cgroups in qemu.conf, The target(s) of
the
multipath device won't be added into qemu-pr-helper's namespace under
certain
situation, It causes the PERSISTENT RESERVE command failure in guest.
While user starts a vm,
To build namespace, The qemuDomainSetupDisk() will be invoked via
threadA(this
thread id will be the qemu's pid),
To build cgroup, The qemuSetupImagePathCgroup() will be invoked via
threadB.
Both of the functions invoke the virDevMapperGetTargets() trying to
parse a
multipath device to target paths string, Then fill the targetPaths[].
The issue I experienced is:
After libvirtd started, Everything works well for the first booted vm
which has
the passthrough multipath device.
But If I shut it down & start it again, OR keep it running & start
another vm
which has other passthrough multipath device, Then the target(s) of the
fresh
started vm won't be added into the related qemu-pr-helper's namespace
and it
causes PERSISTENT RESERVE command failure in the corresponding guest.
I digged into code, In this situation, The targetPaths[] in
qemuDomainSetupDisk()
won't be filled, it keeps NULL after virDevMapperGetTargets() returns.
The virDevMapperGetTargets doesn't fill targetPaths[] because the
dm_task_run()
of libdevmapper returns 0 with errno 9(Bad file descriptor).
So far, I don't understand why the dm_task_run() return 0 in this
situation.
BTW, The virDevMapperGetTargets() can always successfully fill the
targetPaths[]
in qemuSetupImagePathCgroup().
Please refer to the following 2 tests:
The multipath configuration on host:
host:~ # multipath -l
vm1-data (3600140582d9024bc13f4b8db5ff12de0) dm-11 FreeNAS,lv68
size=6.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
`- 2:0:0:2 sdd 8:48 active undef running
vm2-data (36001405fc5f29ace3ec4fb8acd32aae5) dm-8 FreeNAS,lv46
size=4.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
`- 2:0:0:1 sde 8:64 active undef running
===================================================================
Test A:
host:~ # systemctl restart libvirtd
host:~ # virsh list
Id Name State
--------------------
host:~ #
host:~ # virsh domblklist vm1
Target Source
------------------------------------------
vda /opt/vms/vm1/disk0.qcow2
sda /dev/mapper/vm1-data
host:~ #
host:~ # virsh start vm1
Domain vm1 started
host:~ # virsh list
Id Name State
---------------------------
1 vm1 running
host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 48 Jul 14 16:30 /dev/sdd
host:~ # exit
exit
host:~ #
vm1:~ # lsscsi
[0:0:0:0] disk FreeNAS lv68 0123 /dev/sda
vm1:~ #
vm1:~ # sg_persist --in -k /dev/sda
FreeNAS lv68 0123
Peripheral device type: disk
PR generation=0x0, there are NO registered reservation keys
vm1:~ #
host:~ # virsh shutdown vm1
Domain vm1 is being shutdown
host:~ # virsh list
Id Name State
--------------------
host:~ #
host:~ # virsh start vm1
Domain vm1 started
host:~ # virsh list
Id Name State
---------------------------
2 vm1 running
host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory
host:~ # exit
exit
host:~ #
vm1:~ # sg_persist --in -k /dev/sda
FreeNAS lv68 0123
Peripheral device type: disk
PR in (Read keys): Aborted command
Aborted command
vm1:~ #
===================================================================
Test B:
host:~ # systemctl restart libvirtd
host:~ # virsh list
Id Name State
--------------------
host:~ #
host:~ # virsh domblklist vm1
Target Source
------------------------------------------
vda /opt/vms/vm1/disk0.qcow2
sda /dev/mapper/vm1-data
host:~ #
host:~ # virsh start vm1
Domain vm1 started
host:~ # virsh list
Id Name State
---------------------------
1 vm1 running
host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 48 Jul 14 17:28 /dev/sdd
host:~ # exit
exit
host:~ #
vm1:~ # lsscsi
[2:0:0:0] disk FreeNAS lv68 0123 /dev/sda
vm1:~ #
vm1:~ # sg_persist --in -k /dev/sda
FreeNAS lv68 0123
Peripheral device type: disk
PR generation=0x0, there are NO registered reservation keys
vm1:~ #
host:~ # virsh list
Id Name State
---------------------------
1 vm1 running
host:~ #
host:~ # virsh domblklist vm2
Target Source
------------------------------------------
vda /opt/vms/vm2/disk0.qcow2
sda /dev/mapper/vm2-data
host:~ #
host:~ # virsh start vm2
Domain vm2 started
host:~ # virsh list
Id Name State
---------------------------
1 vm1 running
2 vm2 running
host:~ # nsenter -t $(qemu-pr-helper pid of vm2) -a bash
host:~ # ls -l /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory
host:~ # exit
exit
host:~ #
vm2:~ # lsscsi
[0:0:0:0] disk FreeNAS lv46 0123 /dev/sda
vm2:~ #
vm2:~ # sg_persist --in -k /dev/sda
FreeNAS lv46 0123
Peripheral device type: disk
PR in (Read keys): Aborted command
Aborted command
vm2:~ #
===================================================================
Any comments will be much appreciated.
Thanks in advance,
Lin
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: The issue about adding multipath device's targets into qemu-pr-helper's namespace
2020-07-14 13:30 The issue about adding multipath device's targets into qemu-pr-helper's namespace Lin Ma
@ 2020-07-14 13:38 ` Lin Ma
0 siblings, 0 replies; 2+ messages in thread
From: Lin Ma @ 2020-07-14 13:38 UTC (permalink / raw)
To: qemu-devel; +Cc: mprivozn
On 2020-07-14 13:30, Lin Ma wrote:
> Hi all,
>
> I have a namespace question about passthrough disk(multipath device).
> In case of enabling namespace and cgroups in qemu.conf, The target(s)
> of the
> multipath device won't be added into qemu-pr-helper's namespace under
> certain
> situation, It causes the PERSISTENT RESERVE command failure in guest.
>
> While user starts a vm,
> To build namespace, The qemuDomainSetupDisk() will be invoked via
> threadA(this
> thread id will be the qemu's pid),
> To build cgroup, The qemuSetupImagePathCgroup() will be invoked via
> threadB.
>
> Both of the functions invoke the virDevMapperGetTargets() trying to
> parse a
> multipath device to target paths string, Then fill the targetPaths[].
>
> The issue I experienced is:
> After libvirtd started, Everything works well for the first booted vm
> which has
> the passthrough multipath device.
> But If I shut it down & start it again, OR keep it running & start
> another vm
> which has other passthrough multipath device, Then the target(s) of the
> fresh
> started vm won't be added into the related qemu-pr-helper's namespace
> and it
> causes PERSISTENT RESERVE command failure in the corresponding guest.
> I digged into code, In this situation, The targetPaths[] in
> qemuDomainSetupDisk()
> won't be filled, it keeps NULL after virDevMapperGetTargets() returns.
> The virDevMapperGetTargets doesn't fill targetPaths[] because the
> dm_task_run()
> of libdevmapper returns 0 with errno 9(Bad file descriptor).
> So far, I don't understand why the dm_task_run() return 0 in this
> situation.
> BTW, The virDevMapperGetTargets() can always successfully fill the
> targetPaths[]
> in qemuSetupImagePathCgroup().
>
> Please refer to the following 2 tests:
> The multipath configuration on host:
> host:~ # multipath -l
> vm1-data (3600140582d9024bc13f4b8db5ff12de0) dm-11 FreeNAS,lv68
> size=6.0G features='0' hwhandler='1 alua' wp=rw
> `-+- policy='service-time 0' prio=0 status=active
> `- 2:0:0:2 sdd 8:48 active undef running
> vm2-data (36001405fc5f29ace3ec4fb8acd32aae5) dm-8 FreeNAS,lv46
> size=4.0G features='0' hwhandler='1 alua' wp=rw
> `-+- policy='service-time 0' prio=0 status=active
> `- 2:0:0:1 sde 8:64 active undef running
>
> ===================================================================
> Test A:
> host:~ # systemctl restart libvirtd
> host:~ # virsh list
> Id Name State
> --------------------
>
> host:~ #
> host:~ # virsh domblklist vm1
> Target Source
> ------------------------------------------
> vda /opt/vms/vm1/disk0.qcow2
> sda /dev/mapper/vm1-data
>
> host:~ #
> host:~ # virsh start vm1
> Domain vm1 started
>
> host:~ # virsh list
> Id Name State
> ---------------------------
> 1 vm1 running
>
> host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
> host:~ # ls -l /dev/sd*
> brw-rw---- 1 root disk 8, 48 Jul 14 16:30 /dev/sdd
> host:~ # exit
> exit
> host:~ #
>
> vm1:~ # lsscsi
> [0:0:0:0] disk FreeNAS lv68 0123 /dev/sda
> vm1:~ #
> vm1:~ # sg_persist --in -k /dev/sda
> FreeNAS lv68 0123
> Peripheral device type: disk
> PR generation=0x0, there are NO registered reservation keys
> vm1:~ #
>
> host:~ # virsh shutdown vm1
> Domain vm1 is being shutdown
>
> host:~ # virsh list
> Id Name State
> --------------------
>
> host:~ #
> host:~ # virsh start vm1
> Domain vm1 started
>
> host:~ # virsh list
> Id Name State
> ---------------------------
> 2 vm1 running
>
> host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
> host:~ # ls -l /dev/sd*
> ls: cannot access '/dev/sd*': No such file or directory
> host:~ # exit
> exit
> host:~ #
>
> vm1:~ # sg_persist --in -k /dev/sda
> FreeNAS lv68 0123
> Peripheral device type: disk
> PR in (Read keys): Aborted command
> Aborted command
> vm1:~ #
> ===================================================================
> Test B:
> host:~ # systemctl restart libvirtd
> host:~ # virsh list
> Id Name State
> --------------------
>
> host:~ #
> host:~ # virsh domblklist vm1
> Target Source
> ------------------------------------------
> vda /opt/vms/vm1/disk0.qcow2
> sda /dev/mapper/vm1-data
>
> host:~ #
> host:~ # virsh start vm1
> Domain vm1 started
>
> host:~ # virsh list
> Id Name State
> ---------------------------
> 1 vm1 running
>
> host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
> host:~ # ls -l /dev/sd*
> brw-rw---- 1 root disk 8, 48 Jul 14 17:28 /dev/sdd
> host:~ # exit
> exit
> host:~ #
>
> vm1:~ # lsscsi
> [2:0:0:0] disk FreeNAS lv68 0123 /dev/sda
> vm1:~ #
> vm1:~ # sg_persist --in -k /dev/sda
> FreeNAS lv68 0123
> Peripheral device type: disk
> PR generation=0x0, there are NO registered reservation keys
> vm1:~ #
>
> host:~ # virsh list
> Id Name State
> ---------------------------
> 1 vm1 running
>
> host:~ #
> host:~ # virsh domblklist vm2
> Target Source
> ------------------------------------------
> vda /opt/vms/vm2/disk0.qcow2
> sda /dev/mapper/vm2-data
>
> host:~ #
> host:~ # virsh start vm2
> Domain vm2 started
>
> host:~ # virsh list
> Id Name State
> ---------------------------
> 1 vm1 running
> 2 vm2 running
>
> host:~ # nsenter -t $(qemu-pr-helper pid of vm2) -a bash
> host:~ # ls -l /dev/sd*
> ls: cannot access '/dev/sd*': No such file or directory
> host:~ # exit
> exit
> host:~ #
>
> vm2:~ # lsscsi
> [0:0:0:0] disk FreeNAS lv46 0123 /dev/sda
> vm2:~ #
> vm2:~ # sg_persist --in -k /dev/sda
> FreeNAS lv46 0123
> Peripheral device type: disk
> PR in (Read keys): Aborted command
> Aborted command
> vm2:~ #
> ===================================================================
>
> Any comments will be much appreciated.
>
> Thanks in advance,
> Lin
Oops, It shouldn't be sent to qemu ml, It should be sent to libvirt ml,
Sorry about it.
Lin
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2020-07-14 13:40 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-07-14 13:30 The issue about adding multipath device's targets into qemu-pr-helper's namespace Lin Ma
2020-07-14 13:38 ` Lin Ma
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).