From mboxrd@z Thu Jan 1 00:00:00 1970 From: Neo Jia Subject: Re: [PATCH v6 1/4] vfio: Mediated device Core driver Date: Mon, 15 Aug 2016 21:16:55 -0700 Message-ID: <20160816041655.GB19022@nvidia.com> References: <1470251034-1555-2-git-send-email-kwankhede@nvidia.com> <20160809130022.67784f62@t450s.home> <47977462-4a68-f973-a2b8-4c40fbfc9451@nvidia.com> <20160812151608.0db20258@t450s.home> <361a1668-ca11-c46c-7bf6-19c36e703a2d@nvidia.com> <20160815195907.GA14876@nvidia.com> <20160816034551.GA19022@nvidia.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Cc: Kirti Wankhede , Alex Williamson , "pbonzini@redhat.com" , "kraxel@redhat.com" , "qemu-devel@nongnu.org" , "kvm@vger.kernel.org" , "Song, Jike" , "bjsdjshi@linux.vnet.ibm.com" To: "Tian, Kevin" Return-path: Received: from hqemgate16.nvidia.com ([216.228.121.65]:10930 "EHLO hqemgate16.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750917AbcHPEQ5 (ORCPT ); Tue, 16 Aug 2016 00:16:57 -0400 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On Tue, Aug 16, 2016 at 03:50:44AM +0000, Tian, Kevin wrote: > > From: Neo Jia [mailto:cjia@nvidia.com] > > Sent: Tuesday, August 16, 2016 11:46 AM > > > > On Tue, Aug 16, 2016 at 12:30:25AM +0000, Tian, Kevin wrote: > > > > From: Neo Jia [mailto:cjia@nvidia.com] > > > > Sent: Tuesday, August 16, 2016 3:59 AM > > > > > > > > > > > > > > For NVIDIA vGPU solution we need to know all devices assigned to a VM in > > > > > > one shot to commit resources of all vGPUs assigned to a VM along with > > > > > > some common resources. > > > > > > > > > > Kirti, can you elaborate the background about above one-shot commit > > > > > requirement? It's hard to understand such a requirement. > > > > > > > > > > As I relied in another mail, I really hope start/stop become a per-mdev > > > > > attribute instead of global one, e.g.: > > > > > > > > > > echo "0/1" > /sys/class/mdev/12345678-1234-1234-1234-123456789abc/start > > > > > > > > > > In many scenario the user space client may only want to talk to mdev > > > > > instance directly, w/o need to contact its parent device. Still take > > > > > live migration for example, I don't think Qemu wants to know parent > > > > > device of assigned mdev instances. > > > > > > > > Hi Kevin, > > > > > > > > Having a global /sys/class/mdev/mdev_start doesn't require anybody to know > > > > parent device. you can just do > > > > > > > > echo "mdev_UUID" > /sys/class/mdev/mdev_start > > > > > > > > or > > > > > > > > echo "mdev_UUID" > /sys/class/mdev/mdev_stop > > > > > > > > without knowing the parent device. > > > > > > > > > > You can look at some existing sysfs example, e.g.: > > > > > > echo "0/1" > /sys/bus/cpu/devices/cpu1/online > > > > > > You may also argue why not using a global style: > > > > > > echo "cpu1" > /sys/bus/cpu/devices/cpu_online > > > echo "cpu1" > /sys/bus/cpu/devices/cpu_offline > > > > > > There are many similar examples... > > > > Hi Kevin, > > > > My response above is to your question about using the global sysfs entry as you > > don't want to have the global path because > > > > "I don't think Qemu wants to know parent device of assigned mdev instances.". > > > > So I just want to confirm with you that (in case you miss): > > > > /sys/class/mdev/mdev_start | mdev_stop > > > > doesn't require the knowledge of parent device. > > > > Qemu is just one example, where your explanation of parent device > makes sense but still it's not good for Qemu to populate /sys/class/mdev > directly. Qemu is passed with the actual sysfs path of assigned mdev > instance, so any mdev attributes touched by Qemu should be put under > that node (e.g. start/stop for live migration usage as I explained earlier). Exactly, qemu is passed with the actual sysfs path. So, QEMU doesn't touch the file /sys/class/mdev/mdev_start | mdev_stop at all. QEMU will take the sysfs path as input: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/c0b26072-dd1b-4340-84fe-bf338c510818-0,id=vgpu0 As you are saying in live migration, QEMU needs to access "start" and "stop". Could you please share more details, such as how QEMU access the "start" and "stop" sysfs, when and what triggers that? Thanks, Neo > > Thanks > Kevin From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37212) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bZVoF-0001gq-TB for qemu-devel@nongnu.org; Tue, 16 Aug 2016 00:17:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bZVoA-0001K6-Nl for qemu-devel@nongnu.org; Tue, 16 Aug 2016 00:17:02 -0400 Received: from hqemgate16.nvidia.com ([216.228.121.65]:10932) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bZVoA-0001K0-FP for qemu-devel@nongnu.org; Tue, 16 Aug 2016 00:16:58 -0400 Date: Mon, 15 Aug 2016 21:16:55 -0700 From: Neo Jia Message-ID: <20160816041655.GB19022@nvidia.com> References: <1470251034-1555-2-git-send-email-kwankhede@nvidia.com> <20160809130022.67784f62@t450s.home> <47977462-4a68-f973-a2b8-4c40fbfc9451@nvidia.com> <20160812151608.0db20258@t450s.home> <361a1668-ca11-c46c-7bf6-19c36e703a2d@nvidia.com> <20160815195907.GA14876@nvidia.com> <20160816034551.GA19022@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: Subject: Re: [Qemu-devel] [PATCH v6 1/4] vfio: Mediated device Core driver List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Tian, Kevin" Cc: Kirti Wankhede , Alex Williamson , "pbonzini@redhat.com" , "kraxel@redhat.com" , "qemu-devel@nongnu.org" , "kvm@vger.kernel.org" , "Song, Jike" , "bjsdjshi@linux.vnet.ibm.com" On Tue, Aug 16, 2016 at 03:50:44AM +0000, Tian, Kevin wrote: > > From: Neo Jia [mailto:cjia@nvidia.com] > > Sent: Tuesday, August 16, 2016 11:46 AM > > > > On Tue, Aug 16, 2016 at 12:30:25AM +0000, Tian, Kevin wrote: > > > > From: Neo Jia [mailto:cjia@nvidia.com] > > > > Sent: Tuesday, August 16, 2016 3:59 AM > > > > > > > > > > > > > > For NVIDIA vGPU solution we need to know all devices assigned to a VM in > > > > > > one shot to commit resources of all vGPUs assigned to a VM along with > > > > > > some common resources. > > > > > > > > > > Kirti, can you elaborate the background about above one-shot commit > > > > > requirement? It's hard to understand such a requirement. > > > > > > > > > > As I relied in another mail, I really hope start/stop become a per-mdev > > > > > attribute instead of global one, e.g.: > > > > > > > > > > echo "0/1" > /sys/class/mdev/12345678-1234-1234-1234-123456789abc/start > > > > > > > > > > In many scenario the user space client may only want to talk to mdev > > > > > instance directly, w/o need to contact its parent device. Still take > > > > > live migration for example, I don't think Qemu wants to know parent > > > > > device of assigned mdev instances. > > > > > > > > Hi Kevin, > > > > > > > > Having a global /sys/class/mdev/mdev_start doesn't require anybody to know > > > > parent device. you can just do > > > > > > > > echo "mdev_UUID" > /sys/class/mdev/mdev_start > > > > > > > > or > > > > > > > > echo "mdev_UUID" > /sys/class/mdev/mdev_stop > > > > > > > > without knowing the parent device. > > > > > > > > > > You can look at some existing sysfs example, e.g.: > > > > > > echo "0/1" > /sys/bus/cpu/devices/cpu1/online > > > > > > You may also argue why not using a global style: > > > > > > echo "cpu1" > /sys/bus/cpu/devices/cpu_online > > > echo "cpu1" > /sys/bus/cpu/devices/cpu_offline > > > > > > There are many similar examples... > > > > Hi Kevin, > > > > My response above is to your question about using the global sysfs entry as you > > don't want to have the global path because > > > > "I don't think Qemu wants to know parent device of assigned mdev instances.". > > > > So I just want to confirm with you that (in case you miss): > > > > /sys/class/mdev/mdev_start | mdev_stop > > > > doesn't require the knowledge of parent device. > > > > Qemu is just one example, where your explanation of parent device > makes sense but still it's not good for Qemu to populate /sys/class/mdev > directly. Qemu is passed with the actual sysfs path of assigned mdev > instance, so any mdev attributes touched by Qemu should be put under > that node (e.g. start/stop for live migration usage as I explained earlier). Exactly, qemu is passed with the actual sysfs path. So, QEMU doesn't touch the file /sys/class/mdev/mdev_start | mdev_stop at all. QEMU will take the sysfs path as input: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/c0b26072-dd1b-4340-84fe-bf338c510818-0,id=vgpu0 As you are saying in live migration, QEMU needs to access "start" and "stop". Could you please share more details, such as how QEMU access the "start" and "stop" sysfs, when and what triggers that? Thanks, Neo > > Thanks > Kevin