From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kirti Wankhede Subject: Re: [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU. Date: Tue, 2 Feb 2016 14:55:25 +0530 Message-ID: <56B07605.7030905@nvidia.com> References: <56AFD231.3010404@nvidia.com> <56B00AD7.6070103@nvidia.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Cc: "Ruan, Shuai" , "Song, Jike" , "Lv, Zhiyuan" , "kvm@vger.kernel.org" , qemu-devel To: "Tian, Kevin" , Alex Williamson , Neo Jia , Gerd Hoffmann , Paolo Bonzini Return-path: Received: from hqemgate15.nvidia.com ([216.228.121.64]:2093 "EHLO hqemgate15.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754116AbcBBJZh (ORCPT ); Tue, 2 Feb 2016 04:25:37 -0500 In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On 2/2/2016 1:12 PM, Tian, Kevin wrote: >> From: Kirti Wankhede [mailto:kwankhede@nvidia.com] >> Sent: Tuesday, February 02, 2016 9:48 AM >> >> Resending this mail again, somehow my previous mail didn't reached every >> to everyone's inbox. >> >> On 2/2/2016 3:16 AM, Kirti Wankhede wrote: >>> Design for vGPU Driver: >>> Main purpose of vGPU driver is to provide a common interface for vGPU >>> management that can be used by differnt GPU drivers. > > Thanks for composing this design which is a good start. > >>> >>> This module would provide a generic interface to create the device, add >>> it to vGPU bus, add device to IOMMU group and then add it to vfio group. >>> >>> High Level block diagram: >>> >>> >>> +--------------+ vgpu_register_driver()+---------------+ >>> | __init() +------------------------->+ | >>> | | | | >>> | +<-------------------------+ vgpu.ko | >>> | vfio_vgpu.ko | probe()/remove() | | >>> | | +---------+ +---------+ >>> +--------------+ | +-------+-------+ | >>> | ^ | >>> | callback | | >>> | +-------+--------+ | >>> | |vgpu_register_device() | >>> | | | | >>> +---^-----+-----+ +-----+------+-+ >>> | nvidia.ko | | i915.ko | >>> | | | | >>> +-----------+ +------------+ >>> >>> vGPU driver provides two types of registration interfaces: > > Looks you missed callbacks which vgpu.ko provides to vfio_vgpu.ko, > e.g. to retrieve basic region/resource info, etc... > Basic region info or resource info would come from GPU driver. So retrieving such info should be the part of GPU driver interface. Like I mentioned we need to enhance these interfaces during development as and when we find it useful. vfio_vgpu.ko gets dev pointer from which it can reach to vgpu_device structure and then it can use GPU driver interface directly to retrieve such information from GPU driver. This RFC is to focus more on different modules and its structures, how those modules would be inter-linked with each other and have a flexible design to keep the scope for enhancements. We have identified three modules: * vgpu.ko - vGPU core driver that provide registration interface for GPU driver and vGPU VFIO driver, responsible for creating vGPU devices and providing management interface for vGPU devices. * vfio_vgpu.ko - vGPU VFIO driver for vGPU device, provides VFIO interface that is used by QEMU. * vfio_iommu_type1_vgpu.ko - IOMMU TYPE1 driver supporting the IOMMU TYPE1 v1 and v2 interface. The above block diagram gives an overview how vgpu.ko, vfio_vgpu.ko and GPU drivers would be inter-linked with each other. > Also for GPU driver interfaces, better to identify the caller. E.g. it's > easy to understand life-cycle management would come from sysfs > by mgmt. stack like libvirt. What about @read and @write? what's > the connection between this vgpu core driver and specific hypervisor? > etc. Better to connect all necessary dots so we can refine all > necessary requirements on this proposal. > read and write calls are for PCI CFG space amd MMIO space read/write. Read/write access request from QEMU is passed to GPU driver through GPU driver interface. > [...] >>> >>> 2. GPU driver interface >>> >>> /** >>> * struct gpu_device_ops - Structure to be registered for each physical >>> GPU to >>> * register the device to vgpu module. >>> * >>> * @owner: The module owner. >>> * @vgpu_supported_config: Called to get information about supported >>> * vgpu types. >>> * @dev : pci device structure of physical GPU. >>> * @config: should return string listing supported >>> * config >>> * Returns integer: success (0) or error (< 0) >>> * @vgpu_create: Called to allocate basic resouces in graphics >>> * driver for a particular vgpu. >>> * @dev: physical pci device structure on which vgpu >>> * should be created >>> * @uuid: uuid for which VM it is intended to >>> * @instance: vgpu instance in that VM >>> * @vgpu_id: This represents the type of vgpu to be >>> * created >>> * Returns integer: success (0) or error (< 0) > > Specifically for Intel GVT-g we didn't hard partition resource among vGPUs. > Instead we allow user to accurately control how many physical resources > are allocated to a vGPU. So this interface should be extensible to allow > vendor specific resource control. > This interface forwards the create request to vendor/GPU driver informing about which physical GPU this request is intended for and the type of vGPU. Then its vendor/GPU driver's responsibility to do resources allocation and manage resources in their own driver. > And for UUID, I remember Alex had a concern on using it in kernel. > Honestly speaking I don't have a good idea here. In Xen side there is a VM ID > which can be easily used as the index. But for KVM, what would be the best > identifier to associate with a VM? > >>> * @vgpu_destroy: Called to free resources in graphics driver for >>> * a vgpu instance of that VM. >>> * @dev: physical pci device structure to which >>> * this vgpu points to. >>> * @uuid: uuid for which the vgpu belongs to. >>> * @instance: vgpu instance in that VM >>> * Returns integer: success (0) or error (< 0) >>> * If VM is running and vgpu_destroy is called that >>> * means the vGPU is being hotunpluged. Return error >>> * if VM is running and graphics driver doesn't >>> * support vgpu hotplug. >>> * @vgpu_start: Called to do initiate vGPU initialization >>> * process in graphics driver when VM boots before >>> * qemu starts. >>> * @uuid: UUID which is booting. >>> * Returns integer: success (0) or error (< 0) >>> * @vgpu_shutdown: Called to teardown vGPU related resources for >>> * the VM >>> * @uuid: UUID which is shutting down . >>> * Returns integer: success (0) or error (< 0) >>> * @read: Read emulation callback >>> * @vdev: vgpu device structure >>> * @buf: read buffer >>> * @count: number bytes to read >>> * @address_space: specifies for which address space > > and suppose there'll be an 'offset' as required by usual emulation. > Yes, Sorry I missed that in this comment. >>> * the request is: pci_config_space, IO register >>> * space or MMIO space. >>> * Retuns number on bytes read on success or error. >>> * @write: Write emulation callback >>> * @vdev: vgpu device structure >>> * @buf: write buffer >>> * @count: number bytes to be written >>> * @address_space: specifies for which address space >>> * the request is: pci_config_space, IO register >>> * space or MMIO space. >>> * Retuns number on bytes written on success or >>> error. >>> * @vgpu_set_irqs: Called to send about interrupts configuration >>> * information that qemu set. >>> * @vdev: vgpu device structure >>> * @flags, index, start, count and *data : same as >>> * that of struct vfio_irq_set of >>> * VFIO_DEVICE_SET_IRQS API. > > any elaboration how this will be used in your case? > QEMU manage interrupts through VFIO_DEVICE_SET_IRQS ioctl, i.e. configuring, signaling, masking, and unmasking of interrupts. Description of VFIO_DEVICE_SET_IRQS ioctl in include/uapi/linux/vfio.h explains meaning of combination of parameter to VFIO driver. In case of vGPU, interrupts are received by the owner of physical device, i.e. GPU driver. Then GPU driver knows for which vGPU device the interrupt is intended to. So interrupts related information should be available to GPU driver. This interface forwards interrupt related information from vfio_vgpu.ko to GPU driver and GPU driver should accordingly take necessary action. >>> * >>> * Physical GPU that support vGPU should be register with vgpu module with >>> * gpu_device_ops structure. >>> */ >>> > > Also it'd be good design to allow extensible usages, such as statistics, and > other vendor specific control knobs (e.g. foreground/background VM switch > in Intel GVT-g, etc.) > Can you elaborate on what other control knobs that would be needed? Thanks, Kirti. > > Thanks > Kevin > From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36220) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQXDR-0003LV-4l for qemu-devel@nongnu.org; Tue, 02 Feb 2016 04:25:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aQXDN-00032j-T8 for qemu-devel@nongnu.org; Tue, 02 Feb 2016 04:25:41 -0500 Received: from hqemgate15.nvidia.com ([216.228.121.64]:2094) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQXDN-00032O-Gq for qemu-devel@nongnu.org; Tue, 02 Feb 2016 04:25:37 -0500 References: <56AFD231.3010404@nvidia.com> <56B00AD7.6070103@nvidia.com> From: Kirti Wankhede Message-ID: <56B07605.7030905@nvidia.com> Date: Tue, 2 Feb 2016 14:55:25 +0530 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU. List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Tian, Kevin" , Alex Williamson , Neo Jia , Gerd Hoffmann , Paolo Bonzini Cc: "Ruan, Shuai" , "Song, Jike" , "Lv, Zhiyuan" , "kvm@vger.kernel.org" , qemu-devel On 2/2/2016 1:12 PM, Tian, Kevin wrote: >> From: Kirti Wankhede [mailto:kwankhede@nvidia.com] >> Sent: Tuesday, February 02, 2016 9:48 AM >> >> Resending this mail again, somehow my previous mail didn't reached every >> to everyone's inbox. >> >> On 2/2/2016 3:16 AM, Kirti Wankhede wrote: >>> Design for vGPU Driver: >>> Main purpose of vGPU driver is to provide a common interface for vGPU >>> management that can be used by differnt GPU drivers. > > Thanks for composing this design which is a good start. > >>> >>> This module would provide a generic interface to create the device, add >>> it to vGPU bus, add device to IOMMU group and then add it to vfio group. >>> >>> High Level block diagram: >>> >>> >>> +--------------+ vgpu_register_driver()+---------------+ >>> | __init() +------------------------->+ | >>> | | | | >>> | +<-------------------------+ vgpu.ko | >>> | vfio_vgpu.ko | probe()/remove() | | >>> | | +---------+ +---------+ >>> +--------------+ | +-------+-------+ | >>> | ^ | >>> | callback | | >>> | +-------+--------+ | >>> | |vgpu_register_device() | >>> | | | | >>> +---^-----+-----+ +-----+------+-+ >>> | nvidia.ko | | i915.ko | >>> | | | | >>> +-----------+ +------------+ >>> >>> vGPU driver provides two types of registration interfaces: > > Looks you missed callbacks which vgpu.ko provides to vfio_vgpu.ko, > e.g. to retrieve basic region/resource info, etc... > Basic region info or resource info would come from GPU driver. So retrieving such info should be the part of GPU driver interface. Like I mentioned we need to enhance these interfaces during development as and when we find it useful. vfio_vgpu.ko gets dev pointer from which it can reach to vgpu_device structure and then it can use GPU driver interface directly to retrieve such information from GPU driver. This RFC is to focus more on different modules and its structures, how those modules would be inter-linked with each other and have a flexible design to keep the scope for enhancements. We have identified three modules: * vgpu.ko - vGPU core driver that provide registration interface for GPU driver and vGPU VFIO driver, responsible for creating vGPU devices and providing management interface for vGPU devices. * vfio_vgpu.ko - vGPU VFIO driver for vGPU device, provides VFIO interface that is used by QEMU. * vfio_iommu_type1_vgpu.ko - IOMMU TYPE1 driver supporting the IOMMU TYPE1 v1 and v2 interface. The above block diagram gives an overview how vgpu.ko, vfio_vgpu.ko and GPU drivers would be inter-linked with each other. > Also for GPU driver interfaces, better to identify the caller. E.g. it's > easy to understand life-cycle management would come from sysfs > by mgmt. stack like libvirt. What about @read and @write? what's > the connection between this vgpu core driver and specific hypervisor? > etc. Better to connect all necessary dots so we can refine all > necessary requirements on this proposal. > read and write calls are for PCI CFG space amd MMIO space read/write. Read/write access request from QEMU is passed to GPU driver through GPU driver interface. > [...] >>> >>> 2. GPU driver interface >>> >>> /** >>> * struct gpu_device_ops - Structure to be registered for each physical >>> GPU to >>> * register the device to vgpu module. >>> * >>> * @owner: The module owner. >>> * @vgpu_supported_config: Called to get information about supported >>> * vgpu types. >>> * @dev : pci device structure of physical GPU. >>> * @config: should return string listing supported >>> * config >>> * Returns integer: success (0) or error (< 0) >>> * @vgpu_create: Called to allocate basic resouces in graphics >>> * driver for a particular vgpu. >>> * @dev: physical pci device structure on which vgpu >>> * should be created >>> * @uuid: uuid for which VM it is intended to >>> * @instance: vgpu instance in that VM >>> * @vgpu_id: This represents the type of vgpu to be >>> * created >>> * Returns integer: success (0) or error (< 0) > > Specifically for Intel GVT-g we didn't hard partition resource among vGPUs. > Instead we allow user to accurately control how many physical resources > are allocated to a vGPU. So this interface should be extensible to allow > vendor specific resource control. > This interface forwards the create request to vendor/GPU driver informing about which physical GPU this request is intended for and the type of vGPU. Then its vendor/GPU driver's responsibility to do resources allocation and manage resources in their own driver. > And for UUID, I remember Alex had a concern on using it in kernel. > Honestly speaking I don't have a good idea here. In Xen side there is a VM ID > which can be easily used as the index. But for KVM, what would be the best > identifier to associate with a VM? > >>> * @vgpu_destroy: Called to free resources in graphics driver for >>> * a vgpu instance of that VM. >>> * @dev: physical pci device structure to which >>> * this vgpu points to. >>> * @uuid: uuid for which the vgpu belongs to. >>> * @instance: vgpu instance in that VM >>> * Returns integer: success (0) or error (< 0) >>> * If VM is running and vgpu_destroy is called that >>> * means the vGPU is being hotunpluged. Return error >>> * if VM is running and graphics driver doesn't >>> * support vgpu hotplug. >>> * @vgpu_start: Called to do initiate vGPU initialization >>> * process in graphics driver when VM boots before >>> * qemu starts. >>> * @uuid: UUID which is booting. >>> * Returns integer: success (0) or error (< 0) >>> * @vgpu_shutdown: Called to teardown vGPU related resources for >>> * the VM >>> * @uuid: UUID which is shutting down . >>> * Returns integer: success (0) or error (< 0) >>> * @read: Read emulation callback >>> * @vdev: vgpu device structure >>> * @buf: read buffer >>> * @count: number bytes to read >>> * @address_space: specifies for which address space > > and suppose there'll be an 'offset' as required by usual emulation. > Yes, Sorry I missed that in this comment. >>> * the request is: pci_config_space, IO register >>> * space or MMIO space. >>> * Retuns number on bytes read on success or error. >>> * @write: Write emulation callback >>> * @vdev: vgpu device structure >>> * @buf: write buffer >>> * @count: number bytes to be written >>> * @address_space: specifies for which address space >>> * the request is: pci_config_space, IO register >>> * space or MMIO space. >>> * Retuns number on bytes written on success or >>> error. >>> * @vgpu_set_irqs: Called to send about interrupts configuration >>> * information that qemu set. >>> * @vdev: vgpu device structure >>> * @flags, index, start, count and *data : same as >>> * that of struct vfio_irq_set of >>> * VFIO_DEVICE_SET_IRQS API. > > any elaboration how this will be used in your case? > QEMU manage interrupts through VFIO_DEVICE_SET_IRQS ioctl, i.e. configuring, signaling, masking, and unmasking of interrupts. Description of VFIO_DEVICE_SET_IRQS ioctl in include/uapi/linux/vfio.h explains meaning of combination of parameter to VFIO driver. In case of vGPU, interrupts are received by the owner of physical device, i.e. GPU driver. Then GPU driver knows for which vGPU device the interrupt is intended to. So interrupts related information should be available to GPU driver. This interface forwards interrupt related information from vfio_vgpu.ko to GPU driver and GPU driver should accordingly take necessary action. >>> * >>> * Physical GPU that support vGPU should be register with vgpu module with >>> * gpu_device_ops structure. >>> */ >>> > > Also it'd be good design to allow extensible usages, such as statistics, and > other vendor specific control knobs (e.g. foreground/background VM switch > in Intel GVT-g, etc.) > Can you elaborate on what other control knobs that would be needed? Thanks, Kirti. > > Thanks > Kevin >