From mboxrd@z Thu Jan 1 00:00:00 1970 From: Xiao Guangrong Subject: Re: [Qemu-devel] [PATCH v2 06/11] nvdimm acpi: initialize the resource used by NVDIMM ACPI Date: Wed, 17 Feb 2016 10:04:18 +0800 Message-ID: <56C3D522.6090401@linux.intel.com> References: <1452624610-46945-1-git-send-email-guangrong.xiao@linux.intel.com> <1452624610-46945-7-git-send-email-guangrong.xiao@linux.intel.com> <20160208120337.44720b0b@nial.brq.redhat.com> <56C01747.3030102@linux.intel.com> <20160215101105.47e9245a@nial.brq.redhat.com> <20160215111705-mutt-send-email-mst@redhat.com> <56C1A4D2.3060402@linux.intel.com> <20160215114742.382c951e@nial.brq.redhat.com> <20160215133722-mutt-send-email-mst@redhat.com> <20160215143234.29320a5f@nial.brq.redhat.com> <56C1F469.2040602@linux.intel.com> <20160215182404.0878474f@nial.brq.redhat.com> <56C21A7D.5040902@linux.intel.com> <20160216120047.5a50eccf@nial.brq.redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: ehabkost@redhat.com, kvm@vger.kernel.org, "Michael S. Tsirkin" , gleb@kernel.org, mtosatti@redhat.com, qemu-devel@nongnu.org, stefanha@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, rth@twiddle.net To: Igor Mammedov Return-path: Received: from mga02.intel.com ([134.134.136.20]:48651 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S964967AbcBQCMU (ORCPT ); Tue, 16 Feb 2016 21:12:20 -0500 In-Reply-To: <20160216120047.5a50eccf@nial.brq.redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On 02/16/2016 07:00 PM, Igor Mammedov wrote: > On Tue, 16 Feb 2016 02:35:41 +0800 > Xiao Guangrong wrote: > >> On 02/16/2016 01:24 AM, Igor Mammedov wrote: >>> On Mon, 15 Feb 2016 23:53:13 +0800 >>> Xiao Guangrong wrote: >>> >>>> On 02/15/2016 09:32 PM, Igor Mammedov wrote: >>>>> On Mon, 15 Feb 2016 13:45:59 +0200 >>>>> "Michael S. Tsirkin" wrote: >>>>> >>>>>> On Mon, Feb 15, 2016 at 11:47:42AM +0100, Igor Mammedov wrote: >>>>>>> On Mon, 15 Feb 2016 18:13:38 +0800 >>>>>>> Xiao Guangrong wrote: >>>>>>> >>>>>>>> On 02/15/2016 05:18 PM, Michael S. Tsirkin wrote: >>>>>>>>> On Mon, Feb 15, 2016 at 10:11:05AM +0100, Igor Mammedov wrote= : >>>>>>>>>> On Sun, 14 Feb 2016 13:57:27 +0800 >>>>>>>>>> Xiao Guangrong wrote: >>>>>>>>>> >>>>>>>>>>> On 02/08/2016 07:03 PM, Igor Mammedov wrote: >>>>>>>>>>>> On Wed, 13 Jan 2016 02:50:05 +0800 >>>>>>>>>>>> Xiao Guangrong wrote: >>>>>>>>>>>> >>>>>>>>>>>>> 32 bits IO port starting from 0x0a18 in guest is reserved= for NVDIMM >>>>>>>>>>>>> ACPI emulation. The table, NVDIMM_DSM_MEM_FILE, will be p= atched into >>>>>>>>>>>>> NVDIMM ACPI binary code >>>>>>>>>>>>> >>>>>>>>>>>>> OSPM uses this port to tell QEMU the final address of the= DSM memory >>>>>>>>>>>>> and notify QEMU to emulate the DSM method >>>>>>>>>>>> Would you need to pass control to QEMU if each NVDIMM had = its whole >>>>>>>>>>>> label area MemoryRegion mapped right after its storage Mem= oryRegion? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> No, label data is not mapped into guest's address space and= it only >>>>>>>>>>> can be accessed by DSM method indirectly. >>>>>>>>>> Yep, per spec label data should be accessed via _DSM but que= stion >>>>>>>>>> wasn't about it, >>>>>>>> >>>>>>>> Ah, sorry, i missed your question. >>>>>>>> >>>>>>>>>> Why would one map only 4Kb window and serialize label data >>>>>>>>>> via it if it could be mapped as whole, that way _DMS method = will be >>>>>>>>>> much less complicated and there won't be need to add/support= a protocol >>>>>>>>>> for its serialization. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Is it ever accessed on data path? If not I prefer the current= approach: >>>>>>>> >>>>>>>> The label data is only accessed via two DSM commands - Get Nam= espace Label >>>>>>>> Data and Set Namespace Label Data, no other place need to be e= mulated. >>>>>>>> >>>>>>>>> limit the window used, the serialization protocol seems rathe= r simple. >>>>>>>>> >>>>>>>> >>>>>>>> Yes. >>>>>>>> >>>>>>>> Label data is at least 128k which is big enough for BIOS as it= allocates >>>>>>>> memory at 0 ~ 4G which is tight region. It also needs guest OS= to support >>>>>>>> lager max-xfer (the max size that can be transferred one time)= , the size >>>>>>>> in current Linux NVDIMM driver is 4k. >>>>>>>> >>>>>>>> However, using lager DSM buffer can help us to simplify NVDIMM= hotplug for >>>>>>>> the case that too many nvdimm devices present in the system an= d their FIT >>>>>>>> info can not be filled into one page. Each PMEM-only device ne= eds 0xb8 bytes >>>>>>>> and we can append 256 memory devices at most, so 12 pages are = needed to >>>>>>>> contain this info. The prototype we implemented is using ourse= lf-defined >>>>>>>> protocol to read piece of _FIT and concatenate them before ret= urn to Guest, >>>>>>>> please refer to: >>>>>>>> https://github.com/xiaogr/qemu/commit/c46ce01c8433ac0870670304= 360b3c4aa414143a >>>>>>>> >>>>>>>> As 12 pages are not small region for BIOS and the _FIT size ma= y be extended in the >>>>>>>> future development (eg, if PBLK is introduced) i am not sure i= f we need this. Of >>>>>>>> course, another approach to simplify it is that we limit the n= umber of NVDIMM >>>>>>>> device to make sure their _FIT < 4k. >>>>>>> My suggestion is not to have only one label area for every NVDI= MM but >>>>>>> rather to map each label area right after each NVDIMM's data me= mory. >>>>>>> That way _DMS can be made non-serialized and guest could handle >>>>>>> label data in parallel. >>>>>> >>>>>> I think that alignment considerations would mean we are burning = up >>>>>> 1G of phys address space for this. For PAE we only have 64G >>>>>> of this address space, so this would be a problem. >>>>> That's true that it will burning away address space, however that >>>>> just means that PAE guests would not be able to handle as many >>>>> NVDIMMs as 64bit guests. The same applies to DIMMs as well, with >>>>> alignment enforced. If one needs more DIMMs he/she can switch >>>>> to 64bit guest to use them. >>>>> >>>>> It's trade of inefficient GPA consumption vs efficient NVDIMMs ac= cess. >>>>> Also with fully mapped label area for each NVDIMM we don't have t= o >>>>> introduce and maintain any guest visible serialization protocol >>>>> (protocol for serializing _DSM via 4K window) which becomes ABI. >>>> >>>> It's true for label access but it is not for the long term as we w= ill >>>> need to support other _DSM commands such as vendor specific comman= d, >>>> PBLK dsm command, also NVDIMM MCE related commands will be introdu= ced >>>> in the future, so we will come back here at that time. :( >>> I believe for block mode NVDIMM would also need per NVDIMM mapping >>> for performance reasons (parallel access). >>> As for the rest could that commands go via MMIO that we usually >>> use for control path? >> >> So both input data and output data go through single MMIO, we need t= o >> introduce a protocol to pass these data, that is complex? >> >> And is any MMIO we can reuse (more complexer=EF=BC=9F) or we should = allocate this >> MMIO page =EF=BC=88the old question - where to allocated?=EF=BC=89? > Maybe you could reuse/extend memhotplug IO interface, > or alternatively as Michael suggested add a vendor specific PCI_Confi= g, > I'd suggest PM device for that (hw/acpi/[piix4.c|ihc9.c]) > which I like even better since you won't need to care about which por= ts > to allocate at all. Well, if Michael does not object, i will do it in the next version. :)