From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:55039) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S75Wi-0006TU-5v for qemu-devel@nongnu.org; Mon, 12 Mar 2012 09:43:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1S75Wg-00019j-2f for qemu-devel@nongnu.org; Mon, 12 Mar 2012 09:43:03 -0400 Received: from smtp.citrix.com ([66.165.176.89]:50101) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S75Wf-00019D-UH for qemu-devel@nongnu.org; Mon, 12 Mar 2012 09:43:01 -0400 Message-ID: <4F5DFD32.1020107@citrix.com> Date: Mon, 12 Mar 2012 13:42:10 +0000 From: Julien Grall MIME-Version: 1.0 References: <4F4CBE85.5000006@citrix.com> <1330985175.22559.60.camel@cthulhu.hellion.org.uk> In-Reply-To: <1330985175.22559.60.camel@cthulhu.hellion.org.uk> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [Xen-devel] Qemu disaggregation in Xen environment List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Ian Campbell Cc: "xen-devel@lists.xensource.com" , "qemu-devel@nongnu.org" , Stefano Stabellini , Julian Pidancet On 03/05/2012 10:06 PM, Ian Campbell wrote: > I'm not aware of any code existing to do this. > > There's a bunch of interesting stuff to do on the Xen side to make this > stuff work. > > Firstly you would need to add support to the hypervisor for dispatching > I/O requests to multiple qemu instances (via multiple io req rings). I > think at the moment there is only support for a single ring (or maybe > it's one sync and one buffered I/O ring). > I have modified Xen to create "ioreq servers". An ioreq server contains a list of IO ranges and a list of BDFs to trap IOs for a unique instance of QEMU running. Each ioreq server can be associated to an event channel. This way we can deliver IOs events to different processes. For each QEMU, a ioreq server is created. QEMU must specify which pci (with a BDF) and IO range it handles. I added some hypercalls: - to register an ioreq server - to register/unregister BDF - to register/unregister IO range For the moment all QEMUs share the same pages (buffered and IO request). For more security, I would like to privatize these pages for each ioreq server. I saw these pages are allocated by the toolstack. Can we assume that the toolstack know at domain creation time how many QEMU it is going to spawn ? > You'd also need to make sure that qemu explicitly requests all the MMIO > regions it is interested in. Currently the hypervisor forwards any > unknown MMIO to qemu so the explicit registration is probably not done > as consistently as it could be. If you want to have N qemus then you > need to make sure that at least N-1 of register for everything they are > interested in. > I have modified QEMU to register all IO ranges and pci it needs. All unregister IO is discarded by Xen. > Currently the PCI config space decode is done within qemu which is a bit > tricky if you are wanting to have different emulated PCI devices in > different qemu processes. We think it would independently be an > architectural improvement to have the hypervisor do the PCI config space > decode anyway. This would allow it to forward the I/O to the correct > qemu (there are other benefits to this change, e.g. relating to PCI > passthrough and the handling of MSI configuration etc) > I have created a patch which allow Xen to catch cf8 through cff io port registers. Xen goes through the list of ioreq servers to know which server can handle the PCI and prepare the resquest. For that I added a new io request type IOREQ_TYPE_PCI_CONFIG. > Then you'd need to do a bunch of toolstack level work to start and > manage the multiple qemu processes instead of the existing single > process. I have began to modify the toolstack. For the moment, I just handle a new type of device model for my own test.