From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gavin Shan Date: Tue, 06 May 2014 04:26:22 +0000 Subject: Re: [PATCH RFC 00/22] EEH Support for VFIO PCI devices on PowerKVM guest Message-Id: <20140506042622.GA24228@shangw> List-Id: References: <1399253291-3975-1-git-send-email-gwshan@linux.vnet.ibm.com> <53677C6C.8060508@suse.de> <1399298412.24318.521.camel@ul30vt.home> In-Reply-To: <1399298412.24318.521.camel@ul30vt.home> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Alex Williamson Cc: kvm@vger.kernel.org, aik@ozlabs.ru, Alexander Graf , kvm-ppc@vger.kernel.org, Gavin Shan , qiudayu@linux.vnet.ibm.com, linuxppc-dev@lists.ozlabs.org On Mon, May 05, 2014 at 08:00:12AM -0600, Alex Williamson wrote: >On Mon, 2014-05-05 at 13:56 +0200, Alexander Graf wrote: >> On 05/05/2014 03:27 AM, Gavin Shan wrote: >> > The series of patches intends to support EEH for PCI devices, which have been >> > passed through to PowerKVM based guest via VFIO. The implementation is >> > straightforward based on the issues or problems we have to resolve to support >> > EEH for PowerKVM based guest. >> > >> > - Emulation for EEH RTAS requests. Thanksfully, we already have infrastructure >> > to emulate XICS. Without introducing new mechanism, we just extend that >> > existing infrastructure to support EEH RTAS emulation. EEH RTAS requests >> > initiated from guest are posted to host where the requests get handled or >> > delivered to underly firmware for further handling. For that, the host kerenl >> > has to maintain the PCI address (host domain/bus/slot/function to guest's >> > PHB BUID/bus/slot/function) mapping via KVM VFIO device. The address mapping >> > will be built when initializing VFIO device in QEMU and destroied when the >> > VFIO device in QEMU is going to offline, or VM is destroy. >> >> Do you also expose all those interfaces to user space? VFIO is as much >> about user space device drivers as it is about device assignment. >> Yep, all the interfaces are exported to user space. >> I would like to first see an implementation that doesn't touch KVM >> emulation code at all but instead routes everything through QEMU. As a >> second step we can then accelerate performance critical paths inside of KVM. >> Ok. I'll change the implementation. However, the QEMU still has to poll/push information from/to host kerenl. So the best place for that would be tce_iommu_driver_ops::ioctl as EEH is Power specific feature. For the error injection, I guess I have to put the logic token management into QEMU and error injection request will be handled by QEMU and then routed to host kernel via additional syscall as we did for pSeries. >> That way we ensure that user space device drivers have all the power >> over a device they need to drive it. > >+1 > Thanks, Gavin From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from e23smtp03.au.ibm.com (e23smtp03.au.ibm.com [202.81.31.145]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 952F5140E6A for ; Tue, 6 May 2014 14:26:28 +1000 (EST) Received: from /spool/local by e23smtp03.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 6 May 2014 14:26:26 +1000 Received: from d23relay04.au.ibm.com (d23relay04.au.ibm.com [9.190.234.120]) by d23dlp01.au.ibm.com (Postfix) with ESMTP id 0555B2CE8050 for ; Tue, 6 May 2014 14:26:24 +1000 (EST) Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139]) by d23relay04.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id s4645Fhe7602436 for ; Tue, 6 May 2014 14:05:15 +1000 Received: from d23av04.au.ibm.com (localhost [127.0.0.1]) by d23av04.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id s464QNZg014652 for ; Tue, 6 May 2014 14:26:23 +1000 Date: Tue, 6 May 2014 14:26:22 +1000 From: Gavin Shan To: Alex Williamson Subject: Re: [PATCH RFC 00/22] EEH Support for VFIO PCI devices on PowerKVM guest Message-ID: <20140506042622.GA24228@shangw> References: <1399253291-3975-1-git-send-email-gwshan@linux.vnet.ibm.com> <53677C6C.8060508@suse.de> <1399298412.24318.521.camel@ul30vt.home> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <1399298412.24318.521.camel@ul30vt.home> Cc: kvm@vger.kernel.org, aik@ozlabs.ru, Alexander Graf , kvm-ppc@vger.kernel.org, Gavin Shan , qiudayu@linux.vnet.ibm.com, linuxppc-dev@lists.ozlabs.org Reply-To: Gavin Shan List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Mon, May 05, 2014 at 08:00:12AM -0600, Alex Williamson wrote: >On Mon, 2014-05-05 at 13:56 +0200, Alexander Graf wrote: >> On 05/05/2014 03:27 AM, Gavin Shan wrote: >> > The series of patches intends to support EEH for PCI devices, which have been >> > passed through to PowerKVM based guest via VFIO. The implementation is >> > straightforward based on the issues or problems we have to resolve to support >> > EEH for PowerKVM based guest. >> > >> > - Emulation for EEH RTAS requests. Thanksfully, we already have infrastructure >> > to emulate XICS. Without introducing new mechanism, we just extend that >> > existing infrastructure to support EEH RTAS emulation. EEH RTAS requests >> > initiated from guest are posted to host where the requests get handled or >> > delivered to underly firmware for further handling. For that, the host kerenl >> > has to maintain the PCI address (host domain/bus/slot/function to guest's >> > PHB BUID/bus/slot/function) mapping via KVM VFIO device. The address mapping >> > will be built when initializing VFIO device in QEMU and destroied when the >> > VFIO device in QEMU is going to offline, or VM is destroy. >> >> Do you also expose all those interfaces to user space? VFIO is as much >> about user space device drivers as it is about device assignment. >> Yep, all the interfaces are exported to user space. >> I would like to first see an implementation that doesn't touch KVM >> emulation code at all but instead routes everything through QEMU. As a >> second step we can then accelerate performance critical paths inside of KVM. >> Ok. I'll change the implementation. However, the QEMU still has to poll/push information from/to host kerenl. So the best place for that would be tce_iommu_driver_ops::ioctl as EEH is Power specific feature. For the error injection, I guess I have to put the logic token management into QEMU and error injection request will be handled by QEMU and then routed to host kernel via additional syscall as we did for pSeries. >> That way we ensure that user space device drivers have all the power >> over a device they need to drive it. > >+1 > Thanks, Gavin From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gavin Shan Subject: Re: [PATCH RFC 00/22] EEH Support for VFIO PCI devices on PowerKVM guest Date: Tue, 6 May 2014 14:26:22 +1000 Message-ID: <20140506042622.GA24228@shangw> References: <1399253291-3975-1-git-send-email-gwshan@linux.vnet.ibm.com> <53677C6C.8060508@suse.de> <1399298412.24318.521.camel@ul30vt.home> Reply-To: Gavin Shan Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: kvm@vger.kernel.org, aik@ozlabs.ru, Alexander Graf , kvm-ppc@vger.kernel.org, Gavin Shan , qiudayu@linux.vnet.ibm.com, linuxppc-dev@lists.ozlabs.org To: Alex Williamson Return-path: Content-Disposition: inline In-Reply-To: <1399298412.24318.521.camel@ul30vt.home> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+glppd-linuxppc64-dev=m.gmane.org@lists.ozlabs.org Sender: "Linuxppc-dev" List-Id: kvm.vger.kernel.org T24gTW9uLCBNYXkgMDUsIDIwMTQgYXQgMDg6MDA6MTJBTSAtMDYwMCwgQWxleCBXaWxsaWFtc29u IHdyb3RlOgo+T24gTW9uLCAyMDE0LTA1LTA1IGF0IDEzOjU2ICswMjAwLCBBbGV4YW5kZXIgR3Jh ZiB3cm90ZToKPj4gT24gMDUvMDUvMjAxNCAwMzoyNyBBTSwgR2F2aW4gU2hhbiB3cm90ZToKPj4g PiBUaGUgc2VyaWVzIG9mIHBhdGNoZXMgaW50ZW5kcyB0byBzdXBwb3J0IEVFSCBmb3IgUENJIGRl dmljZXMsIHdoaWNoIGhhdmUgYmVlbgo+PiA+IHBhc3NlZCB0aHJvdWdoIHRvIFBvd2VyS1ZNIGJh c2VkIGd1ZXN0IHZpYSBWRklPLiBUaGUgaW1wbGVtZW50YXRpb24gaXMKPj4gPiBzdHJhaWdodGZv cndhcmQgYmFzZWQgb24gdGhlIGlzc3VlcyBvciBwcm9ibGVtcyB3ZSBoYXZlIHRvIHJlc29sdmUg dG8gc3VwcG9ydAo+PiA+IEVFSCBmb3IgUG93ZXJLVk0gYmFzZWQgZ3Vlc3QuCj4+ID4KPj4gPiAt IEVtdWxhdGlvbiBmb3IgRUVIIFJUQVMgcmVxdWVzdHMuIFRoYW5rc2Z1bGx5LCB3ZSBhbHJlYWR5 IGhhdmUgaW5mcmFzdHJ1Y3R1cmUKPj4gPiAgICB0byBlbXVsYXRlIFhJQ1MuIFdpdGhvdXQgaW50 cm9kdWNpbmcgbmV3IG1lY2hhbmlzbSwgd2UganVzdCBleHRlbmQgdGhhdAo+PiA+ICAgIGV4aXN0 aW5nIGluZnJhc3RydWN0dXJlIHRvIHN1cHBvcnQgRUVIIFJUQVMgZW11bGF0aW9uLiBFRUggUlRB UyByZXF1ZXN0cwo+PiA+ICAgIGluaXRpYXRlZCBmcm9tIGd1ZXN0IGFyZSBwb3N0ZWQgdG8gaG9z dCB3aGVyZSB0aGUgcmVxdWVzdHMgZ2V0IGhhbmRsZWQgb3IKPj4gPiAgICBkZWxpdmVyZWQgdG8g dW5kZXJseSBmaXJtd2FyZSBmb3IgZnVydGhlciBoYW5kbGluZy4gRm9yIHRoYXQsIHRoZSBob3N0 IGtlcmVubAo+PiA+ICAgIGhhcyB0byBtYWludGFpbiB0aGUgUENJIGFkZHJlc3MgKGhvc3QgZG9t YWluL2J1cy9zbG90L2Z1bmN0aW9uIHRvIGd1ZXN0J3MKPj4gPiAgICBQSEIgQlVJRC9idXMvc2xv dC9mdW5jdGlvbikgbWFwcGluZyB2aWEgS1ZNIFZGSU8gZGV2aWNlLiBUaGUgYWRkcmVzcyBtYXBw aW5nCj4+ID4gICAgd2lsbCBiZSBidWlsdCB3aGVuIGluaXRpYWxpemluZyBWRklPIGRldmljZSBp biBRRU1VIGFuZCBkZXN0cm9pZWQgd2hlbiB0aGUKPj4gPiAgICBWRklPIGRldmljZSBpbiBRRU1V IGlzIGdvaW5nIHRvIG9mZmxpbmUsIG9yIFZNIGlzIGRlc3Ryb3kuCj4+IAo+PiBEbyB5b3UgYWxz byBleHBvc2UgYWxsIHRob3NlIGludGVyZmFjZXMgdG8gdXNlciBzcGFjZT8gVkZJTyBpcyBhcyBt dWNoIAo+PiBhYm91dCB1c2VyIHNwYWNlIGRldmljZSBkcml2ZXJzIGFzIGl0IGlzIGFib3V0IGRl dmljZSBhc3NpZ25tZW50Lgo+PiAKClllcCwgYWxsIHRoZSBpbnRlcmZhY2VzIGFyZSBleHBvcnRl ZCB0byB1c2VyIHNwYWNlLiAKCj4+IEkgd291bGQgbGlrZSB0byBmaXJzdCBzZWUgYW4gaW1wbGVt ZW50YXRpb24gdGhhdCBkb2Vzbid0IHRvdWNoIEtWTSAKPj4gZW11bGF0aW9uIGNvZGUgYXQgYWxs IGJ1dCBpbnN0ZWFkIHJvdXRlcyBldmVyeXRoaW5nIHRocm91Z2ggUUVNVS4gQXMgYSAKPj4gc2Vj b25kIHN0ZXAgd2UgY2FuIHRoZW4gYWNjZWxlcmF0ZSBwZXJmb3JtYW5jZSBjcml0aWNhbCBwYXRo cyBpbnNpZGUgb2YgS1ZNLgo+PiAKCk9rLiBJJ2xsIGNoYW5nZSB0aGUgaW1wbGVtZW50YXRpb24u IEhvd2V2ZXIsIHRoZSBRRU1VIHN0aWxsIGhhcyB0bwpwb2xsL3B1c2ggaW5mb3JtYXRpb24gZnJv bS90byBob3N0IGtlcmVubC4gU28gdGhlIGJlc3QgcGxhY2UgZm9yIHRoYXQKd291bGQgYmUgdGNl X2lvbW11X2RyaXZlcl9vcHM6OmlvY3RsIGFzIEVFSCBpcyBQb3dlciBzcGVjaWZpYyBmZWF0dXJl LgoKRm9yIHRoZSBlcnJvciBpbmplY3Rpb24sIEkgZ3Vlc3MgSSBoYXZlIHRvIHB1dCB0aGUgbG9n aWMgdG9rZW4gbWFuYWdlbWVudAppbnRvIFFFTVUgYW5kIGVycm9yIGluamVjdGlvbiByZXF1ZXN0 IHdpbGwgYmUgaGFuZGxlZCBieSBRRU1VIGFuZCB0aGVuCnJvdXRlZCB0byBob3N0IGtlcm5lbCB2 aWEgYWRkaXRpb25hbCBzeXNjYWxsIGFzIHdlIGRpZCBmb3IgcFNlcmllcy4KCj4+IFRoYXQgd2F5 IHdlIGVuc3VyZSB0aGF0IHVzZXIgc3BhY2UgZGV2aWNlIGRyaXZlcnMgaGF2ZSBhbGwgdGhlIHBv d2VyIAo+PiBvdmVyIGEgZGV2aWNlIHRoZXkgbmVlZCB0byBkcml2ZSBpdC4KPgo+KzEKPgoKVGhh bmtzLApHYXZpbgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X18KTGludXhwcGMtZGV2IG1haWxpbmcgbGlzdApMaW51eHBwYy1kZXZAbGlzdHMub3psYWJzLm9y ZwpodHRwczovL2xpc3RzLm96bGFicy5vcmcvbGlzdGluZm8vbGludXhwcGMtZGV2