From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: KVM pci-assign - iommu width is not sufficient for mapped address Date: Thu, 07 Jan 2016 21:53:06 -0700 Message-ID: <1452228786.29599.188.camel@redhat.com> References: <1452175812.29599.132.camel@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: kvm@vger.kernel.org To: Shyam Return-path: Received: from mx1.redhat.com ([209.132.183.28]:60916 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753612AbcAHExI (ORCPT ); Thu, 7 Jan 2016 23:53:08 -0500 In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On Fri, 2016-01-08 at 09:47 +0530, Shyam wrote: > Hi Alex, >=20 > Thanks for your inputs. >=20 > We are using Mellanox ConnectX-3 iSER SRIOV capable NICs. We > provision > these VF's into the VM. The VM connects to few SSD drives through > iSER. For this performance test, if we expose the same SSDs through > iSER out of VM to servers & run vdbench 4K read/write workloads we > see > this significant performance drop when using vfio. These VM's have 8 > hyper-threads from Intel E5-2680 v3 server & 32GB RAM. The key > observation is with vfio the cpu saturates much earlier & hence > cannot > allow us to scale IOPs. >=20 > I will open a separate mail thread about this performance degradation > using vfio with numbers. In the meantime if you can suggest how to > look for performance issue or what logs you would prefer for VFIO > debugging it will help in getting the needed info for you. Hi Shyam, =46or the degree of performance loss you're experiencing, I'd suspect some sort of KVM acceleration is disabled. =C2=A0Would it be possible t= o reproduce your testing on a host running something like Fedora 23 or RHEL7/Centos7 where we know that the kernel and QEMU are fully enabled for vfio? Other useful information: * QEMU command line or libvirt logs for VM in each configuration * lspci -vvv of VF from host while in operation in each config * QEMU version * grep VFIO /boot/config-`uname -r` (or wherever the running kernel config is on your system) =46or a well behaved VF, device assignment should mostly setup VM acces= s and get out of the way, there should be little opportunity to inflict such a high performance difference. =C2=A0If we can't spot anything obv= ious and it's reproducible on a known kernel and QEMU, we can look into tracing to see what might be happening. =C2=A0Thanks, Alex