From mboxrd@z Thu Jan 1 00:00:00 1970 From: Don Dutile Subject: Re: decent performance drop for SCSI LLD / SAN initiator when iommu is turned on Date: Fri, 03 May 2013 15:40:28 -0400 Message-ID: <518412AC.3070507@redhat.com> References: <20130502015603.GC26105@redhat.com> <0EE9A1CDC8D6434DB00095CD7DB873462CF9D73E@MTLDAG01.mtl.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <0EE9A1CDC8D6434DB00095CD7DB873462CF9D73E-fViJhHBwANKuSA5JZHE7gA@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Errors-To: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org To: Yan Burman Cc: Roland Dreier , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org" , Or Gerlitz , "Michael S. Tsirkin" List-Id: linux-rdma@vger.kernel.org On 05/02/2013 10:13 AM, Yan Burman wrote: > > >> -----Original Message----- >> From: Michael S. Tsirkin [mailto:mst-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org] >> Sent: Thursday, May 02, 2013 04:56 >> To: Or Gerlitz >> Cc: Roland Dreier; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Yan Burman; linux- >> rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org >> Subject: Re: decent performance drop for SCSI LLD / SAN initiator when >> iommu is turned on >> >> On Thu, May 02, 2013 at 02:11:15AM +0300, Or Gerlitz wrote: >>> Hi Roland, IOMMU folks, >>> >>> So we've noted that when configuring the kernel&& booting with intel >>> iommu set to on on a physical node (non VM, and without enabling SRIOV >>> by the HW device driver) raw performance of the iSER (iSCSI RDMA) SAN >>> initiator is reduced notably, e.g in the testbed we looked today we >>> had ~260K 1KB random IOPS and 5.5GBs BW for 128KB IOs with iommu >>> turned off for single LUN, and ~150K IOPS and 4GBs BW with iommu >>> turned on. No change on the target node between runs. >> >> That's why we have iommu=pt. >> See definition of iommu_pass_through in arch/x86/kernel/pci-dma.c. > > I tried passing "intel_iommu=on iommu=pt" to 3.8.11 kernel and I still get performance degradation. > I get the same numbers with iommu=pt as without it. > > I wanted to send perf output, but currently I seem to have some problem with its output. > Will try to get perf differences next week. > > Yan > > > _______________________________________________ > iommu mailing list > iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu dmesg dump? -- interested to see if x2apic is on, and if MSI is used (or not)