Re: decent performance drop for SCSI LLD / SAN initiator when iommu is turned on

[Don Dutile <ddutile-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>]

On 05/02/2013 10:13 AM, Yan Burman wrote:
>
>
>> -----Original Message-----
>> From: Michael S. Tsirkin [mailto:mst-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
>> Sent: Thursday, May 02, 2013 04:56
>> To: Or Gerlitz
>> Cc: Roland Dreier; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Yan Burman; linux-
>> rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Subject: Re: decent performance drop for SCSI LLD / SAN initiator when
>> iommu is turned on
>>
>> On Thu, May 02, 2013 at 02:11:15AM +0300, Or Gerlitz wrote:
>>> Hi Roland, IOMMU folks,
>>>
>>> So we've noted that when configuring the kernel&&  booting with intel
>>> iommu set to on on a physical node (non VM, and without enabling SRIOV
>>> by the HW device driver) raw performance of the iSER (iSCSI RDMA) SAN
>>> initiator is reduced notably, e.g in the testbed we looked today we
>>> had ~260K 1KB random IOPS and 5.5GBs BW for 128KB IOs with iommu
>>> turned off for single LUN, and ~150K IOPS and 4GBs BW with iommu
>>> turned on. No change on the target node between runs.
>>
>> That's why we have iommu=pt.
>> See definition of iommu_pass_through in arch/x86/kernel/pci-dma.c.
>
> I tried passing "intel_iommu=on iommu=pt" to 3.8.11 kernel and I still get performance degradation.
> I get the same numbers with iommu=pt as without it.
>
> I wanted to send perf output, but currently I seem to have some problem with its output.
> Will try to get perf differences next week.
>
> Yan
>
>
> _______________________________________________
> iommu mailing list
> iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/iommu
dmesg dump? -- interested to see if x2apic is on, and if MSI is used (or not)