Re: Low performance Intel 10GE NIC (3.2.10) on 2.6.38 Kernel

[Alexander Duyck <alexander.h.duyck@intel.com>]

On 4/7/2011 9:03 AM, Eric Dumazet wrote:
> Le jeudi 07 avril 2011 à 08:58 -0700, Alexander Duyck a écrit :
>> On 4/7/2011 4:46 AM, Eric Dumazet wrote:
>>> Le jeudi 07 avril 2011 à 19:15 +0800, Wei Gu a écrit :
>>>> Hi,
>>>> I compile the ixgbe driver into the kernel and run the test again and also change the copy to clone in the fw hook
>>>> This is the perf report while I was forwarding 150Kpps with
>>>> The attached file include the basic info about my test system. Please let me know if I did some thing wrong.
>>>>
>>>> +     71.91%          swapper  [kernel.kallsyms]            [k] poll_idle
>>>> +     10.43%          swapper  [kernel.kallsyms]            [k] intel_idle
>>>> -      8.00%     ksoftirqd/24  [kernel.kallsyms]            [k] _raw_spin_unlock_irqrestore
>>>> \u2592   - _raw_spin_unlock_irqrestore
>>>> \u2592      - 42.25% alloc_iova
>>>> \u2592           intel_alloc_iova
>>>> \u2592           __intel_map_single
>>>> \u2592           intel_map_page
>>
>> I'm almost certain this is the issue here.  I am pretty sure the
>> intel_map_page call indicates that you are running with the Intel IOMMU
>> enabled.  As Eric suggested you can either rebuild your kernel with
>> "CONFIG_DMAR=N", or pass the kernel the parameter "intel_iommu=off" in
>> order to disable it so that it will instead just use SWIOTLB.
>
> What's the purpose of intel_iommu ?
>
> Could this be speedup if ixgbe uses a perqueue iommu context instead of
> a per device, so that we dont hit a single spinlock ?

The intel_iommu is meant to be a security feature.  Primarily it is used 
in virtualzation where it allows KVM or Xen to direct assign a device 
without having to worry about the guest getting access to the hosts 
physical memory by submitting invalid DMA requests.

If virtualzation isn't in use I would recommend turning it off as it can 
have a negative impact especially on small packet performance due to the 
extra locking overhead that is required for DMA map and unmap calls.

Thanks,

Alex