From mboxrd@z Thu Jan  1 00:00:00 1970
From: =?UTF-8?B?QmrDtnJuIFTDtnBlbA==?= <bjorn.topel@gmail.com>
Subject: Re: [RFC PATCH 00/24] Introducing AF_XDP support
Date: Tue, 27 Mar 2018 08:30:57 +0200
Message-ID: <CAJ+HfNhUncAXjp2THADBxyC5b-mYN4RpaEGx2XJxAPoitokd6A@mail.gmail.com>
References: <20180131135356.19134-1-bjorn.topel@gmail.com> <CALDO+SYK_4RfRBe7Z0n00wkPhKc1HwGmcGT34W0tVMVZFLqYpw@mail.gmail.com>
 <20180326183810.2ef4e29f@redhat.com> <ad766caf-4656-befa-d55c-92d9c943fa15@oracle.com>
 <CAKgT0Ufya10bXa4BXFbR2GtwZM6i7x8LPYoodFPZ82=S+t7x2w@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Cc: Tushar Dave <tushar.n.dave@oracle.com>,
        Jesper Dangaard Brouer <brouer@redhat.com>,
        William Tu <u9012063@gmail.com>,
        "Karlsson, Magnus" <magnus.karlsson@intel.com>,
        Alexander Duyck <alexander.h.duyck@intel.com>,
        John Fastabend <john.fastabend@gmail.com>,
        Alexei Starovoitov <ast@fb.com>,
        Willem de Bruijn <willemdebruijn.kernel@gmail.com>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Linux Kernel Network Developers <netdev@vger.kernel.org>,
        =?UTF-8?B?QmrDtnJuIFTDtnBlbA==?= <bjorn.topel@intel.com>,
        michael.lundkvist@ericsson.com,
        "Brandeburg, Jesse" <jesse.brandeburg@intel.com>,
        Anjali Singhai Jain <anjali.singhai@intel.com>,
        "Shaw, Jeffrey B" <jeffrey.b.shaw@intel.com>,
        "Yigit, Ferruh" <ferruh.yigit@intel.com>,
        "Zhang, Qi Z" <qi.z.zhang@intel.com>
To: Alexander Duyck <alexander.duyck@gmail.com>
Return-path: <netdev-owner@vger.kernel.org>
Received: from mail-qt0-f196.google.com ([209.85.216.196]:43123 "EHLO
        mail-qt0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1752401AbeC0Ga7 (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 27 Mar 2018 02:30:59 -0400
Received: by mail-qt0-f196.google.com with SMTP id s48so22388276qtb.10
        for <netdev@vger.kernel.org>; Mon, 26 Mar 2018 23:30:59 -0700 (PDT)
In-Reply-To: <CAKgT0Ufya10bXa4BXFbR2GtwZM6i7x8LPYoodFPZ82=S+t7x2w@mail.gmail.com>
Sender: netdev-owner@vger.kernel.org
List-ID: <netdev.vger.kernel.org>

2018-03-27 1:03 GMT+02:00 Alexander Duyck <alexander.duyck@gmail.com>:
> On Mon, Mar 26, 2018 at 3:54 PM, Tushar Dave <tushar.n.dave@oracle.com> wrote:
[...]
>>
>> Whats the implication here. Should IOMMU be disabled?
>> I'm asking because I do see a huge difference while running pktgen test for
>> my performance benchmarks, with and without intel_iommu.
>>
>>
>> -Tushar
>
> For the Intel parts the IOMMU can be expensive primarily for Tx, since
> it should have minimal impact if the Rx pages are pinned/recycled. I
> am assuming the same is true here for AF_XDP, Bjorn can correct me if
> I am wrong.
>

For the non-zc case the DMA mapping is done in the Tx fast path, so
there, as Alex says, you'll definitely see a performance penalty. For
Rx the page-recycle mechanism (Intel drivers) usally avoids doing any
DMA mappings in the fast-path.

As for AF_XDP zerocopy mode, we do the DMA mapping up front (avoiding
the single-use mappings), to avoid that performance hit. Keep in mind,
though, that the IOTLB is still in play, and usually performs worse
under pressure, than the non-IOMMU case.

> Basically the IOMMU can make creating/destroying a DMA mapping really
> expensive. The easiest way to work around it in the case of the Intel
> IOMMU is to boot with "iommu=pt" which will create an identity mapping
> for the host. The downside is though that you then have the entire
> system accessible to the device unless a new mapping is created for it
> by assigning it to a new IOMMU domain.
>



> Thanks.
>
> - Alex