From mboxrd@z Thu Jan  1 00:00:00 1970
From: Arnd Bergmann <arnd@arndb.de>
Subject: Re: [PATCH v8 10/15] octeontx2-af: Reconfig MSIX base with IOVA
Date: Tue, 9 Oct 2018 09:57:25 +0200
Message-ID: <CAK8P3a01YPk6Bpn4o10za8pebAHA8sfXAEEBAidfgcvbWwzFpA@mail.gmail.com>
References: <1538924364-18781-1-git-send-email-sunil.kovvuri@gmail.com>
 <1538924364-18781-11-git-send-email-sunil.kovvuri@gmail.com>
 <CAK8P3a2KD_ykmwtCYmLAGBAxgLQ4r-UWp00bBejaFZSSi=-cNA@mail.gmail.com> <CA+sq2CcnthcB5S=Wo5bN_40A7==_jOcxD+b7Y-rkU35-gxwuJA@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Cc: Networking <netdev@vger.kernel.org>,
        David Miller <davem@davemloft.net>, linux-soc@vger.kernel.org,
        gakula@marvell.com, sgoutham@marvell.com
To: Sunil Kovvuri <sunil.kovvuri@gmail.com>
Return-path: <netdev-owner@vger.kernel.org>
Received: from mail-qt1-f194.google.com ([209.85.160.194]:33187 "EHLO
        mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1725855AbeJIPNW (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 9 Oct 2018 11:13:22 -0400
In-Reply-To: <CA+sq2CcnthcB5S=Wo5bN_40A7==_jOcxD+b7Y-rkU35-gxwuJA@mail.gmail.com>
Sender: netdev-owner@vger.kernel.org
List-ID: <netdev.vger.kernel.org>

On Tue, Oct 9, 2018 at 9:03 AM Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote:
> On Mon, Oct 8, 2018 at 5:38 PM Arnd Bergmann <arnd@arndb.de> wrote:
> > On Sun, Oct 7, 2018 at 5:01 PM <sunil.kovvuri@gmail.com> wrote:
> >
> > >
> > > +       /* HW interprets RVU_AF_MSIXTR_BASE address as an IOVA, hence
> > > +        * create a IOMMU mapping for the physcial address configured by
> > > +        * firmware and reconfig RVU_AF_MSIXTR_BASE with IOVA.
> > > +        */
> > > +       cfg = rvu_read64(rvu, BLKADDR_RVUM, RVU_PRIV_CONST);
> > > +       max_msix = cfg & 0xFFFFF;
> > > +       phy_addr = rvu_read64(rvu, BLKADDR_RVUM, RVU_AF_MSIXTR_BASE);
> > > +       iova = dma_map_single(rvu->dev, (void *)phy_addr,
> > > +                             max_msix * PCI_MSIX_ENTRY_SIZE,
> > > +                             DMA_BIDIRECTIONAL);
> > > +       if (dma_mapping_error(rvu->dev, iova))
> > > +               return -ENOMEM;
> > > +
> > > +       rvu_write64(rvu, BLKADDR_RVUM, RVU_AF_MSIXTR_BASE, (u64)iova);
> > > +       rvu->msix_base_iova = iova;
> > > +
> >
> > I'm a bit puzzled by how this works. Does this rely on a specific iommu
> > driver implementation? Normally a physical  address makes no sense to the
> > implementation backing dma_map_single() that tries to convert a
> > linear kernel virtual address into a physical address.
> >
> >         Arnd
>
> I understand what you are pointing at, but we did test this.
> IOMMU on this silicon is standard ARM64 SMMUv3.

I didn't doubt that you get the right results, I just couldn't see how ;-)

> All dma_map_single does is virt_to_page and iommu_dma_map_page
> converts it back i.e page_to_phys.
> So the IOMMU driver gets the same physical address passed above and
> creates a iova translation mapping.
>
> For reference below is the captured debug info for the same
> =====
> [   19.435968] rvu_setup_msix_resources: phy_addr 0x3200000 iova 0xfff80000
> [   19.436967] rvu_setup_msix_resources: virt_to_page(phy_addr)
> 0xffff7fe00000c800 page_to_phys(page) 0x3200000
> offset_in_page(phy_addr) 0x00
> =====

I think if you enable CONFIG_DEBUG_VIRTUAL, the virt_to_page()
above should trigger a warning in

phys_addr_t __virt_to_phys(unsigned long x)
{
        WARN(!__is_lm_address(x),
             "virt_to_phys used for non-linear address: %pK (%pS)\n",
              (void *)x,
              (void *)x);

        return __virt_to_phys_nodebug(x);
}

Can you verify that?

        Arnd