From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
Subject: Re: IXGBE, IOMMU DMAR DRHD handling fault issue
Date: Thu, 15 Feb 2018 10:28:33 +0000
Message-ID: <3b6d6cfc-7e2e-70ee-e059-aa31dc23c93b@intel.com>
References: <CAFb4SLA9dJ846c6Eb3fsV4F7NjUm-Nx_e6iUfOXnPE9KDu4_FA@mail.gmail.com>
 <8ddb30a3-1253-ff60-20bb-b735fef5a91c@intel.com>
 <CAFb4SLC5X8nubBW6s4LBGb-kcEp-7hiRzzo2qjNyuiwrPRaiqQ@mail.gmail.com>
 <b667dba1-c390-44b9-bc3b-521fa7f5becf@intel.com>
 <CAFb4SLAOTDnz0bK=9D-dLQJ+x6rVOqMH5zDaP_nAoEXLUByWRQ@mail.gmail.com>
 <c7b7ddec-5a0d-6851-92dd-381b85455c82@intel.com>
 <CAFb4SLAyxYm+iD6wYQBw1fPfnMT7JcjGsR6H+kb7SBYJcFWDKQ@mail.gmail.com>
 <d0682205-c335-14c4-0b0f-8ebbf58bb9f7@intel.com>
 <CAFb4SLB5P5KZ3QxKy6i00yQpSJRQqOE1cwAq0ejnj1raHyWc2Q@mail.gmail.com>
 <4a54fe58-02d8-07ef-b706-f747383f4985@intel.com>
 <CAFb4SLCBCdmAQPCQWjUm53k6Q82eO0uRN6x9JkeSzqdF0rcRUg@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Cc: dev@dpdk.org
To: Ravi Kerur <rkerur@gmail.com>
Return-path: <dev-bounces@dpdk.org>
Received: from mga14.intel.com (mga14.intel.com [192.55.52.115])
 by dpdk.org (Postfix) with ESMTP id 3820C1B28D
 for <dev@dpdk.org>; Thu, 15 Feb 2018 11:28:36 +0100 (CET)
In-Reply-To: <CAFb4SLCBCdmAQPCQWjUm53k6Q82eO0uRN6x9JkeSzqdF0rcRUg@mail.gmail.com>
Content-Language: en-US
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

On 14-Feb-18 8:00 PM, Ravi Kerur wrote:
> 
> Earlier I was focusing only on DMAR errors and I might have said 'it 
> worked' when I didn't notice them on host when dpdk was started on 
> guest. When trying to send packets out of that interface from guest I 
> did see DMAR errors. I am attaching information you requested.  I have 
> enabled log-level=8 and files contain dpdk EAL/PMD logs as well.

Great, now we're on the same page.

> 
> Snippets below
> 
> on host, DMAR fault address from dmesg
> 
> [351576.998109] DMAR: DRHD: handling fault status reg 702
> [351576.998113] DMAR: [DMA Read] Request device [04:10.0] fault addr 
> 257617000 [fault reason 06] PTE Read access is not set
> 
> on guest (dump phys_mem_layout)
> 
> Segment 235: phys:0x257600000, len:2097152, virt:0x7fce87e00000, 
> socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0
> ...
> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce87e0f4c0 
> sw_sc_ring=0x7fce87e07380 hw_ring=0x7fce87e17600 dma_addr=0x257617600
> PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7fce89c67d40 
> sw_sc_ring=0x7fce89c5fc00 hw_ring=0x7fce89c6fe80 dma_addr=0x25406fe80
> ...
> 

To me this looks like host (i.e. either QEMU or the PF driver) is trying 
to do DMA using guest-physical (and not host-physical). I'm not too 
well-versed in how QEMU works, but i'm pretty sure that's not supposed 
to happen.

Is PF also bound to DPDK, or are you using native Linux ixgbe driver?

-- 
Thanks,
Anatoly