From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751913AbdFGO7B (ORCPT ); Wed, 7 Jun 2017 10:59:01 -0400 Received: from 8bytes.org ([81.169.241.247]:45534 "EHLO theia.8bytes.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751850AbdFGO66 (ORCPT ); Wed, 7 Jun 2017 10:58:58 -0400 From: Joerg Roedel To: iommu@lists.linux-foundation.org Cc: linux-kernel@vger.kernel.org, Tom Lendacky , Arindam Nath , Joerg Roedel Subject: [PATCH 3/7] iommu/amd: Make use of the per-domain flush queue Date: Wed, 7 Jun 2017 16:58:16 +0200 Message-Id: <1496847500-6781-4-git-send-email-joro@8bytes.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1496847500-6781-1-git-send-email-joro@8bytes.org> References: <1496847500-6781-1-git-send-email-joro@8bytes.org> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Joerg Roedel Fill the flush-queue on unmap and only flush the IOMMU and device TLBs when a per-cpu queue gets full. Signed-off-by: Joerg Roedel --- drivers/iommu/amd_iommu.c | 60 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 56 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 71c688a..6a5c858 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -1778,6 +1778,61 @@ static int dma_ops_domain_alloc_flush_queue(struct dma_ops_domain *dom) return 0; } +static inline bool queue_ring_full(struct flush_queue *queue) +{ + return (((queue->tail + 1) % FLUSH_QUEUE_SIZE) == queue->head); +} + +#define queue_ring_for_each(i, q) \ + for (i = (q)->head; i != (q)->tail; i = (i + 1) % FLUSH_QUEUE_SIZE) + +static void queue_release(struct dma_ops_domain *dom, + struct flush_queue *queue) +{ + unsigned i; + + queue_ring_for_each(i, queue) + free_iova_fast(&dom->iovad, + queue->entries[i].iova_pfn, + queue->entries[i].pages); + + queue->head = queue->tail = 0; +} + +static inline unsigned queue_ring_add(struct flush_queue *queue) +{ + unsigned idx = queue->tail; + + queue->tail = (idx + 1) % FLUSH_QUEUE_SIZE; + + return idx; +} + +static void queue_add(struct dma_ops_domain *dom, + unsigned long address, unsigned long pages) +{ + struct flush_queue *queue; + int idx; + + pages = __roundup_pow_of_two(pages); + address >>= PAGE_SHIFT; + + queue = get_cpu_ptr(dom->flush_queue); + + if (queue_ring_full(queue)) { + domain_flush_tlb(&dom->domain); + domain_flush_complete(&dom->domain); + queue_release(dom, queue); + } + + idx = queue_ring_add(queue); + + queue->entries[idx].iova_pfn = address; + queue->entries[idx].pages = pages; + + put_cpu_ptr(dom->flush_queue); +} + /* * Free a domain, only used if something went wrong in the * allocation path and we need to free an already allocated page table @@ -2426,10 +2481,7 @@ static void __unmap_single(struct dma_ops_domain *dma_dom, domain_flush_tlb(&dma_dom->domain); domain_flush_complete(&dma_dom->domain); } else { - /* Keep the if() around, we need it later again */ - dma_ops_free_iova(dma_dom, dma_addr, pages); - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); + queue_add(dma_dom, dma_addr, pages); } } -- 2.7.4