* Re: move bus (PCI, PCMCIA, EISA, rapdio) config to drivers/
From: Christoph Hellwig @ 2018-10-15 9:06 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-arch, linux-scsi, Linux Kbuild mailing list, linux-pci,
Linux Kernel Mailing List, Dominik Brodowski, alex.bou9,
linuxppc-dev, Christoph Hellwig
In-Reply-To: <CAK7LNAQ3SNStaA6-tOWhm0QHWpNczO07x9T2F7gMcusduMS7GA@mail.gmail.com>
> Thanks for great work again.
>
> However, I suspect some breakages
> (not testing yet, just quick review by my eyes.)
>
> I left some comments in individual patches.
Thanks. I've updated the git tree based on your comments.
^ permalink raw reply
* Re: [PATCH 4/8] pci: consolidate PCI config entry in drivers/pci
From: Masahiro Yamada @ 2018-10-15 9:17 UTC (permalink / raw)
To: Christoph Hellwig
Cc: linux-arch, linux-scsi, Linux Kbuild mailing list, linux-pci,
Linux Kernel Mailing List, Dominik Brodowski, Alex Bounine,
linuxppc-dev
In-Reply-To: <20181015085737.GC27159@lst.de>
On Mon, Oct 15, 2018 at 5:57 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Mon, Oct 15, 2018 at 03:37:05PM +0900, Masahiro Yamada wrote:
> > I think HAVE_ is a preferred prefix in this case according to this doc:
> > https://github.com/masahir0y/linux/blob/v4.19-rc4/Documentation/kbuild/kconfig-language.txt#L448
>
> Ok, I'll switch everything to HAVE_*
>
> > If you do this for entire powerpc,
> > the other 'select HAS_PCI' addtions to
> > arch/powerpc/platforms/*/Kconfig look redundant.
>
> I'll drop them.
>
> > > -config PCI
> > > - bool "PCI support"
> > > - select PCI_MSI
> >
> >
> > I think this 'select PCI_MSI' for riscv was lost.
>
> Indeed, fixed.
>
> > > menu "Bus options (PCI etc.)"
> > >
> > > -config PCI
> > > - bool "PCI support"
> > > - default y
> >
> >
> > The default is y for x86 (and xtensa as well).
> >
> > With this patch, the default will be flipped.
> >
> > I think most of people want to use PCI for x86,
> > and this change will make people upset.
> >
> > Will you update arch/{x86,xtensa}/configs/*_defconfig?
>
> Yes. Alternatively we could add a new ARCH_PCI_DEFAULT symbol,
> that architectures could select if they want PCI by default. Does
> anyone thing this is a good idea?
No, I do not want to see such an ugly config option.
Instead, you can use 'imply' keyword to set the default y.
config X86
...
imply PCI
But, personally, I'd like the defconfig files updated.
--
Best Regards
Masahiro Yamada
^ permalink raw reply
* [PATCH kernel 1/4] powerpc/mm/iommu: Rename mm_iommu_get
From: Alexey Kardashevskiy @ 2018-10-15 9:24 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Alexey Kardashevskiy, Alex Williamson, kvm-ppc, David Gibson
In-Reply-To: <20181015092416.47380-1-aik@ozlabs.ru>
Normally mm_iommu_get() is supposed to add a reference and
mm_iommu_put() to remove it. However historically mm_iommu_find() does
the referencing and mm_iommu_get() is doing allocation and referencing.
This is step 1 towards simpler mm_iommu_get().
This renames:
- mm_iommu_get to mm_iommu_new;
- mm_iommu_find to mm_iommu_get.
This should cause no behavioural change.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
arch/powerpc/include/asm/mmu_context.h | 4 ++--
arch/powerpc/mm/mmu_context_iommu.c | 8 ++++----
drivers/vfio/vfio_iommu_spapr_tce.c | 6 +++---
3 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h
index b694d6a..59d4941 100644
--- a/arch/powerpc/include/asm/mmu_context.h
+++ b/arch/powerpc/include/asm/mmu_context.h
@@ -21,7 +21,7 @@ struct mm_iommu_table_group_mem_t;
extern int isolate_lru_page(struct page *page); /* from internal.h */
extern bool mm_iommu_preregistered(struct mm_struct *mm);
-extern long mm_iommu_get(struct mm_struct *mm,
+extern long mm_iommu_new(struct mm_struct *mm,
unsigned long ua, unsigned long entries,
struct mm_iommu_table_group_mem_t **pmem);
extern long mm_iommu_put(struct mm_struct *mm,
@@ -32,7 +32,7 @@ extern struct mm_iommu_table_group_mem_t *mm_iommu_lookup(struct mm_struct *mm,
unsigned long ua, unsigned long size);
extern struct mm_iommu_table_group_mem_t *mm_iommu_lookup_rm(
struct mm_struct *mm, unsigned long ua, unsigned long size);
-extern struct mm_iommu_table_group_mem_t *mm_iommu_find(struct mm_struct *mm,
+extern struct mm_iommu_table_group_mem_t *mm_iommu_get(struct mm_struct *mm,
unsigned long ua, unsigned long entries);
extern long mm_iommu_ua_to_hpa(struct mm_iommu_table_group_mem_t *mem,
unsigned long ua, unsigned int pageshift, unsigned long *hpa);
diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c
index 56c2234..8eeb99d 100644
--- a/arch/powerpc/mm/mmu_context_iommu.c
+++ b/arch/powerpc/mm/mmu_context_iommu.c
@@ -126,7 +126,7 @@ static int mm_iommu_move_page_from_cma(struct page *page)
return 0;
}
-long mm_iommu_get(struct mm_struct *mm, unsigned long ua, unsigned long entries,
+long mm_iommu_new(struct mm_struct *mm, unsigned long ua, unsigned long entries,
struct mm_iommu_table_group_mem_t **pmem)
{
struct mm_iommu_table_group_mem_t *mem;
@@ -252,7 +252,7 @@ long mm_iommu_get(struct mm_struct *mm, unsigned long ua, unsigned long entries,
return ret;
}
-EXPORT_SYMBOL_GPL(mm_iommu_get);
+EXPORT_SYMBOL_GPL(mm_iommu_new);
static void mm_iommu_unpin(struct mm_iommu_table_group_mem_t *mem)
{
@@ -368,7 +368,7 @@ struct mm_iommu_table_group_mem_t *mm_iommu_lookup_rm(struct mm_struct *mm,
return ret;
}
-struct mm_iommu_table_group_mem_t *mm_iommu_find(struct mm_struct *mm,
+struct mm_iommu_table_group_mem_t *mm_iommu_get(struct mm_struct *mm,
unsigned long ua, unsigned long entries)
{
struct mm_iommu_table_group_mem_t *mem, *ret = NULL;
@@ -382,7 +382,7 @@ struct mm_iommu_table_group_mem_t *mm_iommu_find(struct mm_struct *mm,
return ret;
}
-EXPORT_SYMBOL_GPL(mm_iommu_find);
+EXPORT_SYMBOL_GPL(mm_iommu_get);
long mm_iommu_ua_to_hpa(struct mm_iommu_table_group_mem_t *mem,
unsigned long ua, unsigned int pageshift, unsigned long *hpa)
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index ad63725..1701798 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -156,7 +156,7 @@ static long tce_iommu_unregister_pages(struct tce_container *container,
if ((vaddr & ~PAGE_MASK) || (size & ~PAGE_MASK))
return -EINVAL;
- mem = mm_iommu_find(container->mm, vaddr, size >> PAGE_SHIFT);
+ mem = mm_iommu_get(container->mm, vaddr, size >> PAGE_SHIFT);
if (!mem)
return -ENOENT;
@@ -185,7 +185,7 @@ static long tce_iommu_register_pages(struct tce_container *container,
((vaddr + size) < vaddr))
return -EINVAL;
- mem = mm_iommu_find(container->mm, vaddr, entries);
+ mem = mm_iommu_get(container->mm, vaddr, entries);
if (mem) {
list_for_each_entry(tcemem, &container->prereg_list, next) {
if (tcemem->mem == mem)
@@ -193,7 +193,7 @@ static long tce_iommu_register_pages(struct tce_container *container,
}
}
- ret = mm_iommu_get(container->mm, vaddr, entries, &mem);
+ ret = mm_iommu_new(container->mm, vaddr, entries, &mem);
if (ret)
return ret;
--
2.11.0
^ permalink raw reply related
* [PATCH kernel 3/4] powerpc/mm/iommu: Make mm_iommu_new() fail on existing regions
From: Alexey Kardashevskiy @ 2018-10-15 9:24 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Alexey Kardashevskiy, Alex Williamson, kvm-ppc, David Gibson
In-Reply-To: <20181015092416.47380-1-aik@ozlabs.ru>
Since we are going to have 2 different preregistering helpers, let's
make it clear that mm_iommu_new() is only for the normal (i.e. not device)
memory and for existing areas mm_iommu_get() should be used instead.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
arch/powerpc/mm/mmu_context_iommu.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c
index a8c4a3c..839dbce 100644
--- a/arch/powerpc/mm/mmu_context_iommu.c
+++ b/arch/powerpc/mm/mmu_context_iommu.c
@@ -141,8 +141,7 @@ long mm_iommu_new(struct mm_struct *mm, unsigned long ua, unsigned long entries,
list_for_each_entry_rcu(mem, &mm->context.iommu_group_mem_list,
next) {
if ((mem->ua == ua) && (mem->entries == entries)) {
- ++mem->used;
- *pmem = mem;
+ ret = -EBUSY;
goto unlock_exit;
}
--
2.11.0
^ permalink raw reply related
* [PATCH kernel 4/4] powerpc/vfio/iommu/kvm: Do not pin device memory
From: Alexey Kardashevskiy @ 2018-10-15 9:24 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Alexey Kardashevskiy, Alex Williamson, kvm-ppc, David Gibson
In-Reply-To: <20181015092416.47380-1-aik@ozlabs.ru>
This new memory does not have page structs as it is not plugged to
the host so gup() will fail anyway.
This adds 2 helpers:
- mm_iommu_newdev() to preregister the "memory device" memory so
the rest of API can still be used;
- mm_iommu_is_devmem() to know if the physical address is one of thise
new regions which we must avoid unpinning of.
This adds @mm to tce_page_is_contained() and iommu_tce_xchg() to test
if the memory is device memory to avoid pfn_to_page().
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
arch/powerpc/include/asm/iommu.h | 5 +-
arch/powerpc/include/asm/mmu_context.h | 5 ++
arch/powerpc/kernel/iommu.c | 9 ++--
arch/powerpc/kvm/book3s_64_vio.c | 18 ++++----
arch/powerpc/mm/mmu_context_iommu.c | 83 ++++++++++++++++++++++++++++++----
drivers/vfio/vfio_iommu_spapr_tce.c | 28 ++++++++----
6 files changed, 116 insertions(+), 32 deletions(-)
diff --git a/arch/powerpc/include/asm/iommu.h b/arch/powerpc/include/asm/iommu.h
index 35db0cb..a8aeac0 100644
--- a/arch/powerpc/include/asm/iommu.h
+++ b/arch/powerpc/include/asm/iommu.h
@@ -218,8 +218,9 @@ extern void iommu_register_group(struct iommu_table_group *table_group,
extern int iommu_add_device(struct device *dev);
extern void iommu_del_device(struct device *dev);
extern int __init tce_iommu_bus_notifier_init(void);
-extern long iommu_tce_xchg(struct iommu_table *tbl, unsigned long entry,
- unsigned long *hpa, enum dma_data_direction *direction);
+extern long iommu_tce_xchg(struct mm_struct *mm, struct iommu_table *tbl,
+ unsigned long entry, unsigned long *hpa,
+ enum dma_data_direction *direction);
#else
static inline void iommu_register_group(struct iommu_table_group *table_group,
int pci_domain_number,
diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h
index 59d4941..45330ff 100644
--- a/arch/powerpc/include/asm/mmu_context.h
+++ b/arch/powerpc/include/asm/mmu_context.h
@@ -24,6 +24,9 @@ extern bool mm_iommu_preregistered(struct mm_struct *mm);
extern long mm_iommu_new(struct mm_struct *mm,
unsigned long ua, unsigned long entries,
struct mm_iommu_table_group_mem_t **pmem);
+extern long mm_iommu_newdev(struct mm_struct *mm, unsigned long ua,
+ unsigned long entries, unsigned long dev_hpa,
+ struct mm_iommu_table_group_mem_t **pmem);
extern long mm_iommu_put(struct mm_struct *mm,
struct mm_iommu_table_group_mem_t *mem);
extern void mm_iommu_init(struct mm_struct *mm);
@@ -39,6 +42,8 @@ extern long mm_iommu_ua_to_hpa(struct mm_iommu_table_group_mem_t *mem,
extern long mm_iommu_ua_to_hpa_rm(struct mm_iommu_table_group_mem_t *mem,
unsigned long ua, unsigned int pageshift, unsigned long *hpa);
extern void mm_iommu_ua_mark_dirty_rm(struct mm_struct *mm, unsigned long ua);
+extern bool mm_iommu_is_devmem(struct mm_struct *mm, unsigned long hpa,
+ unsigned int pageshift);
extern long mm_iommu_mapped_inc(struct mm_iommu_table_group_mem_t *mem);
extern void mm_iommu_mapped_dec(struct mm_iommu_table_group_mem_t *mem);
#endif
diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c
index 19b4c62..ab88b0f 100644
--- a/arch/powerpc/kernel/iommu.c
+++ b/arch/powerpc/kernel/iommu.c
@@ -47,6 +47,7 @@
#include <asm/fadump.h>
#include <asm/vio.h>
#include <asm/tce.h>
+#include <asm/mmu_context.h>
#define DBG(...)
@@ -993,15 +994,17 @@ int iommu_tce_check_gpa(unsigned long page_shift, unsigned long gpa)
}
EXPORT_SYMBOL_GPL(iommu_tce_check_gpa);
-long iommu_tce_xchg(struct iommu_table *tbl, unsigned long entry,
- unsigned long *hpa, enum dma_data_direction *direction)
+long iommu_tce_xchg(struct mm_struct *mm, struct iommu_table *tbl,
+ unsigned long entry, unsigned long *hpa,
+ enum dma_data_direction *direction)
{
long ret;
ret = tbl->it_ops->exchange(tbl, entry, hpa, direction);
if (!ret && ((*direction == DMA_FROM_DEVICE) ||
- (*direction == DMA_BIDIRECTIONAL)))
+ (*direction == DMA_BIDIRECTIONAL)) &&
+ !mm_iommu_is_devmem(mm, *hpa, tbl->it_page_shift))
SetPageDirty(pfn_to_page(*hpa >> PAGE_SHIFT));
/* if (unlikely(ret))
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index 62a8d03..532ab797 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -397,12 +397,13 @@ static long kvmppc_tce_validate(struct kvmppc_spapr_tce_table *stt,
return H_SUCCESS;
}
-static void kvmppc_clear_tce(struct iommu_table *tbl, unsigned long entry)
+static void kvmppc_clear_tce(struct mm_struct *mm, struct iommu_table *tbl,
+ unsigned long entry)
{
unsigned long hpa = 0;
enum dma_data_direction dir = DMA_NONE;
- iommu_tce_xchg(tbl, entry, &hpa, &dir);
+ iommu_tce_xchg(mm, tbl, entry, &hpa, &dir);
}
static long kvmppc_tce_iommu_mapped_dec(struct kvm *kvm,
@@ -433,7 +434,7 @@ static long kvmppc_tce_iommu_do_unmap(struct kvm *kvm,
unsigned long hpa = 0;
long ret;
- if (WARN_ON_ONCE(iommu_tce_xchg(tbl, entry, &hpa, &dir)))
+ if (WARN_ON_ONCE(iommu_tce_xchg(kvm->mm, tbl, entry, &hpa, &dir)))
return H_TOO_HARD;
if (dir == DMA_NONE)
@@ -441,7 +442,7 @@ static long kvmppc_tce_iommu_do_unmap(struct kvm *kvm,
ret = kvmppc_tce_iommu_mapped_dec(kvm, tbl, entry);
if (ret != H_SUCCESS)
- iommu_tce_xchg(tbl, entry, &hpa, &dir);
+ iommu_tce_xchg(kvm->mm, tbl, entry, &hpa, &dir);
return ret;
}
@@ -487,7 +488,7 @@ long kvmppc_tce_iommu_do_map(struct kvm *kvm, struct iommu_table *tbl,
if (mm_iommu_mapped_inc(mem))
return H_TOO_HARD;
- ret = iommu_tce_xchg(tbl, entry, &hpa, &dir);
+ ret = iommu_tce_xchg(kvm->mm, tbl, entry, &hpa, &dir);
if (WARN_ON_ONCE(ret)) {
mm_iommu_mapped_dec(mem);
return H_TOO_HARD;
@@ -566,7 +567,7 @@ long kvmppc_h_put_tce(struct kvm_vcpu *vcpu, unsigned long liobn,
entry, ua, dir);
if (ret != H_SUCCESS) {
- kvmppc_clear_tce(stit->tbl, entry);
+ kvmppc_clear_tce(vcpu->kvm->mm, stit->tbl, entry);
goto unlock_exit;
}
}
@@ -655,7 +656,8 @@ long kvmppc_h_put_tce_indirect(struct kvm_vcpu *vcpu,
iommu_tce_direction(tce));
if (ret != H_SUCCESS) {
- kvmppc_clear_tce(stit->tbl, entry);
+ kvmppc_clear_tce(vcpu->kvm->mm, stit->tbl,
+ entry);
goto unlock_exit;
}
}
@@ -704,7 +706,7 @@ long kvmppc_h_stuff_tce(struct kvm_vcpu *vcpu,
return ret;
WARN_ON_ONCE(1);
- kvmppc_clear_tce(stit->tbl, entry);
+ kvmppc_clear_tce(vcpu->kvm->mm, stit->tbl, entry);
}
}
diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c
index 839dbce..4835b4e 100644
--- a/arch/powerpc/mm/mmu_context_iommu.c
+++ b/arch/powerpc/mm/mmu_context_iommu.c
@@ -36,6 +36,8 @@ struct mm_iommu_table_group_mem_t {
u64 ua; /* userspace address */
u64 entries; /* number of entries in hpas[] */
u64 *hpas; /* vmalloc'ed */
+#define MM_IOMMU_TABLE_INVALID_HPA ((uint64_t)-1)
+ u64 dev_hpa; /* Device memory base address */
};
static long mm_iommu_adjust_locked_vm(struct mm_struct *mm,
@@ -126,7 +128,8 @@ static int mm_iommu_move_page_from_cma(struct page *page)
return 0;
}
-long mm_iommu_new(struct mm_struct *mm, unsigned long ua, unsigned long entries,
+static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,
+ unsigned long entries, unsigned long dev_hpa,
struct mm_iommu_table_group_mem_t **pmem)
{
struct mm_iommu_table_group_mem_t *mem;
@@ -155,11 +158,13 @@ long mm_iommu_new(struct mm_struct *mm, unsigned long ua, unsigned long entries,
}
- ret = mm_iommu_adjust_locked_vm(mm, entries, true);
- if (ret)
- goto unlock_exit;
+ if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) {
+ ret = mm_iommu_adjust_locked_vm(mm, entries, true);
+ if (ret)
+ goto unlock_exit;
- locked_entries = entries;
+ locked_entries = entries;
+ }
mem = kzalloc(sizeof(*mem), GFP_KERNEL);
if (!mem) {
@@ -167,6 +172,13 @@ long mm_iommu_new(struct mm_struct *mm, unsigned long ua, unsigned long entries,
goto unlock_exit;
}
+ if (dev_hpa != MM_IOMMU_TABLE_INVALID_HPA) {
+ mem->pageshift = __ffs(dev_hpa | (entries << PAGE_SHIFT));
+ mem->dev_hpa = dev_hpa;
+ goto good_exit;
+ }
+ mem->dev_hpa = MM_IOMMU_TABLE_INVALID_HPA;
+
/*
* For a starting point for a maximum page size calculation
* we use @ua and @entries natural alignment to allow IOMMU pages
@@ -235,6 +247,7 @@ long mm_iommu_new(struct mm_struct *mm, unsigned long ua, unsigned long entries,
mem->hpas[i] = page_to_pfn(page) << PAGE_SHIFT;
}
+good_exit:
atomic64_set(&mem->mapped, 1);
mem->used = 1;
mem->ua = ua;
@@ -251,13 +264,31 @@ long mm_iommu_new(struct mm_struct *mm, unsigned long ua, unsigned long entries,
return ret;
}
+
+long mm_iommu_new(struct mm_struct *mm, unsigned long ua, unsigned long entries,
+ struct mm_iommu_table_group_mem_t **pmem)
+{
+ return mm_iommu_do_alloc(mm, ua, entries, MM_IOMMU_TABLE_INVALID_HPA,
+ pmem);
+}
EXPORT_SYMBOL_GPL(mm_iommu_new);
+long mm_iommu_newdev(struct mm_struct *mm, unsigned long ua,
+ unsigned long entries, unsigned long dev_hpa,
+ struct mm_iommu_table_group_mem_t **pmem)
+{
+ return mm_iommu_do_alloc(mm, ua, entries, dev_hpa, pmem);
+}
+EXPORT_SYMBOL_GPL(mm_iommu_newdev);
+
static void mm_iommu_unpin(struct mm_iommu_table_group_mem_t *mem)
{
long i;
struct page *page = NULL;
+ if (!mem->hpas)
+ return;
+
for (i = 0; i < mem->entries; ++i) {
if (!mem->hpas[i])
continue;
@@ -299,6 +330,7 @@ static void mm_iommu_release(struct mm_iommu_table_group_mem_t *mem)
long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
{
long ret = 0;
+ unsigned long entries, dev_hpa;
mutex_lock(&mem_list_mutex);
@@ -320,9 +352,12 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem)
}
/* @mapped became 0 so now mappings are disabled, release the region */
+ entries = mem->entries;
+ dev_hpa = mem->dev_hpa;
mm_iommu_release(mem);
- mm_iommu_adjust_locked_vm(mm, mem->entries, false);
+ if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA)
+ mm_iommu_adjust_locked_vm(mm, entries, false);
unlock_exit:
mutex_unlock(&mem_list_mutex);
@@ -392,7 +427,7 @@ long mm_iommu_ua_to_hpa(struct mm_iommu_table_group_mem_t *mem,
unsigned long ua, unsigned int pageshift, unsigned long *hpa)
{
const long entry = (ua - mem->ua) >> PAGE_SHIFT;
- u64 *va = &mem->hpas[entry];
+ u64 *va;
if (entry >= mem->entries)
return -EFAULT;
@@ -400,6 +435,12 @@ long mm_iommu_ua_to_hpa(struct mm_iommu_table_group_mem_t *mem,
if (pageshift > mem->pageshift)
return -EFAULT;
+ if (!mem->hpas) {
+ *hpa = mem->dev_hpa + (ua - mem->ua);
+ return 0;
+ }
+
+ va = &mem->hpas[entry];
*hpa = (*va & MM_IOMMU_TABLE_GROUP_PAGE_MASK) | (ua & ~PAGE_MASK);
return 0;
@@ -410,7 +451,6 @@ long mm_iommu_ua_to_hpa_rm(struct mm_iommu_table_group_mem_t *mem,
unsigned long ua, unsigned int pageshift, unsigned long *hpa)
{
const long entry = (ua - mem->ua) >> PAGE_SHIFT;
- void *va = &mem->hpas[entry];
unsigned long *pa;
if (entry >= mem->entries)
@@ -419,7 +459,12 @@ long mm_iommu_ua_to_hpa_rm(struct mm_iommu_table_group_mem_t *mem,
if (pageshift > mem->pageshift)
return -EFAULT;
- pa = (void *) vmalloc_to_phys(va);
+ if (!mem->hpas) {
+ *hpa = mem->dev_hpa + (ua - mem->ua);
+ return 0;
+ }
+
+ pa = (void *) vmalloc_to_phys(&mem->hpas[entry]);
if (!pa)
return -EFAULT;
@@ -449,6 +494,26 @@ extern void mm_iommu_ua_mark_dirty_rm(struct mm_struct *mm, unsigned long ua)
*pa |= MM_IOMMU_TABLE_GROUP_PAGE_DIRTY;
}
+extern bool mm_iommu_is_devmem(struct mm_struct *mm, unsigned long hpa,
+ unsigned int pageshift)
+{
+ struct mm_iommu_table_group_mem_t *mem;
+ const unsigned long pagesize = 1UL << pageshift;
+
+ list_for_each_entry_rcu(mem, &mm->context.iommu_group_mem_list, next) {
+ if (mem->dev_hpa == MM_IOMMU_TABLE_INVALID_HPA)
+ continue;
+
+ if ((mem->dev_hpa <= hpa) &&
+ (hpa + pagesize <= mem->dev_hpa +
+ (mem->entries << PAGE_SHIFT)))
+ return true;
+ }
+
+ return false;
+}
+EXPORT_SYMBOL_GPL(mm_iommu_is_devmem);
+
long mm_iommu_mapped_inc(struct mm_iommu_table_group_mem_t *mem)
{
if (atomic64_inc_not_zero(&mem->mapped))
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 56db071..ed89137 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -222,8 +222,15 @@ static long tce_iommu_register_pages(struct tce_container *container,
return ret;
}
-static bool tce_page_is_contained(struct page *page, unsigned page_shift)
+static bool tce_page_is_contained(struct mm_struct *mm, unsigned long hpa,
+ unsigned int page_shift)
{
+ struct page *page;
+
+ if (mm_iommu_is_devmem(mm, hpa, page_shift))
+ return true;
+
+ page = pfn_to_page(hpa >> PAGE_SHIFT);
/*
* Check that the TCE table granularity is not bigger than the size of
* a page we just found. Otherwise the hardware can get access to
@@ -499,7 +506,8 @@ static int tce_iommu_clear(struct tce_container *container,
direction = DMA_NONE;
oldhpa = 0;
- ret = iommu_tce_xchg(tbl, entry, &oldhpa, &direction);
+ ret = iommu_tce_xchg(container->mm, tbl, entry, &oldhpa,
+ &direction);
if (ret)
continue;
@@ -537,7 +545,6 @@ static long tce_iommu_build(struct tce_container *container,
enum dma_data_direction direction)
{
long i, ret = 0;
- struct page *page;
unsigned long hpa;
enum dma_data_direction dirtmp;
@@ -548,15 +555,16 @@ static long tce_iommu_build(struct tce_container *container,
if (ret)
break;
- page = pfn_to_page(hpa >> PAGE_SHIFT);
- if (!tce_page_is_contained(page, tbl->it_page_shift)) {
+ if (!tce_page_is_contained(container->mm, hpa,
+ tbl->it_page_shift)) {
ret = -EPERM;
break;
}
hpa |= offset;
dirtmp = direction;
- ret = iommu_tce_xchg(tbl, entry + i, &hpa, &dirtmp);
+ ret = iommu_tce_xchg(container->mm, tbl, entry + i, &hpa,
+ &dirtmp);
if (ret) {
tce_iommu_unuse_page(container, hpa);
pr_err("iommu_tce: %s failed ioba=%lx, tce=%lx, ret=%ld\n",
@@ -583,7 +591,6 @@ static long tce_iommu_build_v2(struct tce_container *container,
enum dma_data_direction direction)
{
long i, ret = 0;
- struct page *page;
unsigned long hpa;
enum dma_data_direction dirtmp;
@@ -596,8 +603,8 @@ static long tce_iommu_build_v2(struct tce_container *container,
if (ret)
break;
- page = pfn_to_page(hpa >> PAGE_SHIFT);
- if (!tce_page_is_contained(page, tbl->it_page_shift)) {
+ if (!tce_page_is_contained(container->mm, hpa,
+ tbl->it_page_shift)) {
ret = -EPERM;
break;
}
@@ -610,7 +617,8 @@ static long tce_iommu_build_v2(struct tce_container *container,
if (mm_iommu_mapped_inc(mem))
break;
- ret = iommu_tce_xchg(tbl, entry + i, &hpa, &dirtmp);
+ ret = iommu_tce_xchg(container->mm, tbl, entry + i, &hpa,
+ &dirtmp);
if (ret) {
/* dirtmp cannot be DMA_NONE here */
tce_iommu_unuse_page_v2(container, tbl, entry + i);
--
2.11.0
^ permalink raw reply related
* [PATCH kernel 0/4] vfio/spapr_tce: Reworks for NVIDIA V100 + P9 passthrough (part 1)
From: Alexey Kardashevskiy @ 2018-10-15 9:24 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Alexey Kardashevskiy, Alex Williamson, kvm-ppc, David Gibson
This is a first set of patches required for passing through NVIDIA V100
with coherent memory. The full patchset is here:
https://github.com/aik/linux/tree/nv2
The matching QEMU is here:
https://github.com/aik/qemu/tree/nv2
This particular patchset prepares for having device memory which is not
backed with page structs.
Does 4/4 really need splitting?
Please comment. Thanks.
Alexey Kardashevskiy (4):
powerpc/mm/iommu: Rename mm_iommu_get
powerpc/mm/iommu/vfio_spapr_tce: Change mm_iommu_get to reference a
region
powerpc/mm/iommu: Make mm_iommu_new() fail on existing regions
powerpc/vfio/iommu/kvm: Do not pin device memory
arch/powerpc/include/asm/iommu.h | 5 +-
arch/powerpc/include/asm/mmu_context.h | 9 +++-
arch/powerpc/kernel/iommu.c | 9 ++--
arch/powerpc/kvm/book3s_64_vio.c | 18 ++++---
arch/powerpc/mm/mmu_context_iommu.c | 97 +++++++++++++++++++++++++++++-----
drivers/vfio/vfio_iommu_spapr_tce.c | 65 +++++++++++++++--------
6 files changed, 151 insertions(+), 52 deletions(-)
--
2.11.0
^ permalink raw reply
* [PATCH kernel 2/4] powerpc/mm/iommu/vfio_spapr_tce: Change mm_iommu_get to reference a region
From: Alexey Kardashevskiy @ 2018-10-15 9:24 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Alexey Kardashevskiy, Alex Williamson, kvm-ppc, David Gibson
In-Reply-To: <20181015092416.47380-1-aik@ozlabs.ru>
We are going to add another helper to preregister device memory so
instead of having mm_iommu_new() which pre-registers the normal memory
and references the region, we need separate helpers for pre-registerign
and referencing.
To make the mm_iommu_get name reflect what it is supposed to do, this
changes mm_iommu_get() to reference the region so from now on for every
mm_iommu_get() we need a matching mm_iommu_put().
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
arch/powerpc/mm/mmu_context_iommu.c | 5 +++++
drivers/vfio/vfio_iommu_spapr_tce.c | 33 ++++++++++++++++++++++-----------
2 files changed, 27 insertions(+), 11 deletions(-)
diff --git a/arch/powerpc/mm/mmu_context_iommu.c b/arch/powerpc/mm/mmu_context_iommu.c
index 8eeb99d..a8c4a3c 100644
--- a/arch/powerpc/mm/mmu_context_iommu.c
+++ b/arch/powerpc/mm/mmu_context_iommu.c
@@ -373,13 +373,18 @@ struct mm_iommu_table_group_mem_t *mm_iommu_get(struct mm_struct *mm,
{
struct mm_iommu_table_group_mem_t *mem, *ret = NULL;
+ mutex_lock(&mem_list_mutex);
+
list_for_each_entry_rcu(mem, &mm->context.iommu_group_mem_list, next) {
if ((mem->ua == ua) && (mem->entries == entries)) {
ret = mem;
+ ++mem->used;
break;
}
}
+ mutex_unlock(&mem_list_mutex);
+
return ret;
}
EXPORT_SYMBOL_GPL(mm_iommu_get);
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 1701798..56db071 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -151,7 +151,8 @@ static long tce_iommu_unregister_pages(struct tce_container *container,
{
struct mm_iommu_table_group_mem_t *mem;
struct tce_iommu_prereg *tcemem;
- bool found = false;
+ bool found;
+ long ret;
if ((vaddr & ~PAGE_MASK) || (size & ~PAGE_MASK))
return -EINVAL;
@@ -168,9 +169,13 @@ static long tce_iommu_unregister_pages(struct tce_container *container,
}
if (!found)
- return -ENOENT;
+ ret = -ENOENT;
+ else
+ ret = tce_iommu_prereg_free(container, tcemem);
- return tce_iommu_prereg_free(container, tcemem);
+ mm_iommu_put(container->mm, mem);
+
+ return ret;
}
static long tce_iommu_register_pages(struct tce_container *container,
@@ -188,19 +193,21 @@ static long tce_iommu_register_pages(struct tce_container *container,
mem = mm_iommu_get(container->mm, vaddr, entries);
if (mem) {
list_for_each_entry(tcemem, &container->prereg_list, next) {
- if (tcemem->mem == mem)
- return -EBUSY;
+ if (tcemem->mem == mem) {
+ ret = -EBUSY;
+ goto put_exit;
+ }
}
+ } else {
+ ret = mm_iommu_new(container->mm, vaddr, entries, &mem);
+ if (ret)
+ return ret;
}
- ret = mm_iommu_new(container->mm, vaddr, entries, &mem);
- if (ret)
- return ret;
-
tcemem = kzalloc(sizeof(*tcemem), GFP_KERNEL);
if (!tcemem) {
- mm_iommu_put(container->mm, mem);
- return -ENOMEM;
+ ret = -ENOMEM;
+ goto put_exit;
}
tcemem->mem = mem;
@@ -209,6 +216,10 @@ static long tce_iommu_register_pages(struct tce_container *container,
container->enabled = true;
return 0;
+
+put_exit:
+ mm_iommu_put(container->mm, mem);
+ return ret;
}
static bool tce_page_is_contained(struct page *page, unsigned page_shift)
--
2.11.0
^ permalink raw reply related
* Re: [PATCH v3 06/24] powerpc/mm: properly set PAGE_KERNEL flags in ioremap()
From: Michael Ellerman @ 2018-10-15 9:25 UTC (permalink / raw)
To: LEROY Christophe
Cc: Jens Axboe, Scott Wood, linux-fbdev, Dominik Brodowski,
Bartlomiej Zolnierkiewicz, Geoff Levand, Nicholas Piggin,
linux-kernel, dri-devel, Li Yang, linux-block, Paul Mackerras,
aneesh.kumar, linuxppc-dev, linux-arm-kernel
In-Reply-To: <20181014115810.Horde.99_1JiRv0nxT_Jq3cbpMhw1@messagerie.si.c-s.fr>
LEROY Christophe <christophe.leroy@c-s.fr> writes:
> Michael Ellerman <mpe@ellerman.id.au> a écrit :
>> Michael Ellerman <mpe@ellerman.id.au> writes:
>>> Christophe Leroy <christophe.leroy@c-s.fr> writes:
>>>
>>>> Set PAGE_KERNEL directly in the caller and do not rely on a
>>>> hack adding PAGE_KERNEL flags when _PAGE_PRESENT is not set.
>>>>
>>>> As already done for PPC64, use pgprot_cache() helpers instead of
>>>> _PAGE_XXX flags in PPC32 ioremap() derived functions.
>>>>
>>>> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
>>>
>>> Something in here is breaking my p5020ds (both 32-bit and 64-bit):
>>
>> Oh duh.
>>
>> That's because I didn't take patch 4.
>>
>> It didn't have any acks, but I guess I'll just merge it rather than
>> breaking things.
>
> Yes indeed. Maybe should I have followed it more carrefully to ensure
> it gets an ack.
That's OK, I should have paid more attention to what it was actually
doing, I thought it was just a cleanup.
cheers
^ permalink raw reply
* [PATCH kernel 1/5] powerpc/powernv/npu: Add helper to access struct npu for NPU device
From: Alexey Kardashevskiy @ 2018-10-15 9:32 UTC (permalink / raw)
To: linuxppc-dev
Cc: Alexey Kardashevskiy, Alistair Popple, Frederic Barrat,
Alex Williamson, kvm-ppc, David Gibson
In-Reply-To: <20181015093301.1007-1-aik@ozlabs.ru>
This step is to help removing the npu struct from pnv_phb so it
can be used by pseries as well.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
arch/powerpc/platforms/powernv/npu-dma.c | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c
index 3a5c4ed..13e5153 100644
--- a/arch/powerpc/platforms/powernv/npu-dma.c
+++ b/arch/powerpc/platforms/powernv/npu-dma.c
@@ -389,6 +389,18 @@ struct pnv_ioda_pe *pnv_pci_npu_setup_iommu(struct pnv_ioda_pe *npe)
return gpe;
}
+/*
+ * NPU2 ATS
+ */
+static struct npu *npdev_to_npu(struct pci_dev *npdev)
+{
+ struct pnv_phb *nphb;
+
+ nphb = pci_bus_to_host(npdev->bus)->private_data;
+
+ return &nphb->npu;
+}
+
/* Maximum number of nvlinks per npu */
#define NV_MAX_LINKS 6
@@ -546,7 +558,6 @@ static void acquire_atsd_reg(struct npu_context *npu_context,
int i, j;
struct npu *npu;
struct pci_dev *npdev;
- struct pnv_phb *nphb;
for (i = 0; i <= max_npu2_index; i++) {
mmio_atsd_reg[i].reg = -1;
@@ -561,8 +572,7 @@ static void acquire_atsd_reg(struct npu_context *npu_context,
if (!npdev)
continue;
- nphb = pci_bus_to_host(npdev->bus)->private_data;
- npu = &nphb->npu;
+ npu = npdev_to_npu(npdev);
mmio_atsd_reg[i].npu = npu;
mmio_atsd_reg[i].reg = get_mmio_atsd_reg(npu);
while (mmio_atsd_reg[i].reg < 0) {
@@ -749,7 +759,7 @@ struct npu_context *pnv_npu2_init_context(struct pci_dev *gpdev,
}
nphb = pci_bus_to_host(npdev->bus)->private_data;
- npu = &nphb->npu;
+ npu = npdev_to_npu(npdev);
/*
* Setup the NPU context table for a particular GPU. These need to be
@@ -869,7 +879,7 @@ void pnv_npu2_destroy_context(struct npu_context *npu_context,
return;
nphb = pci_bus_to_host(npdev->bus)->private_data;
- npu = &nphb->npu;
+ npu = npdev_to_npu(npdev);
nvlink_dn = of_parse_phandle(npdev->dev.of_node, "ibm,nvlink", 0);
if (WARN_ON(of_property_read_u32(nvlink_dn, "ibm,npu-link-index",
&nvlink_index)))
--
2.11.0
^ permalink raw reply related
* [PATCH kernel 5/5] powerpc/powernv/npu: Add helper to map GPU to LPAR
From: Alexey Kardashevskiy @ 2018-10-15 9:33 UTC (permalink / raw)
To: linuxppc-dev
Cc: Alexey Kardashevskiy, Alistair Popple, Frederic Barrat,
Alex Williamson, kvm-ppc, David Gibson
In-Reply-To: <20181015093301.1007-1-aik@ozlabs.ru>
In order to make ATS work and translate addresses for arbitrary
LPID and PID, we need to program an NPU with these.
This implements a helper to assign a GPU to LPAR and program the NPU
with a wildcard for PID. The helper also takes MSR (only DR/HV/PR/SF bits
are allowed) to program them into NPU2 for ATS checkout requests.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
arch/powerpc/include/asm/pci.h | 2 ++
arch/powerpc/platforms/powernv/npu-dma.c | 38 ++++++++++++++++++++++++++++++++
2 files changed, 40 insertions(+)
diff --git a/arch/powerpc/include/asm/pci.h b/arch/powerpc/include/asm/pci.h
index f196df6..c3c9728 100644
--- a/arch/powerpc/include/asm/pci.h
+++ b/arch/powerpc/include/asm/pci.h
@@ -131,5 +131,7 @@ extern struct pci_dev *pnv_pci_get_gpu_dev(struct pci_dev *npdev);
extern struct pci_dev *pnv_pci_get_npu_dev(struct pci_dev *gpdev, int index);
extern void pnv_npu2_devices_init(void);
extern int pnv_npu2_init(struct pci_controller *hose);
+extern int pnv_npu2_map_lpar_dev(struct pci_controller *hose,
+ struct pci_dev *gpdev, unsigned int lparid, unsigned long msr);
#endif /* __ASM_POWERPC_PCI_H */
diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c
index 677f30a..1dde753 100644
--- a/arch/powerpc/platforms/powernv/npu-dma.c
+++ b/arch/powerpc/platforms/powernv/npu-dma.c
@@ -1047,3 +1047,41 @@ void pnv_npu2_map_lpar_phb(struct pnv_phb *nphb, unsigned long msr)
dev_err(&gpdev->dev, "Failed to init context: %d\n", ret);
}
}
+
+int pnv_npu2_map_lpar_dev(struct pci_controller *hose, struct pci_dev *gpdev,
+ unsigned int lparid, unsigned long msr)
+{
+ int ret;
+ struct pnv_phb *nphb = hose->private_data;
+
+ dev_dbg(&gpdev->dev, "Map LPAR opalid=%llu lparid=%u\n",
+ nphb->opal_id, lparid);
+ /*
+ * Currently we only support radix and non-zero LPCR only makes sense
+ * for hash tables so skiboot expects the LPCR parameter to be a zero.
+ */
+ ret = opal_npu_map_lpar(nphb->opal_id,
+ PCI_DEVID(gpdev->bus->number, gpdev->devfn), lparid,
+ 0 /* LPCR bits */);
+ if (ret) {
+ dev_err(&gpdev->dev, "Error %d mapping device to LPAR\n", ret);
+ return ret;
+ }
+
+ dev_dbg(&gpdev->dev, "destroy context opalid=%llu msr=%lx\n",
+ nphb->opal_id, msr);
+ ret = opal_npu_destroy_context(nphb->opal_id, 0/*__unused*/,
+ PCI_DEVID(gpdev->bus->number, gpdev->devfn));
+ if (ret)
+ dev_err(&gpdev->dev, "Failed to destroy context: %d\n", ret);
+
+ dev_dbg(&gpdev->dev, "init context opalid=%llu msr=%lx\n",
+ nphb->opal_id, msr);
+ ret = opal_npu_init_context(nphb->opal_id, 0/*__unused*/, msr,
+ PCI_DEVID(gpdev->bus->number, gpdev->devfn));
+ if (ret)
+ dev_err(&gpdev->dev, "Failed to init context: %d\n", ret);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(pnv_npu2_map_lpar_dev);
--
2.11.0
^ permalink raw reply related
* Re: [PATCH] powerpc/64: properly initialise the stackprotector canary on SMP.
From: Abdul Haleem @ 2018-10-15 9:33 UTC (permalink / raw)
To: Christophe Leroy; +Cc: linuxppc-dev, Paul Mackerras, linux-kernel
In-Reply-To: <a2694b8ffeb6c52d1c755d8a7bfaac689c4e1c6d.1539423893.git.christophe.leroy@c-s.fr>
On Sat, 2018-10-13 at 09:45 +0000, Christophe Leroy wrote:
> commit 06ec27aea9fc ("powerpc/64: add stack protector support")
> doesn't initialise the stack canary on SMP secondary CPU's paca,
> leading to the following false positive report from the
> stack protector.
>
> smp: Bringing up secondary CPUs ...
> Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: __schedule+0x978/0xa80
> CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.0-rc7-next-20181010-autotest-autotest #1
> Call Trace:
> [c000001fed5b3bf0] [c000000000a0ef3c] dump_stack+0xb0/0xf4 (unreliable)
> [c000001fed5b3c30] [c0000000000f9d68] panic+0x140/0x308
> [c000001fed5b3cc0] [c0000000000f9844] __stack_chk_fail+0x24/0x30
> [c000001fed5b3d20] [c000000000a2c3a8] __schedule+0x978/0xa80
> [c000001fed5b3e00] [c000000000a2c9b4] schedule_idle+0x34/0x60
> [c000001fed5b3e30] [c00000000013d344] do_idle+0x224/0x3d0
> [c000001fed5b3ec0] [c00000000013d6e0] cpu_startup_entry+0x30/0x50
> [c000001fed5b3ef0] [c000000000047f34] start_secondary+0x4d4/0x520
> [c000001fed5b3f90] [c00000000000b370] start_secondary_prolog+0x10/0x14
>
> This patch properly initialises the stack_canary of the secondary
> idle tasks.
>
> Reported-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com>
> Fixes: 06ec27aea9fc ("powerpc/64: add stack protector support")
> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
> ---
> arch/powerpc/kernel/smp.c | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> index 61c1fadbc644..e774d3bf3a03 100644
> --- a/arch/powerpc/kernel/smp.c
> +++ b/arch/powerpc/kernel/smp.c
> @@ -34,6 +34,7 @@
> #include <linux/topology.h>
> #include <linux/profile.h>
> #include <linux/processor.h>
> +#include <linux/random.h>
>
> #include <asm/ptrace.h>
> #include <linux/atomic.h>
> @@ -810,9 +811,16 @@ static void cpu_idle_thread_init(unsigned int cpu, struct task_struct *idle)
> {
> struct thread_info *ti = task_thread_info(idle);
>
> +#ifdef CONFIG_STACKPROTECTOR
> + idle->stack_canary = get_random_canary();
> +#endif
> +
> #ifdef CONFIG_PPC64
> paca_ptrs[cpu]->__current = idle;
> paca_ptrs[cpu]->kstack = (unsigned long)ti + THREAD_SIZE - STACK_FRAME_OVERHEAD;
> +#ifdef CONFIG_STACKPROTECTOR
> + paca_ptrs[cpu]->canary = idle->stack_canary;
> +#endif
> #endif
> ti->cpu = cpu;
> secondary_ti = current_set[cpu] = ti;
Thank you Christophe for the fix, Machine boots fine with above patch.
Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com>
--
Regard's
Abdul Haleem
IBM Linux Technology Centre
^ permalink raw reply
* [PATCH kernel 0/5] powerpc/powernv/npu: Reworks for NVIDIA V100 + P9 passthrough (part 2)
From: Alexey Kardashevskiy @ 2018-10-15 9:32 UTC (permalink / raw)
To: linuxppc-dev
Cc: Alexey Kardashevskiy, Alistair Popple, Frederic Barrat,
Alex Williamson, kvm-ppc, David Gibson
This is a second set of patches required for passing through NVIDIA V100
with coherent memory. The full patchset is here:
https://github.com/aik/linux/tree/nv2
The matching QEMU is here:
https://github.com/aik/qemu/tree/nv2
This patchset reworks NPU DMA code to be used with VFIO. The exported symbols
are called from the NVIDIA driver so they must be able to work in both
powernv and pseries platforms.
This depends on skiboot's "npu2: Clear XTS_BDF_MAP when destroying context for next init_context"
posted earlier today.
Please comment. Thanks.
Alexey Kardashevskiy (5):
powerpc/powernv/npu: Add helper to access struct npu for NPU device
powerpc/powernv/npu: Collect all static symbols under one struct
powerpc/powernv: Detach npu struct from pnv_phb
powerpc/powernv/npu: Factor out OPAL calls from context manipulation
powerpc/powernv/npu: Add helper to map GPU to LPAR
arch/powerpc/include/asm/pci.h | 4 +
arch/powerpc/platforms/powernv/pci.h | 18 +-
arch/powerpc/platforms/powernv/npu-dma.c | 298 ++++++++++++++++++++----------
arch/powerpc/platforms/powernv/pci-ioda.c | 9 +-
4 files changed, 212 insertions(+), 117 deletions(-)
--
2.11.0
^ permalink raw reply
* [PATCH kernel 2/5] powerpc/powernv/npu: Collect all static symbols under one struct
From: Alexey Kardashevskiy @ 2018-10-15 9:32 UTC (permalink / raw)
To: linuxppc-dev
Cc: Alexey Kardashevskiy, Alistair Popple, Frederic Barrat,
Alex Williamson, kvm-ppc, David Gibson
In-Reply-To: <20181015093301.1007-1-aik@ozlabs.ru>
We are going to add a global list of NPUs in the system which is going
to be yet another static symbol. Let's reorganise the code and put all
static symbols into one struct for better tracking what is really needed
for NPU (this might become a driver data some day).
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
arch/powerpc/include/asm/pci.h | 1 +
arch/powerpc/platforms/powernv/npu-dma.c | 77 ++++++++++++++++++-------------
arch/powerpc/platforms/powernv/pci-ioda.c | 2 +
3 files changed, 47 insertions(+), 33 deletions(-)
diff --git a/arch/powerpc/include/asm/pci.h b/arch/powerpc/include/asm/pci.h
index 2af9ded..1a96075 100644
--- a/arch/powerpc/include/asm/pci.h
+++ b/arch/powerpc/include/asm/pci.h
@@ -129,5 +129,6 @@ extern void pcibios_scan_phb(struct pci_controller *hose);
extern struct pci_dev *pnv_pci_get_gpu_dev(struct pci_dev *npdev);
extern struct pci_dev *pnv_pci_get_npu_dev(struct pci_dev *gpdev, int index);
+extern void pnv_npu2_devices_init(void);
#endif /* __ASM_POWERPC_PCI_H */
diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c
index 13e5153..01402f9 100644
--- a/arch/powerpc/platforms/powernv/npu-dma.c
+++ b/arch/powerpc/platforms/powernv/npu-dma.c
@@ -22,20 +22,6 @@
#include "pci.h"
/*
- * spinlock to protect initialisation of an npu_context for a particular
- * mm_struct.
- */
-static DEFINE_SPINLOCK(npu_context_lock);
-
-/*
- * When an address shootdown range exceeds this threshold we invalidate the
- * entire TLB on the GPU for the given PID rather than each specific address in
- * the range.
- */
-static uint64_t atsd_threshold = 2 * 1024 * 1024;
-static struct dentry *atsd_threshold_dentry;
-
-/*
* Other types of TCE cache invalidation are not functional in the
* hardware.
*/
@@ -392,6 +378,33 @@ struct pnv_ioda_pe *pnv_pci_npu_setup_iommu(struct pnv_ioda_pe *npe)
/*
* NPU2 ATS
*/
+static struct {
+ /*
+ * spinlock to protect initialisation of an npu_context for
+ * a particular mm_struct.
+ */
+ spinlock_t context_lock;
+
+ /* Maximum index of npu2 hosts in the system. Always < NV_MAX_NPUS */
+ int max_index;
+
+ /*
+ * When an address shootdown range exceeds this threshold we invalidate the
+ * entire TLB on the GPU for the given PID rather than each specific address in
+ * the range.
+ */
+ uint64_t atsd_threshold;
+ struct dentry *atsd_threshold_dentry;
+
+} npu2_devices;
+
+void pnv_npu2_devices_init(void)
+{
+ memset(&npu2_devices, 0, sizeof(npu2_devices));
+ spin_lock_init(&npu2_devices.context_lock);
+ npu2_devices.atsd_threshold = 2 * 1024 * 1024;
+}
+
static struct npu *npdev_to_npu(struct pci_dev *npdev)
{
struct pnv_phb *nphb;
@@ -404,9 +417,6 @@ static struct npu *npdev_to_npu(struct pci_dev *npdev)
/* Maximum number of nvlinks per npu */
#define NV_MAX_LINKS 6
-/* Maximum index of npu2 hosts in the system. Always < NV_MAX_NPUS */
-static int max_npu2_index;
-
struct npu_context {
struct mm_struct *mm;
struct pci_dev *npdev[NV_MAX_NPUS][NV_MAX_LINKS];
@@ -472,7 +482,7 @@ static void mmio_invalidate_pid(struct mmio_atsd_reg mmio_atsd_reg[NV_MAX_NPUS],
int i;
unsigned long launch;
- for (i = 0; i <= max_npu2_index; i++) {
+ for (i = 0; i <= npu2_devices.max_index; i++) {
if (mmio_atsd_reg[i].reg < 0)
continue;
@@ -503,7 +513,7 @@ static void mmio_invalidate_va(struct mmio_atsd_reg mmio_atsd_reg[NV_MAX_NPUS],
int i;
unsigned long launch;
- for (i = 0; i <= max_npu2_index; i++) {
+ for (i = 0; i <= npu2_devices.max_index; i++) {
if (mmio_atsd_reg[i].reg < 0)
continue;
@@ -536,7 +546,7 @@ static void mmio_invalidate_wait(
int i, reg;
/* Wait for all invalidations to complete */
- for (i = 0; i <= max_npu2_index; i++) {
+ for (i = 0; i <= npu2_devices.max_index; i++) {
if (mmio_atsd_reg[i].reg < 0)
continue;
@@ -559,7 +569,7 @@ static void acquire_atsd_reg(struct npu_context *npu_context,
struct npu *npu;
struct pci_dev *npdev;
- for (i = 0; i <= max_npu2_index; i++) {
+ for (i = 0; i <= npu2_devices.max_index; i++) {
mmio_atsd_reg[i].reg = -1;
for (j = 0; j < NV_MAX_LINKS; j++) {
/*
@@ -593,7 +603,7 @@ static void release_atsd_reg(struct mmio_atsd_reg mmio_atsd_reg[NV_MAX_NPUS])
{
int i;
- for (i = 0; i <= max_npu2_index; i++) {
+ for (i = 0; i <= npu2_devices.max_index; i++) {
/*
* We can't rely on npu_context->npdev[][] being the same here
* as when acquire_atsd_reg() was called, hence we use the
@@ -683,7 +693,7 @@ static void pnv_npu2_mn_invalidate_range(struct mmu_notifier *mn,
struct npu_context *npu_context = mn_to_npu_context(mn);
unsigned long address;
- if (end - start > atsd_threshold) {
+ if (end - start > npu2_devices.atsd_threshold) {
/*
* Just invalidate the entire PID if the address range is too
* large.
@@ -777,12 +787,12 @@ struct npu_context *pnv_npu2_init_context(struct pci_dev *gpdev,
* We store the npu pci device so we can more easily get at the
* associated npus.
*/
- spin_lock(&npu_context_lock);
+ spin_lock(&npu2_devices.context_lock);
npu_context = mm->context.npu_context;
if (npu_context) {
if (npu_context->release_cb != cb ||
npu_context->priv != priv) {
- spin_unlock(&npu_context_lock);
+ spin_unlock(&npu2_devices.context_lock);
opal_npu_destroy_context(nphb->opal_id, mm->context.id,
PCI_DEVID(gpdev->bus->number,
gpdev->devfn));
@@ -791,12 +801,12 @@ struct npu_context *pnv_npu2_init_context(struct pci_dev *gpdev,
WARN_ON(!kref_get_unless_zero(&npu_context->kref));
}
- spin_unlock(&npu_context_lock);
+ spin_unlock(&npu2_devices.context_lock);
if (!npu_context) {
/*
* We can set up these fields without holding the
- * npu_context_lock as the npu_context hasn't been returned to
+ * npu2_devices.context_lock as the npu_context hasn't been returned to
* the caller meaning it can't be destroyed. Parallel allocation
* is protected against by mmap_sem.
*/
@@ -887,9 +897,9 @@ void pnv_npu2_destroy_context(struct npu_context *npu_context,
WRITE_ONCE(npu_context->npdev[npu->index][nvlink_index], NULL);
opal_npu_destroy_context(nphb->opal_id, npu_context->mm->context.id,
PCI_DEVID(gpdev->bus->number, gpdev->devfn));
- spin_lock(&npu_context_lock);
+ spin_lock(&npu2_devices.context_lock);
removed = kref_put(&npu_context->kref, pnv_npu2_release_context);
- spin_unlock(&npu_context_lock);
+ spin_unlock(&npu2_devices.context_lock);
/*
* We need to do this outside of pnv_npu2_release_context so that it is
@@ -958,9 +968,10 @@ int pnv_npu2_init(struct pnv_phb *phb)
static int npu_index;
uint64_t rc = 0;
- if (!atsd_threshold_dentry) {
- atsd_threshold_dentry = debugfs_create_x64("atsd_threshold",
- 0600, powerpc_debugfs_root, &atsd_threshold);
+ if (!npu2_devices.atsd_threshold_dentry) {
+ npu2_devices.atsd_threshold_dentry = debugfs_create_x64(
+ "atsd_threshold", 0600, powerpc_debugfs_root,
+ &npu2_devices.atsd_threshold);
}
phb->npu.nmmu_flush =
@@ -988,7 +999,7 @@ int pnv_npu2_init(struct pnv_phb *phb)
npu_index++;
if (WARN_ON(npu_index >= NV_MAX_NPUS))
return -ENOSPC;
- max_npu2_index = npu_index;
+ npu2_devices.max_index = npu_index;
phb->npu.index = npu_index;
return 0;
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index e37b9cc..0cc81c0 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1279,6 +1279,8 @@ static void pnv_pci_ioda_setup_PEs(void)
struct pci_bus *bus;
struct pci_dev *pdev;
+ pnv_npu2_devices_init();
+
list_for_each_entry_safe(hose, tmp, &hose_list, list_node) {
phb = hose->private_data;
if (phb->type == PNV_PHB_NPU_NVLINK) {
--
2.11.0
^ permalink raw reply related
* [PATCH kernel 3/5] powerpc/powernv: Detach npu struct from pnv_phb
From: Alexey Kardashevskiy @ 2018-10-15 9:32 UTC (permalink / raw)
To: linuxppc-dev
Cc: Alexey Kardashevskiy, Alistair Popple, Frederic Barrat,
Alex Williamson, kvm-ppc, David Gibson
In-Reply-To: <20181015093301.1007-1-aik@ozlabs.ru>
The powernv PCI code stores NPU data in the pnv_phb struct. The latter
is referenced by pci_controller::private_data. We are going to have NPU2
support in the pseries platform as well but it does not store any
private_data in in the pci_controller struct; and even if it did,
it would be a different data structure.
This adds a global list of NPUs so each platform can register and use
these in the same fashion.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
arch/powerpc/platforms/powernv/pci.h | 16 -------
arch/powerpc/platforms/powernv/npu-dma.c | 71 +++++++++++++++++++++++++-------
2 files changed, 57 insertions(+), 30 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 8b37b28..3b7617d 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -8,9 +8,6 @@
struct pci_dn;
-/* Maximum possible number of ATSD MMIO registers per NPU */
-#define NV_NMMU_ATSD_REGS 8
-
enum pnv_phb_type {
PNV_PHB_IODA1 = 0,
PNV_PHB_IODA2 = 1,
@@ -180,19 +177,6 @@ struct pnv_phb {
unsigned int diag_data_size;
u8 *diag_data;
- /* Nvlink2 data */
- struct npu {
- int index;
- __be64 *mmio_atsd_regs[NV_NMMU_ATSD_REGS];
- unsigned int mmio_atsd_count;
-
- /* Bitmask for MMIO register usage */
- unsigned long mmio_atsd_usage;
-
- /* Do we need to explicitly flush the nest mmu? */
- bool nmmu_flush;
- } npu;
-
int p2p_target_count;
};
diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c
index 01402f9..cb2b4f9 100644
--- a/arch/powerpc/platforms/powernv/npu-dma.c
+++ b/arch/powerpc/platforms/powernv/npu-dma.c
@@ -378,6 +378,25 @@ struct pnv_ioda_pe *pnv_pci_npu_setup_iommu(struct pnv_ioda_pe *npe)
/*
* NPU2 ATS
*/
+/* Maximum possible number of ATSD MMIO registers per NPU */
+#define NV_NMMU_ATSD_REGS 8
+
+struct npu {
+ int index;
+ __be64 *mmio_atsd_regs[NV_NMMU_ATSD_REGS];
+ unsigned int mmio_atsd_count;
+
+ /* Bitmask for MMIO register usage */
+ unsigned long mmio_atsd_usage;
+
+ /* Do we need to explicitly flush the nest mmu? */
+ bool nmmu_flush;
+
+ struct list_head next;
+
+ struct pci_controller *hose;
+};
+
static struct {
/*
* spinlock to protect initialisation of an npu_context for
@@ -396,22 +415,27 @@ static struct {
uint64_t atsd_threshold;
struct dentry *atsd_threshold_dentry;
+ struct list_head npu_list;
} npu2_devices;
void pnv_npu2_devices_init(void)
{
memset(&npu2_devices, 0, sizeof(npu2_devices));
+ INIT_LIST_HEAD(&npu2_devices.npu_list);
spin_lock_init(&npu2_devices.context_lock);
npu2_devices.atsd_threshold = 2 * 1024 * 1024;
}
static struct npu *npdev_to_npu(struct pci_dev *npdev)
{
- struct pnv_phb *nphb;
+ struct pci_controller *hose = pci_bus_to_host(npdev->bus);
+ struct npu *npu;
- nphb = pci_bus_to_host(npdev->bus)->private_data;
+ list_for_each_entry(npu, &npu2_devices.npu_list, next)
+ if (hose == npu->hose)
+ return npu;
- return &nphb->npu;
+ return NULL;
}
/* Maximum number of nvlinks per npu */
@@ -843,7 +867,7 @@ struct npu_context *pnv_npu2_init_context(struct pci_dev *gpdev,
*/
WRITE_ONCE(npu_context->npdev[npu->index][nvlink_index], npdev);
- if (!nphb->npu.nmmu_flush) {
+ if (!npu->nmmu_flush) {
/*
* If we're not explicitly flushing ourselves we need to mark
* the thread for global flushes
@@ -967,6 +991,13 @@ int pnv_npu2_init(struct pnv_phb *phb)
struct pci_dev *gpdev;
static int npu_index;
uint64_t rc = 0;
+ struct pci_controller *hose = phb->hose;
+ struct npu *npu;
+ int ret;
+
+ npu = kzalloc(sizeof(*npu), GFP_KERNEL);
+ if (!npu)
+ return -ENOMEM;
if (!npu2_devices.atsd_threshold_dentry) {
npu2_devices.atsd_threshold_dentry = debugfs_create_x64(
@@ -974,8 +1005,7 @@ int pnv_npu2_init(struct pnv_phb *phb)
&npu2_devices.atsd_threshold);
}
- phb->npu.nmmu_flush =
- of_property_read_bool(phb->hose->dn, "ibm,nmmu-flush");
+ npu->nmmu_flush = of_property_read_bool(hose->dn, "ibm,nmmu-flush");
for_each_child_of_node(phb->hose->dn, dn) {
gpdev = pnv_pci_get_gpu_dev(get_pci_dev(dn));
if (gpdev) {
@@ -989,18 +1019,31 @@ int pnv_npu2_init(struct pnv_phb *phb)
}
}
- for (i = 0; !of_property_read_u64_index(phb->hose->dn, "ibm,mmio-atsd",
+ for (i = 0; !of_property_read_u64_index(hose->dn, "ibm,mmio-atsd",
i, &mmio_atsd); i++)
- phb->npu.mmio_atsd_regs[i] = ioremap(mmio_atsd, 32);
+ npu->mmio_atsd_regs[i] = ioremap(mmio_atsd, 32);
- pr_info("NPU%lld: Found %d MMIO ATSD registers", phb->opal_id, i);
- phb->npu.mmio_atsd_count = i;
- phb->npu.mmio_atsd_usage = 0;
+ pr_info("NPU%d: Found %d MMIO ATSD registers", hose->global_number, i);
+ npu->mmio_atsd_count = i;
+ npu->mmio_atsd_usage = 0;
npu_index++;
- if (WARN_ON(npu_index >= NV_MAX_NPUS))
- return -ENOSPC;
+ if (WARN_ON(npu_index >= NV_MAX_NPUS)) {
+ ret = -ENOSPC;
+ goto fail_exit;
+ }
npu2_devices.max_index = npu_index;
- phb->npu.index = npu_index;
+ npu->index = npu_index;
+ npu->hose = hose;
+
+ list_add(&npu->next, &npu2_devices.npu_list);
return 0;
+
+fail_exit:
+ for (i = 0; i < npu->mmio_atsd_count; ++i)
+ iounmap(npu->mmio_atsd_regs[i]);
+
+ kfree(npu);
+
+ return ret;
}
--
2.11.0
^ permalink raw reply related
* [PATCH kernel 4/5] powerpc/powernv/npu: Factor out OPAL calls from context manipulation
From: Alexey Kardashevskiy @ 2018-10-15 9:33 UTC (permalink / raw)
To: linuxppc-dev
Cc: Alexey Kardashevskiy, Alistair Popple, Frederic Barrat,
Alex Williamson, kvm-ppc, David Gibson
In-Reply-To: <20181015093301.1007-1-aik@ozlabs.ru>
At the moment the NPU context init/destroy code calls OPAL. The init
handler in OPAL configures the NPU to pass ATS requests to nested MMU,
the destroy handler does nothing besides sanity checks.
Since the init handler programs the NPU with a wildcard for LPID/PID,
this can be done at the point where a GPU is mapped to an LPAR; it also
makes calling opal_npu_destroy_context() unnecessary in this context
(this will change with VFIO later though).
Also, the pnv_npu2_init() helper does not really need to call OPAL as
well as it inialized an NPU structure and does not interact with GPU or
NPU at that moment.
This moves OPAL calls to a separate helper. With this change, the API
for GPUs does not do any OPAL calls and therefore can be used by both
pseries and powernv platforms. The new pnv_npu2_map_lpar_phb() helper
should be called on powernv only as it does OPAL calls and it takes
an MSR mask which NPU adds to ATS requests so nested MMU knows what
translations are permitted; the VFIO/KVM will not set MSR_HV.
This removes the check for FW_FEATURE_OPAL as pnv_npu2_init_context/
pnv_npu2_release_context/pnv_npu2_init do not call OPAL anymore.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
arch/powerpc/include/asm/pci.h | 1 +
arch/powerpc/platforms/powernv/pci.h | 2 +-
arch/powerpc/platforms/powernv/npu-dma.c | 100 +++++++++++++++---------------
arch/powerpc/platforms/powernv/pci-ioda.c | 7 ++-
4 files changed, 57 insertions(+), 53 deletions(-)
diff --git a/arch/powerpc/include/asm/pci.h b/arch/powerpc/include/asm/pci.h
index 1a96075..f196df6 100644
--- a/arch/powerpc/include/asm/pci.h
+++ b/arch/powerpc/include/asm/pci.h
@@ -130,5 +130,6 @@ extern void pcibios_scan_phb(struct pci_controller *hose);
extern struct pci_dev *pnv_pci_get_gpu_dev(struct pci_dev *npdev);
extern struct pci_dev *pnv_pci_get_npu_dev(struct pci_dev *gpdev, int index);
extern void pnv_npu2_devices_init(void);
+extern int pnv_npu2_init(struct pci_controller *hose);
#endif /* __ASM_POWERPC_PCI_H */
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 3b7617d..ca2ce4b 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -224,7 +224,7 @@ extern long pnv_npu_set_window(struct pnv_ioda_pe *npe, int num,
extern long pnv_npu_unset_window(struct pnv_ioda_pe *npe, int num);
extern void pnv_npu_take_ownership(struct pnv_ioda_pe *npe);
extern void pnv_npu_release_ownership(struct pnv_ioda_pe *npe);
-extern int pnv_npu2_init(struct pnv_phb *phb);
+extern void pnv_npu2_map_lpar_phb(struct pnv_phb *nphb, unsigned long msr);
/* pci-ioda-tce.c */
#define POWERNV_IOMMU_DEFAULT_LEVELS 1
diff --git a/arch/powerpc/platforms/powernv/npu-dma.c b/arch/powerpc/platforms/powernv/npu-dma.c
index cb2b4f9..677f30a 100644
--- a/arch/powerpc/platforms/powernv/npu-dma.c
+++ b/arch/powerpc/platforms/powernv/npu-dma.c
@@ -762,7 +762,6 @@ struct npu_context *pnv_npu2_init_context(struct pci_dev *gpdev,
u32 nvlink_index;
struct device_node *nvlink_dn;
struct mm_struct *mm = current->mm;
- struct pnv_phb *nphb;
struct npu *npu;
struct npu_context *npu_context;
@@ -772,9 +771,6 @@ struct npu_context *pnv_npu2_init_context(struct pci_dev *gpdev,
*/
struct pci_dev *npdev = pnv_pci_get_npu_dev(gpdev, 0);
- if (!firmware_has_feature(FW_FEATURE_OPAL))
- return ERR_PTR(-ENODEV);
-
if (!npdev)
/* No nvlink associated with this GPU device */
return ERR_PTR(-ENODEV);
@@ -792,22 +788,9 @@ struct npu_context *pnv_npu2_init_context(struct pci_dev *gpdev,
return ERR_PTR(-EINVAL);
}
- nphb = pci_bus_to_host(npdev->bus)->private_data;
npu = npdev_to_npu(npdev);
/*
- * Setup the NPU context table for a particular GPU. These need to be
- * per-GPU as we need the tables to filter ATSDs when there are no
- * active contexts on a particular GPU. It is safe for these to be
- * called concurrently with destroy as the OPAL call takes appropriate
- * locks and refcounts on init/destroy.
- */
- rc = opal_npu_init_context(nphb->opal_id, mm->context.id, flags,
- PCI_DEVID(gpdev->bus->number, gpdev->devfn));
- if (rc < 0)
- return ERR_PTR(-ENOSPC);
-
- /*
* We store the npu pci device so we can more easily get at the
* associated npus.
*/
@@ -817,9 +800,6 @@ struct npu_context *pnv_npu2_init_context(struct pci_dev *gpdev,
if (npu_context->release_cb != cb ||
npu_context->priv != priv) {
spin_unlock(&npu2_devices.context_lock);
- opal_npu_destroy_context(nphb->opal_id, mm->context.id,
- PCI_DEVID(gpdev->bus->number,
- gpdev->devfn));
return ERR_PTR(-EINVAL);
}
@@ -845,9 +825,6 @@ struct npu_context *pnv_npu2_init_context(struct pci_dev *gpdev,
if (rc) {
kfree(npu_context);
- opal_npu_destroy_context(nphb->opal_id, mm->context.id,
- PCI_DEVID(gpdev->bus->number,
- gpdev->devfn));
return ERR_PTR(rc);
}
@@ -900,7 +877,6 @@ void pnv_npu2_destroy_context(struct npu_context *npu_context,
struct pci_dev *gpdev)
{
int removed;
- struct pnv_phb *nphb;
struct npu *npu;
struct pci_dev *npdev = pnv_pci_get_npu_dev(gpdev, 0);
struct device_node *nvlink_dn;
@@ -909,18 +885,12 @@ void pnv_npu2_destroy_context(struct npu_context *npu_context,
if (WARN_ON(!npdev))
return;
- if (!firmware_has_feature(FW_FEATURE_OPAL))
- return;
-
- nphb = pci_bus_to_host(npdev->bus)->private_data;
npu = npdev_to_npu(npdev);
nvlink_dn = of_parse_phandle(npdev->dev.of_node, "ibm,nvlink", 0);
if (WARN_ON(of_property_read_u32(nvlink_dn, "ibm,npu-link-index",
&nvlink_index)))
return;
WRITE_ONCE(npu_context->npdev[npu->index][nvlink_index], NULL);
- opal_npu_destroy_context(nphb->opal_id, npu_context->mm->context.id,
- PCI_DEVID(gpdev->bus->number, gpdev->devfn));
spin_lock(&npu2_devices.context_lock);
removed = kref_put(&npu_context->kref, pnv_npu2_release_context);
spin_unlock(&npu2_devices.context_lock);
@@ -952,9 +922,6 @@ int pnv_npu2_handle_fault(struct npu_context *context, uintptr_t *ea,
/* mmap_sem should be held so the struct_mm must be present */
struct mm_struct *mm = context->mm;
- if (!firmware_has_feature(FW_FEATURE_OPAL))
- return -ENODEV;
-
WARN_ON(!rwsem_is_locked(&mm->mmap_sem));
for (i = 0; i < count; i++) {
@@ -983,15 +950,11 @@ int pnv_npu2_handle_fault(struct npu_context *context, uintptr_t *ea,
}
EXPORT_SYMBOL(pnv_npu2_handle_fault);
-int pnv_npu2_init(struct pnv_phb *phb)
+int pnv_npu2_init(struct pci_controller *hose)
{
unsigned int i;
u64 mmio_atsd;
- struct device_node *dn;
- struct pci_dev *gpdev;
static int npu_index;
- uint64_t rc = 0;
- struct pci_controller *hose = phb->hose;
struct npu *npu;
int ret;
@@ -1006,18 +969,6 @@ int pnv_npu2_init(struct pnv_phb *phb)
}
npu->nmmu_flush = of_property_read_bool(hose->dn, "ibm,nmmu-flush");
- for_each_child_of_node(phb->hose->dn, dn) {
- gpdev = pnv_pci_get_gpu_dev(get_pci_dev(dn));
- if (gpdev) {
- rc = opal_npu_map_lpar(phb->opal_id,
- PCI_DEVID(gpdev->bus->number, gpdev->devfn),
- 0, 0);
- if (rc)
- dev_err(&gpdev->dev,
- "Error %lld mapping device to LPAR\n",
- rc);
- }
- }
for (i = 0; !of_property_read_u64_index(hose->dn, "ibm,mmio-atsd",
i, &mmio_atsd); i++)
@@ -1047,3 +998,52 @@ int pnv_npu2_init(struct pnv_phb *phb)
return ret;
}
+
+void pnv_npu2_map_lpar_phb(struct pnv_phb *nphb, unsigned long msr)
+{
+ struct pci_dev *gpdev;
+ struct device_node *dn;
+ int ret;
+ struct pci_controller *hose = nphb->hose;
+
+ for_each_child_of_node(hose->dn, dn) {
+ gpdev = pnv_pci_get_gpu_dev(get_pci_dev(dn));
+ if (!gpdev)
+ continue;
+
+ dev_dbg(&gpdev->dev, "Map LPAR opalid=%llu\n", nphb->opal_id);
+ ret = opal_npu_map_lpar(nphb->opal_id,
+ PCI_DEVID(gpdev->bus->number, gpdev->devfn),
+ 0, 0);
+ if (!ret)
+ continue;
+ dev_err(&gpdev->dev, "Error %d mapping device to LPAR\n", ret);
+ }
+
+ /*
+ * It seems that touching NPU2_XTS_BDF_MAP in the way
+ * the opal_npu_map_lpar() does somehow affects the result of
+ * what opal_npu_init_context() does so let's put the latter in
+ * a separate loop.
+ */
+ for_each_child_of_node(hose->dn, dn) {
+ gpdev = pnv_pci_get_gpu_dev(get_pci_dev(dn));
+ if (!gpdev)
+ continue;
+
+ /*
+ * Setup the NPU context table for a particular GPU. These need
+ * to be per-GPU as we need the tables to filter ATSDs when
+ * there are no active contexts on a particular GPU. It is safe
+ * for these to be called concurrently with destroy as the OPAL
+ * call takes appropriate locks and refcounts on init/destroy.
+ */
+ dev_dbg(&gpdev->dev, "init context opalid=%llu\n",
+ nphb->opal_id);
+ ret = opal_npu_init_context(nphb->opal_id, 0/*__unused*/, msr,
+ PCI_DEVID(gpdev->bus->number, gpdev->devfn));
+ if (!ret)
+ continue;
+ dev_err(&gpdev->dev, "Failed to init context: %d\n", ret);
+ }
+}
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 0cc81c0..56a1398 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1287,8 +1287,11 @@ static void pnv_pci_ioda_setup_PEs(void)
/* PE#0 is needed for error reporting */
pnv_ioda_reserve_pe(phb, 0);
pnv_ioda_setup_npu_PEs(hose->bus);
- if (phb->model == PNV_PHB_MODEL_NPU2)
- pnv_npu2_init(phb);
+ if (phb->model == PNV_PHB_MODEL_NPU2) {
+ pnv_npu2_init(hose);
+ pnv_npu2_map_lpar_phb(phb,
+ MSR_DR | MSR_PR | MSR_HV);
+ }
}
if (phb->type == PNV_PHB_NPU_OCAPI) {
bus = hose->bus;
--
2.11.0
^ permalink raw reply related
* Re: [PATCH kernel 0/5] powerpc/powernv/npu: Reworks for NVIDIA V100 + P9 passthrough (part 2)
From: Alexey Kardashevskiy @ 2018-10-15 9:34 UTC (permalink / raw)
To: linuxppc-dev
Cc: Alistair Popple, Frederic Barrat, Alex Williamson, kvm-ppc,
Reza Arbab, David Gibson
In-Reply-To: <20181015093301.1007-1-aik@ozlabs.ru>
Strangely Reza is missed by the get_maintainers.pl, adding in cc: :-/
On 15/10/2018 20:32, Alexey Kardashevskiy wrote:
> This is a second set of patches required for passing through NVIDIA V100
> with coherent memory. The full patchset is here:
> https://github.com/aik/linux/tree/nv2
> The matching QEMU is here:
> https://github.com/aik/qemu/tree/nv2
>
>
> This patchset reworks NPU DMA code to be used with VFIO. The exported symbols
> are called from the NVIDIA driver so they must be able to work in both
> powernv and pseries platforms.
>
> This depends on skiboot's "npu2: Clear XTS_BDF_MAP when destroying context for next init_context"
> posted earlier today.
>
> Please comment. Thanks.
>
>
>
> Alexey Kardashevskiy (5):
> powerpc/powernv/npu: Add helper to access struct npu for NPU device
> powerpc/powernv/npu: Collect all static symbols under one struct
> powerpc/powernv: Detach npu struct from pnv_phb
> powerpc/powernv/npu: Factor out OPAL calls from context manipulation
> powerpc/powernv/npu: Add helper to map GPU to LPAR
>
> arch/powerpc/include/asm/pci.h | 4 +
> arch/powerpc/platforms/powernv/pci.h | 18 +-
> arch/powerpc/platforms/powernv/npu-dma.c | 298 ++++++++++++++++++++----------
> arch/powerpc/platforms/powernv/pci-ioda.c | 9 +-
> 4 files changed, 212 insertions(+), 117 deletions(-)
>
--
Alexey
^ permalink raw reply
* Re: [PATCH] powerpc: Fix stackprotector detection for non-glibc toolchains
From: Michael Ellerman @ 2018-10-15 9:36 UTC (permalink / raw)
To: Segher Boessenkool; +Cc: linuxppc-dev
In-Reply-To: <20181013154829.GO29268@gate.crashing.org>
Segher Boessenkool <segher@kernel.crashing.org> writes:
> On Sat, Oct 13, 2018 at 10:55:01PM +1100, Michael Ellerman wrote:
>> So it's only my cross compilers that don't work.
>>
>> The kernel.org ones are:
>> Configured with: /home/arnd/git/gcc/configure --target=powerpc64-linux
>> --enable-targets=all
>> --prefix=/home/arnd/cross/x86_64/gcc-8.1.0-nolibc/powerpc64-linux
>> --enable-languages=c --without-headers --disable-bootstrap
>> --disable-nls --disable-threads --disable-shared --disable-libmudflap
>> --disable-libssp --disable-libgomp --disable-decimal-float
>> --disable-libquadmath --disable-libatomic --disable-libcc1
>> --disable-libmpx --enable-checking=release
>>
>> Whereas mine is:
>> Configured with: ../../src/gcc/configure
>> --prefix=/home/kerkins/workspace/gcc-build/gcc/gcc-8-branch/target/ppc/build/install/powerpc-linux
>> --disable-multilib --disable-bootstrap --enable-languages=c
>> --with-pkgversion='Custom 2c79ff811dfcee1c' --target=powerpc-linux
>> --enable-targets=all
>>
>>
>> So I wonder if something in there is making the difference?
>
> You have --disable-libssp on the buildall-built compiler, which makes GCC
> assume your libc has the SSP support routines, which gives you these default
> offsets (which are what they are on glibc). Never mind that you explicitly
> do not have a libc ;-)
OK thanks, things just get weirder and weirder :)
>> I guess I'll just rewrite the change log to say "some toolchains".
>
> Or "most".
As it happens I forgot to update the change log anyway :/
Oh well.
cheers
^ permalink raw reply
* [PATCH kernel 0/3] vfio/spapr_tce: Reworks for NVIDIA V100 + P9 passthrough (part 3)
From: Alexey Kardashevskiy @ 2018-10-15 9:42 UTC (permalink / raw)
To: linuxppc-dev
Cc: kvm, Alexey Kardashevskiy, Alistair Popple, kvm-ppc,
Alex Williamson, Reza Arbab, David Gibson
This is a third set of patches required for passing through NVIDIA V100
with coherent memory. The full patchset is here:
https://github.com/aik/linux/tree/nv2
The matching QEMU is here:
https://github.com/aik/qemu/tree/nv2
This implements a subdriver for NVIDIA V100 GPU with coherent memory and
NPU/ATS support available in the POWER9 CPU.
This does not address the interconnected GPUs, this will be posted as
a separate patchset in a few days.
This depends on:
[PATCH kernel 0/4] vfio/spapr_tce: Reworks for NVIDIA V100 + P9 passthrough (part 1)
[PATCH kernel 0/5] powerpc/powernv/npu: Reworks for NVIDIA V100 + P9 passthrough (part 2)
[PATCH kernel RFC 3/3] powerpc/pseries/iommu: Use memory@ nodes in max RAM address calculation
Please comment. Thanks.
Alexey Kardashevskiy (3):
vfio_pci: Allow mapping extra regions
vfio_pci: Allow regions to add own capabilities
vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver
drivers/vfio/pci/Makefile | 1 +
drivers/vfio/pci/vfio_pci_private.h | 8 +
include/uapi/linux/vfio.h | 18 ++
drivers/vfio/pci/vfio_pci.c | 51 ++++-
drivers/vfio/pci/vfio_pci_nvlink2.c | 409 ++++++++++++++++++++++++++++++++++++
drivers/vfio/pci/Kconfig | 4 +
6 files changed, 489 insertions(+), 2 deletions(-)
create mode 100644 drivers/vfio/pci/vfio_pci_nvlink2.c
--
2.11.0
^ permalink raw reply
* [PATCH kernel 2/3] vfio_pci: Allow regions to add own capabilities
From: Alexey Kardashevskiy @ 2018-10-15 9:42 UTC (permalink / raw)
To: linuxppc-dev
Cc: kvm, Alexey Kardashevskiy, Alistair Popple, kvm-ppc,
Alex Williamson, Reza Arbab, David Gibson
In-Reply-To: <20181015094233.1324-1-aik@ozlabs.ru>
VFIO regions already support region capabilities with a limited set of
fields. However the subdriver might have to report to the userspace
additional bits.
This adds an add_capability() hook to vfio_pci_regops.
This is aiming Witherspoon POWER9 machines which have multiple
interconnected NVIDIA V100 GPUs with coherent RAM; each GPU's RAM
is mapped to a system bus and to each of GPU internal system bus and
the GPUs use this for DMA routing as DMA trafic can go via any
of many NVLink2 (GPU-GPU or GPU-CPU) or even stay local within a GPU.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
This is based on top of "vfio_pci: Allow mapping extra regions"
---
drivers/vfio/pci/vfio_pci_private.h | 3 +++
drivers/vfio/pci/vfio_pci.c | 6 ++++++
2 files changed, 9 insertions(+)
diff --git a/drivers/vfio/pci/vfio_pci_private.h b/drivers/vfio/pci/vfio_pci_private.h
index 86aab05..93c1738 100644
--- a/drivers/vfio/pci/vfio_pci_private.h
+++ b/drivers/vfio/pci/vfio_pci_private.h
@@ -62,6 +62,9 @@ struct vfio_pci_regops {
int (*mmap)(struct vfio_pci_device *vdev,
struct vfio_pci_region *region,
struct vm_area_struct *vma);
+ int (*add_capability)(struct vfio_pci_device *vdev,
+ struct vfio_pci_region *region,
+ struct vfio_info_cap *caps);
};
struct vfio_pci_region {
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 7923314..4a3b93e 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -759,6 +759,12 @@ static long vfio_pci_ioctl(void *device_data,
if (ret)
return ret;
+ if (vdev->region[i].ops->add_capability) {
+ ret = vdev->region[i].ops->add_capability(vdev,
+ &vdev->region[i], &caps);
+ if (ret)
+ return ret;
+ }
}
}
--
2.11.0
^ permalink raw reply related
* [PATCH kernel 1/3] vfio_pci: Allow mapping extra regions
From: Alexey Kardashevskiy @ 2018-10-15 9:42 UTC (permalink / raw)
To: linuxppc-dev
Cc: kvm, Alexey Kardashevskiy, Alistair Popple, kvm-ppc,
Alex Williamson, Reza Arbab, David Gibson
In-Reply-To: <20181015094233.1324-1-aik@ozlabs.ru>
So far we only allowed mapping of MMIO BARs to the userspace. However
there there are GPUs with on-board coherent RAM accessible via side
channels which we also want to map to the userspace. The first client
for this is NVIDIA V100 GPU with NVLink2 direct links to a POWER9
NPU-enabled CPU; such GPUs have 16GB RAM which is coherently mapped
to the system address space, we are going to export these as an extra
PCI region.
We already support extra PCI regions and this adds support for mapping
them to the userspace.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
Changes:
v2:
* reverted one of mistakenly removed error checks
---
drivers/vfio/pci/vfio_pci_private.h | 3 +++
drivers/vfio/pci/vfio_pci.c | 8 ++++++++
2 files changed, 11 insertions(+)
diff --git a/drivers/vfio/pci/vfio_pci_private.h b/drivers/vfio/pci/vfio_pci_private.h
index cde3b5d..86aab05 100644
--- a/drivers/vfio/pci/vfio_pci_private.h
+++ b/drivers/vfio/pci/vfio_pci_private.h
@@ -59,6 +59,9 @@ struct vfio_pci_regops {
size_t count, loff_t *ppos, bool iswrite);
void (*release)(struct vfio_pci_device *vdev,
struct vfio_pci_region *region);
+ int (*mmap)(struct vfio_pci_device *vdev,
+ struct vfio_pci_region *region,
+ struct vm_area_struct *vma);
};
struct vfio_pci_region {
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index d9af440..7923314 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -1126,6 +1126,14 @@ static int vfio_pci_mmap(void *device_data, struct vm_area_struct *vma)
return -EINVAL;
if ((vma->vm_flags & VM_SHARED) == 0)
return -EINVAL;
+ if (index >= VFIO_PCI_NUM_REGIONS) {
+ int regnum = index - VFIO_PCI_NUM_REGIONS;
+ struct vfio_pci_region *region = vdev->region + regnum;
+
+ if (region && region->ops && region->ops->mmap)
+ return region->ops->mmap(vdev, region, vma);
+ return -EINVAL;
+ }
if (index >= VFIO_PCI_ROM_REGION_INDEX)
return -EINVAL;
if (!vdev->bar_mmap_supported[index])
--
2.11.0
^ permalink raw reply related
* [PATCH kernel 3/3] vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] [10de:1db1] subdriver
From: Alexey Kardashevskiy @ 2018-10-15 9:42 UTC (permalink / raw)
To: linuxppc-dev
Cc: kvm, Alexey Kardashevskiy, Alistair Popple, kvm-ppc,
Alex Williamson, Reza Arbab, David Gibson
In-Reply-To: <20181015094233.1324-1-aik@ozlabs.ru>
POWER9 Witherspoon machines come with 4 or 6 V100 GPUs which are not
pluggable PCIe devices but implement PCIe links for config space and MMIO.
In addition to that the GPUs are interconnected to each other and also
have direct links to the P9 CPU. The links are NVLink2 and provide direct
access to the system RAM for GPUs via NPU (an NVLink2 "proxy" on P9 chip).
These systems also support ATS (address translation services) which is
a part of the NVLink2 prototol. Such GPUs also share on-board RAM
(16GB in tested config) to the system via the same NVLink2 so a CPU has
cache-coherent access to a GPU RAM.
This exports GPU RAM to the userspace as a new PCI region. This
preregisters the new memory as device memory as it might be used for DMA.
This inserts pfns from the fault handler as the GPU memory is not onlined
until the NVIDIA driver is loaded and trained the links so doing this
earlier produces low level errors which we fence in the firmware so
it does not hurt the host system but still better to avoid.
This exports ATSD (Address Translation Shootdown) register of NPU which
allows the guest to invalidate TLB. The register conviniently occupies
a single 64k page. Since NPU maps the GPU memory, it has a "tgt" property
(which is an abbreviated host system bus address). This exports the "tgt"
as a capability so the guest can program it into the GPU so the GPU can
know how to route DMA trafic.
For ATS to work, the nest MMU (an NVIDIA block in a P9 CPU) needs to
know LPID (a logical partition ID or a KVM guest hardware ID in other
words) and PID (a memory context ID of an userspace process, not to be
confused with a linux pid). This assigns a GPU to LPID in the NPU and
this is why this adds a listener for KVM on an IOMMU group. A PID comes
via NVLink from a GPU and NPU uses a PID wildcard to pass it through.
This requires coherent memory and ATSD to be available on the host as
the GPU vendor only supports configurations with both features enabled
and other configurations are known not to work. Because of this and
because of the ways the features are advertised to the host system
(which is a device tree with very platform specific properties),
this requires enabled POWERNV platform.
This hardcodes the NVLink2 support for specific vendor and device IDs
as there is no reliable way of knowing about coherent memory and ATS
support. The GPU has an unique vendor PCIe capability 0x23 but it was
confirmed that it does not provide required information (and it is still
undisclosed what it actually does).
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
drivers/vfio/pci/Makefile | 1 +
drivers/vfio/pci/vfio_pci_private.h | 2 +
include/uapi/linux/vfio.h | 18 ++
drivers/vfio/pci/vfio_pci.c | 37 +++-
drivers/vfio/pci/vfio_pci_nvlink2.c | 409 ++++++++++++++++++++++++++++++++++++
drivers/vfio/pci/Kconfig | 4 +
6 files changed, 469 insertions(+), 2 deletions(-)
create mode 100644 drivers/vfio/pci/vfio_pci_nvlink2.c
diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
index 76d8ec0..9662c06 100644
--- a/drivers/vfio/pci/Makefile
+++ b/drivers/vfio/pci/Makefile
@@ -1,5 +1,6 @@
vfio-pci-y := vfio_pci.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o
+vfio-pci-$(CONFIG_VFIO_PCI_NVLINK2) += vfio_pci_nvlink2.o
obj-$(CONFIG_VFIO_PCI) += vfio-pci.o
diff --git a/drivers/vfio/pci/vfio_pci_private.h b/drivers/vfio/pci/vfio_pci_private.h
index 93c1738..7639241 100644
--- a/drivers/vfio/pci/vfio_pci_private.h
+++ b/drivers/vfio/pci/vfio_pci_private.h
@@ -163,4 +163,6 @@ static inline int vfio_pci_igd_init(struct vfio_pci_device *vdev)
return -ENODEV;
}
#endif
+extern int vfio_pci_nvdia_v100_nvlink2_init(struct vfio_pci_device *vdev);
+extern int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev);
#endif /* VFIO_PCI_PRIVATE_H */
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index f378b98..9e9a8d3 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -303,6 +303,12 @@ struct vfio_region_info_cap_type {
#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)
#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3)
+/* NVIDIA GPU NVlink2 RAM */
+#define VFIO_REGION_SUBTYPE_NVIDIA_NVLINK2_RAM (1)
+
+/* IBM NPU NVlink2 ATSD */
+#define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD (1)
+
/*
* The MSIX mappable capability informs that MSIX data of a BAR can be mmapped
* which allows direct access to non-MSIX registers which happened to be within
@@ -313,6 +319,18 @@ struct vfio_region_info_cap_type {
*/
#define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3
+/*
+ * Capability with compressed real address (aka SSA - small system address)
+ * where GPU RAM is mapped on a system bus. Used by a GPU for DMA routing.
+ */
+#define VFIO_REGION_INFO_CAP_NPU2 4
+
+struct vfio_region_info_cap_npu2 {
+ struct vfio_info_cap_header header;
+ __u64 tgt;
+ /* size is defined in VFIO_REGION_SUBTYPE_NVIDIA_NVLINK2_RAM */
+};
+
/**
* VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9,
* struct vfio_irq_info)
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 4a3b93e..e9afd43 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -224,6 +224,16 @@ static bool vfio_pci_nointx(struct pci_dev *pdev)
return false;
}
+int __weak vfio_pci_nvdia_v100_nvlink2_init(struct vfio_pci_device *vdev)
+{
+ return -ENODEV;
+}
+
+int __weak vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev)
+{
+ return -ENODEV;
+}
+
static int vfio_pci_enable(struct vfio_pci_device *vdev)
{
struct pci_dev *pdev = vdev->pdev;
@@ -302,14 +312,37 @@ static int vfio_pci_enable(struct vfio_pci_device *vdev)
if (ret) {
dev_warn(&vdev->pdev->dev,
"Failed to setup Intel IGD regions\n");
- vfio_pci_disable(vdev);
- return ret;
+ goto disable_exit;
+ }
+ }
+
+ if (pdev->vendor == PCI_VENDOR_ID_NVIDIA &&
+ pdev->device == 0x1db1) {
+ ret = vfio_pci_nvdia_v100_nvlink2_init(vdev);
+ if (ret) {
+ dev_warn(&vdev->pdev->dev,
+ "Failed to setup NVIDIA NV2 RAM region\n");
+ goto disable_exit;
+ }
+ }
+
+ if (pdev->vendor == PCI_VENDOR_ID_IBM &&
+ pdev->device == 0x04ea) {
+ ret = vfio_pci_ibm_npu2_init(vdev);
+ if (ret) {
+ dev_warn(&vdev->pdev->dev,
+ "Failed to setup NVIDIA NV2 ATSD region\n");
+ goto disable_exit;
}
}
vfio_pci_probe_mmaps(vdev);
return 0;
+
+disable_exit:
+ vfio_pci_disable(vdev);
+ return ret;
}
static void vfio_pci_disable(struct vfio_pci_device *vdev)
diff --git a/drivers/vfio/pci/vfio_pci_nvlink2.c b/drivers/vfio/pci/vfio_pci_nvlink2.c
new file mode 100644
index 0000000..c9d2b55
--- /dev/null
+++ b/drivers/vfio/pci/vfio_pci_nvlink2.c
@@ -0,0 +1,409 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * VFIO PCI NVIDIA Whitherspoon GPU support a.k.a. NVLink2.
+ *
+ * Copyright (C) 2018 IBM Corp. All rights reserved.
+ * Author: Alexey Kardashevskiy <aik@ozlabs.ru>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * Register an on-GPU RAM region for cacheable access.
+ *
+ * Derived from original vfio_pci_igd.c:
+ * Copyright (C) 2016 Red Hat, Inc. All rights reserved.
+ * Author: Alex Williamson <alex.williamson@redhat.com>
+ */
+
+#include <linux/io.h>
+#include <linux/pci.h>
+#include <linux/uaccess.h>
+#include <linux/vfio.h>
+#include <linux/sched/mm.h>
+#include <linux/mmu_context.h>
+#include <asm/kvm_ppc.h>
+#include "vfio_pci_private.h"
+
+struct vfio_pci_nvgpu_data {
+ unsigned long gpu_hpa;
+ unsigned long useraddr;
+ unsigned long size;
+ void *base;
+ struct mm_struct *mm;
+ struct mm_iommu_table_group_mem_t *mem;
+ struct pci_dev *gpdev;
+ struct notifier_block group_notifier;
+};
+
+static size_t vfio_pci_nvgpu_rw(struct vfio_pci_device *vdev,
+ char __user *buf, size_t count, loff_t *ppos, bool iswrite)
+{
+ unsigned int i = VFIO_PCI_OFFSET_TO_INDEX(*ppos) - VFIO_PCI_NUM_REGIONS;
+ struct vfio_pci_nvgpu_data *data = vdev->region[i].data;
+ loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+
+ if (pos >= vdev->region[i].size)
+ return -EINVAL;
+
+ count = min(count, (size_t)(vdev->region[i].size - pos));
+
+ if (iswrite) {
+ if (copy_from_user(data->base + pos, buf, count))
+ return -EFAULT;
+ } else {
+ if (copy_to_user(buf, data->base + pos, count))
+ return -EFAULT;
+ }
+ *ppos += count;
+
+ return count;
+}
+
+static void vfio_pci_nvgpu_release(struct vfio_pci_device *vdev,
+ struct vfio_pci_region *region)
+{
+ struct vfio_pci_nvgpu_data *data = region->data;
+ long ret;
+ struct pci_controller *hose;
+ struct pci_dev *npdev;
+
+ /* If there were any mappings at all... */
+ if (data->mm) {
+ ret = mm_iommu_put(data->mm, data->mem);
+ WARN_ON(ret);
+
+ mmdrop(data->mm);
+ }
+
+ vfio_unregister_notifier(&data->gpdev->dev, VFIO_GROUP_NOTIFY,
+ &data->group_notifier);
+
+ npdev = pnv_pci_get_npu_dev(data->gpdev, 0);
+ hose = pci_bus_to_host(npdev->bus);
+
+ pnv_npu2_map_lpar_dev(hose, data->gpdev, 0, MSR_DR | MSR_PR | MSR_HV);
+
+ memunmap(data->base);
+ kfree(data);
+}
+
+static int vfio_pci_nvgpu_mmap_fault(struct vm_fault *vmf)
+{
+ int ret;
+ struct vm_area_struct *vma = vmf->vma;
+ struct vfio_pci_region *region = vma->vm_private_data;
+ struct vfio_pci_nvgpu_data *data = region->data;
+ unsigned long vmf_off = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
+ unsigned long nv2pg = data->gpu_hpa >> PAGE_SHIFT;
+ unsigned long vm_pgoff = vma->vm_pgoff &
+ ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+ unsigned long pfn = nv2pg + vm_pgoff + vmf_off;
+
+ ret = vm_insert_pfn(vma, vmf->address, pfn);
+ pr_debug("NVLink2: vmf=%lx hpa=%lx ret=%d\n",
+ vmf->address, pfn << PAGE_SHIFT, ret);
+ if (ret)
+ return VM_FAULT_SIGSEGV;
+
+ return VM_FAULT_NOPAGE;
+}
+
+static const struct vm_operations_struct vfio_pci_nvgpu_mmap_vmops = {
+ .fault = vfio_pci_nvgpu_mmap_fault,
+};
+
+static int vfio_pci_nvgpu_mmap(struct vfio_pci_device *vdev,
+ struct vfio_pci_region *region, struct vm_area_struct *vma)
+{
+ long ret;
+ struct vfio_pci_nvgpu_data *data = region->data;
+
+ if (data->useraddr)
+ return -EPERM;
+
+ if (vma->vm_end - vma->vm_start > data->size)
+ return -EINVAL;
+
+ vma->vm_private_data = region;
+ vma->vm_flags |= VM_PFNMAP;
+ vma->vm_ops = &vfio_pci_nvgpu_mmap_vmops;
+
+ /*
+ * Calling mm_iommu_newdev() here once as the region is not
+ * registered yet and therefore right initialization will happen now.
+ * Other places will use mm_iommu_find() which returns
+ * registered @mem and does not go gup().
+ */
+ data->useraddr = vma->vm_start;
+ data->mm = current->mm;
+
+ atomic_inc(&data->mm->mm_count);
+ ret = mm_iommu_newdev(data->mm, data->useraddr,
+ (vma->vm_end - vma->vm_start) >> PAGE_SHIFT,
+ data->gpu_hpa, &data->mem);
+
+ pr_debug("VFIO NVLINK2 mmap: useraddr=%lx hpa=%lx size=%lx ret=%ld\n",
+ data->useraddr, data->gpu_hpa,
+ vma->vm_end - vma->vm_start, ret);
+
+ return ret;
+}
+
+static const struct vfio_pci_regops vfio_pci_nvgpu_regops = {
+ .rw = vfio_pci_nvgpu_rw,
+ .release = vfio_pci_nvgpu_release,
+ .mmap = vfio_pci_nvgpu_mmap,
+};
+
+static int vfio_pci_nvgpu_group_notifier(struct notifier_block *nb,
+ unsigned long action, void *opaque)
+{
+ struct kvm *kvm = opaque;
+ struct vfio_pci_nvgpu_data *data = container_of(nb,
+ struct vfio_pci_nvgpu_data,
+ group_notifier);
+
+ if (action == VFIO_GROUP_NOTIFY_SET_KVM) {
+ struct pci_controller *hose;
+ struct pci_dev *npdev;
+ struct pnv_phb *nphb;
+
+ npdev = pnv_pci_get_npu_dev(data->gpdev, 0);
+ hose = pci_bus_to_host(npdev->bus);
+ nphb = hose->private_data;
+
+ if (!kvm) {
+ if (pnv_npu2_map_lpar_dev(hose, data->gpdev, 0,
+ MSR_DR | MSR_PR | MSR_HV))
+ return NOTIFY_BAD;
+ } else {
+ if (pnv_npu2_map_lpar_dev(hose, data->gpdev,
+ kvm->arch.lpid, MSR_DR | MSR_PR))
+ return NOTIFY_BAD;
+ }
+ }
+
+ return NOTIFY_OK;
+}
+
+int vfio_pci_nvdia_v100_nvlink2_init(struct vfio_pci_device *vdev)
+{
+ int ret;
+ u64 reg[2];
+ struct device_node *npu_node, *mem_node;
+ struct pci_dev *npu_dev;
+ struct vfio_pci_nvgpu_data *data;
+ uint32_t mem_phandle = 0;
+ unsigned long events = VFIO_GROUP_NOTIFY_SET_KVM;
+
+ npu_dev = pnv_pci_get_npu_dev(vdev->pdev, 0);
+ if (!npu_dev)
+ return -EINVAL;
+
+ npu_node = pci_device_to_OF_node(npu_dev);
+ if (!npu_node)
+ return -EINVAL;
+
+ if (of_property_read_u32(npu_node, "memory-region", &mem_phandle))
+ return -EINVAL;
+
+ mem_node = of_find_node_by_phandle(mem_phandle);
+ if (!mem_node)
+ return -EINVAL;
+
+ if (of_property_read_variable_u64_array(mem_node, "reg", reg,
+ ARRAY_SIZE(reg), ARRAY_SIZE(reg)) !=
+ ARRAY_SIZE(reg))
+ return -EINVAL;
+
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ data->gpu_hpa = reg[0];
+ data->size = reg[1];
+ data->base = memremap(data->gpu_hpa, data->size, MEMREMAP_WB);
+ if (!data->base) {
+ ret = -ENOMEM;
+ goto free_exit;
+ }
+
+ dev_dbg(&vdev->pdev->dev, "%lx..%lx\n", data->gpu_hpa,
+ data->gpu_hpa + data->size - 1);
+
+ data->gpdev = vdev->pdev;
+ data->group_notifier.notifier_call = vfio_pci_nvgpu_group_notifier;
+
+ ret = vfio_register_notifier(&data->gpdev->dev, VFIO_GROUP_NOTIFY,
+ &events, &data->group_notifier);
+ if (ret)
+ goto free_exit;
+
+ ret = vfio_pci_register_dev_region(vdev,
+ PCI_VENDOR_ID_NVIDIA | VFIO_REGION_TYPE_PCI_VENDOR_TYPE,
+ VFIO_REGION_SUBTYPE_NVIDIA_NVLINK2_RAM,
+ &vfio_pci_nvgpu_regops, data->size,
+ VFIO_REGION_INFO_FLAG_READ, data);
+ if (ret)
+ goto unreg_exit;
+
+ return 0;
+unreg_exit:
+ vfio_unregister_notifier(&data->gpdev->dev, VFIO_GROUP_NOTIFY,
+ &data->group_notifier);
+free_exit:
+ kfree(data);
+
+ return ret;
+}
+
+/*
+ * IBM NPU2 bridge
+ */
+struct vfio_pci_npu2_data {
+ void *base;
+ unsigned long mmio_atsd;
+ unsigned long gpu_tgt;
+};
+
+static size_t vfio_pci_npu2_rw(struct vfio_pci_device *vdev,
+ char __user *buf, size_t count, loff_t *ppos, bool iswrite)
+{
+ unsigned int i = VFIO_PCI_OFFSET_TO_INDEX(*ppos) - VFIO_PCI_NUM_REGIONS;
+ struct vfio_pci_npu2_data *data = vdev->region[i].data;
+ loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
+
+ if (pos >= vdev->region[i].size)
+ return -EINVAL;
+
+ count = min(count, (size_t)(vdev->region[i].size - pos));
+
+ if (iswrite) {
+ if (copy_from_user(data->base + pos, buf, count))
+ return -EFAULT;
+ } else {
+ if (copy_to_user(buf, data->base + pos, count))
+ return -EFAULT;
+ }
+ *ppos += count;
+
+ return count;
+}
+
+static int vfio_pci_npu2_mmap(struct vfio_pci_device *vdev,
+ struct vfio_pci_region *region, struct vm_area_struct *vma)
+{
+ int ret;
+ struct vfio_pci_npu2_data *data = region->data;
+ unsigned long req_len = vma->vm_end - vma->vm_start;
+
+ if (req_len != PAGE_SIZE)
+ return -EINVAL;
+
+ vma->vm_flags |= VM_PFNMAP;
+ vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+ ret = remap_pfn_range(vma, vma->vm_start, data->mmio_atsd >> PAGE_SHIFT,
+ req_len, vma->vm_page_prot);
+ pr_debug("VFIO NPU2 mmap: %lx %lx size=%lx ret=%d\n",
+ vma->vm_start, data->mmio_atsd,
+ vma->vm_end - vma->vm_start, ret);
+
+ return ret;
+}
+
+static void vfio_pci_npu2_release(struct vfio_pci_device *vdev,
+ struct vfio_pci_region *region)
+{
+ struct vfio_pci_npu2_data *data = region->data;
+
+ memunmap(data->base);
+ kfree(data);
+}
+
+static int vfio_pci_npu2_add_capability(struct vfio_pci_device *vdev,
+ struct vfio_pci_region *region, struct vfio_info_cap *caps)
+{
+ struct vfio_pci_npu2_data *data = region->data;
+ struct vfio_region_info_cap_npu2 cap;
+
+ cap.header.id = VFIO_REGION_INFO_CAP_NPU2;
+ cap.header.version = 1;
+ cap.tgt = data->gpu_tgt;
+
+ return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
+}
+
+static const struct vfio_pci_regops vfio_pci_npu2_regops = {
+ .rw = vfio_pci_npu2_rw,
+ .mmap = vfio_pci_npu2_mmap,
+ .release = vfio_pci_npu2_release,
+ .add_capability = vfio_pci_npu2_add_capability,
+};
+
+int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev)
+{
+ int ret;
+ struct vfio_pci_npu2_data *data;
+ struct device_node *nvlink_dn;
+ u32 nvlink_index = 0;
+ struct pci_dev *npdev = vdev->pdev;
+ struct device_node *npu_node = pci_device_to_OF_node(npdev);
+ struct pci_controller *hose = pci_bus_to_host(npdev->bus);
+ u64 mmio_atsd = 0;
+ u64 tgt = 0;
+
+ /*
+ * NPU2 normally has 8 ATSD registers (for concurrency) and 6 links
+ * so we can allocate one register per link.
+ * Since skiboot only exposes one (a bug), use this as a fallback
+ * which is safe as we do not split GPUs attached to the same NPU.
+ */
+ nvlink_dn = of_parse_phandle(npdev->dev.of_node, "ibm,nvlink", 0);
+ if (WARN_ON(of_property_read_u32(nvlink_dn, "ibm,npu-link-index",
+ &nvlink_index)))
+ return -ENODEV;
+
+ if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", nvlink_index,
+ &mmio_atsd)) {
+ if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", 0,
+ &mmio_atsd)) {
+ dev_warn(&vdev->pdev->dev, "No ATSD found\n");
+ return -EFAULT;
+ }
+ dev_warn(&vdev->pdev->dev, "Fallback to ATSD#0\n");
+ }
+
+ if (of_property_read_u64(npu_node, "ibm,device-tgt-addr", &tgt)) {
+ dev_warn(&vdev->pdev->dev, "No ibm,device-tgt-addr found\n");
+ return -EFAULT;
+ }
+
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ data->mmio_atsd = mmio_atsd;
+ data->gpu_tgt = tgt;
+ data->base = memremap(data->mmio_atsd, SZ_64K, MEMREMAP_WT);
+ if (!data->base) {
+ ret = -ENOMEM;
+ goto free_exit;
+ }
+
+ ret = vfio_pci_register_dev_region(vdev,
+ PCI_VENDOR_ID_IBM | VFIO_REGION_TYPE_PCI_VENDOR_TYPE,
+ VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD,
+ &vfio_pci_npu2_regops, PAGE_SIZE,
+ VFIO_REGION_INFO_FLAG_READ, data);
+ if (ret)
+ goto free_exit;
+
+ return 0;
+
+free_exit:
+ kfree(data);
+
+ return ret;
+}
diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
index 42dc1d3..1a58979 100644
--- a/drivers/vfio/pci/Kconfig
+++ b/drivers/vfio/pci/Kconfig
@@ -38,3 +38,7 @@ config VFIO_PCI_IGD
and LPC bridge config space.
To enable Intel IGD assignment through vfio-pci, say Y.
+
+config VFIO_PCI_NVLINK2
+ bool "VFIO PCI support for P9 Witherspoon machine with NVIDIA V100 GPUs"
+ depends on VFIO_PCI && PPC_POWERNV
--
2.11.0
^ permalink raw reply related
* [PATCH kernel v2] KVM: PPC: Optimize clearing TCEs for sparse tables
From: Alexey Kardashevskiy @ 2018-10-15 10:08 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Alexey Kardashevskiy, kvm-ppc, David Gibson
The powernv platform maintains 2 TCE tables for VFIO - a hardware TCE
table and a table with userspace addresses. These tables are radix trees,
we allocate indirect levels when they are written to. Since
the memory allocation is problematic in real mode, we have 2 accessors
to the entries:
- for virtual mode: it allocates the memory and it is always expected
to return non-NULL;
- fr real mode: it does not allocate and can return NULL.
Also, DMA windows can span to up to 55 bits of the address space and since
we never have this much RAM, such windows are sparse. However currently
the SPAPR TCE IOMMU driver walks through all TCEs to unpin DMA memory.
Since we maintain a userspace addresses table for VFIO which is a mirror
of the hardware table, we can use it to know which parts of the DMA
window have not been mapped and skip these so does this patch.
The bare metal systems do not have this problem as they use a bypass mode
of a PHB which maps RAM directly.
This helps a lot with sparse DMA windows, reducing the shutdown time from
about 3 minutes per 1 billion TCEs to a few seconds for 32GB sparse guest.
Just skipping the last level seems to be good enough.
As non-allocating accessor is used now in virtual mode as well, rename it
from IOMMU_TABLE_USERSPACE_ENTRY_RM (real mode) to _RO (read only).
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
Changes:
v2:
* instead of adding the level size to @entry, now we align the entry
to the beginning of next chunk of the level, ie
"entry += tbl->it_level_size - 1" became "entry |= tbl->it_level_size - 1"
---
arch/powerpc/include/asm/iommu.h | 2 +-
arch/powerpc/kvm/book3s_64_vio.c | 5 ++---
arch/powerpc/kvm/book3s_64_vio_hv.c | 6 +++---
drivers/vfio/vfio_iommu_spapr_tce.c | 23 +++++++++++++++++++++--
4 files changed, 27 insertions(+), 9 deletions(-)
diff --git a/arch/powerpc/include/asm/iommu.h b/arch/powerpc/include/asm/iommu.h
index 3d4b88c..35db0cb 100644
--- a/arch/powerpc/include/asm/iommu.h
+++ b/arch/powerpc/include/asm/iommu.h
@@ -126,7 +126,7 @@ struct iommu_table {
int it_nid;
};
-#define IOMMU_TABLE_USERSPACE_ENTRY_RM(tbl, entry) \
+#define IOMMU_TABLE_USERSPACE_ENTRY_RO(tbl, entry) \
((tbl)->it_ops->useraddrptr((tbl), (entry), false))
#define IOMMU_TABLE_USERSPACE_ENTRY(tbl, entry) \
((tbl)->it_ops->useraddrptr((tbl), (entry), true))
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index c0c64d1..62a8d03 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -410,11 +410,10 @@ static long kvmppc_tce_iommu_mapped_dec(struct kvm *kvm,
{
struct mm_iommu_table_group_mem_t *mem = NULL;
const unsigned long pgsize = 1ULL << tbl->it_page_shift;
- __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY(tbl, entry);
+ __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY_RO(tbl, entry);
if (!pua)
- /* it_userspace allocation might be delayed */
- return H_TOO_HARD;
+ return H_SUCCESS;
mem = mm_iommu_lookup(kvm->mm, be64_to_cpu(*pua), pgsize);
if (!mem)
diff --git a/arch/powerpc/kvm/book3s_64_vio_hv.c b/arch/powerpc/kvm/book3s_64_vio_hv.c
index ec99363..2206bc7 100644
--- a/arch/powerpc/kvm/book3s_64_vio_hv.c
+++ b/arch/powerpc/kvm/book3s_64_vio_hv.c
@@ -214,7 +214,7 @@ static long iommu_tce_xchg_rm(struct mm_struct *mm, struct iommu_table *tbl,
if (!ret && ((*direction == DMA_FROM_DEVICE) ||
(*direction == DMA_BIDIRECTIONAL))) {
- __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY_RM(tbl, entry);
+ __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY_RO(tbl, entry);
/*
* kvmppc_rm_tce_iommu_do_map() updates the UA cache after
* calling this so we still get here a valid UA.
@@ -240,7 +240,7 @@ static long kvmppc_rm_tce_iommu_mapped_dec(struct kvm *kvm,
{
struct mm_iommu_table_group_mem_t *mem = NULL;
const unsigned long pgsize = 1ULL << tbl->it_page_shift;
- __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY_RM(tbl, entry);
+ __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY_RO(tbl, entry);
if (!pua)
/* it_userspace allocation might be delayed */
@@ -304,7 +304,7 @@ static long kvmppc_rm_tce_iommu_do_map(struct kvm *kvm, struct iommu_table *tbl,
{
long ret;
unsigned long hpa = 0;
- __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY_RM(tbl, entry);
+ __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY_RO(tbl, entry);
struct mm_iommu_table_group_mem_t *mem;
if (!pua)
diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c
index 96721b1..b30926e 100644
--- a/drivers/vfio/vfio_iommu_spapr_tce.c
+++ b/drivers/vfio/vfio_iommu_spapr_tce.c
@@ -444,7 +444,7 @@ static void tce_iommu_unuse_page_v2(struct tce_container *container,
struct mm_iommu_table_group_mem_t *mem = NULL;
int ret;
unsigned long hpa = 0;
- __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY(tbl, entry);
+ __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY_RO(tbl, entry);
if (!pua)
return;
@@ -467,8 +467,27 @@ static int tce_iommu_clear(struct tce_container *container,
unsigned long oldhpa;
long ret;
enum dma_data_direction direction;
+ unsigned long lastentry = entry + pages;
+
+ for ( ; entry < lastentry; ++entry) {
+ if (tbl->it_indirect_levels && tbl->it_userspace) {
+ /*
+ * For multilevel tables, we can take a shortcut here
+ * and skip some TCEs as we know that the userspace
+ * addresses cache is a mirror of the real TCE table
+ * and if it is missing some indirect levels, then
+ * the hardware table does not have them allocated
+ * either and therefore does not require updating.
+ */
+ __be64 *pua = IOMMU_TABLE_USERSPACE_ENTRY_RO(tbl,
+ entry);
+ if (!pua) {
+ /* align to level_size which is power of two */
+ entry |= tbl->it_level_size - 1;
+ continue;
+ }
+ }
- for ( ; pages; --pages, ++entry) {
cond_resched();
direction = DMA_NONE;
--
2.11.0
^ permalink raw reply related
* Re: [PATCH v2 2/2] mm: speed up mremap by 500x on large regions
From: Martin Schwidefsky @ 2018-10-15 8:18 UTC (permalink / raw)
To: Christian Borntraeger
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring,
Jonas Bonn, linux-s390, dancol, Yoshinori Sato, sparclinux,
linux-xtensa, linux-hexagon, Helge Deller,
maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd,
James E.J. Bottomley, kasan-dev, kvmarm, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, Fenghua Yu, Jeff Dike, linux-um, Stefan Kristiansson,
Julia Lawall, linux-m68k, openrisc, Borislav Petkov,
Andy Lutomirski, nios2-dev, kirill, Stafford Horne, Guan Xuetao,
linux-arm-kernel, Chris Zankel, Tony Luck, Richard Weinberger,
linux-parisc, pantin, Max Filippov, linux-kernel, minchan,
Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev,
David S. Miller
In-Reply-To: <6580a62b-69c6-f2e3-767c-bd36b977bea2@de.ibm.com>
On Mon, 15 Oct 2018 09:10:53 +0200
Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> On 10/12/2018 03:37 AM, Joel Fernandes (Google) wrote:
> > Android needs to mremap large regions of memory during memory management
> > related operations. The mremap system call can be really slow if THP is
> > not enabled. The bottleneck is move_page_tables, which is copying each
> > pte at a time, and can be really slow across a large map. Turning on THP
> > may not be a viable option, and is not for us. This patch speeds up the
> > performance for non-THP system by copying at the PMD level when possible.
> >
> > The speed up is three orders of magnitude. On a 1GB mremap, the mremap
> > completion times drops from 160-250 millesconds to 380-400 microseconds.
> >
> > Before:
> > Total mremap time for 1GB data: 242321014 nanoseconds.
> > Total mremap time for 1GB data: 196842467 nanoseconds.
> > Total mremap time for 1GB data: 167051162 nanoseconds.
> >
> > After:
> > Total mremap time for 1GB data: 385781 nanoseconds.
> > Total mremap time for 1GB data: 388959 nanoseconds.
> > Total mremap time for 1GB data: 402813 nanoseconds.
> >
> > Incase THP is enabled, the optimization is skipped. I also flush the
> > tlb every time we do this optimization since I couldn't find a way to
> > determine if the low-level PTEs are dirty. It is seen that the cost of
> > doing so is not much compared the improvement, on both x86-64 and arm64.
> >
> > Cc: minchan@kernel.org
> > Cc: pantin@google.com
> > Cc: hughd@google.com
> > Cc: lokeshgidra@google.com
> > Cc: dancol@google.com
> > Cc: mhocko@kernel.org
> > Cc: kirill@shutemov.name
> > Cc: akpm@linux-foundation.org
> > Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> > ---
> > mm/mremap.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 62 insertions(+)
> >
> > diff --git a/mm/mremap.c b/mm/mremap.c
> > index 9e68a02a52b1..d82c485822ef 100644
> > --- a/mm/mremap.c
> > +++ b/mm/mremap.c
> > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
> > drop_rmap_locks(vma);
> > }
> >
> > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
> > + unsigned long new_addr, unsigned long old_end,
> > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush)
> > +{
> > + spinlock_t *old_ptl, *new_ptl;
> > + struct mm_struct *mm = vma->vm_mm;
> > +
> > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
> > + || old_end - old_addr < PMD_SIZE)
> > + return false;
> > +
> > + /*
> > + * The destination pmd shouldn't be established, free_pgtables()
> > + * should have release it.
> > + */
> > + if (WARN_ON(!pmd_none(*new_pmd)))
> > + return false;
> > +
> > + /*
> > + * We don't have to worry about the ordering of src and dst
> > + * ptlocks because exclusive mmap_sem prevents deadlock.
> > + */
> > + old_ptl = pmd_lock(vma->vm_mm, old_pmd);
> > + if (old_ptl) {
> > + pmd_t pmd;
> > +
> > + new_ptl = pmd_lockptr(mm, new_pmd);
> > + if (new_ptl != old_ptl)
> > + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
> > +
> > + /* Clear the pmd */
> > + pmd = *old_pmd;
> > + pmd_clear(old_pmd);
>
> Adding Martin Schwidefsky.
> Is this mapping maybe still in use on other CPUs? If yes, I think for
> s390 we need to flush here as well (in other word we might need to introduce
> pmd_clear_flush). On s390 you have to use instructions like CRDTE,IPTE or IDTE
> to modify page table entries that are still in use. Otherwise you can get a
> delayed access exception which is - in contrast to page faults - not recoverable.
Just clearing an active pmd would be broken for s390. We need the equivalent
of the ptep_get_and_clear() function for pmds. For s390 this function would
look like this:
static inline pte_t pmdp_get_and_clear(struct mm_struct *mm,
unsigned long addr, pmd_t *pmdp)
{
return pmdp_xchg_lazy(mm, addr, pmdp, __pmd(_SEGMENT_ENTRY_INVALID));
}
Just like pmdp_huge_get_and_clear() in fact.
>
>
>
> > +
> > + VM_BUG_ON(!pmd_none(*new_pmd));
> > +
> > + /* Set the new pmd */
> > + set_pmd_at(mm, new_addr, new_pmd, pmd);
> > + if (new_ptl != old_ptl)
> > + spin_unlock(new_ptl);
> > + spin_unlock(old_ptl);
> > +
> > + *need_flush = true;
> > + return true;
> > + }
> > + return false;
> > +}
> > +
So the idea is to move the pmd entry to the new location, dragging
the whole pte table to a new location with a different address.
I wonder if that is safe in regard to get_user_pages_fast().
> > unsigned long move_page_tables(struct vm_area_struct *vma,
> > unsigned long old_addr, struct vm_area_struct *new_vma,
> > unsigned long new_addr, unsigned long len,
> > @@ -239,7 +287,21 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
> > split_huge_pmd(vma, old_pmd, old_addr);
> > if (pmd_trans_unstable(old_pmd))
> > continue;
> > + } else if (extent == PMD_SIZE) {
> > + bool moved;
> > +
> > + /* See comment in move_ptes() */
> > + if (need_rmap_locks)
> > + take_rmap_locks(vma);
> > + moved = move_normal_pmd(vma, old_addr, new_addr,
> > + old_end, old_pmd, new_pmd,
> > + &need_flush);
> > + if (need_rmap_locks)
> > + drop_rmap_locks(vma);
> > + if (moved)
> > + continue;
> > }
> > +
> > if (pte_alloc(new_vma->vm_mm, new_pmd))
> > break;
> > next = (new_addr + PMD_SIZE) & PMD_MASK;
> >
--
blue skies,
Martin.
"Reality continues to ruin my life." - Calvin.
^ permalink raw reply
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2)
From: Christoph Hellwig @ 2018-10-15 9:42 UTC (permalink / raw)
To: Joel Fernandes (Google)
Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra,
Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm,
lokeshgidra, sparclinux, linux-riscv, kvmarm, Jonas Bonn,
linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon,
Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT,
hughd, James E.J. Bottomley, kasan-dev, elfring, Ingo Molnar,
Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team,
Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um,
Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov,
Andy Lutomirski, Ley Foon Tan, kirill, Stafford Horne,
Guan Xuetao, Chris Zankel, Tony Luck, linux-parisc, pantin,
linux-kernel, Fenghua Yu, minchan, Thomas Gleixner,
Richard Weinberger, anton.ivanov, nios2-dev, akpm, linuxppc-dev,
David S. Miller
In-Reply-To: <20181013013200.206928-3-joel@joelfernandes.org>
On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote:
> Android needs to mremap large regions of memory during memory management
> related operations.
Just curious: why?
> + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
> + || old_end - old_addr < PMD_SIZE)
The || goes on the first line.
> + } else if (extent == PMD_SIZE && IS_ENABLED(CONFIG_HAVE_MOVE_PMD)) {
Overly long line.
^ permalink raw reply
* Re: 32-bit poweroc compile failure in v4.19-rc7-166-g7ec21823634d
From: Michael Ellerman @ 2018-10-15 10:33 UTC (permalink / raw)
To: Meelis Roos, linux-kernel, linuxppc-dev
In-Reply-To: <17d36e6c-f99a-abfb-12cd-fa4b14bc7e09@linux.ee>
Meelis Roos <mroos@linux.ee> writes:
> I tried to test the fix to 32-bit poweroc boot hang but found that current git does not compile on 32bit poweroc at all for me:
That's GCC 8 I think?
I have seen that but couldn't work out what the hell GCC is thinking.
I don't think there's an actual bug, the size of the copy is bounded by
the count parameter, which *doesn't* come from user space, it's hard
coded.
I suspect the logic is just to convoluted for GCC.
We should fix it somehow, but I haven't worked out what's the best option.
cheers
> CC arch/powerpc/kernel/ptrace.o
> In file included from ./include/linux/bitmap.h:9,
> from ./include/linux/cpumask.h:12,
> from ./include/linux/rcupdate.h:44,
> from ./include/linux/rculist.h:11,
> from ./include/linux/pid.h:5,
> from ./include/linux/sched.h:14,
> from arch/powerpc/kernel/ptrace.c:19:
> In function ‘memcpy’,
> inlined from ‘user_regset_copyin’ at ./include/linux/regset.h:295:4,
> inlined from ‘vr_set’ at arch/powerpc/kernel/ptrace.c:619:9:
> ./include/linux/string.h:345:9: error: ‘__builtin_memcpy’ offset [-527, -529] is out of the bounds [0, 16] of object ‘vrsave’ with type ‘union <anonymous>’ [-Werror=array-bounds]
> return __builtin_memcpy(p, q, size);
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
> arch/powerpc/kernel/ptrace.c: In function ‘vr_set’:
> arch/powerpc/kernel/ptrace.c:614:5: note: ‘vrsave’ declared here
> } vrsave;
> ^~~~~~
> In file included from ./include/linux/bitmap.h:9,
> from ./include/linux/cpumask.h:12,
> from ./include/linux/rcupdate.h:44,
> from ./include/linux/rculist.h:11,
> from ./include/linux/pid.h:5,
> from ./include/linux/sched.h:14,
> from arch/powerpc/kernel/ptrace.c:19:
> In function ‘memcpy’,
> inlined from ‘user_regset_copyout’ at ./include/linux/regset.h:270:4,
> inlined from ‘vr_get’ at arch/powerpc/kernel/ptrace.c:572:9:
> ./include/linux/string.h:345:9: error: ‘__builtin_memcpy’ offset [-527, -529] is out of the bounds [0, 16] of object ‘vrsave’ with type ‘union <anonymous>’ [-Werror=array-bounds]
> return __builtin_memcpy(p, q, size);
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
> arch/powerpc/kernel/ptrace.c: In function ‘vr_get’:
> arch/powerpc/kernel/ptrace.c:567:5: note: ‘vrsave’ declared here
> } vrsave;
> ^~~~~~
> cc1: all warnings being treated as errors
> make[1]: *** [scripts/Makefile.build:306: arch/powerpc/kernel/ptrace.o] Error 1
> make: *** [Makefile:1052: arch/powerpc/kernel] Error 2
>
> --
> Meelis Roos <mroos@linux.ee>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox