* Re: [PATCH v3 22/33] KVM: PPC: Book3S HV: Handle page fault for a nested guest
From: David Gibson @ 2018-10-05 2:46 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <20181004092120.GA3255@fergus>
[-- Attachment #1: Type: text/plain, Size: 4821 bytes --]
On Thu, Oct 04, 2018 at 07:21:20PM +1000, Paul Mackerras wrote:
> On Wed, Oct 03, 2018 at 03:39:13PM +1000, David Gibson wrote:
> > On Tue, Oct 02, 2018 at 09:31:21PM +1000, Paul Mackerras wrote:
> > > From: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
> > > @@ -367,7 +367,9 @@ struct kvmppc_pte {
> > > bool may_write : 1;
> > > bool may_execute : 1;
> > > unsigned long wimg;
> > > + unsigned long rc;
> > > u8 page_size; /* MMU_PAGE_xxx */
> > > + u16 page_shift;
> >
> > It's a bit ugly that this has both page_size and page_shift, which is
> > redundant information AFAICT. Also, why does page_shift need to be
> > u16 - given that 2^255 bytes is much more than our supported address
> > space, let alone a plausible page size.
>
> These values are all essentially function outputs, so I don't think
> it's ugly to have the same information in different forms. I actually
> don't like using the MMU_PAGE_xxx values, because the information in
> the mmu_psize_defs[] array depends on the MMU mode of the host, but
> KVM needs to be able to work with guests in both MMU modes. More
> generally I don't think it's a good idea that the KVM <-> guest
> interface depends so much on what the host firmware tells us about the
> physical machine we're on. Thus I'm trying to move away from using
> MMU_PSIZE_xxx values and mmu_psize_defs[] in KVM code.
Fair enough.
> I'll change the type to u8.
>
> > > diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c
> > > index bd06a95..ee6f493 100644
> > > --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c
> > > +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c
> > > @@ -29,43 +29,16 @@
> > > */
> > > static int p9_supported_radix_bits[4] = { 5, 9, 9, 13 };
> > >
> > > -/*
> > > - * Used to walk a partition or process table radix tree in guest memory
> > > - * Note: We exploit the fact that a partition table and a process
> > > - * table have the same layout, a partition-scoped page table and a
> > > - * process-scoped page table have the same layout, and the 2nd
> > > - * doubleword of a partition table entry has the same layout as
> > > - * the PTCR register.
> > > - */
> > > -int kvmppc_mmu_radix_translate_table(struct kvm_vcpu *vcpu, gva_t eaddr,
> > > - struct kvmppc_pte *gpte, u64 table,
> > > - int table_index, u64 *pte_ret_p)
> > > +int kvmppc_mmu_walk_radix_tree(struct kvm_vcpu *vcpu, gva_t eaddr,
> > > + struct kvmppc_pte *gpte, u64 root,
> > > + u64 *pte_ret_p)
> > > {
> > > struct kvm *kvm = vcpu->kvm;
> > > int ret, level, ps;
> > > - unsigned long ptbl, root;
> > > - unsigned long rts, bits, offset;
> > > - unsigned long size, index;
> > > - struct prtb_entry entry;
> > > + unsigned long rts, bits, offset, index;
> > > u64 pte, base, gpa;
> > > __be64 rpte;
> > >
> > > - if ((table & PRTS_MASK) > 24)
> > > - return -EINVAL;
> > > - size = 1ul << ((table & PRTS_MASK) + 12);
> > > -
> > > - /* Is the table big enough to contain this entry? */
> > > - if ((table_index * sizeof(entry)) >= size)
> > > - return -EINVAL;
> > > -
> > > - /* Read the table to find the root of the radix tree */
> > > - ptbl = (table & PRTB_MASK) + (table_index * sizeof(entry));
> > > - ret = kvm_read_guest(kvm, ptbl, &entry, sizeof(entry));
> > > - if (ret)
> > > - return ret;
> > > -
> > > - /* Root is stored in the first double word */
> > > - root = be64_to_cpu(entry.prtb0);
> >
> > This refactoring somewhat obscures the changes directly relevant to
> > the nested guest handling. Ideally it would be nice to fold some of
> > this into the earlier reworkings.
>
> True, but given the rapidly approaching merge window, I'm not inclined
> to rework it.
Yeah, ok.
>
> > > + if (ret) {
> > > + /* We didn't find a pte */
> > > + if (ret == -EINVAL) {
> > > + /* Unsupported mmu config */
> > > + flags |= DSISR_UNSUPP_MMU;
> > > + } else if (ret == -ENOENT) {
> > > + /* No translation found */
> > > + flags |= DSISR_NOHPTE;
> > > + } else if (ret == -EFAULT) {
> > > + /* Couldn't access L1 real address */
> > > + flags |= DSISR_PRTABLE_FAULT;
> > > + vcpu->arch.fault_gpa = fault_addr;
> > > + } else {
> > > + /* Unknown error */
> > > + return ret;
> > > + }
> > > + goto resume_host;
> >
> > This is effectively forwarding the fault to L1, yes? In which case a
> > different name might be better than the ambiguous "resume_host".
>
> I'll change it to "forward_to_l1".
Thanks.
>
> Paul.
>
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v4 22/32] KVM: PPC: Book3S HV: Introduce rmap to track nested guest mappings
From: David Gibson @ 2018-10-05 2:49 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <1538654169-15602-23-git-send-email-paulus@ozlabs.org>
[-- Attachment #1: Type: text/plain, Size: 17820 bytes --]
On Thu, Oct 04, 2018 at 09:55:59PM +1000, Paul Mackerras wrote:
> From: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
>
> When a host (L0) page which is mapped into a (L1) guest is in turn
> mapped through to a nested (L2) guest we keep a reverse mapping (rmap)
> so that these mappings can be retrieved later.
>
> Whenever we create an entry in a shadow_pgtable for a nested guest we
> create a corresponding rmap entry and add it to the list for the
> L1 guest memslot at the index of the L1 guest page it maps. This means
> at the L1 guest memslot we end up with lists of rmaps.
>
> When we are notified of a host page being invalidated which has been
> mapped through to a (L1) guest, we can then walk the rmap list for that
> guest page, and find and invalidate all of the corresponding
> shadow_pgtable entries.
>
> In order to reduce memory consumption, we compress the information for
> each rmap entry down to 52 bits -- 12 bits for the LPID and 40 bits
> for the guest real page frame number -- which will fit in a single
> unsigned long. To avoid a scenario where a guest can trigger
> unbounded memory allocations, we scan the list when adding an entry to
> see if there is already an entry with the contents we need. This can
> occur, because we don't ever remove entries from the middle of a list.
>
> A struct nested guest rmap is a list pointer and an rmap entry;
> ----------------
> | next pointer |
> ----------------
> | rmap entry |
> ----------------
>
> Thus the rmap pointer for each guest frame number in the memslot can be
> either NULL, a single entry, or a pointer to a list of nested rmap entries.
>
> gfn memslot rmap array
> -------------------------
> 0 | NULL | (no rmap entry)
> -------------------------
> 1 | single rmap entry | (rmap entry with low bit set)
> -------------------------
> 2 | list head pointer | (list of rmap entries)
> -------------------------
>
> The final entry always has the lowest bit set and is stored in the next
> pointer of the last list entry, or as a single rmap entry.
> With a list of rmap entries looking like;
>
> ----------------- ----------------- -------------------------
> | list head ptr | ----> | next pointer | ----> | single rmap entry |
> ----------------- ----------------- -------------------------
> | rmap entry | | rmap entry |
> ----------------- -------------------------
>
> Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
> arch/powerpc/include/asm/kvm_book3s.h | 3 +
> arch/powerpc/include/asm/kvm_book3s_64.h | 70 ++++++++++++++++-
> arch/powerpc/kvm/book3s_64_mmu_radix.c | 44 +++++++----
> arch/powerpc/kvm/book3s_hv.c | 1 +
> arch/powerpc/kvm/book3s_hv_nested.c | 130 ++++++++++++++++++++++++++++++-
> 5 files changed, 233 insertions(+), 15 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/kvm_book3s.h b/arch/powerpc/include/asm/kvm_book3s.h
> index 63f7ccf..d7aeb6f 100644
> --- a/arch/powerpc/include/asm/kvm_book3s.h
> +++ b/arch/powerpc/include/asm/kvm_book3s.h
> @@ -196,6 +196,9 @@ extern int kvmppc_mmu_radix_translate_table(struct kvm_vcpu *vcpu, gva_t eaddr,
> int table_index, u64 *pte_ret_p);
> extern int kvmppc_mmu_radix_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,
> struct kvmppc_pte *gpte, bool data, bool iswrite);
> +extern void kvmppc_unmap_pte(struct kvm *kvm, pte_t *pte, unsigned long gpa,
> + unsigned int shift, struct kvm_memory_slot *memslot,
> + unsigned int lpid);
> extern bool kvmppc_hv_handle_set_rc(struct kvm *kvm, pgd_t *pgtable,
> bool writing, unsigned long gpa,
> unsigned int lpid);
> diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h
> index 5496152..a02f0b3 100644
> --- a/arch/powerpc/include/asm/kvm_book3s_64.h
> +++ b/arch/powerpc/include/asm/kvm_book3s_64.h
> @@ -53,6 +53,66 @@ struct kvm_nested_guest {
> struct kvm_nested_guest *next;
> };
>
> +/*
> + * We define a nested rmap entry as a single 64-bit quantity
> + * 0xFFF0000000000000 12-bit lpid field
> + * 0x000FFFFFFFFFF000 40-bit guest 4k page frame number
> + * 0x0000000000000001 1-bit single entry flag
> + */
> +#define RMAP_NESTED_LPID_MASK 0xFFF0000000000000UL
> +#define RMAP_NESTED_LPID_SHIFT (52)
> +#define RMAP_NESTED_GPA_MASK 0x000FFFFFFFFFF000UL
> +#define RMAP_NESTED_IS_SINGLE_ENTRY 0x0000000000000001UL
> +
> +/* Structure for a nested guest rmap entry */
> +struct rmap_nested {
> + struct llist_node list;
> + u64 rmap;
> +};
> +
> +/*
> + * for_each_nest_rmap_safe - iterate over the list of nested rmap entries
> + * safe against removal of the list entry or NULL list
> + * @pos: a (struct rmap_nested *) to use as a loop cursor
> + * @node: pointer to the first entry
> + * NOTE: this can be NULL
> + * @rmapp: an (unsigned long *) in which to return the rmap entries on each
> + * iteration
> + * NOTE: this must point to already allocated memory
> + *
> + * The nested_rmap is a llist of (struct rmap_nested) entries pointed to by the
> + * rmap entry in the memslot. The list is always terminated by a "single entry"
> + * stored in the list element of the final entry of the llist. If there is ONLY
> + * a single entry then this is itself in the rmap entry of the memslot, not a
> + * llist head pointer.
> + *
> + * Note that the iterator below assumes that a nested rmap entry is always
> + * non-zero. This is true for our usage because the LPID field is always
> + * non-zero (zero is reserved for the host).
> + *
> + * This should be used to iterate over the list of rmap_nested entries with
> + * processing done on the u64 rmap value given by each iteration. This is safe
> + * against removal of list entries and it is always safe to call free on (pos).
> + *
> + * e.g.
> + * struct rmap_nested *cursor;
> + * struct llist_node *first;
> + * unsigned long rmap;
> + * for_each_nest_rmap_safe(cursor, first, &rmap) {
> + * do_something(rmap);
> + * free(cursor);
> + * }
> + */
> +#define for_each_nest_rmap_safe(pos, node, rmapp) \
> + for ((pos) = llist_entry((node), typeof(*(pos)), list); \
> + (node) && \
> + (*(rmapp) = ((RMAP_NESTED_IS_SINGLE_ENTRY & ((u64) (node))) ? \
> + ((u64) (node)) : ((pos)->rmap))) && \
> + (((node) = ((RMAP_NESTED_IS_SINGLE_ENTRY & ((u64) (node))) ? \
> + ((struct llist_node *) ((pos) = NULL)) : \
> + (pos)->list.next)), true); \
> + (pos) = llist_entry((node), typeof(*(pos)), list))
> +
> struct kvm_nested_guest *kvmhv_get_nested(struct kvm *kvm, int l1_lpid,
> bool create);
> void kvmhv_put_nested(struct kvm_nested_guest *gp);
> @@ -551,7 +611,15 @@ static inline void copy_to_checkpoint(struct kvm_vcpu *vcpu)
>
> extern int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,
> unsigned long gpa, unsigned int level,
> - unsigned long mmu_seq, unsigned int lpid);
> + unsigned long mmu_seq, unsigned int lpid,
> + unsigned long *rmapp, struct rmap_nested **n_rmap);
> +extern void kvmhv_insert_nest_rmap(struct kvm *kvm, unsigned long *rmapp,
> + struct rmap_nested **n_rmap);
> +extern void kvmhv_remove_nest_rmap_range(struct kvm *kvm,
> + struct kvm_memory_slot *memslot,
> + unsigned long gpa, unsigned long hpa,
> + unsigned long nbytes);
> +extern void kvmhv_free_memslot_nest_rmap(struct kvm_memory_slot *free);
>
> #endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */
>
> diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c
> index c4b1a9e..4c1eccb 100644
> --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c
> +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c
> @@ -256,27 +256,38 @@ static void kvmppc_pmd_free(pmd_t *pmdp)
> kmem_cache_free(kvm_pmd_cache, pmdp);
> }
>
> -void kvmppc_unmap_pte(struct kvm *kvm, pte_t *pte,
> - unsigned long gpa, unsigned int shift,
> - struct kvm_memory_slot *memslot,
> +/* Called with kvm->mmu_lock held */
> +void kvmppc_unmap_pte(struct kvm *kvm, pte_t *pte, unsigned long gpa,
> + unsigned int shift, struct kvm_memory_slot *memslot,
> unsigned int lpid)
>
> {
> unsigned long old;
> + unsigned long gfn = gpa >> PAGE_SHIFT;
> + unsigned long page_size = PAGE_SIZE;
> + unsigned long hpa;
>
> old = kvmppc_radix_update_pte(kvm, pte, ~0UL, 0, gpa, shift);
> kvmppc_radix_tlbie_page(kvm, gpa, shift, lpid);
> - if ((old & _PAGE_DIRTY) && (lpid == kvm->arch.lpid)) {
> - unsigned long gfn = gpa >> PAGE_SHIFT;
> - unsigned long page_size = PAGE_SIZE;
>
> - if (shift)
> - page_size = 1ul << shift;
> + /* The following only applies to L1 entries */
> + if (lpid != kvm->arch.lpid)
> + return;
> +
> + if (!memslot) {
> + memslot = gfn_to_memslot(kvm, gfn);
> if (!memslot)
> - memslot = gfn_to_memslot(kvm, gfn);
> - if (memslot && memslot->dirty_bitmap)
> - kvmppc_update_dirty_map(memslot, gfn, page_size);
> + return;
> }
> + if (shift)
> + page_size = 1ul << shift;
> +
> + gpa &= ~(page_size - 1);
> + hpa = old & PTE_RPN_MASK;
> + kvmhv_remove_nest_rmap_range(kvm, memslot, gpa, hpa, page_size);
> +
> + if ((old & _PAGE_DIRTY) && memslot->dirty_bitmap)
> + kvmppc_update_dirty_map(memslot, gfn, page_size);
> }
>
> /*
> @@ -430,7 +441,8 @@ static void kvmppc_unmap_free_pud_entry_table(struct kvm *kvm, pud_t *pud,
>
> int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,
> unsigned long gpa, unsigned int level,
> - unsigned long mmu_seq, unsigned int lpid)
> + unsigned long mmu_seq, unsigned int lpid,
> + unsigned long *rmapp, struct rmap_nested **n_rmap)
> {
> pgd_t *pgd;
> pud_t *pud, *new_pud = NULL;
> @@ -509,6 +521,8 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,
> kvmppc_unmap_free_pud_entry_table(kvm, pud, gpa, lpid);
> }
> kvmppc_radix_set_pte_at(kvm, gpa, (pte_t *)pud, pte);
> + if (rmapp && n_rmap)
> + kvmhv_insert_nest_rmap(kvm, rmapp, n_rmap);
> ret = 0;
> goto out_unlock;
> }
> @@ -559,6 +573,8 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,
> kvmppc_unmap_free_pmd_entry_table(kvm, pmd, gpa, lpid);
> }
> kvmppc_radix_set_pte_at(kvm, gpa, pmdp_ptep(pmd), pte);
> + if (rmapp && n_rmap)
> + kvmhv_insert_nest_rmap(kvm, rmapp, n_rmap);
> ret = 0;
> goto out_unlock;
> }
> @@ -583,6 +599,8 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,
> goto out_unlock;
> }
> kvmppc_radix_set_pte_at(kvm, gpa, ptep, pte);
> + if (rmapp && n_rmap)
> + kvmhv_insert_nest_rmap(kvm, rmapp, n_rmap);
> ret = 0;
>
> out_unlock:
> @@ -710,7 +728,7 @@ int kvmppc_book3s_instantiate_page(struct kvm_vcpu *vcpu,
>
> /* Allocate space in the tree and write the PTE */
> ret = kvmppc_create_pte(kvm, kvm->arch.pgtable, pte, gpa, level,
> - mmu_seq, kvm->arch.lpid);
> + mmu_seq, kvm->arch.lpid, NULL, NULL);
> if (inserted_pte)
> *inserted_pte = pte;
> if (levelp)
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index 134d7c7..2d8209a 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -4269,6 +4269,7 @@ static void kvmppc_core_free_memslot_hv(struct kvm_memory_slot *free,
> struct kvm_memory_slot *dont)
> {
> if (!dont || free->arch.rmap != dont->arch.rmap) {
> + kvmhv_free_memslot_nest_rmap(free);
> vfree(free->arch.rmap);
> free->arch.rmap = NULL;
> }
> diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
> index 9c04242..3947aa5 100644
> --- a/arch/powerpc/kvm/book3s_hv_nested.c
> +++ b/arch/powerpc/kvm/book3s_hv_nested.c
> @@ -10,6 +10,7 @@
>
> #include <linux/kernel.h>
> #include <linux/kvm_host.h>
> +#include <linux/llist.h>
>
> #include <asm/kvm_ppc.h>
> #include <asm/kvm_book3s.h>
> @@ -541,6 +542,123 @@ void kvmhv_put_nested(struct kvm_nested_guest *gp)
> kvmhv_release_nested(gp);
> }
>
> +static struct kvm_nested_guest *kvmhv_find_nested(struct kvm *kvm, int lpid)
> +{
> + if (lpid > kvm->arch.max_nested_lpid)
> + return NULL;
> + return kvm->arch.nested_guests[lpid];
> +}
> +
> +static inline bool kvmhv_n_rmap_is_equal(u64 rmap_1, u64 rmap_2)
> +{
> + return !((rmap_1 ^ rmap_2) & (RMAP_NESTED_LPID_MASK |
> + RMAP_NESTED_GPA_MASK));
> +}
> +
> +void kvmhv_insert_nest_rmap(struct kvm *kvm, unsigned long *rmapp,
> + struct rmap_nested **n_rmap)
> +{
> + struct llist_node *entry = ((struct llist_head *) rmapp)->first;
> + struct rmap_nested *cursor;
> + u64 rmap, new_rmap = (*n_rmap)->rmap;
> +
> + /* Are there any existing entries? */
> + if (!(*rmapp)) {
> + /* No -> use the rmap as a single entry */
> + *rmapp = new_rmap | RMAP_NESTED_IS_SINGLE_ENTRY;
> + return;
> + }
> +
> + /* Do any entries match what we're trying to insert? */
> + for_each_nest_rmap_safe(cursor, entry, &rmap) {
> + if (kvmhv_n_rmap_is_equal(rmap, new_rmap))
> + return;
> + }
> +
> + /* Do we need to create a list or just add the new entry? */
> + rmap = *rmapp;
> + if (rmap & RMAP_NESTED_IS_SINGLE_ENTRY) /* Not previously a list */
> + *rmapp = 0UL;
> + llist_add(&((*n_rmap)->list), (struct llist_head *) rmapp);
> + if (rmap & RMAP_NESTED_IS_SINGLE_ENTRY) /* Not previously a list */
> + (*n_rmap)->list.next = (struct llist_node *) rmap;
> +
> + /* Set NULL so not freed by caller */
> + *n_rmap = NULL;
> +}
> +
> +static void kvmhv_remove_nest_rmap(struct kvm *kvm, u64 n_rmap,
> + unsigned long hpa, unsigned long mask)
> +{
> + struct kvm_nested_guest *gp;
> + unsigned long gpa;
> + unsigned int shift, lpid;
> + pte_t *ptep;
> +
> + gpa = n_rmap & RMAP_NESTED_GPA_MASK;
> + lpid = (n_rmap & RMAP_NESTED_LPID_MASK) >> RMAP_NESTED_LPID_SHIFT;
> + gp = kvmhv_find_nested(kvm, lpid);
> + if (!gp)
> + return;
> +
> + /* Find and invalidate the pte */
> + ptep = __find_linux_pte(gp->shadow_pgtable, gpa, NULL, &shift);
> + /* Don't spuriously invalidate ptes if the pfn has changed */
> + if (ptep && pte_present(*ptep) && ((pte_val(*ptep) & mask) == hpa))
> + kvmppc_unmap_pte(kvm, ptep, gpa, shift, NULL, gp->shadow_lpid);
> +}
> +
> +static void kvmhv_remove_nest_rmap_list(struct kvm *kvm, unsigned long *rmapp,
> + unsigned long hpa, unsigned long mask)
> +{
> + struct llist_node *entry = llist_del_all((struct llist_head *) rmapp);
> + struct rmap_nested *cursor;
> + unsigned long rmap;
> +
> + for_each_nest_rmap_safe(cursor, entry, &rmap) {
> + kvmhv_remove_nest_rmap(kvm, rmap, hpa, mask);
> + kfree(cursor);
> + }
> +}
> +
> +/* called with kvm->mmu_lock held */
> +void kvmhv_remove_nest_rmap_range(struct kvm *kvm,
> + struct kvm_memory_slot *memslot,
> + unsigned long gpa, unsigned long hpa,
> + unsigned long nbytes)
> +{
> + unsigned long gfn, end_gfn;
> + unsigned long addr_mask;
> +
> + if (!memslot)
> + return;
> + gfn = (gpa >> PAGE_SHIFT) - memslot->base_gfn;
> + end_gfn = gfn + (nbytes >> PAGE_SHIFT);
> +
> + addr_mask = PTE_RPN_MASK & ~(nbytes - 1);
> + hpa &= addr_mask;
> +
> + for (; gfn < end_gfn; gfn++) {
> + unsigned long *rmap = &memslot->arch.rmap[gfn];
> + kvmhv_remove_nest_rmap_list(kvm, rmap, hpa, addr_mask);
> + }
> +}
> +
> +void kvmhv_free_memslot_nest_rmap(struct kvm_memory_slot *free)
> +{
> + unsigned long page;
> +
> + for (page = 0; page < free->npages; page++) {
> + unsigned long rmap, *rmapp = &free->arch.rmap[page];
> + struct rmap_nested *cursor;
> + struct llist_node *entry;
> +
> + entry = llist_del_all((struct llist_head *) rmapp);
> + for_each_nest_rmap_safe(cursor, entry, &rmap)
> + kfree(cursor);
> + }
> +}
> +
> static bool kvmhv_invalidate_shadow_pte(struct kvm_vcpu *vcpu,
> struct kvm_nested_guest *gp,
> long gpa, int *shift_ret)
> @@ -692,11 +810,13 @@ static long int __kvmhv_nested_page_fault(struct kvm_vcpu *vcpu,
> {
> struct kvm *kvm = vcpu->kvm;
> struct kvm_memory_slot *memslot;
> + struct rmap_nested *n_rmap;
> struct kvmppc_pte gpte;
> pte_t pte, *pte_p;
> unsigned long mmu_seq;
> unsigned long dsisr = vcpu->arch.fault_dsisr;
> unsigned long ea = vcpu->arch.fault_dar;
> + unsigned long *rmapp;
> unsigned long n_gpa, gpa, gfn, perm = 0UL;
> unsigned int shift, l1_shift, level;
> bool writing = !!(dsisr & DSISR_ISSTORE);
> @@ -830,8 +950,16 @@ static long int __kvmhv_nested_page_fault(struct kvm_vcpu *vcpu,
>
> /* 4. Insert the pte into our shadow_pgtable */
>
> + n_rmap = kzalloc(sizeof(*n_rmap), GFP_KERNEL);
> + if (!n_rmap)
> + return RESUME_GUEST; /* Let the guest try again */
> + n_rmap->rmap = (n_gpa & RMAP_NESTED_GPA_MASK) |
> + (((unsigned long) gp->l1_lpid) << RMAP_NESTED_LPID_SHIFT);
> + rmapp = &memslot->arch.rmap[gfn - memslot->base_gfn];
> ret = kvmppc_create_pte(kvm, gp->shadow_pgtable, pte, n_gpa, level,
> - mmu_seq, gp->shadow_lpid);
> + mmu_seq, gp->shadow_lpid, rmapp, &n_rmap);
> + if (n_rmap)
> + kfree(n_rmap);
> if (ret == -EAGAIN)
> ret = RESUME_GUEST; /* Let the guest try again */
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v3 33/33] KVM: PPC: Book3S HV: Add a VM capability to enable nested virtualization
From: David Gibson @ 2018-10-05 3:29 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <20181004094826.GC3255@fergus>
[-- Attachment #1: Type: text/plain, Size: 1577 bytes --]
On Thu, Oct 04, 2018 at 07:48:26PM +1000, Paul Mackerras wrote:
> On Wed, Oct 03, 2018 at 04:21:44PM +1000, David Gibson wrote:
> > On Tue, Oct 02, 2018 at 09:31:32PM +1000, Paul Mackerras wrote:
> > > With this, userspace can enable a KVM-HV guest to run nested guests
> > > under it.
> [snip]
> > > +/* If set, guests are allowed to create and control nested guests */
> > > +static bool enable_nested = true;
> > > +module_param(enable_nested, bool, S_IRUGO | S_IWUSR);
> > > +MODULE_PARM_DESC(enable_nested, "Enable nested virtualization (only on POWER9)");
> >
> > I'd suggest calling the module parameter just "nested" to match x86.
>
> OK.
>
> > > /* If set, the threads on each CPU core have to be in the same MMU mode */
> > > static bool no_mixing_hpt_and_radix;
> > >
> > > @@ -5188,6 +5193,17 @@ static int kvmhv_configure_mmu(struct kvm *kvm, struct kvm_ppc_mmuv3_cfg *cfg)
> > > return err;
> > > }
> > >
> > > +static int kvmhv_enable_nested(struct kvm *kvm, bool enable)
> > > +{
> > > + if (!(enable_nested && cpu_has_feature(CPU_FTR_ARCH_300)))
> > > + return -EINVAL;
> >
> > Maybe EPERM, rather than EINVAL. There's nothing invalid about the
> > ioctl() parameters - we just can't do what they want.
>
> Just for pedantry's sake, I'll make it EPERM for !enable_nested and
> ENODEV for !POWER9. :)
Sounds fair.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v4 23/32] KVM: PPC: Book3S HV: Implement H_TLB_INVALIDATE hcall
From: David Gibson @ 2018-10-05 3:40 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <1538654169-15602-24-git-send-email-paulus@ozlabs.org>
[-- Attachment #1: Type: text/plain, Size: 3467 bytes --]
On Thu, Oct 04, 2018 at 09:56:00PM +1000, Paul Mackerras wrote:
11;rgb:ffff/ffff/ffff> From: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
>
> When running a nested (L2) guest the guest (L1) hypervisor will use
> the H_TLB_INVALIDATE hcall when it needs to change the partition
> scoped page tables or the partition table which it manages. It will
> use this hcall in the situations where it would use a partition-scoped
> tlbie instruction if it were running in hypervisor mode.
>
> The H_TLB_INVALIDATE hcall can invalidate different scopes:
>
> Invalidate TLB for a given target address:
> - This invalidates a single L2 -> L1 pte
> - We need to invalidate any L2 -> L0 shadow_pgtable ptes which map the L2
> address space which is being invalidated. This is because a single
> L2 -> L1 pte may have been mapped with more than one pte in the
> L2 -> L0 page tables.
>
> Invalidate the entire TLB for a given LPID or for all LPIDs:
> - Invalidate the entire shadow_pgtable for a given nested guest, or
> for all nested guests.
>
> Invalidate the PWC (page walk cache) for a given LPID or for all LPIDs:
> - We don't cache the PWC, so nothing to do.
>
> Invalidate the entire TLB, PWC and partition table for a given/all LPIDs:
> - Here we re-read the partition table entry and remove the nested state
> for any nested guest for which the first doubleword of the partition
> table entry is now zero.
>
> The H_TLB_INVALIDATE hcall takes as parameters the tlbie instruction
> word (of which only the RIC, PRS and R fields are used), the rS value
> (giving the lpid, where required) and the rB value (giving the IS, AP
> and EPN values).
>
> [paulus@ozlabs.org - adapted to having the partition table in guest
> memory, added the H_TLB_INVALIDATE implementation, removed tlbie
> instruction emulation, reworded the commit message.]
>
> Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
That said, there's one change I think could make it substantially
easier to read..
[snip]
> +static int kvmhv_emulate_tlbie_tlb_addr(struct kvm_vcpu *vcpu, int lpid,
> + int ap, long epn)
> +{
> + struct kvm *kvm = vcpu->kvm;
> + struct kvm_nested_guest *gp;
> + long npages;
> + int shift;
> + unsigned long addr;
> +
> + shift = ap_to_shift(ap);
> + addr = epn << 12;
> + if (shift < 0)
> + /* Invalid ap encoding */
> + return -EINVAL;
> +
> + addr &= ~((1UL << shift) - 1);
> + npages = 1UL << (shift - PAGE_SHIFT);
> +
> + gp = kvmhv_get_nested(kvm, lpid, false);
> + if (!gp) /* No such guest -> nothing to do */
> + return 0;
> + mutex_lock(&gp->tlb_lock);
> +
> + /* There may be more than one host page backing this single guest pte */
> + do {
> + kvmhv_invalidate_shadow_pte(vcpu, gp, addr, &shift);
> +
> + npages -= 1UL << (shift - PAGE_SHIFT);
> + addr += 1UL << shift;
I read this about 6 times before realizing that 'shift' here has a
different value to what it did before the loop (which it has to in
order to be correct). I'd suggest a loop local variable with a
different name (maybe 'shadow_shift') to make that more obvious.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v4 24/32] KVM: PPC: Book3S HV: Use hypercalls for TLB invalidation when nested
From: David Gibson @ 2018-10-05 3:45 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <1538654169-15602-25-git-send-email-paulus@ozlabs.org>
[-- Attachment #1: Type: text/plain, Size: 5652 bytes --]
On Thu, Oct 04, 2018 at 09:56:01PM +1000, Paul Mackerras wrote:
> This adds code to call the H_TLB_INVALIDATE hypercall when running as
> a guest, in the cases where we need to invalidate TLBs (or other MMU
> caches) as part of managing the mappings for a nested guest. Calling
> H_TLB_INVALIDATE lets the nested hypervisor inform the parent
> hypervisor about changes to partition-scoped page tables or the
> partition table without needing to do hypervisor-privileged tlbie
> instructions.
>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
> arch/powerpc/include/asm/kvm_book3s_64.h | 5 +++++
> arch/powerpc/kvm/book3s_64_mmu_radix.c | 30 ++++++++++++++++++++++++++++--
> arch/powerpc/kvm/book3s_hv_nested.c | 30 ++++++++++++++++++++++++------
> 3 files changed, 57 insertions(+), 8 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h
> index a02f0b3..aa5bf85 100644
> --- a/arch/powerpc/include/asm/kvm_book3s_64.h
> +++ b/arch/powerpc/include/asm/kvm_book3s_64.h
> @@ -24,6 +24,7 @@
> #include <asm/bitops.h>
> #include <asm/book3s/64/mmu-hash.h>
> #include <asm/cpu_has_feature.h>
> +#include <asm/ppc-opcode.h>
>
> #ifdef CONFIG_PPC_PSERIES
> static inline bool kvmhv_on_pseries(void)
> @@ -117,6 +118,10 @@ struct kvm_nested_guest *kvmhv_get_nested(struct kvm *kvm, int l1_lpid,
> bool create);
> void kvmhv_put_nested(struct kvm_nested_guest *gp);
>
> +/* Encoding of first parameter for H_TLB_INVALIDATE */
> +#define H_TLBIE_P1_ENC(ric, prs, r) (___PPC_RIC(ric) | ___PPC_PRS(prs) | \
> + ___PPC_R(r))
> +
> /* Power architecture requires HPT is at least 256kiB, at most 64TiB */
> #define PPC_MIN_HPT_ORDER 18
> #define PPC_MAX_HPT_ORDER 46
> diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c
> index 4c1eccb..ae0e3ed 100644
> --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c
> +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c
> @@ -201,17 +201,43 @@ static void kvmppc_radix_tlbie_page(struct kvm *kvm, unsigned long addr,
> unsigned int pshift, unsigned int lpid)
> {
> unsigned long psize = PAGE_SIZE;
> + int psi;
> + long rc;
> + unsigned long rb;
>
> if (pshift)
> psize = 1UL << pshift;
> + else
> + pshift = PAGE_SHIFT;
>
> addr &= ~(psize - 1);
> - radix__flush_tlb_lpid_page(lpid, addr, psize);
> +
> + if (!kvmhv_on_pseries()) {
> + radix__flush_tlb_lpid_page(lpid, addr, psize);
> + return;
> + }
> +
> + psi = shift_to_mmu_psize(pshift);
> + rb = addr | (mmu_get_ap(psi) << PPC_BITLSHIFT(58));
> + rc = plpar_hcall_norets(H_TLB_INVALIDATE, H_TLBIE_P1_ENC(0, 0, 1),
> + lpid, rb);
> + if (rc)
> + pr_err("KVM: TLB page invalidation hcall failed, rc=%ld\n", rc);
> }
>
> static void kvmppc_radix_flush_pwc(struct kvm *kvm, unsigned int lpid)
> {
> - radix__flush_pwc_lpid(lpid);
> + long rc;
> +
> + if (!kvmhv_on_pseries()) {
> + radix__flush_pwc_lpid(lpid);
> + return;
> + }
> +
> + rc = plpar_hcall_norets(H_TLB_INVALIDATE, H_TLBIE_P1_ENC(1, 0, 1),
> + lpid, TLBIEL_INVAL_SET_LPID);
> + if (rc)
> + pr_err("KVM: TLB PWC invalidation hcall failed, rc=%ld\n", rc);
> }
>
> static unsigned long kvmppc_radix_update_pte(struct kvm *kvm, pte_t *ptep,
> diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
> index 26151e8..35f8111 100644
> --- a/arch/powerpc/kvm/book3s_hv_nested.c
> +++ b/arch/powerpc/kvm/book3s_hv_nested.c
> @@ -298,14 +298,32 @@ void kvmhv_nested_exit(void)
> }
> }
>
> +static void kvmhv_flush_lpid(unsigned int lpid)
> +{
> + long rc;
> +
> + if (!kvmhv_on_pseries()) {
> + radix__flush_tlb_lpid(lpid);
> + return;
> + }
> +
> + rc = plpar_hcall_norets(H_TLB_INVALIDATE, H_TLBIE_P1_ENC(2, 0, 1),
> + lpid, TLBIEL_INVAL_SET_LPID);
> + if (rc)
> + pr_err("KVM: TLB LPID invalidation hcall failed, rc=%ld\n", rc);
> +}
> +
> void kvmhv_set_ptbl_entry(unsigned int lpid, u64 dw0, u64 dw1)
> {
> - if (cpu_has_feature(CPU_FTR_HVMODE)) {
> + if (!kvmhv_on_pseries()) {
> mmu_partition_table_set_entry(lpid, dw0, dw1);
> - } else {
> - pseries_partition_tb[lpid].patb0 = cpu_to_be64(dw0);
> - pseries_partition_tb[lpid].patb1 = cpu_to_be64(dw1);
> + return;
> }
> +
> + pseries_partition_tb[lpid].patb0 = cpu_to_be64(dw0);
> + pseries_partition_tb[lpid].patb1 = cpu_to_be64(dw1);
> + /* L0 will do the necessary barriers */
> + kvmhv_flush_lpid(lpid);
> }
>
> static void kvmhv_set_nested_ptbl(struct kvm_nested_guest *gp)
> @@ -482,7 +500,7 @@ static void kvmhv_flush_nested(struct kvm_nested_guest *gp)
> spin_lock(&kvm->mmu_lock);
> kvmppc_free_pgtable_radix(kvm, gp->shadow_pgtable, gp->shadow_lpid);
> spin_unlock(&kvm->mmu_lock);
> - radix__flush_tlb_lpid(gp->shadow_lpid);
> + kvmhv_flush_lpid(gp->shadow_lpid);
> kvmhv_update_ptbl_cache(gp);
> if (gp->l1_gr_to_hr == 0)
> kvmhv_remove_nested(gp);
> @@ -766,7 +784,7 @@ static void kvmhv_emulate_tlbie_lpid(struct kvm_vcpu *vcpu,
> spin_lock(&kvm->mmu_lock);
> kvmppc_free_pgtable_radix(kvm, gp->shadow_pgtable,
> gp->shadow_lpid);
> - radix__flush_tlb_lpid(gp->shadow_lpid);
> + kvmhv_flush_lpid(gp->shadow_lpid);
> spin_unlock(&kvm->mmu_lock);
> break;
> case 1:
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v4 25/32] KVM: PPC: Book3S HV: Invalidate TLB when nested vcpu moves physical cpu
From: David Gibson @ 2018-10-05 4:09 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <1538654169-15602-26-git-send-email-paulus@ozlabs.org>
[-- Attachment #1: Type: text/plain, Size: 8667 bytes --]
On Thu, Oct 04, 2018 at 09:56:02PM +1000, Paul Mackerras wrote:
> From: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
>
> This is only done at level 0, since only level 0 knows which physical
> CPU a vcpu is running on. This does for nested guests what L0 already
> did for its own guests, which is to flush the TLB on a pCPU when it
> goes to run a vCPU there, and there is another vCPU in the same VM
> which previously ran on this pCPU and has now started to run on another
> pCPU. This is to handle the situation where the other vCPU touched
> a mapping, moved to another pCPU and did a tlbiel (local-only tlbie)
> on that new pCPU and thus left behind a stale TLB entry on this pCPU.
>
> This introduces a limit on the the vcpu_token values used in the
> H_ENTER_NESTED hcall -- they must now be less than NR_CPUS.
This does make the vcpu tokens no longer entirely opaque to the L0.
It works for now, because the only L1 is Linux and we know basically
how it allocates those tokens. Eventually we probably want some way
to either remove this restriction or to advertise the limit to the L1.
> [paulus@ozlabs.org - made prev_cpu array be unsigned short[] to reduce
> memory consumption.]
>
> Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
> arch/powerpc/include/asm/kvm_book3s_64.h | 3 +
> arch/powerpc/kvm/book3s_hv.c | 101 +++++++++++++++++++------------
> arch/powerpc/kvm/book3s_hv_nested.c | 5 ++
> 3 files changed, 71 insertions(+), 38 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h
> index aa5bf85..1e96027 100644
> --- a/arch/powerpc/include/asm/kvm_book3s_64.h
> +++ b/arch/powerpc/include/asm/kvm_book3s_64.h
> @@ -52,6 +52,9 @@ struct kvm_nested_guest {
> long refcnt; /* number of pointers to this struct */
> struct mutex tlb_lock; /* serialize page faults and tlbies */
> struct kvm_nested_guest *next;
> + cpumask_t need_tlb_flush;
> + cpumask_t cpu_in_guest;
> + unsigned short prev_cpu[NR_CPUS];
> };
>
> /*
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index ba58883..53a967ea 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -2397,10 +2397,18 @@ static void kvmppc_release_hwthread(int cpu)
>
> static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
> {
> + struct kvm_nested_guest *nested = vcpu->arch.nested;
> + cpumask_t *cpu_in_guest;
> int i;
>
> cpu = cpu_first_thread_sibling(cpu);
> - cpumask_set_cpu(cpu, &kvm->arch.need_tlb_flush);
> + if (nested) {
> + cpumask_set_cpu(cpu, &nested->need_tlb_flush);
> + cpu_in_guest = &nested->cpu_in_guest;
> + } else {
> + cpumask_set_cpu(cpu, &kvm->arch.need_tlb_flush);
> + cpu_in_guest = &kvm->arch.cpu_in_guest;
> + }
> /*
> * Make sure setting of bit in need_tlb_flush precedes
> * testing of cpu_in_guest bits. The matching barrier on
> @@ -2408,13 +2416,23 @@ static void radix_flush_cpu(struct kvm *kvm, int cpu, struct kvm_vcpu *vcpu)
> */
> smp_mb();
> for (i = 0; i < threads_per_core; ++i)
> - if (cpumask_test_cpu(cpu + i, &kvm->arch.cpu_in_guest))
> + if (cpumask_test_cpu(cpu + i, cpu_in_guest))
> smp_call_function_single(cpu + i, do_nothing, NULL, 1);
> }
>
> static void kvmppc_prepare_radix_vcpu(struct kvm_vcpu *vcpu, int pcpu)
> {
> + struct kvm_nested_guest *nested = vcpu->arch.nested;
> struct kvm *kvm = vcpu->kvm;
> + int prev_cpu;
> +
> + if (!cpu_has_feature(CPU_FTR_HVMODE))
> + return;
> +
> + if (nested)
> + prev_cpu = nested->prev_cpu[vcpu->arch.nested_vcpu_id];
> + else
> + prev_cpu = vcpu->arch.prev_cpu;
>
> /*
> * With radix, the guest can do TLB invalidations itself,
> @@ -2428,12 +2446,46 @@ static void kvmppc_prepare_radix_vcpu(struct kvm_vcpu *vcpu, int pcpu)
> * ran to flush the TLB. The TLB is shared between threads,
> * so we use a single bit in .need_tlb_flush for all 4 threads.
> */
> - if (vcpu->arch.prev_cpu != pcpu) {
> - if (vcpu->arch.prev_cpu >= 0 &&
> - cpu_first_thread_sibling(vcpu->arch.prev_cpu) !=
> + if (prev_cpu != pcpu) {
> + if (prev_cpu >= 0 &&
> + cpu_first_thread_sibling(prev_cpu) !=
> cpu_first_thread_sibling(pcpu))
> - radix_flush_cpu(kvm, vcpu->arch.prev_cpu, vcpu);
> - vcpu->arch.prev_cpu = pcpu;
> + radix_flush_cpu(kvm, prev_cpu, vcpu);
> + if (nested)
> + nested->prev_cpu[vcpu->arch.nested_vcpu_id] = pcpu;
> + else
> + vcpu->arch.prev_cpu = pcpu;
> + }
> +}
> +
> +static void kvmppc_radix_check_need_tlb_flush(struct kvm *kvm, int pcpu,
> + struct kvm_nested_guest *nested)
> +{
> + cpumask_t *need_tlb_flush;
> + int lpid;
> +
> + if (!cpu_has_feature(CPU_FTR_HVMODE))
> + return;
> +
> + if (cpu_has_feature(CPU_FTR_ARCH_300))
> + pcpu &= ~0x3UL;
> +
> + if (nested) {
> + lpid = nested->shadow_lpid;
> + need_tlb_flush = &nested->need_tlb_flush;
> + } else {
> + lpid = kvm->arch.lpid;
> + need_tlb_flush = &kvm->arch.need_tlb_flush;
> + }
> +
> + mtspr(SPRN_LPID, lpid);
> + isync();
> + smp_mb();
> +
> + if (cpumask_test_cpu(pcpu, need_tlb_flush)) {
> + radix__local_flush_tlb_lpid_guest(lpid);
> + /* Clear the bit after the TLB flush */
> + cpumask_clear_cpu(pcpu, need_tlb_flush);
> }
> }
>
> @@ -3127,8 +3179,6 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc)
> spin_unlock(&core_info.vc[sub]->lock);
>
> if (kvm_is_radix(vc->kvm)) {
> - int tmp = pcpu;
> -
> /*
> * Do we need to flush the process scoped TLB for the LPAR?
> *
> @@ -3139,17 +3189,7 @@ static noinline void kvmppc_run_core(struct kvmppc_vcore *vc)
> *
> * Hash must be flushed in realmode in order to use tlbiel.
> */
> - mtspr(SPRN_LPID, vc->kvm->arch.lpid);
> - isync();
> -
> - if (cpu_has_feature(CPU_FTR_ARCH_300))
> - tmp &= ~0x3UL;
> -
> - if (cpumask_test_cpu(tmp, &vc->kvm->arch.need_tlb_flush)) {
> - radix__local_flush_tlb_lpid_guest(vc->kvm->arch.lpid);
> - /* Clear the bit after the TLB flush */
> - cpumask_clear_cpu(tmp, &vc->kvm->arch.need_tlb_flush);
> - }
> + kvmppc_radix_check_need_tlb_flush(vc->kvm, pcpu, NULL);
> }
>
> /*
> @@ -3868,11 +3908,10 @@ int kvmhv_run_single_vcpu(struct kvm_run *kvm_run,
> struct kvm_vcpu *vcpu, u64 time_limit,
> unsigned long lpcr)
> {
> - int trap, r, pcpu, pcpu0;
> + int trap, r, pcpu;
> int srcu_idx;
> struct kvmppc_vcore *vc;
> struct kvm_nested_guest *nested = vcpu->arch.nested;
> - unsigned long lpid;
>
> trace_kvmppc_run_vcpu_enter(vcpu);
>
> @@ -3945,22 +3984,8 @@ int kvmhv_run_single_vcpu(struct kvm_run *kvm_run,
> vc->vcore_state = VCORE_RUNNING;
> trace_kvmppc_run_core(vc, 0);
>
> - lpid = vc->kvm->arch.lpid;
> - if (nested)
> - lpid = nested->shadow_lpid;
> - mtspr(SPRN_LPID, lpid);
> - isync();
> -
> - /* See comment above in kvmppc_run_core() about this */
> - pcpu0 = pcpu;
> - if (cpu_has_feature(CPU_FTR_ARCH_300))
> - pcpu0 &= ~0x3UL;
> -
> - if (cpumask_test_cpu(pcpu0, &vc->kvm->arch.need_tlb_flush)) {
> - radix__local_flush_tlb_lpid_guest(lpid);
> - /* Clear the bit after the TLB flush */
> - cpumask_clear_cpu(pcpu0, &vc->kvm->arch.need_tlb_flush);
> - }
> + if (cpu_has_feature(CPU_FTR_HVMODE))
> + kvmppc_radix_check_need_tlb_flush(vc->kvm, pcpu, nested);
>
> trace_hardirqs_on();
> guest_enter_irqoff();
> diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
> index 35f8111..1a8c40d 100644
> --- a/arch/powerpc/kvm/book3s_hv_nested.c
> +++ b/arch/powerpc/kvm/book3s_hv_nested.c
> @@ -167,6 +167,9 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)
> if (err)
> return H_PARAMETER;
>
> + if (l2_hv.vcpu_token >= NR_CPUS)
> + return H_PARAMETER;
> +
> /* translate lpid */
> l2 = kvmhv_get_nested(vcpu->kvm, l2_hv.lpid, true);
> if (!l2)
> @@ -411,6 +414,8 @@ struct kvm_nested_guest *kvmhv_alloc_nested(struct kvm *kvm, unsigned int lpid)
> goto out_free2;
> gp->shadow_lpid = shadow_lpid;
>
> + memset(gp->prev_cpu, -1, sizeof(gp->prev_cpu));
> +
> return gp;
>
> out_free2:
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v4 17/32] KVM: PPC: Book3S HV: Framework and hcall stubs for nested virtualization
From: David Gibson @ 2018-10-05 4:12 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <1538654169-15602-18-git-send-email-paulus@ozlabs.org>
[-- Attachment #1: Type: text/plain, Size: 18115 bytes --]
On Thu, Oct 04, 2018 at 09:55:54PM +1000, Paul Mackerras wrote:
> This starts the process of adding the code to support nested HV-style
> virtualization. It defines a new H_SET_PARTITION_TABLE hypercall which
> a nested hypervisor can use to set the base address and size of a
> partition table in its memory (analogous to the PTCR register).
> On the host (level 0 hypervisor) side, the H_SET_PARTITION_TABLE
> hypercall from the guest is handled by code that saves the virtual
> PTCR value for the guest.
>
> This also adds code for creating and destroying nested guests and for
> reading the partition table entry for a nested guest from L1 memory.
> Each nested guest has its own shadow LPID value, different in general
> from the LPID value used by the nested hypervisor to refer to it. The
> shadow LPID value is allocated at nested guest creation time.
>
> Nested hypervisor functionality is only available for a radix guest,
> which therefore means a radix host on a POWER9 (or later) processor.
>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
> arch/powerpc/include/asm/hvcall.h | 5 +
> arch/powerpc/include/asm/kvm_book3s.h | 10 +-
> arch/powerpc/include/asm/kvm_book3s_64.h | 33 ++++
> arch/powerpc/include/asm/kvm_book3s_asm.h | 3 +
> arch/powerpc/include/asm/kvm_host.h | 5 +
> arch/powerpc/kvm/Makefile | 3 +-
> arch/powerpc/kvm/book3s_hv.c | 27 ++-
> arch/powerpc/kvm/book3s_hv_nested.c | 298 ++++++++++++++++++++++++++++++
> 8 files changed, 377 insertions(+), 7 deletions(-)
> create mode 100644 arch/powerpc/kvm/book3s_hv_nested.c
>
> diff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h
> index a0b17f9..c95c651 100644
> --- a/arch/powerpc/include/asm/hvcall.h
> +++ b/arch/powerpc/include/asm/hvcall.h
> @@ -322,6 +322,11 @@
> #define H_GET_24X7_DATA 0xF07C
> #define H_GET_PERF_COUNTER_INFO 0xF080
>
> +/* Platform-specific hcalls used for nested HV KVM */
> +#define H_SET_PARTITION_TABLE 0xF800
> +#define H_ENTER_NESTED 0xF804
> +#define H_TLB_INVALIDATE 0xF808
> +
> /* Values for 2nd argument to H_SET_MODE */
> #define H_SET_MODE_RESOURCE_SET_CIABR 1
> #define H_SET_MODE_RESOURCE_SET_DAWR 2
> diff --git a/arch/powerpc/include/asm/kvm_book3s.h b/arch/powerpc/include/asm/kvm_book3s.h
> index 91c9779..43f212e 100644
> --- a/arch/powerpc/include/asm/kvm_book3s.h
> +++ b/arch/powerpc/include/asm/kvm_book3s.h
> @@ -274,6 +274,13 @@ static inline void kvmppc_save_tm_sprs(struct kvm_vcpu *vcpu) {}
> static inline void kvmppc_restore_tm_sprs(struct kvm_vcpu *vcpu) {}
> #endif
>
> +long kvmhv_nested_init(void);
> +void kvmhv_nested_exit(void);
> +void kvmhv_vm_nested_init(struct kvm *kvm);
> +long kvmhv_set_partition_table(struct kvm_vcpu *vcpu);
> +void kvmhv_set_ptbl_entry(unsigned int lpid, u64 dw0, u64 dw1);
> +void kvmhv_release_all_nested(struct kvm *kvm);
> +
> void kvmppc_giveup_fac(struct kvm_vcpu *vcpu, ulong fac);
>
> extern int kvm_irq_bypass;
> @@ -387,9 +394,6 @@ extern int kvmppc_h_logical_ci_store(struct kvm_vcpu *vcpu);
> /* TO = 31 for unconditional trap */
> #define INS_TW 0x7fe00008
>
> -/* LPIDs we support with this build -- runtime limit may be lower */
> -#define KVMPPC_NR_LPIDS (LPID_RSVD + 1)
> -
> #define SPLIT_HACK_MASK 0xff000000
> #define SPLIT_HACK_OFFS 0xfb000000
>
> diff --git a/arch/powerpc/include/asm/kvm_book3s_64.h b/arch/powerpc/include/asm/kvm_book3s_64.h
> index 5c0e2d9..6d67b6a 100644
> --- a/arch/powerpc/include/asm/kvm_book3s_64.h
> +++ b/arch/powerpc/include/asm/kvm_book3s_64.h
> @@ -23,6 +23,39 @@
> #include <linux/string.h>
> #include <asm/bitops.h>
> #include <asm/book3s/64/mmu-hash.h>
> +#include <asm/cpu_has_feature.h>
> +
> +#ifdef CONFIG_PPC_PSERIES
> +static inline bool kvmhv_on_pseries(void)
> +{
> + return !cpu_has_feature(CPU_FTR_HVMODE);
> +}
> +#else
> +static inline bool kvmhv_on_pseries(void)
> +{
> + return false;
> +}
> +#endif
> +
> +/*
> + * Structure for a nested guest, that is, for a guest that is managed by
> + * one of our guests.
> + */
> +struct kvm_nested_guest {
> + struct kvm *l1_host; /* L1 VM that owns this nested guest */
> + int l1_lpid; /* lpid L1 guest thinks this guest is */
> + int shadow_lpid; /* real lpid of this nested guest */
> + pgd_t *shadow_pgtable; /* our page table for this guest */
> + u64 l1_gr_to_hr; /* L1's addr of part'n-scoped table */
> + u64 process_table; /* process table entry for this guest */
> + long refcnt; /* number of pointers to this struct */
> + struct mutex tlb_lock; /* serialize page faults and tlbies */
> + struct kvm_nested_guest *next;
> +};
> +
> +struct kvm_nested_guest *kvmhv_get_nested(struct kvm *kvm, int l1_lpid,
> + bool create);
> +void kvmhv_put_nested(struct kvm_nested_guest *gp);
>
> /* Power architecture requires HPT is at least 256kiB, at most 64TiB */
> #define PPC_MIN_HPT_ORDER 18
> diff --git a/arch/powerpc/include/asm/kvm_book3s_asm.h b/arch/powerpc/include/asm/kvm_book3s_asm.h
> index d978fdf..eb3ba63 100644
> --- a/arch/powerpc/include/asm/kvm_book3s_asm.h
> +++ b/arch/powerpc/include/asm/kvm_book3s_asm.h
> @@ -25,6 +25,9 @@
> #define XICS_MFRR 0xc
> #define XICS_IPI 2 /* interrupt source # for IPIs */
>
> +/* LPIDs we support with this build -- runtime limit may be lower */
> +#define KVMPPC_NR_LPIDS (LPID_RSVD + 1)
> +
> /* Maximum number of threads per physical core */
> #define MAX_SMT_THREADS 8
>
> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
> index c9cc42f..c35d4f2 100644
> --- a/arch/powerpc/include/asm/kvm_host.h
> +++ b/arch/powerpc/include/asm/kvm_host.h
> @@ -46,6 +46,7 @@
> #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
> #include <asm/kvm_book3s_asm.h> /* for MAX_SMT_THREADS */
> #define KVM_MAX_VCPU_ID (MAX_SMT_THREADS * KVM_MAX_VCORES)
> +#define KVM_MAX_NESTED_GUESTS KVMPPC_NR_LPIDS
>
> #else
> #define KVM_MAX_VCPU_ID KVM_MAX_VCPUS
> @@ -287,6 +288,7 @@ struct kvm_arch {
> u8 radix;
> u8 fwnmi_enabled;
> bool threads_indep;
> + bool nested_enable;
> pgd_t *pgtable;
> u64 process_table;
> struct dentry *debugfs_dir;
> @@ -312,6 +314,9 @@ struct kvm_arch {
> #endif
> struct kvmppc_ops *kvm_ops;
> #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
> + u64 l1_ptcr;
> + int max_nested_lpid;
> + struct kvm_nested_guest *nested_guests[KVM_MAX_NESTED_GUESTS];
> /* This array can grow quite large, keep it at the end */
> struct kvmppc_vcore *vcores[KVM_MAX_VCORES];
> #endif
> diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
> index f872c04..e814f40 100644
> --- a/arch/powerpc/kvm/Makefile
> +++ b/arch/powerpc/kvm/Makefile
> @@ -75,7 +75,8 @@ kvm-hv-y += \
> book3s_hv.o \
> book3s_hv_interrupts.o \
> book3s_64_mmu_hv.o \
> - book3s_64_mmu_radix.o
> + book3s_64_mmu_radix.o \
> + book3s_hv_nested.o
>
> kvm-hv-$(CONFIG_PPC_TRANSACTIONAL_MEM) += \
> book3s_hv_tm.o
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index ca0e4f4..ca2529e 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -934,6 +934,19 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
> if (ret == H_TOO_HARD)
> return RESUME_HOST;
> break;
> +
> + case H_SET_PARTITION_TABLE:
> + ret = H_FUNCTION;
> + if (vcpu->kvm->arch.nested_enable)
> + ret = kvmhv_set_partition_table(vcpu);
> + break;
> + case H_ENTER_NESTED:
> + ret = H_FUNCTION;
> + break;
> + case H_TLB_INVALIDATE:
> + ret = H_FUNCTION;
> + break;
> +
> default:
> return RESUME_HOST;
> }
> @@ -4149,8 +4162,7 @@ void kvmppc_setup_partition_table(struct kvm *kvm)
> __pa(kvm->arch.pgtable) | RADIX_PGD_INDEX_SIZE;
> dw1 = PATB_GR | kvm->arch.process_table;
> }
> -
> - mmu_partition_table_set_entry(kvm->arch.lpid, dw0, dw1);
> + kvmhv_set_ptbl_entry(kvm->arch.lpid, dw0, dw1);
> }
>
> /*
> @@ -4366,6 +4378,8 @@ static int kvmppc_core_init_vm_hv(struct kvm *kvm)
>
> kvmppc_alloc_host_rm_ops();
>
> + kvmhv_vm_nested_init(kvm);
> +
> /*
> * Since we don't flush the TLB when tearing down a VM,
> * and this lpid might have previously been used,
> @@ -4509,8 +4523,10 @@ static void kvmppc_core_destroy_vm_hv(struct kvm *kvm)
>
> /* Perform global invalidation and return lpid to the pool */
> if (cpu_has_feature(CPU_FTR_ARCH_300)) {
> + if (kvm->arch.nested_enable)
> + kvmhv_release_all_nested(kvm);
> kvm->arch.process_table = 0;
> - kvmppc_setup_partition_table(kvm);
> + kvmhv_set_ptbl_entry(kvm->arch.lpid, 0, 0);
> }
> kvmppc_free_lpid(kvm->arch.lpid);
>
> @@ -4981,6 +4997,10 @@ static int kvmppc_book3s_init_hv(void)
> if (r < 0)
> return -ENODEV;
>
> + r = kvmhv_nested_init();
> + if (r)
> + return r;
> +
> r = kvm_init_subcore_bitmap();
> if (r)
> return r;
> @@ -5039,6 +5059,7 @@ static void kvmppc_book3s_exit_hv(void)
> if (kvmppc_radix_possible())
> kvmppc_radix_exit();
> kvmppc_hv_ops = NULL;
> + kvmhv_nested_exit();
> }
>
> module_init(kvmppc_book3s_init_hv);
> diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
> new file mode 100644
> index 0000000..b5e4611
> --- /dev/null
> +++ b/arch/powerpc/kvm/book3s_hv_nested.c
> @@ -0,0 +1,298 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright IBM Corporation, 2018
> + * Authors Suraj Jitindar Singh <sjitindarsingh@gmail.com>
> + * Paul Mackerras <paulus@ozlabs.org>
> + *
> + * Description: KVM functions specific to running nested KVM-HV guests
> + * on Book3S processors (specifically POWER9 and later).
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/kvm_host.h>
> +
> +#include <asm/kvm_ppc.h>
> +#include <asm/mmu.h>
> +#include <asm/pgtable.h>
> +#include <asm/pgalloc.h>
> +
> +static struct patb_entry *pseries_partition_tb;
> +
> +static void kvmhv_update_ptbl_cache(struct kvm_nested_guest *gp);
> +
> +long kvmhv_nested_init(void)
> +{
> + long int ptb_order;
> + unsigned long ptcr;
> + long rc;
> +
> + if (!kvmhv_on_pseries())
> + return 0;
> + if (!radix_enabled())
> + return -ENODEV;
> +
> + /* find log base 2 of KVMPPC_NR_LPIDS, rounding up */
> + ptb_order = __ilog2(KVMPPC_NR_LPIDS - 1) + 1;
> + if (ptb_order < 8)
> + ptb_order = 8;
> + pseries_partition_tb = kmalloc(sizeof(struct patb_entry) << ptb_order,
> + GFP_KERNEL);
> + if (!pseries_partition_tb) {
> + pr_err("kvm-hv: failed to allocated nested partition table\n");
> + return -ENOMEM;
> + }
> +
> + ptcr = __pa(pseries_partition_tb) | (ptb_order - 8);
> + rc = plpar_hcall_norets(H_SET_PARTITION_TABLE, ptcr);
> + if (rc != H_SUCCESS) {
> + pr_err("kvm-hv: Parent hypervisor does not support nesting (rc=%ld)\n",
> + rc);
> + kfree(pseries_partition_tb);
> + pseries_partition_tb = NULL;
> + return -ENODEV;
> + }
> +
> + return 0;
> +}
> +
> +void kvmhv_nested_exit(void)
> +{
> + /*
> + * N.B. the kvmhv_on_pseries() test is there because it enables
> + * the compiler to remove the call to plpar_hcall_norets()
> + * when CONFIG_PPC_PSERIES=n.
> + */
> + if (kvmhv_on_pseries() && pseries_partition_tb) {
> + plpar_hcall_norets(H_SET_PARTITION_TABLE, 0);
> + kfree(pseries_partition_tb);
> + pseries_partition_tb = NULL;
> + }
> +}
> +
> +void kvmhv_set_ptbl_entry(unsigned int lpid, u64 dw0, u64 dw1)
> +{
> + if (cpu_has_feature(CPU_FTR_HVMODE)) {
> + mmu_partition_table_set_entry(lpid, dw0, dw1);
> + } else {
> + pseries_partition_tb[lpid].patb0 = cpu_to_be64(dw0);
> + pseries_partition_tb[lpid].patb1 = cpu_to_be64(dw1);
> + }
> +}
> +
> +static void kvmhv_set_nested_ptbl(struct kvm_nested_guest *gp)
> +{
> + unsigned long dw0;
> +
> + dw0 = PATB_HR | radix__get_tree_size() |
> + __pa(gp->shadow_pgtable) | RADIX_PGD_INDEX_SIZE;
> + kvmhv_set_ptbl_entry(gp->shadow_lpid, dw0, gp->process_table);
> +}
> +
> +void kvmhv_vm_nested_init(struct kvm *kvm)
> +{
> + kvm->arch.max_nested_lpid = -1;
> +}
> +
> +/*
> + * Handle the H_SET_PARTITION_TABLE hcall.
> + * r4 = guest real address of partition table + log_2(size) - 12
> + * (formatted as for the PTCR).
> + */
> +long kvmhv_set_partition_table(struct kvm_vcpu *vcpu)
> +{
> + struct kvm *kvm = vcpu->kvm;
> + unsigned long ptcr = kvmppc_get_gpr(vcpu, 4);
> + int srcu_idx;
> + long ret = H_SUCCESS;
> +
> + srcu_idx = srcu_read_lock(&kvm->srcu);
> + /*
> + * Limit the partition table to 4096 entries (because that's what
> + * hardware supports), and check the base address.
> + */
> + if ((ptcr & PRTS_MASK) > 12 - 8 ||
> + !kvm_is_visible_gfn(vcpu->kvm, (ptcr & PRTB_MASK) >> PAGE_SHIFT))
> + ret = H_PARAMETER;
> + srcu_read_unlock(&kvm->srcu, srcu_idx);
> + if (ret == H_SUCCESS)
> + kvm->arch.l1_ptcr = ptcr;
> + return ret;
> +}
> +
> +/*
> + * Reload the partition table entry for a guest.
> + * Caller must hold gp->tlb_lock.
> + */
> +static void kvmhv_update_ptbl_cache(struct kvm_nested_guest *gp)
> +{
> + int ret;
> + struct patb_entry ptbl_entry;
> + unsigned long ptbl_addr;
> + struct kvm *kvm = gp->l1_host;
> +
> + ret = -EFAULT;
> + ptbl_addr = (kvm->arch.l1_ptcr & PRTB_MASK) + (gp->l1_lpid << 4);
> + if (gp->l1_lpid < (1ul << ((kvm->arch.l1_ptcr & PRTS_MASK) + 8)))
> + ret = kvm_read_guest(kvm, ptbl_addr,
> + &ptbl_entry, sizeof(ptbl_entry));
> + if (ret) {
> + gp->l1_gr_to_hr = 0;
> + gp->process_table = 0;
> + } else {
> + gp->l1_gr_to_hr = be64_to_cpu(ptbl_entry.patb0);
> + gp->process_table = be64_to_cpu(ptbl_entry.patb1);
> + }
> + kvmhv_set_nested_ptbl(gp);
> +}
> +
> +struct kvm_nested_guest *kvmhv_alloc_nested(struct kvm *kvm, unsigned int lpid)
> +{
> + struct kvm_nested_guest *gp;
> + long shadow_lpid;
> +
> + gp = kzalloc(sizeof(*gp), GFP_KERNEL);
> + if (!gp)
> + return NULL;
> + gp->l1_host = kvm;
> + gp->l1_lpid = lpid;
> + mutex_init(&gp->tlb_lock);
> + gp->shadow_pgtable = pgd_alloc(kvm->mm);
> + if (!gp->shadow_pgtable)
> + goto out_free;
> + shadow_lpid = kvmppc_alloc_lpid();
> + if (shadow_lpid < 0)
> + goto out_free2;
> + gp->shadow_lpid = shadow_lpid;
> +
> + return gp;
> +
> + out_free2:
> + pgd_free(kvm->mm, gp->shadow_pgtable);
> + out_free:
> + kfree(gp);
> + return NULL;
> +}
> +
> +/*
> + * Free up any resources allocated for a nested guest.
> + */
> +static void kvmhv_release_nested(struct kvm_nested_guest *gp)
> +{
> + kvmhv_set_ptbl_entry(gp->shadow_lpid, 0, 0);
> + kvmppc_free_lpid(gp->shadow_lpid);
> + if (gp->shadow_pgtable)
> + pgd_free(gp->l1_host->mm, gp->shadow_pgtable);
> + kfree(gp);
> +}
> +
> +static void kvmhv_remove_nested(struct kvm_nested_guest *gp)
> +{
> + struct kvm *kvm = gp->l1_host;
> + int lpid = gp->l1_lpid;
> + long ref;
> +
> + spin_lock(&kvm->mmu_lock);
> + if (gp == kvm->arch.nested_guests[lpid]) {
> + kvm->arch.nested_guests[lpid] = NULL;
> + if (lpid == kvm->arch.max_nested_lpid) {
> + while (--lpid >= 0 && !kvm->arch.nested_guests[lpid])
> + ;
> + kvm->arch.max_nested_lpid = lpid;
> + }
> + --gp->refcnt;
> + }
> + ref = gp->refcnt;
> + spin_unlock(&kvm->mmu_lock);
> + if (ref == 0)
> + kvmhv_release_nested(gp);
> +}
> +
> +/*
> + * Free up all nested resources allocated for this guest.
> + */
> +void kvmhv_release_all_nested(struct kvm *kvm)
> +{
> + int i;
> + struct kvm_nested_guest *gp;
> + struct kvm_nested_guest *freelist = NULL;
> +
> + spin_lock(&kvm->mmu_lock);
> + for (i = 0; i <= kvm->arch.max_nested_lpid; i++) {
> + gp = kvm->arch.nested_guests[i];
> + if (!gp)
> + continue;
> + kvm->arch.nested_guests[i] = NULL;
> + if (--gp->refcnt == 0) {
> + gp->next = freelist;
> + freelist = gp;
> + }
> + }
> + kvm->arch.max_nested_lpid = -1;
> + spin_unlock(&kvm->mmu_lock);
> + while ((gp = freelist) != NULL) {
> + freelist = gp->next;
> + kvmhv_release_nested(gp);
> + }
> +}
> +
> +/* caller must hold gp->tlb_lock */
> +void kvmhv_flush_nested(struct kvm_nested_guest *gp)
> +{
> + kvmhv_update_ptbl_cache(gp);
> + if (gp->l1_gr_to_hr == 0)
> + kvmhv_remove_nested(gp);
> +}
> +
> +struct kvm_nested_guest *kvmhv_get_nested(struct kvm *kvm, int l1_lpid,
> + bool create)
> +{
> + struct kvm_nested_guest *gp, *newgp;
> +
> + if (l1_lpid >= KVM_MAX_NESTED_GUESTS ||
> + l1_lpid >= (1ul << ((kvm->arch.l1_ptcr & PRTS_MASK) + 12 - 4)))
> + return NULL;
> +
> + spin_lock(&kvm->mmu_lock);
> + gp = kvm->arch.nested_guests[l1_lpid];
> + if (gp)
> + ++gp->refcnt;
> + spin_unlock(&kvm->mmu_lock);
> +
> + if (gp || !create)
> + return gp;
> +
> + newgp = kvmhv_alloc_nested(kvm, l1_lpid);
> + if (!newgp)
> + return NULL;
> + spin_lock(&kvm->mmu_lock);
> + if (kvm->arch.nested_guests[l1_lpid]) {
> + /* someone else beat us to it */
> + gp = kvm->arch.nested_guests[l1_lpid];
> + } else {
> + kvm->arch.nested_guests[l1_lpid] = newgp;
> + ++newgp->refcnt;
> + gp = newgp;
> + newgp = NULL;
> + if (l1_lpid > kvm->arch.max_nested_lpid)
> + kvm->arch.max_nested_lpid = l1_lpid;
> + }
> + ++gp->refcnt;
> + spin_unlock(&kvm->mmu_lock);
> +
> + if (newgp)
> + kvmhv_release_nested(newgp);
> +
> + return gp;
> +}
> +
> +void kvmhv_put_nested(struct kvm_nested_guest *gp)
> +{
> + struct kvm *kvm = gp->l1_host;
> + long ref;
> +
> + spin_lock(&kvm->mmu_lock);
> + ref = --gp->refcnt;
> + spin_unlock(&kvm->mmu_lock);
> + if (ref == 0)
> + kvmhv_release_nested(gp);
> +}
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v4 06/32] KVM: PPC: Book3S HV: Simplify real-mode interrupt handling
From: David Gibson @ 2018-10-05 4:18 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <1538654169-15602-7-git-send-email-paulus@ozlabs.org>
[-- Attachment #1: Type: text/plain, Size: 12008 bytes --]
On Thu, Oct 04, 2018 at 09:55:43PM +1000, Paul Mackerras wrote:
> This streamlines the first part of the code that handles a hypervisor
> interrupt that occurred in the guest. With this, all of the real-mode
> handling that occurs is done before the "guest_exit_cont" label; once
> we get to that label we are committed to exiting to host virtual mode.
> Thus the machine check and HMI real-mode handling is moved before that
> label.
>
> Also, the code to handle external interrupts is moved out of line, as
> is the code that calls kvmppc_realmode_hmi_handler().
>
> Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
> arch/powerpc/kvm/book3s_hv_ras.c | 8 ++
> arch/powerpc/kvm/book3s_hv_rmhandlers.S | 220 ++++++++++++++++----------------
> 2 files changed, 119 insertions(+), 109 deletions(-)
>
> diff --git a/arch/powerpc/kvm/book3s_hv_ras.c b/arch/powerpc/kvm/book3s_hv_ras.c
> index b11043b..ee564b6 100644
> --- a/arch/powerpc/kvm/book3s_hv_ras.c
> +++ b/arch/powerpc/kvm/book3s_hv_ras.c
> @@ -331,5 +331,13 @@ long kvmppc_realmode_hmi_handler(void)
> } else {
> wait_for_tb_resync();
> }
> +
> + /*
> + * Reset tb_offset_applied so the guest exit code won't try
> + * to subtract the previous timebase offset from the timebase.
> + */
> + if (local_paca->kvm_hstate.kvm_vcore)
> + local_paca->kvm_hstate.kvm_vcore->tb_offset_applied = 0;
> +
> return 0;
> }
> diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> index 5b2ae34..772740d 100644
> --- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> +++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> @@ -1018,8 +1018,7 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_ARCH_300)
> no_xive:
> #endif /* CONFIG_KVM_XICS */
>
> -deliver_guest_interrupt:
> -kvmppc_cede_reentry: /* r4 = vcpu, r13 = paca */
> +deliver_guest_interrupt: /* r4 = vcpu, r13 = paca */
> /* Check if we can deliver an external or decrementer interrupt now */
> ld r0, VCPU_PENDING_EXC(r4)
> BEGIN_FTR_SECTION
> @@ -1269,18 +1268,26 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
> std r3, VCPU_CTR(r9)
> std r4, VCPU_XER(r9)
>
> -#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
> - /* For softpatch interrupt, go off and do TM instruction emulation */
> - cmpwi r12, BOOK3S_INTERRUPT_HV_SOFTPATCH
> - beq kvmppc_tm_emul
> -#endif
> + /* Save more register state */
> + mfdar r6
> + mfdsisr r7
> + std r6, VCPU_DAR(r9)
> + stw r7, VCPU_DSISR(r9)
>
> /* If this is a page table miss then see if it's theirs or ours */
> cmpwi r12, BOOK3S_INTERRUPT_H_DATA_STORAGE
> beq kvmppc_hdsi
> + std r6, VCPU_FAULT_DAR(r9)
> + stw r7, VCPU_FAULT_DSISR(r9)
> cmpwi r12, BOOK3S_INTERRUPT_H_INST_STORAGE
> beq kvmppc_hisi
>
> +#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
> + /* For softpatch interrupt, go off and do TM instruction emulation */
> + cmpwi r12, BOOK3S_INTERRUPT_HV_SOFTPATCH
> + beq kvmppc_tm_emul
> +#endif
> +
> /* See if this is a leftover HDEC interrupt */
> cmpwi r12,BOOK3S_INTERRUPT_HV_DECREMENTER
> bne 2f
> @@ -1303,7 +1310,7 @@ BEGIN_FTR_SECTION
> END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
> lbz r0, HSTATE_HOST_IPI(r13)
> cmpwi r0, 0
> - beq 4f
> + beq maybe_reenter_guest
> b guest_exit_cont
> 3:
> /* If it's a hypervisor facility unavailable interrupt, save HFSCR */
> @@ -1315,82 +1322,16 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
> 14:
> /* External interrupt ? */
> cmpwi r12, BOOK3S_INTERRUPT_EXTERNAL
> - bne+ guest_exit_cont
> -
> - /* External interrupt, first check for host_ipi. If this is
> - * set, we know the host wants us out so let's do it now
> - */
> - bl kvmppc_read_intr
> -
> - /*
> - * Restore the active volatile registers after returning from
> - * a C function.
> - */
> - ld r9, HSTATE_KVM_VCPU(r13)
> - li r12, BOOK3S_INTERRUPT_EXTERNAL
> -
> - /*
> - * kvmppc_read_intr return codes:
> - *
> - * Exit to host (r3 > 0)
> - * 1 An interrupt is pending that needs to be handled by the host
> - * Exit guest and return to host by branching to guest_exit_cont
> - *
> - * 2 Passthrough that needs completion in the host
> - * Exit guest and return to host by branching to guest_exit_cont
> - * However, we also set r12 to BOOK3S_INTERRUPT_HV_RM_HARD
> - * to indicate to the host to complete handling the interrupt
> - *
> - * Before returning to guest, we check if any CPU is heading out
> - * to the host and if so, we head out also. If no CPUs are heading
> - * check return values <= 0.
> - *
> - * Return to guest (r3 <= 0)
> - * 0 No external interrupt is pending
> - * -1 A guest wakeup IPI (which has now been cleared)
> - * In either case, we return to guest to deliver any pending
> - * guest interrupts.
> - *
> - * -2 A PCI passthrough external interrupt was handled
> - * (interrupt was delivered directly to guest)
> - * Return to guest to deliver any pending guest interrupts.
> - */
> -
> - cmpdi r3, 1
> - ble 1f
> -
> - /* Return code = 2 */
> - li r12, BOOK3S_INTERRUPT_HV_RM_HARD
> - stw r12, VCPU_TRAP(r9)
> - b guest_exit_cont
> -
> -1: /* Return code <= 1 */
> - cmpdi r3, 0
> - bgt guest_exit_cont
> -
> - /* Return code <= 0 */
> -4: ld r5, HSTATE_KVM_VCORE(r13)
> - lwz r0, VCORE_ENTRY_EXIT(r5)
> - cmpwi r0, 0x100
> - mr r4, r9
> - blt deliver_guest_interrupt
> -
> -guest_exit_cont: /* r9 = vcpu, r12 = trap, r13 = paca */
> - /* Save more register state */
> - mfdar r6
> - mfdsisr r7
> - std r6, VCPU_DAR(r9)
> - stw r7, VCPU_DSISR(r9)
> - /* don't overwrite fault_dar/fault_dsisr if HDSI */
> - cmpwi r12,BOOK3S_INTERRUPT_H_DATA_STORAGE
> - beq mc_cont
> - std r6, VCPU_FAULT_DAR(r9)
> - stw r7, VCPU_FAULT_DSISR(r9)
> -
> + beq kvmppc_guest_external
> /* See if it is a machine check */
> cmpwi r12, BOOK3S_INTERRUPT_MACHINE_CHECK
> beq machine_check_realmode
> -mc_cont:
> + /* Or a hypervisor maintenance interrupt */
> + cmpwi r12, BOOK3S_INTERRUPT_HMI
> + beq hmi_realmode
> +
> +guest_exit_cont: /* r9 = vcpu, r12 = trap, r13 = paca */
> +
> #ifdef CONFIG_KVM_BOOK3S_HV_EXIT_TIMING
> addi r3, r9, VCPU_TB_RMEXIT
> mr r4, r9
> @@ -1821,24 +1762,6 @@ BEGIN_FTR_SECTION
> mtspr SPRN_DPDES, r8
> END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
>
> - /* If HMI, call kvmppc_realmode_hmi_handler() */
> - lwz r12, STACK_SLOT_TRAP(r1)
> - cmpwi r12, BOOK3S_INTERRUPT_HMI
> - bne 27f
> - bl kvmppc_realmode_hmi_handler
> - nop
> - cmpdi r3, 0
> - /*
> - * At this point kvmppc_realmode_hmi_handler may have resync-ed
> - * the TB, and if it has, we must not subtract the guest timebase
> - * offset from the timebase. So, skip it.
> - *
> - * Also, do not call kvmppc_subcore_exit_guest() because it has
> - * been invoked as part of kvmppc_realmode_hmi_handler().
> - */
> - beq 30f
> -
> -27:
> /* Subtract timebase offset from timebase */
> ld r8, VCORE_TB_OFFSET_APPL(r5)
> cmpdi r8,0
> @@ -1856,7 +1779,16 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
> addis r8,r8,0x100 /* if so, increment upper 40 bits */
> mtspr SPRN_TBU40,r8
>
> -17: bl kvmppc_subcore_exit_guest
> +17:
> + /*
> + * If this is an HMI, we called kvmppc_realmode_hmi_handler
> + * above, which may or may not have already called
> + * kvmppc_subcore_exit_guest. Fortunately, all that
> + * kvmppc_subcore_exit_guest does is clear a flag, so calling
> + * it again here is benign even if kvmppc_realmode_hmi_handler
> + * has already called it.
> + */
> + bl kvmppc_subcore_exit_guest
> nop
> 30: ld r5,HSTATE_KVM_VCORE(r13)
> ld r4,VCORE_KVM(r5) /* pointer to struct kvm */
> @@ -1910,6 +1842,67 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
> mtlr r0
> blr
>
> +kvmppc_guest_external:
> + /* External interrupt, first check for host_ipi. If this is
> + * set, we know the host wants us out so let's do it now
> + */
> + bl kvmppc_read_intr
> +
> + /*
> + * Restore the active volatile registers after returning from
> + * a C function.
> + */
> + ld r9, HSTATE_KVM_VCPU(r13)
> + li r12, BOOK3S_INTERRUPT_EXTERNAL
> +
> + /*
> + * kvmppc_read_intr return codes:
> + *
> + * Exit to host (r3 > 0)
> + * 1 An interrupt is pending that needs to be handled by the host
> + * Exit guest and return to host by branching to guest_exit_cont
> + *
> + * 2 Passthrough that needs completion in the host
> + * Exit guest and return to host by branching to guest_exit_cont
> + * However, we also set r12 to BOOK3S_INTERRUPT_HV_RM_HARD
> + * to indicate to the host to complete handling the interrupt
> + *
> + * Before returning to guest, we check if any CPU is heading out
> + * to the host and if so, we head out also. If no CPUs are heading
> + * check return values <= 0.
> + *
> + * Return to guest (r3 <= 0)
> + * 0 No external interrupt is pending
> + * -1 A guest wakeup IPI (which has now been cleared)
> + * In either case, we return to guest to deliver any pending
> + * guest interrupts.
> + *
> + * -2 A PCI passthrough external interrupt was handled
> + * (interrupt was delivered directly to guest)
> + * Return to guest to deliver any pending guest interrupts.
> + */
> +
> + cmpdi r3, 1
> + ble 1f
> +
> + /* Return code = 2 */
> + li r12, BOOK3S_INTERRUPT_HV_RM_HARD
> + stw r12, VCPU_TRAP(r9)
> + b guest_exit_cont
> +
> +1: /* Return code <= 1 */
> + cmpdi r3, 0
> + bgt guest_exit_cont
> +
> + /* Return code <= 0 */
> +maybe_reenter_guest:
> + ld r5, HSTATE_KVM_VCORE(r13)
> + lwz r0, VCORE_ENTRY_EXIT(r5)
> + cmpwi r0, 0x100
> + mr r4, r9
> + blt deliver_guest_interrupt
> + b guest_exit_cont
> +
> #ifdef CONFIG_PPC_TRANSACTIONAL_MEM
> /*
> * Softpatch interrupt for transactional memory emulation cases
> @@ -2685,13 +2678,7 @@ END_FTR_SECTION(CPU_FTR_TM | CPU_FTR_P9_TM_HV_ASSIST, 0)
> mr r9, r4
> cmpdi r3, 0
> bgt guest_exit_cont
> -
> - /* see if any other thread is already exiting */
> - lwz r0,VCORE_ENTRY_EXIT(r5)
> - cmpwi r0,0x100
> - bge guest_exit_cont
> -
> - b kvmppc_cede_reentry /* if not go back to guest */
> + b maybe_reenter_guest
>
> /* cede when already previously prodded case */
> kvm_cede_prodded:
> @@ -2758,12 +2745,12 @@ machine_check_realmode:
> */
> ld r11, VCPU_MSR(r9)
> rldicl. r0, r11, 64-MSR_HV_LG, 63 /* check if it happened in HV mode */
> - bne mc_cont /* if so, exit to host */
> + bne guest_exit_cont /* if so, exit to host */
> /* Check if guest is capable of handling NMI exit */
> ld r10, VCPU_KVM(r9)
> lbz r10, KVM_FWNMI(r10)
> cmpdi r10, 1 /* FWNMI capable? */
> - beq mc_cont /* if so, exit with KVM_EXIT_NMI. */
> + beq guest_exit_cont /* if so, exit with KVM_EXIT_NMI. */
>
> /* if not, fall through for backward compatibility. */
> andi. r10, r11, MSR_RI /* check for unrecoverable exception */
> @@ -2777,6 +2764,21 @@ machine_check_realmode:
> 2: b fast_interrupt_c_return
>
> /*
> + * Call C code to handle a HMI in real mode.
> + * Only the primary thread does the call, secondary threads are handled
> + * by calling hmi_exception_realmode() after kvmppc_hv_entry returns.
> + * r9 points to the vcpu on entry
> + */
> +hmi_realmode:
> + lbz r0, HSTATE_PTID(r13)
> + cmpwi r0, 0
> + bne guest_exit_cont
> + bl kvmppc_realmode_hmi_handler
> + ld r9, HSTATE_KVM_VCPU(r13)
> + li r12, BOOK3S_INTERRUPT_HMI
> + b guest_exit_cont
> +
> +/*
> * Check the reason we woke from nap, and take appropriate action.
> * Returns (in r3):
> * 0 if nothing needs to be done
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v4 25/32] KVM: PPC: Book3S HV: Invalidate TLB when nested vcpu moves physical cpu
From: Paul Mackerras @ 2018-10-05 4:23 UTC (permalink / raw)
To: David Gibson; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <20181005040908.GK7004@umbus.fritz.box>
On Fri, Oct 05, 2018 at 02:09:08PM +1000, David Gibson wrote:
> On Thu, Oct 04, 2018 at 09:56:02PM +1000, Paul Mackerras wrote:
> > From: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
> >
> > This is only done at level 0, since only level 0 knows which physical
> > CPU a vcpu is running on. This does for nested guests what L0 already
> > did for its own guests, which is to flush the TLB on a pCPU when it
> > goes to run a vCPU there, and there is another vCPU in the same VM
> > which previously ran on this pCPU and has now started to run on another
> > pCPU. This is to handle the situation where the other vCPU touched
> > a mapping, moved to another pCPU and did a tlbiel (local-only tlbie)
> > on that new pCPU and thus left behind a stale TLB entry on this pCPU.
> >
> > This introduces a limit on the the vcpu_token values used in the
> > H_ENTER_NESTED hcall -- they must now be less than NR_CPUS.
>
> This does make the vcpu tokens no longer entirely opaque to the L0.
> It works for now, because the only L1 is Linux and we know basically
> how it allocates those tokens. Eventually we probably want some way
> to either remove this restriction or to advertise the limit to the L1.
Right, we could use something like a hash table and have it be
basically just as efficient as the array when the set of IDs is dense
while also handling arbitrary ID values. (We'd have to make sure that
L1 couldn't trigger unbounded memory consumption in L0, though.)
Paul.
^ permalink raw reply
* Re: [PATCH v4 25/32] KVM: PPC: Book3S HV: Invalidate TLB when nested vcpu moves physical cpu
From: David Gibson @ 2018-10-05 4:54 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <20181005042350.GA3309@fergus>
[-- Attachment #1: Type: text/plain, Size: 2086 bytes --]
On Fri, Oct 05, 2018 at 02:23:50PM +1000, Paul Mackerras wrote:
> On Fri, Oct 05, 2018 at 02:09:08PM +1000, David Gibson wrote:
> > On Thu, Oct 04, 2018 at 09:56:02PM +1000, Paul Mackerras wrote:
> > > From: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
> > >
> > > This is only done at level 0, since only level 0 knows which physical
> > > CPU a vcpu is running on. This does for nested guests what L0 already
> > > did for its own guests, which is to flush the TLB on a pCPU when it
> > > goes to run a vCPU there, and there is another vCPU in the same VM
> > > which previously ran on this pCPU and has now started to run on another
> > > pCPU. This is to handle the situation where the other vCPU touched
> > > a mapping, moved to another pCPU and did a tlbiel (local-only tlbie)
> > > on that new pCPU and thus left behind a stale TLB entry on this pCPU.
> > >
> > > This introduces a limit on the the vcpu_token values used in the
> > > H_ENTER_NESTED hcall -- they must now be less than NR_CPUS.
> >
> > This does make the vcpu tokens no longer entirely opaque to the L0.
> > It works for now, because the only L1 is Linux and we know basically
> > how it allocates those tokens. Eventually we probably want some way
> > to either remove this restriction or to advertise the limit to the L1.
>
> Right, we could use something like a hash table and have it be
> basically just as efficient as the array when the set of IDs is dense
> while also handling arbitrary ID values. (We'd have to make sure that
> L1 couldn't trigger unbounded memory consumption in L0, though.)
Another approach would be to sacifice some performance for L0
simplicity: when an L1 vCPU changes pCPU, flush all the nested LPIDs
associated with that L1. When an L2 vCPU changes L1 vCPU (and
therefore, indirectly pCPU), the L1 would be responsible for flushing
it.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v4 25/32] KVM: PPC: Book3S HV: Invalidate TLB when nested vcpu moves physical cpu
From: Paul Mackerras @ 2018-10-05 5:32 UTC (permalink / raw)
To: David Gibson; +Cc: linuxppc-dev, kvm-ppc, kvm
In-Reply-To: <20181005045428.GO7004@umbus.fritz.box>
On Fri, Oct 05, 2018 at 02:54:28PM +1000, David Gibson wrote:
> On Fri, Oct 05, 2018 at 02:23:50PM +1000, Paul Mackerras wrote:
> > On Fri, Oct 05, 2018 at 02:09:08PM +1000, David Gibson wrote:
> > > On Thu, Oct 04, 2018 at 09:56:02PM +1000, Paul Mackerras wrote:
> > > > From: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
> > > >
> > > > This is only done at level 0, since only level 0 knows which physical
> > > > CPU a vcpu is running on. This does for nested guests what L0 already
> > > > did for its own guests, which is to flush the TLB on a pCPU when it
> > > > goes to run a vCPU there, and there is another vCPU in the same VM
> > > > which previously ran on this pCPU and has now started to run on another
> > > > pCPU. This is to handle the situation where the other vCPU touched
> > > > a mapping, moved to another pCPU and did a tlbiel (local-only tlbie)
> > > > on that new pCPU and thus left behind a stale TLB entry on this pCPU.
> > > >
> > > > This introduces a limit on the the vcpu_token values used in the
> > > > H_ENTER_NESTED hcall -- they must now be less than NR_CPUS.
> > >
> > > This does make the vcpu tokens no longer entirely opaque to the L0.
> > > It works for now, because the only L1 is Linux and we know basically
> > > how it allocates those tokens. Eventually we probably want some way
> > > to either remove this restriction or to advertise the limit to the L1.
> >
> > Right, we could use something like a hash table and have it be
> > basically just as efficient as the array when the set of IDs is dense
> > while also handling arbitrary ID values. (We'd have to make sure that
> > L1 couldn't trigger unbounded memory consumption in L0, though.)
>
> Another approach would be to sacifice some performance for L0
> simplicity: when an L1 vCPU changes pCPU, flush all the nested LPIDs
> associated with that L1. When an L2 vCPU changes L1 vCPU (and
> therefore, indirectly pCPU), the L1 would be responsible for flushing
> it.
That was one of the approaches I considered initially, but it has
complexities that aren't apparent, and it could be quite inefficient
for a guest with a lot of nested guests. For a start you have to
provide a way for L1 to flush the TLB for another LPID, which guests
can't do themselves (it's a hypervisor privileged operation). Then
there's the fact that it's not the pCPU where the moving vCPU has
moved to that needs the flush, it's the pCPU that it moved from (where
presumably something else is now running). All in all, the simplest
solution was to have L0 do it, because L0 knows unambiguously the real
physical CPU where any given vCPU last ran.
Paul.
^ permalink raw reply
* Re: linux-next: Tree for Oct 4
From: Stephen Rothwell @ 2018-10-05 5:39 UTC (permalink / raw)
To: Guenter Roeck
Cc: Linus PPC, Linux Kernel Mailing List,
open list:INTEL IOMMU (VT-d), Linux-Next, Geert Uytterhoeven,
Greg KH, Alexander Duyck, Robin Murphy, Christoph Hellwig
In-Reply-To: <01c4a73c-7901-2fbb-8f87-95ff21dc5126@roeck-us.net>
[-- Attachment #1: Type: text/plain, Size: 283 bytes --]
Hi Guenter,
On Thu, 4 Oct 2018 18:33:02 -0700 Guenter Roeck <linux@roeck-us.net> wrote:
>
> Most of the boot failures are hopefully fixed with
> https://lore.kernel.org/patchwork/patch/995254/
I have added that commit to linux-next today.
--
Cheers,
Stephen Rothwell
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* linux-next: build failure after merge of the akpm tree
From: Stephen Rothwell @ 2018-10-05 6:14 UTC (permalink / raw)
To: Andrew Morton, Michael Ellerman, Benjamin Herrenschmidt, PowerPC
Cc: Linux-Next Mailing List, Linux Kernel Mailing List
[-- Attachment #1: Type: text/plain, Size: 1272 bytes --]
Hi Andrew,
After merging the akpm tree, today's linux-next build (powerpc
ppc64_defconfig) failed like this:
arch/powerpc/kernel/setup-common.c:36:10: fatal error: linux/bootmem.h: No such file or directory
#include <linux/bootmem.h>
^~~~~~~~~~~~~~~~~
Caused by commit
49353a51a758 ("mm: remove include/linux/bootmem.h")
interacting with commit
d90fe2acd9b2 ("powerpc: Wire up memtest")
from the powerpc tree.
I applied the following patch for today:
From: Stephen Rothwell <sfr@canb.auug.org.au>
Date: Fri, 5 Oct 2018 16:09:34 +1000
Subject: [PATCH] powerpc: fix up for removal of linux/bootmem.h
Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au>
---
arch/powerpc/kernel/setup-common.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c
index 2b56d1f30387..93ee3703b42f 100644
--- a/arch/powerpc/kernel/setup-common.c
+++ b/arch/powerpc/kernel/setup-common.c
@@ -33,7 +33,6 @@
#include <linux/serial_8250.h>
#include <linux/percpu.h>
#include <linux/memblock.h>
-#include <linux/bootmem.h>
#include <linux/of_platform.h>
#include <linux/hugetlb.h>
#include <asm/debugfs.h>
--
2.18.0
--
Cheers,
Stephen Rothwell
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply related
* Re: [PATCH v3 3/6] mm/memory_hotplug: fix online/offline_pages called w.o. mem_hotplug_lock
From: Oscar Salvador @ 2018-10-05 7:07 UTC (permalink / raw)
To: David Hildenbrand
Cc: Kate Stewart, Michal Hocko, linux-doc, Heiko Carstens, linux-mm,
Paul Mackerras, Rashmica Gupta, K. Y. Srinivasan, Thomas Gleixner,
Michael Neuling, Stephen Hemminger, linux-acpi, xen-devel,
Len Brown, Pavel Tatashin, Haiyang Zhang, Dan Williams,
YASUAKI ISHIMATSU, Boris Ostrovsky, Vlastimil Babka,
Oscar Salvador, Juergen Gross, Mathieu Malaterre,
Greg Kroah-Hartman, Rafael J. Wysocki, linux-kernel,
Philippe Ombredanne, Martin Schwidefsky, devel, Andrew Morton,
linuxppc-dev
In-Reply-To: <20180927092554.13567-4-david@redhat.com>
On Thu, Sep 27, 2018 at 11:25:51AM +0200, David Hildenbrand wrote:
> Reviewed-by: Pavel Tatashin <pavel.tatashin@microsoft.com>
> Reviewed-by: Rashmica Gupta <rashmica.g@gmail.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Oscar Salvador <osalvador@suse.de>
--
Oscar Salvador
SUSE L3
^ permalink raw reply
* Re: [PATCH v3 6/6] memory-hotplug.txt: Add some details about locking internals
From: Oscar Salvador @ 2018-10-05 7:08 UTC (permalink / raw)
To: David Hildenbrand
Cc: Jonathan Corbet, Michal Hocko, linux-acpi, linux-doc,
linux-kernel, linux-mm, devel, xen-devel, Andrew Morton,
linuxppc-dev
In-Reply-To: <20180927092554.13567-7-david@redhat.com>
On Thu, Sep 27, 2018 at 11:25:54AM +0200, David Hildenbrand wrote:
> Cc: Jonathan Corbet <corbet@lwn.net>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Reviewed-by: Pavel Tatashin <pavel.tatashin@microsoft.com>
> Reviewed-by: Rashmica Gupta <rashmica.g@gmail.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Oscar Salvador <osalvador@suse.de>
--
Oscar Salvador
SUSE L3
^ permalink raw reply
* Re: [PATCH v3 1/6] mm/memory_hotplug: make remove_memory() take the device_hotplug_lock
From: Oscar Salvador @ 2018-10-05 7:15 UTC (permalink / raw)
To: David Hildenbrand
Cc: Michal Hocko, linux-doc, linux-mm, Paul Mackerras, Rashmica Gupta,
Michael Neuling, Pavel Tatashin, linux-acpi, xen-devel, Len Brown,
YASUAKI ISHIMATSU, Nathan Fontenot, Dan Williams, Joonsoo Kim,
Vlastimil Babka, Oscar Salvador, Mathieu Malaterre,
Greg Kroah-Hartman, Rafael J. Wysocki, linux-kernel, John Allen,
devel, Andrew Morton, linuxppc-dev
In-Reply-To: <20180927092554.13567-2-david@redhat.com>
On Thu, Sep 27, 2018 at 11:25:49AM +0200, David Hildenbrand wrote:
> Reviewed-by: Pavel Tatashin <pavel.tatashin@microsoft.com>
> Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> Reviewed-by: Rashmica Gupta <rashmica.g@gmail.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Oscar Salvador <osalvador@suse.de>
--
Oscar Salvador
SUSE L3
^ permalink raw reply
* Re: [PATCH v3 2/6] mm/memory_hotplug: make add_memory() take the device_hotplug_lock
From: Oscar Salvador @ 2018-10-05 7:16 UTC (permalink / raw)
To: David Hildenbrand
Cc: Michal Hocko, linux-doc, linux-mm, Paul Mackerras, Dan Williams,
linux-acpi, xen-devel, Len Brown, Pavel Tatashin,
YASUAKI ISHIMATSU, Nathan Fontenot, Boris Ostrovsky, Joonsoo Kim,
Vlastimil Babka, Oscar Salvador, Juergen Gross, Mathieu Malaterre,
Greg Kroah-Hartman, Rafael J. Wysocki, linux-kernel, John Allen,
devel, Andrew Morton, linuxppc-dev
In-Reply-To: <20180927092554.13567-3-david@redhat.com>
On Thu, Sep 27, 2018 at 11:25:50AM +0200, David Hildenbrand wrote:
> Reviewed-by: Pavel Tatashin <pavel.tatashin@microsoft.com>
> Reviewed-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> Reviewed-by: Rashmica Gupta <rashmica.g@gmail.com>
> Signed-off-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Oscar Salvador <osalvador@suse.de>
--
Oscar Salvador
SUSE L3
^ permalink raw reply
* Re: [PATCH] dma-direct: Fix return value of dma_direct_supported
From: Christoph Hellwig @ 2018-10-05 7:17 UTC (permalink / raw)
To: Alexander Duyck
Cc: Robin Murphy, LKML, open list:INTEL IOMMU (VT-d), Greg KH,
alexander.h.duyck,
open list:LINUX FOR POWERPC (32-BIT AND 64-BIT),
Christoph Hellwig, Guenter Roeck
In-Reply-To: <CAKgT0UfbAokpR=-xrfNjeYMroEDH4HtPgM6qK_nWmBqK3QOf6A@mail.gmail.com>
On Thu, Oct 04, 2018 at 08:13:26AM -0700, Alexander Duyck wrote:
> Thanks for the review.
>
> - Alex
>
> P.S. It looks like I forgot to add Christoph to the original mail
> since I had just copied the To and Cc from the original submission, so
> I added him to the Cc for this.
Yes, there was some oddness with replies appearing out of nowhere that
I couldn't understand in my jetlagged state yesterday. Now thay I've
caught up on iommu list traffic it all makes sense now. I've applied
this with an Acked-by for Robin added based on his reply.
^ permalink raw reply
* [PATCH v5 0/9] powerpc: Switch to CONFIG_THREAD_INFO_IN_TASK
From: Christophe Leroy @ 2018-10-05 7:32 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
The purpose of this serie is to activate CONFIG_THREAD_INFO_IN_TASK which
moves the thread_info into task_struct.
Moving thread_info into task_struct has the following advantages:
- It protects thread_info from corruption in the case of stack
overflows.
- Its address is harder to determine if stack addresses are
leaked, making a number of attacks more difficult.
Changes since v4:
- Fixed a build failure on 32bits SMP when include/generated/asm-offsets.h is not
already existing, was due to spaces instead of a tab in the Makefile
Changes since RFC v3: (based on Nick's review)
- Renamed task_size.h to task_size_user64.h to better relate to what it contains.
- Handling of the isolation of thread_info cpu field inside CONFIG_SMP #ifdefs moved to a separate patch.
- Removed CURRENT_THREAD_INFO macro completely.
- Added a guard in asm/smp.h to avoid build failure before _TASK_CPU is defined.
- Added a patch at the end to rename 'tp' pointers to 'sp' pointers
- Renamed 'tp' into 'sp' pointers in preparation patch when relevant
- Fixed a few commit logs
- Fixed checkpatch report.
Changes since RFC v2:
- Removed the modification of names in asm-offsets
- Created a rule in arch/powerpc/Makefile to append the offset of current->cpu in CFLAGS
- Modified asm/smp.h to use the offset set in CFLAGS
- Squashed the renaming of THREAD_INFO to TASK_STACK in the preparation patch
- Moved the modification of current_pt_regs in the patch activating CONFIG_THREAD_INFO_IN_TASK
Changes since RFC v1:
- Removed the first patch which was modifying header inclusion order in timer
- Modified some names in asm-offsets to avoid conflicts when including asm-offsets in C files
- Modified asm/smp.h to avoid having to include linux/sched.h (using asm-offsets instead)
- Moved some changes from the activation patch to the preparation patch.
Christophe Leroy (9):
book3s/64: avoid circular header inclusion in mmu-hash.h
powerpc: Only use task_struct 'cpu' field on SMP
powerpc: Prepare for moving thread_info into task_struct
powerpc: Activate CONFIG_THREAD_INFO_IN_TASK
powerpc: regain entire stack space
powerpc: 'current_set' is now a table of task_struct pointers
powerpc/32: Remove CURRENT_THREAD_INFO and rename TI_CPU
powerpc/64: Remove CURRENT_THREAD_INFO
powerpc: clean stack pointers naming
arch/powerpc/Kconfig | 1 +
arch/powerpc/Makefile | 8 ++-
arch/powerpc/include/asm/asm-prototypes.h | 4 +-
arch/powerpc/include/asm/book3s/64/mmu-hash.h | 2 +-
arch/powerpc/include/asm/exception-64s.h | 4 +-
arch/powerpc/include/asm/irq.h | 14 ++---
arch/powerpc/include/asm/livepatch.h | 2 +-
arch/powerpc/include/asm/processor.h | 39 +------------
arch/powerpc/include/asm/ptrace.h | 2 +-
arch/powerpc/include/asm/reg.h | 2 +-
arch/powerpc/include/asm/smp.h | 17 +++++-
arch/powerpc/include/asm/task_size_user64.h | 42 ++++++++++++++
arch/powerpc/include/asm/thread_info.h | 19 -------
arch/powerpc/kernel/asm-offsets.c | 10 ++--
arch/powerpc/kernel/entry_32.S | 66 ++++++++--------------
arch/powerpc/kernel/entry_64.S | 12 ++--
arch/powerpc/kernel/epapr_hcalls.S | 5 +-
arch/powerpc/kernel/exceptions-64e.S | 13 +----
arch/powerpc/kernel/exceptions-64s.S | 2 +-
arch/powerpc/kernel/head_32.S | 14 ++---
arch/powerpc/kernel/head_40x.S | 4 +-
arch/powerpc/kernel/head_44x.S | 8 +--
arch/powerpc/kernel/head_64.S | 1 +
arch/powerpc/kernel/head_8xx.S | 2 +-
arch/powerpc/kernel/head_booke.h | 12 +---
arch/powerpc/kernel/head_fsl_booke.S | 16 +++---
arch/powerpc/kernel/idle_6xx.S | 8 +--
arch/powerpc/kernel/idle_book3e.S | 2 +-
arch/powerpc/kernel/idle_e500.S | 8 +--
arch/powerpc/kernel/idle_power4.S | 2 +-
arch/powerpc/kernel/irq.c | 77 +++++---------------------
arch/powerpc/kernel/kgdb.c | 28 ----------
arch/powerpc/kernel/machine_kexec_64.c | 6 +-
arch/powerpc/kernel/misc_32.S | 17 +++---
arch/powerpc/kernel/process.c | 15 ++---
arch/powerpc/kernel/setup-common.c | 2 +-
arch/powerpc/kernel/setup_32.c | 15 ++---
arch/powerpc/kernel/setup_64.c | 41 ++++----------
arch/powerpc/kernel/smp.c | 16 +++---
arch/powerpc/kernel/trace/ftrace_64_mprofile.S | 6 +-
arch/powerpc/kvm/book3s_hv_hmi.c | 1 +
arch/powerpc/mm/hash_low_32.S | 14 ++---
arch/powerpc/sysdev/6xx-suspend.S | 5 +-
arch/powerpc/xmon/xmon.c | 2 +-
44 files changed, 224 insertions(+), 362 deletions(-)
create mode 100644 arch/powerpc/include/asm/task_size_user64.h
--
2.13.3
^ permalink raw reply
* [PATCH v5 1/9] book3s/64: avoid circular header inclusion in mmu-hash.h
From: Christophe Leroy @ 2018-10-05 7:32 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
In-Reply-To: <cover.1538687073.git.christophe.leroy@c-s.fr>
When activating CONFIG_THREAD_INFO_IN_TASK, linux/sched.h
includes asm/current.h. This generates a circular dependency.
To avoid that, asm/processor.h shall not be included in mmu-hash.h
In order to do that, this patch moves into a new header called
asm/task_size_user64.h the information from asm/processor.h required
by mmu-hash.h
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/book3s/64/mmu-hash.h | 2 +-
arch/powerpc/include/asm/processor.h | 34 +---------------------
arch/powerpc/include/asm/task_size_user64.h | 42 +++++++++++++++++++++++++++
arch/powerpc/kvm/book3s_hv_hmi.c | 1 +
4 files changed, 45 insertions(+), 34 deletions(-)
create mode 100644 arch/powerpc/include/asm/task_size_user64.h
diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h
index e0e4ce8f77d6..02955d867067 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h
@@ -23,7 +23,7 @@
*/
#include <asm/book3s/64/pgtable.h>
#include <asm/bug.h>
-#include <asm/processor.h>
+#include <asm/task_size_user64.h>
#include <asm/cpu_has_feature.h>
/*
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index 52fadded5c1e..13589274fe9b 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -101,40 +101,8 @@ void release_thread(struct task_struct *);
#endif
#ifdef CONFIG_PPC64
-/*
- * 64-bit user address space can have multiple limits
- * For now supported values are:
- */
-#define TASK_SIZE_64TB (0x0000400000000000UL)
-#define TASK_SIZE_128TB (0x0000800000000000UL)
-#define TASK_SIZE_512TB (0x0002000000000000UL)
-#define TASK_SIZE_1PB (0x0004000000000000UL)
-#define TASK_SIZE_2PB (0x0008000000000000UL)
-/*
- * With 52 bits in the address we can support
- * upto 4PB of range.
- */
-#define TASK_SIZE_4PB (0x0010000000000000UL)
-/*
- * For now 512TB is only supported with book3s and 64K linux page size.
- */
-#if defined(CONFIG_PPC_BOOK3S_64) && defined(CONFIG_PPC_64K_PAGES)
-/*
- * Max value currently used:
- */
-#define TASK_SIZE_USER64 TASK_SIZE_4PB
-#define DEFAULT_MAP_WINDOW_USER64 TASK_SIZE_128TB
-#define TASK_CONTEXT_SIZE TASK_SIZE_512TB
-#else
-#define TASK_SIZE_USER64 TASK_SIZE_64TB
-#define DEFAULT_MAP_WINDOW_USER64 TASK_SIZE_64TB
-/*
- * We don't need to allocate extended context ids for 4K page size, because
- * we limit the max effective address on this config to 64TB.
- */
-#define TASK_CONTEXT_SIZE TASK_SIZE_64TB
-#endif
+#include <asm/task_size_user64.h>
/*
* 32-bit user address space is 4GB - 1 page
diff --git a/arch/powerpc/include/asm/task_size_user64.h b/arch/powerpc/include/asm/task_size_user64.h
new file mode 100644
index 000000000000..a4043075864b
--- /dev/null
+++ b/arch/powerpc/include/asm/task_size_user64.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_POWERPC_TASK_SIZE_USER64_H
+#define _ASM_POWERPC_TASK_SIZE_USER64_H
+
+#ifdef CONFIG_PPC64
+/*
+ * 64-bit user address space can have multiple limits
+ * For now supported values are:
+ */
+#define TASK_SIZE_64TB (0x0000400000000000UL)
+#define TASK_SIZE_128TB (0x0000800000000000UL)
+#define TASK_SIZE_512TB (0x0002000000000000UL)
+#define TASK_SIZE_1PB (0x0004000000000000UL)
+#define TASK_SIZE_2PB (0x0008000000000000UL)
+/*
+ * With 52 bits in the address we can support
+ * upto 4PB of range.
+ */
+#define TASK_SIZE_4PB (0x0010000000000000UL)
+
+/*
+ * For now 512TB is only supported with book3s and 64K linux page size.
+ */
+#if defined(CONFIG_PPC_BOOK3S_64) && defined(CONFIG_PPC_64K_PAGES)
+/*
+ * Max value currently used:
+ */
+#define TASK_SIZE_USER64 TASK_SIZE_4PB
+#define DEFAULT_MAP_WINDOW_USER64 TASK_SIZE_128TB
+#define TASK_CONTEXT_SIZE TASK_SIZE_512TB
+#else
+#define TASK_SIZE_USER64 TASK_SIZE_64TB
+#define DEFAULT_MAP_WINDOW_USER64 TASK_SIZE_64TB
+/*
+ * We don't need to allocate extended context ids for 4K page size, because
+ * we limit the max effective address on this config to 64TB.
+ */
+#define TASK_CONTEXT_SIZE TASK_SIZE_64TB
+#endif
+
+#endif /* CONFIG_PPC64 */
+#endif /* _ASM_POWERPC_TASK_SIZE_USER64_H */
diff --git a/arch/powerpc/kvm/book3s_hv_hmi.c b/arch/powerpc/kvm/book3s_hv_hmi.c
index e3f738eb1cac..64b5011475c7 100644
--- a/arch/powerpc/kvm/book3s_hv_hmi.c
+++ b/arch/powerpc/kvm/book3s_hv_hmi.c
@@ -24,6 +24,7 @@
#include <linux/compiler.h>
#include <asm/paca.h>
#include <asm/hmi.h>
+#include <asm/processor.h>
void wait_for_subcore_guest_exit(void)
{
--
2.13.3
^ permalink raw reply related
* [PATCH v5 2/9] powerpc: Only use task_struct 'cpu' field on SMP
From: Christophe Leroy @ 2018-10-05 7:32 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
In-Reply-To: <cover.1538687073.git.christophe.leroy@c-s.fr>
When moving to CONFIG_THREAD_INFO_IN_TASK, the thread_info 'cpu' field
gets moved into task_struct and only defined when CONFIG_SMP is set.
This patch ensures that TI_CPU is only used when CONFIG_SMP is set and
that task_struct 'cpu' field is not used directly out of SMP code.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/head_fsl_booke.S | 2 ++
arch/powerpc/kernel/misc_32.S | 4 ++++
arch/powerpc/xmon/xmon.c | 2 +-
3 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/kernel/head_fsl_booke.S b/arch/powerpc/kernel/head_fsl_booke.S
index e2750b856c8f..05b574f416b3 100644
--- a/arch/powerpc/kernel/head_fsl_booke.S
+++ b/arch/powerpc/kernel/head_fsl_booke.S
@@ -243,8 +243,10 @@ set_ivor:
li r0,0
stwu r0,THREAD_SIZE-STACK_FRAME_OVERHEAD(r1)
+#ifdef CONFIG_SMP
CURRENT_THREAD_INFO(r22, r1)
stw r24, TI_CPU(r22)
+#endif
bl early_init
diff --git a/arch/powerpc/kernel/misc_32.S b/arch/powerpc/kernel/misc_32.S
index 695b24a2d954..2f0fe8bfc078 100644
--- a/arch/powerpc/kernel/misc_32.S
+++ b/arch/powerpc/kernel/misc_32.S
@@ -183,10 +183,14 @@ _GLOBAL(low_choose_750fx_pll)
or r4,r4,r5
mtspr SPRN_HID1,r4
+#ifdef CONFIG_SMP
/* Store new HID1 image */
CURRENT_THREAD_INFO(r6, r1)
lwz r6,TI_CPU(r6)
slwi r6,r6,2
+#else
+ li r6, 0
+#endif
addis r6,r6,nap_save_hid1@ha
stw r4,nap_save_hid1@l(r6)
diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
index c70d17c9a6ba..1731793e1277 100644
--- a/arch/powerpc/xmon/xmon.c
+++ b/arch/powerpc/xmon/xmon.c
@@ -2986,7 +2986,7 @@ static void show_task(struct task_struct *tsk)
printf("%px %016lx %6d %6d %c %2d %s\n", tsk,
tsk->thread.ksp,
tsk->pid, tsk->parent->pid,
- state, task_thread_info(tsk)->cpu,
+ state, task_cpu(tsk),
tsk->comm);
}
--
2.13.3
^ permalink raw reply related
* [PATCH v5 3/9] powerpc: Prepare for moving thread_info into task_struct
From: Christophe Leroy @ 2018-10-05 7:32 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
In-Reply-To: <cover.1538687073.git.christophe.leroy@c-s.fr>
This patch cleans the powerpc kernel before activating
CONFIG_THREAD_INFO_IN_TASK:
- The purpose of the pointer given to call_do_softirq() and
call_do_irq() is to point the new stack ==> change it to void* and
rename it 'sp'
- Don't use CURRENT_THREAD_INFO() to locate the stack.
- Fix a few comments.
- Replace current_thread_info()->task by current
- Remove unnecessary casts to thread_info, as they'll become invalid
once thread_info is not in stack anymore.
- Rename THREAD_INFO to TASK_STASK: as it is in fact the offset of the
pointer to the stack in task_struct, this pointer will not be impacted
by the move of THREAD_INFO.
- Makes TASK_STACK available to PPC64. PPC64 will need it to get the
stack pointer from current once the thread_info have been moved.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/irq.h | 4 ++--
arch/powerpc/include/asm/livepatch.h | 2 +-
arch/powerpc/include/asm/processor.h | 4 ++--
arch/powerpc/include/asm/reg.h | 2 +-
arch/powerpc/kernel/asm-offsets.c | 2 +-
arch/powerpc/kernel/entry_32.S | 2 +-
arch/powerpc/kernel/entry_64.S | 2 +-
arch/powerpc/kernel/head_32.S | 4 ++--
arch/powerpc/kernel/head_40x.S | 4 ++--
arch/powerpc/kernel/head_44x.S | 2 +-
arch/powerpc/kernel/head_8xx.S | 2 +-
arch/powerpc/kernel/head_booke.h | 4 ++--
arch/powerpc/kernel/head_fsl_booke.S | 4 ++--
arch/powerpc/kernel/irq.c | 2 +-
arch/powerpc/kernel/misc_32.S | 4 ++--
arch/powerpc/kernel/process.c | 6 +++---
arch/powerpc/kernel/setup_32.c | 15 +++++----------
arch/powerpc/kernel/smp.c | 4 +++-
18 files changed, 33 insertions(+), 36 deletions(-)
diff --git a/arch/powerpc/include/asm/irq.h b/arch/powerpc/include/asm/irq.h
index ee39ce56b2a2..2efbae8d93be 100644
--- a/arch/powerpc/include/asm/irq.h
+++ b/arch/powerpc/include/asm/irq.h
@@ -63,8 +63,8 @@ extern struct thread_info *hardirq_ctx[NR_CPUS];
extern struct thread_info *softirq_ctx[NR_CPUS];
extern void irq_ctx_init(void);
-extern void call_do_softirq(struct thread_info *tp);
-extern void call_do_irq(struct pt_regs *regs, struct thread_info *tp);
+void call_do_softirq(void *sp);
+void call_do_irq(struct pt_regs *regs, void *sp);
extern void do_IRQ(struct pt_regs *regs);
extern void __init init_IRQ(void);
extern void __do_irq(struct pt_regs *regs);
diff --git a/arch/powerpc/include/asm/livepatch.h b/arch/powerpc/include/asm/livepatch.h
index 47a03b9b528b..818451bf629c 100644
--- a/arch/powerpc/include/asm/livepatch.h
+++ b/arch/powerpc/include/asm/livepatch.h
@@ -49,7 +49,7 @@ static inline void klp_init_thread_info(struct thread_info *ti)
ti->livepatch_sp = (unsigned long *)(ti + 1) + 1;
}
#else
-static void klp_init_thread_info(struct thread_info *ti) { }
+static inline void klp_init_thread_info(struct thread_info *ti) { }
#endif /* CONFIG_LIVEPATCH */
#endif /* _ASM_POWERPC_LIVEPATCH_H */
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index 13589274fe9b..b225c7f7c5a4 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -40,7 +40,7 @@
#ifndef __ASSEMBLY__
#include <linux/types.h>
-#include <asm/thread_info.h>
+#include <linux/thread_info.h>
#include <asm/ptrace.h>
#include <asm/hw_breakpoint.h>
@@ -332,7 +332,7 @@ struct thread_struct {
#define INIT_SP (sizeof(init_stack) + (unsigned long) &init_stack)
#define INIT_SP_LIMIT \
- (_ALIGN_UP(sizeof(init_thread_info), 16) + (unsigned long) &init_stack)
+ (_ALIGN_UP(sizeof(struct thread_info), 16) + (unsigned long)&init_stack)
#ifdef CONFIG_SPE
#define SPEFSCR_INIT \
diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h
index 640a4d818772..d2528a0b2f5b 100644
--- a/arch/powerpc/include/asm/reg.h
+++ b/arch/powerpc/include/asm/reg.h
@@ -1058,7 +1058,7 @@
* - SPRG9 debug exception scratch
*
* All 32-bit:
- * - SPRG3 current thread_info pointer
+ * - SPRG3 current thread_struct physical addr pointer
* (virtual on BookE, physical on others)
*
* 32-bit classic:
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index a6d70fd2e499..c583a02e5a21 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -91,10 +91,10 @@ int main(void)
DEFINE(NMI_MASK, NMI_MASK);
OFFSET(TASKTHREADPPR, task_struct, thread.ppr);
#else
- OFFSET(THREAD_INFO, task_struct, stack);
DEFINE(THREAD_INFO_GAP, _ALIGN_UP(sizeof(struct thread_info), 16));
OFFSET(KSP_LIMIT, thread_struct, ksp_limit);
#endif /* CONFIG_PPC64 */
+ OFFSET(TASK_STACK, task_struct, stack);
#ifdef CONFIG_LIVEPATCH
OFFSET(TI_livepatch_sp, thread_info, livepatch_sp);
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index 77decded1175..7ea1d71f4546 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -1166,7 +1166,7 @@ ret_from_debug_exc:
mfspr r9,SPRN_SPRG_THREAD
lwz r10,SAVED_KSP_LIMIT(r1)
stw r10,KSP_LIMIT(r9)
- lwz r9,THREAD_INFO-THREAD(r9)
+ lwz r9,TASK_STACK-THREAD(r9)
CURRENT_THREAD_INFO(r10, r1)
lwz r10,TI_PREEMPT(r10)
stw r10,TI_PREEMPT(r9)
diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
index ed6f6c7f4264..6fce0f8fd8c4 100644
--- a/arch/powerpc/kernel/entry_64.S
+++ b/arch/powerpc/kernel/entry_64.S
@@ -684,7 +684,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_207S)
2:
#endif /* CONFIG_PPC_BOOK3S_64 */
- CURRENT_THREAD_INFO(r7, r8) /* base of new stack */
+ clrrdi r7, r8, THREAD_SHIFT /* base of new stack */
/* Note: this uses SWITCH_FRAME_SIZE rather than INT_FRAME_SIZE
because we don't need to leave the 288-byte ABI gap at the
top of the kernel stack. */
diff --git a/arch/powerpc/kernel/head_32.S b/arch/powerpc/kernel/head_32.S
index 61ca27929355..dce6f2ff07e5 100644
--- a/arch/powerpc/kernel/head_32.S
+++ b/arch/powerpc/kernel/head_32.S
@@ -261,7 +261,7 @@ __secondary_hold_acknowledge:
tophys(r11,r1); /* use tophys(r1) if kernel */ \
beq 1f; \
mfspr r11,SPRN_SPRG_THREAD; \
- lwz r11,THREAD_INFO-THREAD(r11); \
+ lwz r11,TASK_STACK-THREAD(r11); \
addi r11,r11,THREAD_SIZE; \
tophys(r11,r11); \
1: subi r11,r11,INT_FRAME_SIZE /* alloc exc. frame */
@@ -841,7 +841,7 @@ __secondary_start:
bl init_idle_6xx
#endif /* CONFIG_6xx */
- /* get current_thread_info and current */
+ /* get current's stack and current */
lis r1,secondary_ti@ha
tophys(r1,r1)
lwz r1,secondary_ti@l(r1)
diff --git a/arch/powerpc/kernel/head_40x.S b/arch/powerpc/kernel/head_40x.S
index b19d78410511..3088c9f29f5e 100644
--- a/arch/powerpc/kernel/head_40x.S
+++ b/arch/powerpc/kernel/head_40x.S
@@ -115,7 +115,7 @@ _ENTRY(saved_ksp_limit)
andi. r11,r11,MSR_PR; \
beq 1f; \
mfspr r1,SPRN_SPRG_THREAD; /* if from user, start at top of */\
- lwz r1,THREAD_INFO-THREAD(r1); /* this thread's kernel stack */\
+ lwz r1,TASK_STACK-THREAD(r1); /* this thread's kernel stack */\
addi r1,r1,THREAD_SIZE; \
1: subi r1,r1,INT_FRAME_SIZE; /* Allocate an exception frame */\
tophys(r11,r1); \
@@ -158,7 +158,7 @@ _ENTRY(saved_ksp_limit)
beq 1f; \
/* COMING FROM USER MODE */ \
mfspr r11,SPRN_SPRG_THREAD; /* if from user, start at top of */\
- lwz r11,THREAD_INFO-THREAD(r11); /* this thread's kernel stack */\
+ lwz r11,TASK_STACK-THREAD(r11); /* this thread's kernel stack */\
1: addi r11,r11,THREAD_SIZE-INT_FRAME_SIZE; /* Alloc an excpt frm */\
tophys(r11,r11); \
stw r10,_CCR(r11); /* save various registers */\
diff --git a/arch/powerpc/kernel/head_44x.S b/arch/powerpc/kernel/head_44x.S
index 37e4a7cf0065..15d39b2499de 100644
--- a/arch/powerpc/kernel/head_44x.S
+++ b/arch/powerpc/kernel/head_44x.S
@@ -1020,7 +1020,7 @@ _GLOBAL(start_secondary_47x)
/* Now we can get our task struct and real stack pointer */
- /* Get current_thread_info and current */
+ /* Get current's stack and current */
lis r1,secondary_ti@ha
lwz r1,secondary_ti@l(r1)
lwz r2,TI_TASK(r1)
diff --git a/arch/powerpc/kernel/head_8xx.S b/arch/powerpc/kernel/head_8xx.S
index 6582f824d620..e56e36aa2b3d 100644
--- a/arch/powerpc/kernel/head_8xx.S
+++ b/arch/powerpc/kernel/head_8xx.S
@@ -124,7 +124,7 @@ turn_on_mmu:
tophys(r11,r1); /* use tophys(r1) if kernel */ \
beq 1f; \
mfspr r11,SPRN_SPRG_THREAD; \
- lwz r11,THREAD_INFO-THREAD(r11); \
+ lwz r11,TASK_STACK-THREAD(r11); \
addi r11,r11,THREAD_SIZE; \
tophys(r11,r11); \
1: subi r11,r11,INT_FRAME_SIZE /* alloc exc. frame */
diff --git a/arch/powerpc/kernel/head_booke.h b/arch/powerpc/kernel/head_booke.h
index d0862a100d29..175f812066dc 100644
--- a/arch/powerpc/kernel/head_booke.h
+++ b/arch/powerpc/kernel/head_booke.h
@@ -44,7 +44,7 @@
mr r11, r1; \
beq 1f; \
/* if from user, start at top of this thread's kernel stack */ \
- lwz r11, THREAD_INFO-THREAD(r10); \
+ lwz r11, TASK_STACK - THREAD(r10); \
ALLOC_STACK_FRAME(r11, THREAD_SIZE); \
1 : subi r11, r11, INT_FRAME_SIZE; /* Allocate exception frame */ \
stw r13, _CCR(r11); /* save various registers */ \
@@ -130,7 +130,7 @@
DO_KVM BOOKE_INTERRUPT_##intno exc_level_srr1; \
andi. r11,r11,MSR_PR; \
mfspr r11,SPRN_SPRG_THREAD; /* if from user, start at top of */\
- lwz r11,THREAD_INFO-THREAD(r11); /* this thread's kernel stack */\
+ lwz r11, TASK_STACK - THREAD(r11); /* this thread's kernel stack */\
addi r11,r11,EXC_LVL_FRAME_OVERHEAD; /* allocate stack frame */\
beq 1f; \
/* COMING FROM USER MODE */ \
diff --git a/arch/powerpc/kernel/head_fsl_booke.S b/arch/powerpc/kernel/head_fsl_booke.S
index 05b574f416b3..239ad8a4754e 100644
--- a/arch/powerpc/kernel/head_fsl_booke.S
+++ b/arch/powerpc/kernel/head_fsl_booke.S
@@ -704,7 +704,7 @@ finish_tlb_load:
/* Get the next_tlbcam_idx percpu var */
#ifdef CONFIG_SMP
- lwz r12, THREAD_INFO-THREAD(r12)
+ lwz r12, TASK_STACK-THREAD(r12)
lwz r15, TI_CPU(r12)
lis r14, __per_cpu_offset@h
ori r14, r14, __per_cpu_offset@l
@@ -1076,7 +1076,7 @@ __secondary_start:
mr r4,r24 /* Why? */
bl call_setup_cpu
- /* get current_thread_info and current */
+ /* get current's stack and current */
lis r1,secondary_ti@ha
lwz r1,secondary_ti@l(r1)
lwz r2,TI_TASK(r1)
diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index 916ddc4aac44..9e7dc4468c8c 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -663,7 +663,7 @@ void do_IRQ(struct pt_regs *regs)
struct thread_info *curtp, *irqtp, *sirqtp;
/* Switch to the irq stack to handle this */
- curtp = current_thread_info();
+ curtp = (void *)(current_stack_pointer() & ~(THREAD_SIZE - 1));
irqtp = hardirq_ctx[raw_smp_processor_id()];
sirqtp = softirq_ctx[raw_smp_processor_id()];
diff --git a/arch/powerpc/kernel/misc_32.S b/arch/powerpc/kernel/misc_32.S
index 2f0fe8bfc078..32762f4c3458 100644
--- a/arch/powerpc/kernel/misc_32.S
+++ b/arch/powerpc/kernel/misc_32.S
@@ -60,7 +60,7 @@ _GLOBAL(call_do_softirq)
blr
/*
- * void call_do_irq(struct pt_regs *regs, struct thread_info *irqtp);
+ * void call_do_irq(struct pt_regs *regs, void *sp);
*/
_GLOBAL(call_do_irq)
mflr r0
@@ -603,7 +603,7 @@ EXPORT_SYMBOL(__bswapdi2)
#ifdef CONFIG_SMP
_GLOBAL(start_secondary_resume)
/* Reset stack */
- CURRENT_THREAD_INFO(r1, r1)
+ rlwinm r1, r1, 0, 0, 31 - THREAD_SHIFT
addi r1,r1,THREAD_SIZE-STACK_FRAME_OVERHEAD
li r3,0
stw r3,0(r1) /* Zero the stack frame pointer */
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index ec264a6f0eb3..623f0f6f401c 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -1237,8 +1237,8 @@ struct task_struct *__switch_to(struct task_struct *prev,
batch->active = 1;
}
- if (current_thread_info()->task->thread.regs) {
- restore_math(current_thread_info()->task->thread.regs);
+ if (current->thread.regs) {
+ restore_math(current->thread.regs);
/*
* The copy-paste buffer can only store into foreign real
@@ -1248,7 +1248,7 @@ struct task_struct *__switch_to(struct task_struct *prev,
* mappings, we must issue a cp_abort to clear any state and
* prevent snooping, corruption or a covert channel.
*/
- if (current_thread_info()->task->thread.used_vas)
+ if (current->thread.used_vas)
asm volatile(PPC_CP_ABORT);
}
#endif /* CONFIG_PPC_BOOK3S_64 */
diff --git a/arch/powerpc/kernel/setup_32.c b/arch/powerpc/kernel/setup_32.c
index 8c507be12c3c..81ebf7d6f526 100644
--- a/arch/powerpc/kernel/setup_32.c
+++ b/arch/powerpc/kernel/setup_32.c
@@ -205,10 +205,8 @@ void __init irqstack_early_init(void)
/* interrupt stacks must be in lowmem, we get that for free on ppc32
* as the memblock is limited to lowmem by default */
for_each_possible_cpu(i) {
- softirq_ctx[i] = (struct thread_info *)
- __va(memblock_alloc(THREAD_SIZE, THREAD_SIZE));
- hardirq_ctx[i] = (struct thread_info *)
- __va(memblock_alloc(THREAD_SIZE, THREAD_SIZE));
+ softirq_ctx[i] = __va(memblock_alloc(THREAD_SIZE, THREAD_SIZE));
+ hardirq_ctx[i] = __va(memblock_alloc(THREAD_SIZE, THREAD_SIZE));
}
}
@@ -226,13 +224,10 @@ void __init exc_lvl_early_init(void)
hw_cpu = 0;
#endif
- critirq_ctx[hw_cpu] = (struct thread_info *)
- __va(memblock_alloc(THREAD_SIZE, THREAD_SIZE));
+ critirq_ctx[hw_cpu] = __va(memblock_alloc(THREAD_SIZE, THREAD_SIZE));
#ifdef CONFIG_BOOKE
- dbgirq_ctx[hw_cpu] = (struct thread_info *)
- __va(memblock_alloc(THREAD_SIZE, THREAD_SIZE));
- mcheckirq_ctx[hw_cpu] = (struct thread_info *)
- __va(memblock_alloc(THREAD_SIZE, THREAD_SIZE));
+ dbgirq_ctx[hw_cpu] = __va(memblock_alloc(THREAD_SIZE, THREAD_SIZE));
+ mcheckirq_ctx[hw_cpu] = __va(memblock_alloc(THREAD_SIZE, THREAD_SIZE));
#endif
}
}
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 61c1fadbc644..19dd0ea55714 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -20,6 +20,7 @@
#include <linux/kernel.h>
#include <linux/export.h>
#include <linux/sched/mm.h>
+#include <linux/sched/task_stack.h>
#include <linux/sched/topology.h>
#include <linux/smp.h>
#include <linux/interrupt.h>
@@ -812,7 +813,8 @@ static void cpu_idle_thread_init(unsigned int cpu, struct task_struct *idle)
#ifdef CONFIG_PPC64
paca_ptrs[cpu]->__current = idle;
- paca_ptrs[cpu]->kstack = (unsigned long)ti + THREAD_SIZE - STACK_FRAME_OVERHEAD;
+ paca_ptrs[cpu]->kstack = (unsigned long)task_stack_page(idle) +
+ THREAD_SIZE - STACK_FRAME_OVERHEAD;
#endif
ti->cpu = cpu;
secondary_ti = current_set[cpu] = ti;
--
2.13.3
^ permalink raw reply related
* [PATCH v5 4/9] powerpc: Activate CONFIG_THREAD_INFO_IN_TASK
From: Christophe Leroy @ 2018-10-05 7:32 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
In-Reply-To: <cover.1538687073.git.christophe.leroy@c-s.fr>
This patch activates CONFIG_THREAD_INFO_IN_TASK which
moves the thread_info into task_struct.
Moving thread_info into task_struct has the following advantages:
- It protects thread_info from corruption in the case of stack
overflows.
- Its address is harder to determine if stack addresses are
leaked, making a number of attacks more difficult.
This has the following consequences:
- thread_info is now located at the beginning of task_struct.
- The 'cpu' field is now in task_struct, and only exists when
CONFIG_SMP is active.
- thread_info doesn't have anymore the 'task' field.
This patch:
- Removes all recopy of thread_info struct when the stack changes.
- Changes the CURRENT_THREAD_INFO() macro to point to current.
- Selects CONFIG_THREAD_INFO_IN_TASK.
- Modifies raw_smp_processor_id() to get ->cpu from current without
including linux/sched.h to avoid circular inclusion and without
including asm/asm-offsets.h to avoid symbol names duplication
between ASM constants and C constants.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/Kconfig | 1 +
arch/powerpc/Makefile | 8 +++++-
arch/powerpc/include/asm/ptrace.h | 2 +-
arch/powerpc/include/asm/smp.h | 17 +++++++++++-
arch/powerpc/include/asm/thread_info.h | 17 ++----------
arch/powerpc/kernel/asm-offsets.c | 7 +++--
arch/powerpc/kernel/entry_32.S | 9 +++----
arch/powerpc/kernel/exceptions-64e.S | 11 --------
arch/powerpc/kernel/head_32.S | 6 ++---
arch/powerpc/kernel/head_44x.S | 4 +--
arch/powerpc/kernel/head_64.S | 1 +
arch/powerpc/kernel/head_booke.h | 8 +-----
arch/powerpc/kernel/head_fsl_booke.S | 7 +++--
arch/powerpc/kernel/irq.c | 47 +---------------------------------
arch/powerpc/kernel/kgdb.c | 28 --------------------
arch/powerpc/kernel/machine_kexec_64.c | 6 ++---
arch/powerpc/kernel/setup-common.c | 2 +-
arch/powerpc/kernel/setup_64.c | 21 ---------------
arch/powerpc/kernel/smp.c | 2 +-
19 files changed, 51 insertions(+), 153 deletions(-)
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 602eea723624..3b958cd4e284 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -238,6 +238,7 @@ config PPC
select RTC_LIB
select SPARSE_IRQ
select SYSCTL_EXCEPTION_TRACE
+ select THREAD_INFO_IN_TASK
select VIRT_TO_BUS if !PPC64
#
# Please keep this list sorted alphabetically.
diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile
index 81552c7b46eb..e31db77a4e99 100644
--- a/arch/powerpc/Makefile
+++ b/arch/powerpc/Makefile
@@ -422,6 +422,13 @@ else
endif
endif
+ifdef CONFIG_SMP
+prepare: task_cpu_prepare
+
+task_cpu_prepare: prepare0
+ $(eval KBUILD_CFLAGS += -D_TASK_CPU=$(shell awk '{if ($$2 == "TI_CPU") print $$3;}' include/generated/asm-offsets.h))
+endif
+
# Use the file '.tmp_gas_check' for binutils tests, as gas won't output
# to stdout and these checks are run even on install targets.
TOUT := .tmp_gas_check
@@ -439,4 +446,3 @@ checkbin:
CLEAN_FILES += $(TOUT)
-
diff --git a/arch/powerpc/include/asm/ptrace.h b/arch/powerpc/include/asm/ptrace.h
index 447cbd1bee99..3a7e5561630b 100644
--- a/arch/powerpc/include/asm/ptrace.h
+++ b/arch/powerpc/include/asm/ptrace.h
@@ -120,7 +120,7 @@ extern int ptrace_put_reg(struct task_struct *task, int regno,
unsigned long data);
#define current_pt_regs() \
- ((struct pt_regs *)((unsigned long)current_thread_info() + THREAD_SIZE) - 1)
+ ((struct pt_regs *)((unsigned long)task_stack_page(current) + THREAD_SIZE) - 1)
/*
* We use the least-significant bit of the trap field to indicate
* whether we have saved the full set of registers, or only a
diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h
index 95b66a0c639b..93a8cd120663 100644
--- a/arch/powerpc/include/asm/smp.h
+++ b/arch/powerpc/include/asm/smp.h
@@ -83,7 +83,22 @@ int is_cpu_dead(unsigned int cpu);
/* 32-bit */
extern int smp_hw_index[];
-#define raw_smp_processor_id() (current_thread_info()->cpu)
+/*
+ * This is particularly ugly: it appears we can't actually get the definition
+ * of task_struct here, but we need access to the CPU this task is running on.
+ * Instead of using task_struct we're using _TASK_CPU which is extracted from
+ * asm-offsets.h by kbuild to get the current processor ID.
+ *
+ * This also needs to be safeguarded when building asm-offsets.s because at
+ * that time _TASK_CPU is not defined yet. It could have been guarded by
+ * _TASK_CPU itself, but we want the build to fail if _TASK_CPU is missing
+ * when building something else than asm-offsets.s
+ */
+#ifdef GENERATING_ASM_OFFSETS
+#define raw_smp_processor_id() (0)
+#else
+#define raw_smp_processor_id() (*(unsigned int *)((void *)current + _TASK_CPU))
+#endif
#define hard_smp_processor_id() (smp_hw_index[smp_processor_id()])
static inline int get_hard_smp_processor_id(int cpu)
diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h
index 3185f8ac1182..61c8747cd926 100644
--- a/arch/powerpc/include/asm/thread_info.h
+++ b/arch/powerpc/include/asm/thread_info.h
@@ -18,9 +18,9 @@
#define THREAD_SIZE (1 << THREAD_SHIFT)
#ifdef CONFIG_PPC64
-#define CURRENT_THREAD_INFO(dest, sp) stringify_in_c(clrrdi dest, sp, THREAD_SHIFT)
+#define CURRENT_THREAD_INFO(dest, sp) stringify_in_c(ld dest, PACACURRENT(r13))
#else
-#define CURRENT_THREAD_INFO(dest, sp) stringify_in_c(rlwinm dest, sp, 0, 0, 31-THREAD_SHIFT)
+#define CURRENT_THREAD_INFO(dest, sp) stringify_in_c(mr dest, r2)
#endif
#ifndef __ASSEMBLY__
@@ -33,8 +33,6 @@
* low level task data.
*/
struct thread_info {
- struct task_struct *task; /* main task structure */
- int cpu; /* cpu we're on */
int preempt_count; /* 0 => preemptable,
<0 => BUG */
unsigned long local_flags; /* private flags for thread */
@@ -53,8 +51,6 @@ struct thread_info {
*/
#define INIT_THREAD_INFO(tsk) \
{ \
- .task = &tsk, \
- .cpu = 0, \
.preempt_count = INIT_PREEMPT_COUNT, \
.flags = 0, \
}
@@ -62,15 +58,6 @@ struct thread_info {
#define THREAD_SIZE_ORDER (THREAD_SHIFT - PAGE_SHIFT)
/* how to get the thread information struct from C */
-static inline struct thread_info *current_thread_info(void)
-{
- unsigned long val;
-
- asm (CURRENT_THREAD_INFO(%0,1) : "=r" (val));
-
- return (struct thread_info *)val;
-}
-
extern int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
#endif /* __ASSEMBLY__ */
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index c583a02e5a21..833d189df04c 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -13,6 +13,8 @@
* 2 of the License, or (at your option) any later version.
*/
+#define GENERATING_ASM_OFFSETS /* asm/smp.h */
+
#include <linux/compat.h>
#include <linux/signal.h>
#include <linux/sched.h>
@@ -95,6 +97,9 @@ int main(void)
OFFSET(KSP_LIMIT, thread_struct, ksp_limit);
#endif /* CONFIG_PPC64 */
OFFSET(TASK_STACK, task_struct, stack);
+#ifdef CONFIG_SMP
+ OFFSET(TI_CPU, task_struct, cpu);
+#endif
#ifdef CONFIG_LIVEPATCH
OFFSET(TI_livepatch_sp, thread_info, livepatch_sp);
@@ -162,8 +167,6 @@ int main(void)
OFFSET(TI_FLAGS, thread_info, flags);
OFFSET(TI_LOCAL_FLAGS, thread_info, local_flags);
OFFSET(TI_PREEMPT, thread_info, preempt_count);
- OFFSET(TI_TASK, thread_info, task);
- OFFSET(TI_CPU, thread_info, cpu);
#ifdef CONFIG_PPC64
OFFSET(DCACHEL1BLOCKSIZE, ppc64_caches, l1d.block_size);
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index 7ea1d71f4546..fa7a69ffb37a 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -1166,10 +1166,6 @@ ret_from_debug_exc:
mfspr r9,SPRN_SPRG_THREAD
lwz r10,SAVED_KSP_LIMIT(r1)
stw r10,KSP_LIMIT(r9)
- lwz r9,TASK_STACK-THREAD(r9)
- CURRENT_THREAD_INFO(r10, r1)
- lwz r10,TI_PREEMPT(r10)
- stw r10,TI_PREEMPT(r9)
RESTORE_xSRR(SRR0,SRR1);
RESTORE_xSRR(CSRR0,CSRR1);
RESTORE_MMU_REGS;
@@ -1292,10 +1288,13 @@ BEGIN_FTR_SECTION
END_FTR_SECTION_IFSET(CPU_FTR_601)
lwz r3,_TRAP(r1)
andi. r0,r3,1
- beq 4f
+ beq 5f
SAVE_NVGPRS(r1)
rlwinm r3,r3,0,0,30
stw r3,_TRAP(r1)
+5: mfspr r2,SPRN_SPRG_THREAD
+ addi r2,r2,-THREAD
+ tovirt(r2,r2) /* set back r2 to current */
4: addi r3,r1,STACK_FRAME_OVERHEAD
bl unrecoverable_exception
/* shouldn't return */
diff --git a/arch/powerpc/kernel/exceptions-64e.S b/arch/powerpc/kernel/exceptions-64e.S
index 6d6e144a28ce..231d066b4a3d 100644
--- a/arch/powerpc/kernel/exceptions-64e.S
+++ b/arch/powerpc/kernel/exceptions-64e.S
@@ -77,17 +77,6 @@ special_reg_save:
andi. r3,r3,MSR_PR
bnelr
- /* Copy info into temporary exception thread info */
- ld r11,PACAKSAVE(r13)
- CURRENT_THREAD_INFO(r11, r11)
- CURRENT_THREAD_INFO(r12, r1)
- ld r10,TI_FLAGS(r11)
- std r10,TI_FLAGS(r12)
- ld r10,TI_PREEMPT(r11)
- std r10,TI_PREEMPT(r12)
- ld r10,TI_TASK(r11)
- std r10,TI_TASK(r12)
-
/*
* Advance to the next TLB exception frame for handler
* types that don't do it automatically.
diff --git a/arch/powerpc/kernel/head_32.S b/arch/powerpc/kernel/head_32.S
index dce6f2ff07e5..44dfd73b2a62 100644
--- a/arch/powerpc/kernel/head_32.S
+++ b/arch/powerpc/kernel/head_32.S
@@ -844,9 +844,9 @@ __secondary_start:
/* get current's stack and current */
lis r1,secondary_ti@ha
tophys(r1,r1)
- lwz r1,secondary_ti@l(r1)
- tophys(r2,r1)
- lwz r2,TI_TASK(r2)
+ lwz r2,secondary_ti@l(r1)
+ tophys(r1,r2)
+ lwz r1,TASK_STACK(r1)
/* stack */
addi r1,r1,THREAD_SIZE-STACK_FRAME_OVERHEAD
diff --git a/arch/powerpc/kernel/head_44x.S b/arch/powerpc/kernel/head_44x.S
index 15d39b2499de..2c7e90f36358 100644
--- a/arch/powerpc/kernel/head_44x.S
+++ b/arch/powerpc/kernel/head_44x.S
@@ -1022,8 +1022,8 @@ _GLOBAL(start_secondary_47x)
/* Get current's stack and current */
lis r1,secondary_ti@ha
- lwz r1,secondary_ti@l(r1)
- lwz r2,TI_TASK(r1)
+ lwz r2,secondary_ti@l(r1)
+ lwz r1,TASK_STACK(r2)
/* Current stack pointer */
addi r1,r1,THREAD_SIZE-STACK_FRAME_OVERHEAD
diff --git a/arch/powerpc/kernel/head_64.S b/arch/powerpc/kernel/head_64.S
index 4898e9491a1c..c6a9bf7b34bf 100644
--- a/arch/powerpc/kernel/head_64.S
+++ b/arch/powerpc/kernel/head_64.S
@@ -805,6 +805,7 @@ __secondary_start:
LOAD_REG_ADDR(r3, current_set)
sldi r28,r24,3 /* get current_set[cpu#] */
ldx r14,r3,r28
+ ld r14,TASK_STACK(r14)
addi r14,r14,THREAD_SIZE-STACK_FRAME_OVERHEAD
std r14,PACAKSAVE(r13)
diff --git a/arch/powerpc/kernel/head_booke.h b/arch/powerpc/kernel/head_booke.h
index 175f812066dc..f0c254ab9703 100644
--- a/arch/powerpc/kernel/head_booke.h
+++ b/arch/powerpc/kernel/head_booke.h
@@ -143,13 +143,7 @@
stw r10,GPR11(r11); \
b 2f; \
/* COMING FROM PRIV MODE */ \
-1: lwz r9,TI_FLAGS-EXC_LVL_FRAME_OVERHEAD(r11); \
- lwz r10,TI_PREEMPT-EXC_LVL_FRAME_OVERHEAD(r11); \
- stw r9,TI_FLAGS-EXC_LVL_FRAME_OVERHEAD(r8); \
- stw r10,TI_PREEMPT-EXC_LVL_FRAME_OVERHEAD(r8); \
- lwz r9,TI_TASK-EXC_LVL_FRAME_OVERHEAD(r11); \
- stw r9,TI_TASK-EXC_LVL_FRAME_OVERHEAD(r8); \
- mr r11,r8; \
+1: mr r11, r8; \
2: mfspr r8,SPRN_SPRG_RSCRATCH_##exc_level; \
stw r12,GPR12(r11); /* save various registers */\
mflr r10; \
diff --git a/arch/powerpc/kernel/head_fsl_booke.S b/arch/powerpc/kernel/head_fsl_booke.S
index 239ad8a4754e..b8a2b789677e 100644
--- a/arch/powerpc/kernel/head_fsl_booke.S
+++ b/arch/powerpc/kernel/head_fsl_booke.S
@@ -704,8 +704,7 @@ finish_tlb_load:
/* Get the next_tlbcam_idx percpu var */
#ifdef CONFIG_SMP
- lwz r12, TASK_STACK-THREAD(r12)
- lwz r15, TI_CPU(r12)
+ lwz r15, TI_CPU-THREAD(r12)
lis r14, __per_cpu_offset@h
ori r14, r14, __per_cpu_offset@l
rlwinm r15, r15, 2, 0, 29
@@ -1078,8 +1077,8 @@ __secondary_start:
/* get current's stack and current */
lis r1,secondary_ti@ha
- lwz r1,secondary_ti@l(r1)
- lwz r2,TI_TASK(r1)
+ lwz r2,secondary_ti@l(r1)
+ lwz r1,TASK_STACK(r2)
/* stack */
addi r1,r1,THREAD_SIZE-STACK_FRAME_OVERHEAD
diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index 9e7dc4468c8c..3fdb6b6973cf 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -673,24 +673,9 @@ void do_IRQ(struct pt_regs *regs)
set_irq_regs(old_regs);
return;
}
-
- /* Prepare the thread_info in the irq stack */
- irqtp->task = curtp->task;
- irqtp->flags = 0;
-
- /* Copy the preempt_count so that the [soft]irq checks work. */
- irqtp->preempt_count = curtp->preempt_count;
-
/* Switch stack and call */
call_do_irq(regs, irqtp);
- /* Restore stack limit */
- irqtp->task = NULL;
-
- /* Copy back updates to the thread_info */
- if (irqtp->flags)
- set_bits(irqtp->flags, &curtp->flags);
-
set_irq_regs(old_regs);
}
@@ -711,7 +696,6 @@ struct thread_info *mcheckirq_ctx[NR_CPUS] __read_mostly;
void exc_lvl_ctx_init(void)
{
- struct thread_info *tp;
int i, cpu_nr;
for_each_possible_cpu(i) {
@@ -726,20 +710,9 @@ void exc_lvl_ctx_init(void)
#endif
memset((void *)critirq_ctx[cpu_nr], 0, THREAD_SIZE);
- tp = critirq_ctx[cpu_nr];
- tp->cpu = cpu_nr;
- tp->preempt_count = 0;
-
#ifdef CONFIG_BOOKE
memset((void *)dbgirq_ctx[cpu_nr], 0, THREAD_SIZE);
- tp = dbgirq_ctx[cpu_nr];
- tp->cpu = cpu_nr;
- tp->preempt_count = 0;
-
memset((void *)mcheckirq_ctx[cpu_nr], 0, THREAD_SIZE);
- tp = mcheckirq_ctx[cpu_nr];
- tp->cpu = cpu_nr;
- tp->preempt_count = HARDIRQ_OFFSET;
#endif
}
}
@@ -750,38 +723,20 @@ struct thread_info *hardirq_ctx[NR_CPUS] __read_mostly;
void irq_ctx_init(void)
{
- struct thread_info *tp;
int i;
for_each_possible_cpu(i) {
memset((void *)softirq_ctx[i], 0, THREAD_SIZE);
- tp = softirq_ctx[i];
- tp->cpu = i;
- klp_init_thread_info(tp);
-
memset((void *)hardirq_ctx[i], 0, THREAD_SIZE);
- tp = hardirq_ctx[i];
- tp->cpu = i;
- klp_init_thread_info(tp);
}
}
void do_softirq_own_stack(void)
{
- struct thread_info *curtp, *irqtp;
+ struct thread_info *irqtp;
- curtp = current_thread_info();
irqtp = softirq_ctx[smp_processor_id()];
- irqtp->task = curtp->task;
- irqtp->flags = 0;
call_do_softirq(irqtp);
- irqtp->task = NULL;
-
- /* Set any flag that may have been set on the
- * alternate stack
- */
- if (irqtp->flags)
- set_bits(irqtp->flags, &curtp->flags);
}
irq_hw_number_t virq_to_hw(unsigned int virq)
diff --git a/arch/powerpc/kernel/kgdb.c b/arch/powerpc/kernel/kgdb.c
index 35e240a0a408..f4a34e46ee34 100644
--- a/arch/powerpc/kernel/kgdb.c
+++ b/arch/powerpc/kernel/kgdb.c
@@ -150,41 +150,13 @@ static int kgdb_handle_breakpoint(struct pt_regs *regs)
return 1;
}
-static DEFINE_PER_CPU(struct thread_info, kgdb_thread_info);
static int kgdb_singlestep(struct pt_regs *regs)
{
- struct thread_info *thread_info, *exception_thread_info;
- struct thread_info *backup_current_thread_info =
- this_cpu_ptr(&kgdb_thread_info);
-
if (user_mode(regs))
return 0;
- /*
- * On Book E and perhaps other processors, singlestep is handled on
- * the critical exception stack. This causes current_thread_info()
- * to fail, since it it locates the thread_info by masking off
- * the low bits of the current stack pointer. We work around
- * this issue by copying the thread_info from the kernel stack
- * before calling kgdb_handle_exception, and copying it back
- * afterwards. On most processors the copy is avoided since
- * exception_thread_info == thread_info.
- */
- thread_info = (struct thread_info *)(regs->gpr[1] & ~(THREAD_SIZE-1));
- exception_thread_info = current_thread_info();
-
- if (thread_info != exception_thread_info) {
- /* Save the original current_thread_info. */
- memcpy(backup_current_thread_info, exception_thread_info, sizeof *thread_info);
- memcpy(exception_thread_info, thread_info, sizeof *thread_info);
- }
-
kgdb_handle_exception(0, SIGTRAP, 0, regs);
- if (thread_info != exception_thread_info)
- /* Restore current_thread_info lastly. */
- memcpy(exception_thread_info, backup_current_thread_info, sizeof *thread_info);
-
return 1;
}
diff --git a/arch/powerpc/kernel/machine_kexec_64.c b/arch/powerpc/kernel/machine_kexec_64.c
index a0f6f45005bd..75692c327ba0 100644
--- a/arch/powerpc/kernel/machine_kexec_64.c
+++ b/arch/powerpc/kernel/machine_kexec_64.c
@@ -317,10 +317,8 @@ void default_machine_kexec(struct kimage *image)
* We setup preempt_count to avoid using VMX in memcpy.
* XXX: the task struct will likely be invalid once we do the copy!
*/
- kexec_stack.thread_info.task = current_thread_info()->task;
- kexec_stack.thread_info.flags = 0;
- kexec_stack.thread_info.preempt_count = HARDIRQ_OFFSET;
- kexec_stack.thread_info.cpu = current_thread_info()->cpu;
+ current_thread_info()->flags = 0;
+ current_thread_info()->preempt_count = HARDIRQ_OFFSET;
/* We need a static PACA, too; copy this CPU's PACA over and switch to
* it. Also poison per_cpu_offset and NULL lppaca to catch anyone using
diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c
index 9ca9db707bcb..c991ef972802 100644
--- a/arch/powerpc/kernel/setup-common.c
+++ b/arch/powerpc/kernel/setup-common.c
@@ -940,7 +940,7 @@ void __init setup_arch(char **cmdline_p)
/* Reserve large chunks of memory for use by CMA for KVM. */
kvm_cma_reserve();
- klp_init_thread_info(&init_thread_info);
+ klp_init_thread_info(&init_task.thread_info);
init_mm.start_code = (unsigned long)_stext;
init_mm.end_code = (unsigned long) _etext;
diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
index faf00222b324..2d682f3e31c6 100644
--- a/arch/powerpc/kernel/setup_64.c
+++ b/arch/powerpc/kernel/setup_64.c
@@ -691,24 +691,6 @@ void __init exc_lvl_early_init(void)
#endif
/*
- * Emergency stacks are used for a range of things, from asynchronous
- * NMIs (system reset, machine check) to synchronous, process context.
- * We set preempt_count to zero, even though that isn't necessarily correct. To
- * get the right value we'd need to copy it from the previous thread_info, but
- * doing that might fault causing more problems.
- * TODO: what to do with accounting?
- */
-static void emerg_stack_init_thread_info(struct thread_info *ti, int cpu)
-{
- ti->task = NULL;
- ti->cpu = cpu;
- ti->preempt_count = 0;
- ti->local_flags = 0;
- ti->flags = 0;
- klp_init_thread_info(ti);
-}
-
-/*
* Stack space used when we detect a bad kernel stack pointer, and
* early in SMP boots before relocation is enabled. Exclusive emergency
* stack for machine checks.
@@ -739,20 +721,17 @@ void __init emergency_stack_init(void)
ti = alloc_stack(limit, i);
memset(ti, 0, THREAD_SIZE);
- emerg_stack_init_thread_info(ti, i);
paca_ptrs[i]->emergency_sp = (void *)ti + THREAD_SIZE;
#ifdef CONFIG_PPC_BOOK3S_64
/* emergency stack for NMI exception handling. */
ti = alloc_stack(limit, i);
memset(ti, 0, THREAD_SIZE);
- emerg_stack_init_thread_info(ti, i);
paca_ptrs[i]->nmi_emergency_sp = (void *)ti + THREAD_SIZE;
/* emergency stack for machine check exception handling. */
ti = alloc_stack(limit, i);
memset(ti, 0, THREAD_SIZE);
- emerg_stack_init_thread_info(ti, i);
paca_ptrs[i]->mc_emergency_sp = (void *)ti + THREAD_SIZE;
#endif
}
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 19dd0ea55714..f22fcbeb9898 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -816,7 +816,7 @@ static void cpu_idle_thread_init(unsigned int cpu, struct task_struct *idle)
paca_ptrs[cpu]->kstack = (unsigned long)task_stack_page(idle) +
THREAD_SIZE - STACK_FRAME_OVERHEAD;
#endif
- ti->cpu = cpu;
+ idle->cpu = cpu;
secondary_ti = current_set[cpu] = ti;
}
--
2.13.3
^ permalink raw reply related
* [PATCH v5 5/9] powerpc: regain entire stack space
From: Christophe Leroy @ 2018-10-05 7:32 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
In-Reply-To: <cover.1538687073.git.christophe.leroy@c-s.fr>
thread_info is not anymore in the stack, so the entire stack
can now be used.
In the meantime, with the previous patch all pointers to the stacks
are not anymore pointers to thread_info so this patch changes them
to void*
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
arch/powerpc/include/asm/irq.h | 10 +++++-----
arch/powerpc/include/asm/processor.h | 3 +--
arch/powerpc/kernel/asm-offsets.c | 1 -
arch/powerpc/kernel/entry_32.S | 14 ++++----------
arch/powerpc/kernel/irq.c | 19 +++++++++----------
arch/powerpc/kernel/misc_32.S | 6 ++----
arch/powerpc/kernel/process.c | 9 +++------
arch/powerpc/kernel/setup_64.c | 8 ++++----
8 files changed, 28 insertions(+), 42 deletions(-)
diff --git a/arch/powerpc/include/asm/irq.h b/arch/powerpc/include/asm/irq.h
index 2efbae8d93be..966ddd4d2414 100644
--- a/arch/powerpc/include/asm/irq.h
+++ b/arch/powerpc/include/asm/irq.h
@@ -48,9 +48,9 @@ struct pt_regs;
* Per-cpu stacks for handling critical, debug and machine check
* level interrupts.
*/
-extern struct thread_info *critirq_ctx[NR_CPUS];
-extern struct thread_info *dbgirq_ctx[NR_CPUS];
-extern struct thread_info *mcheckirq_ctx[NR_CPUS];
+extern void *critirq_ctx[NR_CPUS];
+extern void *dbgirq_ctx[NR_CPUS];
+extern void *mcheckirq_ctx[NR_CPUS];
extern void exc_lvl_ctx_init(void);
#else
#define exc_lvl_ctx_init()
@@ -59,8 +59,8 @@ extern void exc_lvl_ctx_init(void);
/*
* Per-cpu stacks for handling hard and soft interrupts.
*/
-extern struct thread_info *hardirq_ctx[NR_CPUS];
-extern struct thread_info *softirq_ctx[NR_CPUS];
+extern void *hardirq_ctx[NR_CPUS];
+extern void *softirq_ctx[NR_CPUS];
extern void irq_ctx_init(void);
void call_do_softirq(void *sp);
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index b225c7f7c5a4..e763342265a2 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -331,8 +331,7 @@ struct thread_struct {
#define ARCH_MIN_TASKALIGN 16
#define INIT_SP (sizeof(init_stack) + (unsigned long) &init_stack)
-#define INIT_SP_LIMIT \
- (_ALIGN_UP(sizeof(struct thread_info), 16) + (unsigned long)&init_stack)
+#define INIT_SP_LIMIT ((unsigned long)&init_stack)
#ifdef CONFIG_SPE
#define SPEFSCR_INIT \
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index 833d189df04c..768ce602d624 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -93,7 +93,6 @@ int main(void)
DEFINE(NMI_MASK, NMI_MASK);
OFFSET(TASKTHREADPPR, task_struct, thread.ppr);
#else
- DEFINE(THREAD_INFO_GAP, _ALIGN_UP(sizeof(struct thread_info), 16));
OFFSET(KSP_LIMIT, thread_struct, ksp_limit);
#endif /* CONFIG_PPC64 */
OFFSET(TASK_STACK, task_struct, stack);
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index fa7a69ffb37a..bd3b146e18a3 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -97,14 +97,11 @@ crit_transfer_to_handler:
mfspr r0,SPRN_SRR1
stw r0,_SRR1(r11)
- /* set the stack limit to the current stack
- * and set the limit to protect the thread_info
- * struct
- */
+ /* set the stack limit to the current stack */
mfspr r8,SPRN_SPRG_THREAD
lwz r0,KSP_LIMIT(r8)
stw r0,SAVED_KSP_LIMIT(r11)
- rlwimi r0,r1,0,0,(31-THREAD_SHIFT)
+ rlwinm r0,r1,0,0,(31 - THREAD_SHIFT)
stw r0,KSP_LIMIT(r8)
/* fall through */
#endif
@@ -121,14 +118,11 @@ crit_transfer_to_handler:
mfspr r0,SPRN_SRR1
stw r0,crit_srr1@l(0)
- /* set the stack limit to the current stack
- * and set the limit to protect the thread_info
- * struct
- */
+ /* set the stack limit to the current stack */
mfspr r8,SPRN_SPRG_THREAD
lwz r0,KSP_LIMIT(r8)
stw r0,saved_ksp_limit@l(0)
- rlwimi r0,r1,0,0,(31-THREAD_SHIFT)
+ rlwinm r0,r1,0,0,(31 - THREAD_SHIFT)
stw r0,KSP_LIMIT(r8)
/* fall through */
#endif
diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index 3fdb6b6973cf..62cfccf4af89 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -618,9 +618,8 @@ static inline void check_stack_overflow(void)
sp = current_stack_pointer() & (THREAD_SIZE-1);
/* check for stack overflow: is there less than 2KB free? */
- if (unlikely(sp < (sizeof(struct thread_info) + 2048))) {
- pr_err("do_IRQ: stack overflow: %ld\n",
- sp - sizeof(struct thread_info));
+ if (unlikely(sp < 2048)) {
+ pr_err("do_IRQ: stack overflow: %ld\n", sp);
dump_stack();
}
#endif
@@ -660,7 +659,7 @@ void __do_irq(struct pt_regs *regs)
void do_IRQ(struct pt_regs *regs)
{
struct pt_regs *old_regs = set_irq_regs(regs);
- struct thread_info *curtp, *irqtp, *sirqtp;
+ void *curtp, *irqtp, *sirqtp;
/* Switch to the irq stack to handle this */
curtp = (void *)(current_stack_pointer() & ~(THREAD_SIZE - 1));
@@ -690,9 +689,9 @@ void __init init_IRQ(void)
}
#if defined(CONFIG_BOOKE) || defined(CONFIG_40x)
-struct thread_info *critirq_ctx[NR_CPUS] __read_mostly;
-struct thread_info *dbgirq_ctx[NR_CPUS] __read_mostly;
-struct thread_info *mcheckirq_ctx[NR_CPUS] __read_mostly;
+void *critirq_ctx[NR_CPUS] __read_mostly;
+void *dbgirq_ctx[NR_CPUS] __read_mostly;
+void *mcheckirq_ctx[NR_CPUS] __read_mostly;
void exc_lvl_ctx_init(void)
{
@@ -718,8 +717,8 @@ void exc_lvl_ctx_init(void)
}
#endif
-struct thread_info *softirq_ctx[NR_CPUS] __read_mostly;
-struct thread_info *hardirq_ctx[NR_CPUS] __read_mostly;
+void *softirq_ctx[NR_CPUS] __read_mostly;
+void *hardirq_ctx[NR_CPUS] __read_mostly;
void irq_ctx_init(void)
{
@@ -733,7 +732,7 @@ void irq_ctx_init(void)
void do_softirq_own_stack(void)
{
- struct thread_info *irqtp;
+ void *irqtp;
irqtp = softirq_ctx[smp_processor_id()];
call_do_softirq(irqtp);
diff --git a/arch/powerpc/kernel/misc_32.S b/arch/powerpc/kernel/misc_32.S
index 32762f4c3458..a3663ad62f16 100644
--- a/arch/powerpc/kernel/misc_32.S
+++ b/arch/powerpc/kernel/misc_32.S
@@ -46,11 +46,10 @@ _GLOBAL(call_do_softirq)
mflr r0
stw r0,4(r1)
lwz r10,THREAD+KSP_LIMIT(r2)
- addi r11,r3,THREAD_INFO_GAP
+ stw r3, THREAD+KSP_LIMIT(r2)
stwu r1,THREAD_SIZE-STACK_FRAME_OVERHEAD(r3)
mr r1,r3
stw r10,8(r1)
- stw r11,THREAD+KSP_LIMIT(r2)
bl __do_softirq
lwz r10,8(r1)
lwz r1,0(r1)
@@ -66,11 +65,10 @@ _GLOBAL(call_do_irq)
mflr r0
stw r0,4(r1)
lwz r10,THREAD+KSP_LIMIT(r2)
- addi r11,r4,THREAD_INFO_GAP
+ stw r4, THREAD+KSP_LIMIT(r2)
stwu r1,THREAD_SIZE-STACK_FRAME_OVERHEAD(r4)
mr r1,r4
stw r10,8(r1)
- stw r11,THREAD+KSP_LIMIT(r2)
bl __do_irq
lwz r10,8(r1)
lwz r1,0(r1)
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index 623f0f6f401c..e973c439ab50 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -1679,8 +1679,7 @@ int copy_thread(unsigned long clone_flags, unsigned long usp,
sp -= STACK_FRAME_OVERHEAD;
p->thread.ksp = sp;
#ifdef CONFIG_PPC32
- p->thread.ksp_limit = (unsigned long)task_stack_page(p) +
- _ALIGN_UP(sizeof(struct thread_info), 16);
+ p->thread.ksp_limit = (unsigned long)task_stack_page(p);
#endif
#ifdef CONFIG_HAVE_HW_BREAKPOINT
p->thread.ptrace_bps[0] = NULL;
@@ -1982,13 +1981,11 @@ static inline int valid_irq_stack(unsigned long sp, struct task_struct *p,
*/
if (cpu < NR_CPUS && cpu_possible(cpu)) {
stack_page = (unsigned long) hardirq_ctx[cpu];
- if (sp >= stack_page + sizeof(struct thread_struct)
- && sp <= stack_page + THREAD_SIZE - nbytes)
+ if (sp >= stack_page && sp <= stack_page + THREAD_SIZE - nbytes)
return 1;
stack_page = (unsigned long) softirq_ctx[cpu];
- if (sp >= stack_page + sizeof(struct thread_struct)
- && sp <= stack_page + THREAD_SIZE - nbytes)
+ if (sp >= stack_page && sp <= stack_page + THREAD_SIZE - nbytes)
return 1;
}
return 0;
diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
index 2d682f3e31c6..6792e9c90689 100644
--- a/arch/powerpc/kernel/setup_64.c
+++ b/arch/powerpc/kernel/setup_64.c
@@ -717,22 +717,22 @@ void __init emergency_stack_init(void)
limit = min(ppc64_bolted_size(), ppc64_rma_size);
for_each_possible_cpu(i) {
- struct thread_info *ti;
+ void *ti;
ti = alloc_stack(limit, i);
memset(ti, 0, THREAD_SIZE);
- paca_ptrs[i]->emergency_sp = (void *)ti + THREAD_SIZE;
+ paca_ptrs[i]->emergency_sp = ti + THREAD_SIZE;
#ifdef CONFIG_PPC_BOOK3S_64
/* emergency stack for NMI exception handling. */
ti = alloc_stack(limit, i);
memset(ti, 0, THREAD_SIZE);
- paca_ptrs[i]->nmi_emergency_sp = (void *)ti + THREAD_SIZE;
+ paca_ptrs[i]->nmi_emergency_sp = ti + THREAD_SIZE;
/* emergency stack for machine check exception handling. */
ti = alloc_stack(limit, i);
memset(ti, 0, THREAD_SIZE);
- paca_ptrs[i]->mc_emergency_sp = (void *)ti + THREAD_SIZE;
+ paca_ptrs[i]->mc_emergency_sp = ti + THREAD_SIZE;
#endif
}
}
--
2.13.3
^ permalink raw reply related
* [PATCH v5 6/9] powerpc: 'current_set' is now a table of task_struct pointers
From: Christophe Leroy @ 2018-10-05 7:32 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
In-Reply-To: <cover.1538687073.git.christophe.leroy@c-s.fr>
The table of pointers 'current_set' has been used for retrieving
the stack and current. They used to be thread_info pointers as
they were pointing to the stack and current was taken from the
'task' field of the thread_info.
Now, the pointers of 'current_set' table are now both pointers
to task_struct and pointers to thread_info.
As they are used to get current, and the stack pointer is
retrieved from current's stack field, this patch changes
their type to task_struct, and renames secondary_ti to
secondary_current.
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
arch/powerpc/include/asm/asm-prototypes.h | 4 ++--
arch/powerpc/kernel/head_32.S | 6 +++---
arch/powerpc/kernel/head_44x.S | 4 ++--
arch/powerpc/kernel/head_fsl_booke.S | 4 ++--
arch/powerpc/kernel/smp.c | 10 ++++------
5 files changed, 13 insertions(+), 15 deletions(-)
diff --git a/arch/powerpc/include/asm/asm-prototypes.h b/arch/powerpc/include/asm/asm-prototypes.h
index 9bc98c239305..ab0541f9da42 100644
--- a/arch/powerpc/include/asm/asm-prototypes.h
+++ b/arch/powerpc/include/asm/asm-prototypes.h
@@ -23,8 +23,8 @@
#include <uapi/asm/ucontext.h>
/* SMP */
-extern struct thread_info *current_set[NR_CPUS];
-extern struct thread_info *secondary_ti;
+extern struct task_struct *current_set[NR_CPUS];
+extern struct task_struct *secondary_current;
void start_secondary(void *unused);
/* kexec */
diff --git a/arch/powerpc/kernel/head_32.S b/arch/powerpc/kernel/head_32.S
index 44dfd73b2a62..ba0341bd5a00 100644
--- a/arch/powerpc/kernel/head_32.S
+++ b/arch/powerpc/kernel/head_32.S
@@ -842,9 +842,9 @@ __secondary_start:
#endif /* CONFIG_6xx */
/* get current's stack and current */
- lis r1,secondary_ti@ha
- tophys(r1,r1)
- lwz r2,secondary_ti@l(r1)
+ lis r2,secondary_current@ha
+ tophys(r2,r2)
+ lwz r2,secondary_current@l(r2)
tophys(r1,r2)
lwz r1,TASK_STACK(r1)
diff --git a/arch/powerpc/kernel/head_44x.S b/arch/powerpc/kernel/head_44x.S
index 2c7e90f36358..48e4de4dfd0c 100644
--- a/arch/powerpc/kernel/head_44x.S
+++ b/arch/powerpc/kernel/head_44x.S
@@ -1021,8 +1021,8 @@ _GLOBAL(start_secondary_47x)
/* Now we can get our task struct and real stack pointer */
/* Get current's stack and current */
- lis r1,secondary_ti@ha
- lwz r2,secondary_ti@l(r1)
+ lis r2,secondary_current@ha
+ lwz r2,secondary_current@l(r2)
lwz r1,TASK_STACK(r2)
/* Current stack pointer */
diff --git a/arch/powerpc/kernel/head_fsl_booke.S b/arch/powerpc/kernel/head_fsl_booke.S
index b8a2b789677e..0d27bfff52dd 100644
--- a/arch/powerpc/kernel/head_fsl_booke.S
+++ b/arch/powerpc/kernel/head_fsl_booke.S
@@ -1076,8 +1076,8 @@ __secondary_start:
bl call_setup_cpu
/* get current's stack and current */
- lis r1,secondary_ti@ha
- lwz r2,secondary_ti@l(r1)
+ lis r2,secondary_current@ha
+ lwz r2,secondary_current@l(r2)
lwz r1,TASK_STACK(r2)
/* stack */
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index f22fcbeb9898..00193643f0da 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -74,7 +74,7 @@
static DEFINE_PER_CPU(int, cpu_state) = { 0 };
#endif
-struct thread_info *secondary_ti;
+struct task_struct *secondary_current;
DEFINE_PER_CPU(cpumask_var_t, cpu_sibling_map);
DEFINE_PER_CPU(cpumask_var_t, cpu_l2_cache_map);
@@ -644,7 +644,7 @@ void smp_send_stop(void)
}
#endif /* CONFIG_NMI_IPI */
-struct thread_info *current_set[NR_CPUS];
+struct task_struct *current_set[NR_CPUS];
static void smp_store_cpu_info(int id)
{
@@ -724,7 +724,7 @@ void smp_prepare_boot_cpu(void)
paca_ptrs[boot_cpuid]->__current = current;
#endif
set_numa_node(numa_cpu_lookup_table[boot_cpuid]);
- current_set[boot_cpuid] = task_thread_info(current);
+ current_set[boot_cpuid] = current;
}
#ifdef CONFIG_HOTPLUG_CPU
@@ -809,15 +809,13 @@ static bool secondaries_inhibited(void)
static void cpu_idle_thread_init(unsigned int cpu, struct task_struct *idle)
{
- struct thread_info *ti = task_thread_info(idle);
-
#ifdef CONFIG_PPC64
paca_ptrs[cpu]->__current = idle;
paca_ptrs[cpu]->kstack = (unsigned long)task_stack_page(idle) +
THREAD_SIZE - STACK_FRAME_OVERHEAD;
#endif
idle->cpu = cpu;
- secondary_ti = current_set[cpu] = ti;
+ secondary_current = current_set[cpu] = idle;
}
int __cpu_up(unsigned int cpu, struct task_struct *tidle)
--
2.13.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox