* Re: [RESEND 6/7] IB/qib: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
From: Dan Williams @ 2019-03-22 22:15 UTC (permalink / raw)
To: ira.weiny
Cc: Michal Hocko, Linux-sh, Peter Zijlstra, James Hogan,
Heiko Carstens, linux-mips, linux-mm, Rich Felker, Paul Mackerras,
sparclinux, linux-s390, Yoshinori Sato, linux-rdma,
Jason Gunthorpe, Ingo Molnar, John Hubbard, Borislav Petkov,
Thomas Gleixner, netdev@vger.kernel.org,
Linux Kernel Mailing List, Ralf Baechle, Martin Schwidefsky,
Andrew Morton, linuxppc-dev, David S. Miller, Kirill A. Shutemov
In-Reply-To: <20190317183438.2057-7-ira.weiny@intel.com>
On Sun, Mar 17, 2019 at 7:36 PM <ira.weiny@intel.com> wrote:
>
> From: Ira Weiny <ira.weiny@intel.com>
>
> Use the new FOLL_LONGTERM to get_user_pages_fast() to protect against
> FS DAX pages being mapped.
>
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>
Looks good modulo potential __get_user_pages_fast() suggestion.
^ permalink raw reply
* Re: [RESEND 5/7] IB/hfi1: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
From: Dan Williams @ 2019-03-22 22:14 UTC (permalink / raw)
To: ira.weiny
Cc: Michal Hocko, Linux-sh, Peter Zijlstra, James Hogan,
Heiko Carstens, linux-mips, linux-mm, Rich Felker, Paul Mackerras,
sparclinux, linux-s390, Yoshinori Sato, linux-rdma,
Jason Gunthorpe, Ingo Molnar, John Hubbard, Borislav Petkov,
Thomas Gleixner, netdev@vger.kernel.org,
Linux Kernel Mailing List, Ralf Baechle, Martin Schwidefsky,
Andrew Morton, linuxppc-dev, David S. Miller, Kirill A. Shutemov
In-Reply-To: <20190317183438.2057-6-ira.weiny@intel.com>
On Sun, Mar 17, 2019 at 7:36 PM <ira.weiny@intel.com> wrote:
>
> From: Ira Weiny <ira.weiny@intel.com>
>
> Use the new FOLL_LONGTERM to get_user_pages_fast() to protect against
> FS DAX pages being mapped.
>
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> ---
> drivers/infiniband/hw/hfi1/user_pages.c | 6 ++++--
> 1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/infiniband/hw/hfi1/user_pages.c b/drivers/infiniband/hw/hfi1/user_pages.c
> index 78ccacaf97d0..6a7f9cd5a94e 100644
> --- a/drivers/infiniband/hw/hfi1/user_pages.c
> +++ b/drivers/infiniband/hw/hfi1/user_pages.c
> @@ -104,9 +104,11 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t np
> bool writable, struct page **pages)
> {
> int ret;
> + unsigned int gup_flags = writable ? FOLL_WRITE : 0;
Maybe:
unsigned int gup_flags = FOLL_LONGTERM | (writable ? FOLL_WRITE : 0);
?
>
> - ret = get_user_pages_fast(vaddr, npages, writable ? FOLL_WRITE : 0,
> - pages);
> + gup_flags |= FOLL_LONGTERM;
> +
> + ret = get_user_pages_fast(vaddr, npages, gup_flags, pages);
> if (ret < 0)
> return ret;
>
> --
> 2.20.1
>
^ permalink raw reply
* Re: [RESEND 4/7] mm/gup: Add FOLL_LONGTERM capability to GUP fast
From: Dan Williams @ 2019-03-22 22:12 UTC (permalink / raw)
To: ira.weiny
Cc: Michal Hocko, Linux-sh, Peter Zijlstra, James Hogan,
Heiko Carstens, linux-mips, linux-mm, Rich Felker, Paul Mackerras,
sparclinux, linux-s390, Yoshinori Sato, linux-rdma,
Jason Gunthorpe, Ingo Molnar, John Hubbard, Borislav Petkov,
Thomas Gleixner, netdev@vger.kernel.org,
Linux Kernel Mailing List, Ralf Baechle, Martin Schwidefsky,
Andrew Morton, linuxppc-dev, David S. Miller, Kirill A. Shutemov
In-Reply-To: <20190317183438.2057-5-ira.weiny@intel.com>
On Sun, Mar 17, 2019 at 7:36 PM <ira.weiny@intel.com> wrote:
>
> From: Ira Weiny <ira.weiny@intel.com>
>
> DAX pages were previously unprotected from longterm pins when users
> called get_user_pages_fast().
>
> Use the new FOLL_LONGTERM flag to check for DEVMAP pages and fall
> back to regular GUP processing if a DEVMAP page is encountered.
>
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> ---
> mm/gup.c | 29 +++++++++++++++++++++++++----
> 1 file changed, 25 insertions(+), 4 deletions(-)
>
> diff --git a/mm/gup.c b/mm/gup.c
> index 0684a9536207..173db0c44678 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -1600,6 +1600,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
> goto pte_unmap;
>
> if (pte_devmap(pte)) {
> + if (unlikely(flags & FOLL_LONGTERM))
> + goto pte_unmap;
> +
> pgmap = get_dev_pagemap(pte_pfn(pte), pgmap);
> if (unlikely(!pgmap)) {
> undo_dev_pagemap(nr, nr_start, pages);
> @@ -1739,8 +1742,11 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,
> if (!pmd_access_permitted(orig, flags & FOLL_WRITE))
> return 0;
>
> - if (pmd_devmap(orig))
> + if (pmd_devmap(orig)) {
> + if (unlikely(flags & FOLL_LONGTERM))
> + return 0;
> return __gup_device_huge_pmd(orig, pmdp, addr, end, pages, nr);
> + }
>
> refs = 0;
> page = pmd_page(orig) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
> @@ -1777,8 +1783,11 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,
> if (!pud_access_permitted(orig, flags & FOLL_WRITE))
> return 0;
>
> - if (pud_devmap(orig))
> + if (pud_devmap(orig)) {
> + if (unlikely(flags & FOLL_LONGTERM))
> + return 0;
> return __gup_device_huge_pud(orig, pudp, addr, end, pages, nr);
> + }
>
> refs = 0;
> page = pud_page(orig) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
> @@ -2066,8 +2075,20 @@ int get_user_pages_fast(unsigned long start, int nr_pages,
> start += nr << PAGE_SHIFT;
> pages += nr;
>
> - ret = get_user_pages_unlocked(start, nr_pages - nr, pages,
> - gup_flags);
> + if (gup_flags & FOLL_LONGTERM) {
> + down_read(¤t->mm->mmap_sem);
> + ret = __gup_longterm_locked(current, current->mm,
> + start, nr_pages - nr,
> + pages, NULL, gup_flags);
> + up_read(¤t->mm->mmap_sem);
> + } else {
> + /*
> + * retain FAULT_FOLL_ALLOW_RETRY optimization if
> + * possible
> + */
> + ret = get_user_pages_unlocked(start, nr_pages - nr,
> + pages, gup_flags);
I couldn't immediately grok why this path needs to branch on
FOLL_LONGTERM? Won't get_user_pages_unlocked(..., FOLL_LONGTERM) do
the right thing?
^ permalink raw reply
* Re: [RESEND 3/7] mm/gup: Change GUP fast to use flags rather than a write 'bool'
From: Dan Williams @ 2019-03-22 22:05 UTC (permalink / raw)
To: ira.weiny
Cc: Michal Hocko, Linux-sh, Peter Zijlstra, James Hogan,
Heiko Carstens, linux-mips, linux-mm, Rich Felker, Paul Mackerras,
sparclinux, linux-s390, Yoshinori Sato, linux-rdma,
Jason Gunthorpe, Ingo Molnar, John Hubbard, Borislav Petkov,
Thomas Gleixner, netdev@vger.kernel.org,
Linux Kernel Mailing List, Ralf Baechle, Martin Schwidefsky,
Andrew Morton, linuxppc-dev, David S. Miller, Kirill A. Shutemov
In-Reply-To: <20190317183438.2057-4-ira.weiny@intel.com>
On Sun, Mar 17, 2019 at 7:36 PM <ira.weiny@intel.com> wrote:
>
> From: Ira Weiny <ira.weiny@intel.com>
>
> To facilitate additional options to get_user_pages_fast() change the
> singular write parameter to be gup_flags.
>
> This patch does not change any functionality. New functionality will
> follow in subsequent patches.
>
> Some of the get_user_pages_fast() call sites were unchanged because they
> already passed FOLL_WRITE or 0 for the write parameter.
>
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>
>
> ---
> Changes from V1:
> Rebase to current merge tree
> arch/powerpc/mm/mmu_context_iommu.c no longer calls gup_fast
> The gup_longterm was converted in patch 1
>
> arch/mips/mm/gup.c | 11 ++++++-----
> arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 ++--
> arch/powerpc/kvm/e500_mmu.c | 2 +-
> arch/s390/kvm/interrupt.c | 2 +-
> arch/s390/mm/gup.c | 12 ++++++------
> arch/sh/mm/gup.c | 11 ++++++-----
> arch/sparc/mm/gup.c | 9 +++++----
> arch/x86/kvm/paging_tmpl.h | 2 +-
> arch/x86/kvm/svm.c | 2 +-
> drivers/fpga/dfl-afu-dma-region.c | 2 +-
> drivers/gpu/drm/via/via_dmablit.c | 3 ++-
> drivers/infiniband/hw/hfi1/user_pages.c | 3 ++-
> drivers/misc/genwqe/card_utils.c | 2 +-
> drivers/misc/vmw_vmci/vmci_host.c | 2 +-
> drivers/misc/vmw_vmci/vmci_queue_pair.c | 6 ++++--
> drivers/platform/goldfish/goldfish_pipe.c | 3 ++-
> drivers/rapidio/devices/rio_mport_cdev.c | 4 +++-
> drivers/sbus/char/oradax.c | 2 +-
> drivers/scsi/st.c | 3 ++-
> drivers/staging/gasket/gasket_page_table.c | 4 ++--
> drivers/tee/tee_shm.c | 2 +-
> drivers/vfio/vfio_iommu_spapr_tce.c | 3 ++-
> drivers/vhost/vhost.c | 2 +-
> drivers/video/fbdev/pvr2fb.c | 2 +-
> drivers/virt/fsl_hypervisor.c | 2 +-
> drivers/xen/gntdev.c | 2 +-
> fs/orangefs/orangefs-bufmap.c | 2 +-
> include/linux/mm.h | 4 ++--
> kernel/futex.c | 2 +-
> lib/iov_iter.c | 7 +++++--
> mm/gup.c | 10 +++++-----
> mm/util.c | 8 ++++----
> net/ceph/pagevec.c | 2 +-
> net/rds/info.c | 2 +-
> net/rds/rdma.c | 3 ++-
> 35 files changed, 79 insertions(+), 63 deletions(-)
>
> diff --git a/arch/mips/mm/gup.c b/arch/mips/mm/gup.c
> index 0d14e0d8eacf..4c2b4483683c 100644
> --- a/arch/mips/mm/gup.c
> +++ b/arch/mips/mm/gup.c
> @@ -235,7 +235,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,
> * get_user_pages_fast() - pin user pages in memory
> * @start: starting user address
> * @nr_pages: number of pages from start to pin
> - * @write: whether pages will be written to
> + * @gup_flags: flags modifying pin behaviour
> * @pages: array that receives pointers to the pages pinned.
> * Should be at least nr_pages long.
> *
> @@ -247,8 +247,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,
> * requested. If nr_pages is 0 or negative, returns 0. If no pages
> * were pinned, returns -errno.
> */
> -int get_user_pages_fast(unsigned long start, int nr_pages, int write,
> - struct page **pages)
> +int get_user_pages_fast(unsigned long start, int nr_pages,
> + unsigned int gup_flags, struct page **pages)
This looks a tad scary given all related thrash especially when it's
only 1 user that wants to do get_user_page_fast_longterm, right? Maybe
something like the following. Note I explicitly moved the flags to the
end so that someone half paying attention that calls
__get_user_pages_fast will get a compile error if they specify the
args in the same order.
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 76ba638ceda8..c6c743bc2c68 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1505,8 +1505,15 @@ static inline long
get_user_pages_longterm(unsigned long start,
}
#endif /* CONFIG_FS_DAX */
-int get_user_pages_fast(unsigned long start, int nr_pages, int write,
- struct page **pages);
+
+int __get_user_pages_fast(unsigned long start, int nr_pages,
+ struct page **pages, unsigned int gup_flags);
+
+static inline int get_user_pages_fast(unsigned long start, int
nr_pages, int write,
+ struct page **pages)
+{
+ return __get_user_pages_fast(start, nr_pages, pages, write ?
FOLL_WRITE);
+}
/* Container for pinned pfns / pages */
struct frame_vector {
^ permalink raw reply related
* Re: [RESEND 2/7] mm/gup: Change write parameter to flags in fast walk
From: Dan Williams @ 2019-03-22 21:30 UTC (permalink / raw)
To: ira.weiny
Cc: Michal Hocko, Linux-sh, Peter Zijlstra, James Hogan,
Heiko Carstens, linux-mips, linux-mm, Rich Felker, Paul Mackerras,
sparclinux, linux-s390, Yoshinori Sato, linux-rdma,
Jason Gunthorpe, Ingo Molnar, John Hubbard, Borislav Petkov,
Thomas Gleixner, netdev@vger.kernel.org,
Linux Kernel Mailing List, Ralf Baechle, Martin Schwidefsky,
Andrew Morton, linuxppc-dev, David S. Miller, Kirill A. Shutemov
In-Reply-To: <20190317183438.2057-3-ira.weiny@intel.com>
On Sun, Mar 17, 2019 at 7:36 PM <ira.weiny@intel.com> wrote:
>
> From: Ira Weiny <ira.weiny@intel.com>
>
> In order to support more options in the GUP fast walk, change
> the write parameter to flags throughout the call stack.
>
> This patch does not change functionality and passes FOLL_WRITE
> where write was previously used.
>
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>
Looks good,
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
^ permalink raw reply
* Re: [RESEND 1/7] mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM
From: Dan Williams @ 2019-03-22 21:24 UTC (permalink / raw)
To: ira.weiny
Cc: Michal Hocko, Linux-sh, Peter Zijlstra, James Hogan,
Heiko Carstens, Michal Hocko, linux-mm, Rich Felker,
Paul Mackerras, sparclinux, linux-s390, Yoshinori Sato,
linux-rdma, Aneesh Kumar K . V, Jason Gunthorpe, Ingo Molnar,
linux-mips, John Hubbard, Borislav Petkov, Thomas Gleixner,
netdev@vger.kernel.org, Linux Kernel Mailing List, Ralf Baechle,
Martin Schwidefsky, Andrew Morton, linuxppc-dev, David S. Miller,
Kirill A. Shutemov
In-Reply-To: <20190317183438.2057-2-ira.weiny@intel.com>
On Sun, Mar 17, 2019 at 7:36 PM <ira.weiny@intel.com> wrote:
>
> From: Ira Weiny <ira.weiny@intel.com>
>
> Rather than have a separate get_user_pages_longterm() call,
> introduce FOLL_LONGTERM and change the longterm callers to use
> it.
>
> This patch does not change any functionality.
>
> FOLL_LONGTERM can only be supported with get_user_pages() as it
> requires vmas to determine if DAX is in use.
>
> CC: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
> CC: Andrew Morton <akpm@linux-foundation.org>
> CC: Michal Hocko <mhocko@kernel.org>
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>
[..]
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 2d483dbdffc0..6831077d126c 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
[..]
> @@ -2609,6 +2596,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,
> #define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */
> #define FOLL_COW 0x4000 /* internal GUP flag */
> #define FOLL_ANON 0x8000 /* don't do file mappings */
> +#define FOLL_LONGTERM 0x10000 /* mapping is intended for a long term pin */
Let's change this comment to say something like /* mapping lifetime is
indefinite / at the discretion of userspace */, since "longterm is not
well defined.
I think it should also include a /* FIXME: */ to say something about
the havoc a long term pin might wreak on fs and mm code paths.
> static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags)
> {
> diff --git a/mm/gup.c b/mm/gup.c
> index f84e22685aaa..8cb4cff067bc 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -1112,26 +1112,7 @@ long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm,
> }
> EXPORT_SYMBOL(get_user_pages_remote);
>
> -/*
> - * This is the same as get_user_pages_remote(), just with a
> - * less-flexible calling convention where we assume that the task
> - * and mm being operated on are the current task's and don't allow
> - * passing of a locked parameter. We also obviously don't pass
> - * FOLL_REMOTE in here.
> - */
> -long get_user_pages(unsigned long start, unsigned long nr_pages,
> - unsigned int gup_flags, struct page **pages,
> - struct vm_area_struct **vmas)
> -{
> - return __get_user_pages_locked(current, current->mm, start, nr_pages,
> - pages, vmas, NULL,
> - gup_flags | FOLL_TOUCH);
> -}
> -EXPORT_SYMBOL(get_user_pages);
> -
> #if defined(CONFIG_FS_DAX) || defined (CONFIG_CMA)
> -
> -#ifdef CONFIG_FS_DAX
> static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages)
> {
> long i;
> @@ -1150,12 +1131,6 @@ static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages)
> }
> return false;
> }
> -#else
> -static inline bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages)
> -{
> - return false;
> -}
> -#endif
>
> #ifdef CONFIG_CMA
> static struct page *new_non_cma_page(struct page *page, unsigned long private)
> @@ -1209,10 +1184,13 @@ static struct page *new_non_cma_page(struct page *page, unsigned long private)
> return __alloc_pages_node(nid, gfp_mask, 0);
> }
>
> -static long check_and_migrate_cma_pages(unsigned long start, long nr_pages,
> - unsigned int gup_flags,
> +static long check_and_migrate_cma_pages(struct task_struct *tsk,
> + struct mm_struct *mm,
> + unsigned long start,
> + unsigned long nr_pages,
> struct page **pages,
> - struct vm_area_struct **vmas)
> + struct vm_area_struct **vmas,
> + unsigned int gup_flags)
> {
> long i;
> bool drain_allow = true;
> @@ -1268,10 +1246,14 @@ static long check_and_migrate_cma_pages(unsigned long start, long nr_pages,
> putback_movable_pages(&cma_page_list);
> }
> /*
> - * We did migrate all the pages, Try to get the page references again
> - * migrating any new CMA pages which we failed to isolate earlier.
> + * We did migrate all the pages, Try to get the page references
> + * again migrating any new CMA pages which we failed to isolate
> + * earlier.
> */
> - nr_pages = get_user_pages(start, nr_pages, gup_flags, pages, vmas);
> + nr_pages = __get_user_pages_locked(tsk, mm, start, nr_pages,
> + pages, vmas, NULL,
> + gup_flags);
> +
Why did this need to change to __get_user_pages_locked?
> if ((nr_pages > 0) && migrate_allow) {
> drain_allow = true;
> goto check_again;
> @@ -1281,66 +1263,115 @@ static long check_and_migrate_cma_pages(unsigned long start, long nr_pages,
> return nr_pages;
> }
> #else
> -static inline long check_and_migrate_cma_pages(unsigned long start, long nr_pages,
> - unsigned int gup_flags,
> - struct page **pages,
> - struct vm_area_struct **vmas)
> +static long check_and_migrate_cma_pages(struct task_struct *tsk,
> + struct mm_struct *mm,
> + unsigned long start,
> + unsigned long nr_pages,
> + struct page **pages,
> + struct vm_area_struct **vmas,
> + unsigned int gup_flags)
> {
> return nr_pages;
> }
> #endif
>
> /*
> - * This is the same as get_user_pages() in that it assumes we are
> - * operating on the current task's mm, but it goes further to validate
> - * that the vmas associated with the address range are suitable for
> - * longterm elevated page reference counts. For example, filesystem-dax
> - * mappings are subject to the lifetime enforced by the filesystem and
> - * we need guarantees that longterm users like RDMA and V4L2 only
> - * establish mappings that have a kernel enforced revocation mechanism.
> + * __gup_longterm_locked() is a wrapper for __get_uer_pages_locked which
s/uer/user/
> + * allows us to process the FOLL_LONGTERM flag if present.
> + *
> + * FOLL_LONGTERM Checks for either DAX VMAs or PPC CMA regions and either fails
> + * the pin or attempts to migrate the page as appropriate.
> + *
> + * In the filesystem-dax case mappings are subject to the lifetime enforced by
> + * the filesystem and we need guarantees that longterm users like RDMA and V4L2
> + * only establish mappings that have a kernel enforced revocation mechanism.
> + *
> + * In the CMA case pages can't be pinned in a CMA region as this would
> + * unnecessarily fragment that region. So CMA attempts to migrate the page
> + * before pinning.
> *
> * "longterm" == userspace controlled elevated page count lifetime.
> * Contrast this to iov_iter_get_pages() usages which are transient.
Ah, here's the longterm documentation, but if I was a developer
considering whether to use FOLL_LONGTERM or not I would expect to find
the documentation at the flag definition site.
I think it has become more clear since get_user_pages_longterm() was
initially merged that we need to warn people not to use it, or at
least seriously reconsider whether they want an interface to support
indefinite pins.
> */
> -long get_user_pages_longterm(unsigned long start, unsigned long nr_pages,
> - unsigned int gup_flags, struct page **pages,
> - struct vm_area_struct **vmas_arg)
> +static __always_inline long __gup_longterm_locked(struct task_struct *tsk,
...why the __always_inline?
^ permalink raw reply
* Re: [GIT PULL] Please pull powerpc/linux.git powerpc-5.1-3 tag
From: pr-tracker-bot @ 2019-03-22 21:20 UTC (permalink / raw)
To: Michael Ellerman
Cc: malat, linuxppc-dev, mark.cave-ayland, linux-kernel,
Linus Torvalds, ben
In-Reply-To: <87h8bvcbhb.fsf@concordia.ellerman.id.au>
The pull request you sent on Fri, 22 Mar 2019 23:58:08 +1100:
> https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git tags/powerpc-5.1-3
has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/a5ed1e96cafde5ba48638f486bfca0685dc6ddc9
Thank you!
--
Deet-doot-dot, I am a bot.
https://korg.wiki.kernel.org/userdoc/prtracker
^ permalink raw reply
* Re: [PATCH v5 1/3] locking/rwsem: Remove arch specific rwsem files
From: Waiman Long @ 2019-03-22 20:27 UTC (permalink / raw)
To: Davidlohr Bueso, Linus Torvalds
Cc: linux-ia64, Linux-sh list, Peter Zijlstra, Will Deacon,
linux-mips, H. Peter Anvin, sparclinux, linux-riscv, linux-arch,
linux-s390, linux-c6x-dev, linux-hexagon,
the arch/x86 maintainers, Ingo Molnar, uclinux-h8-devel,
linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, openrisc,
Borislav Petkov, Thomas Gleixner, linux-alpha@vger.kernel.org,
linux-parisc, Tim Chen, Linux List Kernel Mailing,
linux-alpha@vger.kernel.org, nios2-dev, Andrew Morton,
linuxppc-dev
In-Reply-To: <20190322193010.azb7rmmmaclhal35@linux-r8p5>
On 03/22/2019 03:30 PM, Davidlohr Bueso wrote:
> On Fri, 22 Mar 2019, Linus Torvalds wrote:
>> Some of them _might_ be performance-critical. There's the one on
>> mmap_sem in the fault handling path, for example. And yes, I'd expect
>> the normal case to very much be "no other readers or writers" for that
>> one.
>
> Yeah, the mmap_sem case in the fault path is really expecting an unlocked
> state. To the point that four archs have added branch predictions, ie:
>
> 92181f190b6 (x86: optimise x86's do_page_fault (C entry point for the
> page fault path))
> b15021d994f (powerpc/mm: Add a bunch of (un)likely annotations to
> do_page_fault)
>
> And using PROFILE_ANNOTATED_BRANCHES shows pretty clearly:
> (without resetting the counters)
>
> correct incorrect % Function File Line
> ------- --------- - -------- ---- ----
> 4603685 34 0 do_user_addr_fault fault.c 1416
> (bootup)
> 382327745 449 0 do_user_addr_fault fault.c
> 1416 (kernel build)
> 399446159 461 0 do_user_addr_fault fault.c
> 1416 (redis benchmark)
>
> It would probably wouldn't harm doing the unlikely() for all archs, or
> alternatively, add likely() to the atomic_long_try_cmpxchg_acquire in
> patch 3 and do it implicitly but maybe that would be less flexible(?)
>
> Thanks,
> Davidlohr
I had used the my lock event counting code to count the number of
contended and uncontended trylocks. I tested both bootup and kernel
build. I think I saw less than 1% were contended, the rests were all
uncontended. That is similar to what you got. I thought I had sent the
data out previously, but I couldn't find the email. That was the main
reason why I took Linus' suggestion to optimize it for the uncontended case.
Thanks,
Longman
^ permalink raw reply
* Re: [PATCH v5 1/3] locking/rwsem: Remove arch specific rwsem files
From: Davidlohr Bueso @ 2019-03-22 19:30 UTC (permalink / raw)
To: Linus Torvalds
Cc: linux-ia64, Linux-sh list, Peter Zijlstra, Will Deacon,
linux-mips, H. Peter Anvin, sparclinux, linux-riscv, linux-arch,
linux-s390, linux-c6x-dev, linux-hexagon,
the arch/x86 maintainers, Ingo Molnar, Waiman Long,
uclinux-h8-devel, linux-xtensa, Arnd Bergmann, linux-um,
linux-m68k, openrisc, Borislav Petkov, Thomas Gleixner,
linux-alpha@vger.kernel.org, linux-parisc, Tim Chen,
Linux List Kernel Mailing, linux-alpha@vger.kernel.org, nios2-dev,
Andrew Morton, linuxppc-dev
In-Reply-To: <CAHk-=wikkO-1f1=FEOEzkSnaDg3yJLP=4Vd59UCuLBztFd0KOw@mail.gmail.com>
On Fri, 22 Mar 2019, Linus Torvalds wrote:
>Some of them _might_ be performance-critical. There's the one on
>mmap_sem in the fault handling path, for example. And yes, I'd expect
>the normal case to very much be "no other readers or writers" for that
>one.
Yeah, the mmap_sem case in the fault path is really expecting an unlocked
state. To the point that four archs have added branch predictions, ie:
92181f190b6 (x86: optimise x86's do_page_fault (C entry point for the page fault path))
b15021d994f (powerpc/mm: Add a bunch of (un)likely annotations to do_page_fault)
And using PROFILE_ANNOTATED_BRANCHES shows pretty clearly:
(without resetting the counters)
correct incorrect % Function File Line
------- --------- - -------- ---- ----
4603685 34 0 do_user_addr_fault fault.c 1416 (bootup)
382327745 449 0 do_user_addr_fault fault.c 1416 (kernel build)
399446159 461 0 do_user_addr_fault fault.c 1416 (redis benchmark)
It would probably wouldn't harm doing the unlikely() for all archs, or
alternatively, add likely() to the atomic_long_try_cmpxchg_acquire in
patch 3 and do it implicitly but maybe that would be less flexible(?)
Thanks,
Davidlohr
^ permalink raw reply
* Re: powerpc32 boot crash in 5.1-rc1
From: LEROY Christophe @ 2019-03-22 19:30 UTC (permalink / raw)
To: Meelis Roos; +Cc: linuxppc-dev, linux-kernel
In-Reply-To: <6c3bfa43-4b11-ab27-fcc6-3732ccbed304@linux.ee>
Meelis Roos <mroos@linux.ee> a écrit :
>>> While 5.0.0 worked fine on my PowerMac G4, 5.0 + git (unknown rev
>>> as of Mar 13), 5.0.0-11520-gf261c4e and todays git all fail to boot.
>>>
>>> The problem seems to be in page fault handler in load_elf_binary()
>>> of init process.
>>
>> The patch at https://patchwork.ozlabs.org/patch/1053385/ should fix it
>
> Tested it - yes, it fixes the boot.
Thanks for testing. It will be merged in 5.1-rc2
Christophe
>
> --
> Meelis Roos <mroos@linux.ee>
^ permalink raw reply
* Re: [PATCH v3] powerpc/64: Fix memcmp reading past the end of src/dest
From: Segher Boessenkool @ 2019-03-22 18:29 UTC (permalink / raw)
To: Michael Ellerman; +Cc: linuxppc-dev, chandan
In-Reply-To: <20190322123724.28435-1-mpe@ellerman.id.au>
On Fri, Mar 22, 2019 at 11:37:24PM +1100, Michael Ellerman wrote:
> .Lcmp_rest_lt8bytes:
> - /* Here we have only less than 8 bytes to compare with. at least s1
> - * Address is aligned with 8 bytes.
> - * The next double words are load and shift right with appropriate
> - * bits.
> + /*
> + * Here we have less than 8 bytes to compare. At least s1 is aligned to
> + * 8 bytes, but s2 may not be. We must make sure s2 + 8 doesn't cross a
"s2 + 7"? The code is fine though (bgt, not bge).
> + * page boundary, otherwise we might read past the end of the buffer and
> + * trigger a page fault. We use 4K as the conservative minimum page
> + * size. If we detect that case we go to the byte-by-byte loop.
> + *
> + * Otherwise the next double word is loaded from s1 and s2, and shifted
> + * right to compare the appropriate bits.
> */
> + clrldi r6,r4,(64-12) // r6 = r4 & 0xfff
You can just write
rlwinm r6,r4,0,0x0fff
if that is clearer? Or do you still want a comment with that :-)
> + cmpdi r6,0xff8
> + bgt .Lshort
Reviewed-by: Segher Boessenkool <segher@kernel.crashing.org>
Segher
^ permalink raw reply
* Re: [PATCH v5 3/3] locking/rwsem: Optimize down_read_trylock()
From: Waiman Long @ 2019-03-22 17:41 UTC (permalink / raw)
To: Russell King - ARM Linux admin
Cc: linux-ia64, linux-sh, Peter Zijlstra, Will Deacon, linux-kernel,
H. Peter Anvin, sparclinux, linux-riscv, linux-arch, linux-s390,
Davidlohr Bueso, linux-c6x-dev, linux-hexagon, x86, Ingo Molnar,
uclinux-h8-devel, linux-xtensa, Arnd Bergmann, linux-um,
linuxppc-dev, linux-m68k, openrisc, Borislav Petkov,
Thomas Gleixner, linux-arm-kernel, linux-parisc, Linus Torvalds,
linux-mips, linux-alpha, nios2-dev, Andrew Morton, Tim Chen
In-Reply-To: <20190322172501.3nbjw6e2wqsaisgw@shell.armlinux.org.uk>
On 03/22/2019 01:25 PM, Russell King - ARM Linux admin wrote:
> On Fri, Mar 22, 2019 at 10:30:08AM -0400, Waiman Long wrote:
>> Modify __down_read_trylock() to optimize for an unlocked rwsem and make
>> it generate slightly better code.
>>
>> Before this patch, down_read_trylock:
>>
>> 0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5>
>> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24>
>> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx
>> 0x000000000000000b <+11>: mov %rdx,%rax
>> 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi)
>> 0x0000000000000013 <+19>: cmp %rax,%rdx
>> 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35>
>> 0x0000000000000018 <+24>: mov (%rdi),%rdx
>> 0x000000000000001b <+27>: test %rdx,%rdx
>> 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7>
>> 0x0000000000000020 <+32>: xor %eax,%eax
>> 0x0000000000000022 <+34>: retq
>> 0x0000000000000023 <+35>: mov %gs:0x0,%rax
>> 0x000000000000002c <+44>: or $0x3,%rax
>> 0x0000000000000030 <+48>: mov %rax,0x20(%rdi)
>> 0x0000000000000034 <+52>: mov $0x1,%eax
>> 0x0000000000000039 <+57>: retq
>>
>> After patch, down_read_trylock:
>>
>> 0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5>
>> 0x0000000000000005 <+5>: xor %eax,%eax
>> 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx
>> 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi)
>> 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41>
>> 0x0000000000000012 <+18>: mov %gs:0x0,%rax
>> 0x000000000000001b <+27>: or $0x3,%rax
>> 0x000000000000001f <+31>: mov %rax,0x20(%rdi)
>> 0x0000000000000023 <+35>: mov $0x1,%eax
>> 0x0000000000000028 <+40>: retq
>> 0x0000000000000029 <+41>: test %rax,%rax
>> 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7>
>> 0x000000000000002e <+46>: xor %eax,%eax
>> 0x0000000000000030 <+48>: retq
>>
>> By using a rwsem microbenchmark, the down_read_trylock() rate (with a
>> load of 10 to lengthen the lock critical section) on a x86-64 system
>> before and after the patch were:
>>
>> Before Patch After Patch
>> # of Threads rlock rlock
>> ------------ ----- -----
>> 1 14,496 14,716
>> 2 8,644 8,453
>> 4 6,799 6,983
>> 8 5,664 7,190
>>
>> On a ARM64 system, the performance results were:
>>
>> Before Patch After Patch
>> # of Threads rlock rlock
>> ------------ ----- -----
>> 1 23,676 24,488
>> 2 7,697 9,502
>> 4 4,945 3,440
>> 8 2,641 1,603
>>
>> For the uncontended case (1 thread), the new down_read_trylock() is a
>> little bit faster. For the contended cases, the new down_read_trylock()
>> perform pretty well in x86-64, but performance degrades at high
>> contention level on ARM64.
> So, 70% for 4 threads, 61% for 4 threads - does this trend
> continue tailing off as the number of threads (and cores)
> increase?
>
I didn't try higher number of contending threads. I won't worry too much
about contention as trylock is a one-off event. The chance of having
more than one trylock happening simultaneously is very small.
Cheers,
Longman
^ permalink raw reply
* Re: [PATCH v2 3/7] ocxl: Create a clear delineation between ocxl backend & frontend
From: Frederic Barrat @ 2019-03-22 17:38 UTC (permalink / raw)
To: Alastair D'Silva, alastair
Cc: Greg Kroah-Hartman, linuxppc-dev, Arnd Bergmann, Andrew Donnellan,
linux-kernel
In-Reply-To: <20190320050901.310-4-alastair@au1.ibm.com>
[-- Attachment #1: Type: text/plain, Size: 30718 bytes --]
Hi Alastair,
I'm still seeing problems with the handling of the info structure and
ref counting on the AFU. To make things easier, I'm attaching a patch.
There's also a bunch of other review comments in the patch, check for
the comments with the "fxb" marker.
I've played a bit with driver unload and force unbinds and the free
operations were happening as expected. We're getting there!
Fred
Le 20/03/2019 à 06:08, Alastair D'Silva a écrit :
> From: Alastair D'Silva <alastair@d-silva.org>
>
> The OCXL driver contains both frontend code for interacting with userspace,
> as well as backend code for interacting with the hardware.
>
> This patch separates the backend code from the frontend so that it can be
> used by other device drivers that communicate via OpenCAPI.
>
> Relocate dev, cdev & sysfs files to the frontend code to allow external
> drivers to maintain their own devices.
>
> Reference counting on the device in the backend is replaced with kref
> counting.
>
> Move file & sysfs layer initialisation from core.c (backend) to
> pci.c (frontend).
>
> Create an ocxl_function oriented interface for initing devices &
> enumerating AFUs.
>
> Signed-off-by: Alastair D'Silva <alastair@d-silva.org>
> ---
> drivers/misc/ocxl/context.c | 2 +-
> drivers/misc/ocxl/core.c | 205 +++++++++++++++++++-----------
> drivers/misc/ocxl/file.c | 125 ++++++++++++------
> drivers/misc/ocxl/ocxl_internal.h | 39 +++---
> drivers/misc/ocxl/pci.c | 61 ++++-----
> drivers/misc/ocxl/sysfs.c | 58 +++++----
> include/misc/ocxl.h | 121 ++++++++++++++++--
> 7 files changed, 416 insertions(+), 195 deletions(-)
>
> diff --git a/drivers/misc/ocxl/context.c b/drivers/misc/ocxl/context.c
> index 3498a0199bde..371ef17bba33 100644
> --- a/drivers/misc/ocxl/context.c
> +++ b/drivers/misc/ocxl/context.c
> @@ -238,7 +238,7 @@ int ocxl_context_detach(struct ocxl_context *ctx)
> }
> rc = ocxl_link_remove_pe(ctx->afu->fn->link, ctx->pasid);
> if (rc) {
> - dev_warn(&ctx->afu->dev,
> + dev_warn(&dev->dev,
> "Couldn't remove PE entry cleanly: %d\n", rc);
> }
> return 0;
> diff --git a/drivers/misc/ocxl/core.c b/drivers/misc/ocxl/core.c
> index 2fd0c700e8a0..c632ec372342 100644
> --- a/drivers/misc/ocxl/core.c
> +++ b/drivers/misc/ocxl/core.c
> @@ -13,16 +13,6 @@ static void ocxl_fn_put(struct ocxl_fn *fn)
> put_device(&fn->dev);
> }
>
> -struct ocxl_afu *ocxl_afu_get(struct ocxl_afu *afu)
> -{
> - return (get_device(&afu->dev) == NULL) ? NULL : afu;
> -}
> -
> -void ocxl_afu_put(struct ocxl_afu *afu)
> -{
> - put_device(&afu->dev);
> -}
> -
> static struct ocxl_afu *alloc_afu(struct ocxl_fn *fn)
> {
> struct ocxl_afu *afu;
> @@ -31,6 +21,7 @@ static struct ocxl_afu *alloc_afu(struct ocxl_fn *fn)
> if (!afu)
> return NULL;
>
> + kref_init(&afu->kref);
> mutex_init(&afu->contexts_lock);
> mutex_init(&afu->afu_control_lock);
> idr_init(&afu->contexts_idr);
> @@ -39,32 +30,26 @@ static struct ocxl_afu *alloc_afu(struct ocxl_fn *fn)
> return afu;
> }
>
> -static void free_afu(struct ocxl_afu *afu)
> +static void afu_release(struct kref *kref)
> {
> + struct ocxl_afu *afu = container_of(kref, struct ocxl_afu, kref);
> +
> idr_destroy(&afu->contexts_idr);
> ocxl_fn_put(afu->fn);
> kfree(afu);
> }
>
> -static void free_afu_dev(struct device *dev)
> +void ocxl_afu_get(struct ocxl_afu *afu)
> {
> - struct ocxl_afu *afu = to_ocxl_afu(dev);
> -
> - ocxl_unregister_afu(afu);
> - free_afu(afu);
> + kref_get(&afu->kref);
> }
> +EXPORT_SYMBOL_GPL(ocxl_afu_get);
>
> -static int set_afu_device(struct ocxl_afu *afu, const char *location)
> +void ocxl_afu_put(struct ocxl_afu *afu)
> {
> - struct ocxl_fn *fn = afu->fn;
> - int rc;
> -
> - afu->dev.parent = &fn->dev;
> - afu->dev.release = free_afu_dev;
> - rc = dev_set_name(&afu->dev, "%s.%s.%hhu", afu->config.name, location,
> - afu->config.idx);
> - return rc;
> + kref_put(&afu->kref, afu_release);
> }
> +EXPORT_SYMBOL_GPL(ocxl_afu_put);
>
> static int assign_afu_actag(struct ocxl_afu *afu)
> {
> @@ -233,27 +218,25 @@ static int configure_afu(struct ocxl_afu *afu, u8 afu_idx, struct pci_dev *dev)
> if (rc)
> return rc;
>
> - rc = set_afu_device(afu, dev_name(&dev->dev));
> - if (rc)
> - return rc;
> -
> rc = assign_afu_actag(afu);
> if (rc)
> return rc;
>
> rc = assign_afu_pasid(afu);
> - if (rc) {
> - reclaim_afu_actag(afu);
> - return rc;
> - }
> + if (rc)
> + goto err_free_actag;
>
> rc = map_mmio_areas(afu);
> - if (rc) {
> - reclaim_afu_pasid(afu);
> - reclaim_afu_actag(afu);
> - return rc;
> - }
> + if (rc)
> + goto err_free_pasid;
> +
> return 0;
> +
> +err_free_pasid:
> + reclaim_afu_pasid(afu);
> +err_free_actag:
> + reclaim_afu_actag(afu);
> + return rc;
> }
>
> static void deconfigure_afu(struct ocxl_afu *afu)
> @@ -265,16 +248,8 @@ static void deconfigure_afu(struct ocxl_afu *afu)
>
> static int activate_afu(struct pci_dev *dev, struct ocxl_afu *afu)
> {
> - int rc;
> -
> ocxl_config_set_afu_state(dev, afu->config.dvsec_afu_control_pos, 1);
> - /*
> - * Char device creation is the last step, as processes can
> - * call our driver immediately, so all our inits must be finished.
> - */
> - rc = ocxl_create_cdev(afu);
> - if (rc)
> - return rc;
> +
> return 0;
> }
>
> @@ -282,11 +257,10 @@ static void deactivate_afu(struct ocxl_afu *afu)
> {
> struct pci_dev *dev = to_pci_dev(afu->fn->dev.parent);
>
> - ocxl_destroy_cdev(afu);
> ocxl_config_set_afu_state(dev, afu->config.dvsec_afu_control_pos, 0);
> }
>
> -int init_afu(struct pci_dev *dev, struct ocxl_fn *fn, u8 afu_idx)
> +static int init_afu(struct pci_dev *dev, struct ocxl_fn *fn, u8 afu_idx)
> {
> int rc;
> struct ocxl_afu *afu;
> @@ -297,41 +271,29 @@ int init_afu(struct pci_dev *dev, struct ocxl_fn *fn, u8 afu_idx)
>
> rc = configure_afu(afu, afu_idx, dev);
> if (rc) {
> - free_afu(afu);
> + ocxl_afu_put(afu);
> return rc;
> }
>
> - rc = ocxl_register_afu(afu);
> - if (rc)
> - goto err;
> -
> - rc = ocxl_sysfs_add_afu(afu);
> - if (rc)
> - goto err;
> -
> rc = activate_afu(dev, afu);
> - if (rc)
> - goto err_sys;
> + if (rc) {
> + deconfigure_afu(afu);
> + ocxl_afu_put(afu);
> + return rc;
> + }
>
> list_add_tail(&afu->list, &fn->afu_list);
> - return 0;
>
> -err_sys:
> - ocxl_sysfs_remove_afu(afu);
> -err:
> - deconfigure_afu(afu);
> - device_unregister(&afu->dev);
> - return rc;
> + return 0;
> }
>
> -void remove_afu(struct ocxl_afu *afu)
> +static void remove_afu(struct ocxl_afu *afu)
> {
> list_del(&afu->list);
> ocxl_context_detach_all(afu);
> deactivate_afu(afu);
> - ocxl_sysfs_remove_afu(afu);
> deconfigure_afu(afu);
> - device_unregister(&afu->dev);
> + ocxl_afu_put(afu); // matches the implicit get in alloc_afu
> }
>
> static struct ocxl_fn *alloc_function(void)
> @@ -358,7 +320,7 @@ static void free_function(struct ocxl_fn *fn)
>
> static void free_function_dev(struct device *dev)
> {
> - struct ocxl_fn *fn = to_ocxl_function(dev);
> + struct ocxl_fn *fn = container_of(dev, struct ocxl_fn, dev);
>
> free_function(fn);
> }
> @@ -372,7 +334,6 @@ static int set_function_device(struct ocxl_fn *fn, struct pci_dev *dev)
> rc = dev_set_name(&fn->dev, "ocxlfn.%s", dev_name(&dev->dev));
> if (rc)
> return rc;
> - pci_set_drvdata(dev, fn);
> return 0;
> }
>
> @@ -490,7 +451,7 @@ static void deconfigure_function(struct ocxl_fn *fn)
> pci_disable_device(dev);
> }
>
> -struct ocxl_fn *init_function(struct pci_dev *dev)
> +static struct ocxl_fn *init_function(struct pci_dev *dev)
> {
> struct ocxl_fn *fn;
> int rc;
> @@ -514,8 +475,104 @@ struct ocxl_fn *init_function(struct pci_dev *dev)
> return fn;
> }
>
> -void remove_function(struct ocxl_fn *fn)
> +// Device detection & initialisation
> +
> +struct ocxl_fn *ocxl_function_open(struct pci_dev *dev)
> +{
> + int rc, afu_count = 0;
> + u8 afu;
> + struct ocxl_fn *fn;
> +
> + if (!radix_enabled()) {
> + dev_err(&dev->dev, "Unsupported memory model (hash)\n");
> + return ERR_PTR(-ENODEV);
> + }
> +
> + fn = init_function(dev);
> + if (IS_ERR(fn)) {
> + dev_err(&dev->dev, "function init failed: %li\n",
> + PTR_ERR(fn));
> + return fn;
> + }
> +
> + for (afu = 0; afu <= fn->config.max_afu_index; afu++) {
> + rc = ocxl_config_check_afu_index(dev, &fn->config, afu);
> + if (rc > 0) {
> + rc = init_afu(dev, fn, afu);
> + if (rc) {
> + dev_err(&dev->dev,
> + "Can't initialize AFU index %d\n", afu);
> + continue;
> + }
> + afu_count++;
> + }
> + }
> + dev_info(&dev->dev, "%d AFU(s) configured\n", afu_count);
> + return fn;
> +}
> +EXPORT_SYMBOL_GPL(ocxl_function_open);
> +
> +struct list_head *ocxl_function_afu_list(struct ocxl_fn *fn)
> +{
> + return &fn->afu_list;
> +}
> +EXPORT_SYMBOL_GPL(ocxl_function_afu_list);
> +
> +struct ocxl_afu *ocxl_function_fetch_afu(struct ocxl_fn *fn, u8 afu_idx)
> +{
> + struct ocxl_afu *afu;
> +
> + list_for_each_entry(afu, &fn->afu_list, list) {
> + if (afu->config.idx == afu_idx)
> + return afu;
> + }
> +
> + return NULL;
> +}
> +EXPORT_SYMBOL_GPL(ocxl_function_fetch_afu);
> +
> +const struct ocxl_fn_config *ocxl_function_config(struct ocxl_fn *fn)
> {
> + return &fn->config;
> +}
> +EXPORT_SYMBOL_GPL(ocxl_function_config);
> +
> +void ocxl_function_close(struct ocxl_fn *fn)
> +{
> + struct ocxl_afu *afu, *tmp;
> +
> + list_for_each_entry_safe(afu, tmp, &fn->afu_list, list) {
> + if (afu->private && afu->private_release)
> + afu->private_release(afu->private);
> + remove_afu(afu);
> + }
> +
> deconfigure_function(fn);
> device_unregister(&fn->dev);
> }
> +EXPORT_SYMBOL_GPL(ocxl_function_close);
> +
> +// AFU Metadata
> +
> +struct ocxl_afu_config *ocxl_afu_config(struct ocxl_afu *afu)
> +{
> + return &afu->config;
> +}
> +EXPORT_SYMBOL_GPL(ocxl_afu_config);
> +
> +void ocxl_afu_set_private(struct ocxl_afu *afu, void *private,
> + void (*private_release)(void *private))
> +{
> + afu->private = private;
> + afu->private_release = private_release;
> +}
> +EXPORT_SYMBOL_GPL(ocxl_afu_set_private);
> +
> +void *ocxl_afu_get_private(struct ocxl_afu *afu)
> +{
> + if (afu)
> + return afu->private;
> +
> + return NULL;
> +}
> +EXPORT_SYMBOL_GPL(ocxl_afu_get_private);
> diff --git a/drivers/misc/ocxl/file.c b/drivers/misc/ocxl/file.c
> index 009e09b7ded5..1f17f8706e29 100644
> --- a/drivers/misc/ocxl/file.c
> +++ b/drivers/misc/ocxl/file.c
> @@ -17,12 +17,10 @@ static struct class *ocxl_class;
> static struct mutex minors_idr_lock;
> static struct idr minors_idr;
>
> -static struct ocxl_afu *find_and_get_afu(dev_t devno)
> +static struct ocxl_file_info *find_file_info(dev_t devno)
> {
> - struct ocxl_afu *afu;
> - int afu_minor;
> + struct ocxl_file_info *info;
>
> - afu_minor = MINOR(devno);
> /*
> * We don't declare an RCU critical section here, as our AFU
> * is protected by a reference counter on the device. By the time the
> @@ -30,56 +28,52 @@ static struct ocxl_afu *find_and_get_afu(dev_t devno)
> * the device is already at 0, so no user API will access that AFU and
> * this function can't return it.
> */
> - afu = idr_find(&minors_idr, afu_minor);
> - if (afu)
> - ocxl_afu_get(afu);
> - return afu;
> + info = idr_find(&minors_idr, MINOR(devno));
> + return info;
> }
>
> -static int allocate_afu_minor(struct ocxl_afu *afu)
> +static int allocate_minor(struct ocxl_file_info *info)
> {
> int minor;
>
> mutex_lock(&minors_idr_lock);
> - minor = idr_alloc(&minors_idr, afu, 0, OCXL_NUM_MINORS, GFP_KERNEL);
> + minor = idr_alloc(&minors_idr, info, 0, OCXL_NUM_MINORS, GFP_KERNEL);
> mutex_unlock(&minors_idr_lock);
> return minor;
> }
>
> -static void free_afu_minor(struct ocxl_afu *afu)
> +static void free_minor(struct ocxl_file_info *info)
> {
> mutex_lock(&minors_idr_lock);
> - idr_remove(&minors_idr, MINOR(afu->dev.devt));
> + idr_remove(&minors_idr, MINOR(info->dev.devt));
> mutex_unlock(&minors_idr_lock);
> }
>
> static int afu_open(struct inode *inode, struct file *file)
> {
> - struct ocxl_afu *afu;
> + struct ocxl_file_info *info;
> struct ocxl_context *ctx;
> int rc;
>
> pr_debug("%s for device %x\n", __func__, inode->i_rdev);
>
> - afu = find_and_get_afu(inode->i_rdev);
> - if (!afu)
> + info = find_file_info(inode->i_rdev);
> + if (!info)
> return -ENODEV;
>
> ctx = ocxl_context_alloc();
> if (!ctx) {
> rc = -ENOMEM;
> - goto put_afu;
> + goto err;
> }
>
> - rc = ocxl_context_init(ctx, afu, inode->i_mapping);
> + rc = ocxl_context_init(ctx, info->afu, inode->i_mapping);
> if (rc)
> - goto put_afu;
> + goto err;
> file->private_data = ctx;
> - ocxl_afu_put(afu);
> return 0;
>
> -put_afu:
> - ocxl_afu_put(afu);
> +err:
> return rc;
> }
>
> @@ -204,11 +198,16 @@ static long afu_ioctl(struct file *file, unsigned int cmd,
> struct ocxl_ioctl_irq_fd irq_fd;
> u64 irq_offset;
> long rc;
> + bool closed;
>
> pr_debug("%s for context %d, command %s\n", __func__, ctx->pasid,
> CMD_STR(cmd));
>
> - if (ctx->status == CLOSED)
> + mutex_lock(&ctx->status_mutex);
> + closed = (ctx->status == CLOSED);
> + mutex_unlock(&ctx->status_mutex);
> +
> + if (closed)
> return -EIO;
>
> switch (cmd) {
> @@ -468,39 +467,83 @@ static const struct file_operations ocxl_afu_fops = {
> .release = afu_release,
> };
>
> -int ocxl_create_cdev(struct ocxl_afu *afu)
> +// Called when there are no more consumers of the AFU
> +static void ocxl_file_release(void *private)
> {
> - int rc;
> + struct ocxl_file_info *info = private;
>
> - cdev_init(&afu->cdev, &ocxl_afu_fops);
> - rc = cdev_add(&afu->cdev, afu->dev.devt, 1);
> - if (rc) {
> - dev_err(&afu->dev, "Unable to add afu char device: %d\n", rc);
> - return rc;
> - }
> - return 0;
> + ocxl_sysfs_unregister_afu(info->afu);
> + device_unregister(&info->dev);
> +
> + free_minor(info);
> }
>
> -void ocxl_destroy_cdev(struct ocxl_afu *afu)
> +// Free the info struct
> +static void info_release(struct device *dev)
> {
> - cdev_del(&afu->cdev);
> + struct ocxl_file_info *info = container_of(dev, struct ocxl_file_info, dev);
> + kfree(info);
> }
>
> -int ocxl_register_afu(struct ocxl_afu *afu)
> +int ocxl_file_register_afu(struct ocxl_afu *afu)
> {
> int minor;
> + int rc;
> + struct ocxl_file_info *info;
> + struct ocxl_fn *fn = afu->fn;
> + struct pci_dev *pci_dev = to_pci_dev(fn->dev.parent);
> +
> + info = kzalloc(sizeof(*info), GFP_KERNEL);
> + if (info == NULL)
> + return -ENOMEM;
> +
> + info->afu = afu;
>
> - minor = allocate_afu_minor(afu);
> - if (minor < 0)
> + minor = allocate_minor(info);
> + if (minor < 0) {
> + kfree(info);
> return minor;
> - afu->dev.devt = MKDEV(MAJOR(ocxl_dev), minor);
> - afu->dev.class = ocxl_class;
> - return device_register(&afu->dev);
> + }
> +
> + info->dev.parent = &fn->dev;
> + info->dev.devt = MKDEV(MAJOR(ocxl_dev), minor);
> + info->dev.class = ocxl_class;
> + info->dev.release = info_release;
> +
> + ocxl_afu_set_private(afu, info, ocxl_file_release);
> +
> + rc = dev_set_name(&info->dev, "%s.%s.%hhu",
> + afu->config.name, dev_name(&pci_dev->dev), afu->config.idx);
> + if (rc)
> + return rc;
> +
> + rc = device_register(&info->dev);
> + if (rc)
> + return rc;
> +
> + return ocxl_sysfs_register_afu(afu);
> }
>
> -void ocxl_unregister_afu(struct ocxl_afu *afu)
> +int ocxl_file_make_visible(struct ocxl_afu *afu)
> {
> - free_afu_minor(afu);
> + int rc;
> + struct ocxl_file_info *info = ocxl_afu_get_private(afu);
> +
> + cdev_init(&info->cdev, &ocxl_afu_fops);
> + rc = cdev_add(&info->cdev, info->dev.devt, 1);
> + if (rc) {
> + dev_err(&info->dev, "Unable to add afu char device: %d\n", rc);
> + return rc;
> + }
> +
> + return 0;
> +}
> +
> +void ocxl_file_make_invisible(struct ocxl_afu *afu)
> +{
> + struct ocxl_file_info *info = ocxl_afu_get_private(afu);
> +
> + cdev_del(&info->cdev);
> }
>
> static char *ocxl_devnode(struct device *dev, umode_t *mode)
> diff --git a/drivers/misc/ocxl/ocxl_internal.h b/drivers/misc/ocxl/ocxl_internal.h
> index 81086534dab5..05930a29f606 100644
> --- a/drivers/misc/ocxl/ocxl_internal.h
> +++ b/drivers/misc/ocxl/ocxl_internal.h
> @@ -11,9 +11,6 @@
> #define MAX_IRQ_PER_LINK 2000
> #define MAX_IRQ_PER_CONTEXT MAX_IRQ_PER_LINK
>
> -#define to_ocxl_function(d) container_of(d, struct ocxl_fn, dev)
> -#define to_ocxl_afu(d) container_of(d, struct ocxl_afu, dev)
> -
> extern struct pci_driver ocxl_pci_driver;
>
> struct ocxl_fn {
> @@ -30,11 +27,17 @@ struct ocxl_fn {
> void *link;
> };
>
> +struct ocxl_file_info {
> + struct ocxl_afu *afu;
> + struct device dev;
> + struct cdev cdev;
> + struct bin_attribute attr_global_mmio;
> +};
> +
> struct ocxl_afu {
> + struct kref kref;
> struct ocxl_fn *fn;
> struct list_head list;
> - struct device dev;
> - struct cdev cdev;
> struct ocxl_afu_config config;
> int pasid_base;
> int pasid_count; /* opened contexts */
> @@ -48,7 +51,9 @@ struct ocxl_afu {
> u64 irq_base_offset;
> void __iomem *global_mmio_ptr;
> u64 pp_mmio_start;
> - struct bin_attribute attr_global_mmio;
> + void *private;
> + void (*private_release)(void *private);
> +
> };
>
> enum ocxl_context_status {
> @@ -91,13 +96,11 @@ struct ocxl_process_element {
> __be32 software_state;
> };
>
> -struct ocxl_afu *ocxl_afu_get(struct ocxl_afu *afu);
> -void ocxl_afu_put(struct ocxl_afu *afu);
> -
> int ocxl_create_cdev(struct ocxl_afu *afu);
> void ocxl_destroy_cdev(struct ocxl_afu *afu);
> -int ocxl_register_afu(struct ocxl_afu *afu);
> -void ocxl_unregister_afu(struct ocxl_afu *afu);
> +int ocxl_file_register_afu(struct ocxl_afu *afu);
> +int ocxl_file_make_visible(struct ocxl_afu *afu);
> +void ocxl_file_make_invisible(struct ocxl_afu *afu);
>
> int ocxl_file_init(void);
> void ocxl_file_exit(void);
> @@ -140,8 +143,8 @@ int ocxl_context_detach(struct ocxl_context *ctx);
> void ocxl_context_detach_all(struct ocxl_afu *afu);
> void ocxl_context_free(struct ocxl_context *ctx);
>
> -int ocxl_sysfs_add_afu(struct ocxl_afu *afu);
> -void ocxl_sysfs_remove_afu(struct ocxl_afu *afu);
> +int ocxl_sysfs_register_afu(struct ocxl_afu *afu);
> +void ocxl_sysfs_unregister_afu(struct ocxl_afu *afu);
>
> int ocxl_afu_irq_alloc(struct ocxl_context *ctx, u64 *irq_offset);
> int ocxl_afu_irq_free(struct ocxl_context *ctx, u64 irq_offset);
> @@ -150,9 +153,11 @@ int ocxl_afu_irq_set_fd(struct ocxl_context *ctx, u64 irq_offset,
> int eventfd);
> u64 ocxl_afu_irq_get_addr(struct ocxl_context *ctx, u64 irq_offset);
>
> -struct ocxl_fn *init_function(struct pci_dev *dev);
> -void remove_function(struct ocxl_fn *fn);
> -int init_afu(struct pci_dev *dev, struct ocxl_fn *fn, u8 afu_idx);
> -void remove_afu(struct ocxl_afu *afu);
> +/**
> + * Free an AFU
> + *
> + * afu: The AFU to free
> + */
> +void ocxl_free_afu(struct ocxl_afu *afu);
>
> #endif /* _OCXL_INTERNAL_H_ */
> diff --git a/drivers/misc/ocxl/pci.c b/drivers/misc/ocxl/pci.c
> index 5926c863716c..a1ed2bb02e4b 100644
> --- a/drivers/misc/ocxl/pci.c
> +++ b/drivers/misc/ocxl/pci.c
> @@ -16,47 +16,48 @@ MODULE_DEVICE_TABLE(pci, ocxl_pci_tbl);
>
> static int ocxl_probe(struct pci_dev *dev, const struct pci_device_id *id)
> {
> - int rc, afu_count = 0;
> - u8 afu;
> + int rc;
> + struct ocxl_afu *afu, *tmp;
> struct ocxl_fn *fn;
> + struct list_head *afu_list;
>
> - if (!radix_enabled()) {
> - dev_err(&dev->dev, "Unsupported memory model (hash)\n");
> - return -ENODEV;
> - }
> -
> - fn = init_function(dev);
> - if (IS_ERR(fn)) {
> - dev_err(&dev->dev, "function init failed: %li\n",
> - PTR_ERR(fn));
> + fn = ocxl_function_open(dev);
> + if (IS_ERR(fn))
> return PTR_ERR(fn);
> - }
>
> - for (afu = 0; afu <= fn->config.max_afu_index; afu++) {
> - rc = ocxl_config_check_afu_index(dev, &fn->config, afu);
> - if (rc > 0) {
> - rc = init_afu(dev, fn, afu);
> - if (rc) {
> - dev_err(&dev->dev,
> - "Can't initialize AFU index %d\n", afu);
> - continue;
> - }
> - afu_count++;
> - }
> + pci_set_drvdata(dev, fn);
> +
> + afu_list = ocxl_function_afu_list(fn);
> +
> + list_for_each_entry_safe(afu, tmp, afu_list, list) {
> + rc = ocxl_file_register_afu(afu);
> + if (rc)
> + continue;
> +
> + rc = ocxl_file_make_visible(afu);
> + if (rc)
> + continue;
> +
> + // Defer error cleanup until the device is removed
> }
> - dev_info(&dev->dev, "%d AFU(s) configured\n", afu_count);
> +
> return 0;
> }
>
> -static void ocxl_remove(struct pci_dev *dev)
> +void ocxl_remove(struct pci_dev *dev)
> {
> - struct ocxl_afu *afu, *tmp;
> - struct ocxl_fn *fn = pci_get_drvdata(dev);
> + struct ocxl_fn *fn;
> + struct ocxl_afu *afu;
> + struct list_head *afu_list;
> +
> + fn = pci_get_drvdata(dev);
> + afu_list = ocxl_function_afu_list(fn);
>
> - list_for_each_entry_safe(afu, tmp, &fn->afu_list, list) {
> - remove_afu(afu);
> + list_for_each_entry(afu, afu_list, list) {
> + ocxl_file_make_invisible(afu);
> }
> - remove_function(fn);
> +
> + ocxl_function_close(fn);
> }
>
> struct pci_driver ocxl_pci_driver = {
> diff --git a/drivers/misc/ocxl/sysfs.c b/drivers/misc/ocxl/sysfs.c
> index 0ab1fd1b2682..a67931f1feec 100644
> --- a/drivers/misc/ocxl/sysfs.c
> +++ b/drivers/misc/ocxl/sysfs.c
> @@ -3,11 +3,18 @@
> #include <linux/sysfs.h>
> #include "ocxl_internal.h"
>
> +static inline struct ocxl_afu *to_afu(struct device *device)
> +{
> + struct ocxl_file_info *info = container_of(device, struct ocxl_file_info, dev);
> +
> + return info->afu;
> +}
> +
> static ssize_t global_mmio_size_show(struct device *device,
> struct device_attribute *attr,
> char *buf)
> {
> - struct ocxl_afu *afu = to_ocxl_afu(device);
> + struct ocxl_afu *afu = to_afu(device);
>
> return scnprintf(buf, PAGE_SIZE, "%d\n",
> afu->config.global_mmio_size);
> @@ -17,7 +24,7 @@ static ssize_t pp_mmio_size_show(struct device *device,
> struct device_attribute *attr,
> char *buf)
> {
> - struct ocxl_afu *afu = to_ocxl_afu(device);
> + struct ocxl_afu *afu = to_afu(device);
>
> return scnprintf(buf, PAGE_SIZE, "%d\n",
> afu->config.pp_mmio_stride);
> @@ -27,7 +34,7 @@ static ssize_t afu_version_show(struct device *device,
> struct device_attribute *attr,
> char *buf)
> {
> - struct ocxl_afu *afu = to_ocxl_afu(device);
> + struct ocxl_afu *afu = to_afu(device);
>
> return scnprintf(buf, PAGE_SIZE, "%hhu:%hhu\n",
> afu->config.version_major,
> @@ -38,7 +45,7 @@ static ssize_t contexts_show(struct device *device,
> struct device_attribute *attr,
> char *buf)
> {
> - struct ocxl_afu *afu = to_ocxl_afu(device);
> + struct ocxl_afu *afu = to_afu(device);
>
> return scnprintf(buf, PAGE_SIZE, "%d/%d\n",
> afu->pasid_count, afu->pasid_max);
> @@ -55,7 +62,7 @@ static ssize_t global_mmio_read(struct file *filp, struct kobject *kobj,
> struct bin_attribute *bin_attr, char *buf,
> loff_t off, size_t count)
> {
> - struct ocxl_afu *afu = to_ocxl_afu(kobj_to_dev(kobj));
> + struct ocxl_afu *afu = to_afu(kobj_to_dev(kobj));
>
> if (count == 0 || off < 0 ||
> off >= afu->config.global_mmio_size)
> @@ -86,7 +93,7 @@ static int global_mmio_mmap(struct file *filp, struct kobject *kobj,
> struct bin_attribute *bin_attr,
> struct vm_area_struct *vma)
> {
> - struct ocxl_afu *afu = to_ocxl_afu(kobj_to_dev(kobj));
> + struct ocxl_afu *afu = to_afu(kobj_to_dev(kobj));
>
> if ((vma_pages(vma) + vma->vm_pgoff) >
> (afu->config.global_mmio_size >> PAGE_SHIFT))
> @@ -99,27 +106,29 @@ static int global_mmio_mmap(struct file *filp, struct kobject *kobj,
> return 0;
> }
>
> -int ocxl_sysfs_add_afu(struct ocxl_afu *afu)
> +int ocxl_sysfs_register_afu(struct ocxl_afu *afu)
> {
> int i, rc;
> + struct ocxl_file_info *info = ocxl_afu_get_private(afu);
> +
> + if (!info)
> + return -EFAULT; // ocxl_file_register_afu() must be called first
>
> for (i = 0; i < ARRAY_SIZE(afu_attrs); i++) {
> - rc = device_create_file(&afu->dev, &afu_attrs[i]);
> + rc = device_create_file(&info->dev, &afu_attrs[i]);
> if (rc)
> goto err;
> }
>
> - sysfs_attr_init(&afu->attr_global_mmio.attr);
> - afu->attr_global_mmio.attr.name = "global_mmio_area";
> - afu->attr_global_mmio.attr.mode = 0600;
> - afu->attr_global_mmio.size = afu->config.global_mmio_size;
> - afu->attr_global_mmio.read = global_mmio_read;
> - afu->attr_global_mmio.mmap = global_mmio_mmap;
> - rc = device_create_bin_file(&afu->dev, &afu->attr_global_mmio);
> + sysfs_attr_init(&info->attr_global_mmio.attr);
> + info->attr_global_mmio.attr.name = "global_mmio_area";
> + info->attr_global_mmio.attr.mode = 0600;
> + info->attr_global_mmio.size = afu->config.global_mmio_size;
> + info->attr_global_mmio.read = global_mmio_read;
> + info->attr_global_mmio.mmap = global_mmio_mmap;
> + rc = device_create_bin_file(&info->dev, &info->attr_global_mmio);
> if (rc) {
> - dev_err(&afu->dev,
> - "Unable to create global mmio attr for afu: %d\n",
> - rc);
> + dev_err(&info->dev, "Unable to create global mmio attr for afu: %d\n", rc);
> goto err;
> }
>
> @@ -127,15 +136,20 @@ int ocxl_sysfs_add_afu(struct ocxl_afu *afu)
>
> err:
> for (i--; i >= 0; i--)
> - device_remove_file(&afu->dev, &afu_attrs[i]);
> + device_remove_file(&info->dev, &afu_attrs[i]);
> +
> return rc;
> }
>
> -void ocxl_sysfs_remove_afu(struct ocxl_afu *afu)
> +void ocxl_sysfs_unregister_afu(struct ocxl_afu *afu)
> {
> + struct ocxl_file_info *info = ocxl_afu_get_private(afu);
> int i;
>
> + if (!info)
> + return;
> +
> for (i = 0; i < ARRAY_SIZE(afu_attrs); i++)
> - device_remove_file(&afu->dev, &afu_attrs[i]);
> - device_remove_bin_file(&afu->dev, &afu->attr_global_mmio);
> + device_remove_file(&info->dev, &afu_attrs[i]);
> + device_remove_bin_file(&info->dev, &info->attr_global_mmio);
> }
> diff --git a/include/misc/ocxl.h b/include/misc/ocxl.h
> index 9530d3be1b30..7d611ea68a59 100644
> --- a/include/misc/ocxl.h
> +++ b/include/misc/ocxl.h
> @@ -16,11 +16,7 @@
>
> #define OCXL_AFU_NAME_SZ (24+1) /* add 1 for NULL termination */
>
> -/*
> - * The following 2 structures are a fairly generic way of representing
> - * the configuration data for a function and AFU, as read from the
> - * configuration space.
> - */
> +
> struct ocxl_afu_config {
> u8 idx;
> int dvsec_afu_control_pos; /* offset of AFU control DVSEC */
> @@ -49,12 +45,110 @@ struct ocxl_fn_config {
> s8 max_afu_index;
> };
>
> -/*
> - * Read the configuration space of a function and fill in a
> - * ocxl_fn_config structure with all the function details
> +// These are opaque outside the ocxl driver
> +struct ocxl_afu;
> +struct ocxl_fn;
> +
> +// Device detection & initialisation
> +
> +/**
> + * Open an OpenCAPI function on an OpenCAPI device
> + *
> + * @dev: The PCI device that contains the function
> + *
> + * Returns an opaque pointer to the function, or an error pointer (check with IS_ERR)
> */
> -int ocxl_config_read_function(struct pci_dev *dev,
> - struct ocxl_fn_config *fn);
> +struct ocxl_fn *ocxl_function_open(struct pci_dev *dev);
> +
> +/**
> + * Get the list of AFUs associated with a PCI function device
> + *
> + * Returns a list of struct ocxl_afu *
> + *
> + * @fn: The OpenCAPI function containing the AFUs
> + */
> +struct list_head *ocxl_function_afu_list(struct ocxl_fn *fn);
> +
> +/**
> + * Fetch an AFU instance from an OpenCAPI function
> + *
> + * @fn: The OpenCAPI function to get the AFU from
> + * @afu_idx: The index of the AFU to get
> + *
> + * If successful, the AFU should be released with ocxl_afu_put()
> + *
> + * Returns a pointer to the AFU, or NULL on error
> + */
> +struct ocxl_afu *ocxl_function_fetch_afu(struct ocxl_fn *fn, u8 afu_idx);
> +
> +/**
> + * Take a reference to an AFU
> + *
> + * @afu: The AFU to increment the reference count on
> + */
> +void ocxl_afu_get(struct ocxl_afu *afu);
> +
> +/**
> + * Release a reference to an AFU
> + *
> + * @afu: The AFU to decrement the reference count on
> + */
> +void ocxl_afu_put(struct ocxl_afu *afu);
> +
> +
> +/**
> + * Get the configuration information for an OpenCAPI function
> + *
> + * @fn: The OpenCAPI function to get the config for
> + *
> + * Returns the function config, or NULL on error
> + */
> +const struct ocxl_fn_config *ocxl_function_config(struct ocxl_fn *fn);
> +
> +/**
> + * Close an OpenCAPI function
> + *
> + * This will free any AFUs previously retrieved from the function, and
> + * detach and associated contexts. The contexts must by freed by the caller.
> + *
> + * @fn: The OpenCAPI function to close
> + *
> + */
> +void ocxl_function_close(struct ocxl_fn *fn);
> +
> +// AFU Metadata
> +
> +/**
> + * Get a pointer to the config for an AFU
> + *
> + * @afu: a pointer to the AFU to get the config for
> + *
> + * Returns a pointer to the AFU config
> + */
> +struct ocxl_afu_config *ocxl_afu_config(struct ocxl_afu *afu);
> +
> +/**
> + * Assign opaque hardware specific information to an OpenCAPI AFU.
> + *
> + * @dev: The PCI device associated with the OpenCAPI device
> + * @private: the opaque hardware specific information to assign to the driver
> + * @private_release: A callback to free the private data
> + */
> +void ocxl_afu_set_private(struct ocxl_afu *afu, void *private,
> + void (*private_release)(void *private));
> +
> +/**
> + * Fetch the hardware specific information associated with an external OpenCAPI
> + * AFU. This may be consumed by an external OpenCAPI driver.
> + *
> + * @afu: The AFU
> + *
> + * Returns the opaque pointer associated with the device, or NULL if not set
> + */
> +void *ocxl_afu_get_private(struct ocxl_afu *dev);
> +
> +
> +// Functions left here are for compatibility with the cxlflash driver
>
> /*
> * Read the configuration space of a function for the AFU specified by
> @@ -141,6 +235,13 @@ int ocxl_config_set_TL(struct pci_dev *dev, int tl_dvsec);
> int ocxl_config_terminate_pasid(struct pci_dev *dev,
> int afu_control_offset, int pasid);
>
> +/*
> + * Read the configuration space of a function and fill in a
> + * ocxl_fn_config structure with all the function details
> + */
> +int ocxl_config_read_function(struct pci_dev *dev,
> + struct ocxl_fn_config *fn);
> +
> /*
> * Set up the opencapi link for the function.
> *
>
[-- Attachment #2: 0001-Review-comments.patch --]
[-- Type: text/x-patch, Size: 6386 bytes --]
From 73a937ce583964213f945a60454a3d3666ce098f Mon Sep 17 00:00:00 2001
From: Frederic Barrat <fbarrat@linux.ibm.com>
Date: Fri, 22 Mar 2019 18:29:34 +0100
Subject: [PATCH] Review comments
---
drivers/misc/ocxl/core.c | 8 +++--
drivers/misc/ocxl/file.c | 60 ++++++++++++++++++++++---------
drivers/misc/ocxl/ocxl_internal.h | 1 +
drivers/misc/ocxl/pci.c | 20 +++++++++++
4 files changed, 70 insertions(+), 19 deletions(-)
diff --git a/drivers/misc/ocxl/core.c b/drivers/misc/ocxl/core.c
index c632ec372342..87dedaaddd4f 100644
--- a/drivers/misc/ocxl/core.c
+++ b/drivers/misc/ocxl/core.c
@@ -30,7 +30,11 @@ static struct ocxl_afu *alloc_afu(struct ocxl_fn *fn)
return afu;
}
-static void afu_release(struct kref *kref)
+static void free_afu(struct kref *kref)
+// fxb: I'm trying to keep some kind of naming logic:
+// alloc_afu/free_afu
+// we also have
+// alloc_function/free_function
{
struct ocxl_afu *afu = container_of(kref, struct ocxl_afu, kref);
@@ -47,7 +51,7 @@ EXPORT_SYMBOL_GPL(ocxl_afu_get);
void ocxl_afu_put(struct ocxl_afu *afu)
{
- kref_put(&afu->kref, afu_release);
+ kref_put(&afu->kref, free_afu);
}
EXPORT_SYMBOL_GPL(ocxl_afu_put);
diff --git a/drivers/misc/ocxl/file.c b/drivers/misc/ocxl/file.c
index 42214b0c956a..b17c2bebc127 100644
--- a/drivers/misc/ocxl/file.c
+++ b/drivers/misc/ocxl/file.c
@@ -490,21 +490,18 @@ static const struct file_operations ocxl_afu_fops = {
.release = afu_release,
};
-// Called when there are no more consumers of the AFU
-static void ocxl_file_release(void *private)
-{
- struct ocxl_file_info *info = private;
-
- ocxl_sysfs_unregister_afu(info->afu);
- device_unregister(&info->dev);
-
- free_minor(info);
-}
-
// Free the info struct
static void info_release(struct device *dev)
{
struct ocxl_file_info *info = container_of(dev, struct ocxl_file_info, dev);
+// fxb: we don't need 2 functions to release/free the info
+// structure. Only one, called when the info device is released. So
+// I'm merging the content of ocxl_file_release. When info is
+// released, the afu ref count is decremented, which will free the afu
+// structure, which in turn, will end up freeing the function
+// structure
+ ocxl_afu_put(info->afu);
+ free_minor(info);
kfree(info);
}
@@ -520,8 +517,6 @@ int ocxl_file_register_afu(struct ocxl_afu *afu)
if (info == NULL)
return -ENOMEM;
- info->afu = afu;
-
minor = allocate_minor(info);
if (minor < 0) {
kfree(info);
@@ -533,18 +528,44 @@ int ocxl_file_register_afu(struct ocxl_afu *afu)
info->dev.class = ocxl_class;
info->dev.release = info_release;
- ocxl_afu_set_private(afu, info, ocxl_file_release);
+ info->afu = afu;
+ ocxl_afu_get(afu);
+// fxb: we store a reference to the AFU in the info structure, so we
+// increment the ref count of the AFU to make sure it's not going away
+
+ ocxl_afu_set_private(afu, info, NULL); // fxb: release callback still needed?
rc = dev_set_name(&info->dev, "%s.%s.%hhu",
afu->config.name, dev_name(&pci_dev->dev), afu->config.idx);
if (rc)
- return rc;
+ goto err_put;
rc = device_register(&info->dev);
- if (rc)
- return rc;
+ if (rc) {
+ put_device(&info->dev);
+ goto err_put;
+ }
return ocxl_sysfs_register_afu(afu);
+
+err_put:
+ ocxl_afu_put(afu);
+ free_minor(info);
+ kfree(info);
+ return rc;
+}
+
+void ocxl_file_unregister_afu(struct ocxl_afu *afu)
+{
+ struct ocxl_file_info *info = ocxl_afu_get_private(afu);
+
+// fxb: ocxl_sysfs_unregister_afu() cannot be called from the info
+// device release callback, as it still needs the struct device. Same
+// reason we had to call ocxl_sysfs_register_afu() after the device is
+// registered.
+
+ ocxl_sysfs_unregister_afu(afu);
+ device_unregister(&info->dev);
}
int ocxl_file_make_visible(struct ocxl_afu *afu)
@@ -552,6 +573,11 @@ int ocxl_file_make_visible(struct ocxl_afu *afu)
int rc;
struct ocxl_file_info *info = ocxl_afu_get_private(afu);
+ /*
+ * fxb: couldn't it be merged with ocxl_file_register_afu()? The
+ * requirement is that it must be done last, but it could still
+ * be achieved in ocxl_file_register_afu().
+ */
cdev_init(&info->cdev, &ocxl_afu_fops);
rc = cdev_add(&info->cdev, info->dev.devt, 1);
if (rc) {
diff --git a/drivers/misc/ocxl/ocxl_internal.h b/drivers/misc/ocxl/ocxl_internal.h
index f3775223f4b1..71780e86def8 100644
--- a/drivers/misc/ocxl/ocxl_internal.h
+++ b/drivers/misc/ocxl/ocxl_internal.h
@@ -99,6 +99,7 @@ struct ocxl_process_element {
int ocxl_create_cdev(struct ocxl_afu *afu);
void ocxl_destroy_cdev(struct ocxl_afu *afu);
int ocxl_file_register_afu(struct ocxl_afu *afu);
+void ocxl_file_unregister_afu(struct ocxl_afu *afu);
int ocxl_file_make_visible(struct ocxl_afu *afu);
void ocxl_file_make_invisible(struct ocxl_afu *afu);
diff --git a/drivers/misc/ocxl/pci.c b/drivers/misc/ocxl/pci.c
index a1ed2bb02e4b..329253b5c54a 100644
--- a/drivers/misc/ocxl/pci.c
+++ b/drivers/misc/ocxl/pci.c
@@ -39,6 +39,13 @@ static int ocxl_probe(struct pci_dev *dev, const struct pci_device_id *id)
continue;
// Defer error cleanup until the device is removed
+ // fxb: this is actually dangerous. It means that in
+ // ocxl_file_unregister_afu() and
+ // ocxl_file_make_invisible(), we'll try to undo
+ // things which were not done int the first place, if
+ // something failed during init. It will likely
+ // generate a bunch of errors in ocxl_remove and even
+ // lead to kernel oops with some bad luck
}
return 0;
@@ -54,6 +61,19 @@ void ocxl_remove(struct pci_dev *dev)
afu_list = ocxl_function_afu_list(fn);
list_for_each_entry(afu, afu_list, list) {
+// fxb: the release of the info structure was convoluted. It's simpler
+// to keep things here symmetric with what we do in probe, so
+// ocxl_file_register_afu() needs a matching
+// ocxl_file_unregister_afu(). It also gets rid of the callback on the
+// private data. Though you may still need that for the external
+// driver, I don't know.
+
+// fxb: see comment in probe about calling those functions if we had
+// hit an error patch during probe, there's more work needed (and the
+// problem is not due to the new ocxl_file_unregister_afu(), it was
+// also there before when calling the callback in
+// ocxl_function_close()
+ ocxl_file_unregister_afu(afu);
ocxl_file_make_invisible(afu);
}
--
2.19.1
^ permalink raw reply related
* Re: [PATCH v5 3/3] locking/rwsem: Optimize down_read_trylock()
From: Russell King - ARM Linux admin @ 2019-03-22 17:25 UTC (permalink / raw)
To: Waiman Long
Cc: linux-ia64, linux-sh, Peter Zijlstra, Will Deacon, linux-kernel,
H. Peter Anvin, sparclinux, linux-riscv, linux-arch, linux-s390,
Davidlohr Bueso, linux-c6x-dev, linux-hexagon, x86, Ingo Molnar,
uclinux-h8-devel, linux-xtensa, Arnd Bergmann, linux-um,
linuxppc-dev, linux-m68k, openrisc, Borislav Petkov,
Thomas Gleixner, linux-arm-kernel, linux-parisc, Linus Torvalds,
linux-mips, linux-alpha, nios2-dev, Andrew Morton, Tim Chen
In-Reply-To: <20190322143008.21313-4-longman@redhat.com>
On Fri, Mar 22, 2019 at 10:30:08AM -0400, Waiman Long wrote:
> Modify __down_read_trylock() to optimize for an unlocked rwsem and make
> it generate slightly better code.
>
> Before this patch, down_read_trylock:
>
> 0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5>
> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24>
> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx
> 0x000000000000000b <+11>: mov %rdx,%rax
> 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi)
> 0x0000000000000013 <+19>: cmp %rax,%rdx
> 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35>
> 0x0000000000000018 <+24>: mov (%rdi),%rdx
> 0x000000000000001b <+27>: test %rdx,%rdx
> 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7>
> 0x0000000000000020 <+32>: xor %eax,%eax
> 0x0000000000000022 <+34>: retq
> 0x0000000000000023 <+35>: mov %gs:0x0,%rax
> 0x000000000000002c <+44>: or $0x3,%rax
> 0x0000000000000030 <+48>: mov %rax,0x20(%rdi)
> 0x0000000000000034 <+52>: mov $0x1,%eax
> 0x0000000000000039 <+57>: retq
>
> After patch, down_read_trylock:
>
> 0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5>
> 0x0000000000000005 <+5>: xor %eax,%eax
> 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx
> 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi)
> 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41>
> 0x0000000000000012 <+18>: mov %gs:0x0,%rax
> 0x000000000000001b <+27>: or $0x3,%rax
> 0x000000000000001f <+31>: mov %rax,0x20(%rdi)
> 0x0000000000000023 <+35>: mov $0x1,%eax
> 0x0000000000000028 <+40>: retq
> 0x0000000000000029 <+41>: test %rax,%rax
> 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7>
> 0x000000000000002e <+46>: xor %eax,%eax
> 0x0000000000000030 <+48>: retq
>
> By using a rwsem microbenchmark, the down_read_trylock() rate (with a
> load of 10 to lengthen the lock critical section) on a x86-64 system
> before and after the patch were:
>
> Before Patch After Patch
> # of Threads rlock rlock
> ------------ ----- -----
> 1 14,496 14,716
> 2 8,644 8,453
> 4 6,799 6,983
> 8 5,664 7,190
>
> On a ARM64 system, the performance results were:
>
> Before Patch After Patch
> # of Threads rlock rlock
> ------------ ----- -----
> 1 23,676 24,488
> 2 7,697 9,502
> 4 4,945 3,440
> 8 2,641 1,603
>
> For the uncontended case (1 thread), the new down_read_trylock() is a
> little bit faster. For the contended cases, the new down_read_trylock()
> perform pretty well in x86-64, but performance degrades at high
> contention level on ARM64.
So, 70% for 4 threads, 61% for 4 threads - does this trend
continue tailing off as the number of threads (and cores)
increase?
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
^ permalink raw reply
* Re: [PATCH v5 1/3] locking/rwsem: Remove arch specific rwsem files
From: Waiman Long @ 2019-03-22 17:23 UTC (permalink / raw)
To: Linus Torvalds
Cc: linux-ia64, Linux-sh list, Peter Zijlstra, Will Deacon,
linux-mips, H. Peter Anvin, sparclinux, linux-riscv, linux-arch,
linux-s390, Davidlohr Bueso, linux-c6x-dev, linux-hexagon,
the arch/x86 maintainers, Ingo Molnar, uclinux-h8-devel,
linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, openrisc,
Borislav Petkov, Thomas Gleixner, linux-alpha@vger.kernel.org,
linux-parisc, Tim Chen, Linux List Kernel Mailing,
linux-alpha@vger.kernel.org, nios2-dev, Andrew Morton,
linuxppc-dev
In-Reply-To: <CAHk-=wikkO-1f1=FEOEzkSnaDg3yJLP=4Vd59UCuLBztFd0KOw@mail.gmail.com>
On 03/22/2019 01:01 PM, Linus Torvalds wrote:
> On Fri, Mar 22, 2019 at 7:30 AM Waiman Long <longman@redhat.com> wrote:
>> 19 files changed, 133 insertions(+), 930 deletions(-)
> Lovely. And it all looks sane to me.
>
> So ack.
>
> The only comment I have is about __down_read_trylock(), which probably
> isn't critical enough to actually care about, but:
>
>> +static inline int __down_read_trylock(struct rw_semaphore *sem)
>> +{
>> + long tmp;
>> +
>> + while ((tmp = atomic_long_read(&sem->count)) >= 0) {
>> + if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp,
>> + tmp + RWSEM_ACTIVE_READ_BIAS)) {
>> + return 1;
>> + }
>> + }
>> + return 0;
>> +}
> So this seems to
>
> (a) read the line early (the whole cacheline in shared state issue)
>
> (b) read the line again unnecessarily in the while loop
>
> Now, (a) might be explained by "well, maybe we do trylock even with
> existing readers", although I continue to think that the case we
> should optimize for is simply the uncontended one, where we don't even
> have multiple readers.
>
> But (b) just seems silly.
>
> So I wonder if it shouldn't just be
>
> long tmp = 0;
>
> do {
> long new = atomic_long_cmpxchg_acquire(&sem->count, tmp,
> tmp + RWSEM_ACTIVE_READ_BIAS);
> if (likely(new == tmp))
> return 1;
> tmp = new;
> } while (tmp >= 0);
> return 0;
>
> which would seem simpler and solve both issues. Hmm?
>
> But honestly, I didn't check what our uses of down_read_trylock() look
> like. We have more of them than I expected, and I _think_ the normal
> case is the "nobody else holds the lock", but that's just a gut
> feeling.
>
> Some of them _might_ be performance-critical. There's the one on
> mmap_sem in the fault handling path, for example. And yes, I'd expect
> the normal case to very much be "no other readers or writers" for that
> one.
>
> NOTE! The above code snippet is absolutely untested, and might be
> completely wrong. Take it as a "something like this" rather than
> anything else.
>
> Linus
As you have noticed already, this patch is just for moving code around
without changing it. I optimize __down_read_trylock() in patch 3.
Cheers,
Longman
^ permalink raw reply
* Re: [PATCH v5 3/3] locking/rwsem: Optimize down_read_trylock()
From: Linus Torvalds @ 2019-03-22 17:08 UTC (permalink / raw)
To: Waiman Long
Cc: linux-ia64, Linux-sh list, Peter Zijlstra, Will Deacon,
linux-mips, H. Peter Anvin, sparclinux, linux-riscv, linux-arch,
linux-s390, Davidlohr Bueso, linux-c6x-dev, linux-hexagon,
the arch/x86 maintainers, Ingo Molnar, uclinux-h8-devel,
linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, openrisc,
Borislav Petkov, Thomas Gleixner, linux-alpha@vger.kernel.org,
linux-parisc, Tim Chen, Linux List Kernel Mailing,
linux-alpha@vger.kernel.org, nios2-dev, Andrew Morton,
linuxppc-dev
In-Reply-To: <20190322143008.21313-4-longman@redhat.com>
On Fri, Mar 22, 2019 at 7:30 AM Waiman Long <longman@redhat.com> wrote:
>
> Modify __down_read_trylock() to optimize for an unlocked rwsem and make
> it generate slightly better code.
Oh, that should teach me to read all patches in the series before
starting to comment on them.
So ignore my comment on #1.
Linus
^ permalink raw reply
* Re: [PATCH v5 2/3] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
From: Linus Torvalds @ 2019-03-22 17:07 UTC (permalink / raw)
To: Waiman Long
Cc: linux-ia64, Linux-sh list, Peter Zijlstra, Will Deacon,
linux-mips, H. Peter Anvin, sparclinux, linux-riscv, linux-arch,
linux-s390, Davidlohr Bueso, linux-c6x-dev, linux-hexagon,
the arch/x86 maintainers, Ingo Molnar, uclinux-h8-devel,
linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, openrisc,
Borislav Petkov, Thomas Gleixner, linux-alpha@vger.kernel.org,
linux-parisc, Tim Chen, Linux List Kernel Mailing,
linux-alpha@vger.kernel.org, nios2-dev, Andrew Morton,
linuxppc-dev
In-Reply-To: <20190322143008.21313-3-longman@redhat.com>
On Fri, Mar 22, 2019 at 7:30 AM Waiman Long <longman@redhat.com> wrote:
>
> For simplication, we are going to remove rwsem-spinlock.c and make all
> architectures use a single implementation of rwsem - rwsem-xadd.c.
Ack.
Linus
^ permalink raw reply
* Re: [PATCH v5 1/3] locking/rwsem: Remove arch specific rwsem files
From: Linus Torvalds @ 2019-03-22 17:01 UTC (permalink / raw)
To: Waiman Long
Cc: linux-ia64, Linux-sh list, Peter Zijlstra, Will Deacon,
linux-mips, H. Peter Anvin, sparclinux, linux-riscv, linux-arch,
linux-s390, Davidlohr Bueso, linux-c6x-dev, linux-hexagon,
the arch/x86 maintainers, Ingo Molnar, uclinux-h8-devel,
linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, openrisc,
Borislav Petkov, Thomas Gleixner, linux-alpha@vger.kernel.org,
linux-parisc, Tim Chen, Linux List Kernel Mailing,
linux-alpha@vger.kernel.org, nios2-dev, Andrew Morton,
linuxppc-dev
In-Reply-To: <20190322143008.21313-2-longman@redhat.com>
On Fri, Mar 22, 2019 at 7:30 AM Waiman Long <longman@redhat.com> wrote:
>
> 19 files changed, 133 insertions(+), 930 deletions(-)
Lovely. And it all looks sane to me.
So ack.
The only comment I have is about __down_read_trylock(), which probably
isn't critical enough to actually care about, but:
> +static inline int __down_read_trylock(struct rw_semaphore *sem)
> +{
> + long tmp;
> +
> + while ((tmp = atomic_long_read(&sem->count)) >= 0) {
> + if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp,
> + tmp + RWSEM_ACTIVE_READ_BIAS)) {
> + return 1;
> + }
> + }
> + return 0;
> +}
So this seems to
(a) read the line early (the whole cacheline in shared state issue)
(b) read the line again unnecessarily in the while loop
Now, (a) might be explained by "well, maybe we do trylock even with
existing readers", although I continue to think that the case we
should optimize for is simply the uncontended one, where we don't even
have multiple readers.
But (b) just seems silly.
So I wonder if it shouldn't just be
long tmp = 0;
do {
long new = atomic_long_cmpxchg_acquire(&sem->count, tmp,
tmp + RWSEM_ACTIVE_READ_BIAS);
if (likely(new == tmp))
return 1;
tmp = new;
} while (tmp >= 0);
return 0;
which would seem simpler and solve both issues. Hmm?
But honestly, I didn't check what our uses of down_read_trylock() look
like. We have more of them than I expected, and I _think_ the normal
case is the "nobody else holds the lock", but that's just a gut
feeling.
Some of them _might_ be performance-critical. There's the one on
mmap_sem in the fault handling path, for example. And yes, I'd expect
the normal case to very much be "no other readers or writers" for that
one.
NOTE! The above code snippet is absolutely untested, and might be
completely wrong. Take it as a "something like this" rather than
anything else.
Linus
^ permalink raw reply
* [PATCH v5 3/3] locking/rwsem: Optimize down_read_trylock()
From: Waiman Long @ 2019-03-22 14:30 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Will Deacon, Thomas Gleixner
Cc: linux-ia64, linux-sh, linux-mips, H. Peter Anvin, sparclinux,
linux-riscv, linux-arch, linux-s390, Davidlohr Bueso,
linux-c6x-dev, linux-hexagon, x86, Waiman Long, uclinux-h8-devel,
linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, openrisc,
Borislav Petkov, linux-arm-kernel, Tim Chen, linux-parisc,
Linus Torvalds, linux-kernel, linux-alpha, nios2-dev,
Andrew Morton, linuxppc-dev
In-Reply-To: <20190322143008.21313-1-longman@redhat.com>
Modify __down_read_trylock() to optimize for an unlocked rwsem and make
it generate slightly better code.
Before this patch, down_read_trylock:
0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5>
0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24>
0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx
0x000000000000000b <+11>: mov %rdx,%rax
0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi)
0x0000000000000013 <+19>: cmp %rax,%rdx
0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35>
0x0000000000000018 <+24>: mov (%rdi),%rdx
0x000000000000001b <+27>: test %rdx,%rdx
0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7>
0x0000000000000020 <+32>: xor %eax,%eax
0x0000000000000022 <+34>: retq
0x0000000000000023 <+35>: mov %gs:0x0,%rax
0x000000000000002c <+44>: or $0x3,%rax
0x0000000000000030 <+48>: mov %rax,0x20(%rdi)
0x0000000000000034 <+52>: mov $0x1,%eax
0x0000000000000039 <+57>: retq
After patch, down_read_trylock:
0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5>
0x0000000000000005 <+5>: xor %eax,%eax
0x0000000000000007 <+7>: lea 0x1(%rax),%rdx
0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi)
0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41>
0x0000000000000012 <+18>: mov %gs:0x0,%rax
0x000000000000001b <+27>: or $0x3,%rax
0x000000000000001f <+31>: mov %rax,0x20(%rdi)
0x0000000000000023 <+35>: mov $0x1,%eax
0x0000000000000028 <+40>: retq
0x0000000000000029 <+41>: test %rax,%rax
0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7>
0x000000000000002e <+46>: xor %eax,%eax
0x0000000000000030 <+48>: retq
By using a rwsem microbenchmark, the down_read_trylock() rate (with a
load of 10 to lengthen the lock critical section) on a x86-64 system
before and after the patch were:
Before Patch After Patch
# of Threads rlock rlock
------------ ----- -----
1 14,496 14,716
2 8,644 8,453
4 6,799 6,983
8 5,664 7,190
On a ARM64 system, the performance results were:
Before Patch After Patch
# of Threads rlock rlock
------------ ----- -----
1 23,676 24,488
2 7,697 9,502
4 4,945 3,440
8 2,641 1,603
For the uncontended case (1 thread), the new down_read_trylock() is a
little bit faster. For the contended cases, the new down_read_trylock()
perform pretty well in x86-64, but performance degrades at high
contention level on ARM64.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/locking/rwsem.h | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/kernel/locking/rwsem.h b/kernel/locking/rwsem.h
index 45ee00236e03..1f5775aa6a1d 100644
--- a/kernel/locking/rwsem.h
+++ b/kernel/locking/rwsem.h
@@ -174,14 +174,17 @@ static inline int __down_read_killable(struct rw_semaphore *sem)
static inline int __down_read_trylock(struct rw_semaphore *sem)
{
- long tmp;
+ /*
+ * Optimize for the case when the rwsem is not locked at all.
+ */
+ long tmp = RWSEM_UNLOCKED_VALUE;
- while ((tmp = atomic_long_read(&sem->count)) >= 0) {
- if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp,
- tmp + RWSEM_ACTIVE_READ_BIAS)) {
+ do {
+ if (atomic_long_try_cmpxchg_acquire(&sem->count, &tmp,
+ tmp + RWSEM_ACTIVE_READ_BIAS)) {
return 1;
}
- }
+ } while (tmp >= 0);
return 0;
}
--
2.18.1
^ permalink raw reply related
* [PATCH v5 2/3] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs
From: Waiman Long @ 2019-03-22 14:30 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Will Deacon, Thomas Gleixner
Cc: linux-ia64, linux-sh, linux-mips, H. Peter Anvin, sparclinux,
linux-riscv, linux-arch, linux-s390, Davidlohr Bueso,
linux-c6x-dev, linux-hexagon, x86, Waiman Long, uclinux-h8-devel,
linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, openrisc,
Borislav Petkov, linux-arm-kernel, Tim Chen, linux-parisc,
Linus Torvalds, linux-kernel, linux-alpha, nios2-dev,
Andrew Morton, linuxppc-dev
In-Reply-To: <20190322143008.21313-1-longman@redhat.com>
Currently, we have two different implementation of rwsem:
1) CONFIG_RWSEM_GENERIC_SPINLOCK (rwsem-spinlock.c)
2) CONFIG_RWSEM_XCHGADD_ALGORITHM (rwsem-xadd.c)
As we are going to use a single generic implementation for rwsem-xadd.c
and no architecture-specific code will be needed, there is no point
in keeping two different implementations of rwsem. In most cases, the
performance of rwsem-spinlock.c will be worse. It also doesn't get all
the performance tuning and optimizations that had been implemented in
rwsem-xadd.c over the years.
For simplication, we are going to remove rwsem-spinlock.c and make all
architectures use a single implementation of rwsem - rwsem-xadd.c.
All references to RWSEM_GENERIC_SPINLOCK and RWSEM_XCHGADD_ALGORITHM
in the code are removed.
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Waiman Long <longman@redhat.com>
---
arch/alpha/Kconfig | 7 -
arch/arc/Kconfig | 3 -
arch/arm/Kconfig | 4 -
arch/arm64/Kconfig | 3 -
arch/c6x/Kconfig | 3 -
arch/csky/Kconfig | 3 -
arch/h8300/Kconfig | 3 -
arch/hexagon/Kconfig | 6 -
arch/ia64/Kconfig | 4 -
arch/m68k/Kconfig | 7 -
arch/microblaze/Kconfig | 6 -
arch/mips/Kconfig | 7 -
arch/nds32/Kconfig | 3 -
arch/nios2/Kconfig | 3 -
arch/openrisc/Kconfig | 6 -
arch/parisc/Kconfig | 6 -
arch/powerpc/Kconfig | 7 -
arch/riscv/Kconfig | 3 -
arch/s390/Kconfig | 6 -
arch/sh/Kconfig | 6 -
arch/sparc/Kconfig | 8 -
arch/unicore32/Kconfig | 6 -
arch/x86/Kconfig | 3 -
arch/x86/um/Kconfig | 6 -
arch/xtensa/Kconfig | 3 -
include/linux/rwsem-spinlock.h | 47 -----
include/linux/rwsem.h | 5 -
kernel/Kconfig.locks | 2 +-
kernel/locking/Makefile | 4 +-
kernel/locking/rwsem-spinlock.c | 339 --------------------------------
kernel/locking/rwsem.h | 3 -
31 files changed, 2 insertions(+), 520 deletions(-)
delete mode 100644 include/linux/rwsem-spinlock.h
delete mode 100644 kernel/locking/rwsem-spinlock.c
diff --git a/arch/alpha/Kconfig b/arch/alpha/Kconfig
index 584a6e114853..27c871227eee 100644
--- a/arch/alpha/Kconfig
+++ b/arch/alpha/Kconfig
@@ -49,13 +49,6 @@ config MMU
bool
default y
-config RWSEM_GENERIC_SPINLOCK
- bool
-
-config RWSEM_XCHGADD_ALGORITHM
- bool
- default y
-
config ARCH_HAS_ILOG2_U32
bool
default n
diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig
index c781e45d1d99..23e063df5d2c 100644
--- a/arch/arc/Kconfig
+++ b/arch/arc/Kconfig
@@ -63,9 +63,6 @@ config SCHED_OMIT_FRAME_POINTER
config GENERIC_CSUM
def_bool y
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
config ARCH_DISCONTIGMEM_ENABLE
def_bool n
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 054ead960f98..c11c61093c6c 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -178,10 +178,6 @@ config TRACE_IRQFLAGS_SUPPORT
bool
default !CPU_V7M
-config RWSEM_XCHGADD_ALGORITHM
- bool
- default y
-
config ARCH_HAS_ILOG2_U32
bool
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 7e34b9eba5de..c62b9db2b5e8 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -237,9 +237,6 @@ config LOCKDEP_SUPPORT
config TRACE_IRQFLAGS_SUPPORT
def_bool y
-config RWSEM_XCHGADD_ALGORITHM
- def_bool y
-
config GENERIC_BUG
def_bool y
depends on BUG
diff --git a/arch/c6x/Kconfig b/arch/c6x/Kconfig
index e5cd3c5f8399..ed92b5840c0a 100644
--- a/arch/c6x/Kconfig
+++ b/arch/c6x/Kconfig
@@ -27,9 +27,6 @@ config MMU
config FPU
def_bool n
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
config GENERIC_CALIBRATE_DELAY
def_bool y
diff --git a/arch/csky/Kconfig b/arch/csky/Kconfig
index 725a115759c9..6555d1781132 100644
--- a/arch/csky/Kconfig
+++ b/arch/csky/Kconfig
@@ -92,9 +92,6 @@ config GENERIC_HWEIGHT
config MMU
def_bool y
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
config STACKTRACE_SUPPORT
def_bool y
diff --git a/arch/h8300/Kconfig b/arch/h8300/Kconfig
index c071da34e081..61c01db6c292 100644
--- a/arch/h8300/Kconfig
+++ b/arch/h8300/Kconfig
@@ -27,9 +27,6 @@ config H8300
config CPU_BIG_ENDIAN
def_bool y
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
config GENERIC_HWEIGHT
def_bool y
diff --git a/arch/hexagon/Kconfig b/arch/hexagon/Kconfig
index ac441680dcc0..3e54a53208d5 100644
--- a/arch/hexagon/Kconfig
+++ b/arch/hexagon/Kconfig
@@ -65,12 +65,6 @@ config GENERIC_CSUM
config GENERIC_IRQ_PROBE
def_bool y
-config RWSEM_GENERIC_SPINLOCK
- def_bool n
-
-config RWSEM_XCHGADD_ALGORITHM
- def_bool y
-
config GENERIC_HWEIGHT
def_bool y
diff --git a/arch/ia64/Kconfig b/arch/ia64/Kconfig
index 8d7396bd1790..73a26f04644e 100644
--- a/arch/ia64/Kconfig
+++ b/arch/ia64/Kconfig
@@ -83,10 +83,6 @@ config STACKTRACE_SUPPORT
config GENERIC_LOCKBREAK
def_bool n
-config RWSEM_XCHGADD_ALGORITHM
- bool
- default y
-
config HUGETLB_PAGE_SIZE_VARIABLE
bool
depends on HUGETLB_PAGE
diff --git a/arch/m68k/Kconfig b/arch/m68k/Kconfig
index b54206408f91..f5661f48019c 100644
--- a/arch/m68k/Kconfig
+++ b/arch/m68k/Kconfig
@@ -32,13 +32,6 @@ config M68K
config CPU_BIG_ENDIAN
def_bool y
-config RWSEM_GENERIC_SPINLOCK
- bool
- default y
-
-config RWSEM_XCHGADD_ALGORITHM
- bool
-
config ARCH_HAS_ILOG2_U32
bool
diff --git a/arch/microblaze/Kconfig b/arch/microblaze/Kconfig
index a51b965b3b82..a7a9a9d59f65 100644
--- a/arch/microblaze/Kconfig
+++ b/arch/microblaze/Kconfig
@@ -58,15 +58,9 @@ config CPU_LITTLE_ENDIAN
endchoice
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
config ZONE_DMA
def_bool y
-config RWSEM_XCHGADD_ALGORITHM
- bool
-
config ARCH_HAS_ILOG2_U32
def_bool n
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 4a5f5b0ee9a9..b9c48b27162d 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1037,13 +1037,6 @@ source "arch/mips/paravirt/Kconfig"
endmenu
-config RWSEM_GENERIC_SPINLOCK
- bool
- default y
-
-config RWSEM_XCHGADD_ALGORITHM
- bool
-
config GENERIC_HWEIGHT
bool
default y
diff --git a/arch/nds32/Kconfig b/arch/nds32/Kconfig
index addb7f5f5264..55559ca0efe4 100644
--- a/arch/nds32/Kconfig
+++ b/arch/nds32/Kconfig
@@ -60,9 +60,6 @@ config GENERIC_LOCKBREAK
def_bool y
depends on PREEMPT
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
config TRACE_IRQFLAGS_SUPPORT
def_bool y
diff --git a/arch/nios2/Kconfig b/arch/nios2/Kconfig
index 4ef15a61b7bc..56685fd45ed0 100644
--- a/arch/nios2/Kconfig
+++ b/arch/nios2/Kconfig
@@ -40,9 +40,6 @@ config NO_IOPORT_MAP
config FPU
def_bool n
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
config TRACE_IRQFLAGS_SUPPORT
def_bool n
diff --git a/arch/openrisc/Kconfig b/arch/openrisc/Kconfig
index a5e361fbb75a..683511b8c9df 100644
--- a/arch/openrisc/Kconfig
+++ b/arch/openrisc/Kconfig
@@ -43,12 +43,6 @@ config CPU_BIG_ENDIAN
config MMU
def_bool y
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
-config RWSEM_XCHGADD_ALGORITHM
- def_bool n
-
config GENERIC_HWEIGHT
def_bool y
diff --git a/arch/parisc/Kconfig b/arch/parisc/Kconfig
index c8e621296092..f1ed8ddfe486 100644
--- a/arch/parisc/Kconfig
+++ b/arch/parisc/Kconfig
@@ -75,12 +75,6 @@ config GENERIC_LOCKBREAK
default y
depends on SMP && PREEMPT
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
-config RWSEM_XCHGADD_ALGORITHM
- bool
-
config ARCH_HAS_ILOG2_U32
bool
default n
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 2d0be82c3061..e5dd6aafaf68 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -103,13 +103,6 @@ config LOCKDEP_SUPPORT
bool
default y
-config RWSEM_GENERIC_SPINLOCK
- bool
-
-config RWSEM_XCHGADD_ALGORITHM
- bool
- default y
-
config GENERIC_LOCKBREAK
bool
default y
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index eb56c82d8aa1..0582260fb6c2 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -69,9 +69,6 @@ config STACKTRACE_SUPPORT
config TRACE_IRQFLAGS_SUPPORT
def_bool y
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
config GENERIC_BUG
def_bool y
depends on BUG
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index b6e3d0653002..c9300b437195 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -14,12 +14,6 @@ config LOCKDEP_SUPPORT
config STACKTRACE_SUPPORT
def_bool y
-config RWSEM_GENERIC_SPINLOCK
- bool
-
-config RWSEM_XCHGADD_ALGORITHM
- def_bool y
-
config ARCH_HAS_ILOG2_U32
def_bool n
diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
index b1c91ea9a958..0be08d586d40 100644
--- a/arch/sh/Kconfig
+++ b/arch/sh/Kconfig
@@ -90,12 +90,6 @@ config ARCH_DEFCONFIG
default "arch/sh/configs/shx3_defconfig" if SUPERH32
default "arch/sh/configs/cayman_defconfig" if SUPERH64
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
-config RWSEM_XCHGADD_ALGORITHM
- bool
-
config GENERIC_BUG
def_bool y
depends on BUG && SUPERH32
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 40f8f4f73fe8..16b620237816 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -191,14 +191,6 @@ config NR_CPUS
source "kernel/Kconfig.hz"
-config RWSEM_GENERIC_SPINLOCK
- bool
- default y if SPARC32
-
-config RWSEM_XCHGADD_ALGORITHM
- bool
- default y if SPARC64
-
config GENERIC_HWEIGHT
bool
default y
diff --git a/arch/unicore32/Kconfig b/arch/unicore32/Kconfig
index 817d82608712..0d5869c9bf03 100644
--- a/arch/unicore32/Kconfig
+++ b/arch/unicore32/Kconfig
@@ -38,12 +38,6 @@ config STACKTRACE_SUPPORT
config LOCKDEP_SUPPORT
def_bool y
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
-config RWSEM_XCHGADD_ALGORITHM
- bool
-
config ARCH_HAS_ILOG2_U32
bool
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c1f9b3cf437c..fec892bab654 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -268,9 +268,6 @@ config ARCH_MAY_HAVE_PC_FDC
def_bool y
depends on ISA_DMA_API
-config RWSEM_XCHGADD_ALGORITHM
- def_bool y
-
config GENERIC_CALIBRATE_DELAY
def_bool y
diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
index a9e80e44178c..a8985e1f7432 100644
--- a/arch/x86/um/Kconfig
+++ b/arch/x86/um/Kconfig
@@ -32,12 +32,6 @@ config ARCH_DEFCONFIG
default "arch/um/configs/i386_defconfig" if X86_32
default "arch/um/configs/x86_64_defconfig" if X86_64
-config RWSEM_XCHGADD_ALGORITHM
- def_bool 64BIT
-
-config RWSEM_GENERIC_SPINLOCK
- def_bool !RWSEM_XCHGADD_ALGORITHM
-
config 3_LEVEL_PGTABLES
bool "Three-level pagetables" if !64BIT
default 64BIT
diff --git a/arch/xtensa/Kconfig b/arch/xtensa/Kconfig
index 4b9aafe766c5..35c8d91e6106 100644
--- a/arch/xtensa/Kconfig
+++ b/arch/xtensa/Kconfig
@@ -46,9 +46,6 @@ config XTENSA
with reasonable minimum requirements. The Xtensa Linux project has
a home page at <http://www.linux-xtensa.org/>.
-config RWSEM_XCHGADD_ALGORITHM
- def_bool y
-
config GENERIC_HWEIGHT
def_bool y
diff --git a/include/linux/rwsem-spinlock.h b/include/linux/rwsem-spinlock.h
deleted file mode 100644
index e47568363e5e..000000000000
--- a/include/linux/rwsem-spinlock.h
+++ /dev/null
@@ -1,47 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/* rwsem-spinlock.h: fallback C implementation
- *
- * Copyright (c) 2001 David Howells (dhowells@redhat.com).
- * - Derived partially from ideas by Andrea Arcangeli <andrea@suse.de>
- * - Derived also from comments by Linus
- */
-
-#ifndef _LINUX_RWSEM_SPINLOCK_H
-#define _LINUX_RWSEM_SPINLOCK_H
-
-#ifndef _LINUX_RWSEM_H
-#error "please don't include linux/rwsem-spinlock.h directly, use linux/rwsem.h instead"
-#endif
-
-#ifdef __KERNEL__
-/*
- * the rw-semaphore definition
- * - if count is 0 then there are no active readers or writers
- * - if count is +ve then that is the number of active readers
- * - if count is -1 then there is one active writer
- * - if wait_list is not empty, then there are processes waiting for the semaphore
- */
-struct rw_semaphore {
- __s32 count;
- raw_spinlock_t wait_lock;
- struct list_head wait_list;
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
- struct lockdep_map dep_map;
-#endif
-};
-
-#define RWSEM_UNLOCKED_VALUE 0x00000000
-
-extern void __down_read(struct rw_semaphore *sem);
-extern int __must_check __down_read_killable(struct rw_semaphore *sem);
-extern int __down_read_trylock(struct rw_semaphore *sem);
-extern void __down_write(struct rw_semaphore *sem);
-extern int __must_check __down_write_killable(struct rw_semaphore *sem);
-extern int __down_write_trylock(struct rw_semaphore *sem);
-extern void __up_read(struct rw_semaphore *sem);
-extern void __up_write(struct rw_semaphore *sem);
-extern void __downgrade_write(struct rw_semaphore *sem);
-extern int rwsem_is_locked(struct rw_semaphore *sem);
-
-#endif /* __KERNEL__ */
-#endif /* _LINUX_RWSEM_SPINLOCK_H */
diff --git a/include/linux/rwsem.h b/include/linux/rwsem.h
index 6e56006b2cb6..0fc41062c649 100644
--- a/include/linux/rwsem.h
+++ b/include/linux/rwsem.h
@@ -22,10 +22,6 @@
struct rw_semaphore;
-#ifdef CONFIG_RWSEM_GENERIC_SPINLOCK
-#include <linux/rwsem-spinlock.h> /* use a generic implementation */
-#define __RWSEM_INIT_COUNT(name) .count = RWSEM_UNLOCKED_VALUE
-#else
/* All arch specific implementations share the same struct */
struct rw_semaphore {
atomic_long_t count;
@@ -65,7 +61,6 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem)
#define RWSEM_UNLOCKED_VALUE 0L
#define __RWSEM_INIT_COUNT(name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE)
-#endif
/* Common initializer macros and functions */
diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks
index fbba478ae522..e335953fa704 100644
--- a/kernel/Kconfig.locks
+++ b/kernel/Kconfig.locks
@@ -229,7 +229,7 @@ config MUTEX_SPIN_ON_OWNER
config RWSEM_SPIN_ON_OWNER
def_bool y
- depends on SMP && RWSEM_XCHGADD_ALGORITHM && ARCH_SUPPORTS_ATOMIC_RMW
+ depends on SMP && ARCH_SUPPORTS_ATOMIC_RMW
config LOCK_SPIN_ON_OWNER
def_bool y
diff --git a/kernel/locking/Makefile b/kernel/locking/Makefile
index 392c7f23af76..1af83e9ce57d 100644
--- a/kernel/locking/Makefile
+++ b/kernel/locking/Makefile
@@ -3,7 +3,7 @@
# and is generally not a function of system call inputs.
KCOV_INSTRUMENT := n
-obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o
+obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o rwsem-xadd.o
ifdef CONFIG_FUNCTION_TRACER
CFLAGS_REMOVE_lockdep.o = $(CC_FLAGS_FTRACE)
@@ -25,8 +25,6 @@ obj-$(CONFIG_RT_MUTEXES) += rtmutex.o
obj-$(CONFIG_DEBUG_RT_MUTEXES) += rtmutex-debug.o
obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock.o
obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock_debug.o
-obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o
-obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem-xadd.o
obj-$(CONFIG_QUEUED_RWLOCKS) += qrwlock.o
obj-$(CONFIG_LOCK_TORTURE_TEST) += locktorture.o
obj-$(CONFIG_WW_MUTEX_SELFTEST) += test-ww_mutex.o
diff --git a/kernel/locking/rwsem-spinlock.c b/kernel/locking/rwsem-spinlock.c
deleted file mode 100644
index a7ffb2a96ede..000000000000
--- a/kernel/locking/rwsem-spinlock.c
+++ /dev/null
@@ -1,339 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/* rwsem-spinlock.c: R/W semaphores: contention handling functions for
- * generic spinlock implementation
- *
- * Copyright (c) 2001 David Howells (dhowells@redhat.com).
- * - Derived partially from idea by Andrea Arcangeli <andrea@suse.de>
- * - Derived also from comments by Linus
- */
-#include <linux/rwsem.h>
-#include <linux/sched/signal.h>
-#include <linux/sched/debug.h>
-#include <linux/export.h>
-
-enum rwsem_waiter_type {
- RWSEM_WAITING_FOR_WRITE,
- RWSEM_WAITING_FOR_READ
-};
-
-struct rwsem_waiter {
- struct list_head list;
- struct task_struct *task;
- enum rwsem_waiter_type type;
-};
-
-int rwsem_is_locked(struct rw_semaphore *sem)
-{
- int ret = 1;
- unsigned long flags;
-
- if (raw_spin_trylock_irqsave(&sem->wait_lock, flags)) {
- ret = (sem->count != 0);
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
- }
- return ret;
-}
-EXPORT_SYMBOL(rwsem_is_locked);
-
-/*
- * initialise the semaphore
- */
-void __init_rwsem(struct rw_semaphore *sem, const char *name,
- struct lock_class_key *key)
-{
-#ifdef CONFIG_DEBUG_LOCK_ALLOC
- /*
- * Make sure we are not reinitializing a held semaphore:
- */
- debug_check_no_locks_freed((void *)sem, sizeof(*sem));
- lockdep_init_map(&sem->dep_map, name, key, 0);
-#endif
- sem->count = 0;
- raw_spin_lock_init(&sem->wait_lock);
- INIT_LIST_HEAD(&sem->wait_list);
-}
-EXPORT_SYMBOL(__init_rwsem);
-
-/*
- * handle the lock release when processes blocked on it that can now run
- * - if we come here, then:
- * - the 'active count' _reached_ zero
- * - the 'waiting count' is non-zero
- * - the spinlock must be held by the caller
- * - woken process blocks are discarded from the list after having task zeroed
- * - writers are only woken if wakewrite is non-zero
- */
-static inline struct rw_semaphore *
-__rwsem_do_wake(struct rw_semaphore *sem, int wakewrite)
-{
- struct rwsem_waiter *waiter;
- struct task_struct *tsk;
- int woken;
-
- waiter = list_entry(sem->wait_list.next, struct rwsem_waiter, list);
-
- if (waiter->type == RWSEM_WAITING_FOR_WRITE) {
- if (wakewrite)
- /* Wake up a writer. Note that we do not grant it the
- * lock - it will have to acquire it when it runs. */
- wake_up_process(waiter->task);
- goto out;
- }
-
- /* grant an infinite number of read locks to the front of the queue */
- woken = 0;
- do {
- struct list_head *next = waiter->list.next;
-
- list_del(&waiter->list);
- tsk = waiter->task;
- /*
- * Make sure we do not wakeup the next reader before
- * setting the nil condition to grant the next reader;
- * otherwise we could miss the wakeup on the other
- * side and end up sleeping again. See the pairing
- * in rwsem_down_read_failed().
- */
- smp_mb();
- waiter->task = NULL;
- wake_up_process(tsk);
- put_task_struct(tsk);
- woken++;
- if (next == &sem->wait_list)
- break;
- waiter = list_entry(next, struct rwsem_waiter, list);
- } while (waiter->type != RWSEM_WAITING_FOR_WRITE);
-
- sem->count += woken;
-
- out:
- return sem;
-}
-
-/*
- * wake a single writer
- */
-static inline struct rw_semaphore *
-__rwsem_wake_one_writer(struct rw_semaphore *sem)
-{
- struct rwsem_waiter *waiter;
-
- waiter = list_entry(sem->wait_list.next, struct rwsem_waiter, list);
- wake_up_process(waiter->task);
-
- return sem;
-}
-
-/*
- * get a read lock on the semaphore
- */
-int __sched __down_read_common(struct rw_semaphore *sem, int state)
-{
- struct rwsem_waiter waiter;
- unsigned long flags;
-
- raw_spin_lock_irqsave(&sem->wait_lock, flags);
-
- if (sem->count >= 0 && list_empty(&sem->wait_list)) {
- /* granted */
- sem->count++;
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
- goto out;
- }
-
- /* set up my own style of waitqueue */
- waiter.task = current;
- waiter.type = RWSEM_WAITING_FOR_READ;
- get_task_struct(current);
-
- list_add_tail(&waiter.list, &sem->wait_list);
-
- /* wait to be given the lock */
- for (;;) {
- if (!waiter.task)
- break;
- if (signal_pending_state(state, current))
- goto out_nolock;
- set_current_state(state);
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
- schedule();
- raw_spin_lock_irqsave(&sem->wait_lock, flags);
- }
-
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
- out:
- return 0;
-
-out_nolock:
- /*
- * We didn't take the lock, so that there is a writer, which
- * is owner or the first waiter of the sem. If it's a waiter,
- * it will be woken by current owner. Not need to wake anybody.
- */
- list_del(&waiter.list);
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
- return -EINTR;
-}
-
-void __sched __down_read(struct rw_semaphore *sem)
-{
- __down_read_common(sem, TASK_UNINTERRUPTIBLE);
-}
-
-int __sched __down_read_killable(struct rw_semaphore *sem)
-{
- return __down_read_common(sem, TASK_KILLABLE);
-}
-
-/*
- * trylock for reading -- returns 1 if successful, 0 if contention
- */
-int __down_read_trylock(struct rw_semaphore *sem)
-{
- unsigned long flags;
- int ret = 0;
-
-
- raw_spin_lock_irqsave(&sem->wait_lock, flags);
-
- if (sem->count >= 0 && list_empty(&sem->wait_list)) {
- /* granted */
- sem->count++;
- ret = 1;
- }
-
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
-
- return ret;
-}
-
-/*
- * get a write lock on the semaphore
- */
-int __sched __down_write_common(struct rw_semaphore *sem, int state)
-{
- struct rwsem_waiter waiter;
- unsigned long flags;
- int ret = 0;
-
- raw_spin_lock_irqsave(&sem->wait_lock, flags);
-
- /* set up my own style of waitqueue */
- waiter.task = current;
- waiter.type = RWSEM_WAITING_FOR_WRITE;
- list_add_tail(&waiter.list, &sem->wait_list);
-
- /* wait for someone to release the lock */
- for (;;) {
- /*
- * That is the key to support write lock stealing: allows the
- * task already on CPU to get the lock soon rather than put
- * itself into sleep and waiting for system woke it or someone
- * else in the head of the wait list up.
- */
- if (sem->count == 0)
- break;
- if (signal_pending_state(state, current))
- goto out_nolock;
-
- set_current_state(state);
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
- schedule();
- raw_spin_lock_irqsave(&sem->wait_lock, flags);
- }
- /* got the lock */
- sem->count = -1;
- list_del(&waiter.list);
-
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
-
- return ret;
-
-out_nolock:
- list_del(&waiter.list);
- if (!list_empty(&sem->wait_list) && sem->count >= 0)
- __rwsem_do_wake(sem, 0);
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
-
- return -EINTR;
-}
-
-void __sched __down_write(struct rw_semaphore *sem)
-{
- __down_write_common(sem, TASK_UNINTERRUPTIBLE);
-}
-
-int __sched __down_write_killable(struct rw_semaphore *sem)
-{
- return __down_write_common(sem, TASK_KILLABLE);
-}
-
-/*
- * trylock for writing -- returns 1 if successful, 0 if contention
- */
-int __down_write_trylock(struct rw_semaphore *sem)
-{
- unsigned long flags;
- int ret = 0;
-
- raw_spin_lock_irqsave(&sem->wait_lock, flags);
-
- if (sem->count == 0) {
- /* got the lock */
- sem->count = -1;
- ret = 1;
- }
-
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
-
- return ret;
-}
-
-/*
- * release a read lock on the semaphore
- */
-void __up_read(struct rw_semaphore *sem)
-{
- unsigned long flags;
-
- raw_spin_lock_irqsave(&sem->wait_lock, flags);
-
- if (--sem->count == 0 && !list_empty(&sem->wait_list))
- sem = __rwsem_wake_one_writer(sem);
-
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
-}
-
-/*
- * release a write lock on the semaphore
- */
-void __up_write(struct rw_semaphore *sem)
-{
- unsigned long flags;
-
- raw_spin_lock_irqsave(&sem->wait_lock, flags);
-
- sem->count = 0;
- if (!list_empty(&sem->wait_list))
- sem = __rwsem_do_wake(sem, 1);
-
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
-}
-
-/*
- * downgrade a write lock into a read lock
- * - just wake up any readers at the front of the queue
- */
-void __downgrade_write(struct rw_semaphore *sem)
-{
- unsigned long flags;
-
- raw_spin_lock_irqsave(&sem->wait_lock, flags);
-
- sem->count = 1;
- if (!list_empty(&sem->wait_list))
- sem = __rwsem_do_wake(sem, 0);
-
- raw_spin_unlock_irqrestore(&sem->wait_lock, flags);
-}
-
diff --git a/kernel/locking/rwsem.h b/kernel/locking/rwsem.h
index 067e265fa5c1..45ee00236e03 100644
--- a/kernel/locking/rwsem.h
+++ b/kernel/locking/rwsem.h
@@ -153,7 +153,6 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem)
}
#endif
-#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM
/*
* lock for reading
*/
@@ -260,5 +259,3 @@ static inline void __downgrade_write(struct rw_semaphore *sem)
if (tmp < 0)
rwsem_downgrade_wake(sem);
}
-
-#endif /* CONFIG_RWSEM_XCHGADD_ALGORITHM */
--
2.18.1
^ permalink raw reply related
* [PATCH v5 1/3] locking/rwsem: Remove arch specific rwsem files
From: Waiman Long @ 2019-03-22 14:30 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Will Deacon, Thomas Gleixner
Cc: linux-ia64, linux-sh, linux-mips, H. Peter Anvin, sparclinux,
linux-riscv, linux-arch, linux-s390, Davidlohr Bueso,
linux-c6x-dev, linux-hexagon, x86, Waiman Long, uclinux-h8-devel,
linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, openrisc,
Borislav Petkov, linux-arm-kernel, Tim Chen, linux-parisc,
Linus Torvalds, linux-kernel, linux-alpha, nios2-dev,
Andrew Morton, linuxppc-dev
In-Reply-To: <20190322143008.21313-1-longman@redhat.com>
As the generic rwsem-xadd code is using the appropriate acquire and
release versions of the atomic operations, the arch specific rwsem.h
files will not be that much faster than the generic code as long as the
atomic functions are properly implemented. So we can remove those arch
specific rwsem.h and stop building asm/rwsem.h to reduce maintenance
effort.
Currently, only x86, alpha and ia64 have implemented architecture
specific fast paths. I don't have access to alpha and ia64 systems for
testing, but they are legacy systems that are not likely to be updated
to the latest kernel anyway.
By using a rwsem microbenchmark, the total locking rates on a 4-socket
56-core 112-thread x86-64 system before and after the patch were as
follows (mixed means equal # of read and write locks):
Before Patch After Patch
# of Threads wlock rlock mixed wlock rlock mixed
------------ ----- ----- ----- ----- ----- -----
1 29,201 30,143 29,458 28,615 30,172 29,201
2 6,807 13,299 1,171 7,725 15,025 1,804
4 6,504 12,755 1,520 7,127 14,286 1,345
8 6,762 13,412 764 6,826 13,652 726
16 6,693 15,408 662 6,599 15,938 626
32 6,145 15,286 496 5,549 15,487 511
64 5,812 15,495 60 5,858 15,572 60
There were some run-to-run variations for the multi-thread tests. For
x86-64, using the generic C code fast path seems to be a little bit
faster than the assembly version with low lock contention. Looking at
the assembly version of the fast paths, there are assembly to/from C
code wrappers that save and restore all the callee-clobbered registers
(7 registers on x86-64). The assembly generated from the generic C
code doesn't need to do that. That may explain the slight performance
gain here.
The generic asm rwsem.h can also be merged into kernel/locking/rwsem.h
with no code change as no other code other than those under
kernel/locking needs to access the internal rwsem macros and functions.
Signed-off-by: Waiman Long <longman@redhat.com>
---
MAINTAINERS | 1 -
arch/alpha/include/asm/rwsem.h | 211 ----------------------------
arch/arm/include/asm/Kbuild | 1 -
arch/arm64/include/asm/Kbuild | 1 -
arch/hexagon/include/asm/Kbuild | 1 -
arch/ia64/include/asm/rwsem.h | 172 -----------------------
arch/powerpc/include/asm/Kbuild | 1 -
arch/s390/include/asm/Kbuild | 1 -
arch/sh/include/asm/Kbuild | 1 -
arch/sparc/include/asm/Kbuild | 1 -
arch/x86/include/asm/rwsem.h | 237 --------------------------------
arch/x86/lib/Makefile | 1 -
arch/x86/lib/rwsem.S | 156 ---------------------
arch/x86/um/Makefile | 1 -
arch/xtensa/include/asm/Kbuild | 1 -
include/asm-generic/rwsem.h | 140 -------------------
include/linux/rwsem.h | 4 +-
kernel/locking/percpu-rwsem.c | 2 +
kernel/locking/rwsem.h | 130 ++++++++++++++++++
19 files changed, 133 insertions(+), 930 deletions(-)
delete mode 100644 arch/alpha/include/asm/rwsem.h
delete mode 100644 arch/ia64/include/asm/rwsem.h
delete mode 100644 arch/x86/include/asm/rwsem.h
delete mode 100644 arch/x86/lib/rwsem.S
delete mode 100644 include/asm-generic/rwsem.h
diff --git a/MAINTAINERS b/MAINTAINERS
index e17ebf70b548..6bfd5a94c08e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9089,7 +9089,6 @@ F: arch/*/include/asm/spinlock*.h
F: include/linux/rwlock*.h
F: include/linux/mutex*.h
F: include/linux/rwsem*.h
-F: arch/*/include/asm/rwsem.h
F: include/linux/seqlock.h
F: lib/locking*.[ch]
F: kernel/locking/
diff --git a/arch/alpha/include/asm/rwsem.h b/arch/alpha/include/asm/rwsem.h
deleted file mode 100644
index cf8fc8f9a2ed..000000000000
--- a/arch/alpha/include/asm/rwsem.h
+++ /dev/null
@@ -1,211 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ALPHA_RWSEM_H
-#define _ALPHA_RWSEM_H
-
-/*
- * Written by Ivan Kokshaysky <ink@jurassic.park.msu.ru>, 2001.
- * Based on asm-alpha/semaphore.h and asm-i386/rwsem.h
- */
-
-#ifndef _LINUX_RWSEM_H
-#error "please don't include asm/rwsem.h directly, use linux/rwsem.h instead"
-#endif
-
-#ifdef __KERNEL__
-
-#include <linux/compiler.h>
-
-#define RWSEM_UNLOCKED_VALUE 0x0000000000000000L
-#define RWSEM_ACTIVE_BIAS 0x0000000000000001L
-#define RWSEM_ACTIVE_MASK 0x00000000ffffffffL
-#define RWSEM_WAITING_BIAS (-0x0000000100000000L)
-#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS
-#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS)
-
-static inline int ___down_read(struct rw_semaphore *sem)
-{
- long oldcount;
-#ifndef CONFIG_SMP
- oldcount = sem->count.counter;
- sem->count.counter += RWSEM_ACTIVE_READ_BIAS;
-#else
- long temp;
- __asm__ __volatile__(
- "1: ldq_l %0,%1\n"
- " addq %0,%3,%2\n"
- " stq_c %2,%1\n"
- " beq %2,2f\n"
- " mb\n"
- ".subsection 2\n"
- "2: br 1b\n"
- ".previous"
- :"=&r" (oldcount), "=m" (sem->count), "=&r" (temp)
- :"Ir" (RWSEM_ACTIVE_READ_BIAS), "m" (sem->count) : "memory");
-#endif
- return (oldcount < 0);
-}
-
-static inline void __down_read(struct rw_semaphore *sem)
-{
- if (unlikely(___down_read(sem)))
- rwsem_down_read_failed(sem);
-}
-
-static inline int __down_read_killable(struct rw_semaphore *sem)
-{
- if (unlikely(___down_read(sem)))
- if (IS_ERR(rwsem_down_read_failed_killable(sem)))
- return -EINTR;
-
- return 0;
-}
-
-/*
- * trylock for reading -- returns 1 if successful, 0 if contention
- */
-static inline int __down_read_trylock(struct rw_semaphore *sem)
-{
- long old, new, res;
-
- res = atomic_long_read(&sem->count);
- do {
- new = res + RWSEM_ACTIVE_READ_BIAS;
- if (new <= 0)
- break;
- old = res;
- res = atomic_long_cmpxchg(&sem->count, old, new);
- } while (res != old);
- return res >= 0 ? 1 : 0;
-}
-
-static inline long ___down_write(struct rw_semaphore *sem)
-{
- long oldcount;
-#ifndef CONFIG_SMP
- oldcount = sem->count.counter;
- sem->count.counter += RWSEM_ACTIVE_WRITE_BIAS;
-#else
- long temp;
- __asm__ __volatile__(
- "1: ldq_l %0,%1\n"
- " addq %0,%3,%2\n"
- " stq_c %2,%1\n"
- " beq %2,2f\n"
- " mb\n"
- ".subsection 2\n"
- "2: br 1b\n"
- ".previous"
- :"=&r" (oldcount), "=m" (sem->count), "=&r" (temp)
- :"Ir" (RWSEM_ACTIVE_WRITE_BIAS), "m" (sem->count) : "memory");
-#endif
- return oldcount;
-}
-
-static inline void __down_write(struct rw_semaphore *sem)
-{
- if (unlikely(___down_write(sem)))
- rwsem_down_write_failed(sem);
-}
-
-static inline int __down_write_killable(struct rw_semaphore *sem)
-{
- if (unlikely(___down_write(sem))) {
- if (IS_ERR(rwsem_down_write_failed_killable(sem)))
- return -EINTR;
- }
-
- return 0;
-}
-
-/*
- * trylock for writing -- returns 1 if successful, 0 if contention
- */
-static inline int __down_write_trylock(struct rw_semaphore *sem)
-{
- long ret = atomic_long_cmpxchg(&sem->count, RWSEM_UNLOCKED_VALUE,
- RWSEM_ACTIVE_WRITE_BIAS);
- if (ret == RWSEM_UNLOCKED_VALUE)
- return 1;
- return 0;
-}
-
-static inline void __up_read(struct rw_semaphore *sem)
-{
- long oldcount;
-#ifndef CONFIG_SMP
- oldcount = sem->count.counter;
- sem->count.counter -= RWSEM_ACTIVE_READ_BIAS;
-#else
- long temp;
- __asm__ __volatile__(
- " mb\n"
- "1: ldq_l %0,%1\n"
- " subq %0,%3,%2\n"
- " stq_c %2,%1\n"
- " beq %2,2f\n"
- ".subsection 2\n"
- "2: br 1b\n"
- ".previous"
- :"=&r" (oldcount), "=m" (sem->count), "=&r" (temp)
- :"Ir" (RWSEM_ACTIVE_READ_BIAS), "m" (sem->count) : "memory");
-#endif
- if (unlikely(oldcount < 0))
- if ((int)oldcount - RWSEM_ACTIVE_READ_BIAS == 0)
- rwsem_wake(sem);
-}
-
-static inline void __up_write(struct rw_semaphore *sem)
-{
- long count;
-#ifndef CONFIG_SMP
- sem->count.counter -= RWSEM_ACTIVE_WRITE_BIAS;
- count = sem->count.counter;
-#else
- long temp;
- __asm__ __volatile__(
- " mb\n"
- "1: ldq_l %0,%1\n"
- " subq %0,%3,%2\n"
- " stq_c %2,%1\n"
- " beq %2,2f\n"
- " subq %0,%3,%0\n"
- ".subsection 2\n"
- "2: br 1b\n"
- ".previous"
- :"=&r" (count), "=m" (sem->count), "=&r" (temp)
- :"Ir" (RWSEM_ACTIVE_WRITE_BIAS), "m" (sem->count) : "memory");
-#endif
- if (unlikely(count))
- if ((int)count == 0)
- rwsem_wake(sem);
-}
-
-/*
- * downgrade write lock to read lock
- */
-static inline void __downgrade_write(struct rw_semaphore *sem)
-{
- long oldcount;
-#ifndef CONFIG_SMP
- oldcount = sem->count.counter;
- sem->count.counter -= RWSEM_WAITING_BIAS;
-#else
- long temp;
- __asm__ __volatile__(
- "1: ldq_l %0,%1\n"
- " addq %0,%3,%2\n"
- " stq_c %2,%1\n"
- " beq %2,2f\n"
- " mb\n"
- ".subsection 2\n"
- "2: br 1b\n"
- ".previous"
- :"=&r" (oldcount), "=m" (sem->count), "=&r" (temp)
- :"Ir" (-RWSEM_WAITING_BIAS), "m" (sem->count) : "memory");
-#endif
- if (unlikely(oldcount < 0))
- rwsem_downgrade_wake(sem);
-}
-
-#endif /* __KERNEL__ */
-#endif /* _ALPHA_RWSEM_H */
diff --git a/arch/arm/include/asm/Kbuild b/arch/arm/include/asm/Kbuild
index a8a4eb7f6dae..8fb51b7bf1d5 100644
--- a/arch/arm/include/asm/Kbuild
+++ b/arch/arm/include/asm/Kbuild
@@ -12,7 +12,6 @@ generic-y += mm-arch-hooks.h
generic-y += msi.h
generic-y += parport.h
generic-y += preempt.h
-generic-y += rwsem.h
generic-y += seccomp.h
generic-y += segment.h
generic-y += serial.h
diff --git a/arch/arm64/include/asm/Kbuild b/arch/arm64/include/asm/Kbuild
index 1e17ea5c372b..60a933b07001 100644
--- a/arch/arm64/include/asm/Kbuild
+++ b/arch/arm64/include/asm/Kbuild
@@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h
generic-y += msi.h
generic-y += qrwlock.h
generic-y += qspinlock.h
-generic-y += rwsem.h
generic-y += segment.h
generic-y += serial.h
generic-y += set_memory.h
diff --git a/arch/hexagon/include/asm/Kbuild b/arch/hexagon/include/asm/Kbuild
index b25fd42aa0f4..0a12feb41777 100644
--- a/arch/hexagon/include/asm/Kbuild
+++ b/arch/hexagon/include/asm/Kbuild
@@ -26,7 +26,6 @@ generic-y += mm-arch-hooks.h
generic-y += pci.h
generic-y += percpu.h
generic-y += preempt.h
-generic-y += rwsem.h
generic-y += sections.h
generic-y += segment.h
generic-y += serial.h
diff --git a/arch/ia64/include/asm/rwsem.h b/arch/ia64/include/asm/rwsem.h
deleted file mode 100644
index 917910607e0e..000000000000
--- a/arch/ia64/include/asm/rwsem.h
+++ /dev/null
@@ -1,172 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * R/W semaphores for ia64
- *
- * Copyright (C) 2003 Ken Chen <kenneth.w.chen@intel.com>
- * Copyright (C) 2003 Asit Mallick <asit.k.mallick@intel.com>
- * Copyright (C) 2005 Christoph Lameter <cl@linux.com>
- *
- * Based on asm-i386/rwsem.h and other architecture implementation.
- *
- * The MSW of the count is the negated number of active writers and
- * waiting lockers, and the LSW is the total number of active locks.
- *
- * The lock count is initialized to 0 (no active and no waiting lockers).
- *
- * When a writer subtracts WRITE_BIAS, it'll get 0xffffffff00000001 for
- * the case of an uncontended lock. Readers increment by 1 and see a positive
- * value when uncontended, negative if there are writers (and maybe) readers
- * waiting (in which case it goes to sleep).
- */
-
-#ifndef _ASM_IA64_RWSEM_H
-#define _ASM_IA64_RWSEM_H
-
-#ifndef _LINUX_RWSEM_H
-#error "Please don't include <asm/rwsem.h> directly, use <linux/rwsem.h> instead."
-#endif
-
-#include <asm/intrinsics.h>
-
-#define RWSEM_UNLOCKED_VALUE __IA64_UL_CONST(0x0000000000000000)
-#define RWSEM_ACTIVE_BIAS (1L)
-#define RWSEM_ACTIVE_MASK (0xffffffffL)
-#define RWSEM_WAITING_BIAS (-0x100000000L)
-#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS
-#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS)
-
-/*
- * lock for reading
- */
-static inline int
-___down_read (struct rw_semaphore *sem)
-{
- long result = ia64_fetchadd8_acq((unsigned long *)&sem->count.counter, 1);
-
- return (result < 0);
-}
-
-static inline void
-__down_read (struct rw_semaphore *sem)
-{
- if (___down_read(sem))
- rwsem_down_read_failed(sem);
-}
-
-static inline int
-__down_read_killable (struct rw_semaphore *sem)
-{
- if (___down_read(sem))
- if (IS_ERR(rwsem_down_read_failed_killable(sem)))
- return -EINTR;
-
- return 0;
-}
-
-/*
- * lock for writing
- */
-static inline long
-___down_write (struct rw_semaphore *sem)
-{
- long old, new;
-
- do {
- old = atomic_long_read(&sem->count);
- new = old + RWSEM_ACTIVE_WRITE_BIAS;
- } while (atomic_long_cmpxchg_acquire(&sem->count, old, new) != old);
-
- return old;
-}
-
-static inline void
-__down_write (struct rw_semaphore *sem)
-{
- if (___down_write(sem))
- rwsem_down_write_failed(sem);
-}
-
-static inline int
-__down_write_killable (struct rw_semaphore *sem)
-{
- if (___down_write(sem)) {
- if (IS_ERR(rwsem_down_write_failed_killable(sem)))
- return -EINTR;
- }
-
- return 0;
-}
-
-/*
- * unlock after reading
- */
-static inline void
-__up_read (struct rw_semaphore *sem)
-{
- long result = ia64_fetchadd8_rel((unsigned long *)&sem->count.counter, -1);
-
- if (result < 0 && (--result & RWSEM_ACTIVE_MASK) == 0)
- rwsem_wake(sem);
-}
-
-/*
- * unlock after writing
- */
-static inline void
-__up_write (struct rw_semaphore *sem)
-{
- long old, new;
-
- do {
- old = atomic_long_read(&sem->count);
- new = old - RWSEM_ACTIVE_WRITE_BIAS;
- } while (atomic_long_cmpxchg_release(&sem->count, old, new) != old);
-
- if (new < 0 && (new & RWSEM_ACTIVE_MASK) == 0)
- rwsem_wake(sem);
-}
-
-/*
- * trylock for reading -- returns 1 if successful, 0 if contention
- */
-static inline int
-__down_read_trylock (struct rw_semaphore *sem)
-{
- long tmp;
- while ((tmp = atomic_long_read(&sem->count)) >= 0) {
- if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp, tmp+1)) {
- return 1;
- }
- }
- return 0;
-}
-
-/*
- * trylock for writing -- returns 1 if successful, 0 if contention
- */
-static inline int
-__down_write_trylock (struct rw_semaphore *sem)
-{
- long tmp = atomic_long_cmpxchg_acquire(&sem->count,
- RWSEM_UNLOCKED_VALUE, RWSEM_ACTIVE_WRITE_BIAS);
- return tmp == RWSEM_UNLOCKED_VALUE;
-}
-
-/*
- * downgrade write lock to read lock
- */
-static inline void
-__downgrade_write (struct rw_semaphore *sem)
-{
- long old, new;
-
- do {
- old = atomic_long_read(&sem->count);
- new = old - RWSEM_WAITING_BIAS;
- } while (atomic_long_cmpxchg_release(&sem->count, old, new) != old);
-
- if (old < 0)
- rwsem_downgrade_wake(sem);
-}
-
-#endif /* _ASM_IA64_RWSEM_H */
diff --git a/arch/powerpc/include/asm/Kbuild b/arch/powerpc/include/asm/Kbuild
index a0c132bedfae..36bda391e549 100644
--- a/arch/powerpc/include/asm/Kbuild
+++ b/arch/powerpc/include/asm/Kbuild
@@ -8,6 +8,5 @@ generic-y += irq_regs.h
generic-y += local64.h
generic-y += mcs_spinlock.h
generic-y += preempt.h
-generic-y += rwsem.h
generic-y += vtime.h
generic-y += msi.h
diff --git a/arch/s390/include/asm/Kbuild b/arch/s390/include/asm/Kbuild
index 12d77cb11fe5..d5fadefea33c 100644
--- a/arch/s390/include/asm/Kbuild
+++ b/arch/s390/include/asm/Kbuild
@@ -20,7 +20,6 @@ generic-y += local.h
generic-y += local64.h
generic-y += mcs_spinlock.h
generic-y += mm-arch-hooks.h
-generic-y += rwsem.h
generic-y += trace_clock.h
generic-y += unaligned.h
generic-y += word-at-a-time.h
diff --git a/arch/sh/include/asm/Kbuild b/arch/sh/include/asm/Kbuild
index a6ef3fee5f85..b89fce40878b 100644
--- a/arch/sh/include/asm/Kbuild
+++ b/arch/sh/include/asm/Kbuild
@@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h
generic-y += parport.h
generic-y += percpu.h
generic-y += preempt.h
-generic-y += rwsem.h
generic-y += serial.h
generic-y += sizes.h
generic-y += trace_clock.h
diff --git a/arch/sparc/include/asm/Kbuild b/arch/sparc/include/asm/Kbuild
index b82f64e28f55..e843fc04d10b 100644
--- a/arch/sparc/include/asm/Kbuild
+++ b/arch/sparc/include/asm/Kbuild
@@ -17,7 +17,6 @@ generic-y += mm-arch-hooks.h
generic-y += module.h
generic-y += msi.h
generic-y += preempt.h
-generic-y += rwsem.h
generic-y += serial.h
generic-y += trace_clock.h
generic-y += word-at-a-time.h
diff --git a/arch/x86/include/asm/rwsem.h b/arch/x86/include/asm/rwsem.h
deleted file mode 100644
index 4c25cf6caefa..000000000000
--- a/arch/x86/include/asm/rwsem.h
+++ /dev/null
@@ -1,237 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/* rwsem.h: R/W semaphores implemented using XADD/CMPXCHG for i486+
- *
- * Written by David Howells (dhowells@redhat.com).
- *
- * Derived from asm-x86/semaphore.h
- *
- *
- * The MSW of the count is the negated number of active writers and waiting
- * lockers, and the LSW is the total number of active locks
- *
- * The lock count is initialized to 0 (no active and no waiting lockers).
- *
- * When a writer subtracts WRITE_BIAS, it'll get 0xffff0001 for the case of an
- * uncontended lock. This can be determined because XADD returns the old value.
- * Readers increment by 1 and see a positive value when uncontended, negative
- * if there are writers (and maybe) readers waiting (in which case it goes to
- * sleep).
- *
- * The value of WAITING_BIAS supports up to 32766 waiting processes. This can
- * be extended to 65534 by manually checking the whole MSW rather than relying
- * on the S flag.
- *
- * The value of ACTIVE_BIAS supports up to 65535 active processes.
- *
- * This should be totally fair - if anything is waiting, a process that wants a
- * lock will go to the back of the queue. When the currently active lock is
- * released, if there's a writer at the front of the queue, then that and only
- * that will be woken up; if there's a bunch of consecutive readers at the
- * front, then they'll all be woken up, but no other readers will be.
- */
-
-#ifndef _ASM_X86_RWSEM_H
-#define _ASM_X86_RWSEM_H
-
-#ifndef _LINUX_RWSEM_H
-#error "please don't include asm/rwsem.h directly, use linux/rwsem.h instead"
-#endif
-
-#ifdef __KERNEL__
-#include <asm/asm.h>
-
-/*
- * The bias values and the counter type limits the number of
- * potential readers/writers to 32767 for 32 bits and 2147483647
- * for 64 bits.
- */
-
-#ifdef CONFIG_X86_64
-# define RWSEM_ACTIVE_MASK 0xffffffffL
-#else
-# define RWSEM_ACTIVE_MASK 0x0000ffffL
-#endif
-
-#define RWSEM_UNLOCKED_VALUE 0x00000000L
-#define RWSEM_ACTIVE_BIAS 0x00000001L
-#define RWSEM_WAITING_BIAS (-RWSEM_ACTIVE_MASK-1)
-#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS
-#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS)
-
-/*
- * lock for reading
- */
-#define ____down_read(sem, slow_path) \
-({ \
- struct rw_semaphore* ret; \
- asm volatile("# beginning down_read\n\t" \
- LOCK_PREFIX _ASM_INC "(%[sem])\n\t" \
- /* adds 0x00000001 */ \
- " jns 1f\n" \
- " call " slow_path "\n" \
- "1:\n\t" \
- "# ending down_read\n\t" \
- : "+m" (sem->count), "=a" (ret), \
- ASM_CALL_CONSTRAINT \
- : [sem] "a" (sem) \
- : "memory", "cc"); \
- ret; \
-})
-
-static inline void __down_read(struct rw_semaphore *sem)
-{
- ____down_read(sem, "call_rwsem_down_read_failed");
-}
-
-static inline int __down_read_killable(struct rw_semaphore *sem)
-{
- if (IS_ERR(____down_read(sem, "call_rwsem_down_read_failed_killable")))
- return -EINTR;
- return 0;
-}
-
-/*
- * trylock for reading -- returns 1 if successful, 0 if contention
- */
-static inline bool __down_read_trylock(struct rw_semaphore *sem)
-{
- long result, tmp;
- asm volatile("# beginning __down_read_trylock\n\t"
- " mov %[count],%[result]\n\t"
- "1:\n\t"
- " mov %[result],%[tmp]\n\t"
- " add %[inc],%[tmp]\n\t"
- " jle 2f\n\t"
- LOCK_PREFIX " cmpxchg %[tmp],%[count]\n\t"
- " jnz 1b\n\t"
- "2:\n\t"
- "# ending __down_read_trylock\n\t"
- : [count] "+m" (sem->count), [result] "=&a" (result),
- [tmp] "=&r" (tmp)
- : [inc] "i" (RWSEM_ACTIVE_READ_BIAS)
- : "memory", "cc");
- return result >= 0;
-}
-
-/*
- * lock for writing
- */
-#define ____down_write(sem, slow_path) \
-({ \
- long tmp; \
- struct rw_semaphore* ret; \
- \
- asm volatile("# beginning down_write\n\t" \
- LOCK_PREFIX " xadd %[tmp],(%[sem])\n\t" \
- /* adds 0xffff0001, returns the old value */ \
- " test " __ASM_SEL(%w1,%k1) "," __ASM_SEL(%w1,%k1) "\n\t" \
- /* was the active mask 0 before? */\
- " jz 1f\n" \
- " call " slow_path "\n" \
- "1:\n" \
- "# ending down_write" \
- : "+m" (sem->count), [tmp] "=d" (tmp), \
- "=a" (ret), ASM_CALL_CONSTRAINT \
- : [sem] "a" (sem), "[tmp]" (RWSEM_ACTIVE_WRITE_BIAS) \
- : "memory", "cc"); \
- ret; \
-})
-
-static inline void __down_write(struct rw_semaphore *sem)
-{
- ____down_write(sem, "call_rwsem_down_write_failed");
-}
-
-static inline int __down_write_killable(struct rw_semaphore *sem)
-{
- if (IS_ERR(____down_write(sem, "call_rwsem_down_write_failed_killable")))
- return -EINTR;
-
- return 0;
-}
-
-/*
- * trylock for writing -- returns 1 if successful, 0 if contention
- */
-static inline bool __down_write_trylock(struct rw_semaphore *sem)
-{
- bool result;
- long tmp0, tmp1;
- asm volatile("# beginning __down_write_trylock\n\t"
- " mov %[count],%[tmp0]\n\t"
- "1:\n\t"
- " test " __ASM_SEL(%w1,%k1) "," __ASM_SEL(%w1,%k1) "\n\t"
- /* was the active mask 0 before? */
- " jnz 2f\n\t"
- " mov %[tmp0],%[tmp1]\n\t"
- " add %[inc],%[tmp1]\n\t"
- LOCK_PREFIX " cmpxchg %[tmp1],%[count]\n\t"
- " jnz 1b\n\t"
- "2:\n\t"
- CC_SET(e)
- "# ending __down_write_trylock\n\t"
- : [count] "+m" (sem->count), [tmp0] "=&a" (tmp0),
- [tmp1] "=&r" (tmp1), CC_OUT(e) (result)
- : [inc] "er" (RWSEM_ACTIVE_WRITE_BIAS)
- : "memory");
- return result;
-}
-
-/*
- * unlock after reading
- */
-static inline void __up_read(struct rw_semaphore *sem)
-{
- long tmp;
- asm volatile("# beginning __up_read\n\t"
- LOCK_PREFIX " xadd %[tmp],(%[sem])\n\t"
- /* subtracts 1, returns the old value */
- " jns 1f\n\t"
- " call call_rwsem_wake\n" /* expects old value in %edx */
- "1:\n"
- "# ending __up_read\n"
- : "+m" (sem->count), [tmp] "=d" (tmp)
- : [sem] "a" (sem), "[tmp]" (-RWSEM_ACTIVE_READ_BIAS)
- : "memory", "cc");
-}
-
-/*
- * unlock after writing
- */
-static inline void __up_write(struct rw_semaphore *sem)
-{
- long tmp;
- asm volatile("# beginning __up_write\n\t"
- LOCK_PREFIX " xadd %[tmp],(%[sem])\n\t"
- /* subtracts 0xffff0001, returns the old value */
- " jns 1f\n\t"
- " call call_rwsem_wake\n" /* expects old value in %edx */
- "1:\n\t"
- "# ending __up_write\n"
- : "+m" (sem->count), [tmp] "=d" (tmp)
- : [sem] "a" (sem), "[tmp]" (-RWSEM_ACTIVE_WRITE_BIAS)
- : "memory", "cc");
-}
-
-/*
- * downgrade write lock to read lock
- */
-static inline void __downgrade_write(struct rw_semaphore *sem)
-{
- asm volatile("# beginning __downgrade_write\n\t"
- LOCK_PREFIX _ASM_ADD "%[inc],(%[sem])\n\t"
- /*
- * transitions 0xZZZZ0001 -> 0xYYYY0001 (i386)
- * 0xZZZZZZZZ00000001 -> 0xYYYYYYYY00000001 (x86_64)
- */
- " jns 1f\n\t"
- " call call_rwsem_downgrade_wake\n"
- "1:\n\t"
- "# ending __downgrade_write\n"
- : "+m" (sem->count)
- : [sem] "a" (sem), [inc] "er" (-RWSEM_WAITING_BIAS)
- : "memory", "cc");
-}
-
-#endif /* __KERNEL__ */
-#endif /* _ASM_X86_RWSEM_H */
diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
index 140e61843a07..986652064b15 100644
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -23,7 +23,6 @@ obj-$(CONFIG_SMP) += msr-smp.o cache-smp.o
lib-y := delay.o misc.o cmdline.o cpu.o
lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o
lib-y += memcpy_$(BITS).o
-lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o
lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o
lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o
lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
diff --git a/arch/x86/lib/rwsem.S b/arch/x86/lib/rwsem.S
deleted file mode 100644
index dc2ab6ea6768..000000000000
--- a/arch/x86/lib/rwsem.S
+++ /dev/null
@@ -1,156 +0,0 @@
-/*
- * x86 semaphore implementation.
- *
- * (C) Copyright 1999 Linus Torvalds
- *
- * Portions Copyright 1999 Red Hat, Inc.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version
- * 2 of the License, or (at your option) any later version.
- *
- * rw semaphores implemented November 1999 by Benjamin LaHaise <bcrl@kvack.org>
- */
-
-#include <linux/linkage.h>
-#include <asm/alternative-asm.h>
-#include <asm/frame.h>
-
-#define __ASM_HALF_REG(reg) __ASM_SEL(reg, e##reg)
-#define __ASM_HALF_SIZE(inst) __ASM_SEL(inst##w, inst##l)
-
-#ifdef CONFIG_X86_32
-
-/*
- * The semaphore operations have a special calling sequence that
- * allow us to do a simpler in-line version of them. These routines
- * need to convert that sequence back into the C sequence when
- * there is contention on the semaphore.
- *
- * %eax contains the semaphore pointer on entry. Save the C-clobbered
- * registers (%eax, %edx and %ecx) except %eax which is either a return
- * value or just gets clobbered. Same is true for %edx so make sure GCC
- * reloads it after the slow path, by making it hold a temporary, for
- * example see ____down_write().
- */
-
-#define save_common_regs \
- pushl %ecx
-
-#define restore_common_regs \
- popl %ecx
-
- /* Avoid uglifying the argument copying x86-64 needs to do. */
- .macro movq src, dst
- .endm
-
-#else
-
-/*
- * x86-64 rwsem wrappers
- *
- * This interfaces the inline asm code to the slow-path
- * C routines. We need to save the call-clobbered regs
- * that the asm does not mark as clobbered, and move the
- * argument from %rax to %rdi.
- *
- * NOTE! We don't need to save %rax, because the functions
- * will always return the semaphore pointer in %rax (which
- * is also the input argument to these helpers)
- *
- * The following can clobber %rdx because the asm clobbers it:
- * call_rwsem_down_write_failed
- * call_rwsem_wake
- * but %rdi, %rsi, %rcx, %r8-r11 always need saving.
- */
-
-#define save_common_regs \
- pushq %rdi; \
- pushq %rsi; \
- pushq %rcx; \
- pushq %r8; \
- pushq %r9; \
- pushq %r10; \
- pushq %r11
-
-#define restore_common_regs \
- popq %r11; \
- popq %r10; \
- popq %r9; \
- popq %r8; \
- popq %rcx; \
- popq %rsi; \
- popq %rdi
-
-#endif
-
-/* Fix up special calling conventions */
-ENTRY(call_rwsem_down_read_failed)
- FRAME_BEGIN
- save_common_regs
- __ASM_SIZE(push,) %__ASM_REG(dx)
- movq %rax,%rdi
- call rwsem_down_read_failed
- __ASM_SIZE(pop,) %__ASM_REG(dx)
- restore_common_regs
- FRAME_END
- ret
-ENDPROC(call_rwsem_down_read_failed)
-
-ENTRY(call_rwsem_down_read_failed_killable)
- FRAME_BEGIN
- save_common_regs
- __ASM_SIZE(push,) %__ASM_REG(dx)
- movq %rax,%rdi
- call rwsem_down_read_failed_killable
- __ASM_SIZE(pop,) %__ASM_REG(dx)
- restore_common_regs
- FRAME_END
- ret
-ENDPROC(call_rwsem_down_read_failed_killable)
-
-ENTRY(call_rwsem_down_write_failed)
- FRAME_BEGIN
- save_common_regs
- movq %rax,%rdi
- call rwsem_down_write_failed
- restore_common_regs
- FRAME_END
- ret
-ENDPROC(call_rwsem_down_write_failed)
-
-ENTRY(call_rwsem_down_write_failed_killable)
- FRAME_BEGIN
- save_common_regs
- movq %rax,%rdi
- call rwsem_down_write_failed_killable
- restore_common_regs
- FRAME_END
- ret
-ENDPROC(call_rwsem_down_write_failed_killable)
-
-ENTRY(call_rwsem_wake)
- FRAME_BEGIN
- /* do nothing if still outstanding active readers */
- __ASM_HALF_SIZE(dec) %__ASM_HALF_REG(dx)
- jnz 1f
- save_common_regs
- movq %rax,%rdi
- call rwsem_wake
- restore_common_regs
-1: FRAME_END
- ret
-ENDPROC(call_rwsem_wake)
-
-ENTRY(call_rwsem_downgrade_wake)
- FRAME_BEGIN
- save_common_regs
- __ASM_SIZE(push,) %__ASM_REG(dx)
- movq %rax,%rdi
- call rwsem_downgrade_wake
- __ASM_SIZE(pop,) %__ASM_REG(dx)
- restore_common_regs
- FRAME_END
- ret
-ENDPROC(call_rwsem_downgrade_wake)
diff --git a/arch/x86/um/Makefile b/arch/x86/um/Makefile
index 2d686ae54681..22554c585ced 100644
--- a/arch/x86/um/Makefile
+++ b/arch/x86/um/Makefile
@@ -21,7 +21,6 @@ obj-y += checksum_32.o syscalls_32.o
obj-$(CONFIG_ELF_CORE) += elfcore.o
subarch-y = ../lib/string_32.o ../lib/atomic64_32.o ../lib/atomic64_cx8_32.o
-subarch-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += ../lib/rwsem.o
else
diff --git a/arch/xtensa/include/asm/Kbuild b/arch/xtensa/include/asm/Kbuild
index 42b6cb3d16f7..ea97bdebfba7 100644
--- a/arch/xtensa/include/asm/Kbuild
+++ b/arch/xtensa/include/asm/Kbuild
@@ -24,7 +24,6 @@ generic-y += percpu.h
generic-y += preempt.h
generic-y += qrwlock.h
generic-y += qspinlock.h
-generic-y += rwsem.h
generic-y += sections.h
generic-y += socket.h
generic-y += topology.h
diff --git a/include/asm-generic/rwsem.h b/include/asm-generic/rwsem.h
deleted file mode 100644
index 93e67a055a4d..000000000000
--- a/include/asm-generic/rwsem.h
+++ /dev/null
@@ -1,140 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ASM_GENERIC_RWSEM_H
-#define _ASM_GENERIC_RWSEM_H
-
-#ifndef _LINUX_RWSEM_H
-#error "Please don't include <asm/rwsem.h> directly, use <linux/rwsem.h> instead."
-#endif
-
-#ifdef __KERNEL__
-
-/*
- * R/W semaphores originally for PPC using the stuff in lib/rwsem.c.
- * Adapted largely from include/asm-i386/rwsem.h
- * by Paul Mackerras <paulus@samba.org>.
- */
-
-/*
- * the semaphore definition
- */
-#ifdef CONFIG_64BIT
-# define RWSEM_ACTIVE_MASK 0xffffffffL
-#else
-# define RWSEM_ACTIVE_MASK 0x0000ffffL
-#endif
-
-#define RWSEM_UNLOCKED_VALUE 0x00000000L
-#define RWSEM_ACTIVE_BIAS 0x00000001L
-#define RWSEM_WAITING_BIAS (-RWSEM_ACTIVE_MASK-1)
-#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS
-#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS)
-
-/*
- * lock for reading
- */
-static inline void __down_read(struct rw_semaphore *sem)
-{
- if (unlikely(atomic_long_inc_return_acquire(&sem->count) <= 0))
- rwsem_down_read_failed(sem);
-}
-
-static inline int __down_read_killable(struct rw_semaphore *sem)
-{
- if (unlikely(atomic_long_inc_return_acquire(&sem->count) <= 0)) {
- if (IS_ERR(rwsem_down_read_failed_killable(sem)))
- return -EINTR;
- }
-
- return 0;
-}
-
-static inline int __down_read_trylock(struct rw_semaphore *sem)
-{
- long tmp;
-
- while ((tmp = atomic_long_read(&sem->count)) >= 0) {
- if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp,
- tmp + RWSEM_ACTIVE_READ_BIAS)) {
- return 1;
- }
- }
- return 0;
-}
-
-/*
- * lock for writing
- */
-static inline void __down_write(struct rw_semaphore *sem)
-{
- long tmp;
-
- tmp = atomic_long_add_return_acquire(RWSEM_ACTIVE_WRITE_BIAS,
- &sem->count);
- if (unlikely(tmp != RWSEM_ACTIVE_WRITE_BIAS))
- rwsem_down_write_failed(sem);
-}
-
-static inline int __down_write_killable(struct rw_semaphore *sem)
-{
- long tmp;
-
- tmp = atomic_long_add_return_acquire(RWSEM_ACTIVE_WRITE_BIAS,
- &sem->count);
- if (unlikely(tmp != RWSEM_ACTIVE_WRITE_BIAS))
- if (IS_ERR(rwsem_down_write_failed_killable(sem)))
- return -EINTR;
- return 0;
-}
-
-static inline int __down_write_trylock(struct rw_semaphore *sem)
-{
- long tmp;
-
- tmp = atomic_long_cmpxchg_acquire(&sem->count, RWSEM_UNLOCKED_VALUE,
- RWSEM_ACTIVE_WRITE_BIAS);
- return tmp == RWSEM_UNLOCKED_VALUE;
-}
-
-/*
- * unlock after reading
- */
-static inline void __up_read(struct rw_semaphore *sem)
-{
- long tmp;
-
- tmp = atomic_long_dec_return_release(&sem->count);
- if (unlikely(tmp < -1 && (tmp & RWSEM_ACTIVE_MASK) == 0))
- rwsem_wake(sem);
-}
-
-/*
- * unlock after writing
- */
-static inline void __up_write(struct rw_semaphore *sem)
-{
- if (unlikely(atomic_long_sub_return_release(RWSEM_ACTIVE_WRITE_BIAS,
- &sem->count) < 0))
- rwsem_wake(sem);
-}
-
-/*
- * downgrade write lock to read lock
- */
-static inline void __downgrade_write(struct rw_semaphore *sem)
-{
- long tmp;
-
- /*
- * When downgrading from exclusive to shared ownership,
- * anything inside the write-locked region cannot leak
- * into the read side. In contrast, anything in the
- * read-locked region is ok to be re-ordered into the
- * write side. As such, rely on RELEASE semantics.
- */
- tmp = atomic_long_add_return_release(-RWSEM_WAITING_BIAS, &sem->count);
- if (tmp < 0)
- rwsem_downgrade_wake(sem);
-}
-
-#endif /* __KERNEL__ */
-#endif /* _ASM_GENERIC_RWSEM_H */
diff --git a/include/linux/rwsem.h b/include/linux/rwsem.h
index 67dbb57508b1..6e56006b2cb6 100644
--- a/include/linux/rwsem.h
+++ b/include/linux/rwsem.h
@@ -57,15 +57,13 @@ extern struct rw_semaphore *rwsem_down_write_failed_killable(struct rw_semaphore
extern struct rw_semaphore *rwsem_wake(struct rw_semaphore *);
extern struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem);
-/* Include the arch specific part */
-#include <asm/rwsem.h>
-
/* In all implementations count != 0 means locked */
static inline int rwsem_is_locked(struct rw_semaphore *sem)
{
return atomic_long_read(&sem->count) != 0;
}
+#define RWSEM_UNLOCKED_VALUE 0L
#define __RWSEM_INIT_COUNT(name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE)
#endif
diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c
index 883cf1b92d90..f17dad99eec8 100644
--- a/kernel/locking/percpu-rwsem.c
+++ b/kernel/locking/percpu-rwsem.c
@@ -7,6 +7,8 @@
#include <linux/sched.h>
#include <linux/errno.h>
+#include "rwsem.h"
+
int __percpu_init_rwsem(struct percpu_rw_semaphore *sem,
const char *name, struct lock_class_key *rwsem_key)
{
diff --git a/kernel/locking/rwsem.h b/kernel/locking/rwsem.h
index bad2bca0268b..067e265fa5c1 100644
--- a/kernel/locking/rwsem.h
+++ b/kernel/locking/rwsem.h
@@ -32,6 +32,26 @@
# define DEBUG_RWSEMS_WARN_ON(c)
#endif
+/*
+ * R/W semaphores originally for PPC using the stuff in lib/rwsem.c.
+ * Adapted largely from include/asm-i386/rwsem.h
+ * by Paul Mackerras <paulus@samba.org>.
+ */
+
+/*
+ * the semaphore definition
+ */
+#ifdef CONFIG_64BIT
+# define RWSEM_ACTIVE_MASK 0xffffffffL
+#else
+# define RWSEM_ACTIVE_MASK 0x0000ffffL
+#endif
+
+#define RWSEM_ACTIVE_BIAS 0x00000001L
+#define RWSEM_WAITING_BIAS (-RWSEM_ACTIVE_MASK-1)
+#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS
+#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS)
+
#ifdef CONFIG_RWSEM_SPIN_ON_OWNER
/*
* All writes to owner are protected by WRITE_ONCE() to make sure that
@@ -132,3 +152,113 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem)
{
}
#endif
+
+#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM
+/*
+ * lock for reading
+ */
+static inline void __down_read(struct rw_semaphore *sem)
+{
+ if (unlikely(atomic_long_inc_return_acquire(&sem->count) <= 0))
+ rwsem_down_read_failed(sem);
+}
+
+static inline int __down_read_killable(struct rw_semaphore *sem)
+{
+ if (unlikely(atomic_long_inc_return_acquire(&sem->count) <= 0)) {
+ if (IS_ERR(rwsem_down_read_failed_killable(sem)))
+ return -EINTR;
+ }
+
+ return 0;
+}
+
+static inline int __down_read_trylock(struct rw_semaphore *sem)
+{
+ long tmp;
+
+ while ((tmp = atomic_long_read(&sem->count)) >= 0) {
+ if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp,
+ tmp + RWSEM_ACTIVE_READ_BIAS)) {
+ return 1;
+ }
+ }
+ return 0;
+}
+
+/*
+ * lock for writing
+ */
+static inline void __down_write(struct rw_semaphore *sem)
+{
+ long tmp;
+
+ tmp = atomic_long_add_return_acquire(RWSEM_ACTIVE_WRITE_BIAS,
+ &sem->count);
+ if (unlikely(tmp != RWSEM_ACTIVE_WRITE_BIAS))
+ rwsem_down_write_failed(sem);
+}
+
+static inline int __down_write_killable(struct rw_semaphore *sem)
+{
+ long tmp;
+
+ tmp = atomic_long_add_return_acquire(RWSEM_ACTIVE_WRITE_BIAS,
+ &sem->count);
+ if (unlikely(tmp != RWSEM_ACTIVE_WRITE_BIAS))
+ if (IS_ERR(rwsem_down_write_failed_killable(sem)))
+ return -EINTR;
+ return 0;
+}
+
+static inline int __down_write_trylock(struct rw_semaphore *sem)
+{
+ long tmp;
+
+ tmp = atomic_long_cmpxchg_acquire(&sem->count, RWSEM_UNLOCKED_VALUE,
+ RWSEM_ACTIVE_WRITE_BIAS);
+ return tmp == RWSEM_UNLOCKED_VALUE;
+}
+
+/*
+ * unlock after reading
+ */
+static inline void __up_read(struct rw_semaphore *sem)
+{
+ long tmp;
+
+ tmp = atomic_long_dec_return_release(&sem->count);
+ if (unlikely(tmp < -1 && (tmp & RWSEM_ACTIVE_MASK) == 0))
+ rwsem_wake(sem);
+}
+
+/*
+ * unlock after writing
+ */
+static inline void __up_write(struct rw_semaphore *sem)
+{
+ if (unlikely(atomic_long_sub_return_release(RWSEM_ACTIVE_WRITE_BIAS,
+ &sem->count) < 0))
+ rwsem_wake(sem);
+}
+
+/*
+ * downgrade write lock to read lock
+ */
+static inline void __downgrade_write(struct rw_semaphore *sem)
+{
+ long tmp;
+
+ /*
+ * When downgrading from exclusive to shared ownership,
+ * anything inside the write-locked region cannot leak
+ * into the read side. In contrast, anything in the
+ * read-locked region is ok to be re-ordered into the
+ * write side. As such, rely on RELEASE semantics.
+ */
+ tmp = atomic_long_add_return_release(-RWSEM_WAITING_BIAS, &sem->count);
+ if (tmp < 0)
+ rwsem_downgrade_wake(sem);
+}
+
+#endif /* CONFIG_RWSEM_XCHGADD_ALGORITHM */
--
2.18.1
^ permalink raw reply related
* [PATCH v5 0/3] locking/rwsem: Rwsem rearchitecture part 0
From: Waiman Long @ 2019-03-22 14:30 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar, Will Deacon, Thomas Gleixner
Cc: linux-ia64, linux-sh, linux-mips, H. Peter Anvin, sparclinux,
linux-riscv, linux-arch, linux-s390, Davidlohr Bueso,
linux-c6x-dev, linux-hexagon, x86, Waiman Long, uclinux-h8-devel,
linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, openrisc,
Borislav Petkov, linux-arm-kernel, Tim Chen, linux-parisc,
Linus Torvalds, linux-kernel, linux-alpha, nios2-dev,
Andrew Morton, linuxppc-dev
v5:
- Rebase to the latest v5.1 tree and fix conflicts in
arch/{xtensa,s390}/include/asm/Kbuild.
v4:
- Remove rwsem-spinlock.c and make all archs use rwsem-xadd.c.
v3:
- Optimize __down_read_trylock() for the uncontended case as suggested
by Linus.
v2:
- Add patch 2 to optimize __down_read_trylock() as suggested by PeterZ.
- Update performance test data in patch 1.
The goal of this patchset is to remove the architecture specific files
for rwsem-xadd to make it easer to add enhancements in the later rwsem
patches. It also removes the legacy rwsem-spinlock.c file and make all
the architectures use one single implementation of rwsem - rwsem-xadd.c.
Waiman Long (3):
locking/rwsem: Remove arch specific rwsem files
locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all
archs
locking/rwsem: Optimize down_read_trylock()
MAINTAINERS | 1 -
arch/alpha/Kconfig | 7 -
arch/alpha/include/asm/rwsem.h | 211 --------------------
arch/arc/Kconfig | 3 -
arch/arm/Kconfig | 4 -
arch/arm/include/asm/Kbuild | 1 -
arch/arm64/Kconfig | 3 -
arch/arm64/include/asm/Kbuild | 1 -
arch/c6x/Kconfig | 3 -
arch/csky/Kconfig | 3 -
arch/h8300/Kconfig | 3 -
arch/hexagon/Kconfig | 6 -
arch/hexagon/include/asm/Kbuild | 1 -
arch/ia64/Kconfig | 4 -
arch/ia64/include/asm/rwsem.h | 172 ----------------
arch/m68k/Kconfig | 7 -
arch/microblaze/Kconfig | 6 -
arch/mips/Kconfig | 7 -
arch/nds32/Kconfig | 3 -
arch/nios2/Kconfig | 3 -
arch/openrisc/Kconfig | 6 -
arch/parisc/Kconfig | 6 -
arch/powerpc/Kconfig | 7 -
arch/powerpc/include/asm/Kbuild | 1 -
arch/riscv/Kconfig | 3 -
arch/s390/Kconfig | 6 -
arch/s390/include/asm/Kbuild | 1 -
arch/sh/Kconfig | 6 -
arch/sh/include/asm/Kbuild | 1 -
arch/sparc/Kconfig | 8 -
arch/sparc/include/asm/Kbuild | 1 -
arch/unicore32/Kconfig | 6 -
arch/x86/Kconfig | 3 -
arch/x86/include/asm/rwsem.h | 237 ----------------------
arch/x86/lib/Makefile | 1 -
arch/x86/lib/rwsem.S | 156 ---------------
arch/x86/um/Kconfig | 6 -
arch/x86/um/Makefile | 1 -
arch/xtensa/Kconfig | 3 -
arch/xtensa/include/asm/Kbuild | 1 -
include/asm-generic/rwsem.h | 140 -------------
include/linux/rwsem-spinlock.h | 47 -----
include/linux/rwsem.h | 9 +-
kernel/Kconfig.locks | 2 +-
kernel/locking/Makefile | 4 +-
kernel/locking/percpu-rwsem.c | 2 +
kernel/locking/rwsem-spinlock.c | 339 --------------------------------
kernel/locking/rwsem.h | 130 ++++++++++++
48 files changed, 135 insertions(+), 1447 deletions(-)
delete mode 100644 arch/alpha/include/asm/rwsem.h
delete mode 100644 arch/ia64/include/asm/rwsem.h
delete mode 100644 arch/x86/include/asm/rwsem.h
delete mode 100644 arch/x86/lib/rwsem.S
delete mode 100644 include/asm-generic/rwsem.h
delete mode 100644 include/linux/rwsem-spinlock.h
delete mode 100644 kernel/locking/rwsem-spinlock.c
--
2.18.1
^ permalink raw reply
* Re: powerpc32 boot crash in 5.1-rc1
From: Meelis Roos @ 2019-03-22 14:21 UTC (permalink / raw)
To: LEROY Christophe; +Cc: linuxppc-dev, linux-kernel
In-Reply-To: <20190321221632.Horde.sUFPCFV5Nrk0KQFVNqmMjw2@messagerie.si.c-s.fr>
>> While 5.0.0 worked fine on my PowerMac G4, 5.0 + git (unknown rev as of Mar 13), 5.0.0-11520-gf261c4e and todays git all fail to boot.
>>
>> The problem seems to be in page fault handler in load_elf_binary() of init process.
>
> The patch at https://patchwork.ozlabs.org/patch/1053385/ should fix it
Tested it - yes, it fixes the boot.
--
Meelis Roos <mroos@linux.ee>
^ permalink raw reply
* [RFC PATCH v1 3/3] kasan: add interceptors for all string functions
From: Christophe Leroy @ 2019-03-22 14:00 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
Nicholas Piggin, Aneesh Kumar K.V, Andrey Ryabinin,
Alexander Potapenko, Dmitry Vyukov, Daniel Axtens
Cc: linux-mm, linuxppc-dev, linux-kernel, kasan-dev
In-Reply-To: <45a5e13683694fc8d4574b52c4851ffb7f5e5fbd.1553263058.git.christophe.leroy@c-s.fr>
In the same spirit as commit 393f203f5fd5 ("x86_64: kasan: add
interceptors for memset/memmove/memcpy functions"), this patch
adds interceptors for string manipulation functions so that we
can compile lib/string.o without kasan support hence allow the
string functions to also be used from places where kasan has
to be disabled.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
This is the generic part. If we agree on the principle, then I'll
go through the arches and see if adaptations need to be done there.
include/linux/string.h | 79 ++++++++++++
lib/Makefile | 2 +
mm/kasan/string.c | 334 +++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 415 insertions(+)
diff --git a/include/linux/string.h b/include/linux/string.h
index 7927b875f80c..7e7441f4c420 100644
--- a/include/linux/string.h
+++ b/include/linux/string.h
@@ -19,54 +19,117 @@ extern void *memdup_user_nul(const void __user *, size_t);
*/
#include <asm/string.h>
+#if defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__)
+/*
+ * For files that are not instrumented (e.g. mm/slub.c) we
+ * should use not instrumented version of mem* functions.
+ */
+#define memset16 __memset16
+#define memset32 __memset32
+#define memset64 __memset64
+#define memzero_explicit __memzero_explicit
+#define strcpy __strcpy
+#define strncpy __strncpy
+#define strlcpy __strlcpy
+#define strscpy __strscpy
+#define strcat __strcat
+#define strncat __strncat
+#define strlcat __strlcat
+#define strcmp __strcmp
+#define strncmp __strncmp
+#define strcasecmp __strcasecmp
+#define strncasecmp __strncasecmp
+#define strchr __strchr
+#define strchrnul __strchrnul
+#define strrchr __strrchr
+#define strnchr __strnchr
+#define skip_spaces __skip_spaces
+#define strim __strim
+#define strstr __strstr
+#define strnstr __strnstr
+#define strlen __strlen
+#define strnlen __strnlen
+#define strpbrk __strpbrk
+#define strsep __strsep
+#define strspn __strspn
+#define strcspn __strcspn
+#define memscan __memscan
+#define memcmp __memcmp
+#define memchr __memchr
+#define memchr_inv __memchr_inv
+#define strreplace __strreplace
+
+#ifndef __NO_FORTIFY
+#define __NO_FORTIFY /* FORTIFY_SOURCE uses __builtin_memcpy, etc. */
+#endif
+
+#endif
+
#ifndef __HAVE_ARCH_STRCPY
extern char * strcpy(char *,const char *);
+char *__strcpy(char *,const char *);
#endif
#ifndef __HAVE_ARCH_STRNCPY
extern char * strncpy(char *,const char *, __kernel_size_t);
+char *__strncpy(char *,const char *, __kernel_size_t);
#endif
#ifndef __HAVE_ARCH_STRLCPY
size_t strlcpy(char *, const char *, size_t);
+size_t __strlcpy(char *, const char *, size_t);
#endif
#ifndef __HAVE_ARCH_STRSCPY
ssize_t strscpy(char *, const char *, size_t);
+ssize_t __strscpy(char *, const char *, size_t);
#endif
#ifndef __HAVE_ARCH_STRCAT
extern char * strcat(char *, const char *);
+char *__strcat(char *, const char *);
#endif
#ifndef __HAVE_ARCH_STRNCAT
extern char * strncat(char *, const char *, __kernel_size_t);
+char *__strncat(char *, const char *, __kernel_size_t);
#endif
#ifndef __HAVE_ARCH_STRLCAT
extern size_t strlcat(char *, const char *, __kernel_size_t);
+size_t __strlcat(char *, const char *, __kernel_size_t);
#endif
#ifndef __HAVE_ARCH_STRCMP
extern int strcmp(const char *,const char *);
+int __strcmp(const char *,const char *);
#endif
#ifndef __HAVE_ARCH_STRNCMP
extern int strncmp(const char *,const char *,__kernel_size_t);
+int __strncmp(const char *,const char *,__kernel_size_t);
#endif
#ifndef __HAVE_ARCH_STRCASECMP
extern int strcasecmp(const char *s1, const char *s2);
+int __strcasecmp(const char *s1, const char *s2);
#endif
#ifndef __HAVE_ARCH_STRNCASECMP
extern int strncasecmp(const char *s1, const char *s2, size_t n);
+int __strncasecmp(const char *s1, const char *s2, size_t n);
#endif
#ifndef __HAVE_ARCH_STRCHR
extern char * strchr(const char *,int);
+char *__strchr(const char *,int);
#endif
#ifndef __HAVE_ARCH_STRCHRNUL
extern char * strchrnul(const char *,int);
+char *__strchrnul(const char *,int);
#endif
#ifndef __HAVE_ARCH_STRNCHR
extern char * strnchr(const char *, size_t, int);
+char *__strnchr(const char *, size_t, int);
#endif
#ifndef __HAVE_ARCH_STRRCHR
extern char * strrchr(const char *,int);
+char *__strrchr(const char *,int);
#endif
extern char * __must_check skip_spaces(const char *);
+char * __must_check __skip_spaces(const char *);
extern char *strim(char *);
+char *__strim(char *);
static inline __must_check char *strstrip(char *str)
{
@@ -75,27 +138,35 @@ static inline __must_check char *strstrip(char *str)
#ifndef __HAVE_ARCH_STRSTR
extern char * strstr(const char *, const char *);
+char *__strstr(const char *, const char *);
#endif
#ifndef __HAVE_ARCH_STRNSTR
extern char * strnstr(const char *, const char *, size_t);
+char *__strnstr(const char *, const char *, size_t);
#endif
#ifndef __HAVE_ARCH_STRLEN
extern __kernel_size_t strlen(const char *);
+__kernel_size_t __strlen(const char *);
#endif
#ifndef __HAVE_ARCH_STRNLEN
extern __kernel_size_t strnlen(const char *,__kernel_size_t);
+__kernel_size_t __strnlen(const char *,__kernel_size_t);
#endif
#ifndef __HAVE_ARCH_STRPBRK
extern char * strpbrk(const char *,const char *);
+char *__strpbrk(const char *,const char *);
#endif
#ifndef __HAVE_ARCH_STRSEP
extern char * strsep(char **,const char *);
+char *__strsep(char **,const char *);
#endif
#ifndef __HAVE_ARCH_STRSPN
extern __kernel_size_t strspn(const char *,const char *);
+__kernel_size_t __strspn(const char *,const char *);
#endif
#ifndef __HAVE_ARCH_STRCSPN
extern __kernel_size_t strcspn(const char *,const char *);
+__kernel_size_t __strcspn(const char *,const char *);
#endif
#ifndef __HAVE_ARCH_MEMSET
@@ -104,14 +175,17 @@ extern void * memset(void *,int,__kernel_size_t);
#ifndef __HAVE_ARCH_MEMSET16
extern void *memset16(uint16_t *, uint16_t, __kernel_size_t);
+void *__memset16(uint16_t *, uint16_t, __kernel_size_t);
#endif
#ifndef __HAVE_ARCH_MEMSET32
extern void *memset32(uint32_t *, uint32_t, __kernel_size_t);
+void *__memset32(uint32_t *, uint32_t, __kernel_size_t);
#endif
#ifndef __HAVE_ARCH_MEMSET64
extern void *memset64(uint64_t *, uint64_t, __kernel_size_t);
+void *__memset64(uint64_t *, uint64_t, __kernel_size_t);
#endif
static inline void *memset_l(unsigned long *p, unsigned long v,
@@ -146,12 +220,15 @@ extern void * memmove(void *,const void *,__kernel_size_t);
#endif
#ifndef __HAVE_ARCH_MEMSCAN
extern void * memscan(void *,int,__kernel_size_t);
+void *__memscan(void *,int,__kernel_size_t);
#endif
#ifndef __HAVE_ARCH_MEMCMP
extern int memcmp(const void *,const void *,__kernel_size_t);
+int __memcmp(const void *,const void *,__kernel_size_t);
#endif
#ifndef __HAVE_ARCH_MEMCHR
extern void * memchr(const void *,int,__kernel_size_t);
+void *__memchr(const void *,int,__kernel_size_t);
#endif
#ifndef __HAVE_ARCH_MEMCPY_MCSAFE
static inline __must_check unsigned long memcpy_mcsafe(void *dst,
@@ -168,7 +245,9 @@ static inline void memcpy_flushcache(void *dst, const void *src, size_t cnt)
}
#endif
void *memchr_inv(const void *s, int c, size_t n);
+void *__memchr_inv(const void *s, int c, size_t n);
char *strreplace(char *s, char old, char new);
+char *__strreplace(char *s, char old, char new);
extern void kfree_const(const void *x);
diff --git a/lib/Makefile b/lib/Makefile
index 30b9b0bfbba9..19d0237f9b9c 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -18,6 +18,8 @@ KCOV_INSTRUMENT_list_debug.o := n
KCOV_INSTRUMENT_debugobjects.o := n
KCOV_INSTRUMENT_dynamic_debug.o := n
+KASAN_SANITIZE_string.o := n
+
lib-y := ctype.o string.o string_sysfs.o vsprintf.o cmdline.o \
rbtree.o radix-tree.o timerqueue.o xarray.o \
idr.o int_sqrt.o extable.o \
diff --git a/mm/kasan/string.c b/mm/kasan/string.c
index f23a740ff985..9a86b422e2e7 100644
--- a/mm/kasan/string.c
+++ b/mm/kasan/string.c
@@ -16,6 +16,38 @@ void *memset(void *addr, int c, size_t len)
return __memset(addr, c, len);
}
+#undef memset16
+void *memset16(uint16_t *s, uint16_t v, size_t count)
+{
+ check_memory_region((unsigned long)s, count << 1, true, _RET_IP_);
+
+ return __memset16(s, v, count);
+}
+
+#undef memset32
+void *memset32(uint32_t *s, uint32_t v, size_t count)
+{
+ check_memory_region((unsigned long)s, count << 2, true, _RET_IP_);
+
+ return __memset32(s, v, count);
+}
+
+#undef memset64
+void *memset64(uint64_t *s, uint64_t v, size_t count)
+{
+ check_memory_region((unsigned long)s, count << 3, true, _RET_IP_);
+
+ return __memset64(s, v, count);
+}
+
+#undef memzero_explicit
+void memzero_explicit(void *s, size_t count)
+{
+ check_memory_region((unsigned long)s, count, true, _RET_IP_);
+
+ return __memzero_explicit(s, count);
+}
+
#undef memmove
void *memmove(void *dest, const void *src, size_t len)
{
@@ -33,3 +65,305 @@ void *memcpy(void *dest, const void *src, size_t len)
return __memcpy(dest, src, len);
}
+
+#undef strcpy
+char *strcpy(char *dest, const char *src)
+{
+ size_t len = __strlen(src) + 1;
+
+ check_memory_region((unsigned long)src, len, false, _RET_IP_);
+ check_memory_region((unsigned long)dest, len, true, _RET_IP_);
+
+ return __strcpy(dest, src);
+}
+
+#undef strncpy
+char *strncpy(char *dest, const char *src, size_t count)
+{
+ size_t len = min(__strlen(src) + 1, count);
+
+ check_memory_region((unsigned long)src, len, false, _RET_IP_);
+ check_memory_region((unsigned long)dest, count, true, _RET_IP_);
+
+ return __strncpy(dest, src, count);
+}
+
+#undef strlcpy
+size_t strlcpy(char *dest, const char *src, size_t size)
+{
+ size_t len = __strlen(src) + 1;
+
+ check_memory_region((unsigned long)src, len, false, _RET_IP_);
+ check_memory_region((unsigned long)dest, min(len, size), true, _RET_IP_);
+
+ return __strlcpy(dest, src, size);
+}
+
+#undef strscpy
+ssize_t strscpy(char *dest, const char *src, size_t count)
+{
+ int len = min(__strlen(src) + 1, count);
+
+ check_memory_region((unsigned long)src, len, false, _RET_IP_);
+ check_memory_region((unsigned long)dest, len, true, _RET_IP_);
+
+ return __strscpy(dest, src, count);
+}
+
+#undef strcat
+char *strcat(char *dest, const char *src)
+{
+ size_t slen = __strlen(src) + 1;
+ size_t dlen = __strlen(dest);
+
+ check_memory_region((unsigned long)src, slen, false, _RET_IP_);
+ check_memory_region((unsigned long)dest, dlen, false, _RET_IP_);
+ check_memory_region((unsigned long)(dest + dlen), slen, true, _RET_IP_);
+
+ return __strcat(dest, src);
+}
+
+char *strncat(char *dest, const char *src, size_t count)
+{
+ size_t slen = min(__strlen(src) + 1, count);
+ size_t dlen = __strlen(dest);
+
+ check_memory_region((unsigned long)src, slen, false, _RET_IP_);
+ check_memory_region((unsigned long)dest, dlen, false, _RET_IP_);
+ check_memory_region((unsigned long)(dest + dlen), slen , true, _RET_IP_);
+
+ return __strncat(dest, src, count);
+}
+
+size_t strlcat(char *dest, const char *src, size_t count)
+{
+ size_t slen = min(__strlen(src) + 1, count);
+ size_t dlen = __strlen(dest);
+
+ check_memory_region((unsigned long)src, slen, false, _RET_IP_);
+ check_memory_region((unsigned long)dest, dlen, false, _RET_IP_);
+ check_memory_region((unsigned long)(dest + dlen), slen , true, _RET_IP_);
+
+ return __strlcat(dest, src, count);
+}
+
+int strcmp(const char *cs, const char *ct)
+{
+ size_t len = min(__strlen(cs) + 1, __strlen(ct) + 1);
+
+ check_memory_region((unsigned long)cs, len, false, _RET_IP_);
+ check_memory_region((unsigned long)ct, len, false, _RET_IP_);
+
+ return __strcmp(cs, ct);
+}
+
+int strncmp(const char *cs, const char *ct, size_t count)
+{
+ size_t len = min3(__strlen(cs) + 1, __strlen(ct) + 1, count);
+
+ check_memory_region((unsigned long)cs, len, false, _RET_IP_);
+ check_memory_region((unsigned long)ct, len, false, _RET_IP_);
+
+ return __strncmp(cs, ct, count);
+}
+
+int strcasecmp(const char *s1, const char *s2)
+{
+ size_t len = min(__strlen(s1) + 1, __strlen(s2) + 1);
+
+ check_memory_region((unsigned long)s1, len, false, _RET_IP_);
+ check_memory_region((unsigned long)s2, len, false, _RET_IP_);
+
+ return __strcasecmp(s1, s2);
+}
+
+int strncasecmp(const char *s1, const char *s2, size_t len)
+{
+ size_t sz = min3(__strlen(s1) + 1, __strlen(s2) + 1, len);
+
+ check_memory_region((unsigned long)s1, sz, false, _RET_IP_);
+ check_memory_region((unsigned long)s2, sz, false, _RET_IP_);
+
+ return __strncasecmp(s1, s2, len);
+}
+
+char *strchr(const char *s, int c)
+{
+ size_t len = __strlen(s) + 1;
+
+ check_memory_region((unsigned long)s, len, false, _RET_IP_);
+
+ return __strchr(s, c);
+}
+
+char *strchrnul(const char *s, int c)
+{
+ size_t len = __strlen(s) + 1;
+
+ check_memory_region((unsigned long)s, len, false, _RET_IP_);
+
+ return __strchrnul(s, c);
+}
+
+char *strrchr(const char *s, int c)
+{
+ size_t len = __strlen(s) + 1;
+
+ check_memory_region((unsigned long)s, len, false, _RET_IP_);
+
+ return __strrchr(s, c);
+}
+
+char *strnchr(const char *s, size_t count, int c)
+{
+ size_t len = __strlen(s) + 1;
+
+ check_memory_region((unsigned long)s, len, false, _RET_IP_);
+
+ return __strnchr(s, count, c);
+}
+
+char *skip_spaces(const char *str)
+{
+ size_t len = __strlen(str) + 1;
+
+ check_memory_region((unsigned long)str, len, false, _RET_IP_);
+
+ return __skip_spaces(str);
+}
+
+char *strim(char *s)
+{
+ size_t len = __strlen(s) + 1;
+
+ check_memory_region((unsigned long)s, len, false, _RET_IP_);
+
+ return __strim(s);
+}
+
+char *strstr(const char *s1, const char *s2)
+{
+ size_t l1 = __strlen(s1) + 1;
+ size_t l2 = __strlen(s2) + 1;
+
+ check_memory_region((unsigned long)s1, l1, false, _RET_IP_);
+ check_memory_region((unsigned long)s2, l2, false, _RET_IP_);
+
+ return __strstr(s1, s2);
+}
+
+char *strnstr(const char *s1, const char *s2, size_t len)
+{
+ size_t l1 = min(__strlen(s1) + 1, len);
+ size_t l2 = __strlen(s2) + 1;
+
+ check_memory_region((unsigned long)s1, l1, false, _RET_IP_);
+ check_memory_region((unsigned long)s2, l2, false, _RET_IP_);
+
+ return __strnstr(s1, s2, len);
+}
+
+size_t strlen(const char *s)
+{
+ size_t len = __strlen(s);
+
+ check_memory_region((unsigned long)s, len + 1, false, _RET_IP_);
+
+ return len;
+}
+
+size_t strnlen(const char *s, size_t count)
+{
+ size_t len = __strnlen(s, count);
+
+ check_memory_region((unsigned long)s, min(len + 1, count), false, _RET_IP_);
+
+ return len;
+}
+
+char *strpbrk(const char *cs, const char *ct)
+{
+ size_t ls = __strlen(cs) + 1;
+ size_t lt = __strlen(ct) + 1;
+
+ check_memory_region((unsigned long)cs, ls, false, _RET_IP_);
+ check_memory_region((unsigned long)ct, lt, false, _RET_IP_);
+
+ return __strpbrk(cs, ct);
+}
+char *strsep(char **s, const char *ct)
+{
+ char *cs = *s;
+
+ check_memory_region((unsigned long)s, sizeof(*s), true, _RET_IP_);
+
+ if (cs) {
+ int ls = __strlen(cs) + 1;
+ int lt = __strlen(ct) + 1;
+
+ check_memory_region((unsigned long)cs, ls, false, _RET_IP_);
+ check_memory_region((unsigned long)ct, lt, false, _RET_IP_);
+ }
+
+ return __strsep(s, ct);
+}
+
+size_t strspn(const char *s, const char *accept)
+{
+ size_t ls = __strlen(s) + 1;
+ size_t la = __strlen(accept) + 1;
+
+ check_memory_region((unsigned long)s, ls, false, _RET_IP_);
+ check_memory_region((unsigned long)accept, la, false, _RET_IP_);
+
+ return __strspn(s, accept);
+}
+
+size_t strcspn(const char *s, const char *reject)
+{
+ size_t ls = __strlen(s) + 1;
+ size_t lr = __strlen(reject) + 1;
+
+ check_memory_region((unsigned long)s, ls, false, _RET_IP_);
+ check_memory_region((unsigned long)reject, lr, false, _RET_IP_);
+
+ return __strcspn(s, reject);
+}
+
+void *memscan(void *addr, int c, size_t size)
+{
+ check_memory_region((unsigned long)addr, size, false, _RET_IP_);
+
+ return __memscan(addr, c, size);
+}
+
+int memcmp(const void *cs, const void *ct, size_t count)
+{
+ check_memory_region((unsigned long)cs, count, false, _RET_IP_);
+ check_memory_region((unsigned long)ct, count, false, _RET_IP_);
+
+ return __memcmp(cs, ct, count);
+}
+
+void *memchr(const void *s, int c, size_t n)
+{
+ check_memory_region((unsigned long)s, n, false, _RET_IP_);
+
+ return __memchr(s, c, n);
+}
+
+void *memchr_inv(const void *start, int c, size_t bytes)
+{
+ check_memory_region((unsigned long)start, bytes, false, _RET_IP_);
+
+ return __memchr_inv(start, c, bytes);
+}
+
+char *strreplace(char *s, char old, char new)
+{
+ size_t len = __strlen(s) + 1;
+
+ check_memory_region((unsigned long)s, len, true, _RET_IP_);
+
+ return __strreplace(s, old, new);
+}
--
2.13.3
^ permalink raw reply related
* [RFC PATCH v1 1/3] kasan: move memset/memmove/memcpy interceptors in a dedicated file
From: Christophe Leroy @ 2019-03-22 14:00 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
Nicholas Piggin, Aneesh Kumar K.V, Andrey Ryabinin,
Alexander Potapenko, Dmitry Vyukov, Daniel Axtens
Cc: linux-mm, linuxppc-dev, linux-kernel, kasan-dev
In preparation of the addition of interceptors for other string functions,
this patch moves memset/memmove/memcpy interceptions in string.c
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
mm/kasan/Makefile | 5 ++++-
mm/kasan/common.c | 26 --------------------------
mm/kasan/string.c | 35 +++++++++++++++++++++++++++++++++++
3 files changed, 39 insertions(+), 27 deletions(-)
create mode 100644 mm/kasan/string.c
diff --git a/mm/kasan/Makefile b/mm/kasan/Makefile
index 5d1065efbd47..85e91e301404 100644
--- a/mm/kasan/Makefile
+++ b/mm/kasan/Makefile
@@ -1,11 +1,13 @@
# SPDX-License-Identifier: GPL-2.0
KASAN_SANITIZE := n
UBSAN_SANITIZE_common.o := n
+UBSAN_SANITIZE_string.o := n
UBSAN_SANITIZE_generic.o := n
UBSAN_SANITIZE_tags.o := n
KCOV_INSTRUMENT := n
CFLAGS_REMOVE_common.o = -pg
+CFLAGS_REMOVE_string.o = -pg
CFLAGS_REMOVE_generic.o = -pg
CFLAGS_REMOVE_tags.o = -pg
@@ -13,9 +15,10 @@ CFLAGS_REMOVE_tags.o = -pg
# see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63533
CFLAGS_common.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector)
+CFLAGS_string.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector)
CFLAGS_generic.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector)
CFLAGS_tags.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector)
-obj-$(CONFIG_KASAN) := common.o init.o report.o
+obj-$(CONFIG_KASAN) := common.o init.o report.o string.o
obj-$(CONFIG_KASAN_GENERIC) += generic.o generic_report.o quarantine.o
obj-$(CONFIG_KASAN_SW_TAGS) += tags.o tags_report.o
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 80bbe62b16cd..3b94f484bf78 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -109,32 +109,6 @@ void kasan_check_write(const volatile void *p, unsigned int size)
}
EXPORT_SYMBOL(kasan_check_write);
-#undef memset
-void *memset(void *addr, int c, size_t len)
-{
- check_memory_region((unsigned long)addr, len, true, _RET_IP_);
-
- return __memset(addr, c, len);
-}
-
-#undef memmove
-void *memmove(void *dest, const void *src, size_t len)
-{
- check_memory_region((unsigned long)src, len, false, _RET_IP_);
- check_memory_region((unsigned long)dest, len, true, _RET_IP_);
-
- return __memmove(dest, src, len);
-}
-
-#undef memcpy
-void *memcpy(void *dest, const void *src, size_t len)
-{
- check_memory_region((unsigned long)src, len, false, _RET_IP_);
- check_memory_region((unsigned long)dest, len, true, _RET_IP_);
-
- return __memcpy(dest, src, len);
-}
-
/*
* Poisons the shadow memory for 'size' bytes starting from 'addr'.
* Memory addresses should be aligned to KASAN_SHADOW_SCALE_SIZE.
diff --git a/mm/kasan/string.c b/mm/kasan/string.c
new file mode 100644
index 000000000000..f23a740ff985
--- /dev/null
+++ b/mm/kasan/string.c
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file contains strings functions for KASAN
+ *
+ */
+
+#include <linux/string.h>
+
+#include "kasan.h"
+
+#undef memset
+void *memset(void *addr, int c, size_t len)
+{
+ check_memory_region((unsigned long)addr, len, true, _RET_IP_);
+
+ return __memset(addr, c, len);
+}
+
+#undef memmove
+void *memmove(void *dest, const void *src, size_t len)
+{
+ check_memory_region((unsigned long)src, len, false, _RET_IP_);
+ check_memory_region((unsigned long)dest, len, true, _RET_IP_);
+
+ return __memmove(dest, src, len);
+}
+
+#undef memcpy
+void *memcpy(void *dest, const void *src, size_t len)
+{
+ check_memory_region((unsigned long)src, len, false, _RET_IP_);
+ check_memory_region((unsigned long)dest, len, true, _RET_IP_);
+
+ return __memcpy(dest, src, len);
+}
--
2.13.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox