* Re: [PATCH v5 3/6] mm: make alloc_contig_range work at pageblock granularity
From: Zi Yan @ 2022-02-14 15:46 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Mel Gorman, David Hildenbrand, linuxppc-dev, linux-kernel,
virtualization, linux-mm, iommu, Mike Rapoport, Eric Ren,
Oscar Salvador, Robin Murphy, Vlastimil Babka, Marek Szyprowski
In-Reply-To: <20220214072601.GA17306@lst.de>
[-- Attachment #1: Type: text/plain, Size: 586 bytes --]
On 14 Feb 2022, at 2:26, Christoph Hellwig wrote:
>> +int
>> +isolate_single_pageblock(unsigned long boundary_pfn, gfp_t gfp_flags, int isolate_before_boundary);
>
> Please avoid the completely unreadably long line. i.e.
>
> int isolate_single_pageblock(unsigned long boundary_pfn, gfp_t gfp_flags,
> int isolate_before_boundary);
>
> Same in various other spots.
OK. Thanks for pointing it out. checkpatch.pl did not report any
warning about this. It seems that the column limit has been relaxed
to 100. Anyway, I will make it shorter.
--
Best Regards,
Yan, Zi
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 854 bytes --]
^ permalink raw reply
* Re: [PATCH v2 09/13] powerpc/ftrace: Implement CONFIG_DYNAMIC_FTRACE_WITH_ARGS
From: Naveen N. Rao @ 2022-02-14 15:25 UTC (permalink / raw)
To: Christophe Leroy, Jiri Kosina, Joe Lawrence, Josh Poimboeuf,
Miroslav Benes, Ingo Molnar, Petr Mladek, Steven Rostedt
Cc: live-patching@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
linux-kernel@vger.kernel.org
In-Reply-To: <5831f711a778fcd6eb51eb5898f1faae4378b35b.1640017960.git.christophe.leroy@csgroup.eu>
Christophe Leroy wrote:
> Implement CONFIG_DYNAMIC_FTRACE_WITH_ARGS. It accelerates the call
> of livepatching.
>
> Also note that powerpc being the last one to convert to
> CONFIG_DYNAMIC_FTRACE_WITH_ARGS, it will now be possible to remove
> klp_arch_set_pc() on all architectures.
>
> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
> ---
> arch/powerpc/Kconfig | 1 +
> arch/powerpc/include/asm/ftrace.h | 17 +++++++++++++++++
> arch/powerpc/include/asm/livepatch.h | 4 +---
> 3 files changed, 19 insertions(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index cdac2115eb00..e2b1792b2aae 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -210,6 +210,7 @@ config PPC
> select HAVE_DEBUG_KMEMLEAK
> select HAVE_DEBUG_STACKOVERFLOW
> select HAVE_DYNAMIC_FTRACE
> + select HAVE_DYNAMIC_FTRACE_WITH_ARGS if MPROFILE_KERNEL || PPC32
> select HAVE_DYNAMIC_FTRACE_WITH_REGS if MPROFILE_KERNEL || PPC32
> select HAVE_EBPF_JIT
> select HAVE_EFFICIENT_UNALIGNED_ACCESS if !(CPU_LITTLE_ENDIAN && POWER7_CPU)
> diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h
> index b3f6184f77ea..45c3d6f11daa 100644
> --- a/arch/powerpc/include/asm/ftrace.h
> +++ b/arch/powerpc/include/asm/ftrace.h
> @@ -22,6 +22,23 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr)
> struct dyn_arch_ftrace {
> struct module *mod;
> };
> +
> +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
> +struct ftrace_regs {
> + struct pt_regs regs;
> +};
> +
> +static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs *fregs)
> +{
> + return &fregs->regs;
> +}
I think this is wrong. We need to differentiate between ftrace_caller()
and ftrace_regs_caller() here, and only return pt_regs if coming in
through ftrace_regs_caller() (i.e., FL_SAVE_REGS is set).
> +
> +static __always_inline void ftrace_instruction_pointer_set(struct ftrace_regs *fregs,
> + unsigned long ip)
> +{
> + regs_set_return_ip(&fregs->regs, ip);
Should we use that helper here? regs_set_return_ip() also updates some
other state related to taking interrupts and I don't think it makes
sense for use with ftrace.
- Naveen
^ permalink raw reply
* Re: [PATCH v2 08/13] powerpc/ftrace: Prepare PPC64's ftrace_caller() for CONFIG_DYNAMIC_FTRACE_WITH_ARGS
From: Naveen N. Rao @ 2022-02-14 15:19 UTC (permalink / raw)
To: Christophe Leroy, Jiri Kosina, Joe Lawrence, Josh Poimboeuf,
Miroslav Benes, Ingo Molnar, Petr Mladek, Steven Rostedt
Cc: live-patching@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
linux-kernel@vger.kernel.org
In-Reply-To: <850817333cc76593699032e8e9a70d8c36e1af1e.1640017960.git.christophe.leroy@csgroup.eu>
Hi Christophe,
Thanks for your work enabling DYNAMIC_FTRACE_WITH_ARGS on powerpc. Sorry
for the late review on this series, but I have a few comments below.
Christophe Leroy wrote:
> In order to implement CONFIG_DYNAMIC_FTRACE_WITH_ARGS, change ftrace_caller()
> to handle LIVEPATCH the same way as frace_caller_regs().
>
> Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
> ---
> .../powerpc/kernel/trace/ftrace_64_mprofile.S | 25 ++++++++++++++-----
> 1 file changed, 19 insertions(+), 6 deletions(-)
I think we also need to save r1 into pt_regs so that the stack pointer
is available in the callbacks.
Other than that, a few minor nits below...
>
> diff --git a/arch/powerpc/kernel/trace/ftrace_64_mprofile.S b/arch/powerpc/kernel/trace/ftrace_64_mprofile.S
> index d636fc755f60..f6f787819273 100644
> --- a/arch/powerpc/kernel/trace/ftrace_64_mprofile.S
> +++ b/arch/powerpc/kernel/trace/ftrace_64_mprofile.S
> @@ -172,14 +172,19 @@ _GLOBAL(ftrace_caller)
> addi r3, r3, function_trace_op@toc@l
> ld r5, 0(r3)
>
> +#ifdef CONFIG_LIVEPATCH_64
> + SAVE_GPR(14, r1)
> + mr r14,r7 /* remember old NIP */
^ add a space
> +#endif
Please add a blank line here, to match the formatting for the rest of
this file.
> /* Calculate ip from nip-4 into r3 for call below */
> subi r3, r7, MCOUNT_INSN_SIZE
>
> /* Put the original return address in r4 as parent_ip */
> + std r0, _LINK(r1)
> mr r4, r0
>
> - /* Set pt_regs to NULL */
> - li r6, 0
> + /* Load &pt_regs in r6 for call below */
> + addi r6, r1 ,STACK_FRAME_OVERHEAD
^^ incorrect spacing
>
> /* ftrace_call(r3, r4, r5, r6) */
> .globl ftrace_call
> @@ -189,6 +194,10 @@ ftrace_call:
>
> ld r3, _NIP(r1)
> mtctr r3
Another blank line here.
> +#ifdef CONFIG_LIVEPATCH_64
> + cmpd r14, r3 /* has NIP been altered? */
> + REST_GPR(14, r1)
> +#endif
>
> /* Restore gprs */
> REST_GPRS(3, 10, r1)
> @@ -196,13 +205,17 @@ ftrace_call:
> /* Restore callee's TOC */
> ld r2, 24(r1)
>
> + /* Restore possibly modified LR */
> + ld r0, _LINK(r1)
> + mtlr r0
> +
> /* Pop our stack frame */
> addi r1, r1, SWITCH_FRAME_SIZE
>
> - /* Reload original LR */
> - ld r0, LRSAVE(r1)
> - mtlr r0
> -
> +#ifdef CONFIG_LIVEPATCH_64
> + /* Based on the cmpd above, if the NIP was altered handle livepatch */
> + bne- livepatch_handler
> +#endif
Here too.
> /* Handle function_graph or go back */
> b ftrace_caller_common
>
- Naveen
^ permalink raw reply
* Re: [PATCH v5 2/6] powerpc/kexec_file: Add KEXEC_SIG support.
From: Mimi Zohar @ 2022-02-14 15:14 UTC (permalink / raw)
To: Michal Suchanek, keyrings, linux-crypto, linux-integrity
Cc: Nayna, David Howells, Paul Mackerras, Alexander Gordeev,
linux-s390, Herbert Xu, Baoquan He, Christian Borntraeger,
James Morris, Lakshmi Ramasubramanian, Christian Borntraeger,
Serge E. Hallyn, Vasily Gorbik, Rob Herring, Heiko Carstens,
Dmitry Kasatkin, Hari Bathini, Daniel Axtens, Philipp Rudo,
Frank van der Linden, kexec, linux-kernel, Luis Chamberlain,
Sven Schnelle, linux-security-module, Jessica Yu, linuxppc-dev,
David S. Miller, Thiago Jung Bauermann, buendgen
In-Reply-To: <cff97dbe262919ff709a5ad2c4af6a702cc72a95.camel@linux.ibm.com>
Hi Michal,
On Sun, 2022-02-13 at 21:59 -0500, Mimi Zohar wrote:
>
> On Tue, 2022-01-11 at 12:37 +0100, Michal Suchanek wrote:
> > diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> > index dea74d7717c0..1cde9b6c5987 100644
> > --- a/arch/powerpc/Kconfig
> > +++ b/arch/powerpc/Kconfig
> > @@ -560,6 +560,22 @@ config KEXEC_FILE
> > config ARCH_HAS_KEXEC_PURGATORY
> > def_bool KEXEC_FILE
> >
> > +config KEXEC_SIG
> > + bool "Verify kernel signature during kexec_file_load() syscall"
> > + depends on KEXEC_FILE && MODULE_SIG_FORMAT
> > + help
> > + This option makes kernel signature verification mandatory for
> > + the kexec_file_load() syscall.
>
> When KEXEC_SIG is enabled on other architectures, IMA does not define a
> kexec 'appraise' policy rule. Refer to the policy rules in
> security/ima/ima_efi.c. Similarly the kexec 'appraise' policy rule in
> arch/powerpc/kernel/ima_policy.c should not be defined.
The discussion shouldn't only be about IMA vs. KEXEC_SIG kernel image
signature verification. Let's try and reframe the problem a bit.
1. Unify and simply the existing kexec signature verification so
verifying the KEXEC kernel image signature works irrespective of
signature type - PE, appended signature.
solution: enable KEXEC_SIG (This patch set, with the above powerpc IMA
policy changes.)
2. Measure and include the kexec kernel image in a log for attestation,
if desired.
solution: enable IMA_ARCH_POLICY
- Powerpc: requires trusted boot to be enabled.
- EFI: requires secure boot to be enabled. The IMA efi policy
doesn't differentiate between secure and trusted boot.
3. Carry the kexec kernel image measurement across kexec, if desired
and supported on the architecture.
solution: enable IMA_KEXEC
Comparison:
- Are there any differences between IMA vs. KEXEC_SIG measuring the
kexec kernel image?
One of the main differences is "what" is included in the measurement
list differs. In both cases, the 'd-ng' field of the IMA measurement
list template (e.g. ima-ng, ima-sig, ima-modsig) is the full file hash
including the appended signature. With IMA and the 'ima-modsig'
template, an additional hash without the appended signature is defined,
as well as including the appended signature in the 'sig' field.
Including the file hash and appended signature in the measurement list
allows an attestation server, for example, to verify the appended
signature without having to know the file hash without the signature.
Other differences are already included in the Kconfig KEXEC_SIG "Notes"
section.
--
thanks,
Mimi
^ permalink raw reply
* Re: No Linux logs when doing `ppc64_cpu --smt=off/8`
From: Michal Suchánek @ 2022-02-14 13:56 UTC (permalink / raw)
To: Paul Menzel; +Cc: linuxppc-dev
In-Reply-To: <e6701fa9-a51b-3706-5aa5-bb6c7ae76cf0@molgen.mpg.de>
On Mon, Feb 14, 2022 at 01:33:24PM +0100, Paul Menzel wrote:
> Dear Michal,
>
>
> Thank you for your reply.
>
> Am 14.02.22 um 10:43 schrieb Michal Suchánek:
>
> > On Mon, Feb 14, 2022 at 07:08:07AM +0100, Paul Menzel wrote:
> > > Dear PPC folks,
> > >
> > >
> > > On the POWER8 server IBM S822LC running `ppc64_cpu --smt=off` or `ppc64_cpu
> > > --smt=8`, Linux 5.17-rc4 does not log anything. I would have expected a
> > > message about the change in number of processing units.
> >
> > IIRC it was considered too noisy for systems with many CPUs and the
> > message was dropped. You can always check the resulting state with
> > ppc64_cpu or examining sysfs.
>
> Yes, simple `nproc` suffice, but I was more thinking about, that the Linux
> log is often used for debugging and the changes of amount of processing
> units might be good to have. `ppc64_cpu --smt=off` or `=8` seems to block
> for quite some time, and each thread/processing unit seems to powered
> down/on sequentially, so it takes quite some time and it blocks. So 140
> messages would indeed be quite noise. No idea how `ppc64_cpu` works, and if
> it could log a message at the beginning and end.
Yes, it enables/disables threads one by one. AFAICT the kernel cannot know that
ppc64_cpu will enable/disable more threads later, it can either log each
or none. Rate limiting would not show the whole picture so it's not
great solution either.
Thanks
Michal
^ permalink raw reply
* Re: No Linux logs when doing `ppc64_cpu --smt=off/8`
From: Paul Menzel @ 2022-02-14 12:33 UTC (permalink / raw)
To: Michal Suchánek; +Cc: linuxppc-dev
In-Reply-To: <20220214094327.GL3113@kunlun.suse.cz>
Dear Michal,
Thank you for your reply.
Am 14.02.22 um 10:43 schrieb Michal Suchánek:
> On Mon, Feb 14, 2022 at 07:08:07AM +0100, Paul Menzel wrote:
>> Dear PPC folks,
>>
>>
>> On the POWER8 server IBM S822LC running `ppc64_cpu --smt=off` or `ppc64_cpu
>> --smt=8`, Linux 5.17-rc4 does not log anything. I would have expected a
>> message about the change in number of processing units.
>
> IIRC it was considered too noisy for systems with many CPUs and the
> message was dropped. You can always check the resulting state with
> ppc64_cpu or examining sysfs.
Yes, simple `nproc` suffice, but I was more thinking about, that the
Linux log is often used for debugging and the changes of amount of
processing units might be good to have. `ppc64_cpu --smt=off` or `=8`
seems to block for quite some time, and each thread/processing unit
seems to powered down/on sequentially, so it takes quite some time and
it blocks. So 140 messages would indeed be quite noise. No idea how
`ppc64_cpu` works, and if it could log a message at the beginning and end.
Kind regards,
Paul
^ permalink raw reply
* Re: [RFC PATCH 0/3] powerpc64/bpf: Add support for BPF Trampolines
From: Naveen N. Rao @ 2022-02-14 10:47 UTC (permalink / raw)
To: Alexei Starovoitov, Christophe Leroy, Daniel Borkmann,
Michael Ellerman, Steven Rostedt
Cc: Yauheni Kaliuta, Jordan Niethe, linuxppc-dev@lists.ozlabs.org,
bpf@vger.kernel.org, Jiri Olsa, Hari Bathini
In-Reply-To: <918b842d-f674-bbdb-c772-b43a00f1b155@csgroup.eu>
Christophe Leroy wrote:
>
>
> Le 07/02/2022 à 08:07, Naveen N. Rao a écrit :
>> This is an early RFC series that adds support for BPF Trampolines on
>> powerpc64. Some of the selftests are passing for me, but this needs more
>> testing and I've likely missed a few things as well. A review of the
>> patches and feedback about the overall approach will be great.
>>
>> This series depends on some of the other BPF JIT fixes and enhancements
>> posted previously, as well as on ftrace direct enablement on powerpc
>> which has also been posted in the past.
>
> Is there any reason to limit this to powerpc64 ?
I have limited this to elf v2, and we won't be able to get this working
on elf v1, since we don't have DYNAMIC_FTRACE_WITH_REGS supported there.
We should be able to get this working on ppc32 though.
- Naveen
^ permalink raw reply
* Re: [PATCH v5 1/6] mm: page_isolation: move has_unmovable_pages() to mm/page_isolation.c
From: Mike Rapoport @ 2022-02-14 10:44 UTC (permalink / raw)
To: Zi Yan
Cc: Mel Gorman, David Hildenbrand, linuxppc-dev, linux-kernel,
virtualization, linux-mm, iommu, Eric Ren, Oscar Salvador,
Robin Murphy, Christoph Hellwig, Vlastimil Babka,
Marek Szyprowski
In-Reply-To: <20220211164135.1803616-2-zi.yan@sent.com>
On Fri, Feb 11, 2022 at 11:41:30AM -0500, Zi Yan wrote:
> From: Zi Yan <ziy@nvidia.com>
>
> has_unmovable_pages() is only used in mm/page_isolation.c. Move it from
> mm/page_alloc.c and make it static.
>
> Signed-off-by: Zi Yan <ziy@nvidia.com>
> Reviewed-by: Oscar Salvador <osalvador@suse.de>
Reviewed-by: Mike Rapoport <rppt@linux.ibm.com>
> ---
> include/linux/page-isolation.h | 2 -
> mm/page_alloc.c | 119 ---------------------------------
> mm/page_isolation.c | 119 +++++++++++++++++++++++++++++++++
> 3 files changed, 119 insertions(+), 121 deletions(-)
>
> diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h
> index 572458016331..e14eddf6741a 100644
> --- a/include/linux/page-isolation.h
> +++ b/include/linux/page-isolation.h
> @@ -33,8 +33,6 @@ static inline bool is_migrate_isolate(int migratetype)
> #define MEMORY_OFFLINE 0x1
> #define REPORT_FAILURE 0x2
>
> -struct page *has_unmovable_pages(struct zone *zone, struct page *page,
> - int migratetype, int flags);
> void set_pageblock_migratetype(struct page *page, int migratetype);
> int move_freepages_block(struct zone *zone, struct page *page,
> int migratetype, int *num_movable);
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index cface1d38093..e2c6a67fc386 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -8962,125 +8962,6 @@ void *__init alloc_large_system_hash(const char *tablename,
> return table;
> }
>
> -/*
> - * This function checks whether pageblock includes unmovable pages or not.
> - *
> - * PageLRU check without isolation or lru_lock could race so that
> - * MIGRATE_MOVABLE block might include unmovable pages. And __PageMovable
> - * check without lock_page also may miss some movable non-lru pages at
> - * race condition. So you can't expect this function should be exact.
> - *
> - * Returns a page without holding a reference. If the caller wants to
> - * dereference that page (e.g., dumping), it has to make sure that it
> - * cannot get removed (e.g., via memory unplug) concurrently.
> - *
> - */
> -struct page *has_unmovable_pages(struct zone *zone, struct page *page,
> - int migratetype, int flags)
> -{
> - unsigned long iter = 0;
> - unsigned long pfn = page_to_pfn(page);
> - unsigned long offset = pfn % pageblock_nr_pages;
> -
> - if (is_migrate_cma_page(page)) {
> - /*
> - * CMA allocations (alloc_contig_range) really need to mark
> - * isolate CMA pageblocks even when they are not movable in fact
> - * so consider them movable here.
> - */
> - if (is_migrate_cma(migratetype))
> - return NULL;
> -
> - return page;
> - }
> -
> - for (; iter < pageblock_nr_pages - offset; iter++) {
> - page = pfn_to_page(pfn + iter);
> -
> - /*
> - * Both, bootmem allocations and memory holes are marked
> - * PG_reserved and are unmovable. We can even have unmovable
> - * allocations inside ZONE_MOVABLE, for example when
> - * specifying "movablecore".
> - */
> - if (PageReserved(page))
> - return page;
> -
> - /*
> - * If the zone is movable and we have ruled out all reserved
> - * pages then it should be reasonably safe to assume the rest
> - * is movable.
> - */
> - if (zone_idx(zone) == ZONE_MOVABLE)
> - continue;
> -
> - /*
> - * Hugepages are not in LRU lists, but they're movable.
> - * THPs are on the LRU, but need to be counted as #small pages.
> - * We need not scan over tail pages because we don't
> - * handle each tail page individually in migration.
> - */
> - if (PageHuge(page) || PageTransCompound(page)) {
> - struct page *head = compound_head(page);
> - unsigned int skip_pages;
> -
> - if (PageHuge(page)) {
> - if (!hugepage_migration_supported(page_hstate(head)))
> - return page;
> - } else if (!PageLRU(head) && !__PageMovable(head)) {
> - return page;
> - }
> -
> - skip_pages = compound_nr(head) - (page - head);
> - iter += skip_pages - 1;
> - continue;
> - }
> -
> - /*
> - * We can't use page_count without pin a page
> - * because another CPU can free compound page.
> - * This check already skips compound tails of THP
> - * because their page->_refcount is zero at all time.
> - */
> - if (!page_ref_count(page)) {
> - if (PageBuddy(page))
> - iter += (1 << buddy_order(page)) - 1;
> - continue;
> - }
> -
> - /*
> - * The HWPoisoned page may be not in buddy system, and
> - * page_count() is not 0.
> - */
> - if ((flags & MEMORY_OFFLINE) && PageHWPoison(page))
> - continue;
> -
> - /*
> - * We treat all PageOffline() pages as movable when offlining
> - * to give drivers a chance to decrement their reference count
> - * in MEM_GOING_OFFLINE in order to indicate that these pages
> - * can be offlined as there are no direct references anymore.
> - * For actually unmovable PageOffline() where the driver does
> - * not support this, we will fail later when trying to actually
> - * move these pages that still have a reference count > 0.
> - * (false negatives in this function only)
> - */
> - if ((flags & MEMORY_OFFLINE) && PageOffline(page))
> - continue;
> -
> - if (__PageMovable(page) || PageLRU(page))
> - continue;
> -
> - /*
> - * If there are RECLAIMABLE pages, we need to check
> - * it. But now, memory offline itself doesn't call
> - * shrink_node_slabs() and it still to be fixed.
> - */
> - return page;
> - }
> - return NULL;
> -}
> -
> #ifdef CONFIG_CONTIG_ALLOC
> static unsigned long pfn_max_align_down(unsigned long pfn)
> {
> diff --git a/mm/page_isolation.c b/mm/page_isolation.c
> index f67c4c70f17f..b34f1310aeaa 100644
> --- a/mm/page_isolation.c
> +++ b/mm/page_isolation.c
> @@ -15,6 +15,125 @@
> #define CREATE_TRACE_POINTS
> #include <trace/events/page_isolation.h>
>
> +/*
> + * This function checks whether pageblock includes unmovable pages or not.
> + *
> + * PageLRU check without isolation or lru_lock could race so that
> + * MIGRATE_MOVABLE block might include unmovable pages. And __PageMovable
> + * check without lock_page also may miss some movable non-lru pages at
> + * race condition. So you can't expect this function should be exact.
> + *
> + * Returns a page without holding a reference. If the caller wants to
> + * dereference that page (e.g., dumping), it has to make sure that it
> + * cannot get removed (e.g., via memory unplug) concurrently.
> + *
> + */
> +static struct page *has_unmovable_pages(struct zone *zone, struct page *page,
> + int migratetype, int flags)
> +{
> + unsigned long iter = 0;
> + unsigned long pfn = page_to_pfn(page);
> + unsigned long offset = pfn % pageblock_nr_pages;
> +
> + if (is_migrate_cma_page(page)) {
> + /*
> + * CMA allocations (alloc_contig_range) really need to mark
> + * isolate CMA pageblocks even when they are not movable in fact
> + * so consider them movable here.
> + */
> + if (is_migrate_cma(migratetype))
> + return NULL;
> +
> + return page;
> + }
> +
> + for (; iter < pageblock_nr_pages - offset; iter++) {
> + page = pfn_to_page(pfn + iter);
> +
> + /*
> + * Both, bootmem allocations and memory holes are marked
> + * PG_reserved and are unmovable. We can even have unmovable
> + * allocations inside ZONE_MOVABLE, for example when
> + * specifying "movablecore".
> + */
> + if (PageReserved(page))
> + return page;
> +
> + /*
> + * If the zone is movable and we have ruled out all reserved
> + * pages then it should be reasonably safe to assume the rest
> + * is movable.
> + */
> + if (zone_idx(zone) == ZONE_MOVABLE)
> + continue;
> +
> + /*
> + * Hugepages are not in LRU lists, but they're movable.
> + * THPs are on the LRU, but need to be counted as #small pages.
> + * We need not scan over tail pages because we don't
> + * handle each tail page individually in migration.
> + */
> + if (PageHuge(page) || PageTransCompound(page)) {
> + struct page *head = compound_head(page);
> + unsigned int skip_pages;
> +
> + if (PageHuge(page)) {
> + if (!hugepage_migration_supported(page_hstate(head)))
> + return page;
> + } else if (!PageLRU(head) && !__PageMovable(head)) {
> + return page;
> + }
> +
> + skip_pages = compound_nr(head) - (page - head);
> + iter += skip_pages - 1;
> + continue;
> + }
> +
> + /*
> + * We can't use page_count without pin a page
> + * because another CPU can free compound page.
> + * This check already skips compound tails of THP
> + * because their page->_refcount is zero at all time.
> + */
> + if (!page_ref_count(page)) {
> + if (PageBuddy(page))
> + iter += (1 << buddy_order(page)) - 1;
> + continue;
> + }
> +
> + /*
> + * The HWPoisoned page may be not in buddy system, and
> + * page_count() is not 0.
> + */
> + if ((flags & MEMORY_OFFLINE) && PageHWPoison(page))
> + continue;
> +
> + /*
> + * We treat all PageOffline() pages as movable when offlining
> + * to give drivers a chance to decrement their reference count
> + * in MEM_GOING_OFFLINE in order to indicate that these pages
> + * can be offlined as there are no direct references anymore.
> + * For actually unmovable PageOffline() where the driver does
> + * not support this, we will fail later when trying to actually
> + * move these pages that still have a reference count > 0.
> + * (false negatives in this function only)
> + */
> + if ((flags & MEMORY_OFFLINE) && PageOffline(page))
> + continue;
> +
> + if (__PageMovable(page) || PageLRU(page))
> + continue;
> +
> + /*
> + * If there are RECLAIMABLE pages, we need to check
> + * it. But now, memory offline itself doesn't call
> + * shrink_node_slabs() and it still to be fixed.
> + */
> + return page;
> + }
> + return NULL;
> +}
> +
> static int set_migratetype_isolate(struct page *page, int migratetype, int isol_flags)
> {
> struct zone *zone = page_zone(page);
> --
> 2.34.1
>
--
Sincerely yours,
Mike.
^ permalink raw reply
* [PATCH powerpc/next 17/17] powerpc/bpf: Simplify bpf_to_ppc() and adopt it for powerpc64
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
Convert bpf_to_ppc() to a macro to help simplify its usage since
codegen_context is available in all places it is used. Adopt it also for
powerpc64 for uniformity and get rid of the global b2p structure.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit.h | 11 ++--
arch/powerpc/net/bpf_jit_comp.c | 8 +--
arch/powerpc/net/bpf_jit_comp32.c | 90 ++++++++++++++----------------
arch/powerpc/net/bpf_jit_comp64.c | 93 ++++++++++++++++---------------
4 files changed, 98 insertions(+), 104 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index d9bdc9df2e48ed..c16271a456b343 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -122,12 +122,6 @@
#define SEEN_VREG_MASK 0x1ff80000 /* Volatile registers r3-r12 */
#define SEEN_NVREG_MASK 0x0003ffff /* Non volatile registers r14-r31 */
-#ifdef CONFIG_PPC64
-extern const int b2p[MAX_BPF_JIT_REG + 2];
-#else
-extern const int b2p[MAX_BPF_JIT_REG + 1];
-#endif
-
struct codegen_context {
/*
* This is used to track register usage as well
@@ -141,11 +135,13 @@ struct codegen_context {
unsigned int seen;
unsigned int idx;
unsigned int stack_size;
- int b2p[ARRAY_SIZE(b2p)];
+ int b2p[MAX_BPF_JIT_REG + 2];
unsigned int exentry_idx;
unsigned int alt_exit_addr;
};
+#define bpf_to_ppc(r) (ctx->b2p[r])
+
#ifdef CONFIG_PPC32
#define BPF_FIXUP_LEN 3 /* Three instructions => 12 bytes */
#else
@@ -173,6 +169,7 @@ static inline void bpf_clear_seen_register(struct codegen_context *ctx, int i)
ctx->seen &= ~(1 << (31 - i));
}
+void bpf_jit_init_reg_mapping(struct codegen_context *ctx);
int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func);
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
u32 *addrs, int pass);
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 635f7448ff7952..fc160d33c83960 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -72,13 +72,13 @@ static int bpf_jit_fixup_addresses(struct bpf_prog *fp, u32 *image,
tmp_idx = ctx->idx;
ctx->idx = addrs[i] / 4;
#ifdef CONFIG_PPC32
- PPC_LI32(ctx->b2p[insn[i].dst_reg] - 1, (u32)insn[i + 1].imm);
- PPC_LI32(ctx->b2p[insn[i].dst_reg], (u32)insn[i].imm);
+ PPC_LI32(bpf_to_ppc(insn[i].dst_reg) - 1, (u32)insn[i + 1].imm);
+ PPC_LI32(bpf_to_ppc(insn[i].dst_reg), (u32)insn[i].imm);
for (j = ctx->idx - addrs[i] / 4; j < 4; j++)
EMIT(PPC_RAW_NOP());
#else
func_addr = ((u64)(u32)insn[i].imm) | (((u64)(u32)insn[i + 1].imm) << 32);
- PPC_LI64(b2p[insn[i].dst_reg], func_addr);
+ PPC_LI64(bpf_to_ppc(insn[i].dst_reg), func_addr);
/* overwrite rest with nops */
for (j = ctx->idx - addrs[i] / 4; j < 5; j++)
EMIT(PPC_RAW_NOP());
@@ -179,7 +179,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
}
memset(&cgctx, 0, sizeof(struct codegen_context));
- memcpy(cgctx.b2p, b2p, sizeof(cgctx.b2p));
+ bpf_jit_init_reg_mapping(&cgctx);
/* Make sure that the stack is quadword aligned. */
cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index fe4e0eca017ede..7dc716cd64bcbc 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -33,42 +33,38 @@
/* stack frame, ensure this is quadword aligned */
#define BPF_PPC_STACKFRAME(ctx) (STACK_FRAME_MIN_SIZE + BPF_PPC_STACK_SAVE + (ctx)->stack_size)
-/* BPF register usage */
-#define TMP_REG (MAX_BPF_JIT_REG + 0)
-
#define PPC_EX32(r, i) EMIT(PPC_RAW_LI((r), (i) < 0 ? -1 : 0))
+/* PPC NVR range -- update this if we ever use NVRs below r17 */
+#define BPF_PPC_NVR_MIN _R17
+#define BPF_PPC_TC _R16
+
+/* BPF register usage */
+#define TMP_REG (MAX_BPF_JIT_REG + 0)
+
/* BPF to ppc register mappings */
-const int b2p[MAX_BPF_JIT_REG + 1] = {
+void bpf_jit_init_reg_mapping(struct codegen_context *ctx)
+{
/* function return value */
- [BPF_REG_0] = _R12,
+ ctx->b2p[BPF_REG_0] = _R12;
/* function arguments */
- [BPF_REG_1] = _R4,
- [BPF_REG_2] = _R6,
- [BPF_REG_3] = _R8,
- [BPF_REG_4] = _R10,
- [BPF_REG_5] = _R22,
+ ctx->b2p[BPF_REG_1] = _R4;
+ ctx->b2p[BPF_REG_2] = _R6;
+ ctx->b2p[BPF_REG_3] = _R8;
+ ctx->b2p[BPF_REG_4] = _R10;
+ ctx->b2p[BPF_REG_5] = _R22;
/* non volatile registers */
- [BPF_REG_6] = _R24,
- [BPF_REG_7] = _R26,
- [BPF_REG_8] = _R28,
- [BPF_REG_9] = _R30,
+ ctx->b2p[BPF_REG_6] = _R24;
+ ctx->b2p[BPF_REG_7] = _R26;
+ ctx->b2p[BPF_REG_8] = _R28;
+ ctx->b2p[BPF_REG_9] = _R30;
/* frame pointer aka BPF_REG_10 */
- [BPF_REG_FP] = _R18,
+ ctx->b2p[BPF_REG_FP] = _R18;
/* eBPF jit internal registers */
- [BPF_REG_AX] = _R20,
- [TMP_REG] = _R31, /* 32 bits */
-};
-
-static int bpf_to_ppc(struct codegen_context *ctx, int reg)
-{
- return ctx->b2p[reg];
+ ctx->b2p[BPF_REG_AX] = _R20;
+ ctx->b2p[TMP_REG] = _R31; /* 32 bits */
}
-/* PPC NVR range -- update this if we ever use NVRs below r17 */
-#define BPF_PPC_NVR_MIN _R17
-#define BPF_PPC_TC _R16
-
static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
{
if ((reg >= BPF_PPC_NVR_MIN && reg < 32) || reg == BPF_PPC_TC)
@@ -110,8 +106,8 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
int i;
/* First arg comes in as a 32 bits pointer. */
- EMIT(PPC_RAW_MR(bpf_to_ppc(ctx, BPF_REG_1), _R3));
- EMIT(PPC_RAW_LI(bpf_to_ppc(ctx, BPF_REG_1) - 1, 0));
+ EMIT(PPC_RAW_MR(bpf_to_ppc(BPF_REG_1), _R3));
+ EMIT(PPC_RAW_LI(bpf_to_ppc(BPF_REG_1) - 1, 0));
EMIT(PPC_RAW_STWU(_R1, _R1, -BPF_PPC_STACKFRAME(ctx)));
/*
@@ -120,7 +116,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
* invoked through a tail call.
*/
if (ctx->seen & SEEN_TAILCALL)
- EMIT(PPC_RAW_STW(bpf_to_ppc(ctx, BPF_REG_1) - 1, _R1,
+ EMIT(PPC_RAW_STW(bpf_to_ppc(BPF_REG_1) - 1, _R1,
bpf_jit_stack_offsetof(ctx, BPF_PPC_TC)));
else
EMIT(PPC_RAW_NOP());
@@ -142,15 +138,15 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
EMIT(PPC_RAW_STW(i, _R1, bpf_jit_stack_offsetof(ctx, i)));
/* If needed retrieve arguments 9 and 10, ie 5th 64 bits arg.*/
- if (bpf_is_seen_register(ctx, bpf_to_ppc(ctx, BPF_REG_5))) {
- EMIT(PPC_RAW_LWZ(bpf_to_ppc(ctx, BPF_REG_5) - 1, _R1, BPF_PPC_STACKFRAME(ctx)) + 8);
- EMIT(PPC_RAW_LWZ(bpf_to_ppc(ctx, BPF_REG_5), _R1, BPF_PPC_STACKFRAME(ctx)) + 12);
+ if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_5))) {
+ EMIT(PPC_RAW_LWZ(bpf_to_ppc(BPF_REG_5) - 1, _R1, BPF_PPC_STACKFRAME(ctx)) + 8);
+ EMIT(PPC_RAW_LWZ(bpf_to_ppc(BPF_REG_5), _R1, BPF_PPC_STACKFRAME(ctx)) + 12);
}
/* Setup frame pointer to point to the bpf stack area */
- if (bpf_is_seen_register(ctx, bpf_to_ppc(ctx, BPF_REG_FP))) {
- EMIT(PPC_RAW_LI(bpf_to_ppc(ctx, BPF_REG_FP) - 1, 0));
- EMIT(PPC_RAW_ADDI(bpf_to_ppc(ctx, BPF_REG_FP), _R1,
+ if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP))) {
+ EMIT(PPC_RAW_LI(bpf_to_ppc(BPF_REG_FP) - 1, 0));
+ EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
STACK_FRAME_MIN_SIZE + ctx->stack_size));
}
@@ -170,7 +166,7 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
{
- EMIT(PPC_RAW_MR(_R3, bpf_to_ppc(ctx, BPF_REG_0)));
+ EMIT(PPC_RAW_MR(_R3, bpf_to_ppc(BPF_REG_0)));
bpf_jit_emit_common_epilogue(image, ctx);
@@ -215,8 +211,8 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
* r5-r6/BPF_REG_2 - pointer to bpf_array
* r7-r8/BPF_REG_3 - index in bpf_array
*/
- int b2p_bpf_array = bpf_to_ppc(ctx, BPF_REG_2);
- int b2p_index = bpf_to_ppc(ctx, BPF_REG_3);
+ int b2p_bpf_array = bpf_to_ppc(BPF_REG_2);
+ int b2p_index = bpf_to_ppc(BPF_REG_3);
/*
* if (index >= array->map.max_entries)
@@ -262,7 +258,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
EMIT(PPC_RAW_MTCTR(_R3));
- EMIT(PPC_RAW_MR(_R3, bpf_to_ppc(ctx, BPF_REG_1)));
+ EMIT(PPC_RAW_MR(_R3, bpf_to_ppc(BPF_REG_1)));
/* tear restore NVRs, ... */
bpf_jit_emit_common_epilogue(image, ctx);
@@ -286,11 +282,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
for (i = 0; i < flen; i++) {
u32 code = insn[i].code;
- u32 dst_reg = bpf_to_ppc(ctx, insn[i].dst_reg);
+ u32 dst_reg = bpf_to_ppc(insn[i].dst_reg);
u32 dst_reg_h = dst_reg - 1;
- u32 src_reg = bpf_to_ppc(ctx, insn[i].src_reg);
+ u32 src_reg = bpf_to_ppc(insn[i].src_reg);
u32 src_reg_h = src_reg - 1;
- u32 tmp_reg = bpf_to_ppc(ctx, TMP_REG);
+ u32 tmp_reg = bpf_to_ppc(TMP_REG);
u32 size = BPF_SIZE(code);
s16 off = insn[i].off;
s32 imm = insn[i].imm;
@@ -952,17 +948,17 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
if (ret < 0)
return ret;
- if (bpf_is_seen_register(ctx, bpf_to_ppc(ctx, BPF_REG_5))) {
- EMIT(PPC_RAW_STW(bpf_to_ppc(ctx, BPF_REG_5) - 1, _R1, 8));
- EMIT(PPC_RAW_STW(bpf_to_ppc(ctx, BPF_REG_5), _R1, 12));
+ if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_5))) {
+ EMIT(PPC_RAW_STW(bpf_to_ppc(BPF_REG_5) - 1, _R1, 8));
+ EMIT(PPC_RAW_STW(bpf_to_ppc(BPF_REG_5), _R1, 12));
}
ret = bpf_jit_emit_func_call_rel(image, ctx, func_addr);
if (ret)
return ret;
- EMIT(PPC_RAW_MR(bpf_to_ppc(ctx, BPF_REG_0) - 1, _R3));
- EMIT(PPC_RAW_MR(bpf_to_ppc(ctx, BPF_REG_0), _R4));
+ EMIT(PPC_RAW_MR(bpf_to_ppc(BPF_REG_0) - 1, _R3));
+ EMIT(PPC_RAW_MR(bpf_to_ppc(BPF_REG_0), _R4));
break;
/*
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index b4de0c35c8a4ab..585f257da04587 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -46,27 +46,28 @@
#define TMP_REG_2 (MAX_BPF_JIT_REG + 1)
/* BPF to ppc register mappings */
-const int b2p[MAX_BPF_JIT_REG + 2] = {
+void bpf_jit_init_reg_mapping(struct codegen_context *ctx)
+{
/* function return value */
- [BPF_REG_0] = _R8,
+ ctx->b2p[BPF_REG_0] = _R8;
/* function arguments */
- [BPF_REG_1] = _R3,
- [BPF_REG_2] = _R4,
- [BPF_REG_3] = _R5,
- [BPF_REG_4] = _R6,
- [BPF_REG_5] = _R7,
+ ctx->b2p[BPF_REG_1] = _R3;
+ ctx->b2p[BPF_REG_2] = _R4;
+ ctx->b2p[BPF_REG_3] = _R5;
+ ctx->b2p[BPF_REG_4] = _R6;
+ ctx->b2p[BPF_REG_5] = _R7;
/* non volatile registers */
- [BPF_REG_6] = _R27,
- [BPF_REG_7] = _R28,
- [BPF_REG_8] = _R29,
- [BPF_REG_9] = _R30,
+ ctx->b2p[BPF_REG_6] = _R27;
+ ctx->b2p[BPF_REG_7] = _R28;
+ ctx->b2p[BPF_REG_8] = _R29;
+ ctx->b2p[BPF_REG_9] = _R30;
/* frame pointer aka BPF_REG_10 */
- [BPF_REG_FP] = _R31,
+ ctx->b2p[BPF_REG_FP] = _R31;
/* eBPF jit internal registers */
- [BPF_REG_AX] = _R12,
- [TMP_REG_1] = _R9,
- [TMP_REG_2] = _R10
-};
+ ctx->b2p[BPF_REG_AX] = _R12;
+ ctx->b2p[TMP_REG_1] = _R9;
+ ctx->b2p[TMP_REG_2] = _R10;
+}
/* PPC NVR range -- update this if we ever use NVRs below r27 */
#define BPF_PPC_NVR_MIN _R27
@@ -79,7 +80,7 @@ static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
* - the bpf program uses its stack area
* The latter condition is deduced from the usage of BPF_REG_FP
*/
- return ctx->seen & SEEN_FUNC || bpf_is_seen_register(ctx, b2p[BPF_REG_FP]);
+ return ctx->seen & SEEN_FUNC || bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP));
}
/*
@@ -134,9 +135,9 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
* invoked through a tail call.
*/
if (ctx->seen & SEEN_TAILCALL) {
- EMIT(PPC_RAW_LI(b2p[TMP_REG_1], 0));
+ EMIT(PPC_RAW_LI(bpf_to_ppc(TMP_REG_1), 0));
/* this goes in the redzone */
- EMIT(PPC_RAW_STD(b2p[TMP_REG_1], _R1, -(BPF_PPC_STACK_SAVE + 8)));
+ EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, -(BPF_PPC_STACK_SAVE + 8)));
} else {
EMIT(PPC_RAW_NOP());
EMIT(PPC_RAW_NOP());
@@ -161,12 +162,12 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
* in the protected zone below the previous stack frame
*/
for (i = BPF_REG_6; i <= BPF_REG_10; i++)
- if (bpf_is_seen_register(ctx, b2p[i]))
- EMIT(PPC_RAW_STD(b2p[i], _R1, bpf_jit_stack_offsetof(ctx, b2p[i])));
+ if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+ EMIT(PPC_RAW_STD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
/* Setup frame pointer to point to the bpf stack area */
- if (bpf_is_seen_register(ctx, b2p[BPF_REG_FP]))
- EMIT(PPC_RAW_ADDI(b2p[BPF_REG_FP], _R1,
+ if (bpf_is_seen_register(ctx, bpf_to_ppc(BPF_REG_FP)))
+ EMIT(PPC_RAW_ADDI(bpf_to_ppc(BPF_REG_FP), _R1,
STACK_FRAME_MIN_SIZE + ctx->stack_size));
}
@@ -176,8 +177,8 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
/* Restore NVRs */
for (i = BPF_REG_6; i <= BPF_REG_10; i++)
- if (bpf_is_seen_register(ctx, b2p[i]))
- EMIT(PPC_RAW_LD(b2p[i], _R1, bpf_jit_stack_offsetof(ctx, b2p[i])));
+ if (bpf_is_seen_register(ctx, bpf_to_ppc(i)))
+ EMIT(PPC_RAW_LD(bpf_to_ppc(i), _R1, bpf_jit_stack_offsetof(ctx, bpf_to_ppc(i))));
/* Tear down our stack frame */
if (bpf_has_stack_frame(ctx)) {
@@ -194,7 +195,7 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
bpf_jit_emit_common_epilogue(image, ctx);
/* Move result to r3 */
- EMIT(PPC_RAW_MR(_R3, b2p[BPF_REG_0]));
+ EMIT(PPC_RAW_MR(_R3, bpf_to_ppc(BPF_REG_0)));
EMIT(PPC_RAW_BLR());
}
@@ -261,8 +262,8 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
* r4/BPF_REG_2 - pointer to bpf_array
* r5/BPF_REG_3 - index in bpf_array
*/
- int b2p_bpf_array = b2p[BPF_REG_2];
- int b2p_index = b2p[BPF_REG_3];
+ int b2p_bpf_array = bpf_to_ppc(BPF_REG_2);
+ int b2p_index = bpf_to_ppc(BPF_REG_3);
int bpf_tailcall_prologue_size = 8;
if (__is_defined(PPC64_ELF_ABI_v2))
@@ -272,42 +273,42 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
* if (index >= array->map.max_entries)
* goto out;
*/
- EMIT(PPC_RAW_LWZ(b2p[TMP_REG_1], b2p_bpf_array, offsetof(struct bpf_array, map.max_entries)));
+ EMIT(PPC_RAW_LWZ(bpf_to_ppc(TMP_REG_1), b2p_bpf_array, offsetof(struct bpf_array, map.max_entries)));
EMIT(PPC_RAW_RLWINM(b2p_index, b2p_index, 0, 0, 31));
- EMIT(PPC_RAW_CMPLW(b2p_index, b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_CMPLW(b2p_index, bpf_to_ppc(TMP_REG_1)));
PPC_BCC_SHORT(COND_GE, out);
/*
* if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
* goto out;
*/
- EMIT(PPC_RAW_LD(b2p[TMP_REG_1], _R1, bpf_jit_stack_tailcallcnt(ctx)));
- EMIT(PPC_RAW_CMPLWI(b2p[TMP_REG_1], MAX_TAIL_CALL_CNT));
+ EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
+ EMIT(PPC_RAW_CMPLWI(bpf_to_ppc(TMP_REG_1), MAX_TAIL_CALL_CNT));
PPC_BCC_SHORT(COND_GE, out);
/*
* tail_call_cnt++;
*/
- EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], b2p[TMP_REG_1], 1));
- EMIT(PPC_RAW_STD(b2p[TMP_REG_1], _R1, bpf_jit_stack_tailcallcnt(ctx)));
+ EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), 1));
+ EMIT(PPC_RAW_STD(bpf_to_ppc(TMP_REG_1), _R1, bpf_jit_stack_tailcallcnt(ctx)));
/* prog = array->ptrs[index]; */
- EMIT(PPC_RAW_MULI(b2p[TMP_REG_1], b2p_index, 8));
- EMIT(PPC_RAW_ADD(b2p[TMP_REG_1], b2p[TMP_REG_1], b2p_bpf_array));
- EMIT(PPC_RAW_LD(b2p[TMP_REG_1], b2p[TMP_REG_1], offsetof(struct bpf_array, ptrs)));
+ EMIT(PPC_RAW_MULI(bpf_to_ppc(TMP_REG_1), b2p_index, 8));
+ EMIT(PPC_RAW_ADD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), b2p_bpf_array));
+ EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), offsetof(struct bpf_array, ptrs)));
/*
* if (prog == NULL)
* goto out;
*/
- EMIT(PPC_RAW_CMPLDI(b2p[TMP_REG_1], 0));
+ EMIT(PPC_RAW_CMPLDI(bpf_to_ppc(TMP_REG_1), 0));
PPC_BCC_SHORT(COND_EQ, out);
/* goto *(prog->bpf_func + prologue_size); */
- EMIT(PPC_RAW_LD(b2p[TMP_REG_1], b2p[TMP_REG_1], offsetof(struct bpf_prog, bpf_func)));
- EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], b2p[TMP_REG_1],
+ EMIT(PPC_RAW_LD(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1), offsetof(struct bpf_prog, bpf_func)));
+ EMIT(PPC_RAW_ADDI(bpf_to_ppc(TMP_REG_1), bpf_to_ppc(TMP_REG_1),
FUNCTION_DESCR_SIZE + bpf_tailcall_prologue_size));
- EMIT(PPC_RAW_MTCTR(b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_MTCTR(bpf_to_ppc(TMP_REG_1)));
/* tear down stack, restore NVRs, ... */
bpf_jit_emit_common_epilogue(image, ctx);
@@ -354,11 +355,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
for (i = 0; i < flen; i++) {
u32 code = insn[i].code;
- u32 dst_reg = b2p[insn[i].dst_reg];
- u32 src_reg = b2p[insn[i].src_reg];
+ u32 dst_reg = bpf_to_ppc(insn[i].dst_reg);
+ u32 src_reg = bpf_to_ppc(insn[i].src_reg);
u32 size = BPF_SIZE(code);
- u32 tmp1_reg = b2p[TMP_REG_1];
- u32 tmp2_reg = b2p[TMP_REG_2];
+ u32 tmp1_reg = bpf_to_ppc(TMP_REG_1);
+ u32 tmp2_reg = bpf_to_ppc(TMP_REG_2);
s16 off = insn[i].off;
s32 imm = insn[i].imm;
bool func_addr_fixed;
@@ -938,7 +939,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
return ret;
/* move return value from r3 to BPF_REG_0 */
- EMIT(PPC_RAW_MR(b2p[BPF_REG_0], _R3));
+ EMIT(PPC_RAW_MR(bpf_to_ppc(BPF_REG_0), _R3));
break;
/*
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 16/17] powerpc64/bpf: Store temp registers' bpf to ppc mapping
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
From: Jordan Niethe <jniethe5@gmail.com>
In bpf_jit_build_body(), the mapping of TMP_REG_1 and TMP_REG_2's bpf
register to ppc register is evalulated at every use despite not
changing. Instead, determine the ppc register once and store the result.
Signed-off-by: Jordan Niethe <jniethe5@gmail.com>
[Rebased, converted additional usage sites]
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit_comp64.c | 197 +++++++++++++-----------------
1 file changed, 86 insertions(+), 111 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index ac06efa7022379..b4de0c35c8a4ab 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -357,6 +357,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
u32 dst_reg = b2p[insn[i].dst_reg];
u32 src_reg = b2p[insn[i].src_reg];
u32 size = BPF_SIZE(code);
+ u32 tmp1_reg = b2p[TMP_REG_1];
+ u32 tmp2_reg = b2p[TMP_REG_2];
s16 off = insn[i].off;
s32 imm = insn[i].imm;
bool func_addr_fixed;
@@ -407,8 +409,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
} else if (imm >= -32768 && imm < 32768) {
EMIT(PPC_RAW_ADDI(dst_reg, dst_reg, IMM_L(imm)));
} else {
- PPC_LI32(b2p[TMP_REG_1], imm);
- EMIT(PPC_RAW_ADD(dst_reg, dst_reg, b2p[TMP_REG_1]));
+ PPC_LI32(tmp1_reg, imm);
+ EMIT(PPC_RAW_ADD(dst_reg, dst_reg, tmp1_reg));
}
goto bpf_alu32_trunc;
case BPF_ALU | BPF_SUB | BPF_K: /* (u32) dst -= (u32) imm */
@@ -418,8 +420,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
} else if (imm > -32768 && imm <= 32768) {
EMIT(PPC_RAW_ADDI(dst_reg, dst_reg, IMM_L(-imm)));
} else {
- PPC_LI32(b2p[TMP_REG_1], imm);
- EMIT(PPC_RAW_SUB(dst_reg, dst_reg, b2p[TMP_REG_1]));
+ PPC_LI32(tmp1_reg, imm);
+ EMIT(PPC_RAW_SUB(dst_reg, dst_reg, tmp1_reg));
}
goto bpf_alu32_trunc;
case BPF_ALU | BPF_MUL | BPF_X: /* (u32) dst *= (u32) src */
@@ -434,32 +436,28 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
if (imm >= -32768 && imm < 32768)
EMIT(PPC_RAW_MULI(dst_reg, dst_reg, IMM_L(imm)));
else {
- PPC_LI32(b2p[TMP_REG_1], imm);
+ PPC_LI32(tmp1_reg, imm);
if (BPF_CLASS(code) == BPF_ALU)
- EMIT(PPC_RAW_MULW(dst_reg, dst_reg,
- b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_MULW(dst_reg, dst_reg, tmp1_reg));
else
- EMIT(PPC_RAW_MULD(dst_reg, dst_reg,
- b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_MULD(dst_reg, dst_reg, tmp1_reg));
}
goto bpf_alu32_trunc;
case BPF_ALU | BPF_DIV | BPF_X: /* (u32) dst /= (u32) src */
case BPF_ALU | BPF_MOD | BPF_X: /* (u32) dst %= (u32) src */
if (BPF_OP(code) == BPF_MOD) {
- EMIT(PPC_RAW_DIVWU(b2p[TMP_REG_1], dst_reg, src_reg));
- EMIT(PPC_RAW_MULW(b2p[TMP_REG_1], src_reg,
- b2p[TMP_REG_1]));
- EMIT(PPC_RAW_SUB(dst_reg, dst_reg, b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_DIVWU(tmp1_reg, dst_reg, src_reg));
+ EMIT(PPC_RAW_MULW(tmp1_reg, src_reg, tmp1_reg));
+ EMIT(PPC_RAW_SUB(dst_reg, dst_reg, tmp1_reg));
} else
EMIT(PPC_RAW_DIVWU(dst_reg, dst_reg, src_reg));
goto bpf_alu32_trunc;
case BPF_ALU64 | BPF_DIV | BPF_X: /* dst /= src */
case BPF_ALU64 | BPF_MOD | BPF_X: /* dst %= src */
if (BPF_OP(code) == BPF_MOD) {
- EMIT(PPC_RAW_DIVDU(b2p[TMP_REG_1], dst_reg, src_reg));
- EMIT(PPC_RAW_MULD(b2p[TMP_REG_1], src_reg,
- b2p[TMP_REG_1]));
- EMIT(PPC_RAW_SUB(dst_reg, dst_reg, b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_DIVDU(tmp1_reg, dst_reg, src_reg));
+ EMIT(PPC_RAW_MULD(tmp1_reg, src_reg, tmp1_reg));
+ EMIT(PPC_RAW_SUB(dst_reg, dst_reg, tmp1_reg));
} else
EMIT(PPC_RAW_DIVDU(dst_reg, dst_reg, src_reg));
break;
@@ -478,35 +476,23 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
}
}
- PPC_LI32(b2p[TMP_REG_1], imm);
+ PPC_LI32(tmp1_reg, imm);
switch (BPF_CLASS(code)) {
case BPF_ALU:
if (BPF_OP(code) == BPF_MOD) {
- EMIT(PPC_RAW_DIVWU(b2p[TMP_REG_2],
- dst_reg,
- b2p[TMP_REG_1]));
- EMIT(PPC_RAW_MULW(b2p[TMP_REG_1],
- b2p[TMP_REG_1],
- b2p[TMP_REG_2]));
- EMIT(PPC_RAW_SUB(dst_reg, dst_reg,
- b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_DIVWU(tmp2_reg, dst_reg, tmp1_reg));
+ EMIT(PPC_RAW_MULW(tmp1_reg, tmp1_reg, tmp2_reg));
+ EMIT(PPC_RAW_SUB(dst_reg, dst_reg, tmp1_reg));
} else
- EMIT(PPC_RAW_DIVWU(dst_reg, dst_reg,
- b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_DIVWU(dst_reg, dst_reg, tmp1_reg));
break;
case BPF_ALU64:
if (BPF_OP(code) == BPF_MOD) {
- EMIT(PPC_RAW_DIVDU(b2p[TMP_REG_2],
- dst_reg,
- b2p[TMP_REG_1]));
- EMIT(PPC_RAW_MULD(b2p[TMP_REG_1],
- b2p[TMP_REG_1],
- b2p[TMP_REG_2]));
- EMIT(PPC_RAW_SUB(dst_reg, dst_reg,
- b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_DIVDU(tmp2_reg, dst_reg, tmp1_reg));
+ EMIT(PPC_RAW_MULD(tmp1_reg, tmp1_reg, tmp2_reg));
+ EMIT(PPC_RAW_SUB(dst_reg, dst_reg, tmp1_reg));
} else
- EMIT(PPC_RAW_DIVDU(dst_reg, dst_reg,
- b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_DIVDU(dst_reg, dst_reg, tmp1_reg));
break;
}
goto bpf_alu32_trunc;
@@ -528,8 +514,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
EMIT(PPC_RAW_ANDI(dst_reg, dst_reg, IMM_L(imm)));
else {
/* Sign-extended */
- PPC_LI32(b2p[TMP_REG_1], imm);
- EMIT(PPC_RAW_AND(dst_reg, dst_reg, b2p[TMP_REG_1]));
+ PPC_LI32(tmp1_reg, imm);
+ EMIT(PPC_RAW_AND(dst_reg, dst_reg, tmp1_reg));
}
goto bpf_alu32_trunc;
case BPF_ALU | BPF_OR | BPF_X: /* dst = (u32) dst | (u32) src */
@@ -540,8 +526,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
case BPF_ALU64 | BPF_OR | BPF_K:/* dst = dst | imm */
if (imm < 0 && BPF_CLASS(code) == BPF_ALU64) {
/* Sign-extended */
- PPC_LI32(b2p[TMP_REG_1], imm);
- EMIT(PPC_RAW_OR(dst_reg, dst_reg, b2p[TMP_REG_1]));
+ PPC_LI32(tmp1_reg, imm);
+ EMIT(PPC_RAW_OR(dst_reg, dst_reg, tmp1_reg));
} else {
if (IMM_L(imm))
EMIT(PPC_RAW_ORI(dst_reg, dst_reg, IMM_L(imm)));
@@ -557,8 +543,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
case BPF_ALU64 | BPF_XOR | BPF_K: /* dst ^= imm */
if (imm < 0 && BPF_CLASS(code) == BPF_ALU64) {
/* Sign-extended */
- PPC_LI32(b2p[TMP_REG_1], imm);
- EMIT(PPC_RAW_XOR(dst_reg, dst_reg, b2p[TMP_REG_1]));
+ PPC_LI32(tmp1_reg, imm);
+ EMIT(PPC_RAW_XOR(dst_reg, dst_reg, tmp1_reg));
} else {
if (IMM_L(imm))
EMIT(PPC_RAW_XORI(dst_reg, dst_reg, IMM_L(imm)));
@@ -659,11 +645,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
switch (imm) {
case 16:
/* Rotate 8 bits left & mask with 0x0000ff00 */
- EMIT(PPC_RAW_RLWINM(b2p[TMP_REG_1], dst_reg, 8, 16, 23));
+ EMIT(PPC_RAW_RLWINM(tmp1_reg, dst_reg, 8, 16, 23));
/* Rotate 8 bits right & insert LSB to reg */
- EMIT(PPC_RAW_RLWIMI(b2p[TMP_REG_1], dst_reg, 24, 24, 31));
+ EMIT(PPC_RAW_RLWIMI(tmp1_reg, dst_reg, 24, 24, 31));
/* Move result back to dst_reg */
- EMIT(PPC_RAW_MR(dst_reg, b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_MR(dst_reg, tmp1_reg));
break;
case 32:
/*
@@ -671,28 +657,28 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
* 2 bytes are already in their final position
* -- byte 2 and 4 (of bytes 1, 2, 3 and 4)
*/
- EMIT(PPC_RAW_RLWINM(b2p[TMP_REG_1], dst_reg, 8, 0, 31));
+ EMIT(PPC_RAW_RLWINM(tmp1_reg, dst_reg, 8, 0, 31));
/* Rotate 24 bits and insert byte 1 */
- EMIT(PPC_RAW_RLWIMI(b2p[TMP_REG_1], dst_reg, 24, 0, 7));
+ EMIT(PPC_RAW_RLWIMI(tmp1_reg, dst_reg, 24, 0, 7));
/* Rotate 24 bits and insert byte 3 */
- EMIT(PPC_RAW_RLWIMI(b2p[TMP_REG_1], dst_reg, 24, 16, 23));
- EMIT(PPC_RAW_MR(dst_reg, b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_RLWIMI(tmp1_reg, dst_reg, 24, 16, 23));
+ EMIT(PPC_RAW_MR(dst_reg, tmp1_reg));
break;
case 64:
/* Store the value to stack and then use byte-reverse loads */
EMIT(PPC_RAW_STD(dst_reg, _R1, bpf_jit_stack_local(ctx)));
- EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], _R1, bpf_jit_stack_local(ctx)));
+ EMIT(PPC_RAW_ADDI(tmp1_reg, _R1, bpf_jit_stack_local(ctx)));
if (cpu_has_feature(CPU_FTR_ARCH_206)) {
- EMIT(PPC_RAW_LDBRX(dst_reg, 0, b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_LDBRX(dst_reg, 0, tmp1_reg));
} else {
- EMIT(PPC_RAW_LWBRX(dst_reg, 0, b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_LWBRX(dst_reg, 0, tmp1_reg));
if (IS_ENABLED(CONFIG_CPU_LITTLE_ENDIAN))
EMIT(PPC_RAW_SLDI(dst_reg, dst_reg, 32));
- EMIT(PPC_RAW_LI(b2p[TMP_REG_2], 4));
- EMIT(PPC_RAW_LWBRX(b2p[TMP_REG_2], b2p[TMP_REG_2], b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_LI(tmp2_reg, 4));
+ EMIT(PPC_RAW_LWBRX(tmp2_reg, tmp2_reg, tmp1_reg));
if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN))
- EMIT(PPC_RAW_SLDI(b2p[TMP_REG_2], b2p[TMP_REG_2], 32));
- EMIT(PPC_RAW_OR(dst_reg, dst_reg, b2p[TMP_REG_2]));
+ EMIT(PPC_RAW_SLDI(tmp2_reg, tmp2_reg, 32));
+ EMIT(PPC_RAW_OR(dst_reg, dst_reg, tmp2_reg));
}
break;
}
@@ -731,7 +717,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
break;
case STF_BARRIER_SYNC_ORI:
EMIT(PPC_RAW_SYNC());
- EMIT(PPC_RAW_LD(b2p[TMP_REG_1], _R13, 0));
+ EMIT(PPC_RAW_LD(tmp1_reg, _R13, 0));
EMIT(PPC_RAW_ORI(_R31, _R31, 0));
break;
case STF_BARRIER_FALLBACK:
@@ -751,36 +737,36 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
case BPF_STX | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = imm */
if (BPF_CLASS(code) == BPF_ST) {
- EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
- src_reg = b2p[TMP_REG_1];
+ EMIT(PPC_RAW_LI(tmp1_reg, imm));
+ src_reg = tmp1_reg;
}
EMIT(PPC_RAW_STB(src_reg, dst_reg, off));
break;
case BPF_STX | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = imm */
if (BPF_CLASS(code) == BPF_ST) {
- EMIT(PPC_RAW_LI(b2p[TMP_REG_1], imm));
- src_reg = b2p[TMP_REG_1];
+ EMIT(PPC_RAW_LI(tmp1_reg, imm));
+ src_reg = tmp1_reg;
}
EMIT(PPC_RAW_STH(src_reg, dst_reg, off));
break;
case BPF_STX | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = imm */
if (BPF_CLASS(code) == BPF_ST) {
- PPC_LI32(b2p[TMP_REG_1], imm);
- src_reg = b2p[TMP_REG_1];
+ PPC_LI32(tmp1_reg, imm);
+ src_reg = tmp1_reg;
}
EMIT(PPC_RAW_STW(src_reg, dst_reg, off));
break;
case BPF_STX | BPF_MEM | BPF_DW: /* (u64 *)(dst + off) = src */
case BPF_ST | BPF_MEM | BPF_DW: /* *(u64 *)(dst + off) = imm */
if (BPF_CLASS(code) == BPF_ST) {
- PPC_LI32(b2p[TMP_REG_1], imm);
- src_reg = b2p[TMP_REG_1];
+ PPC_LI32(tmp1_reg, imm);
+ src_reg = tmp1_reg;
}
if (off % 4) {
- EMIT(PPC_RAW_LI(b2p[TMP_REG_2], off));
- EMIT(PPC_RAW_STDX(src_reg, dst_reg, b2p[TMP_REG_2]));
+ EMIT(PPC_RAW_LI(tmp2_reg, off));
+ EMIT(PPC_RAW_STDX(src_reg, dst_reg, tmp2_reg));
} else {
EMIT(PPC_RAW_STD(src_reg, dst_reg, off));
}
@@ -800,14 +786,14 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
/* *(u32 *)(dst + off) += src */
/* Get EA into TMP_REG_1 */
- EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], dst_reg, off));
+ EMIT(PPC_RAW_ADDI(tmp1_reg, dst_reg, off));
tmp_idx = ctx->idx * 4;
/* load value from memory into TMP_REG_2 */
- EMIT(PPC_RAW_LWARX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1], 0));
+ EMIT(PPC_RAW_LWARX(tmp2_reg, 0, tmp1_reg, 0));
/* add value from src_reg into this */
- EMIT(PPC_RAW_ADD(b2p[TMP_REG_2], b2p[TMP_REG_2], src_reg));
+ EMIT(PPC_RAW_ADD(tmp2_reg, tmp2_reg, src_reg));
/* store result back */
- EMIT(PPC_RAW_STWCX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_STWCX(tmp2_reg, 0, tmp1_reg));
/* we're done if this succeeded */
PPC_BCC_SHORT(COND_NE, tmp_idx);
break;
@@ -820,11 +806,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
}
/* *(u64 *)(dst + off) += src */
- EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], dst_reg, off));
+ EMIT(PPC_RAW_ADDI(tmp1_reg, dst_reg, off));
tmp_idx = ctx->idx * 4;
- EMIT(PPC_RAW_LDARX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1], 0));
- EMIT(PPC_RAW_ADD(b2p[TMP_REG_2], b2p[TMP_REG_2], src_reg));
- EMIT(PPC_RAW_STDCX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_LDARX(tmp2_reg, 0, tmp1_reg, 0));
+ EMIT(PPC_RAW_ADD(tmp2_reg, tmp2_reg, src_reg));
+ EMIT(PPC_RAW_STDCX(tmp2_reg, 0, tmp1_reg));
PPC_BCC_SHORT(COND_NE, tmp_idx);
break;
@@ -850,12 +836,12 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
* set dst_reg=0 and move on.
*/
if (BPF_MODE(code) == BPF_PROBE_MEM) {
- EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], src_reg, off));
+ EMIT(PPC_RAW_ADDI(tmp1_reg, src_reg, off));
if (IS_ENABLED(CONFIG_PPC_BOOK3E_64))
- PPC_LI64(b2p[TMP_REG_2], 0x8000000000000000ul);
+ PPC_LI64(tmp2_reg, 0x8000000000000000ul);
else /* BOOK3S_64 */
- PPC_LI64(b2p[TMP_REG_2], PAGE_OFFSET);
- EMIT(PPC_RAW_CMPLD(b2p[TMP_REG_1], b2p[TMP_REG_2]));
+ PPC_LI64(tmp2_reg, PAGE_OFFSET);
+ EMIT(PPC_RAW_CMPLD(tmp1_reg, tmp2_reg));
PPC_BCC_SHORT(COND_GT, (ctx->idx + 3) * 4);
EMIT(PPC_RAW_LI(dst_reg, 0));
/*
@@ -880,8 +866,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
break;
case BPF_DW:
if (off % 4) {
- EMIT(PPC_RAW_LI(b2p[TMP_REG_1], off));
- EMIT(PPC_RAW_LDX(dst_reg, src_reg, b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_LI(tmp1_reg, off));
+ EMIT(PPC_RAW_LDX(dst_reg, src_reg, tmp1_reg));
} else {
EMIT(PPC_RAW_LD(dst_reg, src_reg, off));
}
@@ -925,7 +911,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
* we'll just fall through to the epilogue.
*/
if (i != flen - 1) {
- ret = bpf_jit_emit_exit_insn(image, ctx, b2p[TMP_REG_1], exit_addr);
+ ret = bpf_jit_emit_exit_insn(image, ctx, tmp1_reg, exit_addr);
if (ret)
return ret;
}
@@ -1058,14 +1044,10 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
case BPF_JMP | BPF_JSET | BPF_X:
case BPF_JMP32 | BPF_JSET | BPF_X:
if (BPF_CLASS(code) == BPF_JMP) {
- EMIT(PPC_RAW_AND_DOT(b2p[TMP_REG_1], dst_reg,
- src_reg));
+ EMIT(PPC_RAW_AND_DOT(tmp1_reg, dst_reg, src_reg));
} else {
- int tmp_reg = b2p[TMP_REG_1];
-
- EMIT(PPC_RAW_AND(tmp_reg, dst_reg, src_reg));
- EMIT(PPC_RAW_RLWINM_DOT(tmp_reg, tmp_reg, 0, 0,
- 31));
+ EMIT(PPC_RAW_AND(tmp1_reg, dst_reg, src_reg));
+ EMIT(PPC_RAW_RLWINM_DOT(tmp1_reg, tmp1_reg, 0, 0, 31));
}
break;
case BPF_JMP | BPF_JNE | BPF_K:
@@ -1094,14 +1076,12 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
EMIT(PPC_RAW_CMPLDI(dst_reg, imm));
} else {
/* sign-extending load */
- PPC_LI32(b2p[TMP_REG_1], imm);
+ PPC_LI32(tmp1_reg, imm);
/* ... but unsigned comparison */
if (is_jmp32)
- EMIT(PPC_RAW_CMPLW(dst_reg,
- b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_CMPLW(dst_reg, tmp1_reg));
else
- EMIT(PPC_RAW_CMPLD(dst_reg,
- b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_CMPLD(dst_reg, tmp1_reg));
}
break;
}
@@ -1126,13 +1106,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
else
EMIT(PPC_RAW_CMPDI(dst_reg, imm));
} else {
- PPC_LI32(b2p[TMP_REG_1], imm);
+ PPC_LI32(tmp1_reg, imm);
if (is_jmp32)
- EMIT(PPC_RAW_CMPW(dst_reg,
- b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_CMPW(dst_reg, tmp1_reg));
else
- EMIT(PPC_RAW_CMPD(dst_reg,
- b2p[TMP_REG_1]));
+ EMIT(PPC_RAW_CMPD(dst_reg, tmp1_reg));
}
break;
}
@@ -1141,19 +1119,16 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
/* andi does not sign-extend the immediate */
if (imm >= 0 && imm < 32768)
/* PPC_ANDI is _only/always_ dot-form */
- EMIT(PPC_RAW_ANDI(b2p[TMP_REG_1], dst_reg, imm));
+ EMIT(PPC_RAW_ANDI(tmp1_reg, dst_reg, imm));
else {
- int tmp_reg = b2p[TMP_REG_1];
-
- PPC_LI32(tmp_reg, imm);
+ PPC_LI32(tmp1_reg, imm);
if (BPF_CLASS(code) == BPF_JMP) {
- EMIT(PPC_RAW_AND_DOT(tmp_reg, dst_reg,
- tmp_reg));
+ EMIT(PPC_RAW_AND_DOT(tmp1_reg, dst_reg,
+ tmp1_reg));
} else {
- EMIT(PPC_RAW_AND(tmp_reg, dst_reg,
- tmp_reg));
- EMIT(PPC_RAW_RLWINM_DOT(tmp_reg, tmp_reg,
- 0, 0, 31));
+ EMIT(PPC_RAW_AND(tmp1_reg, dst_reg, tmp1_reg));
+ EMIT(PPC_RAW_RLWINM_DOT(tmp1_reg, tmp1_reg,
+ 0, 0, 31));
}
}
break;
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 14/17] powerpc/bpf: Move bpf_jit64.h into bpf_jit_comp64.c
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
There is no need for a separate header anymore. Move the contents of
bpf_jit64.h into bpf_jit_comp64.c
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit64.h | 69 -------------------------------
arch/powerpc/net/bpf_jit_comp64.c | 54 +++++++++++++++++++++++-
2 files changed, 53 insertions(+), 70 deletions(-)
delete mode 100644 arch/powerpc/net/bpf_jit64.h
diff --git a/arch/powerpc/net/bpf_jit64.h b/arch/powerpc/net/bpf_jit64.h
deleted file mode 100644
index 199348b7296653..00000000000000
--- a/arch/powerpc/net/bpf_jit64.h
+++ /dev/null
@@ -1,69 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * bpf_jit64.h: BPF JIT compiler for PPC64
- *
- * Copyright 2016 Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
- * IBM Corporation
- */
-#ifndef _BPF_JIT64_H
-#define _BPF_JIT64_H
-
-#include "bpf_jit.h"
-
-/*
- * Stack layout:
- * Ensure the top half (upto local_tmp_var) stays consistent
- * with our redzone usage.
- *
- * [ prev sp ] <-------------
- * [ nv gpr save area ] 5*8 |
- * [ tail_call_cnt ] 8 |
- * [ local_tmp_var ] 16 |
- * fp (r31) --> [ ebpf stack space ] upto 512 |
- * [ frame header ] 32/112 |
- * sp (r1) ---> [ stack pointer ] --------------
- */
-
-/* for gpr non volatile registers BPG_REG_6 to 10 */
-#define BPF_PPC_STACK_SAVE (5*8)
-/* for bpf JIT code internal usage */
-#define BPF_PPC_STACK_LOCALS 24
-/* stack frame excluding BPF stack, ensure this is quadword aligned */
-#define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \
- BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE)
-
-#ifndef __ASSEMBLY__
-
-/* BPF register usage */
-#define TMP_REG_1 (MAX_BPF_JIT_REG + 0)
-#define TMP_REG_2 (MAX_BPF_JIT_REG + 1)
-
-/* BPF to ppc register mappings */
-const int b2p[MAX_BPF_JIT_REG + 2] = {
- /* function return value */
- [BPF_REG_0] = 8,
- /* function arguments */
- [BPF_REG_1] = 3,
- [BPF_REG_2] = 4,
- [BPF_REG_3] = 5,
- [BPF_REG_4] = 6,
- [BPF_REG_5] = 7,
- /* non volatile registers */
- [BPF_REG_6] = 27,
- [BPF_REG_7] = 28,
- [BPF_REG_8] = 29,
- [BPF_REG_9] = 30,
- /* frame pointer aka BPF_REG_10 */
- [BPF_REG_FP] = 31,
- /* eBPF jit internal registers */
- [BPF_REG_AX] = 12,
- [TMP_REG_1] = 9,
- [TMP_REG_2] = 10
-};
-
-/* PPC NVR range -- update this if we ever use NVRs below r27 */
-#define BPF_PPC_NVR_MIN 27
-
-#endif /* !__ASSEMBLY__ */
-
-#endif
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index eeda636cd7be64..3e4ed556094770 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -17,7 +17,59 @@
#include <linux/bpf.h>
#include <asm/security_features.h>
-#include "bpf_jit64.h"
+#include "bpf_jit.h"
+
+/*
+ * Stack layout:
+ * Ensure the top half (upto local_tmp_var) stays consistent
+ * with our redzone usage.
+ *
+ * [ prev sp ] <-------------
+ * [ nv gpr save area ] 5*8 |
+ * [ tail_call_cnt ] 8 |
+ * [ local_tmp_var ] 16 |
+ * fp (r31) --> [ ebpf stack space ] upto 512 |
+ * [ frame header ] 32/112 |
+ * sp (r1) ---> [ stack pointer ] --------------
+ */
+
+/* for gpr non volatile registers BPG_REG_6 to 10 */
+#define BPF_PPC_STACK_SAVE (5*8)
+/* for bpf JIT code internal usage */
+#define BPF_PPC_STACK_LOCALS 24
+/* stack frame excluding BPF stack, ensure this is quadword aligned */
+#define BPF_PPC_STACKFRAME (STACK_FRAME_MIN_SIZE + \
+ BPF_PPC_STACK_LOCALS + BPF_PPC_STACK_SAVE)
+
+/* BPF register usage */
+#define TMP_REG_1 (MAX_BPF_JIT_REG + 0)
+#define TMP_REG_2 (MAX_BPF_JIT_REG + 1)
+
+/* BPF to ppc register mappings */
+const int b2p[MAX_BPF_JIT_REG + 2] = {
+ /* function return value */
+ [BPF_REG_0] = 8,
+ /* function arguments */
+ [BPF_REG_1] = 3,
+ [BPF_REG_2] = 4,
+ [BPF_REG_3] = 5,
+ [BPF_REG_4] = 6,
+ [BPF_REG_5] = 7,
+ /* non volatile registers */
+ [BPF_REG_6] = 27,
+ [BPF_REG_7] = 28,
+ [BPF_REG_8] = 29,
+ [BPF_REG_9] = 30,
+ /* frame pointer aka BPF_REG_10 */
+ [BPF_REG_FP] = 31,
+ /* eBPF jit internal registers */
+ [BPF_REG_AX] = 12,
+ [TMP_REG_1] = 9,
+ [TMP_REG_2] = 10
+};
+
+/* PPC NVR range -- update this if we ever use NVRs below r27 */
+#define BPF_PPC_NVR_MIN 27
static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
{
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 15/17] powerpc/bpf: Use _Rn macros for GPRs
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
Use _Rn macros to specify register names to make their usage clear.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit_comp32.c | 30 +++++++-------
arch/powerpc/net/bpf_jit_comp64.c | 68 +++++++++++++++----------------
2 files changed, 49 insertions(+), 49 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index 063e3a1be9270d..fe4e0eca017ede 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -41,23 +41,23 @@
/* BPF to ppc register mappings */
const int b2p[MAX_BPF_JIT_REG + 1] = {
/* function return value */
- [BPF_REG_0] = 12,
+ [BPF_REG_0] = _R12,
/* function arguments */
- [BPF_REG_1] = 4,
- [BPF_REG_2] = 6,
- [BPF_REG_3] = 8,
- [BPF_REG_4] = 10,
- [BPF_REG_5] = 22,
+ [BPF_REG_1] = _R4,
+ [BPF_REG_2] = _R6,
+ [BPF_REG_3] = _R8,
+ [BPF_REG_4] = _R10,
+ [BPF_REG_5] = _R22,
/* non volatile registers */
- [BPF_REG_6] = 24,
- [BPF_REG_7] = 26,
- [BPF_REG_8] = 28,
- [BPF_REG_9] = 30,
+ [BPF_REG_6] = _R24,
+ [BPF_REG_7] = _R26,
+ [BPF_REG_8] = _R28,
+ [BPF_REG_9] = _R30,
/* frame pointer aka BPF_REG_10 */
- [BPF_REG_FP] = 18,
+ [BPF_REG_FP] = _R18,
/* eBPF jit internal registers */
- [BPF_REG_AX] = 20,
- [TMP_REG] = 31, /* 32 bits */
+ [BPF_REG_AX] = _R20,
+ [TMP_REG] = _R31, /* 32 bits */
};
static int bpf_to_ppc(struct codegen_context *ctx, int reg)
@@ -66,8 +66,8 @@ static int bpf_to_ppc(struct codegen_context *ctx, int reg)
}
/* PPC NVR range -- update this if we ever use NVRs below r17 */
-#define BPF_PPC_NVR_MIN 17
-#define BPF_PPC_TC 16
+#define BPF_PPC_NVR_MIN _R17
+#define BPF_PPC_TC _R16
static int bpf_jit_stack_offsetof(struct codegen_context *ctx, int reg)
{
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 3e4ed556094770..ac06efa7022379 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -48,28 +48,28 @@
/* BPF to ppc register mappings */
const int b2p[MAX_BPF_JIT_REG + 2] = {
/* function return value */
- [BPF_REG_0] = 8,
+ [BPF_REG_0] = _R8,
/* function arguments */
- [BPF_REG_1] = 3,
- [BPF_REG_2] = 4,
- [BPF_REG_3] = 5,
- [BPF_REG_4] = 6,
- [BPF_REG_5] = 7,
+ [BPF_REG_1] = _R3,
+ [BPF_REG_2] = _R4,
+ [BPF_REG_3] = _R5,
+ [BPF_REG_4] = _R6,
+ [BPF_REG_5] = _R7,
/* non volatile registers */
- [BPF_REG_6] = 27,
- [BPF_REG_7] = 28,
- [BPF_REG_8] = 29,
- [BPF_REG_9] = 30,
+ [BPF_REG_6] = _R27,
+ [BPF_REG_7] = _R28,
+ [BPF_REG_8] = _R29,
+ [BPF_REG_9] = _R30,
/* frame pointer aka BPF_REG_10 */
- [BPF_REG_FP] = 31,
+ [BPF_REG_FP] = _R31,
/* eBPF jit internal registers */
- [BPF_REG_AX] = 12,
- [TMP_REG_1] = 9,
- [TMP_REG_2] = 10
+ [BPF_REG_AX] = _R12,
+ [TMP_REG_1] = _R9,
+ [TMP_REG_2] = _R10
};
/* PPC NVR range -- update this if we ever use NVRs below r27 */
-#define BPF_PPC_NVR_MIN 27
+#define BPF_PPC_NVR_MIN _R27
static inline bool bpf_has_stack_frame(struct codegen_context *ctx)
{
@@ -136,7 +136,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
if (ctx->seen & SEEN_TAILCALL) {
EMIT(PPC_RAW_LI(b2p[TMP_REG_1], 0));
/* this goes in the redzone */
- EMIT(PPC_RAW_STD(b2p[TMP_REG_1], 1, -(BPF_PPC_STACK_SAVE + 8)));
+ EMIT(PPC_RAW_STD(b2p[TMP_REG_1], _R1, -(BPF_PPC_STACK_SAVE + 8)));
} else {
EMIT(PPC_RAW_NOP());
EMIT(PPC_RAW_NOP());
@@ -149,10 +149,10 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
*/
if (ctx->seen & SEEN_FUNC) {
EMIT(PPC_RAW_MFLR(_R0));
- EMIT(PPC_RAW_STD(0, 1, PPC_LR_STKOFF));
+ EMIT(PPC_RAW_STD(_R0, _R1, PPC_LR_STKOFF));
}
- EMIT(PPC_RAW_STDU(1, 1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
+ EMIT(PPC_RAW_STDU(_R1, _R1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
}
/*
@@ -162,11 +162,11 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
*/
for (i = BPF_REG_6; i <= BPF_REG_10; i++)
if (bpf_is_seen_register(ctx, b2p[i]))
- EMIT(PPC_RAW_STD(b2p[i], 1, bpf_jit_stack_offsetof(ctx, b2p[i])));
+ EMIT(PPC_RAW_STD(b2p[i], _R1, bpf_jit_stack_offsetof(ctx, b2p[i])));
/* Setup frame pointer to point to the bpf stack area */
if (bpf_is_seen_register(ctx, b2p[BPF_REG_FP]))
- EMIT(PPC_RAW_ADDI(b2p[BPF_REG_FP], 1,
+ EMIT(PPC_RAW_ADDI(b2p[BPF_REG_FP], _R1,
STACK_FRAME_MIN_SIZE + ctx->stack_size));
}
@@ -177,14 +177,14 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
/* Restore NVRs */
for (i = BPF_REG_6; i <= BPF_REG_10; i++)
if (bpf_is_seen_register(ctx, b2p[i]))
- EMIT(PPC_RAW_LD(b2p[i], 1, bpf_jit_stack_offsetof(ctx, b2p[i])));
+ EMIT(PPC_RAW_LD(b2p[i], _R1, bpf_jit_stack_offsetof(ctx, b2p[i])));
/* Tear down our stack frame */
if (bpf_has_stack_frame(ctx)) {
- EMIT(PPC_RAW_ADDI(1, 1, BPF_PPC_STACKFRAME + ctx->stack_size));
+ EMIT(PPC_RAW_ADDI(_R1, _R1, BPF_PPC_STACKFRAME + ctx->stack_size));
if (ctx->seen & SEEN_FUNC) {
- EMIT(PPC_RAW_LD(0, 1, PPC_LR_STKOFF));
- EMIT(PPC_RAW_MTLR(0));
+ EMIT(PPC_RAW_LD(_R0, _R1, PPC_LR_STKOFF));
+ EMIT(PPC_RAW_MTLR(_R0));
}
}
}
@@ -194,7 +194,7 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
bpf_jit_emit_common_epilogue(image, ctx);
/* Move result to r3 */
- EMIT(PPC_RAW_MR(3, b2p[BPF_REG_0]));
+ EMIT(PPC_RAW_MR(_R3, b2p[BPF_REG_0]));
EMIT(PPC_RAW_BLR());
}
@@ -232,7 +232,7 @@ int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func
func += FUNCTION_DESCR_SIZE;
/* Load function address into r12 */
- PPC_LI64(12, func);
+ PPC_LI64(_R12, func);
/* For bpf-to-bpf function calls, the callee's address is unknown
* until the last extra pass. As seen above, we use PPC_LI64() to
@@ -247,7 +247,7 @@ int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func
for (i = ctx->idx - ctx_idx; i < 5; i++)
EMIT(PPC_RAW_NOP());
- EMIT(PPC_RAW_MTCTR(12));
+ EMIT(PPC_RAW_MTCTR(_R12));
EMIT(PPC_RAW_BCTRL());
return 0;
@@ -281,7 +281,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
* if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
* goto out;
*/
- EMIT(PPC_RAW_LD(b2p[TMP_REG_1], 1, bpf_jit_stack_tailcallcnt(ctx)));
+ EMIT(PPC_RAW_LD(b2p[TMP_REG_1], _R1, bpf_jit_stack_tailcallcnt(ctx)));
EMIT(PPC_RAW_CMPLWI(b2p[TMP_REG_1], MAX_TAIL_CALL_CNT));
PPC_BCC_SHORT(COND_GE, out);
@@ -289,7 +289,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
* tail_call_cnt++;
*/
EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], b2p[TMP_REG_1], 1));
- EMIT(PPC_RAW_STD(b2p[TMP_REG_1], 1, bpf_jit_stack_tailcallcnt(ctx)));
+ EMIT(PPC_RAW_STD(b2p[TMP_REG_1], _R1, bpf_jit_stack_tailcallcnt(ctx)));
/* prog = array->ptrs[index]; */
EMIT(PPC_RAW_MULI(b2p[TMP_REG_1], b2p_index, 8));
@@ -680,8 +680,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
break;
case 64:
/* Store the value to stack and then use byte-reverse loads */
- EMIT(PPC_RAW_STD(dst_reg, 1, bpf_jit_stack_local(ctx)));
- EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], 1, bpf_jit_stack_local(ctx)));
+ EMIT(PPC_RAW_STD(dst_reg, _R1, bpf_jit_stack_local(ctx)));
+ EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], _R1, bpf_jit_stack_local(ctx)));
if (cpu_has_feature(CPU_FTR_ARCH_206)) {
EMIT(PPC_RAW_LDBRX(dst_reg, 0, b2p[TMP_REG_1]));
} else {
@@ -736,8 +736,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
break;
case STF_BARRIER_FALLBACK:
ctx->seen |= SEEN_FUNC;
- PPC_LI64(12, dereference_kernel_function_descriptor(bpf_stf_barrier));
- EMIT(PPC_RAW_MTCTR(12));
+ PPC_LI64(_R12, dereference_kernel_function_descriptor(bpf_stf_barrier));
+ EMIT(PPC_RAW_MTCTR(_R12));
EMIT(PPC_RAW_BCTRL());
break;
case STF_BARRIER_NONE:
@@ -952,7 +952,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
return ret;
/* move return value from r3 to BPF_REG_0 */
- EMIT(PPC_RAW_MR(b2p[BPF_REG_0], 3));
+ EMIT(PPC_RAW_MR(b2p[BPF_REG_0], _R3));
break;
/*
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 13/17] powerpc/bpf: Cleanup bpf_jit.h
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
- PPC_EX32() is only used by ppc32 JIT. Move it to bpf_jit_comp32.c
- PPC_LI64() is only valid in ppc64. #ifdef it
- PPC_FUNC_ADDR() is not used anymore. Remove it.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit.h | 10 +---------
arch/powerpc/net/bpf_jit_comp32.c | 2 ++
2 files changed, 3 insertions(+), 9 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 0832235a274983..d9bdc9df2e48ed 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -59,10 +59,7 @@
EMIT(PPC_RAW_ORI(d, d, IMM_L(i))); \
} } while(0)
-#ifdef CONFIG_PPC32
-#define PPC_EX32(r, i) EMIT(PPC_RAW_LI((r), (i) < 0 ? -1 : 0))
-#endif
-
+#ifdef CONFIG_PPC64
#define PPC_LI64(d, i) do { \
if ((long)(i) >= -2147483648 && \
(long)(i) < 2147483648) \
@@ -85,11 +82,6 @@
EMIT(PPC_RAW_ORI(d, d, (uintptr_t)(i) & \
0xffff)); \
} } while (0)
-
-#ifdef CONFIG_PPC64
-#define PPC_FUNC_ADDR(d,i) do { PPC_LI64(d, i); } while(0)
-#else
-#define PPC_FUNC_ADDR(d,i) do { PPC_LI32(d, i); } while(0)
#endif
/*
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index cf66b25ed7c865..063e3a1be9270d 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -36,6 +36,8 @@
/* BPF register usage */
#define TMP_REG (MAX_BPF_JIT_REG + 0)
+#define PPC_EX32(r, i) EMIT(PPC_RAW_LI((r), (i) < 0 ? -1 : 0))
+
/* BPF to ppc register mappings */
const int b2p[MAX_BPF_JIT_REG + 1] = {
/* function return value */
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 12/17] powerpc64/bpf: Get rid of PPC_BPF_[LL|STL|STLU] macros
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
All these macros now have a single user. Expand their usage in place.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit64.h | 22 ----------------------
arch/powerpc/net/bpf_jit_comp64.c | 21 +++++++++++++++------
2 files changed, 15 insertions(+), 28 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit64.h b/arch/powerpc/net/bpf_jit64.h
index 82cdfee412784a..199348b7296653 100644
--- a/arch/powerpc/net/bpf_jit64.h
+++ b/arch/powerpc/net/bpf_jit64.h
@@ -64,28 +64,6 @@ const int b2p[MAX_BPF_JIT_REG + 2] = {
/* PPC NVR range -- update this if we ever use NVRs below r27 */
#define BPF_PPC_NVR_MIN 27
-/*
- * WARNING: These can use TMP_REG_2 if the offset is not at word boundary,
- * so ensure that it isn't in use already.
- */
-#define PPC_BPF_LL(r, base, i) do { \
- if ((i) % 4) { \
- EMIT(PPC_RAW_LI(b2p[TMP_REG_2], (i)));\
- EMIT(PPC_RAW_LDX(r, base, \
- b2p[TMP_REG_2])); \
- } else \
- EMIT(PPC_RAW_LD(r, base, i)); \
- } while(0)
-#define PPC_BPF_STL(r, base, i) do { \
- if ((i) % 4) { \
- EMIT(PPC_RAW_LI(b2p[TMP_REG_2], (i)));\
- EMIT(PPC_RAW_STDX(r, base, \
- b2p[TMP_REG_2])); \
- } else \
- EMIT(PPC_RAW_STD(r, base, i)); \
- } while(0)
-#define PPC_BPF_STLU(r, base, i) do { EMIT(PPC_RAW_STDU(r, base, i)); } while(0)
-
#endif /* !__ASSEMBLY__ */
#endif
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 411ac41dba4293..eeda636cd7be64 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -100,7 +100,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
EMIT(PPC_RAW_STD(0, 1, PPC_LR_STKOFF));
}
- PPC_BPF_STLU(1, 1, -(BPF_PPC_STACKFRAME + ctx->stack_size));
+ EMIT(PPC_RAW_STDU(1, 1, -(BPF_PPC_STACKFRAME + ctx->stack_size)));
}
/*
@@ -726,7 +726,12 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
PPC_LI32(b2p[TMP_REG_1], imm);
src_reg = b2p[TMP_REG_1];
}
- PPC_BPF_STL(src_reg, dst_reg, off);
+ if (off % 4) {
+ EMIT(PPC_RAW_LI(b2p[TMP_REG_2], off));
+ EMIT(PPC_RAW_STDX(src_reg, dst_reg, b2p[TMP_REG_2]));
+ } else {
+ EMIT(PPC_RAW_STD(src_reg, dst_reg, off));
+ }
break;
/*
@@ -802,9 +807,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
PPC_BCC_SHORT(COND_GT, (ctx->idx + 3) * 4);
EMIT(PPC_RAW_LI(dst_reg, 0));
/*
- * Check if 'off' is word aligned because PPC_BPF_LL()
- * (BPF_DW case) generates two instructions if 'off' is not
- * word-aligned and one instruction otherwise.
+ * Check if 'off' is word aligned for BPF_DW, because
+ * we might generate two instructions.
*/
if (BPF_SIZE(code) == BPF_DW && (off & 3))
PPC_JMP((ctx->idx + 3) * 4);
@@ -823,7 +827,12 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
EMIT(PPC_RAW_LWZ(dst_reg, src_reg, off));
break;
case BPF_DW:
- PPC_BPF_LL(dst_reg, src_reg, off);
+ if (off % 4) {
+ EMIT(PPC_RAW_LI(b2p[TMP_REG_1], off));
+ EMIT(PPC_RAW_LDX(dst_reg, src_reg, b2p[TMP_REG_1]));
+ } else {
+ EMIT(PPC_RAW_LD(dst_reg, src_reg, off));
+ }
break;
}
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 03/17] powerpc/bpf: Handle large branch ranges with BPF_EXIT
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
In some scenarios, it is possible that the program epilogue is outside
the branch range for a BPF_EXIT instruction. Instead of rejecting such
programs, emit epilogue as an alternate exit point from the program.
Track the location of the same so that subsequent exits can take either
of the two paths.
Reported-by: Jordan Niethe <jniethe5@gmail.com>
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit.h | 2 ++
arch/powerpc/net/bpf_jit_comp.c | 22 +++++++++++++++++++++-
arch/powerpc/net/bpf_jit_comp32.c | 7 +++++--
arch/powerpc/net/bpf_jit_comp64.c | 7 +++++--
4 files changed, 33 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 9cdd33d6be4cc0..3b5c44c0b6638d 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -151,6 +151,7 @@ struct codegen_context {
unsigned int stack_size;
int b2p[ARRAY_SIZE(b2p)];
unsigned int exentry_idx;
+ unsigned int alt_exit_addr;
};
#ifdef CONFIG_PPC32
@@ -186,6 +187,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
void bpf_jit_realloc_regs(struct codegen_context *ctx);
+int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr);
int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx,
int insn_idx, int jmp_off, int dst_reg);
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 56dd1f4e3e4447..141e64585b6458 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -89,6 +89,22 @@ static int bpf_jit_fixup_addresses(struct bpf_prog *fp, u32 *image,
return 0;
}
+int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr)
+{
+ if (!exit_addr || is_offset_in_branch_range(exit_addr - (ctx->idx * 4))) {
+ PPC_JMP(exit_addr);
+ } else if (ctx->alt_exit_addr) {
+ if (WARN_ON(!is_offset_in_branch_range((long)ctx->alt_exit_addr - (ctx->idx * 4))))
+ return -1;
+ PPC_JMP(ctx->alt_exit_addr);
+ } else {
+ ctx->alt_exit_addr = ctx->idx * 4;
+ bpf_jit_build_epilogue(image, ctx);
+ }
+
+ return 0;
+}
+
struct powerpc64_jit_data {
struct bpf_binary_header *header;
u32 *addrs;
@@ -177,8 +193,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
* If we have seen a tail call, we need a second pass.
* This is because bpf_jit_emit_common_epilogue() is called
* from bpf_jit_emit_tail_call() with a not yet stable ctx->seen.
+ * We also need a second pass if we ended up with too large
+ * a program so as to ensure BPF_EXIT branches are in range.
*/
- if (cgctx.seen & SEEN_TAILCALL) {
+ if (cgctx.seen & SEEN_TAILCALL || !is_offset_in_branch_range((long)cgctx.idx * 4)) {
cgctx.idx = 0;
if (bpf_jit_build_body(fp, 0, &cgctx, addrs, 0)) {
fp = org_fp;
@@ -193,6 +211,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
* calculate total size from idx.
*/
bpf_jit_build_prologue(0, &cgctx);
+ addrs[fp->len] = cgctx.idx * 4;
bpf_jit_build_epilogue(0, &cgctx);
fixup_len = fp->aux->num_exentries * BPF_FIXUP_LEN * 4;
@@ -233,6 +252,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
for (pass = 1; pass < 3; pass++) {
/* Now build the prologue, body code & epilogue for real. */
cgctx.idx = 0;
+ cgctx.alt_exit_addr = 0;
bpf_jit_build_prologue(code_base, &cgctx);
if (bpf_jit_build_body(fp, code_base, &cgctx, addrs, pass)) {
bpf_jit_binary_free(bpf_hdr);
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index 81e0c56661ddf2..f401bfc5a67684 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -929,8 +929,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
* the epilogue. If we _are_ the last instruction,
* we'll just fall through to the epilogue.
*/
- if (i != flen - 1)
- PPC_JMP(exit_addr);
+ if (i != flen - 1) {
+ ret = bpf_jit_emit_exit_insn(image, ctx, _R0, exit_addr);
+ if (ret)
+ return ret;
+ }
/* else fall through to the epilogue */
break;
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index b1ed8611091d2b..371bd5a16859c7 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -871,8 +871,11 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
* the epilogue. If we _are_ the last instruction,
* we'll just fall through to the epilogue.
*/
- if (i != flen - 1)
- PPC_JMP(exit_addr);
+ if (i != flen - 1) {
+ ret = bpf_jit_emit_exit_insn(image, ctx, b2p[TMP_REG_1], exit_addr);
+ if (ret)
+ return ret;
+ }
/* else fall through to the epilogue */
break;
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 11/17] powerpc64/bpf: Convert some of the uses of PPC_BPF_[LL|STL] to PPC_BPF_[LD|STD]
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
PPC_BPF_[LL|STL] are macros meant for scenarios where we may have to
deal with a non-word aligned offset. Limit their usage to only those
scenarios by converting the rest to just use PPC_BPF_[LD|STD].
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit_comp64.c | 22 +++++++++++-----------
1 file changed, 11 insertions(+), 11 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index bff200723e7282..411ac41dba4293 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -74,7 +74,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
int i;
if (__is_defined(PPC64_ELF_ABI_v2))
- PPC_BPF_LL(_R2, _R13, offsetof(struct paca_struct, kernel_toc));
+ EMIT(PPC_RAW_LD(_R2, _R13, offsetof(struct paca_struct, kernel_toc)));
/*
* Initialize tail_call_cnt if we do tail calls.
@@ -84,7 +84,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
if (ctx->seen & SEEN_TAILCALL) {
EMIT(PPC_RAW_LI(b2p[TMP_REG_1], 0));
/* this goes in the redzone */
- PPC_BPF_STL(b2p[TMP_REG_1], 1, -(BPF_PPC_STACK_SAVE + 8));
+ EMIT(PPC_RAW_STD(b2p[TMP_REG_1], 1, -(BPF_PPC_STACK_SAVE + 8)));
} else {
EMIT(PPC_RAW_NOP());
EMIT(PPC_RAW_NOP());
@@ -97,7 +97,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
*/
if (ctx->seen & SEEN_FUNC) {
EMIT(PPC_RAW_MFLR(_R0));
- PPC_BPF_STL(0, 1, PPC_LR_STKOFF);
+ EMIT(PPC_RAW_STD(0, 1, PPC_LR_STKOFF));
}
PPC_BPF_STLU(1, 1, -(BPF_PPC_STACKFRAME + ctx->stack_size));
@@ -110,7 +110,7 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
*/
for (i = BPF_REG_6; i <= BPF_REG_10; i++)
if (bpf_is_seen_register(ctx, b2p[i]))
- PPC_BPF_STL(b2p[i], 1, bpf_jit_stack_offsetof(ctx, b2p[i]));
+ EMIT(PPC_RAW_STD(b2p[i], 1, bpf_jit_stack_offsetof(ctx, b2p[i])));
/* Setup frame pointer to point to the bpf stack area */
if (bpf_is_seen_register(ctx, b2p[BPF_REG_FP]))
@@ -125,13 +125,13 @@ static void bpf_jit_emit_common_epilogue(u32 *image, struct codegen_context *ctx
/* Restore NVRs */
for (i = BPF_REG_6; i <= BPF_REG_10; i++)
if (bpf_is_seen_register(ctx, b2p[i]))
- PPC_BPF_LL(b2p[i], 1, bpf_jit_stack_offsetof(ctx, b2p[i]));
+ EMIT(PPC_RAW_LD(b2p[i], 1, bpf_jit_stack_offsetof(ctx, b2p[i])));
/* Tear down our stack frame */
if (bpf_has_stack_frame(ctx)) {
EMIT(PPC_RAW_ADDI(1, 1, BPF_PPC_STACKFRAME + ctx->stack_size));
if (ctx->seen & SEEN_FUNC) {
- PPC_BPF_LL(0, 1, PPC_LR_STKOFF);
+ EMIT(PPC_RAW_LD(0, 1, PPC_LR_STKOFF));
EMIT(PPC_RAW_MTLR(0));
}
}
@@ -229,7 +229,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
* if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
* goto out;
*/
- PPC_BPF_LL(b2p[TMP_REG_1], 1, bpf_jit_stack_tailcallcnt(ctx));
+ EMIT(PPC_RAW_LD(b2p[TMP_REG_1], 1, bpf_jit_stack_tailcallcnt(ctx)));
EMIT(PPC_RAW_CMPLWI(b2p[TMP_REG_1], MAX_TAIL_CALL_CNT));
PPC_BCC_SHORT(COND_GE, out);
@@ -237,12 +237,12 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
* tail_call_cnt++;
*/
EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], b2p[TMP_REG_1], 1));
- PPC_BPF_STL(b2p[TMP_REG_1], 1, bpf_jit_stack_tailcallcnt(ctx));
+ EMIT(PPC_RAW_STD(b2p[TMP_REG_1], 1, bpf_jit_stack_tailcallcnt(ctx)));
/* prog = array->ptrs[index]; */
EMIT(PPC_RAW_MULI(b2p[TMP_REG_1], b2p_index, 8));
EMIT(PPC_RAW_ADD(b2p[TMP_REG_1], b2p[TMP_REG_1], b2p_bpf_array));
- PPC_BPF_LL(b2p[TMP_REG_1], b2p[TMP_REG_1], offsetof(struct bpf_array, ptrs));
+ EMIT(PPC_RAW_LD(b2p[TMP_REG_1], b2p[TMP_REG_1], offsetof(struct bpf_array, ptrs)));
/*
* if (prog == NULL)
@@ -252,7 +252,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
PPC_BCC_SHORT(COND_EQ, out);
/* goto *(prog->bpf_func + prologue_size); */
- PPC_BPF_LL(b2p[TMP_REG_1], b2p[TMP_REG_1], offsetof(struct bpf_prog, bpf_func));
+ EMIT(PPC_RAW_LD(b2p[TMP_REG_1], b2p[TMP_REG_1], offsetof(struct bpf_prog, bpf_func)));
EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], b2p[TMP_REG_1],
FUNCTION_DESCR_SIZE + bpf_tailcall_prologue_size));
EMIT(PPC_RAW_MTCTR(b2p[TMP_REG_1]));
@@ -628,7 +628,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
break;
case 64:
/* Store the value to stack and then use byte-reverse loads */
- PPC_BPF_STL(dst_reg, 1, bpf_jit_stack_local(ctx));
+ EMIT(PPC_RAW_STD(dst_reg, 1, bpf_jit_stack_local(ctx)));
EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], 1, bpf_jit_stack_local(ctx)));
if (cpu_has_feature(CPU_FTR_ARCH_206)) {
EMIT(PPC_RAW_LDBRX(dst_reg, 0, b2p[TMP_REG_1]));
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 10/17] powerpc/bpf: Rename PPC_BL_ABS() to PPC_BL()
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
PPC_BL_ABS() is just doing a relative branch with link. The name
suggests that it is for branching to an absolute address, which is
incorrect. Rename the macro to a more appropriate PPC_BL().
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit.h | 6 +++---
arch/powerpc/net/bpf_jit_comp32.c | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 5cb3efd76715a9..0832235a274983 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -34,9 +34,9 @@
EMIT(PPC_RAW_BRANCH(offset)); \
} while (0)
-/* blr; (unconditional 'branch' with link) to absolute address */
-#define PPC_BL_ABS(dest) EMIT(PPC_INST_BL | \
- (((dest) - (unsigned long)(image + ctx->idx)) & 0x03fffffc))
+/* bl (unconditional 'branch' with link) */
+#define PPC_BL(dest) EMIT(PPC_INST_BL | (((dest) - (unsigned long)(image + ctx->idx)) & 0x03fffffc))
+
/* "cond" here covers BO:BI fields. */
#define PPC_BCC_SHORT(cond, dest) \
do { \
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index 014cf893ce90d6..cf66b25ed7c865 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -190,7 +190,7 @@ int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func
s32 rel = (s32)func - (s32)(image + ctx->idx);
if (image && rel < 0x2000000 && rel >= -0x2000000) {
- PPC_BL_ABS(func);
+ PPC_BL(func);
EMIT(PPC_RAW_NOP());
EMIT(PPC_RAW_NOP());
EMIT(PPC_RAW_NOP());
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 09/17] powerpc64/bpf: Optimize instruction sequence used for function calls
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
When calling BPF helpers, we load the function address to call into a
register. This can result in upto 5 instructions. Optimize this by
instead using the kernel toc in r2 and adjusting offset to the BPF
helper. This works since all BPF helpers are part of kernel text, and
all BPF programs/functions utilize the kernel TOC.
Further more:
- load the actual function entry address in elf v1, rather than loading
it through the function descriptor address.
- load the Local Entry Point (LEP) in elf v2 skipping TOC setup.
- consolidate code across elf abi v1 and v2 by using r12 on both.
Reported-by: Anton Blanchard <anton@ozlabs.org>
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit_comp64.c | 30 +++++++++++++-----------------
1 file changed, 13 insertions(+), 17 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index e9fd4694226fe0..bff200723e7282 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -150,22 +150,20 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
static int bpf_jit_emit_func_call_hlp(u32 *image, struct codegen_context *ctx, u64 func)
{
unsigned long func_addr = func ? ppc_function_entry((void *)func) : 0;
+ long reladdr;
if (WARN_ON_ONCE(!core_kernel_text(func_addr)))
return -EINVAL;
-#ifdef PPC64_ELF_ABI_v1
- /* func points to the function descriptor */
- PPC_LI64(b2p[TMP_REG_2], func);
- /* Load actual entry point from function descriptor */
- PPC_BPF_LL(b2p[TMP_REG_1], b2p[TMP_REG_2], 0);
- /* ... and move it to CTR */
- EMIT(PPC_RAW_MTCTR(b2p[TMP_REG_1]));
-#else
- /* We can clobber r12 */
- PPC_FUNC_ADDR(12, func);
- EMIT(PPC_RAW_MTCTR(12));
-#endif
+ reladdr = func_addr - kernel_toc_addr();
+ if (reladdr > 0x7FFFFFFF || reladdr < -(0x80000000L)) {
+ pr_err("eBPF: address of %ps out of range of kernel_toc.\n", (void *)func);
+ return -ERANGE;
+ }
+
+ EMIT(PPC_RAW_ADDIS(_R12, _R2, PPC_HA(reladdr)));
+ EMIT(PPC_RAW_ADDI(_R12, _R12, PPC_LO(reladdr)));
+ EMIT(PPC_RAW_MTCTR(_R12));
EMIT(PPC_RAW_BCTRL());
return 0;
@@ -178,6 +176,9 @@ int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func
if (WARN_ON_ONCE(func && is_module_text_address(func)))
return -EINVAL;
+ /* skip past descriptor if elf v1 */
+ func += FUNCTION_DESCR_SIZE;
+
/* Load function address into r12 */
PPC_LI64(12, func);
@@ -194,11 +195,6 @@ int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func
for (i = ctx->idx - ctx_idx; i < 5; i++)
EMIT(PPC_RAW_NOP());
-#ifdef PPC64_ELF_ABI_v1
- /* Load actual entry point from function descriptor */
- PPC_BPF_LL(12, 12, 0);
-#endif
-
EMIT(PPC_RAW_MTCTR(12));
EMIT(PPC_RAW_BCTRL());
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 08/17] powerpc64/bpf elfv1: Do not load TOC before calling functions
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
BPF helpers always reside in core kernel and all BPF programs use the
kernel TOC. As such, there is no need to load the TOC before calling
helpers or other BPF functions. Drop code to do the same.
Add a check to ensure we don't proceed if this assumption ever changes
in future.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit.h | 2 +-
arch/powerpc/net/bpf_jit_comp.c | 4 +++-
arch/powerpc/net/bpf_jit_comp32.c | 8 +++++--
arch/powerpc/net/bpf_jit_comp64.c | 39 ++++++++++++++++---------------
4 files changed, 30 insertions(+), 23 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index 3b5c44c0b6638d..5cb3efd76715a9 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -181,7 +181,7 @@ static inline void bpf_clear_seen_register(struct codegen_context *ctx, int i)
ctx->seen &= ~(1 << (31 - i));
}
-void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func);
+int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func);
int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
u32 *addrs, int pass);
void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 141e64585b6458..635f7448ff7952 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -59,7 +59,9 @@ static int bpf_jit_fixup_addresses(struct bpf_prog *fp, u32 *image,
*/
tmp_idx = ctx->idx;
ctx->idx = addrs[i] / 4;
- bpf_jit_emit_func_call_rel(image, ctx, func_addr);
+ ret = bpf_jit_emit_func_call_rel(image, ctx, func_addr);
+ if (ret)
+ return ret;
/*
* Restore ctx->idx here. This is safe as the length
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index f401bfc5a67684..014cf893ce90d6 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -185,7 +185,7 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
EMIT(PPC_RAW_BLR());
}
-void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func)
+int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func)
{
s32 rel = (s32)func - (s32)(image + ctx->idx);
@@ -201,6 +201,8 @@ void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 fun
EMIT(PPC_RAW_MTCTR(_R0));
EMIT(PPC_RAW_BCTRL());
}
+
+ return 0;
}
static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 out)
@@ -953,7 +955,9 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
EMIT(PPC_RAW_STW(bpf_to_ppc(ctx, BPF_REG_5), _R1, 12));
}
- bpf_jit_emit_func_call_rel(image, ctx, func_addr);
+ ret = bpf_jit_emit_func_call_rel(image, ctx, func_addr);
+ if (ret)
+ return ret;
EMIT(PPC_RAW_MR(bpf_to_ppc(ctx, BPF_REG_0) - 1, _R3));
EMIT(PPC_RAW_MR(bpf_to_ppc(ctx, BPF_REG_0), _R4));
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 44314ee60155e4..e9fd4694226fe0 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -147,9 +147,13 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
EMIT(PPC_RAW_BLR());
}
-static void bpf_jit_emit_func_call_hlp(u32 *image, struct codegen_context *ctx,
- u64 func)
+static int bpf_jit_emit_func_call_hlp(u32 *image, struct codegen_context *ctx, u64 func)
{
+ unsigned long func_addr = func ? ppc_function_entry((void *)func) : 0;
+
+ if (WARN_ON_ONCE(!core_kernel_text(func_addr)))
+ return -EINVAL;
+
#ifdef PPC64_ELF_ABI_v1
/* func points to the function descriptor */
PPC_LI64(b2p[TMP_REG_2], func);
@@ -157,25 +161,23 @@ static void bpf_jit_emit_func_call_hlp(u32 *image, struct codegen_context *ctx,
PPC_BPF_LL(b2p[TMP_REG_1], b2p[TMP_REG_2], 0);
/* ... and move it to CTR */
EMIT(PPC_RAW_MTCTR(b2p[TMP_REG_1]));
- /*
- * Load TOC from function descriptor at offset 8.
- * We can clobber r2 since we get called through a
- * function pointer (so caller will save/restore r2)
- * and since we don't use a TOC ourself.
- */
- PPC_BPF_LL(2, b2p[TMP_REG_2], 8);
#else
/* We can clobber r12 */
PPC_FUNC_ADDR(12, func);
EMIT(PPC_RAW_MTCTR(12));
#endif
EMIT(PPC_RAW_BCTRL());
+
+ return 0;
}
-void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func)
+int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func)
{
unsigned int i, ctx_idx = ctx->idx;
+ if (WARN_ON_ONCE(func && is_module_text_address(func)))
+ return -EINVAL;
+
/* Load function address into r12 */
PPC_LI64(12, func);
@@ -193,19 +195,14 @@ void bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 fun
EMIT(PPC_RAW_NOP());
#ifdef PPC64_ELF_ABI_v1
- /*
- * Load TOC from function descriptor at offset 8.
- * We can clobber r2 since we get called through a
- * function pointer (so caller will save/restore r2)
- * and since we don't use a TOC ourself.
- */
- PPC_BPF_LL(2, 12, 8);
/* Load actual entry point from function descriptor */
PPC_BPF_LL(12, 12, 0);
#endif
EMIT(PPC_RAW_MTCTR(12));
EMIT(PPC_RAW_BCTRL());
+
+ return 0;
}
static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 out)
@@ -890,9 +887,13 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
return ret;
if (func_addr_fixed)
- bpf_jit_emit_func_call_hlp(image, ctx, func_addr);
+ ret = bpf_jit_emit_func_call_hlp(image, ctx, func_addr);
else
- bpf_jit_emit_func_call_rel(image, ctx, func_addr);
+ ret = bpf_jit_emit_func_call_rel(image, ctx, func_addr);
+
+ if (ret)
+ return ret;
+
/* move return value from r3 to BPF_REG_0 */
EMIT(PPC_RAW_MR(b2p[BPF_REG_0], 3));
break;
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 07/17] powerpc64/bpf elfv2: Setup kernel TOC in r2 on entry
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
In preparation for using kernel TOC, load the same in r2 on entry. With
elfv1, the kernel TOC is already setup by our caller.
We adjust the number of instructions to skip on a tail call accordingly.
We get rid of the #ifdef in bpf_jit_emit_tail_call() since
FUNCTION_DESCR_SIZE is itself under a #ifdef.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit_comp64.c | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 27ac2fc7670298..44314ee60155e4 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -73,6 +73,9 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
{
int i;
+ if (__is_defined(PPC64_ELF_ABI_v2))
+ PPC_BPF_LL(_R2, _R13, offsetof(struct paca_struct, kernel_toc));
+
/*
* Initialize tail_call_cnt if we do tail calls.
* Otherwise, put in NOPs so that it can be skipped when we are
@@ -87,8 +90,6 @@ void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx)
EMIT(PPC_RAW_NOP());
}
-#define BPF_TAILCALL_PROLOGUE_SIZE 8
-
if (bpf_has_stack_frame(ctx)) {
/*
* We need a stack frame, but we don't necessarily need to
@@ -217,6 +218,10 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
*/
int b2p_bpf_array = b2p[BPF_REG_2];
int b2p_index = b2p[BPF_REG_3];
+ int bpf_tailcall_prologue_size = 8;
+
+ if (__is_defined(PPC64_ELF_ABI_v2))
+ bpf_tailcall_prologue_size += 4; /* skip past the toc load */
/*
* if (index >= array->map.max_entries)
@@ -255,13 +260,8 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
/* goto *(prog->bpf_func + prologue_size); */
PPC_BPF_LL(b2p[TMP_REG_1], b2p[TMP_REG_1], offsetof(struct bpf_prog, bpf_func));
-#ifdef PPC64_ELF_ABI_v1
- /* skip past the function descriptor */
EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], b2p[TMP_REG_1],
- FUNCTION_DESCR_SIZE + BPF_TAILCALL_PROLOGUE_SIZE));
-#else
- EMIT(PPC_RAW_ADDI(b2p[TMP_REG_1], b2p[TMP_REG_1], BPF_TAILCALL_PROLOGUE_SIZE));
-#endif
+ FUNCTION_DESCR_SIZE + bpf_tailcall_prologue_size));
EMIT(PPC_RAW_MTCTR(b2p[TMP_REG_1]));
/* tear down stack, restore NVRs, ... */
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 06/17] powerpc64: Set PPC64_ELF_ABI_v[1|2] macros to 1
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
Set macros to 1 so that they can be used with __is_defined().
Suggested-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/types.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/asm/types.h b/arch/powerpc/include/asm/types.h
index f1630c553efe43..6c0411ce406255 100644
--- a/arch/powerpc/include/asm/types.h
+++ b/arch/powerpc/include/asm/types.h
@@ -13,9 +13,9 @@
#ifdef __powerpc64__
#if defined(_CALL_ELF) && _CALL_ELF == 2
-#define PPC64_ELF_ABI_v2
+#define PPC64_ELF_ABI_v2 1
#else
-#define PPC64_ELF_ABI_v1
+#define PPC64_ELF_ABI_v1 1
#endif
#endif /* __powerpc64__ */
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 05/17] powerpc64/bpf: Use r12 for constant blinding
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
In preparation for preserving kernel toc in r2, switch BPF_REG_AX from
r2 to r12. r12 is not used by bpf JIT except during external helper/bpf
calls, or with BPF_NOSPEC. These sequences aren't emitted when
BPF_REG_AX is used for constant blinding and other purposes.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit64.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/powerpc/net/bpf_jit64.h b/arch/powerpc/net/bpf_jit64.h
index b63b35e45e558c..82cdfee412784a 100644
--- a/arch/powerpc/net/bpf_jit64.h
+++ b/arch/powerpc/net/bpf_jit64.h
@@ -56,7 +56,7 @@ const int b2p[MAX_BPF_JIT_REG + 2] = {
/* frame pointer aka BPF_REG_10 */
[BPF_REG_FP] = 31,
/* eBPF jit internal registers */
- [BPF_REG_AX] = 2,
+ [BPF_REG_AX] = 12,
[TMP_REG_1] = 9,
[TMP_REG_2] = 10
};
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 04/17] powerpc64/bpf: Do not save/restore LR on each call to bpf_stf_barrier()
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
Instead of saving and restoring LR before each invocation to
bpf_stf_barrier(), set SEEN_FUNC flag so that we save/restore LR in
prologue/epilogue.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit_comp64.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 371bd5a16859c7..27ac2fc7670298 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -690,11 +690,10 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
EMIT(PPC_RAW_ORI(_R31, _R31, 0));
break;
case STF_BARRIER_FALLBACK:
- EMIT(PPC_RAW_MFLR(b2p[TMP_REG_1]));
+ ctx->seen |= SEEN_FUNC;
PPC_LI64(12, dereference_kernel_function_descriptor(bpf_stf_barrier));
EMIT(PPC_RAW_MTCTR(12));
EMIT(PPC_RAW_BCTRL());
- EMIT(PPC_RAW_MTLR(b2p[TMP_REG_1]));
break;
case STF_BARRIER_NONE:
break;
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 02/17] powerpc/bpf: Emit a single branch instruction for known short branch ranges
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
PPC_BCC() emits two instructions to accommodate scenarios where we need
to branch outside the range of a conditional branch. PPC_BCC_SHORT()
emits a single branch instruction and can be used when the branch is
known to be within a conditional branch range.
Convert some of the uses of PPC_BCC() in the powerpc BPF JIT over to
PPC_BCC_SHORT() where we know the branch range.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit_comp32.c | 8 ++++----
arch/powerpc/net/bpf_jit_comp64.c | 8 ++++----
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp32.c b/arch/powerpc/net/bpf_jit_comp32.c
index cf8dd8aea386c4..81e0c56661ddf2 100644
--- a/arch/powerpc/net/bpf_jit_comp32.c
+++ b/arch/powerpc/net/bpf_jit_comp32.c
@@ -221,7 +221,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
EMIT(PPC_RAW_LWZ(_R0, b2p_bpf_array, offsetof(struct bpf_array, map.max_entries)));
EMIT(PPC_RAW_CMPLW(b2p_index, _R0));
EMIT(PPC_RAW_LWZ(_R0, _R1, bpf_jit_stack_offsetof(ctx, BPF_PPC_TC)));
- PPC_BCC(COND_GE, out);
+ PPC_BCC_SHORT(COND_GE, out);
/*
* if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
@@ -230,7 +230,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
EMIT(PPC_RAW_CMPLWI(_R0, MAX_TAIL_CALL_CNT));
/* tail_call_cnt++; */
EMIT(PPC_RAW_ADDIC(_R0, _R0, 1));
- PPC_BCC(COND_GE, out);
+ PPC_BCC_SHORT(COND_GE, out);
/* prog = array->ptrs[index]; */
EMIT(PPC_RAW_RLWINM(_R3, b2p_index, 2, 0, 29));
@@ -243,7 +243,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
* goto out;
*/
EMIT(PPC_RAW_CMPLWI(_R3, 0));
- PPC_BCC(COND_EQ, out);
+ PPC_BCC_SHORT(COND_EQ, out);
/* goto *(prog->bpf_func + prologue_size); */
EMIT(PPC_RAW_LWZ(_R3, _R3, offsetof(struct bpf_prog, bpf_func)));
@@ -834,7 +834,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
if (BPF_MODE(code) == BPF_PROBE_MEM) {
PPC_LI32(_R0, TASK_SIZE - off);
EMIT(PPC_RAW_CMPLW(src_reg, _R0));
- PPC_BCC(COND_GT, (ctx->idx + 5) * 4);
+ PPC_BCC_SHORT(COND_GT, (ctx->idx + 4) * 4);
EMIT(PPC_RAW_LI(dst_reg, 0));
/*
* For BPF_DW case, "li reg_h,0" would be needed when
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index e1e8c934308adb..b1ed8611091d2b 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -225,7 +225,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
EMIT(PPC_RAW_LWZ(b2p[TMP_REG_1], b2p_bpf_array, offsetof(struct bpf_array, map.max_entries)));
EMIT(PPC_RAW_RLWINM(b2p_index, b2p_index, 0, 0, 31));
EMIT(PPC_RAW_CMPLW(b2p_index, b2p[TMP_REG_1]));
- PPC_BCC(COND_GE, out);
+ PPC_BCC_SHORT(COND_GE, out);
/*
* if (tail_call_cnt >= MAX_TAIL_CALL_CNT)
@@ -233,7 +233,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
*/
PPC_BPF_LL(b2p[TMP_REG_1], 1, bpf_jit_stack_tailcallcnt(ctx));
EMIT(PPC_RAW_CMPLWI(b2p[TMP_REG_1], MAX_TAIL_CALL_CNT));
- PPC_BCC(COND_GE, out);
+ PPC_BCC_SHORT(COND_GE, out);
/*
* tail_call_cnt++;
@@ -251,7 +251,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
* goto out;
*/
EMIT(PPC_RAW_CMPLDI(b2p[TMP_REG_1], 0));
- PPC_BCC(COND_EQ, out);
+ PPC_BCC_SHORT(COND_EQ, out);
/* goto *(prog->bpf_func + prologue_size); */
PPC_BPF_LL(b2p[TMP_REG_1], b2p[TMP_REG_1], offsetof(struct bpf_prog, bpf_func));
@@ -807,7 +807,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
else /* BOOK3S_64 */
PPC_LI64(b2p[TMP_REG_2], PAGE_OFFSET);
EMIT(PPC_RAW_CMPLD(b2p[TMP_REG_1], b2p[TMP_REG_2]));
- PPC_BCC(COND_GT, (ctx->idx + 4) * 4);
+ PPC_BCC_SHORT(COND_GT, (ctx->idx + 3) * 4);
EMIT(PPC_RAW_LI(dst_reg, 0));
/*
* Check if 'off' is word aligned because PPC_BPF_LL()
--
2.35.1
^ permalink raw reply related
* [PATCH powerpc/next 01/17] powerpc/bpf: Skip branch range validation during first pass
From: Naveen N. Rao @ 2022-02-14 10:41 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Yauheni Kaliuta, bpf, linuxppc-dev,
Alexei Starovoitov, Hari Bathini
In-Reply-To: <cover.1644834730.git.naveen.n.rao@linux.vnet.ibm.com>
During the first pass, addrs[] is still being populated. So, all
branches to following instructions will appear to be going to the start
of the JIT program. Ignore branch range validation for such instructions
and assume those to be in range. Branch range validation will happen
during the second pass after addrs[] is setup properly.
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
---
arch/powerpc/net/bpf_jit.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit.h b/arch/powerpc/net/bpf_jit.h
index b20a2a83a6e75b..9cdd33d6be4cc0 100644
--- a/arch/powerpc/net/bpf_jit.h
+++ b/arch/powerpc/net/bpf_jit.h
@@ -27,7 +27,7 @@
#define PPC_JMP(dest) \
do { \
long offset = (long)(dest) - (ctx->idx * 4); \
- if (!is_offset_in_branch_range(offset)) { \
+ if ((dest) != 0 && !is_offset_in_branch_range(offset)) { \
pr_err_ratelimited("Branch offset 0x%lx (@%u) out of range\n", offset, ctx->idx); \
return -ERANGE; \
} \
@@ -41,7 +41,7 @@
#define PPC_BCC_SHORT(cond, dest) \
do { \
long offset = (long)(dest) - (ctx->idx * 4); \
- if (!is_offset_in_cond_branch_range(offset)) { \
+ if ((dest) != 0 && !is_offset_in_cond_branch_range(offset)) { \
pr_err_ratelimited("Conditional branch offset 0x%lx (@%u) out of range\n", offset, ctx->idx); \
return -ERANGE; \
} \
--
2.35.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox