* Re: [PATCHv3 bpf-next 07/12] bpf: Add helper to retrieve socket in BPF
From: Alexei Starovoitov @ 2018-09-28 9:46 UTC (permalink / raw)
To: Joe Stringer
Cc: daniel, netdev, ast, john.fastabend, tgraf, kafai, nitin.hande,
mauricio.vasquez
In-Reply-To: <20180927232659.14348-8-joe@wand.net.nz>
On Thu, Sep 27, 2018 at 04:26:54PM -0700, Joe Stringer wrote:
> This patch adds new BPF helper functions, bpf_sk_lookup_tcp() and
> bpf_sk_lookup_udp() which allows BPF programs to find out if there is a
> socket listening on this host, and returns a socket pointer which the
> BPF program can then access to determine, for instance, whether to
> forward or drop traffic. bpf_sk_lookup_xxx() may take a reference on the
> socket, so when a BPF program makes use of this function, it must
> subsequently pass the returned pointer into the newly added sk_release()
> to return the reference.
>
> By way of example, the following pseudocode would filter inbound
> connections at XDP if there is no corresponding service listening for
> the traffic:
>
> struct bpf_sock_tuple tuple;
> struct bpf_sock_ops *sk;
>
> populate_tuple(ctx, &tuple); // Extract the 5tuple from the packet
> sk = bpf_sk_lookup_tcp(ctx, &tuple, sizeof tuple, netns, 0);
> if (!sk) {
> // Couldn't find a socket listening for this traffic. Drop.
> return TC_ACT_SHOT;
> }
> bpf_sk_release(sk);
> return TC_ACT_OK;
>
> Signed-off-by: Joe Stringer <joe@wand.net.nz>
> ---
> v2: Rework 'struct bpf_sock_tuple' to allow passing a packet pointer
> Limit netns_id field to 32 bits
> Fix compile error with CONFIG_IPV6 enabled
> Allow direct packet access from helper
>
> v3: Fix release of caller_net when netns is not specified.
> Use skb->sk to find caller net when skb->dev is unavailable.
> Remove flags argument to sk_release()
> Define the semantics of the new helpers more clearly.
Acked-by: Alexei Starovoitov <ast@kernel.org>
^ permalink raw reply
* Re: [PATCHv3 bpf-next 04/12] bpf: Add PTR_TO_SOCKET verifier type
From: Alexei Starovoitov @ 2018-09-28 9:44 UTC (permalink / raw)
To: Joe Stringer
Cc: daniel, netdev, ast, john.fastabend, tgraf, kafai, nitin.hande,
mauricio.vasquez
In-Reply-To: <20180927232659.14348-5-joe@wand.net.nz>
On Thu, Sep 27, 2018 at 04:26:51PM -0700, Joe Stringer wrote:
> Teach the verifier a little bit about a new type of pointer, a
> PTR_TO_SOCKET. This pointer type is accessed from BPF through the
> 'struct bpf_sock' structure.
>
> Signed-off-by: Joe Stringer <joe@wand.net.nz>
Acked-by: Alexei Starovoitov <ast@kernel.org>
^ permalink raw reply
* Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
From: Ard Biesheuvel @ 2018-09-28 16:01 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: Linux Kernel Mailing List, <netdev@vger.kernel.org>,
open list:HARDWARE RANDOM NUMBER GENERATOR CORE, David S. Miller,
Greg Kroah-Hartman, Samuel Neves, Andy Lutomirski,
Jean-Philippe Aumasson, Russell King, linux-arm-kernel
In-Reply-To: <20180925145622.29959-8-Jason@zx2c4.com>
On 25 September 2018 at 16:56, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> These wire Andy Polyakov's implementations up to the kernel for ARMv7,8
> NEON, and introduce Eric Biggers' ultra-fast scalar implementation for
> CPUs without NEON or for CPUs with slow NEON (Cortex-A5,7).
>
> This commit does the following:
> - Adds the glue code for the assembly implementations.
> - Renames the ARMv8 code into place, since it can at this point be
> used wholesale.
> - Merges Andy Polyakov's ARMv7 NEON code with Eric Biggers' <=ARMv7
> scalar code.
>
> Commit note: Eric Biggers' scalar code is brand new, and quite possibly
> prematurely added to this commit, and so it may require a bit of revision.
>
Please put comments like this below the ---
> This commit delivers approximately the same or much better performance than
> the existing crypto API's code and has been measured to do as such on:
>
> - ARM1176JZF-S [ARMv6]
> - Cortex-A7 [ARMv7]
> - Cortex-A8 [ARMv7]
> - Cortex-A9 [ARMv7]
> - Cortex-A17 [ARMv7]
> - Cortex-A53 [ARMv8]
> - Cortex-A55 [ARMv8]
> - Cortex-A73 [ARMv8]
> - Cortex-A75 [ARMv8]
>
> Interestingly, Andy Polyakov's scalar code is slower than Eric Biggers',
> but is also significantly shorter. This has the advantage that it does
> not evict other code from L1 cache -- particularly on ARM11 chips -- and
> so in certain circumstances it can actually be faster. However, it wasn't
> found that this had an affect on any code existing in the kernel today.
>
> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
> Co-authored-by: Eric Biggers <ebiggers@google.com>
> Cc: Samuel Neves <sneves@dei.uc.pt>
> Cc: Andy Lutomirski <luto@kernel.org>
> Cc: Greg KH <gregkh@linuxfoundation.org>
> Cc: Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
> Cc: Russell King <linux@armlinux.org.uk>
> Cc: linux-arm-kernel@lists.infradead.org
> ---
> lib/zinc/Makefile | 2 +
> lib/zinc/chacha20/chacha20-arm-glue.h | 88 +++
> ...acha20-arm-cryptogams.S => chacha20-arm.S} | 502 ++++++++++++++++--
> ...20-arm64-cryptogams.S => chacha20-arm64.S} | 0
> lib/zinc/chacha20/chacha20.c | 2 +
> 5 files changed, 556 insertions(+), 38 deletions(-)
> create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h
> rename lib/zinc/chacha20/{chacha20-arm-cryptogams.S => chacha20-arm.S} (71%)
> rename lib/zinc/chacha20/{chacha20-arm64-cryptogams.S => chacha20-arm64.S} (100%)
>
> diff --git a/lib/zinc/Makefile b/lib/zinc/Makefile
> index 223a0816c918..e47f64e12bbd 100644
> --- a/lib/zinc/Makefile
> +++ b/lib/zinc/Makefile
> @@ -4,4 +4,6 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
>
> zinc_chacha20-y := chacha20/chacha20.o
> zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o
> +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o
> +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o
Are these CONFIG_ symbols defined anywhere at this point?
In any case, I don't think these is a reason for these, at least not
on ARM/arm64. The 64-bitness is implied in both cases, and the
dependency on !CPU_32v3 you introduce (looking at the version of
Kconfig at the end of the series) seems spurious to me. Was that added
because of some kbuild robot report? (we don't support ARMv3 in the
kernel but ARCH_RPC is built in v3 mode because of historical reasons
while the actual core is a v4)
> obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
> diff --git a/lib/zinc/chacha20/chacha20-arm-glue.h b/lib/zinc/chacha20/chacha20-arm-glue.h
> new file mode 100644
> index 000000000000..86cce851ed02
> --- /dev/null
> +++ b/lib/zinc/chacha20/chacha20-arm-glue.h
> @@ -0,0 +1,88 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
> +/*
> + * Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
> + */
> +
> +#include <asm/hwcap.h>
> +#include <asm/neon.h>
> +#if defined(CONFIG_ARM)
> +#include <asm/system_info.h>
> +#include <asm/cputype.h>
> +#endif
> +
> +asmlinkage void chacha20_arm(u8 *out, const u8 *in, const size_t len,
> + const u32 key[8], const u32 counter[4]);
> +#if defined(CONFIG_ARM)
> +asmlinkage void hchacha20_arm(const u32 state[16], u32 out[8]);
> +#endif
> +#if defined(CONFIG_KERNEL_MODE_NEON)
> +asmlinkage void chacha20_neon(u8 *out, const u8 *in, const size_t len,
> + const u32 key[8], const u32 counter[4]);
> +#endif
> +
No need to make asmlinkage declarations conditional
> +static bool chacha20_use_neon __ro_after_init;
> +
> +static void __init chacha20_fpu_init(void)
> +{
> +#if defined(CONFIG_ARM64)
> + chacha20_use_neon = elf_hwcap & HWCAP_ASIMD;
> +#elif defined(CONFIG_ARM)
> + switch (read_cpuid_part()) {
> + case ARM_CPU_PART_CORTEX_A7:
> + case ARM_CPU_PART_CORTEX_A5:
> + /* The Cortex-A7 and Cortex-A5 do not perform well with the NEON
> + * implementation but do incredibly with the scalar one and use
> + * less power.
> + */
> + break;
> + default:
> + chacha20_use_neon = elf_hwcap & HWCAP_NEON;
> + }
> +#endif
> +}
> +
> +static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
> + const u8 *src, size_t len,
> + simd_context_t *simd_context)
> +{
> +#if defined(CONFIG_KERNEL_MODE_NEON)
if (IS_ENABLED())
> + if (chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
> + simd_use(simd_context))
> + chacha20_neon(dst, src, len, state->key, state->counter);
> + else
> +#endif
> + chacha20_arm(dst, src, len, state->key, state->counter);
> +
> + state->counter[0] += (len + 63) / 64;
> + return true;
> +}
> +
> +static inline bool hchacha20_arch(u32 derived_key[CHACHA20_KEY_WORDS],
> + const u8 nonce[HCHACHA20_NONCE_SIZE],
> + const u8 key[HCHACHA20_KEY_SIZE],
> + simd_context_t *simd_context)
> +{
> +#if defined(CONFIG_ARM)
> + u32 x[] = { CHACHA20_CONSTANT_EXPA,
> + CHACHA20_CONSTANT_ND_3,
> + CHACHA20_CONSTANT_2_BY,
> + CHACHA20_CONSTANT_TE_K,
> + get_unaligned_le32(key + 0),
> + get_unaligned_le32(key + 4),
> + get_unaligned_le32(key + 8),
> + get_unaligned_le32(key + 12),
> + get_unaligned_le32(key + 16),
> + get_unaligned_le32(key + 20),
> + get_unaligned_le32(key + 24),
> + get_unaligned_le32(key + 28),
> + get_unaligned_le32(nonce + 0),
> + get_unaligned_le32(nonce + 4),
> + get_unaligned_le32(nonce + 8),
> + get_unaligned_le32(nonce + 12)
> + };
> + hchacha20_arm(x, derived_key);
"""
if (!IS_ENABLED(CONFIG_ARM))
return false;
hchacha20_arm(x, derived_key);
return true;
"""
and drop the #ifdefs
> + return true;
> +#else
> + return false;
> +#endif
> +}
> diff --git a/lib/zinc/chacha20/chacha20-arm-cryptogams.S b/lib/zinc/chacha20/chacha20-arm.S
> similarity index 71%
> rename from lib/zinc/chacha20/chacha20-arm-cryptogams.S
> rename to lib/zinc/chacha20/chacha20-arm.S
> index 770bab469171..5abedafcf129 100644
> --- a/lib/zinc/chacha20/chacha20-arm-cryptogams.S
> +++ b/lib/zinc/chacha20/chacha20-arm.S
> @@ -1,13 +1,475 @@
> /* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
> /*
> + * Copyright (C) 2018 Google, Inc.
> * Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
> * Copyright (C) 2006-2017 CRYPTOGAMS by <appro@openssl.org>. All Rights Reserved.
> - *
> - * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS.
> */
>
> #include <linux/linkage.h>
>
> +/*
> + * The following scalar routine was written by Eric Biggers.
> + *
> + * Design notes:
> + *
> + * 16 registers would be needed to hold the state matrix, but only 14 are
> + * available because 'sp' and 'pc' cannot be used. So we spill the elements
> + * (x8, x9) to the stack and swap them out with (x10, x11). This adds one
> + * 'ldrd' and one 'strd' instruction per round.
> + *
> + * All rotates are performed using the implicit rotate operand accepted by the
> + * 'add' and 'eor' instructions. This is faster than using explicit rotate
> + * instructions. To make this work, we allow the values in the second and last
> + * rows of the ChaCha state matrix (rows 'b' and 'd') to temporarily have the
> + * wrong rotation amount. The rotation amount is then fixed up just in time
> + * when the values are used. 'brot' is the number of bits the values in row 'b'
> + * need to be rotated right to arrive at the correct values, and 'drot'
> + * similarly for row 'd'. (brot, drot) start out as (0, 0) but we make it such
> + * that they end up as (25, 24) after every round.
> + */
> +
> + // ChaCha state registers
> + X0 .req r0
> + X1 .req r1
> + X2 .req r2
> + X3 .req r3
> + X4 .req r4
> + X5 .req r5
> + X6 .req r6
> + X7 .req r7
> + X8_X10 .req r8 // shared by x8 and x10
> + X9_X11 .req r9 // shared by x9 and x11
> + X12 .req r10
> + X13 .req r11
> + X14 .req r12
> + X15 .req r14
> +
> +.Lexpand_32byte_k:
> + // "expand 32-byte k"
> + .word 0x61707865, 0x3320646e, 0x79622d32, 0x6b206574
> +
> +#ifdef __thumb2__
> +# define adrl adr
> +#endif
> +
> +.macro __rev out, in, t0, t1, t2
> +.if __LINUX_ARM_ARCH__ >= 6
> + rev \out, \in
> +.else
> + lsl \t0, \in, #24
> + and \t1, \in, #0xff00
> + and \t2, \in, #0xff0000
> + orr \out, \t0, \in, lsr #24
> + orr \out, \out, \t1, lsl #8
> + orr \out, \out, \t2, lsr #8
> +.endif
> +.endm
> +
> +.macro _le32_bswap x, t0, t1, t2
> +#ifdef __ARMEB__
> + __rev \x, \x, \t0, \t1, \t2
> +#endif
> +.endm
> +
> +.macro _le32_bswap_4x a, b, c, d, t0, t1, t2
> + _le32_bswap \a, \t0, \t1, \t2
> + _le32_bswap \b, \t0, \t1, \t2
> + _le32_bswap \c, \t0, \t1, \t2
> + _le32_bswap \d, \t0, \t1, \t2
> +.endm
> +
> +.macro __ldrd a, b, src, offset
> +#if __LINUX_ARM_ARCH__ >= 6
> + ldrd \a, \b, [\src, #\offset]
> +#else
> + ldr \a, [\src, #\offset]
> + ldr \b, [\src, #\offset + 4]
> +#endif
> +.endm
> +
> +.macro __strd a, b, dst, offset
> +#if __LINUX_ARM_ARCH__ >= 6
> + strd \a, \b, [\dst, #\offset]
> +#else
> + str \a, [\dst, #\offset]
> + str \b, [\dst, #\offset + 4]
> +#endif
> +.endm
> +
> +.macro _halfround a1, b1, c1, d1, a2, b2, c2, d2
> +
> + // a += b; d ^= a; d = rol(d, 16);
> + add \a1, \a1, \b1, ror #brot
> + add \a2, \a2, \b2, ror #brot
> + eor \d1, \a1, \d1, ror #drot
> + eor \d2, \a2, \d2, ror #drot
> + // drot == 32 - 16 == 16
> +
> + // c += d; b ^= c; b = rol(b, 12);
> + add \c1, \c1, \d1, ror #16
> + add \c2, \c2, \d2, ror #16
> + eor \b1, \c1, \b1, ror #brot
> + eor \b2, \c2, \b2, ror #brot
> + // brot == 32 - 12 == 20
> +
> + // a += b; d ^= a; d = rol(d, 8);
> + add \a1, \a1, \b1, ror #20
> + add \a2, \a2, \b2, ror #20
> + eor \d1, \a1, \d1, ror #16
> + eor \d2, \a2, \d2, ror #16
> + // drot == 32 - 8 == 24
> +
> + // c += d; b ^= c; b = rol(b, 7);
> + add \c1, \c1, \d1, ror #24
> + add \c2, \c2, \d2, ror #24
> + eor \b1, \c1, \b1, ror #20
> + eor \b2, \c2, \b2, ror #20
> + // brot == 32 - 7 == 25
> +.endm
> +
> +.macro _doubleround
> +
> + // column round
> +
> + // quarterrounds: (x0, x4, x8, x12) and (x1, x5, x9, x13)
> + _halfround X0, X4, X8_X10, X12, X1, X5, X9_X11, X13
> +
> + // save (x8, x9); restore (x10, x11)
> + __strd X8_X10, X9_X11, sp, 0
> + __ldrd X8_X10, X9_X11, sp, 8
> +
> + // quarterrounds: (x2, x6, x10, x14) and (x3, x7, x11, x15)
> + _halfround X2, X6, X8_X10, X14, X3, X7, X9_X11, X15
> +
> + .set brot, 25
> + .set drot, 24
> +
> + // diagonal round
> +
> + // quarterrounds: (x0, x5, x10, x15) and (x1, x6, x11, x12)
> + _halfround X0, X5, X8_X10, X15, X1, X6, X9_X11, X12
> +
> + // save (x10, x11); restore (x8, x9)
> + __strd X8_X10, X9_X11, sp, 8
> + __ldrd X8_X10, X9_X11, sp, 0
> +
> + // quarterrounds: (x2, x7, x8, x13) and (x3, x4, x9, x14)
> + _halfround X2, X7, X8_X10, X13, X3, X4, X9_X11, X14
> +.endm
> +
> +.macro _chacha_permute nrounds
> + .set brot, 0
> + .set drot, 0
> + .rept \nrounds / 2
> + _doubleround
> + .endr
> +.endm
> +
> +.macro _chacha nrounds
> +
> +.Lnext_block\@:
> + // Stack: unused0-unused1 x10-x11 x0-x15 OUT IN LEN
> + // Registers contain x0-x9,x12-x15.
> +
> + // Do the core ChaCha permutation to update x0-x15.
> + _chacha_permute \nrounds
> +
> + add sp, #8
> + // Stack: x10-x11 orig_x0-orig_x15 OUT IN LEN
> + // Registers contain x0-x9,x12-x15.
> + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
> +
> + // Free up some registers (r8-r12,r14) by pushing (x8-x9,x12-x15).
> + push {X8_X10, X9_X11, X12, X13, X14, X15}
> +
> + // Load (OUT, IN, LEN).
> + ldr r14, [sp, #96]
> + ldr r12, [sp, #100]
> + ldr r11, [sp, #104]
> +
> + orr r10, r14, r12
> +
> + // Use slow path if fewer than 64 bytes remain.
> + cmp r11, #64
> + blt .Lxor_slowpath\@
> +
> + // Use slow path if IN and/or OUT isn't 4-byte aligned. Needed even on
> + // ARMv6+, since ldmia and stmia (used below) still require alignment.
> + tst r10, #3
> + bne .Lxor_slowpath\@
> +
> + // Fast path: XOR 64 bytes of aligned data.
> +
> + // Stack: x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN
> + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is OUT.
> + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
> +
> + // x0-x3
> + __ldrd r8, r9, sp, 32
> + __ldrd r10, r11, sp, 40
> + add X0, X0, r8
> + add X1, X1, r9
> + add X2, X2, r10
> + add X3, X3, r11
> + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10
> + ldmia r12!, {r8-r11}
> + eor X0, X0, r8
> + eor X1, X1, r9
> + eor X2, X2, r10
> + eor X3, X3, r11
> + stmia r14!, {X0-X3}
> +
> + // x4-x7
> + __ldrd r8, r9, sp, 48
> + __ldrd r10, r11, sp, 56
> + add X4, r8, X4, ror #brot
> + add X5, r9, X5, ror #brot
> + ldmia r12!, {X0-X3}
> + add X6, r10, X6, ror #brot
> + add X7, r11, X7, ror #brot
> + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10
> + eor X4, X4, X0
> + eor X5, X5, X1
> + eor X6, X6, X2
> + eor X7, X7, X3
> + stmia r14!, {X4-X7}
> +
> + // x8-x15
> + pop {r0-r7} // (x8-x9,x12-x15,x10-x11)
> + __ldrd r8, r9, sp, 32
> + __ldrd r10, r11, sp, 40
> + add r0, r0, r8 // x8
> + add r1, r1, r9 // x9
> + add r6, r6, r10 // x10
> + add r7, r7, r11 // x11
> + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10
> + ldmia r12!, {r8-r11}
> + eor r0, r0, r8 // x8
> + eor r1, r1, r9 // x9
> + eor r6, r6, r10 // x10
> + eor r7, r7, r11 // x11
> + stmia r14!, {r0,r1,r6,r7}
> + ldmia r12!, {r0,r1,r6,r7}
> + __ldrd r8, r9, sp, 48
> + __ldrd r10, r11, sp, 56
> + add r2, r8, r2, ror #drot // x12
> + add r3, r9, r3, ror #drot // x13
> + add r4, r10, r4, ror #drot // x14
> + add r5, r11, r5, ror #drot // x15
> + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11
> + ldr r9, [sp, #72] // load LEN
> + eor r2, r2, r0 // x12
> + eor r3, r3, r1 // x13
> + eor r4, r4, r6 // x14
> + eor r5, r5, r7 // x15
> + subs r9, #64 // decrement and check LEN
> + stmia r14!, {r2-r5}
> +
> + beq .Ldone\@
> +
> +.Lprepare_for_next_block\@:
> +
> + // Stack: x0-x15 OUT IN LEN
> +
> + // Increment block counter (x12)
> + add r8, #1
> +
> + // Store updated (OUT, IN, LEN)
> + str r14, [sp, #64]
> + str r12, [sp, #68]
> + str r9, [sp, #72]
> +
> + mov r14, sp
> +
> + // Store updated block counter (x12)
> + str r8, [sp, #48]
> +
> + sub sp, #16
> +
> + // Reload state and do next block
> + ldmia r14!, {r0-r11} // load x0-x11
> + __strd r10, r11, sp, 8 // store x10-x11 before state
> + ldmia r14, {r10-r12,r14} // load x12-x15
> + b .Lnext_block\@
> +
> +.Lxor_slowpath\@:
> + // Slow path: < 64 bytes remaining, or unaligned input or output buffer.
> + // We handle it by storing the 64 bytes of keystream to the stack, then
> + // XOR-ing the needed portion with the data.
> +
> + // Allocate keystream buffer
> + sub sp, #64
> + mov r14, sp
> +
> + // Stack: ks0-ks15 x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN
> + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is &ks0.
> + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
> +
> + // Save keystream for x0-x3
> + __ldrd r8, r9, sp, 96
> + __ldrd r10, r11, sp, 104
> + add X0, X0, r8
> + add X1, X1, r9
> + add X2, X2, r10
> + add X3, X3, r11
> + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10
> + stmia r14!, {X0-X3}
> +
> + // Save keystream for x4-x7
> + __ldrd r8, r9, sp, 112
> + __ldrd r10, r11, sp, 120
> + add X4, r8, X4, ror #brot
> + add X5, r9, X5, ror #brot
> + add X6, r10, X6, ror #brot
> + add X7, r11, X7, ror #brot
> + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10
> + add r8, sp, #64
> + stmia r14!, {X4-X7}
> +
> + // Save keystream for x8-x15
> + ldm r8, {r0-r7} // (x8-x9,x12-x15,x10-x11)
> + __ldrd r8, r9, sp, 128
> + __ldrd r10, r11, sp, 136
> + add r0, r0, r8 // x8
> + add r1, r1, r9 // x9
> + add r6, r6, r10 // x10
> + add r7, r7, r11 // x11
> + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10
> + stmia r14!, {r0,r1,r6,r7}
> + __ldrd r8, r9, sp, 144
> + __ldrd r10, r11, sp, 152
> + add r2, r8, r2, ror #drot // x12
> + add r3, r9, r3, ror #drot // x13
> + add r4, r10, r4, ror #drot // x14
> + add r5, r11, r5, ror #drot // x15
> + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11
> + stmia r14, {r2-r5}
> +
> + // Stack: ks0-ks15 unused0-unused7 x0-x15 OUT IN LEN
> + // Registers: r8 is block counter, r12 is IN.
> +
> + ldr r9, [sp, #168] // LEN
> + ldr r14, [sp, #160] // OUT
> + cmp r9, #64
> + mov r0, sp
> + movle r1, r9
> + movgt r1, #64
> + // r1 is number of bytes to XOR, in range [1, 64]
> +
> +.if __LINUX_ARM_ARCH__ < 6
> + orr r2, r12, r14
> + tst r2, #3 // IN or OUT misaligned?
> + bne .Lxor_next_byte\@
> +.endif
> +
> + // XOR a word at a time
> +.rept 16
> + subs r1, #4
> + blt .Lxor_words_done\@
> + ldr r2, [r12], #4
> + ldr r3, [r0], #4
> + eor r2, r2, r3
> + str r2, [r14], #4
> +.endr
> + b .Lxor_slowpath_done\@
> +.Lxor_words_done\@:
> + ands r1, r1, #3
> + beq .Lxor_slowpath_done\@
> +
> + // XOR a byte at a time
> +.Lxor_next_byte\@:
> + ldrb r2, [r12], #1
> + ldrb r3, [r0], #1
> + eor r2, r2, r3
> + strb r2, [r14], #1
> + subs r1, #1
> + bne .Lxor_next_byte\@
> +
> +.Lxor_slowpath_done\@:
> + subs r9, #64
> + add sp, #96
> + bgt .Lprepare_for_next_block\@
> +
> +.Ldone\@:
> +.endm // _chacha
> +
> +/*
> + * void chacha20_arm(u8 *out, const u8 *in, size_t len, const u32 key[8],
> + * const u32 iv[4]);
> + */
> +ENTRY(chacha20_arm)
> + cmp r2, #0 // len == 0?
> + bxeq lr
> +
> + push {r0-r2,r4-r11,lr}
> +
> + // Push state x0-x15 onto stack.
> + // Also store an extra copy of x10-x11 just before the state.
> +
> + ldr r4, [sp, #48] // iv
> + mov r0, sp
> + sub sp, #80
> +
> + // iv: x12-x15
> + ldm r4, {X12,X13,X14,X15}
> + stmdb r0!, {X12,X13,X14,X15}
> +
> + // key: x4-x11
> + __ldrd X8_X10, X9_X11, r3, 24
> + __strd X8_X10, X9_X11, sp, 8
> + stmdb r0!, {X8_X10, X9_X11}
> + ldm r3, {X4-X9_X11}
> + stmdb r0!, {X4-X9_X11}
> +
> + // constants: x0-x3
> + adrl X3, .Lexpand_32byte_k
> + ldm X3, {X0-X3}
> + __strd X0, X1, sp, 16
> + __strd X2, X3, sp, 24
> +
> + _chacha 20
> +
> + add sp, #76
> + pop {r4-r11, pc}
> +ENDPROC(chacha20_arm)
> +
> +/*
> + * void hchacha20_arm(const u32 state[16], u32 out[8]);
> + */
> +ENTRY(hchacha20_arm)
> + push {r1,r4-r11,lr}
> +
> + mov r14, r0
> + ldmia r14!, {r0-r11} // load x0-x11
> + push {r10-r11} // store x10-x11 to stack
> + ldm r14, {r10-r12,r14} // load x12-x15
> + sub sp, #8
> +
> + _chacha_permute 20
> +
> + // Skip over (unused0-unused1, x10-x11)
> + add sp, #16
> +
> + // Fix up rotations of x12-x15
> + ror X12, X12, #drot
> + ror X13, X13, #drot
> + pop {r4} // load 'out'
> + ror X14, X14, #drot
> + ror X15, X15, #drot
> +
> + // Store (x0-x3,x12-x15) to 'out'
> + stm r4, {X0,X1,X2,X3,X12,X13,X14,X15}
> +
> + pop {r4-r11,pc}
> +ENDPROC(hchacha20_arm)
> +
> +#ifdef CONFIG_KERNEL_MODE_NEON
> +/*
> + * This following NEON routine was ported from Andy Polyakov's implementation
> + * from CRYPTOGAMS. It begins with parts of the CRYPTOGAMS scalar routine,
> + * since certain NEON code paths actually branch to it.
> + */
> +
> .text
> #if defined(__thumb2__) || defined(__clang__)
> .syntax unified
> @@ -22,39 +484,6 @@
> #define ldrhsb ldrbhs
> #endif
>
> -.align 5
> -.Lsigma:
> -.long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral
> -.Lone:
> -.long 1,0,0,0
> -.word -1
> -
> -.align 5
> -ENTRY(chacha20_arm)
> - ldr r12,[sp,#0] @ pull pointer to counter and nonce
> - stmdb sp!,{r0-r2,r4-r11,lr}
> - cmp r2,#0 @ len==0?
> -#ifdef __thumb2__
> - itt eq
> -#endif
> - addeq sp,sp,#4*3
> - beq .Lno_data_arm
> - ldmia r12,{r4-r7} @ load counter and nonce
> - sub sp,sp,#4*(16) @ off-load area
> -#if __LINUX_ARM_ARCH__ < 7 && !defined(__thumb2__)
> - sub r14,pc,#100 @ .Lsigma
> -#else
> - adr r14,.Lsigma @ .Lsigma
> -#endif
> - stmdb sp!,{r4-r7} @ copy counter and nonce
> - ldmia r3,{r4-r11} @ load key
> - ldmia r14,{r0-r3} @ load sigma
> - stmdb sp!,{r4-r11} @ copy key
> - stmdb sp!,{r0-r3} @ copy sigma
> - str r10,[sp,#4*(16+10)] @ off-load "rx"
> - str r11,[sp,#4*(16+11)] @ off-load "rx"
> - b .Loop_outer_enter
> -
> .align 4
> .Loop_outer:
> ldmia sp,{r0-r9} @ load key material
> @@ -748,11 +1177,8 @@ ENTRY(chacha20_arm)
>
> .Ldone:
> add sp,sp,#4*(32+3)
> -.Lno_data_arm:
> ldmia sp!,{r4-r11,pc}
> -ENDPROC(chacha20_arm)
>
> -#ifdef CONFIG_KERNEL_MODE_NEON
> .align 5
> .Lsigma2:
> .long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral
> diff --git a/lib/zinc/chacha20/chacha20-arm64-cryptogams.S b/lib/zinc/chacha20/chacha20-arm64.S
> similarity index 100%
> rename from lib/zinc/chacha20/chacha20-arm64-cryptogams.S
> rename to lib/zinc/chacha20/chacha20-arm64.S
> diff --git a/lib/zinc/chacha20/chacha20.c b/lib/zinc/chacha20/chacha20.c
> index 4354b874a6a5..fc4f74fca653 100644
> --- a/lib/zinc/chacha20/chacha20.c
> +++ b/lib/zinc/chacha20/chacha20.c
> @@ -16,6 +16,8 @@
>
> #if defined(CONFIG_ZINC_ARCH_X86_64)
> #include "chacha20-x86_64-glue.h"
> +#elif defined(CONFIG_ZINC_ARCH_ARM) || defined(CONFIG_ZINC_ARCH_ARM64)
As above, just use CONFIG_ARM / CONFIG_ARM64 directly
> +#include "chacha20-arm-glue.h"
> #else
> void __init chacha20_fpu_init(void)
> {
> --
> 2.19.0
>
^ permalink raw reply
* Re: [PATCH 00/15] octeontx2-af: Add RVU Admin Function driver
From: Sunil Kovvuri @ 2018-09-28 9:35 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Linux Netdev List, David S. Miller, linux-soc, Sunil Goutham
In-Reply-To: <CAK8P3a2051NXGnuOyOGuXTovxQ9UfiZPrgEwJLP4PrXO-9DBPw@mail.gmail.com>
On Fri, Sep 28, 2018 at 1:57 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Fri, Sep 28, 2018 at 8:08 AM <sunil.kovvuri@gmail.com> wrote:
> >
> > From: Sunil Goutham <sgoutham@marvell.com>
> >
> > Resource virtualization unit (RVU) on Marvell's OcteonTX2 SOC maps HW
> > resources from the network, crypto and other functional blocks into
> > PCI-compatible physical and virtual functions. Each functional block
> > again has multiple local functions (LFs) for provisioning to PCI devices.
> > RVU supports multiple PCIe SRIOV physical functions (PFs) and virtual
> > functions (VFs). PF0 is called the administrative / admin function (AF)
> > and has privileges to provision RVU functional block's LFs to each of the
> > PF/VF.
> >
> > RVU managed networking functional blocks
> > - Network pool allocator (NPA)
> > - Network interface controller (NIX)
> > - Network parser CAM (NPC)
> > - Schedule/Synchronize/Order unit (SSO)
> >
> > RVU managed non-networking functional blocks
> > - Crypto accelerator (CPT)
> > - Scheduled timers unit (TIM)
> > - Schedule/Synchronize/Order unit (SSO)
> > Used for both networking and non networking usecases
> > - Compression (upcoming in future variants of the silicons)
> >
> > Resource provisioning examples
> > - A PF/VF with NIX-LF & NPA-LF resources works as a pure network device
> > - A PF/VF with CPT-LF resource works as a pure cyrpto offload device.
> >
> > This admin function driver neither receives any data nor processes it i.e
> > no I/O, a configuration only driver.
> >
> > PF/VFs communicates with AF via a shared memory region (mailbox). Upon
> > receiving requests from PF/VF, AF does resource provisioning and other
> > HW configuration. AF is always attached to host, but PF/VFs may be used
> > by host kernel itself, or attached to VMs or to userspace applications
> > like DPDK etc. So AF has to handle provisioning/configuration requests
> > sent by any device from any domain.
> >
> > This patch series adds logic for the following
> > - RVU AF driver with functional blocks provisioning support.
> > - Mailbox infrastructure for communication between AF and PFs.
> > - CGX (MAC controller) driver which communicates with firmware for
> > managing physical ethernet interfaces. AF collects info from this
> > driver and forwards the same to the PF/VFs uaing these interfaces.
> >
> > This is the first set of patches out of 80+ patches.
>
> Hi Sunil,
>
> This looks good overall, thanks for moving it to drivers/net
> as I suggested. I found two more minor remaining issues here,
> it should not be a problem to change the code accordingly.
>
> One thing I was missing here is a revision history for the patch
> series, it would have been helpful to get a list of things you have
> changed compared to the previous submission, what triggered
> those changes, and which issues (if any) are still remaining.
> Best add the full revision history in the next (hopefully final)
> round for these patches, and use 'git format-patch --reroll-count'
> to add the patchset revision number.
Not much changes done to the patches since the v3 patchset i submitted
on drivers/soc. But yes, i will add the old revision history while re-submitting
patches after addressing your suggestions.
>
> I assume that the next set of patches will be more controversial
> as you get to the point that adds the user interface. I have
> not looked at those patches yet, so it may just be fine, assuming
> all the configuration is done using devlink. Please keep me
> on Cc for the future series, even if I don't participate in those
> reviews as much.
Sure I will keep you in CC.
>
> Thanks,
>
> Arnd
^ permalink raw reply
* Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations
From: Jason A. Donenfeld @ 2018-09-28 15:57 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: LKML, Netdev, Linux Crypto Mailing List, David Miller,
Greg Kroah-Hartman, Samuel Neves, Andrew Lutomirski,
Jean-Philippe Aumasson, Andy Polyakov, Russell King - ARM Linux,
linux-arm-kernel
In-Reply-To: <CAKv+Gu8-EwxFhQSUPxjEvTA5ZPz34RieMokM6CUqwURDr74jtg@mail.gmail.com>
Hi Ard,
On Fri, Sep 28, 2018 at 5:49 PM Ard Biesheuvel
<ard.biesheuvel@linaro.org> wrote:
> As I mentioned before, I'd prefer this to be based on the original .pl
> but if I am the only one objecting to this, I guess I can live with
> it.
We're working on that, actually. It's not obvious when it'll be ready
to ship -- perhaps after the initial merge, but perhaps way sooner --
but that is something we're trying to do for arm/arm64/mips64.
Jason
^ permalink raw reply
* Re: [PATCH 13/15] octeontx2-af: Add support for CGX link management
From: Sunil Kovvuri @ 2018-09-28 9:31 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Linux Netdev List, David S. Miller, linux-soc, Linu Cherian,
Nithya Mani
In-Reply-To: <CAK8P3a36irQQXSZSACiNDi04AzA2Fb+ZUkWj10mR7Xxu+b0DbA@mail.gmail.com>
On Fri, Sep 28, 2018 at 1:49 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Fri, Sep 28, 2018 at 8:09 AM <sunil.kovvuri@gmail.com> wrote:
> >
> > +/* scratchx(0) CSR used for ATF->non-secure SW communication.
> > + * This acts as the status register
> > + * Provides details on command ack/status, link status, error details
> > + */
> > +struct cgx_evt_sts {
> > + uint64_t ack:1;
> > + uint64_t evt_type:1; /* cgx_evt_type */
> > + uint64_t stat:1; /* cgx_stat */
> > + uint64_t id:6; /* cgx_evt_id/cgx_cmd_id */
> > + uint64_t reserved:55;
> > +};
> > +
> > +/* Response to command IDs with command status as CGX_STAT_FAIL
> > + *
> > + * Not applicable for commands :
> > + * CGX_CMD_LINK_BRING_UP/DOWN/CGX_EVT_LINK_CHANGE
> > + * check struct cgx_lnk_sts comments
> > + */
> > +struct cgx_err_sts_s {
> > + uint64_t reserved1:9;
> > + uint64_t type:10; /* cgx_error_type */
> > + uint64_t reserved2:35;
> > +};
> > +
> > +/* Response to cmd ID as CGX_CMD_GET_FW_VER with cmd status as
> > + * CGX_STAT_SUCCESS
> > + */
> > +struct cgx_ver_s {
> > + uint64_t reserved1:9;
> > + uint64_t major_ver:4;
> > + uint64_t minor_ver:4;
> > + uint64_t reserved2:47;
> > +};
>
> From what I can tell, you pass these structures to the device, so they
> are a binary interface. I don't think you can rely on bitfields to work
> correctly here, they are generally not portable and I wouldn't rely
> on them doing the right thing if you build a big-endian kernel.
>
> It's better to use bitmasks on a u64 value instead to make it obviously
> portable.
>
> Arnd
This communication between firmware and kernel driver is done using couple of
scratch registers. With limited space available we had to resort to bitfields.
Your point about endianness is correct. As you might be aware that the device to
which this driver registers to, is only found on OcteonTx2 SOC which operates
in a standalone mode. As of now we are not targeting to make these drivers
work in big-endian mode.
We would prefer to make big-endian related changes later on, test them
fully and
submit patches, would this be okay ?
If not we will define big endian bit fields in all command structures
and re-submit.
Sunil.
^ permalink raw reply
* Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations
From: Ard Biesheuvel @ 2018-09-28 15:51 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: Linux Kernel Mailing List, <netdev@vger.kernel.org>,
open list:HARDWARE RANDOM NUMBER GENERATOR CORE, David S. Miller,
Greg Kroah-Hartman, Samuel Neves, Andy Lutomirski,
Jean-Philippe Aumasson, Andy Polyakov, Russell King,
linux-arm-kernel
In-Reply-To: <CAKv+Gu8-EwxFhQSUPxjEvTA5ZPz34RieMokM6CUqwURDr74jtg@mail.gmail.com>
On 28 September 2018 at 17:49, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> On 25 September 2018 at 16:56, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>> These NEON and non-NEON implementations come from Andy Polyakov's
>> implementation, and are included here in raw form without modification,
>> so that subsequent commits that fix these up for the kernel can see how
>> it has changed. This awkward commit splitting has been requested for the
>> ARM[64] implementations in particular.
>>
>> While this is CRYPTOGAMS code, the originating code for this happens to
>> be the same as OpenSSL's commit 87cc649f30aaf69b351701875b9dac07c29ce8a2
>>
>> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
>> Based-on-code-from: Andy Polyakov <appro@openssl.org>
>> Cc: Samuel Neves <sneves@dei.uc.pt>
>> Cc: Andy Lutomirski <luto@kernel.org>
>> Cc: Greg KH <gregkh@linuxfoundation.org>
>> Cc: Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
>> Cc: Andy Polyakov <appro@openssl.org>
>> Cc: Russell King <linux@armlinux.org.uk>
>> Cc: linux-arm-kernel@lists.infradead.org
>
> As I mentioned before, I'd prefer this to be based on the original .pl
> but if I am the only one objecting to this, I guess I can live with
> it.
>
Note that I am getting bounces from LAKML because the patch is too big.
^ permalink raw reply
* Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations
From: Ard Biesheuvel @ 2018-09-28 15:49 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: Linux Kernel Mailing List, <netdev@vger.kernel.org>,
open list:HARDWARE RANDOM NUMBER GENERATOR CORE, David S. Miller,
Greg Kroah-Hartman, Samuel Neves, Andy Lutomirski,
Jean-Philippe Aumasson, Andy Polyakov, Russell King,
linux-arm-kernel
In-Reply-To: <20180925145622.29959-6-Jason@zx2c4.com>
On 25 September 2018 at 16:56, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> These NEON and non-NEON implementations come from Andy Polyakov's
> implementation, and are included here in raw form without modification,
> so that subsequent commits that fix these up for the kernel can see how
> it has changed. This awkward commit splitting has been requested for the
> ARM[64] implementations in particular.
>
> While this is CRYPTOGAMS code, the originating code for this happens to
> be the same as OpenSSL's commit 87cc649f30aaf69b351701875b9dac07c29ce8a2
>
> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
> Based-on-code-from: Andy Polyakov <appro@openssl.org>
> Cc: Samuel Neves <sneves@dei.uc.pt>
> Cc: Andy Lutomirski <luto@kernel.org>
> Cc: Greg KH <gregkh@linuxfoundation.org>
> Cc: Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
> Cc: Andy Polyakov <appro@openssl.org>
> Cc: Russell King <linux@armlinux.org.uk>
> Cc: linux-arm-kernel@lists.infradead.org
As I mentioned before, I'd prefer this to be based on the original .pl
but if I am the only one objecting to this, I guess I can live with
it.
> ---
> lib/zinc/chacha20/chacha20-arm-cryptogams.S | 1440 ++++++++++++
> lib/zinc/chacha20/chacha20-arm64-cryptogams.S | 1973 +++++++++++++++++
> 2 files changed, 3413 insertions(+)
> create mode 100644 lib/zinc/chacha20/chacha20-arm-cryptogams.S
> create mode 100644 lib/zinc/chacha20/chacha20-arm64-cryptogams.S
>
> diff --git a/lib/zinc/chacha20/chacha20-arm-cryptogams.S b/lib/zinc/chacha20/chacha20-arm-cryptogams.S
> new file mode 100644
> index 000000000000..05a3a9e6e93f
> --- /dev/null
> +++ b/lib/zinc/chacha20/chacha20-arm-cryptogams.S
> @@ -0,0 +1,1440 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
> +/*
> + * Copyright (C) 2006-2017 CRYPTOGAMS by <appro@openssl.org>. All Rights Reserved.
> + */
> +
> +#include "arm_arch.h"
> +
> +.text
> +#if defined(__thumb2__) || defined(__clang__)
> +.syntax unified
> +#endif
> +#if defined(__thumb2__)
> +.thumb
> +#else
> +.code 32
> +#endif
> +
> +#if defined(__thumb2__) || defined(__clang__)
> +#define ldrhsb ldrbhs
> +#endif
> +
> +.align 5
> +.Lsigma:
> +.long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral
> +.Lone:
> +.long 1,0,0,0
> +.Lrot8:
> +.long 0x02010003,0x06050407
> +#if __ARM_MAX_ARCH__>=7
> +.LOPENSSL_armcap:
> +.word OPENSSL_armcap_P-.LChaCha20_ctr32
> +#else
> +.word -1
> +#endif
> +
> +.globl ChaCha20_ctr32
> +.type ChaCha20_ctr32,%function
> +.align 5
> +ChaCha20_ctr32:
> +.LChaCha20_ctr32:
> + ldr r12,[sp,#0] @ pull pointer to counter and nonce
> + stmdb sp!,{r0-r2,r4-r11,lr}
> +#if __ARM_ARCH__<7 && !defined(__thumb2__)
> + sub r14,pc,#16 @ ChaCha20_ctr32
> +#else
> + adr r14,.LChaCha20_ctr32
> +#endif
> + cmp r2,#0 @ len==0?
> +#ifdef __thumb2__
> + itt eq
> +#endif
> + addeq sp,sp,#4*3
> + beq .Lno_data
> +#if __ARM_MAX_ARCH__>=7
> + cmp r2,#192 @ test len
> + bls .Lshort
> + ldr r4,[r14,#-24]
> + ldr r4,[r14,r4]
> +# ifdef __APPLE__
> + ldr r4,[r4]
> +# endif
> + tst r4,#ARMV7_NEON
> + bne .LChaCha20_neon
> +.Lshort:
> +#endif
> + ldmia r12,{r4-r7} @ load counter and nonce
> + sub sp,sp,#4*(16) @ off-load area
> + sub r14,r14,#64 @ .Lsigma
> + stmdb sp!,{r4-r7} @ copy counter and nonce
> + ldmia r3,{r4-r11} @ load key
> + ldmia r14,{r0-r3} @ load sigma
> + stmdb sp!,{r4-r11} @ copy key
> + stmdb sp!,{r0-r3} @ copy sigma
> + str r10,[sp,#4*(16+10)] @ off-load "rx"
> + str r11,[sp,#4*(16+11)] @ off-load "rx"
> + b .Loop_outer_enter
> +
> +.align 4
> +.Loop_outer:
> + ldmia sp,{r0-r9} @ load key material
> + str r11,[sp,#4*(32+2)] @ save len
> + str r12, [sp,#4*(32+1)] @ save inp
> + str r14, [sp,#4*(32+0)] @ save out
> +.Loop_outer_enter:
> + ldr r11, [sp,#4*(15)]
> + mov r4,r4,ror#19 @ twist b[0..3]
> + ldr r12,[sp,#4*(12)] @ modulo-scheduled load
> + mov r5,r5,ror#19
> + ldr r10, [sp,#4*(13)]
> + mov r6,r6,ror#19
> + ldr r14,[sp,#4*(14)]
> + mov r7,r7,ror#19
> + mov r11,r11,ror#8 @ twist d[0..3]
> + mov r12,r12,ror#8
> + mov r10,r10,ror#8
> + mov r14,r14,ror#8
> + str r11, [sp,#4*(16+15)]
> + mov r11,#10
> + b .Loop
> +
> +.align 4
> +.Loop:
> + subs r11,r11,#1
> + add r0,r0,r4,ror#13
> + add r1,r1,r5,ror#13
> + eor r12,r0,r12,ror#24
> + eor r10,r1,r10,ror#24
> + add r8,r8,r12,ror#16
> + add r9,r9,r10,ror#16
> + eor r4,r8,r4,ror#13
> + eor r5,r9,r5,ror#13
> + add r0,r0,r4,ror#20
> + add r1,r1,r5,ror#20
> + eor r12,r0,r12,ror#16
> + eor r10,r1,r10,ror#16
> + add r8,r8,r12,ror#24
> + str r10,[sp,#4*(16+13)]
> + add r9,r9,r10,ror#24
> + ldr r10,[sp,#4*(16+15)]
> + str r8,[sp,#4*(16+8)]
> + eor r4,r4,r8,ror#12
> + str r9,[sp,#4*(16+9)]
> + eor r5,r5,r9,ror#12
> + ldr r8,[sp,#4*(16+10)]
> + add r2,r2,r6,ror#13
> + ldr r9,[sp,#4*(16+11)]
> + add r3,r3,r7,ror#13
> + eor r14,r2,r14,ror#24
> + eor r10,r3,r10,ror#24
> + add r8,r8,r14,ror#16
> + add r9,r9,r10,ror#16
> + eor r6,r8,r6,ror#13
> + eor r7,r9,r7,ror#13
> + add r2,r2,r6,ror#20
> + add r3,r3,r7,ror#20
> + eor r14,r2,r14,ror#16
> + eor r10,r3,r10,ror#16
> + add r8,r8,r14,ror#24
> + add r9,r9,r10,ror#24
> + eor r6,r6,r8,ror#12
> + eor r7,r7,r9,ror#12
> + add r0,r0,r5,ror#13
> + add r1,r1,r6,ror#13
> + eor r10,r0,r10,ror#24
> + eor r12,r1,r12,ror#24
> + add r8,r8,r10,ror#16
> + add r9,r9,r12,ror#16
> + eor r5,r8,r5,ror#13
> + eor r6,r9,r6,ror#13
> + add r0,r0,r5,ror#20
> + add r1,r1,r6,ror#20
> + eor r10,r0,r10,ror#16
> + eor r12,r1,r12,ror#16
> + str r10,[sp,#4*(16+15)]
> + add r8,r8,r10,ror#24
> + ldr r10,[sp,#4*(16+13)]
> + add r9,r9,r12,ror#24
> + str r8,[sp,#4*(16+10)]
> + eor r5,r5,r8,ror#12
> + str r9,[sp,#4*(16+11)]
> + eor r6,r6,r9,ror#12
> + ldr r8,[sp,#4*(16+8)]
> + add r2,r2,r7,ror#13
> + ldr r9,[sp,#4*(16+9)]
> + add r3,r3,r4,ror#13
> + eor r10,r2,r10,ror#24
> + eor r14,r3,r14,ror#24
> + add r8,r8,r10,ror#16
> + add r9,r9,r14,ror#16
> + eor r7,r8,r7,ror#13
> + eor r4,r9,r4,ror#13
> + add r2,r2,r7,ror#20
> + add r3,r3,r4,ror#20
> + eor r10,r2,r10,ror#16
> + eor r14,r3,r14,ror#16
> + add r8,r8,r10,ror#24
> + add r9,r9,r14,ror#24
> + eor r7,r7,r8,ror#12
> + eor r4,r4,r9,ror#12
> + bne .Loop
> +
> + ldr r11,[sp,#4*(32+2)] @ load len
> +
> + str r8, [sp,#4*(16+8)] @ modulo-scheduled store
> + str r9, [sp,#4*(16+9)]
> + str r12,[sp,#4*(16+12)]
> + str r10, [sp,#4*(16+13)]
> + str r14,[sp,#4*(16+14)]
> +
> + @ at this point we have first half of 512-bit result in
> + @ rx and second half at sp+4*(16+8)
> +
> + cmp r11,#64 @ done yet?
> +#ifdef __thumb2__
> + itete lo
> +#endif
> + addlo r12,sp,#4*(0) @ shortcut or ...
> + ldrhs r12,[sp,#4*(32+1)] @ ... load inp
> + addlo r14,sp,#4*(0) @ shortcut or ...
> + ldrhs r14,[sp,#4*(32+0)] @ ... load out
> +
> + ldr r8,[sp,#4*(0)] @ load key material
> + ldr r9,[sp,#4*(1)]
> +
> +#if __ARM_ARCH__>=6 || !defined(__ARMEB__)
> +# if __ARM_ARCH__<7
> + orr r10,r12,r14
> + tst r10,#3 @ are input and output aligned?
> + ldr r10,[sp,#4*(2)]
> + bne .Lunaligned
> + cmp r11,#64 @ restore flags
> +# else
> + ldr r10,[sp,#4*(2)]
> +# endif
> + ldr r11,[sp,#4*(3)]
> +
> + add r0,r0,r8 @ accumulate key material
> + add r1,r1,r9
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r8,[r12],#16 @ load input
> + ldrhs r9,[r12,#-12]
> +
> + add r2,r2,r10
> + add r3,r3,r11
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r10,[r12,#-8]
> + ldrhs r11,[r12,#-4]
> +# if __ARM_ARCH__>=6 && defined(__ARMEB__)
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> +# endif
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r0,r0,r8 @ xor with input
> + eorhs r1,r1,r9
> + add r8,sp,#4*(4)
> + str r0,[r14],#16 @ store output
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r2,r2,r10
> + eorhs r3,r3,r11
> + ldmia r8,{r8-r11} @ load key material
> + str r1,[r14,#-12]
> + str r2,[r14,#-8]
> + str r3,[r14,#-4]
> +
> + add r4,r8,r4,ror#13 @ accumulate key material
> + add r5,r9,r5,ror#13
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r8,[r12],#16 @ load input
> + ldrhs r9,[r12,#-12]
> + add r6,r10,r6,ror#13
> + add r7,r11,r7,ror#13
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r10,[r12,#-8]
> + ldrhs r11,[r12,#-4]
> +# if __ARM_ARCH__>=6 && defined(__ARMEB__)
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r4,r4,r8
> + eorhs r5,r5,r9
> + add r8,sp,#4*(8)
> + str r4,[r14],#16 @ store output
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r6,r6,r10
> + eorhs r7,r7,r11
> + str r5,[r14,#-12]
> + ldmia r8,{r8-r11} @ load key material
> + str r6,[r14,#-8]
> + add r0,sp,#4*(16+8)
> + str r7,[r14,#-4]
> +
> + ldmia r0,{r0-r7} @ load second half
> +
> + add r0,r0,r8 @ accumulate key material
> + add r1,r1,r9
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r8,[r12],#16 @ load input
> + ldrhs r9,[r12,#-12]
> +# ifdef __thumb2__
> + itt hi
> +# endif
> + strhi r10,[sp,#4*(16+10)] @ copy "rx" while at it
> + strhi r11,[sp,#4*(16+11)] @ copy "rx" while at it
> + add r2,r2,r10
> + add r3,r3,r11
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r10,[r12,#-8]
> + ldrhs r11,[r12,#-4]
> +# if __ARM_ARCH__>=6 && defined(__ARMEB__)
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> +# endif
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r0,r0,r8
> + eorhs r1,r1,r9
> + add r8,sp,#4*(12)
> + str r0,[r14],#16 @ store output
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r2,r2,r10
> + eorhs r3,r3,r11
> + str r1,[r14,#-12]
> + ldmia r8,{r8-r11} @ load key material
> + str r2,[r14,#-8]
> + str r3,[r14,#-4]
> +
> + add r4,r8,r4,ror#24 @ accumulate key material
> + add r5,r9,r5,ror#24
> +# ifdef __thumb2__
> + itt hi
> +# endif
> + addhi r8,r8,#1 @ next counter value
> + strhi r8,[sp,#4*(12)] @ save next counter value
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r8,[r12],#16 @ load input
> + ldrhs r9,[r12,#-12]
> + add r6,r10,r6,ror#24
> + add r7,r11,r7,ror#24
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r10,[r12,#-8]
> + ldrhs r11,[r12,#-4]
> +# if __ARM_ARCH__>=6 && defined(__ARMEB__)
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r4,r4,r8
> + eorhs r5,r5,r9
> +# ifdef __thumb2__
> + it ne
> +# endif
> + ldrne r8,[sp,#4*(32+2)] @ re-load len
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r6,r6,r10
> + eorhs r7,r7,r11
> + str r4,[r14],#16 @ store output
> + str r5,[r14,#-12]
> +# ifdef __thumb2__
> + it hs
> +# endif
> + subhs r11,r8,#64 @ len-=64
> + str r6,[r14,#-8]
> + str r7,[r14,#-4]
> + bhi .Loop_outer
> +
> + beq .Ldone
> +# if __ARM_ARCH__<7
> + b .Ltail
> +
> +.align 4
> +.Lunaligned: @ unaligned endian-neutral path
> + cmp r11,#64 @ restore flags
> +# endif
> +#endif
> +#if __ARM_ARCH__<7
> + ldr r11,[sp,#4*(3)]
> + add r0,r8,r0 @ accumulate key material
> + add r1,r9,r1
> + add r2,r10,r2
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r8,r8,r8 @ zero or ...
> + ldrhsb r8,[r12],#16 @ ... load input
> + eorlo r9,r9,r9
> + ldrhsb r9,[r12,#-12]
> +
> + add r3,r11,r3
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r10,r10,r10
> + ldrhsb r10,[r12,#-8]
> + eorlo r11,r11,r11
> + ldrhsb r11,[r12,#-4]
> +
> + eor r0,r8,r0 @ xor with input (or zero)
> + eor r1,r9,r1
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-15] @ load more input
> + ldrhsb r9,[r12,#-11]
> + eor r2,r10,r2
> + strb r0,[r14],#16 @ store output
> + eor r3,r11,r3
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-7]
> + ldrhsb r11,[r12,#-3]
> + strb r1,[r14,#-12]
> + eor r0,r8,r0,lsr#8
> + strb r2,[r14,#-8]
> + eor r1,r9,r1,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-14] @ load more input
> + ldrhsb r9,[r12,#-10]
> + strb r3,[r14,#-4]
> + eor r2,r10,r2,lsr#8
> + strb r0,[r14,#-15]
> + eor r3,r11,r3,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-6]
> + ldrhsb r11,[r12,#-2]
> + strb r1,[r14,#-11]
> + eor r0,r8,r0,lsr#8
> + strb r2,[r14,#-7]
> + eor r1,r9,r1,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-13] @ load more input
> + ldrhsb r9,[r12,#-9]
> + strb r3,[r14,#-3]
> + eor r2,r10,r2,lsr#8
> + strb r0,[r14,#-14]
> + eor r3,r11,r3,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-5]
> + ldrhsb r11,[r12,#-1]
> + strb r1,[r14,#-10]
> + strb r2,[r14,#-6]
> + eor r0,r8,r0,lsr#8
> + strb r3,[r14,#-2]
> + eor r1,r9,r1,lsr#8
> + strb r0,[r14,#-13]
> + eor r2,r10,r2,lsr#8
> + strb r1,[r14,#-9]
> + eor r3,r11,r3,lsr#8
> + strb r2,[r14,#-5]
> + strb r3,[r14,#-1]
> + add r8,sp,#4*(4+0)
> + ldmia r8,{r8-r11} @ load key material
> + add r0,sp,#4*(16+8)
> + add r4,r8,r4,ror#13 @ accumulate key material
> + add r5,r9,r5,ror#13
> + add r6,r10,r6,ror#13
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r8,r8,r8 @ zero or ...
> + ldrhsb r8,[r12],#16 @ ... load input
> + eorlo r9,r9,r9
> + ldrhsb r9,[r12,#-12]
> +
> + add r7,r11,r7,ror#13
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r10,r10,r10
> + ldrhsb r10,[r12,#-8]
> + eorlo r11,r11,r11
> + ldrhsb r11,[r12,#-4]
> +
> + eor r4,r8,r4 @ xor with input (or zero)
> + eor r5,r9,r5
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-15] @ load more input
> + ldrhsb r9,[r12,#-11]
> + eor r6,r10,r6
> + strb r4,[r14],#16 @ store output
> + eor r7,r11,r7
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-7]
> + ldrhsb r11,[r12,#-3]
> + strb r5,[r14,#-12]
> + eor r4,r8,r4,lsr#8
> + strb r6,[r14,#-8]
> + eor r5,r9,r5,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-14] @ load more input
> + ldrhsb r9,[r12,#-10]
> + strb r7,[r14,#-4]
> + eor r6,r10,r6,lsr#8
> + strb r4,[r14,#-15]
> + eor r7,r11,r7,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-6]
> + ldrhsb r11,[r12,#-2]
> + strb r5,[r14,#-11]
> + eor r4,r8,r4,lsr#8
> + strb r6,[r14,#-7]
> + eor r5,r9,r5,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-13] @ load more input
> + ldrhsb r9,[r12,#-9]
> + strb r7,[r14,#-3]
> + eor r6,r10,r6,lsr#8
> + strb r4,[r14,#-14]
> + eor r7,r11,r7,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-5]
> + ldrhsb r11,[r12,#-1]
> + strb r5,[r14,#-10]
> + strb r6,[r14,#-6]
> + eor r4,r8,r4,lsr#8
> + strb r7,[r14,#-2]
> + eor r5,r9,r5,lsr#8
> + strb r4,[r14,#-13]
> + eor r6,r10,r6,lsr#8
> + strb r5,[r14,#-9]
> + eor r7,r11,r7,lsr#8
> + strb r6,[r14,#-5]
> + strb r7,[r14,#-1]
> + add r8,sp,#4*(4+4)
> + ldmia r8,{r8-r11} @ load key material
> + ldmia r0,{r0-r7} @ load second half
> +# ifdef __thumb2__
> + itt hi
> +# endif
> + strhi r10,[sp,#4*(16+10)] @ copy "rx"
> + strhi r11,[sp,#4*(16+11)] @ copy "rx"
> + add r0,r8,r0 @ accumulate key material
> + add r1,r9,r1
> + add r2,r10,r2
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r8,r8,r8 @ zero or ...
> + ldrhsb r8,[r12],#16 @ ... load input
> + eorlo r9,r9,r9
> + ldrhsb r9,[r12,#-12]
> +
> + add r3,r11,r3
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r10,r10,r10
> + ldrhsb r10,[r12,#-8]
> + eorlo r11,r11,r11
> + ldrhsb r11,[r12,#-4]
> +
> + eor r0,r8,r0 @ xor with input (or zero)
> + eor r1,r9,r1
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-15] @ load more input
> + ldrhsb r9,[r12,#-11]
> + eor r2,r10,r2
> + strb r0,[r14],#16 @ store output
> + eor r3,r11,r3
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-7]
> + ldrhsb r11,[r12,#-3]
> + strb r1,[r14,#-12]
> + eor r0,r8,r0,lsr#8
> + strb r2,[r14,#-8]
> + eor r1,r9,r1,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-14] @ load more input
> + ldrhsb r9,[r12,#-10]
> + strb r3,[r14,#-4]
> + eor r2,r10,r2,lsr#8
> + strb r0,[r14,#-15]
> + eor r3,r11,r3,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-6]
> + ldrhsb r11,[r12,#-2]
> + strb r1,[r14,#-11]
> + eor r0,r8,r0,lsr#8
> + strb r2,[r14,#-7]
> + eor r1,r9,r1,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-13] @ load more input
> + ldrhsb r9,[r12,#-9]
> + strb r3,[r14,#-3]
> + eor r2,r10,r2,lsr#8
> + strb r0,[r14,#-14]
> + eor r3,r11,r3,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-5]
> + ldrhsb r11,[r12,#-1]
> + strb r1,[r14,#-10]
> + strb r2,[r14,#-6]
> + eor r0,r8,r0,lsr#8
> + strb r3,[r14,#-2]
> + eor r1,r9,r1,lsr#8
> + strb r0,[r14,#-13]
> + eor r2,r10,r2,lsr#8
> + strb r1,[r14,#-9]
> + eor r3,r11,r3,lsr#8
> + strb r2,[r14,#-5]
> + strb r3,[r14,#-1]
> + add r8,sp,#4*(4+8)
> + ldmia r8,{r8-r11} @ load key material
> + add r4,r8,r4,ror#24 @ accumulate key material
> +# ifdef __thumb2__
> + itt hi
> +# endif
> + addhi r8,r8,#1 @ next counter value
> + strhi r8,[sp,#4*(12)] @ save next counter value
> + add r5,r9,r5,ror#24
> + add r6,r10,r6,ror#24
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r8,r8,r8 @ zero or ...
> + ldrhsb r8,[r12],#16 @ ... load input
> + eorlo r9,r9,r9
> + ldrhsb r9,[r12,#-12]
> +
> + add r7,r11,r7,ror#24
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r10,r10,r10
> + ldrhsb r10,[r12,#-8]
> + eorlo r11,r11,r11
> + ldrhsb r11,[r12,#-4]
> +
> + eor r4,r8,r4 @ xor with input (or zero)
> + eor r5,r9,r5
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-15] @ load more input
> + ldrhsb r9,[r12,#-11]
> + eor r6,r10,r6
> + strb r4,[r14],#16 @ store output
> + eor r7,r11,r7
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-7]
> + ldrhsb r11,[r12,#-3]
> + strb r5,[r14,#-12]
> + eor r4,r8,r4,lsr#8
> + strb r6,[r14,#-8]
> + eor r5,r9,r5,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-14] @ load more input
> + ldrhsb r9,[r12,#-10]
> + strb r7,[r14,#-4]
> + eor r6,r10,r6,lsr#8
> + strb r4,[r14,#-15]
> + eor r7,r11,r7,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-6]
> + ldrhsb r11,[r12,#-2]
> + strb r5,[r14,#-11]
> + eor r4,r8,r4,lsr#8
> + strb r6,[r14,#-7]
> + eor r5,r9,r5,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-13] @ load more input
> + ldrhsb r9,[r12,#-9]
> + strb r7,[r14,#-3]
> + eor r6,r10,r6,lsr#8
> + strb r4,[r14,#-14]
> + eor r7,r11,r7,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-5]
> + ldrhsb r11,[r12,#-1]
> + strb r5,[r14,#-10]
> + strb r6,[r14,#-6]
> + eor r4,r8,r4,lsr#8
> + strb r7,[r14,#-2]
> + eor r5,r9,r5,lsr#8
> + strb r4,[r14,#-13]
> + eor r6,r10,r6,lsr#8
> + strb r5,[r14,#-9]
> + eor r7,r11,r7,lsr#8
> + strb r6,[r14,#-5]
> + strb r7,[r14,#-1]
> +# ifdef __thumb2__
> + it ne
> +# endif
> + ldrne r8,[sp,#4*(32+2)] @ re-load len
> +# ifdef __thumb2__
> + it hs
> +# endif
> + subhs r11,r8,#64 @ len-=64
> + bhi .Loop_outer
> +
> + beq .Ldone
> +#endif
> +
> +.Ltail:
> + ldr r12,[sp,#4*(32+1)] @ load inp
> + add r9,sp,#4*(0)
> + ldr r14,[sp,#4*(32+0)] @ load out
> +
> +.Loop_tail:
> + ldrb r10,[r9],#1 @ read buffer on stack
> + ldrb r11,[r12],#1 @ read input
> + subs r8,r8,#1
> + eor r11,r11,r10
> + strb r11,[r14],#1 @ store output
> + bne .Loop_tail
> +
> +.Ldone:
> + add sp,sp,#4*(32+3)
> +.Lno_data:
> + ldmia sp!,{r4-r11,pc}
> +.size ChaCha20_ctr32,.-ChaCha20_ctr32
> +#if __ARM_MAX_ARCH__>=7
> +.arch armv7-a
> +.fpu neon
> +
> +.type ChaCha20_neon,%function
> +.align 5
> +ChaCha20_neon:
> + ldr r12,[sp,#0] @ pull pointer to counter and nonce
> + stmdb sp!,{r0-r2,r4-r11,lr}
> +.LChaCha20_neon:
> + adr r14,.Lsigma
> + vstmdb sp!,{d8-d15} @ ABI spec says so
> + stmdb sp!,{r0-r3}
> +
> + vld1.32 {q1-q2},[r3] @ load key
> + ldmia r3,{r4-r11} @ load key
> +
> + sub sp,sp,#4*(16+16)
> + vld1.32 {q3},[r12] @ load counter and nonce
> + add r12,sp,#4*8
> + ldmia r14,{r0-r3} @ load sigma
> + vld1.32 {q0},[r14]! @ load sigma
> + vld1.32 {q12},[r14]! @ one
> + @ vld1.32 {d30},[r14] @ rot8
> + vst1.32 {q2-q3},[r12] @ copy 1/2key|counter|nonce
> + vst1.32 {q0-q1},[sp] @ copy sigma|1/2key
> +
> + str r10,[sp,#4*(16+10)] @ off-load "rx"
> + str r11,[sp,#4*(16+11)] @ off-load "rx"
> + vshl.i32 d26,d24,#1 @ two
> + vstr d24,[sp,#4*(16+0)]
> + vshl.i32 d28,d24,#2 @ four
> + vstr d26,[sp,#4*(16+2)]
> + vmov q4,q0
> + vstr d28,[sp,#4*(16+4)]
> + vmov q8,q0
> + @ vstr d30,[sp,#4*(16+6)]
> + vmov q5,q1
> + vmov q9,q1
> + b .Loop_neon_enter
> +
> +.align 4
> +.Loop_neon_outer:
> + ldmia sp,{r0-r9} @ load key material
> + cmp r11,#64*2 @ if len<=64*2
> + bls .Lbreak_neon @ switch to integer-only
> + @ vldr d30,[sp,#4*(16+6)] @ rot8
> + vmov q4,q0
> + str r11,[sp,#4*(32+2)] @ save len
> + vmov q8,q0
> + str r12, [sp,#4*(32+1)] @ save inp
> + vmov q5,q1
> + str r14, [sp,#4*(32+0)] @ save out
> + vmov q9,q1
> +.Loop_neon_enter:
> + ldr r11, [sp,#4*(15)]
> + mov r4,r4,ror#19 @ twist b[0..3]
> + vadd.i32 q7,q3,q12 @ counter+1
> + ldr r12,[sp,#4*(12)] @ modulo-scheduled load
> + mov r5,r5,ror#19
> + vmov q6,q2
> + ldr r10, [sp,#4*(13)]
> + mov r6,r6,ror#19
> + vmov q10,q2
> + ldr r14,[sp,#4*(14)]
> + mov r7,r7,ror#19
> + vadd.i32 q11,q7,q12 @ counter+2
> + add r12,r12,#3 @ counter+3
> + mov r11,r11,ror#8 @ twist d[0..3]
> + mov r12,r12,ror#8
> + mov r10,r10,ror#8
> + mov r14,r14,ror#8
> + str r11, [sp,#4*(16+15)]
> + mov r11,#10
> + b .Loop_neon
> +
> +.align 4
> +.Loop_neon:
> + subs r11,r11,#1
> + vadd.i32 q0,q0,q1
> + add r0,r0,r4,ror#13
> + vadd.i32 q4,q4,q5
> + add r1,r1,r5,ror#13
> + vadd.i32 q8,q8,q9
> + eor r12,r0,r12,ror#24
> + veor q3,q3,q0
> + eor r10,r1,r10,ror#24
> + veor q7,q7,q4
> + add r8,r8,r12,ror#16
> + veor q11,q11,q8
> + add r9,r9,r10,ror#16
> + vrev32.16 q3,q3
> + eor r4,r8,r4,ror#13
> + vrev32.16 q7,q7
> + eor r5,r9,r5,ror#13
> + vrev32.16 q11,q11
> + add r0,r0,r4,ror#20
> + vadd.i32 q2,q2,q3
> + add r1,r1,r5,ror#20
> + vadd.i32 q6,q6,q7
> + eor r12,r0,r12,ror#16
> + vadd.i32 q10,q10,q11
> + eor r10,r1,r10,ror#16
> + veor q12,q1,q2
> + add r8,r8,r12,ror#24
> + veor q13,q5,q6
> + str r10,[sp,#4*(16+13)]
> + veor q14,q9,q10
> + add r9,r9,r10,ror#24
> + vshr.u32 q1,q12,#20
> + ldr r10,[sp,#4*(16+15)]
> + vshr.u32 q5,q13,#20
> + str r8,[sp,#4*(16+8)]
> + vshr.u32 q9,q14,#20
> + eor r4,r4,r8,ror#12
> + vsli.32 q1,q12,#12
> + str r9,[sp,#4*(16+9)]
> + vsli.32 q5,q13,#12
> + eor r5,r5,r9,ror#12
> + vsli.32 q9,q14,#12
> + ldr r8,[sp,#4*(16+10)]
> + vadd.i32 q0,q0,q1
> + add r2,r2,r6,ror#13
> + vadd.i32 q4,q4,q5
> + ldr r9,[sp,#4*(16+11)]
> + vadd.i32 q8,q8,q9
> + add r3,r3,r7,ror#13
> + veor q12,q3,q0
> + eor r14,r2,r14,ror#24
> + veor q13,q7,q4
> + eor r10,r3,r10,ror#24
> + veor q14,q11,q8
> + add r8,r8,r14,ror#16
> + vshr.u32 q3,q12,#24
> + add r9,r9,r10,ror#16
> + vshr.u32 q7,q13,#24
> + eor r6,r8,r6,ror#13
> + vshr.u32 q11,q14,#24
> + eor r7,r9,r7,ror#13
> + vsli.32 q3,q12,#8
> + add r2,r2,r6,ror#20
> + vsli.32 q7,q13,#8
> + add r3,r3,r7,ror#20
> + vsli.32 q11,q14,#8
> + eor r14,r2,r14,ror#16
> + vadd.i32 q2,q2,q3
> + eor r10,r3,r10,ror#16
> + vadd.i32 q6,q6,q7
> + add r8,r8,r14,ror#24
> + vadd.i32 q10,q10,q11
> + add r9,r9,r10,ror#24
> + veor q12,q1,q2
> + eor r6,r6,r8,ror#12
> + veor q13,q5,q6
> + eor r7,r7,r9,ror#12
> + veor q14,q9,q10
> + vshr.u32 q1,q12,#25
> + vshr.u32 q5,q13,#25
> + vshr.u32 q9,q14,#25
> + vsli.32 q1,q12,#7
> + vsli.32 q5,q13,#7
> + vsli.32 q9,q14,#7
> + vext.8 q2,q2,q2,#8
> + vext.8 q6,q6,q6,#8
> + vext.8 q10,q10,q10,#8
> + vext.8 q1,q1,q1,#4
> + vext.8 q5,q5,q5,#4
> + vext.8 q9,q9,q9,#4
> + vext.8 q3,q3,q3,#12
> + vext.8 q7,q7,q7,#12
> + vext.8 q11,q11,q11,#12
> + vadd.i32 q0,q0,q1
> + add r0,r0,r5,ror#13
> + vadd.i32 q4,q4,q5
> + add r1,r1,r6,ror#13
> + vadd.i32 q8,q8,q9
> + eor r10,r0,r10,ror#24
> + veor q3,q3,q0
> + eor r12,r1,r12,ror#24
> + veor q7,q7,q4
> + add r8,r8,r10,ror#16
> + veor q11,q11,q8
> + add r9,r9,r12,ror#16
> + vrev32.16 q3,q3
> + eor r5,r8,r5,ror#13
> + vrev32.16 q7,q7
> + eor r6,r9,r6,ror#13
> + vrev32.16 q11,q11
> + add r0,r0,r5,ror#20
> + vadd.i32 q2,q2,q3
> + add r1,r1,r6,ror#20
> + vadd.i32 q6,q6,q7
> + eor r10,r0,r10,ror#16
> + vadd.i32 q10,q10,q11
> + eor r12,r1,r12,ror#16
> + veor q12,q1,q2
> + str r10,[sp,#4*(16+15)]
> + veor q13,q5,q6
> + add r8,r8,r10,ror#24
> + veor q14,q9,q10
> + ldr r10,[sp,#4*(16+13)]
> + vshr.u32 q1,q12,#20
> + add r9,r9,r12,ror#24
> + vshr.u32 q5,q13,#20
> + str r8,[sp,#4*(16+10)]
> + vshr.u32 q9,q14,#20
> + eor r5,r5,r8,ror#12
> + vsli.32 q1,q12,#12
> + str r9,[sp,#4*(16+11)]
> + vsli.32 q5,q13,#12
> + eor r6,r6,r9,ror#12
> + vsli.32 q9,q14,#12
> + ldr r8,[sp,#4*(16+8)]
> + vadd.i32 q0,q0,q1
> + add r2,r2,r7,ror#13
> + vadd.i32 q4,q4,q5
> + ldr r9,[sp,#4*(16+9)]
> + vadd.i32 q8,q8,q9
> + add r3,r3,r4,ror#13
> + veor q12,q3,q0
> + eor r10,r2,r10,ror#24
> + veor q13,q7,q4
> + eor r14,r3,r14,ror#24
> + veor q14,q11,q8
> + add r8,r8,r10,ror#16
> + vshr.u32 q3,q12,#24
> + add r9,r9,r14,ror#16
> + vshr.u32 q7,q13,#24
> + eor r7,r8,r7,ror#13
> + vshr.u32 q11,q14,#24
> + eor r4,r9,r4,ror#13
> + vsli.32 q3,q12,#8
> + add r2,r2,r7,ror#20
> + vsli.32 q7,q13,#8
> + add r3,r3,r4,ror#20
> + vsli.32 q11,q14,#8
> + eor r10,r2,r10,ror#16
> + vadd.i32 q2,q2,q3
> + eor r14,r3,r14,ror#16
> + vadd.i32 q6,q6,q7
> + add r8,r8,r10,ror#24
> + vadd.i32 q10,q10,q11
> + add r9,r9,r14,ror#24
> + veor q12,q1,q2
> + eor r7,r7,r8,ror#12
> + veor q13,q5,q6
> + eor r4,r4,r9,ror#12
> + veor q14,q9,q10
> + vshr.u32 q1,q12,#25
> + vshr.u32 q5,q13,#25
> + vshr.u32 q9,q14,#25
> + vsli.32 q1,q12,#7
> + vsli.32 q5,q13,#7
> + vsli.32 q9,q14,#7
> + vext.8 q2,q2,q2,#8
> + vext.8 q6,q6,q6,#8
> + vext.8 q10,q10,q10,#8
> + vext.8 q1,q1,q1,#12
> + vext.8 q5,q5,q5,#12
> + vext.8 q9,q9,q9,#12
> + vext.8 q3,q3,q3,#4
> + vext.8 q7,q7,q7,#4
> + vext.8 q11,q11,q11,#4
> + bne .Loop_neon
> +
> + add r11,sp,#32
> + vld1.32 {q12-q13},[sp] @ load key material
> + vld1.32 {q14-q15},[r11]
> +
> + ldr r11,[sp,#4*(32+2)] @ load len
> +
> + str r8, [sp,#4*(16+8)] @ modulo-scheduled store
> + str r9, [sp,#4*(16+9)]
> + str r12,[sp,#4*(16+12)]
> + str r10, [sp,#4*(16+13)]
> + str r14,[sp,#4*(16+14)]
> +
> + @ at this point we have first half of 512-bit result in
> + @ rx and second half at sp+4*(16+8)
> +
> + ldr r12,[sp,#4*(32+1)] @ load inp
> + ldr r14,[sp,#4*(32+0)] @ load out
> +
> + vadd.i32 q0,q0,q12 @ accumulate key material
> + vadd.i32 q4,q4,q12
> + vadd.i32 q8,q8,q12
> + vldr d24,[sp,#4*(16+0)] @ one
> +
> + vadd.i32 q1,q1,q13
> + vadd.i32 q5,q5,q13
> + vadd.i32 q9,q9,q13
> + vldr d26,[sp,#4*(16+2)] @ two
> +
> + vadd.i32 q2,q2,q14
> + vadd.i32 q6,q6,q14
> + vadd.i32 q10,q10,q14
> + vadd.i32 d14,d14,d24 @ counter+1
> + vadd.i32 d22,d22,d26 @ counter+2
> +
> + vadd.i32 q3,q3,q15
> + vadd.i32 q7,q7,q15
> + vadd.i32 q11,q11,q15
> +
> + cmp r11,#64*4
> + blo .Ltail_neon
> +
> + vld1.8 {q12-q13},[r12]! @ load input
> + mov r11,sp
> + vld1.8 {q14-q15},[r12]!
> + veor q0,q0,q12 @ xor with input
> + veor q1,q1,q13
> + vld1.8 {q12-q13},[r12]!
> + veor q2,q2,q14
> + veor q3,q3,q15
> + vld1.8 {q14-q15},[r12]!
> +
> + veor q4,q4,q12
> + vst1.8 {q0-q1},[r14]! @ store output
> + veor q5,q5,q13
> + vld1.8 {q12-q13},[r12]!
> + veor q6,q6,q14
> + vst1.8 {q2-q3},[r14]!
> + veor q7,q7,q15
> + vld1.8 {q14-q15},[r12]!
> +
> + veor q8,q8,q12
> + vld1.32 {q0-q1},[r11]! @ load for next iteration
> + veor d25,d25,d25
> + vldr d24,[sp,#4*(16+4)] @ four
> + veor q9,q9,q13
> + vld1.32 {q2-q3},[r11]
> + veor q10,q10,q14
> + vst1.8 {q4-q5},[r14]!
> + veor q11,q11,q15
> + vst1.8 {q6-q7},[r14]!
> +
> + vadd.i32 d6,d6,d24 @ next counter value
> + vldr d24,[sp,#4*(16+0)] @ one
> +
> + ldmia sp,{r8-r11} @ load key material
> + add r0,r0,r8 @ accumulate key material
> + ldr r8,[r12],#16 @ load input
> + vst1.8 {q8-q9},[r14]!
> + add r1,r1,r9
> + ldr r9,[r12,#-12]
> + vst1.8 {q10-q11},[r14]!
> + add r2,r2,r10
> + ldr r10,[r12,#-8]
> + add r3,r3,r11
> + ldr r11,[r12,#-4]
> +# ifdef __ARMEB__
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> +# endif
> + eor r0,r0,r8 @ xor with input
> + add r8,sp,#4*(4)
> + eor r1,r1,r9
> + str r0,[r14],#16 @ store output
> + eor r2,r2,r10
> + str r1,[r14,#-12]
> + eor r3,r3,r11
> + ldmia r8,{r8-r11} @ load key material
> + str r2,[r14,#-8]
> + str r3,[r14,#-4]
> +
> + add r4,r8,r4,ror#13 @ accumulate key material
> + ldr r8,[r12],#16 @ load input
> + add r5,r9,r5,ror#13
> + ldr r9,[r12,#-12]
> + add r6,r10,r6,ror#13
> + ldr r10,[r12,#-8]
> + add r7,r11,r7,ror#13
> + ldr r11,[r12,#-4]
> +# ifdef __ARMEB__
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> + eor r4,r4,r8
> + add r8,sp,#4*(8)
> + eor r5,r5,r9
> + str r4,[r14],#16 @ store output
> + eor r6,r6,r10
> + str r5,[r14,#-12]
> + eor r7,r7,r11
> + ldmia r8,{r8-r11} @ load key material
> + str r6,[r14,#-8]
> + add r0,sp,#4*(16+8)
> + str r7,[r14,#-4]
> +
> + ldmia r0,{r0-r7} @ load second half
> +
> + add r0,r0,r8 @ accumulate key material
> + ldr r8,[r12],#16 @ load input
> + add r1,r1,r9
> + ldr r9,[r12,#-12]
> +# ifdef __thumb2__
> + it hi
> +# endif
> + strhi r10,[sp,#4*(16+10)] @ copy "rx" while at it
> + add r2,r2,r10
> + ldr r10,[r12,#-8]
> +# ifdef __thumb2__
> + it hi
> +# endif
> + strhi r11,[sp,#4*(16+11)] @ copy "rx" while at it
> + add r3,r3,r11
> + ldr r11,[r12,#-4]
> +# ifdef __ARMEB__
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> +# endif
> + eor r0,r0,r8
> + add r8,sp,#4*(12)
> + eor r1,r1,r9
> + str r0,[r14],#16 @ store output
> + eor r2,r2,r10
> + str r1,[r14,#-12]
> + eor r3,r3,r11
> + ldmia r8,{r8-r11} @ load key material
> + str r2,[r14,#-8]
> + str r3,[r14,#-4]
> +
> + add r4,r8,r4,ror#24 @ accumulate key material
> + add r8,r8,#4 @ next counter value
> + add r5,r9,r5,ror#24
> + str r8,[sp,#4*(12)] @ save next counter value
> + ldr r8,[r12],#16 @ load input
> + add r6,r10,r6,ror#24
> + add r4,r4,#3 @ counter+3
> + ldr r9,[r12,#-12]
> + add r7,r11,r7,ror#24
> + ldr r10,[r12,#-8]
> + ldr r11,[r12,#-4]
> +# ifdef __ARMEB__
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> + eor r4,r4,r8
> +# ifdef __thumb2__
> + it hi
> +# endif
> + ldrhi r8,[sp,#4*(32+2)] @ re-load len
> + eor r5,r5,r9
> + eor r6,r6,r10
> + str r4,[r14],#16 @ store output
> + eor r7,r7,r11
> + str r5,[r14,#-12]
> + sub r11,r8,#64*4 @ len-=64*4
> + str r6,[r14,#-8]
> + str r7,[r14,#-4]
> + bhi .Loop_neon_outer
> +
> + b .Ldone_neon
> +
> +.align 4
> +.Lbreak_neon:
> + @ harmonize NEON and integer-only stack frames: load data
> + @ from NEON frame, but save to integer-only one; distance
> + @ between the two is 4*(32+4+16-32)=4*(20).
> +
> + str r11, [sp,#4*(20+32+2)] @ save len
> + add r11,sp,#4*(32+4)
> + str r12, [sp,#4*(20+32+1)] @ save inp
> + str r14, [sp,#4*(20+32+0)] @ save out
> +
> + ldr r12,[sp,#4*(16+10)]
> + ldr r14,[sp,#4*(16+11)]
> + vldmia r11,{d8-d15} @ fulfill ABI requirement
> + str r12,[sp,#4*(20+16+10)] @ copy "rx"
> + str r14,[sp,#4*(20+16+11)] @ copy "rx"
> +
> + ldr r11, [sp,#4*(15)]
> + mov r4,r4,ror#19 @ twist b[0..3]
> + ldr r12,[sp,#4*(12)] @ modulo-scheduled load
> + mov r5,r5,ror#19
> + ldr r10, [sp,#4*(13)]
> + mov r6,r6,ror#19
> + ldr r14,[sp,#4*(14)]
> + mov r7,r7,ror#19
> + mov r11,r11,ror#8 @ twist d[0..3]
> + mov r12,r12,ror#8
> + mov r10,r10,ror#8
> + mov r14,r14,ror#8
> + str r11, [sp,#4*(20+16+15)]
> + add r11,sp,#4*(20)
> + vst1.32 {q0-q1},[r11]! @ copy key
> + add sp,sp,#4*(20) @ switch frame
> + vst1.32 {q2-q3},[r11]
> + mov r11,#10
> + b .Loop @ go integer-only
> +
> +.align 4
> +.Ltail_neon:
> + cmp r11,#64*3
> + bhs .L192_or_more_neon
> + cmp r11,#64*2
> + bhs .L128_or_more_neon
> + cmp r11,#64*1
> + bhs .L64_or_more_neon
> +
> + add r8,sp,#4*(8)
> + vst1.8 {q0-q1},[sp]
> + add r10,sp,#4*(0)
> + vst1.8 {q2-q3},[r8]
> + b .Loop_tail_neon
> +
> +.align 4
> +.L64_or_more_neon:
> + vld1.8 {q12-q13},[r12]!
> + vld1.8 {q14-q15},[r12]!
> + veor q0,q0,q12
> + veor q1,q1,q13
> + veor q2,q2,q14
> + veor q3,q3,q15
> + vst1.8 {q0-q1},[r14]!
> + vst1.8 {q2-q3},[r14]!
> +
> + beq .Ldone_neon
> +
> + add r8,sp,#4*(8)
> + vst1.8 {q4-q5},[sp]
> + add r10,sp,#4*(0)
> + vst1.8 {q6-q7},[r8]
> + sub r11,r11,#64*1 @ len-=64*1
> + b .Loop_tail_neon
> +
> +.align 4
> +.L128_or_more_neon:
> + vld1.8 {q12-q13},[r12]!
> + vld1.8 {q14-q15},[r12]!
> + veor q0,q0,q12
> + veor q1,q1,q13
> + vld1.8 {q12-q13},[r12]!
> + veor q2,q2,q14
> + veor q3,q3,q15
> + vld1.8 {q14-q15},[r12]!
> +
> + veor q4,q4,q12
> + veor q5,q5,q13
> + vst1.8 {q0-q1},[r14]!
> + veor q6,q6,q14
> + vst1.8 {q2-q3},[r14]!
> + veor q7,q7,q15
> + vst1.8 {q4-q5},[r14]!
> + vst1.8 {q6-q7},[r14]!
> +
> + beq .Ldone_neon
> +
> + add r8,sp,#4*(8)
> + vst1.8 {q8-q9},[sp]
> + add r10,sp,#4*(0)
> + vst1.8 {q10-q11},[r8]
> + sub r11,r11,#64*2 @ len-=64*2
> + b .Loop_tail_neon
> +
> +.align 4
> +.L192_or_more_neon:
> + vld1.8 {q12-q13},[r12]!
> + vld1.8 {q14-q15},[r12]!
> + veor q0,q0,q12
> + veor q1,q1,q13
> + vld1.8 {q12-q13},[r12]!
> + veor q2,q2,q14
> + veor q3,q3,q15
> + vld1.8 {q14-q15},[r12]!
> +
> + veor q4,q4,q12
> + veor q5,q5,q13
> + vld1.8 {q12-q13},[r12]!
> + veor q6,q6,q14
> + vst1.8 {q0-q1},[r14]!
> + veor q7,q7,q15
> + vld1.8 {q14-q15},[r12]!
> +
> + veor q8,q8,q12
> + vst1.8 {q2-q3},[r14]!
> + veor q9,q9,q13
> + vst1.8 {q4-q5},[r14]!
> + veor q10,q10,q14
> + vst1.8 {q6-q7},[r14]!
> + veor q11,q11,q15
> + vst1.8 {q8-q9},[r14]!
> + vst1.8 {q10-q11},[r14]!
> +
> + beq .Ldone_neon
> +
> + ldmia sp,{r8-r11} @ load key material
> + add r0,r0,r8 @ accumulate key material
> + add r8,sp,#4*(4)
> + add r1,r1,r9
> + add r2,r2,r10
> + add r3,r3,r11
> + ldmia r8,{r8-r11} @ load key material
> +
> + add r4,r8,r4,ror#13 @ accumulate key material
> + add r8,sp,#4*(8)
> + add r5,r9,r5,ror#13
> + add r6,r10,r6,ror#13
> + add r7,r11,r7,ror#13
> + ldmia r8,{r8-r11} @ load key material
> +# ifdef __ARMEB__
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> + stmia sp,{r0-r7}
> + add r0,sp,#4*(16+8)
> +
> + ldmia r0,{r0-r7} @ load second half
> +
> + add r0,r0,r8 @ accumulate key material
> + add r8,sp,#4*(12)
> + add r1,r1,r9
> + add r2,r2,r10
> + add r3,r3,r11
> + ldmia r8,{r8-r11} @ load key material
> +
> + add r4,r8,r4,ror#24 @ accumulate key material
> + add r8,sp,#4*(8)
> + add r5,r9,r5,ror#24
> + add r4,r4,#3 @ counter+3
> + add r6,r10,r6,ror#24
> + add r7,r11,r7,ror#24
> + ldr r11,[sp,#4*(32+2)] @ re-load len
> +# ifdef __ARMEB__
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> + stmia r8,{r0-r7}
> + add r10,sp,#4*(0)
> + sub r11,r11,#64*3 @ len-=64*3
> +
> +.Loop_tail_neon:
> + ldrb r8,[r10],#1 @ read buffer on stack
> + ldrb r9,[r12],#1 @ read input
> + subs r11,r11,#1
> + eor r8,r8,r9
> + strb r8,[r14],#1 @ store output
> + bne .Loop_tail_neon
> +
> +.Ldone_neon:
> + add sp,sp,#4*(32+4)
> + vldmia sp,{d8-d15}
> + add sp,sp,#4*(16+3)
> + ldmia sp!,{r4-r11,pc}
> +.size ChaCha20_neon,.-ChaCha20_neon
> +.comm OPENSSL_armcap_P,4,4
> +#endif
> diff --git a/lib/zinc/chacha20/chacha20-arm64-cryptogams.S b/lib/zinc/chacha20/chacha20-arm64-cryptogams.S
> new file mode 100644
> index 000000000000..4d029bfdad3a
> --- /dev/null
> +++ b/lib/zinc/chacha20/chacha20-arm64-cryptogams.S
> @@ -0,0 +1,1973 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
> +/*
> + * Copyright (C) 2006-2017 CRYPTOGAMS by <appro@openssl.org>. All Rights Reserved.
> + */
> +
> +#include "arm_arch.h"
> +
> +.text
> +
> +
> +
> +.align 5
> +.Lsigma:
> +.quad 0x3320646e61707865,0x6b20657479622d32 // endian-neutral
> +.Lone:
> +.long 1,0,0,0
> +.LOPENSSL_armcap_P:
> +#ifdef __ILP32__
> +.long OPENSSL_armcap_P-.
> +#else
> +.quad OPENSSL_armcap_P-.
> +#endif
> +.byte 67,104,97,67,104,97,50,48,32,102,111,114,32,65,82,77,118,56,44,32,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0
> +.align 2
> +
> +.globl ChaCha20_ctr32
> +.type ChaCha20_ctr32,%function
> +.align 5
> +ChaCha20_ctr32:
> + cbz x2,.Labort
> + adr x5,.LOPENSSL_armcap_P
> + cmp x2,#192
> + b.lo .Lshort
> +#ifdef __ILP32__
> + ldrsw x6,[x5]
> +#else
> + ldr x6,[x5]
> +#endif
> + ldr w17,[x6,x5]
> + tst w17,#ARMV7_NEON
> + b.ne ChaCha20_neon
> +
> +.Lshort:
> + stp x29,x30,[sp,#-96]!
> + add x29,sp,#0
> +
> + adr x5,.Lsigma
> + stp x19,x20,[sp,#16]
> + stp x21,x22,[sp,#32]
> + stp x23,x24,[sp,#48]
> + stp x25,x26,[sp,#64]
> + stp x27,x28,[sp,#80]
> + sub sp,sp,#64
> +
> + ldp x22,x23,[x5] // load sigma
> + ldp x24,x25,[x3] // load key
> + ldp x26,x27,[x3,#16]
> + ldp x28,x30,[x4] // load counter
> +#ifdef __ARMEB__
> + ror x24,x24,#32
> + ror x25,x25,#32
> + ror x26,x26,#32
> + ror x27,x27,#32
> + ror x28,x28,#32
> + ror x30,x30,#32
> +#endif
> +
> +.Loop_outer:
> + mov w5,w22 // unpack key block
> + lsr x6,x22,#32
> + mov w7,w23
> + lsr x8,x23,#32
> + mov w9,w24
> + lsr x10,x24,#32
> + mov w11,w25
> + lsr x12,x25,#32
> + mov w13,w26
> + lsr x14,x26,#32
> + mov w15,w27
> + lsr x16,x27,#32
> + mov w17,w28
> + lsr x19,x28,#32
> + mov w20,w30
> + lsr x21,x30,#32
> +
> + mov x4,#10
> + subs x2,x2,#64
> +.Loop:
> + sub x4,x4,#1
> + add w5,w5,w9
> + add w6,w6,w10
> + add w7,w7,w11
> + add w8,w8,w12
> + eor w17,w17,w5
> + eor w19,w19,w6
> + eor w20,w20,w7
> + eor w21,w21,w8
> + ror w17,w17,#16
> + ror w19,w19,#16
> + ror w20,w20,#16
> + ror w21,w21,#16
> + add w13,w13,w17
> + add w14,w14,w19
> + add w15,w15,w20
> + add w16,w16,w21
> + eor w9,w9,w13
> + eor w10,w10,w14
> + eor w11,w11,w15
> + eor w12,w12,w16
> + ror w9,w9,#20
> + ror w10,w10,#20
> + ror w11,w11,#20
> + ror w12,w12,#20
> + add w5,w5,w9
> + add w6,w6,w10
> + add w7,w7,w11
> + add w8,w8,w12
> + eor w17,w17,w5
> + eor w19,w19,w6
> + eor w20,w20,w7
> + eor w21,w21,w8
> + ror w17,w17,#24
> + ror w19,w19,#24
> + ror w20,w20,#24
> + ror w21,w21,#24
> + add w13,w13,w17
> + add w14,w14,w19
> + add w15,w15,w20
> + add w16,w16,w21
> + eor w9,w9,w13
> + eor w10,w10,w14
> + eor w11,w11,w15
> + eor w12,w12,w16
> + ror w9,w9,#25
> + ror w10,w10,#25
> + ror w11,w11,#25
> + ror w12,w12,#25
> + add w5,w5,w10
> + add w6,w6,w11
> + add w7,w7,w12
> + add w8,w8,w9
> + eor w21,w21,w5
> + eor w17,w17,w6
> + eor w19,w19,w7
> + eor w20,w20,w8
> + ror w21,w21,#16
> + ror w17,w17,#16
> + ror w19,w19,#16
> + ror w20,w20,#16
> + add w15,w15,w21
> + add w16,w16,w17
> + add w13,w13,w19
> + add w14,w14,w20
> + eor w10,w10,w15
> + eor w11,w11,w16
> + eor w12,w12,w13
> + eor w9,w9,w14
> + ror w10,w10,#20
> + ror w11,w11,#20
> + ror w12,w12,#20
> + ror w9,w9,#20
> + add w5,w5,w10
> + add w6,w6,w11
> + add w7,w7,w12
> + add w8,w8,w9
> + eor w21,w21,w5
> + eor w17,w17,w6
> + eor w19,w19,w7
> + eor w20,w20,w8
> + ror w21,w21,#24
> + ror w17,w17,#24
> + ror w19,w19,#24
> + ror w20,w20,#24
> + add w15,w15,w21
> + add w16,w16,w17
> + add w13,w13,w19
> + add w14,w14,w20
> + eor w10,w10,w15
> + eor w11,w11,w16
> + eor w12,w12,w13
> + eor w9,w9,w14
> + ror w10,w10,#25
> + ror w11,w11,#25
> + ror w12,w12,#25
> + ror w9,w9,#25
> + cbnz x4,.Loop
> +
> + add w5,w5,w22 // accumulate key block
> + add x6,x6,x22,lsr#32
> + add w7,w7,w23
> + add x8,x8,x23,lsr#32
> + add w9,w9,w24
> + add x10,x10,x24,lsr#32
> + add w11,w11,w25
> + add x12,x12,x25,lsr#32
> + add w13,w13,w26
> + add x14,x14,x26,lsr#32
> + add w15,w15,w27
> + add x16,x16,x27,lsr#32
> + add w17,w17,w28
> + add x19,x19,x28,lsr#32
> + add w20,w20,w30
> + add x21,x21,x30,lsr#32
> +
> + b.lo .Ltail
> +
> + add x5,x5,x6,lsl#32 // pack
> + add x7,x7,x8,lsl#32
> + ldp x6,x8,[x1,#0] // load input
> + add x9,x9,x10,lsl#32
> + add x11,x11,x12,lsl#32
> + ldp x10,x12,[x1,#16]
> + add x13,x13,x14,lsl#32
> + add x15,x15,x16,lsl#32
> + ldp x14,x16,[x1,#32]
> + add x17,x17,x19,lsl#32
> + add x20,x20,x21,lsl#32
> + ldp x19,x21,[x1,#48]
> + add x1,x1,#64
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + eor x5,x5,x6
> + eor x7,x7,x8
> + eor x9,x9,x10
> + eor x11,x11,x12
> + eor x13,x13,x14
> + eor x15,x15,x16
> + eor x17,x17,x19
> + eor x20,x20,x21
> +
> + stp x5,x7,[x0,#0] // store output
> + add x28,x28,#1 // increment counter
> + stp x9,x11,[x0,#16]
> + stp x13,x15,[x0,#32]
> + stp x17,x20,[x0,#48]
> + add x0,x0,#64
> +
> + b.hi .Loop_outer
> +
> + ldp x19,x20,[x29,#16]
> + add sp,sp,#64
> + ldp x21,x22,[x29,#32]
> + ldp x23,x24,[x29,#48]
> + ldp x25,x26,[x29,#64]
> + ldp x27,x28,[x29,#80]
> + ldp x29,x30,[sp],#96
> +.Labort:
> + ret
> +
> +.align 4
> +.Ltail:
> + add x2,x2,#64
> +.Less_than_64:
> + sub x0,x0,#1
> + add x1,x1,x2
> + add x0,x0,x2
> + add x4,sp,x2
> + neg x2,x2
> +
> + add x5,x5,x6,lsl#32 // pack
> + add x7,x7,x8,lsl#32
> + add x9,x9,x10,lsl#32
> + add x11,x11,x12,lsl#32
> + add x13,x13,x14,lsl#32
> + add x15,x15,x16,lsl#32
> + add x17,x17,x19,lsl#32
> + add x20,x20,x21,lsl#32
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + stp x5,x7,[sp,#0]
> + stp x9,x11,[sp,#16]
> + stp x13,x15,[sp,#32]
> + stp x17,x20,[sp,#48]
> +
> +.Loop_tail:
> + ldrb w10,[x1,x2]
> + ldrb w11,[x4,x2]
> + add x2,x2,#1
> + eor w10,w10,w11
> + strb w10,[x0,x2]
> + cbnz x2,.Loop_tail
> +
> + stp xzr,xzr,[sp,#0]
> + stp xzr,xzr,[sp,#16]
> + stp xzr,xzr,[sp,#32]
> + stp xzr,xzr,[sp,#48]
> +
> + ldp x19,x20,[x29,#16]
> + add sp,sp,#64
> + ldp x21,x22,[x29,#32]
> + ldp x23,x24,[x29,#48]
> + ldp x25,x26,[x29,#64]
> + ldp x27,x28,[x29,#80]
> + ldp x29,x30,[sp],#96
> + ret
> +.size ChaCha20_ctr32,.-ChaCha20_ctr32
> +
> +.type ChaCha20_neon,%function
> +.align 5
> +ChaCha20_neon:
> + stp x29,x30,[sp,#-96]!
> + add x29,sp,#0
> +
> + adr x5,.Lsigma
> + stp x19,x20,[sp,#16]
> + stp x21,x22,[sp,#32]
> + stp x23,x24,[sp,#48]
> + stp x25,x26,[sp,#64]
> + stp x27,x28,[sp,#80]
> + cmp x2,#512
> + b.hs .L512_or_more_neon
> +
> + sub sp,sp,#64
> +
> + ldp x22,x23,[x5] // load sigma
> + ld1 {v24.4s},[x5],#16
> + ldp x24,x25,[x3] // load key
> + ldp x26,x27,[x3,#16]
> + ld1 {v25.4s,v26.4s},[x3]
> + ldp x28,x30,[x4] // load counter
> + ld1 {v27.4s},[x4]
> + ld1 {v31.4s},[x5]
> +#ifdef __ARMEB__
> + rev64 v24.4s,v24.4s
> + ror x24,x24,#32
> + ror x25,x25,#32
> + ror x26,x26,#32
> + ror x27,x27,#32
> + ror x28,x28,#32
> + ror x30,x30,#32
> +#endif
> + add v27.4s,v27.4s,v31.4s // += 1
> + add v28.4s,v27.4s,v31.4s
> + add v29.4s,v28.4s,v31.4s
> + shl v31.4s,v31.4s,#2 // 1 -> 4
> +
> +.Loop_outer_neon:
> + mov w5,w22 // unpack key block
> + lsr x6,x22,#32
> + mov v0.16b,v24.16b
> + mov w7,w23
> + lsr x8,x23,#32
> + mov v4.16b,v24.16b
> + mov w9,w24
> + lsr x10,x24,#32
> + mov v16.16b,v24.16b
> + mov w11,w25
> + mov v1.16b,v25.16b
> + lsr x12,x25,#32
> + mov v5.16b,v25.16b
> + mov w13,w26
> + mov v17.16b,v25.16b
> + lsr x14,x26,#32
> + mov v3.16b,v27.16b
> + mov w15,w27
> + mov v7.16b,v28.16b
> + lsr x16,x27,#32
> + mov v19.16b,v29.16b
> + mov w17,w28
> + mov v2.16b,v26.16b
> + lsr x19,x28,#32
> + mov v6.16b,v26.16b
> + mov w20,w30
> + mov v18.16b,v26.16b
> + lsr x21,x30,#32
> +
> + mov x4,#10
> + subs x2,x2,#256
> +.Loop_neon:
> + sub x4,x4,#1
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w9
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w10
> + add v16.4s,v16.4s,v17.4s
> + add w7,w7,w11
> + eor v3.16b,v3.16b,v0.16b
> + add w8,w8,w12
> + eor v7.16b,v7.16b,v4.16b
> + eor w17,w17,w5
> + eor v19.16b,v19.16b,v16.16b
> + eor w19,w19,w6
> + rev32 v3.8h,v3.8h
> + eor w20,w20,w7
> + rev32 v7.8h,v7.8h
> + eor w21,w21,w8
> + rev32 v19.8h,v19.8h
> + ror w17,w17,#16
> + add v2.4s,v2.4s,v3.4s
> + ror w19,w19,#16
> + add v6.4s,v6.4s,v7.4s
> + ror w20,w20,#16
> + add v18.4s,v18.4s,v19.4s
> + ror w21,w21,#16
> + eor v20.16b,v1.16b,v2.16b
> + add w13,w13,w17
> + eor v21.16b,v5.16b,v6.16b
> + add w14,w14,w19
> + eor v22.16b,v17.16b,v18.16b
> + add w15,w15,w20
> + ushr v1.4s,v20.4s,#20
> + add w16,w16,w21
> + ushr v5.4s,v21.4s,#20
> + eor w9,w9,w13
> + ushr v17.4s,v22.4s,#20
> + eor w10,w10,w14
> + sli v1.4s,v20.4s,#12
> + eor w11,w11,w15
> + sli v5.4s,v21.4s,#12
> + eor w12,w12,w16
> + sli v17.4s,v22.4s,#12
> + ror w9,w9,#20
> + add v0.4s,v0.4s,v1.4s
> + ror w10,w10,#20
> + add v4.4s,v4.4s,v5.4s
> + ror w11,w11,#20
> + add v16.4s,v16.4s,v17.4s
> + ror w12,w12,#20
> + eor v20.16b,v3.16b,v0.16b
> + add w5,w5,w9
> + eor v21.16b,v7.16b,v4.16b
> + add w6,w6,w10
> + eor v22.16b,v19.16b,v16.16b
> + add w7,w7,w11
> + ushr v3.4s,v20.4s,#24
> + add w8,w8,w12
> + ushr v7.4s,v21.4s,#24
> + eor w17,w17,w5
> + ushr v19.4s,v22.4s,#24
> + eor w19,w19,w6
> + sli v3.4s,v20.4s,#8
> + eor w20,w20,w7
> + sli v7.4s,v21.4s,#8
> + eor w21,w21,w8
> + sli v19.4s,v22.4s,#8
> + ror w17,w17,#24
> + add v2.4s,v2.4s,v3.4s
> + ror w19,w19,#24
> + add v6.4s,v6.4s,v7.4s
> + ror w20,w20,#24
> + add v18.4s,v18.4s,v19.4s
> + ror w21,w21,#24
> + eor v20.16b,v1.16b,v2.16b
> + add w13,w13,w17
> + eor v21.16b,v5.16b,v6.16b
> + add w14,w14,w19
> + eor v22.16b,v17.16b,v18.16b
> + add w15,w15,w20
> + ushr v1.4s,v20.4s,#25
> + add w16,w16,w21
> + ushr v5.4s,v21.4s,#25
> + eor w9,w9,w13
> + ushr v17.4s,v22.4s,#25
> + eor w10,w10,w14
> + sli v1.4s,v20.4s,#7
> + eor w11,w11,w15
> + sli v5.4s,v21.4s,#7
> + eor w12,w12,w16
> + sli v17.4s,v22.4s,#7
> + ror w9,w9,#25
> + ext v2.16b,v2.16b,v2.16b,#8
> + ror w10,w10,#25
> + ext v6.16b,v6.16b,v6.16b,#8
> + ror w11,w11,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w12,w12,#25
> + ext v3.16b,v3.16b,v3.16b,#12
> + ext v7.16b,v7.16b,v7.16b,#12
> + ext v19.16b,v19.16b,v19.16b,#12
> + ext v1.16b,v1.16b,v1.16b,#4
> + ext v5.16b,v5.16b,v5.16b,#4
> + ext v17.16b,v17.16b,v17.16b,#4
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w10
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w11
> + add v16.4s,v16.4s,v17.4s
> + add w7,w7,w12
> + eor v3.16b,v3.16b,v0.16b
> + add w8,w8,w9
> + eor v7.16b,v7.16b,v4.16b
> + eor w21,w21,w5
> + eor v19.16b,v19.16b,v16.16b
> + eor w17,w17,w6
> + rev32 v3.8h,v3.8h
> + eor w19,w19,w7
> + rev32 v7.8h,v7.8h
> + eor w20,w20,w8
> + rev32 v19.8h,v19.8h
> + ror w21,w21,#16
> + add v2.4s,v2.4s,v3.4s
> + ror w17,w17,#16
> + add v6.4s,v6.4s,v7.4s
> + ror w19,w19,#16
> + add v18.4s,v18.4s,v19.4s
> + ror w20,w20,#16
> + eor v20.16b,v1.16b,v2.16b
> + add w15,w15,w21
> + eor v21.16b,v5.16b,v6.16b
> + add w16,w16,w17
> + eor v22.16b,v17.16b,v18.16b
> + add w13,w13,w19
> + ushr v1.4s,v20.4s,#20
> + add w14,w14,w20
> + ushr v5.4s,v21.4s,#20
> + eor w10,w10,w15
> + ushr v17.4s,v22.4s,#20
> + eor w11,w11,w16
> + sli v1.4s,v20.4s,#12
> + eor w12,w12,w13
> + sli v5.4s,v21.4s,#12
> + eor w9,w9,w14
> + sli v17.4s,v22.4s,#12
> + ror w10,w10,#20
> + add v0.4s,v0.4s,v1.4s
> + ror w11,w11,#20
> + add v4.4s,v4.4s,v5.4s
> + ror w12,w12,#20
> + add v16.4s,v16.4s,v17.4s
> + ror w9,w9,#20
> + eor v20.16b,v3.16b,v0.16b
> + add w5,w5,w10
> + eor v21.16b,v7.16b,v4.16b
> + add w6,w6,w11
> + eor v22.16b,v19.16b,v16.16b
> + add w7,w7,w12
> + ushr v3.4s,v20.4s,#24
> + add w8,w8,w9
> + ushr v7.4s,v21.4s,#24
> + eor w21,w21,w5
> + ushr v19.4s,v22.4s,#24
> + eor w17,w17,w6
> + sli v3.4s,v20.4s,#8
> + eor w19,w19,w7
> + sli v7.4s,v21.4s,#8
> + eor w20,w20,w8
> + sli v19.4s,v22.4s,#8
> + ror w21,w21,#24
> + add v2.4s,v2.4s,v3.4s
> + ror w17,w17,#24
> + add v6.4s,v6.4s,v7.4s
> + ror w19,w19,#24
> + add v18.4s,v18.4s,v19.4s
> + ror w20,w20,#24
> + eor v20.16b,v1.16b,v2.16b
> + add w15,w15,w21
> + eor v21.16b,v5.16b,v6.16b
> + add w16,w16,w17
> + eor v22.16b,v17.16b,v18.16b
> + add w13,w13,w19
> + ushr v1.4s,v20.4s,#25
> + add w14,w14,w20
> + ushr v5.4s,v21.4s,#25
> + eor w10,w10,w15
> + ushr v17.4s,v22.4s,#25
> + eor w11,w11,w16
> + sli v1.4s,v20.4s,#7
> + eor w12,w12,w13
> + sli v5.4s,v21.4s,#7
> + eor w9,w9,w14
> + sli v17.4s,v22.4s,#7
> + ror w10,w10,#25
> + ext v2.16b,v2.16b,v2.16b,#8
> + ror w11,w11,#25
> + ext v6.16b,v6.16b,v6.16b,#8
> + ror w12,w12,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w9,w9,#25
> + ext v3.16b,v3.16b,v3.16b,#4
> + ext v7.16b,v7.16b,v7.16b,#4
> + ext v19.16b,v19.16b,v19.16b,#4
> + ext v1.16b,v1.16b,v1.16b,#12
> + ext v5.16b,v5.16b,v5.16b,#12
> + ext v17.16b,v17.16b,v17.16b,#12
> + cbnz x4,.Loop_neon
> +
> + add w5,w5,w22 // accumulate key block
> + add v0.4s,v0.4s,v24.4s
> + add x6,x6,x22,lsr#32
> + add v4.4s,v4.4s,v24.4s
> + add w7,w7,w23
> + add v16.4s,v16.4s,v24.4s
> + add x8,x8,x23,lsr#32
> + add v2.4s,v2.4s,v26.4s
> + add w9,w9,w24
> + add v6.4s,v6.4s,v26.4s
> + add x10,x10,x24,lsr#32
> + add v18.4s,v18.4s,v26.4s
> + add w11,w11,w25
> + add v3.4s,v3.4s,v27.4s
> + add x12,x12,x25,lsr#32
> + add w13,w13,w26
> + add v7.4s,v7.4s,v28.4s
> + add x14,x14,x26,lsr#32
> + add w15,w15,w27
> + add v19.4s,v19.4s,v29.4s
> + add x16,x16,x27,lsr#32
> + add w17,w17,w28
> + add v1.4s,v1.4s,v25.4s
> + add x19,x19,x28,lsr#32
> + add w20,w20,w30
> + add v5.4s,v5.4s,v25.4s
> + add x21,x21,x30,lsr#32
> + add v17.4s,v17.4s,v25.4s
> +
> + b.lo .Ltail_neon
> +
> + add x5,x5,x6,lsl#32 // pack
> + add x7,x7,x8,lsl#32
> + ldp x6,x8,[x1,#0] // load input
> + add x9,x9,x10,lsl#32
> + add x11,x11,x12,lsl#32
> + ldp x10,x12,[x1,#16]
> + add x13,x13,x14,lsl#32
> + add x15,x15,x16,lsl#32
> + ldp x14,x16,[x1,#32]
> + add x17,x17,x19,lsl#32
> + add x20,x20,x21,lsl#32
> + ldp x19,x21,[x1,#48]
> + add x1,x1,#64
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
> + eor x5,x5,x6
> + eor x7,x7,x8
> + eor x9,x9,x10
> + eor x11,x11,x12
> + eor x13,x13,x14
> + eor v0.16b,v0.16b,v20.16b
> + eor x15,x15,x16
> + eor v1.16b,v1.16b,v21.16b
> + eor x17,x17,x19
> + eor v2.16b,v2.16b,v22.16b
> + eor x20,x20,x21
> + eor v3.16b,v3.16b,v23.16b
> + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
> +
> + stp x5,x7,[x0,#0] // store output
> + add x28,x28,#4 // increment counter
> + stp x9,x11,[x0,#16]
> + add v27.4s,v27.4s,v31.4s // += 4
> + stp x13,x15,[x0,#32]
> + add v28.4s,v28.4s,v31.4s
> + stp x17,x20,[x0,#48]
> + add v29.4s,v29.4s,v31.4s
> + add x0,x0,#64
> +
> + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[x0],#64
> + ld1 {v0.16b,v1.16b,v2.16b,v3.16b},[x1],#64
> +
> + eor v4.16b,v4.16b,v20.16b
> + eor v5.16b,v5.16b,v21.16b
> + eor v6.16b,v6.16b,v22.16b
> + eor v7.16b,v7.16b,v23.16b
> + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[x0],#64
> +
> + eor v16.16b,v16.16b,v0.16b
> + eor v17.16b,v17.16b,v1.16b
> + eor v18.16b,v18.16b,v2.16b
> + eor v19.16b,v19.16b,v3.16b
> + st1 {v16.16b,v17.16b,v18.16b,v19.16b},[x0],#64
> +
> + b.hi .Loop_outer_neon
> +
> + ldp x19,x20,[x29,#16]
> + add sp,sp,#64
> + ldp x21,x22,[x29,#32]
> + ldp x23,x24,[x29,#48]
> + ldp x25,x26,[x29,#64]
> + ldp x27,x28,[x29,#80]
> + ldp x29,x30,[sp],#96
> + ret
> +
> +.Ltail_neon:
> + add x2,x2,#256
> + cmp x2,#64
> + b.lo .Less_than_64
> +
> + add x5,x5,x6,lsl#32 // pack
> + add x7,x7,x8,lsl#32
> + ldp x6,x8,[x1,#0] // load input
> + add x9,x9,x10,lsl#32
> + add x11,x11,x12,lsl#32
> + ldp x10,x12,[x1,#16]
> + add x13,x13,x14,lsl#32
> + add x15,x15,x16,lsl#32
> + ldp x14,x16,[x1,#32]
> + add x17,x17,x19,lsl#32
> + add x20,x20,x21,lsl#32
> + ldp x19,x21,[x1,#48]
> + add x1,x1,#64
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + eor x5,x5,x6
> + eor x7,x7,x8
> + eor x9,x9,x10
> + eor x11,x11,x12
> + eor x13,x13,x14
> + eor x15,x15,x16
> + eor x17,x17,x19
> + eor x20,x20,x21
> +
> + stp x5,x7,[x0,#0] // store output
> + add x28,x28,#4 // increment counter
> + stp x9,x11,[x0,#16]
> + stp x13,x15,[x0,#32]
> + stp x17,x20,[x0,#48]
> + add x0,x0,#64
> + b.eq .Ldone_neon
> + sub x2,x2,#64
> + cmp x2,#64
> + b.lo .Less_than_128
> +
> + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
> + eor v0.16b,v0.16b,v20.16b
> + eor v1.16b,v1.16b,v21.16b
> + eor v2.16b,v2.16b,v22.16b
> + eor v3.16b,v3.16b,v23.16b
> + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[x0],#64
> + b.eq .Ldone_neon
> + sub x2,x2,#64
> + cmp x2,#64
> + b.lo .Less_than_192
> +
> + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
> + eor v4.16b,v4.16b,v20.16b
> + eor v5.16b,v5.16b,v21.16b
> + eor v6.16b,v6.16b,v22.16b
> + eor v7.16b,v7.16b,v23.16b
> + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[x0],#64
> + b.eq .Ldone_neon
> + sub x2,x2,#64
> +
> + st1 {v16.16b,v17.16b,v18.16b,v19.16b},[sp]
> + b .Last_neon
> +
> +.Less_than_128:
> + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[sp]
> + b .Last_neon
> +.Less_than_192:
> + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[sp]
> + b .Last_neon
> +
> +.align 4
> +.Last_neon:
> + sub x0,x0,#1
> + add x1,x1,x2
> + add x0,x0,x2
> + add x4,sp,x2
> + neg x2,x2
> +
> +.Loop_tail_neon:
> + ldrb w10,[x1,x2]
> + ldrb w11,[x4,x2]
> + add x2,x2,#1
> + eor w10,w10,w11
> + strb w10,[x0,x2]
> + cbnz x2,.Loop_tail_neon
> +
> + stp xzr,xzr,[sp,#0]
> + stp xzr,xzr,[sp,#16]
> + stp xzr,xzr,[sp,#32]
> + stp xzr,xzr,[sp,#48]
> +
> +.Ldone_neon:
> + ldp x19,x20,[x29,#16]
> + add sp,sp,#64
> + ldp x21,x22,[x29,#32]
> + ldp x23,x24,[x29,#48]
> + ldp x25,x26,[x29,#64]
> + ldp x27,x28,[x29,#80]
> + ldp x29,x30,[sp],#96
> + ret
> +.size ChaCha20_neon,.-ChaCha20_neon
> +.type ChaCha20_512_neon,%function
> +.align 5
> +ChaCha20_512_neon:
> + stp x29,x30,[sp,#-96]!
> + add x29,sp,#0
> +
> + adr x5,.Lsigma
> + stp x19,x20,[sp,#16]
> + stp x21,x22,[sp,#32]
> + stp x23,x24,[sp,#48]
> + stp x25,x26,[sp,#64]
> + stp x27,x28,[sp,#80]
> +
> +.L512_or_more_neon:
> + sub sp,sp,#128+64
> +
> + ldp x22,x23,[x5] // load sigma
> + ld1 {v24.4s},[x5],#16
> + ldp x24,x25,[x3] // load key
> + ldp x26,x27,[x3,#16]
> + ld1 {v25.4s,v26.4s},[x3]
> + ldp x28,x30,[x4] // load counter
> + ld1 {v27.4s},[x4]
> + ld1 {v31.4s},[x5]
> +#ifdef __ARMEB__
> + rev64 v24.4s,v24.4s
> + ror x24,x24,#32
> + ror x25,x25,#32
> + ror x26,x26,#32
> + ror x27,x27,#32
> + ror x28,x28,#32
> + ror x30,x30,#32
> +#endif
> + add v27.4s,v27.4s,v31.4s // += 1
> + stp q24,q25,[sp,#0] // off-load key block, invariant part
> + add v27.4s,v27.4s,v31.4s // not typo
> + str q26,[sp,#32]
> + add v28.4s,v27.4s,v31.4s
> + add v29.4s,v28.4s,v31.4s
> + add v30.4s,v29.4s,v31.4s
> + shl v31.4s,v31.4s,#2 // 1 -> 4
> +
> + stp d8,d9,[sp,#128+0] // meet ABI requirements
> + stp d10,d11,[sp,#128+16]
> + stp d12,d13,[sp,#128+32]
> + stp d14,d15,[sp,#128+48]
> +
> + sub x2,x2,#512 // not typo
> +
> +.Loop_outer_512_neon:
> + mov v0.16b,v24.16b
> + mov v4.16b,v24.16b
> + mov v8.16b,v24.16b
> + mov v12.16b,v24.16b
> + mov v16.16b,v24.16b
> + mov v20.16b,v24.16b
> + mov v1.16b,v25.16b
> + mov w5,w22 // unpack key block
> + mov v5.16b,v25.16b
> + lsr x6,x22,#32
> + mov v9.16b,v25.16b
> + mov w7,w23
> + mov v13.16b,v25.16b
> + lsr x8,x23,#32
> + mov v17.16b,v25.16b
> + mov w9,w24
> + mov v21.16b,v25.16b
> + lsr x10,x24,#32
> + mov v3.16b,v27.16b
> + mov w11,w25
> + mov v7.16b,v28.16b
> + lsr x12,x25,#32
> + mov v11.16b,v29.16b
> + mov w13,w26
> + mov v15.16b,v30.16b
> + lsr x14,x26,#32
> + mov v2.16b,v26.16b
> + mov w15,w27
> + mov v6.16b,v26.16b
> + lsr x16,x27,#32
> + add v19.4s,v3.4s,v31.4s // +4
> + mov w17,w28
> + add v23.4s,v7.4s,v31.4s // +4
> + lsr x19,x28,#32
> + mov v10.16b,v26.16b
> + mov w20,w30
> + mov v14.16b,v26.16b
> + lsr x21,x30,#32
> + mov v18.16b,v26.16b
> + stp q27,q28,[sp,#48] // off-load key block, variable part
> + mov v22.16b,v26.16b
> + str q29,[sp,#80]
> +
> + mov x4,#5
> + subs x2,x2,#512
> +.Loop_upper_neon:
> + sub x4,x4,#1
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w9
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w10
> + add v8.4s,v8.4s,v9.4s
> + add w7,w7,w11
> + add v12.4s,v12.4s,v13.4s
> + add w8,w8,w12
> + add v16.4s,v16.4s,v17.4s
> + eor w17,w17,w5
> + add v20.4s,v20.4s,v21.4s
> + eor w19,w19,w6
> + eor v3.16b,v3.16b,v0.16b
> + eor w20,w20,w7
> + eor v7.16b,v7.16b,v4.16b
> + eor w21,w21,w8
> + eor v11.16b,v11.16b,v8.16b
> + ror w17,w17,#16
> + eor v15.16b,v15.16b,v12.16b
> + ror w19,w19,#16
> + eor v19.16b,v19.16b,v16.16b
> + ror w20,w20,#16
> + eor v23.16b,v23.16b,v20.16b
> + ror w21,w21,#16
> + rev32 v3.8h,v3.8h
> + add w13,w13,w17
> + rev32 v7.8h,v7.8h
> + add w14,w14,w19
> + rev32 v11.8h,v11.8h
> + add w15,w15,w20
> + rev32 v15.8h,v15.8h
> + add w16,w16,w21
> + rev32 v19.8h,v19.8h
> + eor w9,w9,w13
> + rev32 v23.8h,v23.8h
> + eor w10,w10,w14
> + add v2.4s,v2.4s,v3.4s
> + eor w11,w11,w15
> + add v6.4s,v6.4s,v7.4s
> + eor w12,w12,w16
> + add v10.4s,v10.4s,v11.4s
> + ror w9,w9,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w10,w10,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w11,w11,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w12,w12,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w9
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w10
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w11
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w12
> + eor v28.16b,v17.16b,v18.16b
> + eor w17,w17,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w19,w19,w6
> + ushr v1.4s,v24.4s,#20
> + eor w20,w20,w7
> + ushr v5.4s,v25.4s,#20
> + eor w21,w21,w8
> + ushr v9.4s,v26.4s,#20
> + ror w17,w17,#24
> + ushr v13.4s,v27.4s,#20
> + ror w19,w19,#24
> + ushr v17.4s,v28.4s,#20
> + ror w20,w20,#24
> + ushr v21.4s,v29.4s,#20
> + ror w21,w21,#24
> + sli v1.4s,v24.4s,#12
> + add w13,w13,w17
> + sli v5.4s,v25.4s,#12
> + add w14,w14,w19
> + sli v9.4s,v26.4s,#12
> + add w15,w15,w20
> + sli v13.4s,v27.4s,#12
> + add w16,w16,w21
> + sli v17.4s,v28.4s,#12
> + eor w9,w9,w13
> + sli v21.4s,v29.4s,#12
> + eor w10,w10,w14
> + add v0.4s,v0.4s,v1.4s
> + eor w11,w11,w15
> + add v4.4s,v4.4s,v5.4s
> + eor w12,w12,w16
> + add v8.4s,v8.4s,v9.4s
> + ror w9,w9,#25
> + add v12.4s,v12.4s,v13.4s
> + ror w10,w10,#25
> + add v16.4s,v16.4s,v17.4s
> + ror w11,w11,#25
> + add v20.4s,v20.4s,v21.4s
> + ror w12,w12,#25
> + eor v24.16b,v3.16b,v0.16b
> + add w5,w5,w10
> + eor v25.16b,v7.16b,v4.16b
> + add w6,w6,w11
> + eor v26.16b,v11.16b,v8.16b
> + add w7,w7,w12
> + eor v27.16b,v15.16b,v12.16b
> + add w8,w8,w9
> + eor v28.16b,v19.16b,v16.16b
> + eor w21,w21,w5
> + eor v29.16b,v23.16b,v20.16b
> + eor w17,w17,w6
> + ushr v3.4s,v24.4s,#24
> + eor w19,w19,w7
> + ushr v7.4s,v25.4s,#24
> + eor w20,w20,w8
> + ushr v11.4s,v26.4s,#24
> + ror w21,w21,#16
> + ushr v15.4s,v27.4s,#24
> + ror w17,w17,#16
> + ushr v19.4s,v28.4s,#24
> + ror w19,w19,#16
> + ushr v23.4s,v29.4s,#24
> + ror w20,w20,#16
> + sli v3.4s,v24.4s,#8
> + add w15,w15,w21
> + sli v7.4s,v25.4s,#8
> + add w16,w16,w17
> + sli v11.4s,v26.4s,#8
> + add w13,w13,w19
> + sli v15.4s,v27.4s,#8
> + add w14,w14,w20
> + sli v19.4s,v28.4s,#8
> + eor w10,w10,w15
> + sli v23.4s,v29.4s,#8
> + eor w11,w11,w16
> + add v2.4s,v2.4s,v3.4s
> + eor w12,w12,w13
> + add v6.4s,v6.4s,v7.4s
> + eor w9,w9,w14
> + add v10.4s,v10.4s,v11.4s
> + ror w10,w10,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w11,w11,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w12,w12,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w9,w9,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w10
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w11
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w12
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w9
> + eor v28.16b,v17.16b,v18.16b
> + eor w21,w21,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w17,w17,w6
> + ushr v1.4s,v24.4s,#25
> + eor w19,w19,w7
> + ushr v5.4s,v25.4s,#25
> + eor w20,w20,w8
> + ushr v9.4s,v26.4s,#25
> + ror w21,w21,#24
> + ushr v13.4s,v27.4s,#25
> + ror w17,w17,#24
> + ushr v17.4s,v28.4s,#25
> + ror w19,w19,#24
> + ushr v21.4s,v29.4s,#25
> + ror w20,w20,#24
> + sli v1.4s,v24.4s,#7
> + add w15,w15,w21
> + sli v5.4s,v25.4s,#7
> + add w16,w16,w17
> + sli v9.4s,v26.4s,#7
> + add w13,w13,w19
> + sli v13.4s,v27.4s,#7
> + add w14,w14,w20
> + sli v17.4s,v28.4s,#7
> + eor w10,w10,w15
> + sli v21.4s,v29.4s,#7
> + eor w11,w11,w16
> + ext v2.16b,v2.16b,v2.16b,#8
> + eor w12,w12,w13
> + ext v6.16b,v6.16b,v6.16b,#8
> + eor w9,w9,w14
> + ext v10.16b,v10.16b,v10.16b,#8
> + ror w10,w10,#25
> + ext v14.16b,v14.16b,v14.16b,#8
> + ror w11,w11,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w12,w12,#25
> + ext v22.16b,v22.16b,v22.16b,#8
> + ror w9,w9,#25
> + ext v3.16b,v3.16b,v3.16b,#12
> + ext v7.16b,v7.16b,v7.16b,#12
> + ext v11.16b,v11.16b,v11.16b,#12
> + ext v15.16b,v15.16b,v15.16b,#12
> + ext v19.16b,v19.16b,v19.16b,#12
> + ext v23.16b,v23.16b,v23.16b,#12
> + ext v1.16b,v1.16b,v1.16b,#4
> + ext v5.16b,v5.16b,v5.16b,#4
> + ext v9.16b,v9.16b,v9.16b,#4
> + ext v13.16b,v13.16b,v13.16b,#4
> + ext v17.16b,v17.16b,v17.16b,#4
> + ext v21.16b,v21.16b,v21.16b,#4
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w9
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w10
> + add v8.4s,v8.4s,v9.4s
> + add w7,w7,w11
> + add v12.4s,v12.4s,v13.4s
> + add w8,w8,w12
> + add v16.4s,v16.4s,v17.4s
> + eor w17,w17,w5
> + add v20.4s,v20.4s,v21.4s
> + eor w19,w19,w6
> + eor v3.16b,v3.16b,v0.16b
> + eor w20,w20,w7
> + eor v7.16b,v7.16b,v4.16b
> + eor w21,w21,w8
> + eor v11.16b,v11.16b,v8.16b
> + ror w17,w17,#16
> + eor v15.16b,v15.16b,v12.16b
> + ror w19,w19,#16
> + eor v19.16b,v19.16b,v16.16b
> + ror w20,w20,#16
> + eor v23.16b,v23.16b,v20.16b
> + ror w21,w21,#16
> + rev32 v3.8h,v3.8h
> + add w13,w13,w17
> + rev32 v7.8h,v7.8h
> + add w14,w14,w19
> + rev32 v11.8h,v11.8h
> + add w15,w15,w20
> + rev32 v15.8h,v15.8h
> + add w16,w16,w21
> + rev32 v19.8h,v19.8h
> + eor w9,w9,w13
> + rev32 v23.8h,v23.8h
> + eor w10,w10,w14
> + add v2.4s,v2.4s,v3.4s
> + eor w11,w11,w15
> + add v6.4s,v6.4s,v7.4s
> + eor w12,w12,w16
> + add v10.4s,v10.4s,v11.4s
> + ror w9,w9,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w10,w10,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w11,w11,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w12,w12,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w9
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w10
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w11
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w12
> + eor v28.16b,v17.16b,v18.16b
> + eor w17,w17,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w19,w19,w6
> + ushr v1.4s,v24.4s,#20
> + eor w20,w20,w7
> + ushr v5.4s,v25.4s,#20
> + eor w21,w21,w8
> + ushr v9.4s,v26.4s,#20
> + ror w17,w17,#24
> + ushr v13.4s,v27.4s,#20
> + ror w19,w19,#24
> + ushr v17.4s,v28.4s,#20
> + ror w20,w20,#24
> + ushr v21.4s,v29.4s,#20
> + ror w21,w21,#24
> + sli v1.4s,v24.4s,#12
> + add w13,w13,w17
> + sli v5.4s,v25.4s,#12
> + add w14,w14,w19
> + sli v9.4s,v26.4s,#12
> + add w15,w15,w20
> + sli v13.4s,v27.4s,#12
> + add w16,w16,w21
> + sli v17.4s,v28.4s,#12
> + eor w9,w9,w13
> + sli v21.4s,v29.4s,#12
> + eor w10,w10,w14
> + add v0.4s,v0.4s,v1.4s
> + eor w11,w11,w15
> + add v4.4s,v4.4s,v5.4s
> + eor w12,w12,w16
> + add v8.4s,v8.4s,v9.4s
> + ror w9,w9,#25
> + add v12.4s,v12.4s,v13.4s
> + ror w10,w10,#25
> + add v16.4s,v16.4s,v17.4s
> + ror w11,w11,#25
> + add v20.4s,v20.4s,v21.4s
> + ror w12,w12,#25
> + eor v24.16b,v3.16b,v0.16b
> + add w5,w5,w10
> + eor v25.16b,v7.16b,v4.16b
> + add w6,w6,w11
> + eor v26.16b,v11.16b,v8.16b
> + add w7,w7,w12
> + eor v27.16b,v15.16b,v12.16b
> + add w8,w8,w9
> + eor v28.16b,v19.16b,v16.16b
> + eor w21,w21,w5
> + eor v29.16b,v23.16b,v20.16b
> + eor w17,w17,w6
> + ushr v3.4s,v24.4s,#24
> + eor w19,w19,w7
> + ushr v7.4s,v25.4s,#24
> + eor w20,w20,w8
> + ushr v11.4s,v26.4s,#24
> + ror w21,w21,#16
> + ushr v15.4s,v27.4s,#24
> + ror w17,w17,#16
> + ushr v19.4s,v28.4s,#24
> + ror w19,w19,#16
> + ushr v23.4s,v29.4s,#24
> + ror w20,w20,#16
> + sli v3.4s,v24.4s,#8
> + add w15,w15,w21
> + sli v7.4s,v25.4s,#8
> + add w16,w16,w17
> + sli v11.4s,v26.4s,#8
> + add w13,w13,w19
> + sli v15.4s,v27.4s,#8
> + add w14,w14,w20
> + sli v19.4s,v28.4s,#8
> + eor w10,w10,w15
> + sli v23.4s,v29.4s,#8
> + eor w11,w11,w16
> + add v2.4s,v2.4s,v3.4s
> + eor w12,w12,w13
> + add v6.4s,v6.4s,v7.4s
> + eor w9,w9,w14
> + add v10.4s,v10.4s,v11.4s
> + ror w10,w10,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w11,w11,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w12,w12,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w9,w9,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w10
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w11
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w12
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w9
> + eor v28.16b,v17.16b,v18.16b
> + eor w21,w21,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w17,w17,w6
> + ushr v1.4s,v24.4s,#25
> + eor w19,w19,w7
> + ushr v5.4s,v25.4s,#25
> + eor w20,w20,w8
> + ushr v9.4s,v26.4s,#25
> + ror w21,w21,#24
> + ushr v13.4s,v27.4s,#25
> + ror w17,w17,#24
> + ushr v17.4s,v28.4s,#25
> + ror w19,w19,#24
> + ushr v21.4s,v29.4s,#25
> + ror w20,w20,#24
> + sli v1.4s,v24.4s,#7
> + add w15,w15,w21
> + sli v5.4s,v25.4s,#7
> + add w16,w16,w17
> + sli v9.4s,v26.4s,#7
> + add w13,w13,w19
> + sli v13.4s,v27.4s,#7
> + add w14,w14,w20
> + sli v17.4s,v28.4s,#7
> + eor w10,w10,w15
> + sli v21.4s,v29.4s,#7
> + eor w11,w11,w16
> + ext v2.16b,v2.16b,v2.16b,#8
> + eor w12,w12,w13
> + ext v6.16b,v6.16b,v6.16b,#8
> + eor w9,w9,w14
> + ext v10.16b,v10.16b,v10.16b,#8
> + ror w10,w10,#25
> + ext v14.16b,v14.16b,v14.16b,#8
> + ror w11,w11,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w12,w12,#25
> + ext v22.16b,v22.16b,v22.16b,#8
> + ror w9,w9,#25
> + ext v3.16b,v3.16b,v3.16b,#4
> + ext v7.16b,v7.16b,v7.16b,#4
> + ext v11.16b,v11.16b,v11.16b,#4
> + ext v15.16b,v15.16b,v15.16b,#4
> + ext v19.16b,v19.16b,v19.16b,#4
> + ext v23.16b,v23.16b,v23.16b,#4
> + ext v1.16b,v1.16b,v1.16b,#12
> + ext v5.16b,v5.16b,v5.16b,#12
> + ext v9.16b,v9.16b,v9.16b,#12
> + ext v13.16b,v13.16b,v13.16b,#12
> + ext v17.16b,v17.16b,v17.16b,#12
> + ext v21.16b,v21.16b,v21.16b,#12
> + cbnz x4,.Loop_upper_neon
> +
> + add w5,w5,w22 // accumulate key block
> + add x6,x6,x22,lsr#32
> + add w7,w7,w23
> + add x8,x8,x23,lsr#32
> + add w9,w9,w24
> + add x10,x10,x24,lsr#32
> + add w11,w11,w25
> + add x12,x12,x25,lsr#32
> + add w13,w13,w26
> + add x14,x14,x26,lsr#32
> + add w15,w15,w27
> + add x16,x16,x27,lsr#32
> + add w17,w17,w28
> + add x19,x19,x28,lsr#32
> + add w20,w20,w30
> + add x21,x21,x30,lsr#32
> +
> + add x5,x5,x6,lsl#32 // pack
> + add x7,x7,x8,lsl#32
> + ldp x6,x8,[x1,#0] // load input
> + add x9,x9,x10,lsl#32
> + add x11,x11,x12,lsl#32
> + ldp x10,x12,[x1,#16]
> + add x13,x13,x14,lsl#32
> + add x15,x15,x16,lsl#32
> + ldp x14,x16,[x1,#32]
> + add x17,x17,x19,lsl#32
> + add x20,x20,x21,lsl#32
> + ldp x19,x21,[x1,#48]
> + add x1,x1,#64
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + eor x5,x5,x6
> + eor x7,x7,x8
> + eor x9,x9,x10
> + eor x11,x11,x12
> + eor x13,x13,x14
> + eor x15,x15,x16
> + eor x17,x17,x19
> + eor x20,x20,x21
> +
> + stp x5,x7,[x0,#0] // store output
> + add x28,x28,#1 // increment counter
> + mov w5,w22 // unpack key block
> + lsr x6,x22,#32
> + stp x9,x11,[x0,#16]
> + mov w7,w23
> + lsr x8,x23,#32
> + stp x13,x15,[x0,#32]
> + mov w9,w24
> + lsr x10,x24,#32
> + stp x17,x20,[x0,#48]
> + add x0,x0,#64
> + mov w11,w25
> + lsr x12,x25,#32
> + mov w13,w26
> + lsr x14,x26,#32
> + mov w15,w27
> + lsr x16,x27,#32
> + mov w17,w28
> + lsr x19,x28,#32
> + mov w20,w30
> + lsr x21,x30,#32
> +
> + mov x4,#5
> +.Loop_lower_neon:
> + sub x4,x4,#1
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w9
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w10
> + add v8.4s,v8.4s,v9.4s
> + add w7,w7,w11
> + add v12.4s,v12.4s,v13.4s
> + add w8,w8,w12
> + add v16.4s,v16.4s,v17.4s
> + eor w17,w17,w5
> + add v20.4s,v20.4s,v21.4s
> + eor w19,w19,w6
> + eor v3.16b,v3.16b,v0.16b
> + eor w20,w20,w7
> + eor v7.16b,v7.16b,v4.16b
> + eor w21,w21,w8
> + eor v11.16b,v11.16b,v8.16b
> + ror w17,w17,#16
> + eor v15.16b,v15.16b,v12.16b
> + ror w19,w19,#16
> + eor v19.16b,v19.16b,v16.16b
> + ror w20,w20,#16
> + eor v23.16b,v23.16b,v20.16b
> + ror w21,w21,#16
> + rev32 v3.8h,v3.8h
> + add w13,w13,w17
> + rev32 v7.8h,v7.8h
> + add w14,w14,w19
> + rev32 v11.8h,v11.8h
> + add w15,w15,w20
> + rev32 v15.8h,v15.8h
> + add w16,w16,w21
> + rev32 v19.8h,v19.8h
> + eor w9,w9,w13
> + rev32 v23.8h,v23.8h
> + eor w10,w10,w14
> + add v2.4s,v2.4s,v3.4s
> + eor w11,w11,w15
> + add v6.4s,v6.4s,v7.4s
> + eor w12,w12,w16
> + add v10.4s,v10.4s,v11.4s
> + ror w9,w9,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w10,w10,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w11,w11,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w12,w12,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w9
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w10
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w11
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w12
> + eor v28.16b,v17.16b,v18.16b
> + eor w17,w17,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w19,w19,w6
> + ushr v1.4s,v24.4s,#20
> + eor w20,w20,w7
> + ushr v5.4s,v25.4s,#20
> + eor w21,w21,w8
> + ushr v9.4s,v26.4s,#20
> + ror w17,w17,#24
> + ushr v13.4s,v27.4s,#20
> + ror w19,w19,#24
> + ushr v17.4s,v28.4s,#20
> + ror w20,w20,#24
> + ushr v21.4s,v29.4s,#20
> + ror w21,w21,#24
> + sli v1.4s,v24.4s,#12
> + add w13,w13,w17
> + sli v5.4s,v25.4s,#12
> + add w14,w14,w19
> + sli v9.4s,v26.4s,#12
> + add w15,w15,w20
> + sli v13.4s,v27.4s,#12
> + add w16,w16,w21
> + sli v17.4s,v28.4s,#12
> + eor w9,w9,w13
> + sli v21.4s,v29.4s,#12
> + eor w10,w10,w14
> + add v0.4s,v0.4s,v1.4s
> + eor w11,w11,w15
> + add v4.4s,v4.4s,v5.4s
> + eor w12,w12,w16
> + add v8.4s,v8.4s,v9.4s
> + ror w9,w9,#25
> + add v12.4s,v12.4s,v13.4s
> + ror w10,w10,#25
> + add v16.4s,v16.4s,v17.4s
> + ror w11,w11,#25
> + add v20.4s,v20.4s,v21.4s
> + ror w12,w12,#25
> + eor v24.16b,v3.16b,v0.16b
> + add w5,w5,w10
> + eor v25.16b,v7.16b,v4.16b
> + add w6,w6,w11
> + eor v26.16b,v11.16b,v8.16b
> + add w7,w7,w12
> + eor v27.16b,v15.16b,v12.16b
> + add w8,w8,w9
> + eor v28.16b,v19.16b,v16.16b
> + eor w21,w21,w5
> + eor v29.16b,v23.16b,v20.16b
> + eor w17,w17,w6
> + ushr v3.4s,v24.4s,#24
> + eor w19,w19,w7
> + ushr v7.4s,v25.4s,#24
> + eor w20,w20,w8
> + ushr v11.4s,v26.4s,#24
> + ror w21,w21,#16
> + ushr v15.4s,v27.4s,#24
> + ror w17,w17,#16
> + ushr v19.4s,v28.4s,#24
> + ror w19,w19,#16
> + ushr v23.4s,v29.4s,#24
> + ror w20,w20,#16
> + sli v3.4s,v24.4s,#8
> + add w15,w15,w21
> + sli v7.4s,v25.4s,#8
> + add w16,w16,w17
> + sli v11.4s,v26.4s,#8
> + add w13,w13,w19
> + sli v15.4s,v27.4s,#8
> + add w14,w14,w20
> + sli v19.4s,v28.4s,#8
> + eor w10,w10,w15
> + sli v23.4s,v29.4s,#8
> + eor w11,w11,w16
> + add v2.4s,v2.4s,v3.4s
> + eor w12,w12,w13
> + add v6.4s,v6.4s,v7.4s
> + eor w9,w9,w14
> + add v10.4s,v10.4s,v11.4s
> + ror w10,w10,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w11,w11,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w12,w12,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w9,w9,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w10
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w11
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w12
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w9
> + eor v28.16b,v17.16b,v18.16b
> + eor w21,w21,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w17,w17,w6
> + ushr v1.4s,v24.4s,#25
> + eor w19,w19,w7
> + ushr v5.4s,v25.4s,#25
> + eor w20,w20,w8
> + ushr v9.4s,v26.4s,#25
> + ror w21,w21,#24
> + ushr v13.4s,v27.4s,#25
> + ror w17,w17,#24
> + ushr v17.4s,v28.4s,#25
> + ror w19,w19,#24
> + ushr v21.4s,v29.4s,#25
> + ror w20,w20,#24
> + sli v1.4s,v24.4s,#7
> + add w15,w15,w21
> + sli v5.4s,v25.4s,#7
> + add w16,w16,w17
> + sli v9.4s,v26.4s,#7
> + add w13,w13,w19
> + sli v13.4s,v27.4s,#7
> + add w14,w14,w20
> + sli v17.4s,v28.4s,#7
> + eor w10,w10,w15
> + sli v21.4s,v29.4s,#7
> + eor w11,w11,w16
> + ext v2.16b,v2.16b,v2.16b,#8
> + eor w12,w12,w13
> + ext v6.16b,v6.16b,v6.16b,#8
> + eor w9,w9,w14
> + ext v10.16b,v10.16b,v10.16b,#8
> + ror w10,w10,#25
> + ext v14.16b,v14.16b,v14.16b,#8
> + ror w11,w11,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w12,w12,#25
> + ext v22.16b,v22.16b,v22.16b,#8
> + ror w9,w9,#25
> + ext v3.16b,v3.16b,v3.16b,#12
> + ext v7.16b,v7.16b,v7.16b,#12
> + ext v11.16b,v11.16b,v11.16b,#12
> + ext v15.16b,v15.16b,v15.16b,#12
> + ext v19.16b,v19.16b,v19.16b,#12
> + ext v23.16b,v23.16b,v23.16b,#12
> + ext v1.16b,v1.16b,v1.16b,#4
> + ext v5.16b,v5.16b,v5.16b,#4
> + ext v9.16b,v9.16b,v9.16b,#4
> + ext v13.16b,v13.16b,v13.16b,#4
> + ext v17.16b,v17.16b,v17.16b,#4
> + ext v21.16b,v21.16b,v21.16b,#4
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w9
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w10
> + add v8.4s,v8.4s,v9.4s
> + add w7,w7,w11
> + add v12.4s,v12.4s,v13.4s
> + add w8,w8,w12
> + add v16.4s,v16.4s,v17.4s
> + eor w17,w17,w5
> + add v20.4s,v20.4s,v21.4s
> + eor w19,w19,w6
> + eor v3.16b,v3.16b,v0.16b
> + eor w20,w20,w7
> + eor v7.16b,v7.16b,v4.16b
> + eor w21,w21,w8
> + eor v11.16b,v11.16b,v8.16b
> + ror w17,w17,#16
> + eor v15.16b,v15.16b,v12.16b
> + ror w19,w19,#16
> + eor v19.16b,v19.16b,v16.16b
> + ror w20,w20,#16
> + eor v23.16b,v23.16b,v20.16b
> + ror w21,w21,#16
> + rev32 v3.8h,v3.8h
> + add w13,w13,w17
> + rev32 v7.8h,v7.8h
> + add w14,w14,w19
> + rev32 v11.8h,v11.8h
> + add w15,w15,w20
> + rev32 v15.8h,v15.8h
> + add w16,w16,w21
> + rev32 v19.8h,v19.8h
> + eor w9,w9,w13
> + rev32 v23.8h,v23.8h
> + eor w10,w10,w14
> + add v2.4s,v2.4s,v3.4s
> + eor w11,w11,w15
> + add v6.4s,v6.4s,v7.4s
> + eor w12,w12,w16
> + add v10.4s,v10.4s,v11.4s
> + ror w9,w9,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w10,w10,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w11,w11,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w12,w12,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w9
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w10
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w11
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w12
> + eor v28.16b,v17.16b,v18.16b
> + eor w17,w17,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w19,w19,w6
> + ushr v1.4s,v24.4s,#20
> + eor w20,w20,w7
> + ushr v5.4s,v25.4s,#20
> + eor w21,w21,w8
> + ushr v9.4s,v26.4s,#20
> + ror w17,w17,#24
> + ushr v13.4s,v27.4s,#20
> + ror w19,w19,#24
> + ushr v17.4s,v28.4s,#20
> + ror w20,w20,#24
> + ushr v21.4s,v29.4s,#20
> + ror w21,w21,#24
> + sli v1.4s,v24.4s,#12
> + add w13,w13,w17
> + sli v5.4s,v25.4s,#12
> + add w14,w14,w19
> + sli v9.4s,v26.4s,#12
> + add w15,w15,w20
> + sli v13.4s,v27.4s,#12
> + add w16,w16,w21
> + sli v17.4s,v28.4s,#12
> + eor w9,w9,w13
> + sli v21.4s,v29.4s,#12
> + eor w10,w10,w14
> + add v0.4s,v0.4s,v1.4s
> + eor w11,w11,w15
> + add v4.4s,v4.4s,v5.4s
> + eor w12,w12,w16
> + add v8.4s,v8.4s,v9.4s
> + ror w9,w9,#25
> + add v12.4s,v12.4s,v13.4s
> + ror w10,w10,#25
> + add v16.4s,v16.4s,v17.4s
> + ror w11,w11,#25
> + add v20.4s,v20.4s,v21.4s
> + ror w12,w12,#25
> + eor v24.16b,v3.16b,v0.16b
> + add w5,w5,w10
> + eor v25.16b,v7.16b,v4.16b
> + add w6,w6,w11
> + eor v26.16b,v11.16b,v8.16b
> + add w7,w7,w12
> + eor v27.16b,v15.16b,v12.16b
> + add w8,w8,w9
> + eor v28.16b,v19.16b,v16.16b
> + eor w21,w21,w5
> + eor v29.16b,v23.16b,v20.16b
> + eor w17,w17,w6
> + ushr v3.4s,v24.4s,#24
> + eor w19,w19,w7
> + ushr v7.4s,v25.4s,#24
> + eor w20,w20,w8
> + ushr v11.4s,v26.4s,#24
> + ror w21,w21,#16
> + ushr v15.4s,v27.4s,#24
> + ror w17,w17,#16
> + ushr v19.4s,v28.4s,#24
> + ror w19,w19,#16
> + ushr v23.4s,v29.4s,#24
> + ror w20,w20,#16
> + sli v3.4s,v24.4s,#8
> + add w15,w15,w21
> + sli v7.4s,v25.4s,#8
> + add w16,w16,w17
> + sli v11.4s,v26.4s,#8
> + add w13,w13,w19
> + sli v15.4s,v27.4s,#8
> + add w14,w14,w20
> + sli v19.4s,v28.4s,#8
> + eor w10,w10,w15
> + sli v23.4s,v29.4s,#8
> + eor w11,w11,w16
> + add v2.4s,v2.4s,v3.4s
> + eor w12,w12,w13
> + add v6.4s,v6.4s,v7.4s
> + eor w9,w9,w14
> + add v10.4s,v10.4s,v11.4s
> + ror w10,w10,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w11,w11,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w12,w12,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w9,w9,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w10
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w11
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w12
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w9
> + eor v28.16b,v17.16b,v18.16b
> + eor w21,w21,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w17,w17,w6
> + ushr v1.4s,v24.4s,#25
> + eor w19,w19,w7
> + ushr v5.4s,v25.4s,#25
> + eor w20,w20,w8
> + ushr v9.4s,v26.4s,#25
> + ror w21,w21,#24
> + ushr v13.4s,v27.4s,#25
> + ror w17,w17,#24
> + ushr v17.4s,v28.4s,#25
> + ror w19,w19,#24
> + ushr v21.4s,v29.4s,#25
> + ror w20,w20,#24
> + sli v1.4s,v24.4s,#7
> + add w15,w15,w21
> + sli v5.4s,v25.4s,#7
> + add w16,w16,w17
> + sli v9.4s,v26.4s,#7
> + add w13,w13,w19
> + sli v13.4s,v27.4s,#7
> + add w14,w14,w20
> + sli v17.4s,v28.4s,#7
> + eor w10,w10,w15
> + sli v21.4s,v29.4s,#7
> + eor w11,w11,w16
> + ext v2.16b,v2.16b,v2.16b,#8
> + eor w12,w12,w13
> + ext v6.16b,v6.16b,v6.16b,#8
> + eor w9,w9,w14
> + ext v10.16b,v10.16b,v10.16b,#8
> + ror w10,w10,#25
> + ext v14.16b,v14.16b,v14.16b,#8
> + ror w11,w11,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w12,w12,#25
> + ext v22.16b,v22.16b,v22.16b,#8
> + ror w9,w9,#25
> + ext v3.16b,v3.16b,v3.16b,#4
> + ext v7.16b,v7.16b,v7.16b,#4
> + ext v11.16b,v11.16b,v11.16b,#4
> + ext v15.16b,v15.16b,v15.16b,#4
> + ext v19.16b,v19.16b,v19.16b,#4
> + ext v23.16b,v23.16b,v23.16b,#4
> + ext v1.16b,v1.16b,v1.16b,#12
> + ext v5.16b,v5.16b,v5.16b,#12
> + ext v9.16b,v9.16b,v9.16b,#12
> + ext v13.16b,v13.16b,v13.16b,#12
> + ext v17.16b,v17.16b,v17.16b,#12
> + ext v21.16b,v21.16b,v21.16b,#12
> + cbnz x4,.Loop_lower_neon
> +
> + add w5,w5,w22 // accumulate key block
> + ldp q24,q25,[sp,#0]
> + add x6,x6,x22,lsr#32
> + ldp q26,q27,[sp,#32]
> + add w7,w7,w23
> + ldp q28,q29,[sp,#64]
> + add x8,x8,x23,lsr#32
> + add v0.4s,v0.4s,v24.4s
> + add w9,w9,w24
> + add v4.4s,v4.4s,v24.4s
> + add x10,x10,x24,lsr#32
> + add v8.4s,v8.4s,v24.4s
> + add w11,w11,w25
> + add v12.4s,v12.4s,v24.4s
> + add x12,x12,x25,lsr#32
> + add v16.4s,v16.4s,v24.4s
> + add w13,w13,w26
> + add v20.4s,v20.4s,v24.4s
> + add x14,x14,x26,lsr#32
> + add v2.4s,v2.4s,v26.4s
> + add w15,w15,w27
> + add v6.4s,v6.4s,v26.4s
> + add x16,x16,x27,lsr#32
> + add v10.4s,v10.4s,v26.4s
> + add w17,w17,w28
> + add v14.4s,v14.4s,v26.4s
> + add x19,x19,x28,lsr#32
> + add v18.4s,v18.4s,v26.4s
> + add w20,w20,w30
> + add v22.4s,v22.4s,v26.4s
> + add x21,x21,x30,lsr#32
> + add v19.4s,v19.4s,v31.4s // +4
> + add x5,x5,x6,lsl#32 // pack
> + add v23.4s,v23.4s,v31.4s // +4
> + add x7,x7,x8,lsl#32
> + add v3.4s,v3.4s,v27.4s
> + ldp x6,x8,[x1,#0] // load input
> + add v7.4s,v7.4s,v28.4s
> + add x9,x9,x10,lsl#32
> + add v11.4s,v11.4s,v29.4s
> + add x11,x11,x12,lsl#32
> + add v15.4s,v15.4s,v30.4s
> + ldp x10,x12,[x1,#16]
> + add v19.4s,v19.4s,v27.4s
> + add x13,x13,x14,lsl#32
> + add v23.4s,v23.4s,v28.4s
> + add x15,x15,x16,lsl#32
> + add v1.4s,v1.4s,v25.4s
> + ldp x14,x16,[x1,#32]
> + add v5.4s,v5.4s,v25.4s
> + add x17,x17,x19,lsl#32
> + add v9.4s,v9.4s,v25.4s
> + add x20,x20,x21,lsl#32
> + add v13.4s,v13.4s,v25.4s
> + ldp x19,x21,[x1,#48]
> + add v17.4s,v17.4s,v25.4s
> + add x1,x1,#64
> + add v21.4s,v21.4s,v25.4s
> +
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + ld1 {v24.16b,v25.16b,v26.16b,v27.16b},[x1],#64
> + eor x5,x5,x6
> + eor x7,x7,x8
> + eor x9,x9,x10
> + eor x11,x11,x12
> + eor x13,x13,x14
> + eor v0.16b,v0.16b,v24.16b
> + eor x15,x15,x16
> + eor v1.16b,v1.16b,v25.16b
> + eor x17,x17,x19
> + eor v2.16b,v2.16b,v26.16b
> + eor x20,x20,x21
> + eor v3.16b,v3.16b,v27.16b
> + ld1 {v24.16b,v25.16b,v26.16b,v27.16b},[x1],#64
> +
> + stp x5,x7,[x0,#0] // store output
> + add x28,x28,#7 // increment counter
> + stp x9,x11,[x0,#16]
> + stp x13,x15,[x0,#32]
> + stp x17,x20,[x0,#48]
> + add x0,x0,#64
> + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[x0],#64
> +
> + ld1 {v0.16b,v1.16b,v2.16b,v3.16b},[x1],#64
> + eor v4.16b,v4.16b,v24.16b
> + eor v5.16b,v5.16b,v25.16b
> + eor v6.16b,v6.16b,v26.16b
> + eor v7.16b,v7.16b,v27.16b
> + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[x0],#64
> +
> + ld1 {v4.16b,v5.16b,v6.16b,v7.16b},[x1],#64
> + eor v8.16b,v8.16b,v0.16b
> + ldp q24,q25,[sp,#0]
> + eor v9.16b,v9.16b,v1.16b
> + ldp q26,q27,[sp,#32]
> + eor v10.16b,v10.16b,v2.16b
> + eor v11.16b,v11.16b,v3.16b
> + st1 {v8.16b,v9.16b,v10.16b,v11.16b},[x0],#64
> +
> + ld1 {v8.16b,v9.16b,v10.16b,v11.16b},[x1],#64
> + eor v12.16b,v12.16b,v4.16b
> + eor v13.16b,v13.16b,v5.16b
> + eor v14.16b,v14.16b,v6.16b
> + eor v15.16b,v15.16b,v7.16b
> + st1 {v12.16b,v13.16b,v14.16b,v15.16b},[x0],#64
> +
> + ld1 {v12.16b,v13.16b,v14.16b,v15.16b},[x1],#64
> + eor v16.16b,v16.16b,v8.16b
> + eor v17.16b,v17.16b,v9.16b
> + eor v18.16b,v18.16b,v10.16b
> + eor v19.16b,v19.16b,v11.16b
> + st1 {v16.16b,v17.16b,v18.16b,v19.16b},[x0],#64
> +
> + shl v0.4s,v31.4s,#1 // 4 -> 8
> + eor v20.16b,v20.16b,v12.16b
> + eor v21.16b,v21.16b,v13.16b
> + eor v22.16b,v22.16b,v14.16b
> + eor v23.16b,v23.16b,v15.16b
> + st1 {v20.16b,v21.16b,v22.16b,v23.16b},[x0],#64
> +
> + add v27.4s,v27.4s,v0.4s // += 8
> + add v28.4s,v28.4s,v0.4s
> + add v29.4s,v29.4s,v0.4s
> + add v30.4s,v30.4s,v0.4s
> +
> + b.hs .Loop_outer_512_neon
> +
> + adds x2,x2,#512
> + ushr v0.4s,v31.4s,#2 // 4 -> 1
> +
> + ldp d8,d9,[sp,#128+0] // meet ABI requirements
> + ldp d10,d11,[sp,#128+16]
> + ldp d12,d13,[sp,#128+32]
> + ldp d14,d15,[sp,#128+48]
> +
> + stp q24,q31,[sp,#0] // wipe off-load area
> + stp q24,q31,[sp,#32]
> + stp q24,q31,[sp,#64]
> +
> + b.eq .Ldone_512_neon
> +
> + cmp x2,#192
> + sub v27.4s,v27.4s,v0.4s // -= 1
> + sub v28.4s,v28.4s,v0.4s
> + sub v29.4s,v29.4s,v0.4s
> + add sp,sp,#128
> + b.hs .Loop_outer_neon
> +
> + eor v25.16b,v25.16b,v25.16b
> + eor v26.16b,v26.16b,v26.16b
> + eor v27.16b,v27.16b,v27.16b
> + eor v28.16b,v28.16b,v28.16b
> + eor v29.16b,v29.16b,v29.16b
> + eor v30.16b,v30.16b,v30.16b
> + b .Loop_outer
> +
> +.Ldone_512_neon:
> + ldp x19,x20,[x29,#16]
> + add sp,sp,#128+64
> + ldp x21,x22,[x29,#32]
> + ldp x23,x24,[x29,#48]
> + ldp x25,x26,[x29,#64]
> + ldp x27,x28,[x29,#80]
> + ldp x29,x30,[sp],#96
> + ret
> +.size ChaCha20_512_neon,.-ChaCha20_512_neon
> --
> 2.19.0
>
^ permalink raw reply
* Re: [PATCH 11/15] octeontx2-af: Add Marvell OcteonTX2 CGX driver
From: Sunil Kovvuri @ 2018-09-28 9:19 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Linux Netdev List, David S. Miller, linux-soc, Sunil Goutham
In-Reply-To: <CAK8P3a1-a_io1Ene58R6i4z99s_u0XTTkdrms9m0Z0q0Y3eb+g@mail.gmail.com>
On Fri, Sep 28, 2018 at 1:45 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Fri, Sep 28, 2018 at 8:09 AM <sunil.kovvuri@gmail.com> wrote:
> >
> > From: Sunil Goutham <sgoutham@marvell.com>
> >
> > This patch adds basic template for Marvell OcteonTX2's
> > CGX ethernet interface driver. Just the probe.
> > RVU AF driver will use APIs exported by this driver
> > for various things like PF to physical interface mapping,
> > loopback mode, interface stats etc. Hence marged both
> > drivers into a single module.
> >
> > Signed-off-by: Sunil Goutham <sgoutham@marvell.com>
> > ---
> > drivers/net/ethernet/marvell/octeontx2/af/Makefile | 2 +-
> > drivers/net/ethernet/marvell/octeontx2/af/cgx.c | 100 +++++++++++++++++++++
> > drivers/net/ethernet/marvell/octeontx2/af/cgx.h | 22 +++++
> > drivers/net/ethernet/marvell/octeontx2/af/rvu.c | 14 ++-
> > 4 files changed, 136 insertions(+), 2 deletions(-)
> > create mode 100644 drivers/net/ethernet/marvell/octeontx2/af/cgx.c
> > create mode 100644 drivers/net/ethernet/marvell/octeontx2/af/cgx.h
> >
> > diff --git a/drivers/net/ethernet/marvell/octeontx2/af/Makefile b/drivers/net/ethernet/marvell/octeontx2/af/Makefile
> > index ac17cb9..8646421 100644
> > --- a/drivers/net/ethernet/marvell/octeontx2/af/Makefile
> > +++ b/drivers/net/ethernet/marvell/octeontx2/af/Makefile
> > @@ -7,4 +7,4 @@ obj-$(CONFIG_OCTEONTX2_MBOX) += octeontx2_mbox.o
> > obj-$(CONFIG_OCTEONTX2_AF) += octeontx2_af.o
> >
> > octeontx2_mbox-y := mbox.o
> > -octeontx2_af-y := rvu.o
> > +octeontx2_af-y := cgx.o rvu.o
> > diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
> > new file mode 100644
> > index 0000000..cfd80d2
> > --- /dev/null
> > +++ b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
>
> > +MODULE_AUTHOR("Marvell International Ltd.");
> > +MODULE_DESCRIPTION(DRV_STRING);
> > +MODULE_LICENSE("GPL v2");
> > +MODULE_DEVICE_TABLE(pci, cgx_id_table);
>
> Since you now link both pci_driver instances in one module, I think
> you need to remove the duplicate author/description/license tags.
> Multiple device tables are fine I think.
>
> Arnd
Thanks, will remove these and re-submit.
Sunil.
^ permalink raw reply
* Re: [PATCH bpf-next] bpf: permit CGROUP_DEVICE programs accessing helper bpf_get_current_cgroup_id()
From: Alexei Starovoitov @ 2018-09-28 9:11 UTC (permalink / raw)
To: Yonghong Song; +Cc: ast, daniel, netdev, guro, kernel-team
In-Reply-To: <20180927213730.1224816-1-yhs@fb.com>
On Thu, Sep 27, 2018 at 02:37:30PM -0700, Yonghong Song wrote:
> Currently, helper bpf_get_current_cgroup_id() is not permitted
> for CGROUP_DEVICE type of programs. If the helper is used
> in such cases, the verifier will log the following error:
>
> 0: (bf) r6 = r1
> 1: (69) r7 = *(u16 *)(r6 +0)
> 2: (85) call bpf_get_current_cgroup_id#80
> unknown func bpf_get_current_cgroup_id#80
>
> The bpf_get_current_cgroup_id() is useful for CGROUP_DEVICE
> type of programs in order to customize action based on cgroup id.
> This patch added such a support.
>
> Cc: Roman Gushchin <guro@fb.com>
> Signed-off-by: Yonghong Song <yhs@fb.com>
Acked-by: Alexei Starovoitov <ast@kernel.org>
^ permalink raw reply
* Re: [PATCH net 0/2] s390/qeth: fixes 2019-09-26
From: Julian Wiedmann @ 2018-09-28 9:05 UTC (permalink / raw)
To: David Miller
Cc: netdev, linux-s390, schwidefsky, heiko.carstens, raspl, ubraun
In-Reply-To: <20180926.103948.934917523873266650.davem@davemloft.net>
On 26.09.2018 19:39, David Miller wrote:
> From: Julian Wiedmann <jwi@linux.ibm.com>
> Date: Wed, 26 Sep 2018 18:07:08 +0200
>
>> please apply two qeth patches for -net. The first is a trivial cleanup
>> required for patch #2 by Jean, which fixes a potential endless loop.
>
> Series applied, thank you.
>
Odd, doesn't look like they arrived in -net though. Forgot to push them out?
Thanks.
^ permalink raw reply
* Re: [net-next 1/8] ice: use [sr]q.count when checking if queue is initialized
From: Sergei Shtylyov @ 2018-09-28 9:01 UTC (permalink / raw)
To: Jeff Kirsher, davem
Cc: Jacob Keller, netdev, nhorman, sassmann, jogreene,
Anirudh Venkataramanan
In-Reply-To: <20180927162201.30900-2-jeffrey.t.kirsher@intel.com>
Hello!
On 9/27/2018 7:21 PM, Jeff Kirsher wrote:
> From: Jacob Keller <jacob.e.keller@intel.com>
>
> When shutting down the controlqs, we check if they are initialized
> before we shut them down and destroy the lock. This is important, as it
> prevents attempts to access the lock of an already shutdown queue.
>
> Unfortunately, we checked rq.head and sq.head as the value to determine
> if the queue was initialized. This doesn't work, because head is not
> reset when the queue is shutdown. In some flows, the adminq will have
> already been shut down prior to calling ice_shutdown_all_ctrlqs. This
> can result in a crash due to attempting to access the already destroyed
> mutex.
>
> Fix this by using rq.count and sq.count instead. Indeed, ice_shutdown_sq
> and ice_shutdown_rq already indicate that this is the value we should be
> using to determine of the queue was initialized.
s/of/if/?
> Signed-off-by: Jacob Keller <jacob.e.keller@intel.com>
> Signed-off-by: Anirudh Venkataramanan <anirudh.venkataramanan@intel.com>
> Tested-by: Andrew Bowers <andrewx.bowers@intel.com>
> Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
[...]
MBR, Sergei
^ permalink raw reply
* Re: [PATCH v3 2/2] netfilter: nf_tables: add requirements for connsecmark support
From: Pablo Neira Ayuso @ 2018-09-28 9:01 UTC (permalink / raw)
To: Christian Göttsche
Cc: kadlec, fw, davem, netfilter-devel, coreteam, netdev,
linux-kernel, paul, sds, eparis, jmorris, serge, selinux,
linux-security-module
In-Reply-To: <20180923182616.11398-2-cgzones@googlemail.com>
On Sun, Sep 23, 2018 at 08:26:16PM +0200, Christian Göttsche wrote:
> Add ability to set the connection tracking secmark value.
>
> Add ability to set the meta secmark value.
Also applied, thanks.
^ permalink raw reply
* Re: [PATCH 1/2] net: dpaa2: move DPAA2 PTP driver out of staging/
From: Andrew Lunn @ 2018-09-28 15:17 UTC (permalink / raw)
To: Y.b. Lu
Cc: linux-kernel@vger.kernel.org, devel@driverdev.osuosl.org,
netdev@vger.kernel.org, Richard Cochran, David S . Miller,
Ioana Ciocoi Radulescu, Greg Kroah-Hartman
In-Reply-To: <VI1PR0401MB223737CACE4E4064AE8325A4F8EC0@VI1PR0401MB2237.eurprd04.prod.outlook.com>
> > struct dprtc_cmd_get_irq - Putting pad at the beginning of a struct seems very
> > odd. And it is not the only example.
>
> [Y.b. Lu] This should depended on MC firmware and APIs I think. Once the MC improves this, the APIs could be updated to fix this.
That is going to be hard to do. Ideally the driver should work with
any firmware version. You don't really want to force the user to
upgrade the driver/kernel and the firmware at the same time. So you
cannot for example remove this pad. What you might be able to do in
newer versions is actually use the space. But you have to be sure the
current code is correctly ignoring it and setting it to zero.
Andrew
^ permalink raw reply
* INFO: trying to register non-static key in tun_chr_write_iter
From: syzbot @ 2018-09-28 15:05 UTC (permalink / raw)
To: brouer, davem, edumazet, jasowang, linux-kernel, mst, netdev, sd,
syzkaller-bugs
Hello,
syzbot found the following crash on:
HEAD commit: 100811936f89 bpf: test_bpf: add init_net to dev for flow_d..
git tree: bpf-next
console output: https://syzkaller.appspot.com/x/log.txt?x=145378e6400000
kernel config: https://syzkaller.appspot.com/x/.config?x=443816db871edd66
dashboard link: https://syzkaller.appspot.com/bug?extid=e662df0ac1d753b57e80
compiler: gcc (GCC) 8.0.1 20180413 (experimental)
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+e662df0ac1d753b57e80@syzkaller.appspotmail.com
device nr0\x01 entered promiscuous mode
INFO: trying to register non-static key.
kobject: 'tx-1' (00000000e8c66ef6): kobject_add_internal: parent: 'queues',
set: 'queues'
the code is fine but needs lockdep annotation.
turning off the locking correctness validator.
CPU: 0 PID: 11684 Comm: syz-executor3 Not tainted 4.19.0-rc5+ #104
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
assign_lock_key kernel/locking/lockdep.c:727 [inline]
register_lock_class+0x21a7/0x29b0 kernel/locking/lockdep.c:753
__lock_acquire+0x1c1/0x4ec0 kernel/locking/lockdep.c:3299
lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3900
__mutex_lock_common kernel/locking/mutex.c:925 [inline]
__mutex_lock+0x166/0x1700 kernel/locking/mutex.c:1072
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1087
tun_get_user+0x1e02/0x42a0 drivers/net/tun.c:1826
tun_chr_write_iter+0xb9/0x160 drivers/net/tun.c:2001
call_write_iter include/linux/fs.h:1808 [inline]
new_sync_write fs/read_write.c:474 [inline]
__vfs_write+0x6b8/0x9f0 fs/read_write.c:487
vfs_write+0x1fc/0x560 fs/read_write.c:549
ksys_write+0x101/0x260 fs/read_write.c:598
__do_sys_write fs/read_write.c:610 [inline]
__se_sys_write fs/read_write.c:607 [inline]
__x64_sys_write+0x73/0xb0 fs/read_write.c:607
do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x457579
Code: 1d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff
ff 0f 83 eb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f681315ec78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457579
RDX: 0000000000000012 RSI: 0000000020000000 RDI: 000000000000000b
RBP: 000000000072c180 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f681315f6d4
R13: 00000000004c5574 R14: 00000000004d8e98 R15: 00000000ffffffff
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'tx-1' (00000000e8c66ef6): kobject_uevent_env
kobject: 'tx-1' (00000000e8c66ef6): fill_kobj_path: path
= '/devices/virtual/net/nr0\x01/queues/tx-1'
kobject: 'rx-1' (00000000b39a08f5): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'rx-1' (00000000b39a08f5): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'rx-1' (00000000b39a08f5): fill_kobj_path: path
= '/devices/virtual/net/nr0\x01/queues/rx-1'
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'tx-1' (00000000e8c66ef6): kobject_cleanup, parent 000000007699ee79
kobject: 'tx-1' (00000000e8c66ef6): auto cleanup 'remove' event
kobject: 'tx-1' (00000000e8c66ef6): kobject_uevent_env
kobject: 'tx-1' (00000000e8c66ef6): fill_kobj_path: path
= '/devices/virtual/net/nr0\x01/queues/tx-1'
kobject: 'tx-1' (00000000e8c66ef6): auto cleanup kobject_del
kobject: 'tx-1' (00000000e8c66ef6): calling ktype release
kobject: 'tx-1': free name
kobject: 'rx-1' (00000000b39a08f5): kobject_cleanup, parent 000000007699ee79
kobject: 'rx-1' (00000000b39a08f5): auto cleanup 'remove' event
kobject: 'rx-1' (00000000b39a08f5): kobject_uevent_env
kobject: 'rx-1' (00000000b39a08f5): fill_kobj_path: path
= '/devices/virtual/net/nr0\x01/queues/rx-1'
kobject: 'rx-1' (00000000b39a08f5): auto cleanup kobject_del
kobject: 'rx-1' (00000000b39a08f5): calling ktype release
kobject: 'rx-1': free name
kobject: 'rx-0' (000000003e93d61e): kobject_cleanup, parent 000000007699ee79
kobject: 'rx-0' (000000003e93d61e): auto cleanup 'remove' event
kobject: 'rx-0' (000000003e93d61e): kobject_uevent_env
kobject: 'rx-0' (000000003e93d61e): fill_kobj_path: path
= '/devices/virtual/net/nr0\x01/queues/rx-0'
kobject: 'rx-0' (000000003e93d61e): auto cleanup kobject_del
kobject: 'rx-0' (000000003e93d61e): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (000000003e264bf3): kobject_cleanup, parent 000000007699ee79
kobject: 'tx-0' (000000003e264bf3): auto cleanup 'remove' event
kobject: 'tx-0' (000000003e264bf3): kobject_uevent_env
kobject: 'tx-0' (000000003e264bf3): fill_kobj_path: path
= '/devices/virtual/net/nr0\x01/queues/tx-0'
kobject: 'tx-0' (000000003e264bf3): auto cleanup kobject_del
kobject: 'tx-0' (000000003e264bf3): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (000000007699ee79): kobject_cleanup, parent
(null)
kobject: 'queues' (000000007699ee79): calling ktype release
kobject: 'queues' (000000007699ee79): kset_release
kobject: 'queues': free name
kobject: 'nr0\x01' (00000000ee24d4b5): kobject_uevent_env
kobject: 'nr0\x01' (00000000ee24d4b5): fill_kobj_path: path
= '/devices/virtual/net/nr0\x01'
kobject: 'nr0\x01' (00000000ee24d4b5): kobject_cleanup, parent (null)
kobject: 'nr0\x01' (00000000ee24d4b5): calling ktype release
kobject: 'nr0\x01': free name
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'i' (00000000c801699c): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'i' (00000000c801699c): kobject_uevent_env
kobject: 'i' (00000000c801699c): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'queues' (0000000043f5dc52): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'queues' (0000000043f5dc52): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'queues' (0000000043f5dc52): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (00000000600dcf17): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'rx-0' (00000000600dcf17): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'rx-0' (00000000600dcf17): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'tx-0' (0000000047ae8cc6): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'tx-0' (0000000047ae8cc6): kobject_uevent_env
kobject: 'tx-0' (0000000047ae8cc6): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'rx-0' (00000000600dcf17): kobject_cleanup, parent 0000000043f5dc52
kobject: 'rx-0' (00000000600dcf17): auto cleanup 'remove' event
kobject: 'rx-0' (00000000600dcf17): kobject_uevent_env
kobject: 'rx-0' (00000000600dcf17): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (00000000600dcf17): auto cleanup kobject_del
kobject: 'rx-0' (00000000600dcf17): calling ktype release
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'rx-0': free name
kobject: 'tx-0' (0000000047ae8cc6): kobject_cleanup, parent 0000000043f5dc52
kobject: 'tx-0' (0000000047ae8cc6): auto cleanup 'remove' event
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'tx-0' (0000000047ae8cc6): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'tx-0' (0000000047ae8cc6): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'tx-0' (0000000047ae8cc6): auto cleanup kobject_del
kobject: 'tx-0' (0000000047ae8cc6): calling ktype release
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'tx-0': free name
kobject: 'queues' (0000000043f5dc52): kobject_cleanup, parent
(null)
kobject: 'queues' (0000000043f5dc52): calling ktype release
kobject: 'queues' (0000000043f5dc52): kset_release
kobject: 'queues': free name
kobject: 'i' (00000000c801699c): kobject_uevent_env
kobject: 'i' (00000000c801699c): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'i' (00000000c801699c): kobject_cleanup, parent (null)
kobject: 'i' (00000000c801699c): calling ktype release
kobject: 'i': free name
kobject: 'i' (00000000a7641c59): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'i' (00000000a7641c59): kobject_uevent_env
kobject: 'i' (00000000a7641c59): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'queues' (0000000095fbc889): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'queues' (0000000095fbc889): kobject_uevent_env
kobject: 'queues' (0000000095fbc889): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (00000000eef86eb6): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'rx-0' (00000000eef86eb6): kobject_uevent_env
kobject: 'rx-0' (00000000eef86eb6): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (0000000077222671): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'tx-0' (0000000077222671): kobject_uevent_env
kobject: 'tx-0' (0000000077222671): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'rx-0' (00000000eef86eb6): kobject_cleanup, parent 0000000095fbc889
kobject: 'rx-0' (00000000eef86eb6): auto cleanup 'remove' event
kobject: 'rx-0' (00000000eef86eb6): kobject_uevent_env
kobject: 'rx-0' (00000000eef86eb6): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (00000000eef86eb6): auto cleanup kobject_del
kobject: 'rx-0' (00000000eef86eb6): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (0000000077222671): kobject_cleanup, parent 0000000095fbc889
kobject: 'tx-0' (0000000077222671): auto cleanup 'remove' event
kobject: 'tx-0' (0000000077222671): kobject_uevent_env
kobject: 'tx-0' (0000000077222671): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (0000000077222671): auto cleanup kobject_del
kobject: 'tx-0' (0000000077222671): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (0000000095fbc889): kobject_cleanup, parent
(null)
kobject: 'queues' (0000000095fbc889): calling ktype release
kobject: 'queues' (0000000095fbc889): kset_release
kobject: 'queues': free name
kobject: 'i' (00000000a7641c59): kobject_uevent_env
kobject: 'i' (00000000a7641c59): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'i' (00000000a7641c59): kobject_cleanup, parent (null)
kobject: 'i' (00000000a7641c59): calling ktype release
kobject: 'i': free name
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'i' (000000000d9ae1e5): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'i' (000000000d9ae1e5): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'i' (000000000d9ae1e5): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'queues' (00000000e089a529): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'queues' (00000000e089a529): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'queues' (00000000e089a529): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'rx-0' (0000000095823a56): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'rx-0' (0000000095823a56): kobject_uevent_env
kobject: 'rx-0' (0000000095823a56): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (000000002f7588ed): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'tx-0' (000000002f7588ed): kobject_uevent_env
kobject: 'tx-0' (000000002f7588ed): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'rx-0' (0000000095823a56): kobject_cleanup, parent 00000000e089a529
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'rx-0' (0000000095823a56): auto cleanup 'remove' event
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'rx-0' (0000000095823a56): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'rx-0' (0000000095823a56): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'rx-0' (0000000095823a56): auto cleanup kobject_del
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'rx-0' (0000000095823a56): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (000000002f7588ed): kobject_cleanup, parent 00000000e089a529
kobject: 'tx-0' (000000002f7588ed): auto cleanup 'remove' event
kobject: 'tx-0' (000000002f7588ed): kobject_uevent_env
kobject: 'tx-0' (000000002f7588ed): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (000000002f7588ed): auto cleanup kobject_del
kobject: 'tx-0' (000000002f7588ed): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (00000000e089a529): kobject_cleanup, parent
(null)
kobject: 'queues' (00000000e089a529): calling ktype release
kobject: 'queues' (00000000e089a529): kset_release
kobject: 'queues': free name
kobject: 'i' (000000000d9ae1e5): kobject_uevent_env
kobject: 'i' (000000000d9ae1e5): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'i' (000000000d9ae1e5): kobject_cleanup, parent (null)
kobject: 'i' (000000000d9ae1e5): calling ktype release
kobject: 'i': free name
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'i' (000000003db311aa): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'i' (000000003db311aa): kobject_uevent_env
kobject: 'i' (000000003db311aa): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'queues' (0000000006c3516d): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'queues' (0000000006c3516d): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'queues' (0000000006c3516d): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'rx-0' (00000000ca97a144): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'rx-0' (00000000ca97a144): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'rx-0' (00000000ca97a144): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'tx-0' (00000000a18be29e): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'tx-0' (00000000a18be29e): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'tx-0' (00000000a18be29e): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'rx-0' (00000000ca97a144): kobject_cleanup, parent 0000000006c3516d
kobject: 'rx-0' (00000000ca97a144): auto cleanup 'remove' event
kobject: 'rx-0' (00000000ca97a144): kobject_uevent_env
kobject: 'rx-0' (00000000ca97a144): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (00000000ca97a144): auto cleanup kobject_del
kobject: 'rx-0' (00000000ca97a144): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (00000000a18be29e): kobject_cleanup, parent 0000000006c3516d
kobject: 'tx-0' (00000000a18be29e): auto cleanup 'remove' event
kobject: 'tx-0' (00000000a18be29e): kobject_uevent_env
kobject: 'tx-0' (00000000a18be29e): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (00000000a18be29e): auto cleanup kobject_del
kobject: 'tx-0' (00000000a18be29e): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (0000000006c3516d): kobject_cleanup, parent
(null)
kobject: 'queues' (0000000006c3516d): calling ktype release
kobject: 'queues' (0000000006c3516d): kset_release
kobject: 'queues': free name
kobject: 'i' (000000003db311aa): kobject_uevent_env
kobject: 'i' (000000003db311aa): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'i' (000000003db311aa): kobject_cleanup, parent (null)
kobject: 'i' (000000003db311aa): calling ktype release
kobject: 'i': free name
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'i' (00000000d8b5fb5b): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'i' (00000000d8b5fb5b): kobject_uevent_env
kobject: 'i' (00000000d8b5fb5b): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'queues' (0000000029da0284): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'queues' (0000000029da0284): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'queues' (0000000029da0284): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'rx-0' (000000009326241e): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'rx-0' (000000009326241e): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'rx-0' (000000009326241e): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (00000000c0e108a6): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'tx-0' (00000000c0e108a6): kobject_uevent_env
kobject: 'tx-0' (00000000c0e108a6): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'i' (00000000983a79d1): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'i' (00000000983a79d1): kobject_uevent_env
kobject: 'i' (00000000983a79d1): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'queues' (0000000067601488): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'queues' (0000000067601488): kobject_uevent_env
kobject: 'queues' (0000000067601488): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (0000000081097f67): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'rx-0' (0000000081097f67): kobject_uevent_env
kobject: 'rx-0' (0000000081097f67): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (000000005f03e7ce): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'tx-0' (000000005f03e7ce): kobject_uevent_env
kobject: 'tx-0' (000000005f03e7ce): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'rx-0' (0000000081097f67): kobject_cleanup, parent 0000000067601488
kobject: 'rx-0' (0000000081097f67): auto cleanup 'remove' event
kobject: 'rx-0' (0000000081097f67): kobject_uevent_env
kobject: 'rx-0' (0000000081097f67): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (0000000081097f67): auto cleanup kobject_del
kobject: 'rx-0' (0000000081097f67): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (000000005f03e7ce): kobject_cleanup, parent 0000000067601488
kobject: 'tx-0' (000000005f03e7ce): auto cleanup 'remove' event
kobject: 'tx-0' (000000005f03e7ce): kobject_uevent_env
kobject: 'tx-0' (000000005f03e7ce): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (000000005f03e7ce): auto cleanup kobject_del
kobject: 'tx-0' (000000005f03e7ce): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (0000000067601488): kobject_cleanup, parent
(null)
kobject: 'queues' (0000000067601488): calling ktype release
kobject: 'queues' (0000000067601488): kset_release
kobject: 'queues': free name
kobject: 'i' (00000000983a79d1): kobject_uevent_env
kobject: 'i' (00000000983a79d1): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'rx-0' (000000009326241e): kobject_cleanup, parent 0000000029da0284
kobject: 'rx-0' (000000009326241e): auto cleanup 'remove' event
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'rx-0' (000000009326241e): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'rx-0' (000000009326241e): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (000000009326241e): auto cleanup kobject_del
kobject: 'rx-0' (000000009326241e): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (00000000c0e108a6): kobject_cleanup, parent 0000000029da0284
kobject: 'tx-0' (00000000c0e108a6): auto cleanup 'remove' event
kobject: 'tx-0' (00000000c0e108a6): kobject_uevent_env
kobject: 'tx-0' (00000000c0e108a6): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (00000000c0e108a6): auto cleanup kobject_del
kobject: 'tx-0' (00000000c0e108a6): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (0000000029da0284): kobject_cleanup, parent
(null)
kobject: 'queues' (0000000029da0284): calling ktype release
kobject: 'queues' (0000000029da0284): kset_release
kobject: 'queues': free name
kobject: 'i' (00000000d8b5fb5b): kobject_uevent_env
kobject: 'i' (00000000d8b5fb5b): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'i' (00000000983a79d1): kobject_cleanup, parent (null)
kobject: 'i' (00000000983a79d1): calling ktype release
kobject: 'i': free name
kobject: 'i' (00000000d8b5fb5b): kobject_cleanup, parent (null)
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'i' (00000000d8b5fb5b): calling ktype release
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'i': free name
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'i' (00000000127717e2): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'i' (00000000127717e2): kobject_uevent_env
kobject: 'i' (00000000127717e2): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'queues' (00000000d6787087): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'queues' (00000000d6787087): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'queues' (00000000d6787087): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (0000000066d53395): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'rx-0' (0000000066d53395): kobject_uevent_env
kobject: 'rx-0' (0000000066d53395): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (00000000cd4658a4): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'tx-0' (00000000cd4658a4): kobject_uevent_env
kobject: 'tx-0' (00000000cd4658a4): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'i' (000000008ace66eb): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'i' (000000008ace66eb): kobject_uevent_env
kobject: 'i' (000000008ace66eb): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'queues' (00000000261c8543): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'queues' (00000000261c8543): kobject_uevent_env
kobject: 'queues' (00000000261c8543): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (000000004a180861): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'rx-0' (000000004a180861): kobject_uevent_env
kobject: 'rx-0' (000000004a180861): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (00000000063907b4): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'tx-0' (00000000063907b4): kobject_uevent_env
kobject: 'tx-0' (00000000063907b4): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'rx-0' (000000004a180861): kobject_cleanup, parent 00000000261c8543
kobject: 'rx-0' (000000004a180861): auto cleanup 'remove' event
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'rx-0' (000000004a180861): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'rx-0' (000000004a180861): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (000000004a180861): auto cleanup kobject_del
kobject: 'rx-0' (000000004a180861): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (00000000063907b4): kobject_cleanup, parent 00000000261c8543
kobject: 'tx-0' (00000000063907b4): auto cleanup 'remove' event
kobject: 'tx-0' (00000000063907b4): kobject_uevent_env
kobject: 'tx-0' (00000000063907b4): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (00000000063907b4): auto cleanup kobject_del
kobject: 'tx-0' (00000000063907b4): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (00000000261c8543): kobject_cleanup, parent
(null)
kobject: 'queues' (00000000261c8543): calling ktype release
kobject: 'queues' (00000000261c8543): kset_release
kobject: 'queues': free name
kobject: 'i' (000000008ace66eb): kobject_uevent_env
kobject: 'i' (000000008ace66eb): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'rx-0' (0000000066d53395): kobject_cleanup, parent 00000000d6787087
kobject: 'rx-0' (0000000066d53395): auto cleanup 'remove' event
kobject: 'rx-0' (0000000066d53395): kobject_uevent_env
kobject: 'rx-0' (0000000066d53395): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (0000000066d53395): auto cleanup kobject_del
kobject: 'rx-0' (0000000066d53395): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (00000000cd4658a4): kobject_cleanup, parent 00000000d6787087
kobject: 'tx-0' (00000000cd4658a4): auto cleanup 'remove' event
kobject: 'tx-0' (00000000cd4658a4): kobject_uevent_env
kobject: 'tx-0' (00000000cd4658a4): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (00000000cd4658a4): auto cleanup kobject_del
kobject: 'tx-0' (00000000cd4658a4): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (00000000d6787087): kobject_cleanup, parent
(null)
kobject: 'queues' (00000000d6787087): calling ktype release
kobject: 'queues' (00000000d6787087): kset_release
kobject: 'queues': free name
kobject: 'i' (00000000127717e2): kobject_uevent_env
kobject: 'i' (00000000127717e2): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'i' (000000008ace66eb): kobject_cleanup, parent (null)
kobject: 'i' (000000008ace66eb): calling ktype release
kobject: 'i': free name
kobject: 'i' (00000000127717e2): kobject_cleanup, parent (null)
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'i' (00000000127717e2): calling ktype release
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'i': free name
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'i' (000000002a5ba534): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'i' (000000002a5ba534): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'i' (000000002a5ba534): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'queues' (000000002bd3c0c9): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'queues' (000000002bd3c0c9): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'queues' (000000002bd3c0c9): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (000000001a0ac9ca): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'rx-0' (000000001a0ac9ca): kobject_uevent_env
kobject: 'rx-0' (000000001a0ac9ca): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (000000004afdf9ed): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'tx-0' (000000004afdf9ed): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'tx-0' (000000004afdf9ed): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'i' (0000000089867b8d): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'i' (0000000089867b8d): kobject_uevent_env
kobject: 'i' (0000000089867b8d): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'queues' (00000000f9cebe04): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'queues' (00000000f9cebe04): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'queues' (00000000f9cebe04): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'rx-0' (00000000eb735641): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'rx-0' (00000000eb735641): kobject_uevent_env
kobject: 'rx-0' (00000000eb735641): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (00000000d8b46f99): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'tx-0' (00000000d8b46f99): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'tx-0' (00000000d8b46f99): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'rx-0' (00000000eb735641): kobject_cleanup, parent 00000000f9cebe04
kobject: 'rx-0' (00000000eb735641): auto cleanup 'remove' event
kobject: 'rx-0' (00000000eb735641): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'rx-0' (00000000eb735641): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'rx-0' (00000000eb735641): auto cleanup kobject_del
kobject: 'rx-0' (00000000eb735641): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (00000000d8b46f99): kobject_cleanup, parent 00000000f9cebe04
kobject: 'tx-0' (00000000d8b46f99): auto cleanup 'remove' event
kobject: 'tx-0' (00000000d8b46f99): kobject_uevent_env
kobject: 'tx-0' (00000000d8b46f99): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (00000000d8b46f99): auto cleanup kobject_del
kobject: 'tx-0' (00000000d8b46f99): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (00000000f9cebe04): kobject_cleanup, parent
(null)
kobject: 'queues' (00000000f9cebe04): calling ktype release
kobject: 'queues' (00000000f9cebe04): kset_release
kobject: 'queues': free name
kobject: 'i' (0000000089867b8d): kobject_uevent_env
kobject: 'i' (0000000089867b8d): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'rx-0' (000000001a0ac9ca): kobject_cleanup, parent 000000002bd3c0c9
kobject: 'rx-0' (000000001a0ac9ca): auto cleanup 'remove' event
kobject: 'rx-0' (000000001a0ac9ca): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'rx-0' (000000001a0ac9ca): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'rx-0' (000000001a0ac9ca): auto cleanup kobject_del
kobject: 'rx-0' (000000001a0ac9ca): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (000000004afdf9ed): kobject_cleanup, parent 000000002bd3c0c9
kobject: 'tx-0' (000000004afdf9ed): auto cleanup 'remove' event
kobject: 'tx-0' (000000004afdf9ed): kobject_uevent_env
kobject: 'tx-0' (000000004afdf9ed): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (000000004afdf9ed): auto cleanup kobject_del
kobject: 'tx-0' (000000004afdf9ed): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (000000002bd3c0c9): kobject_cleanup, parent
(null)
kobject: 'queues' (000000002bd3c0c9): calling ktype release
kobject: 'queues' (000000002bd3c0c9): kset_release
kobject: 'queues': free name
kobject: 'i' (000000002a5ba534): kobject_uevent_env
kobject: 'i' (000000002a5ba534): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'i' (00000000bdbce1b0): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'i' (00000000bdbce1b0): kobject_uevent_env
kobject: 'i' (00000000bdbce1b0): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'queues' (0000000086c1dba6): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'queues' (0000000086c1dba6): kobject_uevent_env
kobject: 'queues' (0000000086c1dba6): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (0000000004c94468): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'rx-0' (0000000004c94468): kobject_uevent_env
kobject: 'rx-0' (0000000004c94468): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (0000000004076f2f): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'tx-0' (0000000004076f2f): kobject_uevent_env
kobject: 'tx-0' (0000000004076f2f): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'rx-0' (0000000004c94468): kobject_cleanup, parent 0000000086c1dba6
kobject: 'rx-0' (0000000004c94468): auto cleanup 'remove' event
kobject: 'rx-0' (0000000004c94468): kobject_uevent_env
kobject: 'rx-0' (0000000004c94468): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (0000000004c94468): auto cleanup kobject_del
kobject: 'rx-0' (0000000004c94468): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (0000000004076f2f): kobject_cleanup, parent 0000000086c1dba6
kobject: 'tx-0' (0000000004076f2f): auto cleanup 'remove' event
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'tx-0' (0000000004076f2f): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'tx-0' (0000000004076f2f): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (0000000004076f2f): auto cleanup kobject_del
kobject: 'tx-0' (0000000004076f2f): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (0000000086c1dba6): kobject_cleanup, parent
(null)
kobject: 'queues' (0000000086c1dba6): calling ktype release
kobject: 'queues' (0000000086c1dba6): kset_release
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'queues': free name
kobject: 'i' (00000000bdbce1b0): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'i' (00000000bdbce1b0): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'i' (0000000089867b8d): kobject_cleanup, parent (null)
kobject: 'i' (0000000089867b8d): calling ktype release
kobject: 'i' (000000002a5ba534): kobject_cleanup, parent (null)
kobject: 'i': free name
kobject: 'i' (000000002a5ba534): calling ktype release
kobject: 'i': free name
kobject: 'i' (00000000bdbce1b0): kobject_cleanup, parent (null)
kobject: 'i' (00000000bdbce1b0): calling ktype release
kobject: 'i': free name
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'i' (000000006d24ca63): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'i' (000000006d24ca63): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'i' (000000006d24ca63): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'queues' (0000000007a511a1): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'queues' (0000000007a511a1): kobject_uevent_env
kobject: 'queues' (0000000007a511a1): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (000000001946dc59): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'rx-0' (000000001946dc59): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'rx-0' (000000001946dc59): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (00000000ef0cc92d): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'tx-0' (00000000ef0cc92d): kobject_uevent_env
kobject: 'tx-0' (00000000ef0cc92d): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'i' (00000000917865da): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'i' (00000000917865da): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'i' (00000000917865da): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'queues' (0000000088d9fc56): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'queues' (0000000088d9fc56): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'queues' (0000000088d9fc56): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'rx-0' (0000000098e9720c): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'rx-0' (0000000098e9720c): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'rx-0' (0000000098e9720c): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'tx-0' (000000003a747871): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'tx-0' (000000003a747871): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'tx-0' (000000003a747871): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'i' (00000000d2824c3a): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'i' (00000000d2824c3a): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'i' (00000000d2824c3a): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'queues' (00000000f8d9b04d): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'queues' (00000000f8d9b04d): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'queues' (00000000f8d9b04d): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (0000000076725752): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'rx-0' (0000000076725752): kobject_uevent_env
kobject: 'rx-0' (0000000076725752): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (00000000dfa5fc07): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'tx-0' (00000000dfa5fc07): kobject_uevent_env
kobject: 'tx-0' (00000000dfa5fc07): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'rx-0' (000000001946dc59): kobject_cleanup, parent 0000000007a511a1
kobject: 'rx-0' (000000001946dc59): auto cleanup 'remove' event
kobject: 'rx-0' (000000001946dc59): kobject_uevent_env
kobject: 'rx-0' (000000001946dc59): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (000000001946dc59): auto cleanup kobject_del
kobject: 'rx-0' (000000001946dc59): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (00000000ef0cc92d): kobject_cleanup, parent 0000000007a511a1
kobject: 'tx-0' (00000000ef0cc92d): auto cleanup 'remove' event
kobject: 'tx-0' (00000000ef0cc92d): kobject_uevent_env
kobject: 'tx-0' (00000000ef0cc92d): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (00000000ef0cc92d): auto cleanup kobject_del
kobject: 'tx-0' (00000000ef0cc92d): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (0000000007a511a1): kobject_cleanup, parent
(null)
kobject: 'queues' (0000000007a511a1): calling ktype release
kobject: 'queues' (0000000007a511a1): kset_release
kobject: 'queues': free name
kobject: 'i' (000000006d24ca63): kobject_uevent_env
kobject: 'i' (000000006d24ca63): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'rx-0' (0000000098e9720c): kobject_cleanup, parent 0000000088d9fc56
kobject: 'rx-0' (0000000098e9720c): auto cleanup 'remove' event
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'rx-0' (0000000098e9720c): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'rx-0' (0000000098e9720c): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (0000000098e9720c): auto cleanup kobject_del
kobject: 'rx-0' (0000000098e9720c): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (000000003a747871): kobject_cleanup, parent 0000000088d9fc56
kobject: 'tx-0' (000000003a747871): auto cleanup 'remove' event
kobject: 'tx-0' (000000003a747871): kobject_uevent_env
kobject: 'tx-0' (000000003a747871): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (000000003a747871): auto cleanup kobject_del
kobject: 'tx-0' (000000003a747871): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (0000000088d9fc56): kobject_cleanup, parent
(null)
kobject: 'queues' (0000000088d9fc56): calling ktype release
kobject: 'queues' (0000000088d9fc56): kset_release
kobject: 'queues': free name
kobject: 'i' (00000000917865da): kobject_uevent_env
kobject: 'i' (00000000917865da): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'rx-0' (0000000076725752): kobject_cleanup, parent 00000000f8d9b04d
kobject: 'rx-0' (0000000076725752): auto cleanup 'remove' event
kobject: 'rx-0' (0000000076725752): kobject_uevent_env
kobject: 'rx-0' (0000000076725752): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (0000000076725752): auto cleanup kobject_del
kobject: 'rx-0' (0000000076725752): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (00000000dfa5fc07): kobject_cleanup, parent 00000000f8d9b04d
kobject: 'tx-0' (00000000dfa5fc07): auto cleanup 'remove' event
kobject: 'tx-0' (00000000dfa5fc07): kobject_uevent_env
kobject: 'tx-0' (00000000dfa5fc07): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (00000000dfa5fc07): auto cleanup kobject_del
kobject: 'tx-0' (00000000dfa5fc07): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (00000000f8d9b04d): kobject_cleanup, parent
(null)
kobject: 'queues' (00000000f8d9b04d): calling ktype release
kobject: 'queues' (00000000f8d9b04d): kset_release
kobject: 'queues': free name
kobject: 'i' (00000000d2824c3a): kobject_uevent_env
kobject: 'i' (00000000d2824c3a): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'i' (000000006d24ca63): kobject_cleanup, parent (null)
kobject: 'i' (00000000917865da): kobject_cleanup, parent (null)
kobject: 'i' (000000006d24ca63): calling ktype release
kobject: 'i' (00000000917865da): calling ktype release
kobject: 'i': free name
kobject: 'i': free name
kobject: 'i' (00000000d2824c3a): kobject_cleanup, parent (null)
kobject: 'i' (00000000d2824c3a): calling ktype release
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'i': free name
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'i' (000000009af5bdee): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'i' (000000009af5bdee): kobject_uevent_env
kobject: 'i' (000000009af5bdee): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'queues' (000000004c6d032d): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'queues' (000000004c6d032d): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'queues' (000000004c6d032d): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (00000000e5074add): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'rx-0' (00000000e5074add): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'rx-0' (00000000e5074add): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (00000000b544d9b3): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'tx-0' (00000000b544d9b3): kobject_uevent_env
kobject: 'tx-0' (00000000b544d9b3): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'i' (000000009fd50b90): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'i' (000000009fd50b90): kobject_uevent_env
kobject: 'i' (000000009fd50b90): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'queues' (0000000054942d99): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'queues' (0000000054942d99): kobject_uevent_env
kobject: 'queues' (0000000054942d99): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'rx-0' (0000000035ecd1ba): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'rx-0' (0000000035ecd1ba): kobject_uevent_env
kobject: 'rx-0' (0000000035ecd1ba): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (0000000055425850): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'tx-0' (0000000055425850): kobject_uevent_env
kobject: 'tx-0' (0000000055425850): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'rx-0' (0000000035ecd1ba): kobject_cleanup, parent 0000000054942d99
kobject: 'rx-0' (0000000035ecd1ba): auto cleanup 'remove' event
kobject: 'rx-0' (0000000035ecd1ba): kobject_uevent_env
kobject: 'rx-0' (0000000035ecd1ba): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (0000000035ecd1ba): auto cleanup kobject_del
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'rx-0' (0000000035ecd1ba): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (0000000055425850): kobject_cleanup, parent 0000000054942d99
kobject: 'tx-0' (0000000055425850): auto cleanup 'remove' event
kobject: 'tx-0' (0000000055425850): kobject_uevent_env
kobject: 'tx-0' (0000000055425850): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'tx-0' (0000000055425850): auto cleanup kobject_del
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'tx-0' (0000000055425850): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (0000000054942d99): kobject_cleanup, parent
(null)
kobject: 'queues' (0000000054942d99): calling ktype release
kobject: 'queues' (0000000054942d99): kset_release
kobject: 'queues': free name
kobject: 'i' (000000009fd50b90): kobject_uevent_env
kobject: 'i' (000000009fd50b90): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'rx-0' (00000000e5074add): kobject_cleanup, parent 000000004c6d032d
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'rx-0' (00000000e5074add): auto cleanup 'remove' event
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'rx-0' (00000000e5074add): kobject_uevent_env
kobject: 'rx-0' (00000000e5074add): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (00000000e5074add): auto cleanup kobject_del
kobject: 'rx-0' (00000000e5074add): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (00000000b544d9b3): kobject_cleanup, parent 000000004c6d032d
kobject: 'tx-0' (00000000b544d9b3): auto cleanup 'remove' event
kobject: 'tx-0' (00000000b544d9b3): kobject_uevent_env
kobject: 'tx-0' (00000000b544d9b3): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (00000000b544d9b3): auto cleanup kobject_del
kobject: 'tx-0' (00000000b544d9b3): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (000000004c6d032d): kobject_cleanup, parent
(null)
kobject: 'queues' (000000004c6d032d): calling ktype release
kobject: 'queues' (000000004c6d032d): kset_release
kobject: 'queues': free name
kobject: 'i' (000000009af5bdee): kobject_uevent_env
kobject: 'i' (000000009af5bdee): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'i' (000000009fd50b90): kobject_cleanup, parent (null)
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'i' (000000009fd50b90): calling ktype release
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'i' (000000009af5bdee): kobject_cleanup, parent (null)
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'i': free name
kobject: 'i' (000000009af5bdee): calling ktype release
kobject: 'i': free name
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'i' (0000000070ce36e1): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'i' (0000000070ce36e1): kobject_uevent_env
kobject: 'i' (0000000070ce36e1): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'queues' (000000005c80c146): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'queues' (000000005c80c146): kobject_uevent_env
kobject: 'queues' (000000005c80c146): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (000000002363462e): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'rx-0' (000000002363462e): kobject_uevent_env
kobject: 'rx-0' (000000002363462e): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (000000000c8c9968): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'tx-0' (000000000c8c9968): kobject_uevent_env
kobject: 'tx-0' (000000000c8c9968): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'i' (0000000040059e2c): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'i' (0000000040059e2c): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'i' (0000000040059e2c): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'queues' (00000000f51a7dec): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'queues' (00000000f51a7dec): kobject_uevent_env
kobject: 'queues' (00000000f51a7dec): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'rx-0' (0000000087391310): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'rx-0' (0000000087391310): kobject_uevent_env
kobject: 'rx-0' (0000000087391310): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'tx-0' (00000000fdc0981c): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'tx-0' (00000000fdc0981c): kobject_uevent_env
kobject: 'tx-0' (00000000fdc0981c): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'rx-0' (000000002363462e): kobject_cleanup, parent 000000005c80c146
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'rx-0' (000000002363462e): auto cleanup 'remove' event
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'rx-0' (000000002363462e): kobject_uevent_env
kobject: 'rx-0' (000000002363462e): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (000000002363462e): auto cleanup kobject_del
kobject: 'rx-0' (000000002363462e): calling ktype release
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'rx-0': free name
kobject: 'tx-0' (000000000c8c9968): kobject_cleanup, parent 000000005c80c146
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'tx-0' (000000000c8c9968): auto cleanup 'remove' event
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'tx-0' (000000000c8c9968): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'tx-0' (000000000c8c9968): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'tx-0' (000000000c8c9968): auto cleanup kobject_del
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'tx-0' (000000000c8c9968): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (000000005c80c146): kobject_cleanup, parent
(null)
kobject: 'queues' (000000005c80c146): calling ktype release
kobject: 'queues' (000000005c80c146): kset_release
kobject: 'queues': free name
kobject: 'i' (0000000070ce36e1): kobject_uevent_env
kobject: 'i' (0000000070ce36e1): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'rx-0' (0000000087391310): kobject_cleanup, parent 00000000f51a7dec
kobject: 'rx-0' (0000000087391310): auto cleanup 'remove' event
kobject: 'rx-0' (0000000087391310): kobject_uevent_env
kobject: 'rx-0' (0000000087391310): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'rx-0' (0000000087391310): auto cleanup kobject_del
kobject: 'rx-0' (0000000087391310): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (00000000fdc0981c): kobject_cleanup, parent 00000000f51a7dec
kobject: 'tx-0' (00000000fdc0981c): auto cleanup 'remove' event
kobject: 'tx-0' (00000000fdc0981c): kobject_uevent_env
kobject: 'tx-0' (00000000fdc0981c): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (00000000fdc0981c): auto cleanup kobject_del
kobject: 'tx-0' (00000000fdc0981c): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (00000000f51a7dec): kobject_cleanup, parent
(null)
kobject: 'queues' (00000000f51a7dec): calling ktype release
kobject: 'queues' (00000000f51a7dec): kset_release
kobject: 'queues': free name
kobject: 'i' (0000000040059e2c): kobject_uevent_env
kobject: 'i' (0000000040059e2c): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'i' (0000000070ce36e1): kobject_cleanup, parent (null)
kobject: 'i' (0000000070ce36e1): calling ktype release
kobject: 'i': free name
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'i' (0000000040059e2c): kobject_cleanup, parent (null)
kobject: 'i' (0000000040059e2c): calling ktype release
kobject: 'i': free name
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'i' (00000000141c63e1): kobject_add_internal: parent: 'net',
set: 'devices'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'i' (00000000141c63e1): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'i' (00000000141c63e1): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'queues' (000000008e34733f): kobject_add_internal: parent: 'i',
set: '<NULL>'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'queues' (000000008e34733f): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'queues' (000000008e34733f): kobject_uevent_env: filter function
caused the event to drop!
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'rx-0' (00000000753bdf04): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'rx-0' (00000000753bdf04): kobject_uevent_env
kobject: 'rx-0' (00000000753bdf04): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'tx-0' (000000004bbe743b): kobject_add_internal: parent: 'queues',
set: 'queues'
kobject: 'tx-0' (000000004bbe743b): kobject_uevent_env
kobject: 'tx-0' (000000004bbe743b): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'loop0' (000000009f832ae0): kobject_uevent_env
kobject: 'loop0' (000000009f832ae0): fill_kobj_path: path
= '/devices/virtual/block/loop0'
kobject: 'rx-0' (00000000753bdf04): kobject_cleanup, parent 000000008e34733f
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'rx-0' (00000000753bdf04): auto cleanup 'remove' event
kobject: 'loop4' (00000000a9f26271): fill_kobj_path: path
= '/devices/virtual/block/loop4'
kobject: 'rx-0' (00000000753bdf04): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'rx-0' (00000000753bdf04): fill_kobj_path: path
= '/devices/virtual/net/i/queues/rx-0'
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'rx-0' (00000000753bdf04): auto cleanup kobject_del
kobject: 'rx-0' (00000000753bdf04): calling ktype release
kobject: 'rx-0': free name
kobject: 'tx-0' (000000004bbe743b): kobject_cleanup, parent 000000008e34733f
kobject: 'tx-0' (000000004bbe743b): auto cleanup 'remove' event
kobject: 'tx-0' (000000004bbe743b): kobject_uevent_env
kobject: 'tx-0' (000000004bbe743b): fill_kobj_path: path
= '/devices/virtual/net/i/queues/tx-0'
kobject: 'tx-0' (000000004bbe743b): auto cleanup kobject_del
kobject: 'tx-0' (000000004bbe743b): calling ktype release
kobject: 'tx-0': free name
kobject: 'queues' (000000008e34733f): kobject_cleanup, parent
(null)
kobject: 'queues' (000000008e34733f): calling ktype release
kobject: 'queues' (000000008e34733f): kset_release
kobject: 'queues': free name
kobject: 'i' (00000000141c63e1): kobject_uevent_env
kobject: 'i' (00000000141c63e1): fill_kobj_path: path
= '/devices/virtual/net/i'
kobject: 'i' (00000000141c63e1): kobject_cleanup, parent (null)
kobject: 'i' (00000000141c63e1): calling ktype release
kobject: 'i': free name
kobject: 'loop3' (00000000b2c7bab5): kobject_uevent_env
kobject: 'loop3' (00000000b2c7bab5): fill_kobj_path: path
= '/devices/virtual/block/loop3'
kobject: 'loop2' (00000000b2d1fc99): kobject_uevent_env
kobject: 'loop2' (00000000b2d1fc99): fill_kobj_path: path
= '/devices/virtual/block/loop2'
kobject: 'loop5' (000000005737c1db): kobject_uevent_env
kobject: 'loop5' (000000005737c1db): fill_kobj_path: path
= '/devices/virtual/block/loop5'
kobject: 'loop1' (0000000028db64e1): kobject_uevent_env
kobject: 'loop1' (0000000028db64e1): fill_kobj_path: path
= '/devices/virtual/block/loop1'
kobject: 'loop4' (00000000a9f26271): kobject_uevent_env
kobject: 'loop4' (00000000a9f26271): fill_kobj_p
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
syzbot.
^ permalink raw reply
* Re: [PATCH net-next v6 23/23] net: WireGuard secure network tunnel
From: Jason A. Donenfeld @ 2018-09-28 15:04 UTC (permalink / raw)
To: Andrew Lunn
Cc: LKML, Netdev, Linux Crypto Mailing List, David Miller,
Greg Kroah-Hartman
In-Reply-To: <20180928150117.GA19396@lunn.ch>
On Fri, Sep 28, 2018 at 5:01 PM Andrew Lunn <andrew@lunn.ch> wrote:
> The namespace is more than just about the linker. I see an Opps stack
> trace with wg_ symbols, i know i need to talk to Jason. Without any
> prefix, i have to go digging into the code to find out who i need to
> talk to. This is one reason why often every symbol has the prefix, not
> just the global scope ones.
Good point. I'll see what the wg_ prefixing looks like for all the
static functions too.
^ permalink raw reply
* Re: [PATCH net-next v6 23/23] net: WireGuard secure network tunnel
From: Andrew Lunn @ 2018-09-28 15:01 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: LKML, Netdev, Linux Crypto Mailing List, David Miller,
Greg Kroah-Hartman
In-Reply-To: <CAHmME9q7WRGF3TTRhhy0i_EB4ad2DaSD=tnHM92zfV4Cckyw=A@mail.gmail.com>
On Fri, Sep 28, 2018 at 12:37:03AM +0200, Jason A. Donenfeld wrote:
> Hi Andrew,
>
> Thanks for following up with this.
>
> On Thu, Sep 27, 2018 at 3:15 AM Andrew Lunn <andrew@lunn.ch> wrote:
> > I know you have been concentrating on the crypto code, so i'm not
> > expecting too many changes at the moment in the network code.
>
> I should be addressing things in parallel, actually, so I'm happy to
> work on this.
>
> > WARNING: Avoid crashing the kernel - try using WARN_ON & recovery code rather than BUG() or BUG_ON()
> > #2984: FILE: drivers/net/wireguard/noise.c:293:
> > + BUG_ON(first_len > BLAKE2S_HASH_SIZE || second_len > BLAKE2S_HASH_SIZE ||
>
> I was actually going to ask you about this, because it applies
> similarly in another context too that I'm trying to refine. The above
> function you quote has the following properties:
We have the above case, and we have the general case. The general case
is, only use BUG_ON() when you know things have already gone bad and
there is no recovery for the machine. I doubt that ever applies to
wireguard. So i expect all the BUG_ON() to be removed. This is
something which Linus keeps ranting about....
In this case:
You have it inside a #ifdef. Meaning you don't really care, you can
keep going anyway if debugging is turned of. So just turn it into a
WARN_ON() so you get the splat, but the kernel keeps running.
You have some other options. first_len, second_len, third_len are all
parameter coming from #defines. As you suggested, you could do
BUILD_BUG_ON(), but you have to do it at the caller. Which is fine,
this is debug code, not user input validation code...
BUILD_BUG_ON(NOISE_HASH_LEN > BLAKE2S_HAS_SIZE)
BUILD_BUG_ON(NOISE_SYMMETRIC_KEY_LEN > BLAKE2S_HAS_SIZE)
BUILD_BUG_ON(NOISE_PUBLIC_KEY_LEN > BLAKE2S_HAS_SIZE)
kdf(chaining_key, key, NULL, dh_calculation, NOISE_HASH_LEN,
NOISE_SYMMETRIC_KEY_LEN, 0, NOISE_PUBLIC_KEY_LEN, chaining_key);
> > WARNING: Macros with flow control statements should be avoided
> > #5471: FILE: drivers/net/wireguard/selftest/allowedips.h:456:
> > +#define init_peer(name) do { \
> > + name = kzalloc(sizeof(*name), GFP_KERNEL); \
> > + if (unlikely(!name)) { \
> > + pr_info("allowedips self-test: out of memory\n"); \
> > + goto free; \
> > + } \
> > + kref_init(&name->refcount); \
> > + } while (0)
>
> This is part of a debugging selftest, where I'm initing a bunch of
> peers one after another, and this macro helps keep the test clear
> while offloading the actual irrelevant coding part to this macro. The
> test itself then just has code like:
Please don't focus on just the examples i give you. Look at the bigger
issue. We cannot see the woods for the trees. checkpatch is giving out
lots of warnings. There are so many, it is hard to see the interesting
ones from the simple ones where you need to add an extra space, or add
missing {}.
If checkpatch was just issues one or two warnings about a goto in a
macro, and nothing else, i probably would let it pass. This is bad,
but it is not terrible. I personally have fallen into a trap caused by
a goto in a macro. I changed the locking, not knowing about the goto,
causing an error path not to unlock the lock. It took a while, but
eventually the error happened, and soon after the machine
deadlocked. At the time static checkers did not expand macros, so they
did not detect the issue.
> On a slightly related note, out of curiosity, any idea what's up with
> the future of LTO in the kernel?
Sorry, i've no idea. Not my corner of the kernel.
> It sounds like that'd be nice to have
> on a module-by-module basis. IOW, I'd love to LTO all of my .c files
> in wireguard together, and then only ever expose mod_init/exit and
> whatever I explicitly EXPORT_SYMBOL
The namespace is more than just about the linker. I see an Opps stack
trace with wg_ symbols, i know i need to talk to Jason. Without any
prefix, i have to go digging into the code to find out who i need to
talk to. This is one reason why often every symbol has the prefix, not
just the global scope ones. Go look at other code in driver/net. You
will find that most drivers use a prefix everywhere, both local and
global.
Andrew
^ permalink raw reply
* Re: [PATCH] netfilter: check if the socket netns is correct.
From: Pablo Neira Ayuso @ 2018-09-28 8:33 UTC (permalink / raw)
To: Flavio Leitner
Cc: Guenter Roeck, Jozsef Kadlecsik, Florian Westphal,
David S. Miller, Alexey Kuznetsov, Hideaki YOSHIFUJI,
netfilter-devel, coreteam, netdev, linux-kernel
In-Reply-To: <20180927225824.GT2702@plex.lan>
On Thu, Sep 27, 2018 at 07:58:24PM -0300, Flavio Leitner wrote:
[...]
> From a5f927e7f1368d753f87cb978d630d786d5adb62 Mon Sep 17 00:00:00 2001
> From: Flavio Leitner <fbl@redhat.com>
> Date: Thu, 27 Sep 2018 19:36:28 -0300
> Subject: [PATCH] xt_socket: check sk before checking for netns.
>
> Only check for the network namespace if the socket is available.
Applied.
Please, Cc netfilter-devel@vger.kernel.org next time for your
netfilter patches.
Thanks.
^ permalink raw reply
* RE: [PATCH v2 net-next 0/2] Add support for Microchip Technology KSZ9131 10/100/1000 Ethernet PHY
From: Yuiko.Oshino @ 2018-09-28 14:51 UTC (permalink / raw)
To: f.fainelli, davem, robh+dt, devicetree, andrew
Cc: linux-kernel, mark.rutland, m.felsch, Markus.Niebel, netdev,
UNGLinuxDriver
In-Reply-To: <7c84dfed-6864-7b25-38d1-9bed1cfea0bc@gmail.com>
>On 09/27/2018 01:15 PM, Yuiko Oshino wrote:
>> This is the initial driver for Microchip KSZ9131 10/100/1000 Ethernet
>> PHY
>>
>> v2:
>> - Creating a series from two related patches.
>
>When people give you Acked-by or Reviewed-by tags in prior versions, it is usually
>a good practice to add them to your next version of the patch submission. Also,
>your patches 1 and 2 did not show up as as reply to this cover letter, while they
>should, please fix this if you ever need to submit new patches in the future.
>
>Thank you!
Florian,
Thank you for your reviews. Sorry I forgot to add Acked-by.
Is seems like I am modifying the dt-binding doc, so I will submit v3.
Thank you!
>
>--
>Florian
Yuiko
^ permalink raw reply
* Re: [PATCH net-next v6 01/23] asm: simd context helper API
From: Ard Biesheuvel @ 2018-09-28 8:28 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: Linux Kernel Mailing List, <netdev@vger.kernel.org>,
open list:HARDWARE RANDOM NUMBER GENERATOR CORE, David S. Miller,
Greg Kroah-Hartman, Samuel Neves, Andy Lutomirski,
Thomas Gleixner, linux-arch
In-Reply-To: <20180925145622.29959-2-Jason@zx2c4.com>
On 25 September 2018 at 16:56, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> Sometimes it's useful to amortize calls to XSAVE/XRSTOR and the related
> FPU/SIMD functions over a number of calls, because FPU restoration is
> quite expensive. This adds a simple header for carrying out this pattern:
>
> simd_context_t simd_context;
>
> simd_get(&simd_context);
> while ((item = get_item_from_queue()) != NULL) {
> encrypt_item(item, simd_context);
> simd_relax(&simd_context);
> }
> simd_put(&simd_context);
>
> The relaxation step ensures that we don't trample over preemption, and
> the get/put API should be a familiar paradigm in the kernel.
>
> On the other end, code that actually wants to use SIMD instructions can
> accept this as a parameter and check it via:
>
> void encrypt_item(struct item *item, simd_context_t *simd_context)
> {
> if (item->len > LARGE_FOR_SIMD && simd_use(simd_context))
> wild_simd_code(item);
> else
> boring_scalar_code(item);
> }
>
> The actual XSAVE happens during simd_use (and only on the first time),
> so that if the context is never actually used, no performance penalty is
> hit.
>
> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
> Cc: Samuel Neves <sneves@dei.uc.pt>
> Cc: Andy Lutomirski <luto@kernel.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Greg KH <gregkh@linuxfoundation.org>
> Cc: linux-arch@vger.kernel.org
> ---
> arch/alpha/include/asm/Kbuild | 5 ++-
> arch/arc/include/asm/Kbuild | 1 +
> arch/arm/include/asm/simd.h | 63 ++++++++++++++++++++++++++++++
> arch/arm64/include/asm/simd.h | 51 +++++++++++++++++++++---
> arch/c6x/include/asm/Kbuild | 3 +-
> arch/h8300/include/asm/Kbuild | 3 +-
> arch/hexagon/include/asm/Kbuild | 1 +
> arch/ia64/include/asm/Kbuild | 1 +
> arch/m68k/include/asm/Kbuild | 1 +
> arch/microblaze/include/asm/Kbuild | 1 +
> arch/mips/include/asm/Kbuild | 1 +
> arch/nds32/include/asm/Kbuild | 7 ++--
> arch/nios2/include/asm/Kbuild | 1 +
> arch/openrisc/include/asm/Kbuild | 7 ++--
> arch/parisc/include/asm/Kbuild | 1 +
> arch/powerpc/include/asm/Kbuild | 3 +-
> arch/riscv/include/asm/Kbuild | 3 +-
> arch/s390/include/asm/Kbuild | 3 +-
> arch/sh/include/asm/Kbuild | 1 +
> arch/sparc/include/asm/Kbuild | 1 +
> arch/um/include/asm/Kbuild | 3 +-
> arch/unicore32/include/asm/Kbuild | 1 +
> arch/x86/include/asm/simd.h | 44 ++++++++++++++++++++-
> arch/xtensa/include/asm/Kbuild | 1 +
> include/asm-generic/simd.h | 20 ++++++++++
> include/linux/simd.h | 28 +++++++++++++
> 26 files changed, 234 insertions(+), 21 deletions(-)
> create mode 100644 arch/arm/include/asm/simd.h
> create mode 100644 include/linux/simd.h
>
> diff --git a/arch/alpha/include/asm/Kbuild b/arch/alpha/include/asm/Kbuild
> index 0580cb8c84b2..07b2c1025d34 100644
> --- a/arch/alpha/include/asm/Kbuild
> +++ b/arch/alpha/include/asm/Kbuild
> @@ -2,14 +2,15 @@
>
>
> generic-y += compat.h
> +generic-y += current.h
> generic-y += exec.h
> generic-y += export.h
> generic-y += fb.h
> generic-y += irq_work.h
> +generic-y += kprobes.h
> generic-y += mcs_spinlock.h
> generic-y += mm-arch-hooks.h
> generic-y += preempt.h
> generic-y += sections.h
> +generic-y += simd.h
> generic-y += trace_clock.h
> -generic-y += current.h
> -generic-y += kprobes.h
Given that this patch applies to all architectures at once, it is
probably better to drop the unrelated reordering hunks to avoid
conflicts.
> diff --git a/arch/arc/include/asm/Kbuild b/arch/arc/include/asm/Kbuild
> index feed50ce89fa..a7f4255f1649 100644
> --- a/arch/arc/include/asm/Kbuild
> +++ b/arch/arc/include/asm/Kbuild
> @@ -22,6 +22,7 @@ generic-y += parport.h
> generic-y += pci.h
> generic-y += percpu.h
> generic-y += preempt.h
> +generic-y += simd.h
> generic-y += topology.h
> generic-y += trace_clock.h
> generic-y += user.h
> diff --git a/arch/arm/include/asm/simd.h b/arch/arm/include/asm/simd.h
> new file mode 100644
> index 000000000000..263950dd69cb
> --- /dev/null
> +++ b/arch/arm/include/asm/simd.h
> @@ -0,0 +1,63 @@
> +/* SPDX-License-Identifier: GPL-2.0
> + *
> + * Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
> + */
> +
> +#include <linux/simd.h>
> +#ifndef _ASM_SIMD_H
> +#define _ASM_SIMD_H
> +
> +#ifdef CONFIG_KERNEL_MODE_NEON
> +#include <asm/neon.h>
> +
> +static __must_check inline bool may_use_simd(void)
> +{
> + return !in_interrupt();
> +}
> +
Remember this guy?
https://marc.info/?l=linux-arch&m=149631094625176&w=2
That was never merged, so let's get it right this time.
> +static inline void simd_get(simd_context_t *ctx)
> +{
> + *ctx = may_use_simd() ? HAVE_FULL_SIMD : HAVE_NO_SIMD;
> +}
> +
> +static inline void simd_put(simd_context_t *ctx)
> +{
> + if (*ctx & HAVE_SIMD_IN_USE)
> + kernel_neon_end();
> + *ctx = HAVE_NO_SIMD;
> +}
> +
> +static __must_check inline bool simd_use(simd_context_t *ctx)
> +{
> + if (!(*ctx & HAVE_FULL_SIMD))
> + return false;
> + if (*ctx & HAVE_SIMD_IN_USE)
> + return true;
> + kernel_neon_begin();
> + *ctx |= HAVE_SIMD_IN_USE;
> + return true;
> +}
> +
> +#else
> +
> +static __must_check inline bool may_use_simd(void)
> +{
> + return false;
> +}
> +
> +static inline void simd_get(simd_context_t *ctx)
> +{
> + *ctx = HAVE_NO_SIMD;
> +}
> +
> +static inline void simd_put(simd_context_t *ctx)
> +{
> +}
> +
> +static __must_check inline bool simd_use(simd_context_t *ctx)
> +{
> + return false;
> +}
> +#endif
> +
> +#endif /* _ASM_SIMD_H */
> diff --git a/arch/arm64/include/asm/simd.h b/arch/arm64/include/asm/simd.h
> index 6495cc51246f..a45ff1600040 100644
> --- a/arch/arm64/include/asm/simd.h
> +++ b/arch/arm64/include/asm/simd.h
> @@ -1,11 +1,10 @@
> -/*
> - * Copyright (C) 2017 Linaro Ltd. <ard.biesheuvel@linaro.org>
> +/* SPDX-License-Identifier: GPL-2.0
> *
> - * This program is free software; you can redistribute it and/or modify it
> - * under the terms of the GNU General Public License version 2 as published
> - * by the Free Software Foundation.
> + * Copyright (C) 2017 Linaro Ltd. <ard.biesheuvel@linaro.org>
> + * Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
> */
>
> +#include <linux/simd.h>
> #ifndef __ASM_SIMD_H
> #define __ASM_SIMD_H
>
> @@ -16,6 +15,8 @@
> #include <linux/types.h>
>
> #ifdef CONFIG_KERNEL_MODE_NEON
> +#include <asm/neon.h>
> +#include <asm/simd.h>
>
> DECLARE_PER_CPU(bool, kernel_neon_busy);
>
> @@ -40,9 +41,47 @@ static __must_check inline bool may_use_simd(void)
> !this_cpu_read(kernel_neon_busy);
> }
>
> +static inline void simd_get(simd_context_t *ctx)
> +{
> + *ctx = may_use_simd() ? HAVE_FULL_SIMD : HAVE_NO_SIMD;
> +}
> +
> +static inline void simd_put(simd_context_t *ctx)
> +{
> + if (*ctx & HAVE_SIMD_IN_USE)
> + kernel_neon_end();
> + *ctx = HAVE_NO_SIMD;
> +}
> +
> +static __must_check inline bool simd_use(simd_context_t *ctx)
> +{
> + if (!(*ctx & HAVE_FULL_SIMD))
> + return false;
> + if (*ctx & HAVE_SIMD_IN_USE)
> + return true;
> + kernel_neon_begin();
> + *ctx |= HAVE_SIMD_IN_USE;
> + return true;
> +}
> +
> #else /* ! CONFIG_KERNEL_MODE_NEON */
>
> -static __must_check inline bool may_use_simd(void) {
> +static __must_check inline bool may_use_simd(void)
> +{
> + return false;
> +}
> +
> +static inline void simd_get(simd_context_t *ctx)
> +{
> + *ctx = HAVE_NO_SIMD;
> +}
> +
> +static inline void simd_put(simd_context_t *ctx)
> +{
> +}
> +
> +static __must_check inline bool simd_use(simd_context_t *ctx)
> +{
> return false;
> }
>
> diff --git a/arch/c6x/include/asm/Kbuild b/arch/c6x/include/asm/Kbuild
> index 33a2c94fed0d..22f3d8333c74 100644
> --- a/arch/c6x/include/asm/Kbuild
> +++ b/arch/c6x/include/asm/Kbuild
> @@ -5,8 +5,8 @@ generic-y += compat.h
> generic-y += current.h
> generic-y += device.h
> generic-y += div64.h
> -generic-y += dma.h
> generic-y += dma-mapping.h
> +generic-y += dma.h
> generic-y += emergency-restart.h
> generic-y += exec.h
> generic-y += extable.h
> @@ -30,6 +30,7 @@ generic-y += pgalloc.h
> generic-y += preempt.h
> generic-y += segment.h
> generic-y += serial.h
> +generic-y += simd.h
> generic-y += tlbflush.h
> generic-y += topology.h
> generic-y += trace_clock.h
> diff --git a/arch/h8300/include/asm/Kbuild b/arch/h8300/include/asm/Kbuild
> index a5d0b2991f47..f5c2f12d593e 100644
> --- a/arch/h8300/include/asm/Kbuild
> +++ b/arch/h8300/include/asm/Kbuild
> @@ -8,8 +8,8 @@ generic-y += current.h
> generic-y += delay.h
> generic-y += device.h
> generic-y += div64.h
> -generic-y += dma.h
> generic-y += dma-mapping.h
> +generic-y += dma.h
> generic-y += emergency-restart.h
> generic-y += exec.h
> generic-y += extable.h
> @@ -39,6 +39,7 @@ generic-y += preempt.h
> generic-y += scatterlist.h
> generic-y += sections.h
> generic-y += serial.h
> +generic-y += simd.h
> generic-y += sizes.h
> generic-y += spinlock.h
> generic-y += timex.h
> diff --git a/arch/hexagon/include/asm/Kbuild b/arch/hexagon/include/asm/Kbuild
> index dd2fd9c0d292..217d4695fd8a 100644
> --- a/arch/hexagon/include/asm/Kbuild
> +++ b/arch/hexagon/include/asm/Kbuild
> @@ -29,6 +29,7 @@ generic-y += rwsem.h
> generic-y += sections.h
> generic-y += segment.h
> generic-y += serial.h
> +generic-y += simd.h
> generic-y += sizes.h
> generic-y += topology.h
> generic-y += trace_clock.h
> diff --git a/arch/ia64/include/asm/Kbuild b/arch/ia64/include/asm/Kbuild
> index 557bbc8ba9f5..41c5ebdf79e5 100644
> --- a/arch/ia64/include/asm/Kbuild
> +++ b/arch/ia64/include/asm/Kbuild
> @@ -4,6 +4,7 @@ generic-y += irq_work.h
> generic-y += mcs_spinlock.h
> generic-y += mm-arch-hooks.h
> generic-y += preempt.h
> +generic-y += simd.h
> generic-y += trace_clock.h
> generic-y += vtime.h
> generic-y += word-at-a-time.h
> diff --git a/arch/m68k/include/asm/Kbuild b/arch/m68k/include/asm/Kbuild
> index a4b8d3331a9e..73898dd1a4d0 100644
> --- a/arch/m68k/include/asm/Kbuild
> +++ b/arch/m68k/include/asm/Kbuild
> @@ -19,6 +19,7 @@ generic-y += mm-arch-hooks.h
> generic-y += percpu.h
> generic-y += preempt.h
> generic-y += sections.h
> +generic-y += simd.h
> generic-y += spinlock.h
> generic-y += topology.h
> generic-y += trace_clock.h
> diff --git a/arch/microblaze/include/asm/Kbuild b/arch/microblaze/include/asm/Kbuild
> index 569ba9e670c1..7a877eea99d3 100644
> --- a/arch/microblaze/include/asm/Kbuild
> +++ b/arch/microblaze/include/asm/Kbuild
> @@ -25,6 +25,7 @@ generic-y += parport.h
> generic-y += percpu.h
> generic-y += preempt.h
> generic-y += serial.h
> +generic-y += simd.h
> generic-y += syscalls.h
> generic-y += topology.h
> generic-y += trace_clock.h
> diff --git a/arch/mips/include/asm/Kbuild b/arch/mips/include/asm/Kbuild
> index 58351e48421e..e8868e0fb2c3 100644
> --- a/arch/mips/include/asm/Kbuild
> +++ b/arch/mips/include/asm/Kbuild
> @@ -16,6 +16,7 @@ generic-y += qrwlock.h
> generic-y += qspinlock.h
> generic-y += sections.h
> generic-y += segment.h
> +generic-y += simd.h
> generic-y += trace_clock.h
> generic-y += unaligned.h
> generic-y += user.h
> diff --git a/arch/nds32/include/asm/Kbuild b/arch/nds32/include/asm/Kbuild
> index dbc4e5422550..603c1d020620 100644
> --- a/arch/nds32/include/asm/Kbuild
> +++ b/arch/nds32/include/asm/Kbuild
> @@ -7,14 +7,14 @@ generic-y += bug.h
> generic-y += bugs.h
> generic-y += checksum.h
> generic-y += clkdev.h
> -generic-y += cmpxchg.h
> generic-y += cmpxchg-local.h
> +generic-y += cmpxchg.h
> generic-y += compat.h
> generic-y += cputime.h
> generic-y += device.h
> generic-y += div64.h
> -generic-y += dma.h
> generic-y += dma-mapping.h
> +generic-y += dma.h
> generic-y += emergency-restart.h
> generic-y += errno.h
> generic-y += exec.h
> @@ -46,14 +46,15 @@ generic-y += sections.h
> generic-y += segment.h
> generic-y += serial.h
> generic-y += shmbuf.h
> +generic-y += simd.h
> generic-y += sizes.h
> generic-y += stat.h
> generic-y += switch_to.h
> generic-y += timex.h
> generic-y += topology.h
> generic-y += trace_clock.h
> -generic-y += xor.h
> generic-y += unaligned.h
> generic-y += user.h
> generic-y += vga.h
> generic-y += word-at-a-time.h
> +generic-y += xor.h
> diff --git a/arch/nios2/include/asm/Kbuild b/arch/nios2/include/asm/Kbuild
> index 8fde4fa2c34f..571a9d9ad107 100644
> --- a/arch/nios2/include/asm/Kbuild
> +++ b/arch/nios2/include/asm/Kbuild
> @@ -33,6 +33,7 @@ generic-y += preempt.h
> generic-y += sections.h
> generic-y += segment.h
> generic-y += serial.h
> +generic-y += simd.h
> generic-y += spinlock.h
> generic-y += topology.h
> generic-y += trace_clock.h
> diff --git a/arch/openrisc/include/asm/Kbuild b/arch/openrisc/include/asm/Kbuild
> index eb87cd8327c8..5e9f2f4c4d39 100644
> --- a/arch/openrisc/include/asm/Kbuild
> +++ b/arch/openrisc/include/asm/Kbuild
> @@ -28,12 +28,13 @@ generic-y += module.h
> generic-y += pci.h
> generic-y += percpu.h
> generic-y += preempt.h
> -generic-y += qspinlock_types.h
> -generic-y += qspinlock.h
> -generic-y += qrwlock_types.h
> generic-y += qrwlock.h
> +generic-y += qrwlock_types.h
> +generic-y += qspinlock.h
> +generic-y += qspinlock_types.h
> generic-y += sections.h
> generic-y += segment.h
> +generic-y += simd.h
> generic-y += string.h
> generic-y += switch_to.h
> generic-y += topology.h
> diff --git a/arch/parisc/include/asm/Kbuild b/arch/parisc/include/asm/Kbuild
> index 2013d639e735..97970b4d05ab 100644
> --- a/arch/parisc/include/asm/Kbuild
> +++ b/arch/parisc/include/asm/Kbuild
> @@ -17,6 +17,7 @@ generic-y += percpu.h
> generic-y += preempt.h
> generic-y += seccomp.h
> generic-y += segment.h
> +generic-y += simd.h
> generic-y += topology.h
> generic-y += trace_clock.h
> generic-y += user.h
> diff --git a/arch/powerpc/include/asm/Kbuild b/arch/powerpc/include/asm/Kbuild
> index 3196d227e351..64290f48e733 100644
> --- a/arch/powerpc/include/asm/Kbuild
> +++ b/arch/powerpc/include/asm/Kbuild
> @@ -4,7 +4,8 @@ generic-y += irq_regs.h
> generic-y += irq_work.h
> generic-y += local64.h
> generic-y += mcs_spinlock.h
> +generic-y += msi.h
> generic-y += preempt.h
> generic-y += rwsem.h
> +generic-y += simd.h
> generic-y += vtime.h
> -generic-y += msi.h
> diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild
> index efdbe311e936..6669b7374c0a 100644
> --- a/arch/riscv/include/asm/Kbuild
> +++ b/arch/riscv/include/asm/Kbuild
> @@ -5,9 +5,9 @@ generic-y += compat.h
> generic-y += cputime.h
> generic-y += device.h
> generic-y += div64.h
> -generic-y += dma.h
> generic-y += dma-contiguous.h
> generic-y += dma-mapping.h
> +generic-y += dma.h
> generic-y += emergency-restart.h
> generic-y += errno.h
> generic-y += exec.h
> @@ -46,6 +46,7 @@ generic-y += setup.h
> generic-y += shmbuf.h
> generic-y += shmparam.h
> generic-y += signal.h
> +generic-y += simd.h
> generic-y += socket.h
> generic-y += sockios.h
> generic-y += stat.h
> diff --git a/arch/s390/include/asm/Kbuild b/arch/s390/include/asm/Kbuild
> index e3239772887a..7a26dc6ce815 100644
> --- a/arch/s390/include/asm/Kbuild
> +++ b/arch/s390/include/asm/Kbuild
> @@ -7,9 +7,9 @@ generated-y += unistd_nr.h
> generic-y += asm-offsets.h
> generic-y += cacheflush.h
> generic-y += device.h
> +generic-y += div64.h
> generic-y += dma-contiguous.h
> generic-y += dma-mapping.h
> -generic-y += div64.h
> generic-y += emergency-restart.h
> generic-y += export.h
> generic-y += fb.h
> @@ -22,6 +22,7 @@ generic-y += mcs_spinlock.h
> generic-y += mm-arch-hooks.h
> generic-y += preempt.h
> generic-y += rwsem.h
> +generic-y += simd.h
> generic-y += trace_clock.h
> generic-y += unaligned.h
> generic-y += word-at-a-time.h
> diff --git a/arch/sh/include/asm/Kbuild b/arch/sh/include/asm/Kbuild
> index 6a5609a55965..8e64ff35a933 100644
> --- a/arch/sh/include/asm/Kbuild
> +++ b/arch/sh/include/asm/Kbuild
> @@ -16,6 +16,7 @@ generic-y += percpu.h
> generic-y += preempt.h
> generic-y += rwsem.h
> generic-y += serial.h
> +generic-y += simd.h
> generic-y += sizes.h
> generic-y += trace_clock.h
> generic-y += xor.h
> diff --git a/arch/sparc/include/asm/Kbuild b/arch/sparc/include/asm/Kbuild
> index 410b263ef5c8..72b9e08fb350 100644
> --- a/arch/sparc/include/asm/Kbuild
> +++ b/arch/sparc/include/asm/Kbuild
> @@ -17,5 +17,6 @@ generic-y += msi.h
> generic-y += preempt.h
> generic-y += rwsem.h
> generic-y += serial.h
> +generic-y += simd.h
> generic-y += trace_clock.h
> generic-y += word-at-a-time.h
> diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild
> index b10dde6cb793..d37288b08dd2 100644
> --- a/arch/um/include/asm/Kbuild
> +++ b/arch/um/include/asm/Kbuild
> @@ -16,15 +16,16 @@ generic-y += io.h
> generic-y += irq_regs.h
> generic-y += irq_work.h
> generic-y += kdebug.h
> +generic-y += kprobes.h
> generic-y += mcs_spinlock.h
> generic-y += mm-arch-hooks.h
> generic-y += param.h
> generic-y += pci.h
> generic-y += percpu.h
> generic-y += preempt.h
> +generic-y += simd.h
> generic-y += switch_to.h
> generic-y += topology.h
> generic-y += trace_clock.h
> generic-y += word-at-a-time.h
> generic-y += xor.h
> -generic-y += kprobes.h
> diff --git a/arch/unicore32/include/asm/Kbuild b/arch/unicore32/include/asm/Kbuild
> index bfc7abe77905..98a908720bbd 100644
> --- a/arch/unicore32/include/asm/Kbuild
> +++ b/arch/unicore32/include/asm/Kbuild
> @@ -27,6 +27,7 @@ generic-y += preempt.h
> generic-y += sections.h
> generic-y += segment.h
> generic-y += serial.h
> +generic-y += simd.h
> generic-y += sizes.h
> generic-y += syscalls.h
> generic-y += topology.h
> diff --git a/arch/x86/include/asm/simd.h b/arch/x86/include/asm/simd.h
> index a341c878e977..4aad7f158dcb 100644
> --- a/arch/x86/include/asm/simd.h
> +++ b/arch/x86/include/asm/simd.h
> @@ -1,4 +1,11 @@
> -/* SPDX-License-Identifier: GPL-2.0 */
> +/* SPDX-License-Identifier: GPL-2.0
> + *
> + * Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
> + */
> +
> +#include <linux/simd.h>
> +#ifndef _ASM_SIMD_H
> +#define _ASM_SIMD_H
>
> #include <asm/fpu/api.h>
>
> @@ -10,3 +17,38 @@ static __must_check inline bool may_use_simd(void)
> {
> return irq_fpu_usable();
> }
> +
> +static inline void simd_get(simd_context_t *ctx)
> +{
> +#if !defined(CONFIG_UML)
> + *ctx = may_use_simd() ? HAVE_FULL_SIMD : HAVE_NO_SIMD;
> +#else
> + *ctx = HAVE_NO_SIMD;
> +#endif
> +}
> +
> +static inline void simd_put(simd_context_t *ctx)
> +{
> +#if !defined(CONFIG_UML)
> + if (*ctx & HAVE_SIMD_IN_USE)
> + kernel_fpu_end();
> +#endif
> + *ctx = HAVE_NO_SIMD;
> +}
> +
> +static __must_check inline bool simd_use(simd_context_t *ctx)
> +{
> +#if !defined(CONFIG_UML)
> + if (!(*ctx & HAVE_FULL_SIMD))
> + return false;
> + if (*ctx & HAVE_SIMD_IN_USE)
> + return true;
> + kernel_fpu_begin();
> + *ctx |= HAVE_SIMD_IN_USE;
> + return true;
> +#else
> + return false;
> +#endif
> +}
> +
> +#endif /* _ASM_SIMD_H */
> diff --git a/arch/xtensa/include/asm/Kbuild b/arch/xtensa/include/asm/Kbuild
> index 82c756431b49..7950f359649d 100644
> --- a/arch/xtensa/include/asm/Kbuild
> +++ b/arch/xtensa/include/asm/Kbuild
> @@ -24,6 +24,7 @@ generic-y += percpu.h
> generic-y += preempt.h
> generic-y += rwsem.h
> generic-y += sections.h
> +generic-y += simd.h
> generic-y += topology.h
> generic-y += trace_clock.h
> generic-y += word-at-a-time.h
> diff --git a/include/asm-generic/simd.h b/include/asm-generic/simd.h
> index d0343d58a74a..b3dd61ac010e 100644
> --- a/include/asm-generic/simd.h
> +++ b/include/asm-generic/simd.h
> @@ -1,5 +1,9 @@
> /* SPDX-License-Identifier: GPL-2.0 */
>
> +#include <linux/simd.h>
> +#ifndef _ASM_SIMD_H
> +#define _ASM_SIMD_H
> +
> #include <linux/hardirq.h>
>
> /*
> @@ -13,3 +17,19 @@ static __must_check inline bool may_use_simd(void)
> {
> return !in_interrupt();
> }
> +
> +static inline void simd_get(simd_context_t *ctx)
> +{
> + *ctx = HAVE_NO_SIMD;
> +}
> +
> +static inline void simd_put(simd_context_t *ctx)
> +{
> +}
> +
> +static __must_check inline bool simd_use(simd_context_t *ctx)
> +{
> + return false;
> +}
> +
> +#endif /* _ASM_SIMD_H */
> diff --git a/include/linux/simd.h b/include/linux/simd.h
> new file mode 100644
> index 000000000000..33bba21012ff
> --- /dev/null
> +++ b/include/linux/simd.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0
> + *
> + * Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
> + */
> +
> +#ifndef _SIMD_H
> +#define _SIMD_H
> +
> +typedef enum {
> + HAVE_NO_SIMD = 1 << 0,
> + HAVE_FULL_SIMD = 1 << 1,
> + HAVE_SIMD_IN_USE = 1 << 31
> +} simd_context_t;
> +
> +#include <linux/sched.h>
> +#include <asm/simd.h>
> +
> +static inline void simd_relax(simd_context_t *ctx)
> +{
> +#ifdef CONFIG_PREEMPT
> + if ((*ctx & HAVE_SIMD_IN_USE) && need_resched()) {
> + simd_put(ctx);
> + simd_get(ctx);
> + }
> +#endif
Could we return a bool here indicating whether we rescheduled or not?
In some cases, we could pass that into the asm code as a 'reload'
param, allowing repeated loads of key schedules, round constant tables
or S-boxes to be elided.
> +}
> +
> +#endif /* _SIMD_H */
> --
> 2.19.0
>
^ permalink raw reply
* Re: [PATCH 00/15] octeontx2-af: Add RVU Admin Function driver
From: Arnd Bergmann @ 2018-09-28 8:27 UTC (permalink / raw)
To: Sunil Kovvuri; +Cc: Networking, David Miller, linux-soc, sgoutham
In-Reply-To: <1538114872-10820-1-git-send-email-sunil.kovvuri@gmail.com>
On Fri, Sep 28, 2018 at 8:08 AM <sunil.kovvuri@gmail.com> wrote:
>
> From: Sunil Goutham <sgoutham@marvell.com>
>
> Resource virtualization unit (RVU) on Marvell's OcteonTX2 SOC maps HW
> resources from the network, crypto and other functional blocks into
> PCI-compatible physical and virtual functions. Each functional block
> again has multiple local functions (LFs) for provisioning to PCI devices.
> RVU supports multiple PCIe SRIOV physical functions (PFs) and virtual
> functions (VFs). PF0 is called the administrative / admin function (AF)
> and has privileges to provision RVU functional block's LFs to each of the
> PF/VF.
>
> RVU managed networking functional blocks
> - Network pool allocator (NPA)
> - Network interface controller (NIX)
> - Network parser CAM (NPC)
> - Schedule/Synchronize/Order unit (SSO)
>
> RVU managed non-networking functional blocks
> - Crypto accelerator (CPT)
> - Scheduled timers unit (TIM)
> - Schedule/Synchronize/Order unit (SSO)
> Used for both networking and non networking usecases
> - Compression (upcoming in future variants of the silicons)
>
> Resource provisioning examples
> - A PF/VF with NIX-LF & NPA-LF resources works as a pure network device
> - A PF/VF with CPT-LF resource works as a pure cyrpto offload device.
>
> This admin function driver neither receives any data nor processes it i.e
> no I/O, a configuration only driver.
>
> PF/VFs communicates with AF via a shared memory region (mailbox). Upon
> receiving requests from PF/VF, AF does resource provisioning and other
> HW configuration. AF is always attached to host, but PF/VFs may be used
> by host kernel itself, or attached to VMs or to userspace applications
> like DPDK etc. So AF has to handle provisioning/configuration requests
> sent by any device from any domain.
>
> This patch series adds logic for the following
> - RVU AF driver with functional blocks provisioning support.
> - Mailbox infrastructure for communication between AF and PFs.
> - CGX (MAC controller) driver which communicates with firmware for
> managing physical ethernet interfaces. AF collects info from this
> driver and forwards the same to the PF/VFs uaing these interfaces.
>
> This is the first set of patches out of 80+ patches.
Hi Sunil,
This looks good overall, thanks for moving it to drivers/net
as I suggested. I found two more minor remaining issues here,
it should not be a problem to change the code accordingly.
One thing I was missing here is a revision history for the patch
series, it would have been helpful to get a list of things you have
changed compared to the previous submission, what triggered
those changes, and which issues (if any) are still remaining.
Best add the full revision history in the next (hopefully final)
round for these patches, and use 'git format-patch --reroll-count'
to add the patchset revision number.
I assume that the next set of patches will be more controversial
as you get to the point that adds the user interface. I have
not looked at those patches yet, so it may just be fine, assuming
all the configuration is done using devlink. Please keep me
on Cc for the future series, even if I don't participate in those
reviews as much.
Thanks,
Arnd
^ permalink raw reply
* [PATCH v4 bpf-next 09/10] samples/bpf: extend test_cgrp2_attach2 test to use per-cpu cgroup storage
From: Roman Gushchin @ 2018-09-28 14:45 UTC (permalink / raw)
To: netdev@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, Kernel Team, Roman Gushchin,
Daniel Borkmann, Alexei Starovoitov
In-Reply-To: <20180928144452.5284-1-guro@fb.com>
This commit extends the test_cgrp2_attach2 test to cover per-cpu
cgroup storage. Bpf program will use shared and per-cpu cgroup
storages simultaneously, so a better coverage of corresponding
core code will be achieved.
Expected output:
$ ./test_cgrp2_attach2
Attached DROP prog. This ping in cgroup /foo should fail...
ping: sendmsg: Operation not permitted
Attached DROP prog. This ping in cgroup /foo/bar should fail...
ping: sendmsg: Operation not permitted
Attached PASS prog. This ping in cgroup /foo/bar should pass...
Detached PASS from /foo/bar while DROP is attached to /foo.
This ping in cgroup /foo/bar should fail...
ping: sendmsg: Operation not permitted
Attached PASS from /foo/bar and detached DROP from /foo.
This ping in cgroup /foo/bar should pass...
### override:PASS
### multi:PASS
Signed-off-by: Roman Gushchin <guro@fb.com>
Acked-by: Song Liu <songliubraving@fb.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Alexei Starovoitov <ast@kernel.org>
---
samples/bpf/test_cgrp2_attach2.c | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/samples/bpf/test_cgrp2_attach2.c b/samples/bpf/test_cgrp2_attach2.c
index 180f9d813bca..d7b68ef5ba79 100644
--- a/samples/bpf/test_cgrp2_attach2.c
+++ b/samples/bpf/test_cgrp2_attach2.c
@@ -209,7 +209,7 @@ static int map_fd = -1;
static int prog_load_cnt(int verdict, int val)
{
- int cgroup_storage_fd;
+ int cgroup_storage_fd, percpu_cgroup_storage_fd;
if (map_fd < 0)
map_fd = bpf_create_map(BPF_MAP_TYPE_ARRAY, 4, 8, 1, 0);
@@ -225,6 +225,14 @@ static int prog_load_cnt(int verdict, int val)
return -1;
}
+ percpu_cgroup_storage_fd = bpf_create_map(
+ BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE,
+ sizeof(struct bpf_cgroup_storage_key), 8, 0, 0);
+ if (percpu_cgroup_storage_fd < 0) {
+ printf("failed to create map '%s'\n", strerror(errno));
+ return -1;
+ }
+
struct bpf_insn prog[] = {
BPF_MOV32_IMM(BPF_REG_0, 0),
BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_0, -4), /* *(u32 *)(fp - 4) = r0 */
@@ -235,11 +243,20 @@ static int prog_load_cnt(int verdict, int val)
BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
BPF_MOV64_IMM(BPF_REG_1, val), /* r1 = 1 */
BPF_RAW_INSN(BPF_STX | BPF_XADD | BPF_DW, BPF_REG_0, BPF_REG_1, 0, 0), /* xadd r0 += r1 */
+
BPF_LD_MAP_FD(BPF_REG_1, cgroup_storage_fd),
BPF_MOV64_IMM(BPF_REG_2, 0),
BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_local_storage),
BPF_MOV64_IMM(BPF_REG_1, val),
BPF_RAW_INSN(BPF_STX | BPF_XADD | BPF_W, BPF_REG_0, BPF_REG_1, 0, 0),
+
+ BPF_LD_MAP_FD(BPF_REG_1, percpu_cgroup_storage_fd),
+ BPF_MOV64_IMM(BPF_REG_2, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_get_local_storage),
+ BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_0, 0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, 0x1),
+ BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_3, 0),
+
BPF_MOV64_IMM(BPF_REG_0, verdict), /* r0 = verdict */
BPF_EXIT_INSN(),
};
--
2.17.1
^ permalink raw reply related
* [PATCH v4 bpf-next 08/10] selftests/bpf: extend the storage test to test per-cpu cgroup storage
From: Roman Gushchin @ 2018-09-28 14:45 UTC (permalink / raw)
To: netdev@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, Kernel Team, Roman Gushchin,
Daniel Borkmann, Alexei Starovoitov
In-Reply-To: <20180928144452.5284-1-guro@fb.com>
This test extends the cgroup storage test to use per-cpu flavor
of the cgroup storage as well.
The test initializes a per-cpu cgroup storage to some non-zero initial
value (1000), and then simple bumps a per-cpu counter each time
the shared counter is atomically incremented. Then it reads all
per-cpu areas from the userspace side, and checks that the sum
of values adds to the expected sum.
Expected output:
$ ./test_cgroup_storage
test_cgroup_storage:PASS
Signed-off-by: Roman Gushchin <guro@fb.com>
Acked-by: Song Liu <songliubraving@fb.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/test_cgroup_storage.c | 60 ++++++++++++++++++-
1 file changed, 57 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/bpf/test_cgroup_storage.c b/tools/testing/selftests/bpf/test_cgroup_storage.c
index 4e196e3bfecf..f44834155f25 100644
--- a/tools/testing/selftests/bpf/test_cgroup_storage.c
+++ b/tools/testing/selftests/bpf/test_cgroup_storage.c
@@ -4,6 +4,7 @@
#include <linux/filter.h>
#include <stdio.h>
#include <stdlib.h>
+#include <sys/sysinfo.h>
#include "bpf_rlimit.h"
#include "cgroup_helpers.h"
@@ -15,6 +16,14 @@ char bpf_log_buf[BPF_LOG_BUF_SIZE];
int main(int argc, char **argv)
{
struct bpf_insn prog[] = {
+ BPF_LD_MAP_FD(BPF_REG_1, 0), /* percpu map fd */
+ BPF_MOV64_IMM(BPF_REG_2, 0), /* flags, not used */
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_local_storage),
+ BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_0, 0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, 0x1),
+ BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_3, 0),
+
BPF_LD_MAP_FD(BPF_REG_1, 0), /* map fd */
BPF_MOV64_IMM(BPF_REG_2, 0), /* flags, not used */
BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
@@ -28,9 +37,18 @@ int main(int argc, char **argv)
};
size_t insns_cnt = sizeof(prog) / sizeof(struct bpf_insn);
int error = EXIT_FAILURE;
- int map_fd, prog_fd, cgroup_fd;
+ int map_fd, percpu_map_fd, prog_fd, cgroup_fd;
struct bpf_cgroup_storage_key key;
unsigned long long value;
+ unsigned long long *percpu_value;
+ int cpu, nproc;
+
+ nproc = get_nprocs_conf();
+ percpu_value = malloc(sizeof(*percpu_value) * nproc);
+ if (!percpu_value) {
+ printf("Not enough memory for per-cpu area (%d cpus)\n", nproc);
+ goto err;
+ }
map_fd = bpf_create_map(BPF_MAP_TYPE_CGROUP_STORAGE, sizeof(key),
sizeof(value), 0, 0);
@@ -39,7 +57,15 @@ int main(int argc, char **argv)
goto out;
}
- prog[0].imm = map_fd;
+ percpu_map_fd = bpf_create_map(BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE,
+ sizeof(key), sizeof(value), 0, 0);
+ if (percpu_map_fd < 0) {
+ printf("Failed to create map: %s\n", strerror(errno));
+ goto out;
+ }
+
+ prog[0].imm = percpu_map_fd;
+ prog[7].imm = map_fd;
prog_fd = bpf_load_program(BPF_PROG_TYPE_CGROUP_SKB,
prog, insns_cnt, "GPL", 0,
bpf_log_buf, BPF_LOG_BUF_SIZE);
@@ -77,7 +103,15 @@ int main(int argc, char **argv)
}
if (bpf_map_lookup_elem(map_fd, &key, &value)) {
- printf("Failed to lookup cgroup storage\n");
+ printf("Failed to lookup cgroup storage 0\n");
+ goto err;
+ }
+
+ for (cpu = 0; cpu < nproc; cpu++)
+ percpu_value[cpu] = 1000;
+
+ if (bpf_map_update_elem(percpu_map_fd, &key, percpu_value, 0)) {
+ printf("Failed to update the data in the cgroup storage\n");
goto err;
}
@@ -120,11 +154,31 @@ int main(int argc, char **argv)
goto err;
}
+ /* Check the final value of the counter in the percpu local storage */
+
+ for (cpu = 0; cpu < nproc; cpu++)
+ percpu_value[cpu] = 0;
+
+ if (bpf_map_lookup_elem(percpu_map_fd, &key, percpu_value)) {
+ printf("Failed to lookup the per-cpu cgroup storage\n");
+ goto err;
+ }
+
+ value = 0;
+ for (cpu = 0; cpu < nproc; cpu++)
+ value += percpu_value[cpu];
+
+ if (value != nproc * 1000 + 6) {
+ printf("Unexpected data in the per-cpu cgroup storage\n");
+ goto err;
+ }
+
error = 0;
printf("test_cgroup_storage:PASS\n");
err:
cleanup_cgroup_environment();
+ free(percpu_value);
out:
return error;
--
2.17.1
^ permalink raw reply related
* [PATCH v4 bpf-next 07/10] selftests/bpf: add verifier per-cpu cgroup storage tests
From: Roman Gushchin @ 2018-09-28 14:45 UTC (permalink / raw)
To: netdev@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, Kernel Team, Roman Gushchin,
Daniel Borkmann, Alexei Starovoitov
In-Reply-To: <20180928144452.5284-1-guro@fb.com>
This commits adds verifier tests covering per-cpu cgroup storage
functionality. There are 6 new tests, which are exactly the same
as for shared cgroup storage, but do use per-cpu cgroup storage
map.
Expected output:
$ ./test_verifier
#0/u add+sub+mul OK
#0/p add+sub+mul OK
...
#286/p invalid cgroup storage access 6 OK
#287/p valid per-cpu cgroup storage access OK
#288/p invalid per-cpu cgroup storage access 1 OK
#289/p invalid per-cpu cgroup storage access 2 OK
#290/p invalid per-cpu cgroup storage access 3 OK
#291/p invalid per-cpu cgroup storage access 4 OK
#292/p invalid per-cpu cgroup storage access 5 OK
#293/p invalid per-cpu cgroup storage access 6 OK
#294/p multiple registers share map_lookup_elem result OK
...
#662/p mov64 src == dst OK
#663/p mov64 src != dst OK
Summary: 914 PASSED, 0 SKIPPED, 0 FAILED
Signed-off-by: Roman Gushchin <guro@fb.com>
Acked-by: Song Liu <songliubraving@fb.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/test_verifier.c | 139 +++++++++++++++++++-
1 file changed, 133 insertions(+), 6 deletions(-)
diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c
index 67c412d19c09..c7d25f23baf9 100644
--- a/tools/testing/selftests/bpf/test_verifier.c
+++ b/tools/testing/selftests/bpf/test_verifier.c
@@ -68,6 +68,7 @@ struct bpf_test {
int fixup_prog2[MAX_FIXUPS];
int fixup_map_in_map[MAX_FIXUPS];
int fixup_cgroup_storage[MAX_FIXUPS];
+ int fixup_percpu_cgroup_storage[MAX_FIXUPS];
const char *errstr;
const char *errstr_unpriv;
uint32_t retval;
@@ -4676,7 +4677,7 @@ static struct bpf_test tests[] = {
.prog_type = BPF_PROG_TYPE_CGROUP_SKB,
},
{
- "invalid per-cgroup storage access 3",
+ "invalid cgroup storage access 3",
.insns = {
BPF_MOV64_IMM(BPF_REG_2, 0),
BPF_LD_MAP_FD(BPF_REG_1, 0),
@@ -4743,6 +4744,121 @@ static struct bpf_test tests[] = {
.errstr = "get_local_storage() doesn't support non-zero flags",
.prog_type = BPF_PROG_TYPE_CGROUP_SKB,
},
+ {
+ "valid per-cpu cgroup storage access",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_2, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_local_storage),
+ BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_1),
+ BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_percpu_cgroup_storage = { 1 },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "invalid per-cpu cgroup storage access 1",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_2, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_local_storage),
+ BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_1),
+ BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map1 = { 1 },
+ .result = REJECT,
+ .errstr = "cannot pass map_type 1 into func bpf_get_local_storage",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "invalid per-cpu cgroup storage access 2",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_2, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 1),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_local_storage),
+ BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .result = REJECT,
+ .errstr = "fd 1 is not pointing to valid bpf_map",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "invalid per-cpu cgroup storage access 3",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_2, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_local_storage),
+ BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 256),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_percpu_cgroup_storage = { 1 },
+ .result = REJECT,
+ .errstr = "invalid access to map value, value_size=64 off=256 size=4",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "invalid per-cpu cgroup storage access 4",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_2, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_local_storage),
+ BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, -2),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_1),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 1),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_cgroup_storage = { 1 },
+ .result = REJECT,
+ .errstr = "invalid access to map value, value_size=64 off=-2 size=4",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "invalid per-cpu cgroup storage access 5",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_2, 7),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_local_storage),
+ BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_1),
+ BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_percpu_cgroup_storage = { 1 },
+ .result = REJECT,
+ .errstr = "get_local_storage() doesn't support non-zero flags",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "invalid per-cpu cgroup storage access 6",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_1),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_local_storage),
+ BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_1),
+ BPF_ALU64_IMM(BPF_AND, BPF_REG_0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_percpu_cgroup_storage = { 1 },
+ .result = REJECT,
+ .errstr = "get_local_storage() doesn't support non-zero flags",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
{
"multiple registers share map_lookup_elem result",
.insns = {
@@ -12615,15 +12731,17 @@ static int create_map_in_map(void)
return outer_map_fd;
}
-static int create_cgroup_storage(void)
+static int create_cgroup_storage(bool percpu)
{
+ enum bpf_map_type type = percpu ? BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE :
+ BPF_MAP_TYPE_CGROUP_STORAGE;
int fd;
- fd = bpf_create_map(BPF_MAP_TYPE_CGROUP_STORAGE,
- sizeof(struct bpf_cgroup_storage_key),
+ fd = bpf_create_map(type, sizeof(struct bpf_cgroup_storage_key),
TEST_DATA_LEN, 0, 0);
if (fd < 0)
- printf("Failed to create array '%s'!\n", strerror(errno));
+ printf("Failed to create cgroup storage '%s'!\n",
+ strerror(errno));
return fd;
}
@@ -12641,6 +12759,7 @@ static void do_test_fixup(struct bpf_test *test, struct bpf_insn *prog,
int *fixup_prog2 = test->fixup_prog2;
int *fixup_map_in_map = test->fixup_map_in_map;
int *fixup_cgroup_storage = test->fixup_cgroup_storage;
+ int *fixup_percpu_cgroup_storage = test->fixup_percpu_cgroup_storage;
if (test->fill_helper)
test->fill_helper(test);
@@ -12710,12 +12829,20 @@ static void do_test_fixup(struct bpf_test *test, struct bpf_insn *prog,
}
if (*fixup_cgroup_storage) {
- map_fds[7] = create_cgroup_storage();
+ map_fds[7] = create_cgroup_storage(false);
do {
prog[*fixup_cgroup_storage].imm = map_fds[7];
fixup_cgroup_storage++;
} while (*fixup_cgroup_storage);
}
+
+ if (*fixup_percpu_cgroup_storage) {
+ map_fds[8] = create_cgroup_storage(true);
+ do {
+ prog[*fixup_percpu_cgroup_storage].imm = map_fds[8];
+ fixup_percpu_cgroup_storage++;
+ } while (*fixup_percpu_cgroup_storage);
+ }
}
static void do_test_single(struct bpf_test *test, bool unpriv,
--
2.17.1
^ permalink raw reply related
* [PATCH v4 bpf-next 06/10] bpftool: add support for PERCPU_CGROUP_STORAGE maps
From: Roman Gushchin @ 2018-09-28 14:45 UTC (permalink / raw)
To: netdev@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, Kernel Team, Roman Gushchin,
Daniel Borkmann, Alexei Starovoitov
In-Reply-To: <20180928144452.5284-1-guro@fb.com>
This commit adds support for BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE
map type.
Signed-off-by: Roman Gushchin <guro@fb.com>
Acked-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Acked-by: Song Liu <songliubraving@fb.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Alexei Starovoitov <ast@kernel.org>
---
tools/bpf/bpftool/map.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c
index e22fbe8b975f..6003e9598973 100644
--- a/tools/bpf/bpftool/map.c
+++ b/tools/bpf/bpftool/map.c
@@ -72,13 +72,15 @@ static const char * const map_type_name[] = {
[BPF_MAP_TYPE_SOCKHASH] = "sockhash",
[BPF_MAP_TYPE_CGROUP_STORAGE] = "cgroup_storage",
[BPF_MAP_TYPE_REUSEPORT_SOCKARRAY] = "reuseport_sockarray",
+ [BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE] = "percpu_cgroup_storage",
};
static bool map_is_per_cpu(__u32 type)
{
return type == BPF_MAP_TYPE_PERCPU_HASH ||
type == BPF_MAP_TYPE_PERCPU_ARRAY ||
- type == BPF_MAP_TYPE_LRU_PERCPU_HASH;
+ type == BPF_MAP_TYPE_LRU_PERCPU_HASH ||
+ type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE;
}
static bool map_is_map_of_maps(__u32 type)
--
2.17.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox