* Re: [PATCH net-next 00/10] Cleanups, minor additions & fixes for HNS3 driver
From: David Miller @ 2018-09-28 17:38 UTC (permalink / raw)
To: salil.mehta
Cc: yisen.zhuang, lipeng321, mehta.salil, netdev, linux-kernel,
linuxarm
In-Reply-To: <20180926182840.28392-1-salil.mehta@huawei.com>
From: Salil Mehta <salil.mehta@huawei.com>
Date: Wed, 26 Sep 2018 19:28:30 +0100
> This patch-set contains cleans-ups, minor changes and fixes to the HNS3 driver.
Series applied, thank you.
^ permalink raw reply
* Re: [PATCH net-next] net: dsa: b53: Fix build with B53_SRAB enabled and B53_SERDES=m
From: David Miller @ 2018-09-28 17:35 UTC (permalink / raw)
To: f.fainelli; +Cc: arnd, andrew, vivien.didelot, netdev, linux-kernel
In-Reply-To: <c0af4a4c-6656-e014-a4bb-0bacfbe1d468@gmail.com>
From: Florian Fainelli <f.fainelli@gmail.com>
Date: Thu, 27 Sep 2018 12:07:01 -0700
> On 09/27/2018 03:02 AM, Arnd Bergmann wrote:
>> When B53_SERDES is a loadable module, a built-in srab driver still
>> cannot reach it, so the previous fix is incomplete:
>>
>> b53_srab.c:(.text+0x3f4): undefined reference to `b53_serdes_init'
>> drivers/net/dsa/b53/b53_srab.o:(.rodata+0xe64): undefined reference to `b53_serdes_link_state'
>> drivers/net/dsa/b53/b53_srab.o:(.rodata+0xe74): undefined reference to `b53_serdes_link_set'
>> drivers/net/dsa/b53/b53_srab.o:(.rodata+0xe88): undefined reference to `b53_serdes_an_restart'
>> drivers/net/dsa/b53/b53_srab.o:(.rodata+0xea0): undefined reference to `b53_serdes_phylink_validate'
>> drivers/net/dsa/b53/b53_srab.o:(.rodata+0xea4): undefined reference to `b53_serdes_config'
>>
>> Add a Kconfig dependency that forces srab to also be a module
>> in this case, but allow it to be built-in when serdes is
>> disabled or built-in.
>>
>> Fixes: 7a8c7f5c30f9 ("net: dsa: b53: Fix build with B53_SRAB enabled and not B53_SERDES")
>> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
>
> Acked-by: Florian Fainelli <f.fainelli@gmail.com>
Applied, thanks Arnd.
^ permalink raw reply
* Re: [PATCH net-next] net: dsa: b53: Fix build with B53_SRAB enabled and B53_SERDES=m
From: David Miller @ 2018-09-28 17:33 UTC (permalink / raw)
To: arnd; +Cc: f.fainelli, andrew, vivien.didelot, netdev, linux-kernel
In-Reply-To: <20180927100244.692546-1-arnd@arndb.de>
From: Arnd Bergmann <arnd@arndb.de>
Date: Thu, 27 Sep 2018 12:02:38 +0200
> When B53_SERDES is a loadable module, a built-in srab driver still
> cannot reach it, so the previous fix is incomplete:
>
> b53_srab.c:(.text+0x3f4): undefined reference to `b53_serdes_init'
> drivers/net/dsa/b53/b53_srab.o:(.rodata+0xe64): undefined reference to `b53_serdes_link_state'
> drivers/net/dsa/b53/b53_srab.o:(.rodata+0xe74): undefined reference to `b53_serdes_link_set'
> drivers/net/dsa/b53/b53_srab.o:(.rodata+0xe88): undefined reference to `b53_serdes_an_restart'
> drivers/net/dsa/b53/b53_srab.o:(.rodata+0xea0): undefined reference to `b53_serdes_phylink_validate'
> drivers/net/dsa/b53/b53_srab.o:(.rodata+0xea4): undefined reference to `b53_serdes_config'
>
> Add a Kconfig dependency that forces srab to also be a module
> in this case, but allow it to be built-in when serdes is
> disabled or built-in.
>
> Fixes: 7a8c7f5c30f9 ("net: dsa: b53: Fix build with B53_SRAB enabled and not B53_SERDES")
> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Applied, thanks Arnd.
^ permalink raw reply
* Re: [PATCH RFC net-next] openvswitch: Queue upcalls to userspace in per-port round-robin order
From: Pravin Shelar @ 2018-09-28 17:15 UTC (permalink / raw)
To: Stefano Brivio
Cc: ovs dev, Justin Pettit, Linux Kernel Network Developers,
Jiri Benc
In-Reply-To: <20180926115821.7cd9fed8@epycfail>
On Wed, Sep 26, 2018 at 2:58 AM Stefano Brivio <sbrivio-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
> Hi Pravin,
>
> On Wed, 15 Aug 2018 00:19:39 -0700
> Pravin Shelar <pshelar-LZ6Gd1LRuIk@public.gmane.org> wrote:
>
> > I understand fairness has cost, but we need to find right balance
> > between performance and fairness. Current fairness scheme is a
> > lockless algorithm without much computational overhead, did you try to
> > improve current algorithm so that it uses less number of ports.
>
> In the end, we tried harder as you suggested, and found out a way to
> avoid the need for per-thread sets of per-vport sockets: with a few
> changes, a process-wide set of per-vport sockets is actually enough to
> maintain the current fairness behaviour.
>
> Further, we can get rid of duplicate fd events if the EPOLLEXCLUSIVE
> epoll() flag is available, which improves performance noticeably. This
> is explained in more detail in the commit message of the ovs-vswitchd
> patch Matteo wrote [1], now merged.
>
> This approach solves the biggest issue (i.e. huge amount of netlink
> sockets) in ovs-vswitchd itself, without the need for kernel changes.
>
Thanks for working on this issue.
> It doesn't address some proposed improvements though, that is, it does
> nothing to improve the current fairness scheme, it won't allow neither
> the configurable fairness criteria proposed by Ben, nor usage of BPF
> maps for extensibility as suggested by William.
>
> I would however say that those topics bear definitely lower priority,
> and perhaps addressing them at all becomes overkill now that we don't
> any longer have a serious issue.
>
> [1] https://patchwork.ozlabs.org/patch/974304/
Nice!
Thanks,
Pravin.
^ permalink raw reply
* Re: INFO: trying to register non-static key in tun_chr_write_iter
From: Eric Dumazet @ 2018-09-28 17:06 UTC (permalink / raw)
To: syzbot, brouer, davem, edumazet, jasowang, linux-kernel, mst,
netdev, sd, syzkaller-bugs
In-Reply-To: <000000000000c1119b0576efc7f6@google.com>
On 09/28/2018 08:05 AM, syzbot wrote:
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit: 100811936f89 bpf: test_bpf: add init_net to dev for flow_d..
> git tree: bpf-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=145378e6400000
> kernel config: https://syzkaller.appspot.com/x/.config?x=443816db871edd66
> dashboard link: https://syzkaller.appspot.com/bug?extid=e662df0ac1d753b57e80
> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>
> Unfortunately, I don't have any reproducer for this crash yet.
>
I took a look at this report, I will send a patch series fixing the issues
I found in tun driver.
^ permalink raw reply
* Re: [PATCH net-next] geneve: fix ttl inherit type
From: Phil Sutter @ 2018-09-28 10:38 UTC (permalink / raw)
To: Hangbin Liu; +Cc: netdev, David Miller, Stephen Hemminger, David Ahern
In-Reply-To: <1538096998-20937-1-git-send-email-liuhangbin@gmail.com>
On Fri, Sep 28, 2018 at 09:09:58AM +0800, Hangbin Liu wrote:
> Phil pointed out that there is a mismatch between vxlan and geneve ttl
> inherit. We should define it as a flag and use nla_put_flag to export this
> opiton.
Same typo here. :)
Apart from that, LGTM!
Thanks, Phil
^ permalink raw reply
* Re: [PATCH net] vxlan: use nla_put_flag for ttl inherit
From: Phil Sutter @ 2018-09-28 10:37 UTC (permalink / raw)
To: Hangbin Liu; +Cc: netdev, David Miller, Stephen Hemminger, David Ahern
In-Reply-To: <1538096906-20866-1-git-send-email-liuhangbin@gmail.com>
On Fri, Sep 28, 2018 at 09:08:26AM +0800, Hangbin Liu wrote:
> Phil pointed out that there is a mismatch between vxlan and geneve ttl inherit.
> We should define it as a flag and use nla_put_flag to export this opiton.
s/opiton/option/
Apart from that, LGTM!
Thanks, Phil
^ permalink raw reply
* pull-request: ieee802154 for net 2018-09-28
From: Stefan Schmidt @ 2018-09-28 10:20 UTC (permalink / raw)
To: davem; +Cc: linux-wpan, alex.aring, netdev
Hello Dave.
An update from ieee802154 for your *net* tree.
Some cleanup patches throughout the drivers from the Huawei tag team
Yue Haibing and Zhong Jiang.
Xue is replacing some magic numbers with defines in his mcr20a driver.
regards
Stefan Schmidt
The following changes since commit 10bc6a6042c900934a097988b5d50e3cf3f91781:
r8169: fix autoneg issue on resume with RTL8168E (2018-09-20 19:58:47 -0700)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/sschmidt/wpan.git ieee802154-for-davem-2018-09-28
for you to fetch changes up to d6d1cd2578c4da0764ad334e3411c1c1b1557f58:
ieee802154: mcr20a: Replace magic number with constants (2018-09-27 17:22:48 +0200)
----------------------------------------------------------------
Xue Liu (1):
ieee802154: mcr20a: Replace magic number with constants
YueHaibing (1):
ieee802154: Use kmemdup instead of duplicating it in ca8210_test_int_driver_write
zhong jiang (2):
ieee802154: remove unecessary condition check before debugfs_remove_recursive
ieee802154: ca8210: remove redundant condition check before debugfs_remove
drivers/net/ieee802154/adf7242.c | 3 +--
drivers/net/ieee802154/ca8210.c | 6 ++----
drivers/net/ieee802154/mcr20a.c | 8 ++++----
3 files changed, 7 insertions(+), 10 deletions(-)
^ permalink raw reply
* [PATCH net 4/8] rxrpc: Emit BUSY packets when supposed to rather than ABORTs
From: David Howells @ 2018-09-28 10:10 UTC (permalink / raw)
To: netdev; +Cc: dhowells, linux-afs, linux-kernel
In-Reply-To: <153812939111.10469.179989653628053410.stgit@warthog.procyon.org.uk>
In the input path, a received sk_buff can be marked for rejection by
setting RXRPC_SKB_MARK_* in skb->mark and, if needed, some auxiliary data
(such as an abort code) in skb->priority. The rejection is handled by
queueing the sk_buff up for dealing with in process context. The output
code reads the mark and priority and, theoretically, generates an
appropriate response packet.
However, if RXRPC_SKB_MARK_BUSY is set, this isn't noticed and an ABORT
message with a random abort code is generated (since skb->priority wasn't
set to anything).
Fix this by outputting the appropriate sort of packet.
Also, whilst we're at it, most of the marks are no longer used, so remove
them and rename the remaining two to something more obvious.
Fixes: 248f219cb8bc ("rxrpc: Rewrite the data and ack handling code")
Signed-off-by: David Howells <dhowells@redhat.com>
---
net/rxrpc/ar-internal.h | 13 ++++---------
net/rxrpc/call_accept.c | 6 +++---
net/rxrpc/input.c | 2 +-
net/rxrpc/output.c | 23 ++++++++++++++++++-----
4 files changed, 26 insertions(+), 18 deletions(-)
diff --git a/net/rxrpc/ar-internal.h b/net/rxrpc/ar-internal.h
index 9fcb3e197b14..e8861cb78070 100644
--- a/net/rxrpc/ar-internal.h
+++ b/net/rxrpc/ar-internal.h
@@ -40,17 +40,12 @@ struct rxrpc_crypt {
struct rxrpc_connection;
/*
- * Mark applied to socket buffers.
+ * Mark applied to socket buffers in skb->mark. skb->priority is used
+ * to pass supplementary information.
*/
enum rxrpc_skb_mark {
- RXRPC_SKB_MARK_DATA, /* data message */
- RXRPC_SKB_MARK_FINAL_ACK, /* final ACK received message */
- RXRPC_SKB_MARK_BUSY, /* server busy message */
- RXRPC_SKB_MARK_REMOTE_ABORT, /* remote abort message */
- RXRPC_SKB_MARK_LOCAL_ABORT, /* local abort message */
- RXRPC_SKB_MARK_NET_ERROR, /* network error message */
- RXRPC_SKB_MARK_LOCAL_ERROR, /* local error message */
- RXRPC_SKB_MARK_NEW_CALL, /* local error message */
+ RXRPC_SKB_MARK_REJECT_BUSY, /* Reject with BUSY */
+ RXRPC_SKB_MARK_REJECT_ABORT, /* Reject with ABORT (code in skb->priority) */
};
/*
diff --git a/net/rxrpc/call_accept.c b/net/rxrpc/call_accept.c
index 9d1e298b784c..e88f131c1d7f 100644
--- a/net/rxrpc/call_accept.c
+++ b/net/rxrpc/call_accept.c
@@ -353,7 +353,7 @@ struct rxrpc_call *rxrpc_new_incoming_call(struct rxrpc_local *local,
trace_rxrpc_abort(0, "INV", sp->hdr.cid, sp->hdr.callNumber, sp->hdr.seq,
RX_INVALID_OPERATION, EOPNOTSUPP);
- skb->mark = RXRPC_SKB_MARK_LOCAL_ABORT;
+ skb->mark = RXRPC_SKB_MARK_REJECT_ABORT;
skb->priority = RX_INVALID_OPERATION;
_leave(" = NULL [service]");
return NULL;
@@ -364,7 +364,7 @@ struct rxrpc_call *rxrpc_new_incoming_call(struct rxrpc_local *local,
rx->sk.sk_state == RXRPC_CLOSE) {
trace_rxrpc_abort(0, "CLS", sp->hdr.cid, sp->hdr.callNumber,
sp->hdr.seq, RX_INVALID_OPERATION, ESHUTDOWN);
- skb->mark = RXRPC_SKB_MARK_LOCAL_ABORT;
+ skb->mark = RXRPC_SKB_MARK_REJECT_ABORT;
skb->priority = RX_INVALID_OPERATION;
_leave(" = NULL [close]");
call = NULL;
@@ -373,7 +373,7 @@ struct rxrpc_call *rxrpc_new_incoming_call(struct rxrpc_local *local,
call = rxrpc_alloc_incoming_call(rx, local, conn, skb);
if (!call) {
- skb->mark = RXRPC_SKB_MARK_BUSY;
+ skb->mark = RXRPC_SKB_MARK_REJECT_BUSY;
_leave(" = NULL [busy]");
call = NULL;
goto out;
diff --git a/net/rxrpc/input.c b/net/rxrpc/input.c
index 7f9ed3a60b9a..b0f12471f5e7 100644
--- a/net/rxrpc/input.c
+++ b/net/rxrpc/input.c
@@ -1354,7 +1354,7 @@ void rxrpc_data_ready(struct sock *udp_sk)
protocol_error:
skb->priority = RX_PROTOCOL_ERROR;
post_abort:
- skb->mark = RXRPC_SKB_MARK_LOCAL_ABORT;
+ skb->mark = RXRPC_SKB_MARK_REJECT_ABORT;
reject_packet:
trace_rxrpc_rx_done(skb->mark, skb->priority);
rxrpc_reject_packet(local, skb);
diff --git a/net/rxrpc/output.c b/net/rxrpc/output.c
index 8a4da3fe96df..e8fb8922bca8 100644
--- a/net/rxrpc/output.c
+++ b/net/rxrpc/output.c
@@ -524,7 +524,7 @@ void rxrpc_reject_packets(struct rxrpc_local *local)
struct kvec iov[2];
size_t size;
__be32 code;
- int ret;
+ int ret, ioc;
_enter("%d", local->debug_id);
@@ -532,7 +532,6 @@ void rxrpc_reject_packets(struct rxrpc_local *local)
iov[0].iov_len = sizeof(whdr);
iov[1].iov_base = &code;
iov[1].iov_len = sizeof(code);
- size = sizeof(whdr) + sizeof(code);
msg.msg_name = &srx.transport;
msg.msg_control = NULL;
@@ -540,17 +539,31 @@ void rxrpc_reject_packets(struct rxrpc_local *local)
msg.msg_flags = 0;
memset(&whdr, 0, sizeof(whdr));
- whdr.type = RXRPC_PACKET_TYPE_ABORT;
while ((skb = skb_dequeue(&local->reject_queue))) {
rxrpc_see_skb(skb, rxrpc_skb_rx_seen);
sp = rxrpc_skb(skb);
+ switch (skb->mark) {
+ case RXRPC_SKB_MARK_REJECT_BUSY:
+ whdr.type = RXRPC_PACKET_TYPE_BUSY;
+ size = sizeof(whdr);
+ ioc = 1;
+ break;
+ case RXRPC_SKB_MARK_REJECT_ABORT:
+ whdr.type = RXRPC_PACKET_TYPE_ABORT;
+ code = htonl(skb->priority);
+ size = sizeof(whdr) + sizeof(code);
+ ioc = 2;
+ break;
+ default:
+ rxrpc_free_skb(skb, rxrpc_skb_rx_freed);
+ continue;
+ }
+
if (rxrpc_extract_addr_from_skb(local, &srx, skb) == 0) {
msg.msg_namelen = srx.transport_len;
- code = htonl(skb->priority);
-
whdr.epoch = htonl(sp->hdr.epoch);
whdr.cid = htonl(sp->hdr.cid);
whdr.callNumber = htonl(sp->hdr.callNumber);
^ permalink raw reply related
* Re: [PATCH 13/15] octeontx2-af: Add support for CGX link management
From: Arnd Bergmann @ 2018-09-28 10:00 UTC (permalink / raw)
To: Sunil Kovvuri; +Cc: Networking, David Miller, linux-soc, lcherian, nmani
In-Reply-To: <CA+sq2Cc5p+Hdm573is2rAWMeGZKS+=iUiYHU3+soTR4CSpgcHg@mail.gmail.com>
On Fri, Sep 28, 2018 at 11:31 AM Sunil Kovvuri <sunil.kovvuri@gmail.com> wrote:
> On Fri, Sep 28, 2018 at 1:49 PM Arnd Bergmann <arnd@arndb.de> wrote:
> > On Fri, Sep 28, 2018 at 8:09 AM <sunil.kovvuri@gmail.com> wrote:
> This communication between firmware and kernel driver is done using couple of
> scratch registers. With limited space available we had to resort to bitfields.
> Your point about endianness is correct. As you might be aware that the device to
> which this driver registers to, is only found on OcteonTx2 SOC which operates
> in a standalone mode. As of now we are not targeting to make these drivers
> work in big-endian mode.
>
> We would prefer to make big-endian related changes later on, test them
> fully and
> submit patches, would this be okay ?
>
> If not we will define big endian bit fields in all command structures
> and re-submit.
Generally speaking, I think all drivers should be written in a portable way,
since you never know whether they will be reused in a different way later,
copied into other drivers, or used in creative ways by your users.
I would therefore recommend to change the bitfields now, but not
necessarily test big-endian builds. Well-written code should just work
out of the box in either endianess, and if someone finds a problem
there later, they can fix it themselves or report it. Just don't use
nonportable code intentionally.
Arnd
^ permalink raw reply
* [PATCH net-next] selftests/tls: Fix recv(MSG_PEEK) & splice() test cases
From: Vakul Garg @ 2018-09-28 16:18 UTC (permalink / raw)
To: netdev
Cc: linux-kselftest, linux-kernel, shuah, davejwatson, davem, doronrk,
Vakul Garg
TLS test cases splice_from_pipe, send_and_splice &
recv_peek_multiple_records expect to receive a given nummber of bytes
and then compare them against the number of bytes which were sent.
Therefore, system call recv() must not return before receiving the
requested number of bytes, otherwise the subsequent memcmp() fails.
This patch passes MSG_WAITALL flag to recv() so that it does not return
prematurely before requested number of bytes are copied to receive
buffer.
Signed-off-by: Vakul Garg <vakul.garg@nxp.com>
---
tools/testing/selftests/net/tls.c | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/tools/testing/selftests/net/tls.c b/tools/testing/selftests/net/tls.c
index 11d54c36ae49..fac68d710f35 100644
--- a/tools/testing/selftests/net/tls.c
+++ b/tools/testing/selftests/net/tls.c
@@ -288,7 +288,7 @@ TEST_F(tls, splice_from_pipe)
ASSERT_GE(pipe(p), 0);
EXPECT_GE(write(p[1], mem_send, send_len), 0);
EXPECT_GE(splice(p[0], NULL, self->fd, NULL, send_len, 0), 0);
- EXPECT_GE(recv(self->cfd, mem_recv, send_len, 0), 0);
+ EXPECT_EQ(recv(self->cfd, mem_recv, send_len, MSG_WAITALL), send_len);
EXPECT_EQ(memcmp(mem_send, mem_recv, send_len), 0);
}
@@ -322,13 +322,13 @@ TEST_F(tls, send_and_splice)
ASSERT_GE(pipe(p), 0);
EXPECT_EQ(send(self->fd, test_str, send_len2, 0), send_len2);
- EXPECT_NE(recv(self->cfd, buf, send_len2, 0), -1);
+ EXPECT_EQ(recv(self->cfd, buf, send_len2, MSG_WAITALL), send_len2);
EXPECT_EQ(memcmp(test_str, buf, send_len2), 0);
EXPECT_GE(write(p[1], mem_send, send_len), send_len);
EXPECT_GE(splice(p[0], NULL, self->fd, NULL, send_len, 0), send_len);
- EXPECT_GE(recv(self->cfd, mem_recv, send_len, 0), 0);
+ EXPECT_EQ(recv(self->cfd, mem_recv, send_len, MSG_WAITALL), send_len);
EXPECT_EQ(memcmp(mem_send, mem_recv, send_len), 0);
}
@@ -516,17 +516,17 @@ TEST_F(tls, recv_peek_multiple_records)
len = strlen(test_str_second) + 1;
EXPECT_EQ(send(self->fd, test_str_second, len, 0), len);
- len = sizeof(buf);
+ len = strlen(test_str_first);
memset(buf, 0, len);
- EXPECT_NE(recv(self->cfd, buf, len, MSG_PEEK), -1);
+ EXPECT_EQ(recv(self->cfd, buf, len, MSG_PEEK | MSG_WAITALL), len);
/* MSG_PEEK can only peek into the current record. */
- len = strlen(test_str_first) + 1;
+ len = strlen(test_str_first);
EXPECT_EQ(memcmp(test_str_first, buf, len), 0);
- len = sizeof(buf);
+ len = strlen(test_str) + 1;
memset(buf, 0, len);
- EXPECT_NE(recv(self->cfd, buf, len, 0), -1);
+ EXPECT_EQ(recv(self->cfd, buf, len, MSG_WAITALL), len);
/* Non-MSG_PEEK will advance strparser (and therefore record)
* however.
@@ -543,9 +543,9 @@ TEST_F(tls, recv_peek_multiple_records)
len = strlen(test_str_second) + 1;
EXPECT_EQ(send(self->fd, test_str_second, len, 0), len);
- len = sizeof(buf);
+ len = strlen(test_str) + 1;
memset(buf, 0, len);
- EXPECT_NE(recv(self->cfd, buf, len, MSG_PEEK), -1);
+ EXPECT_EQ(recv(self->cfd, buf, len, MSG_PEEK | MSG_WAITALL), len);
len = strlen(test_str) + 1;
EXPECT_EQ(memcmp(test_str, buf, len), 0);
--
2.13.6
^ permalink raw reply related
* Re: [PATCH bpf-next] bpf: permit CGROUP_DEVICE programs accessing helper bpf_get_current_cgroup_id()
From: Roman Gushchin @ 2018-09-28 9:53 UTC (permalink / raw)
To: Yonghong Song; +Cc: ast, daniel, netdev, kernel-team
In-Reply-To: <20180927213730.1224816-1-yhs@fb.com>
On Thu, Sep 27, 2018 at 02:37:30PM -0700, Yonghong Song wrote:
> Currently, helper bpf_get_current_cgroup_id() is not permitted
> for CGROUP_DEVICE type of programs. If the helper is used
> in such cases, the verifier will log the following error:
>
> 0: (bf) r6 = r1
> 1: (69) r7 = *(u16 *)(r6 +0)
> 2: (85) call bpf_get_current_cgroup_id#80
> unknown func bpf_get_current_cgroup_id#80
>
> The bpf_get_current_cgroup_id() is useful for CGROUP_DEVICE
> type of programs in order to customize action based on cgroup id.
> This patch added such a support.
>
> Cc: Roman Gushchin <guro@fb.com>
> Signed-off-by: Yonghong Song <yhs@fb.com>
Acked-by: Roman Gushchin <guro@fb.com>
Thanks, Yonghong!
^ permalink raw reply
* Re: [PATCHv3 bpf-next 07/12] bpf: Add helper to retrieve socket in BPF
From: Alexei Starovoitov @ 2018-09-28 9:46 UTC (permalink / raw)
To: Joe Stringer
Cc: daniel, netdev, ast, john.fastabend, tgraf, kafai, nitin.hande,
mauricio.vasquez
In-Reply-To: <20180927232659.14348-8-joe@wand.net.nz>
On Thu, Sep 27, 2018 at 04:26:54PM -0700, Joe Stringer wrote:
> This patch adds new BPF helper functions, bpf_sk_lookup_tcp() and
> bpf_sk_lookup_udp() which allows BPF programs to find out if there is a
> socket listening on this host, and returns a socket pointer which the
> BPF program can then access to determine, for instance, whether to
> forward or drop traffic. bpf_sk_lookup_xxx() may take a reference on the
> socket, so when a BPF program makes use of this function, it must
> subsequently pass the returned pointer into the newly added sk_release()
> to return the reference.
>
> By way of example, the following pseudocode would filter inbound
> connections at XDP if there is no corresponding service listening for
> the traffic:
>
> struct bpf_sock_tuple tuple;
> struct bpf_sock_ops *sk;
>
> populate_tuple(ctx, &tuple); // Extract the 5tuple from the packet
> sk = bpf_sk_lookup_tcp(ctx, &tuple, sizeof tuple, netns, 0);
> if (!sk) {
> // Couldn't find a socket listening for this traffic. Drop.
> return TC_ACT_SHOT;
> }
> bpf_sk_release(sk);
> return TC_ACT_OK;
>
> Signed-off-by: Joe Stringer <joe@wand.net.nz>
> ---
> v2: Rework 'struct bpf_sock_tuple' to allow passing a packet pointer
> Limit netns_id field to 32 bits
> Fix compile error with CONFIG_IPV6 enabled
> Allow direct packet access from helper
>
> v3: Fix release of caller_net when netns is not specified.
> Use skb->sk to find caller net when skb->dev is unavailable.
> Remove flags argument to sk_release()
> Define the semantics of the new helpers more clearly.
Acked-by: Alexei Starovoitov <ast@kernel.org>
^ permalink raw reply
* Re: [PATCHv3 bpf-next 04/12] bpf: Add PTR_TO_SOCKET verifier type
From: Alexei Starovoitov @ 2018-09-28 9:44 UTC (permalink / raw)
To: Joe Stringer
Cc: daniel, netdev, ast, john.fastabend, tgraf, kafai, nitin.hande,
mauricio.vasquez
In-Reply-To: <20180927232659.14348-5-joe@wand.net.nz>
On Thu, Sep 27, 2018 at 04:26:51PM -0700, Joe Stringer wrote:
> Teach the verifier a little bit about a new type of pointer, a
> PTR_TO_SOCKET. This pointer type is accessed from BPF through the
> 'struct bpf_sock' structure.
>
> Signed-off-by: Joe Stringer <joe@wand.net.nz>
Acked-by: Alexei Starovoitov <ast@kernel.org>
^ permalink raw reply
* Re: [PATCH net-next v6 07/23] zinc: ChaCha20 ARM and ARM64 implementations
From: Ard Biesheuvel @ 2018-09-28 16:01 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: Linux Kernel Mailing List, <netdev@vger.kernel.org>,
open list:HARDWARE RANDOM NUMBER GENERATOR CORE, David S. Miller,
Greg Kroah-Hartman, Samuel Neves, Andy Lutomirski,
Jean-Philippe Aumasson, Russell King, linux-arm-kernel
In-Reply-To: <20180925145622.29959-8-Jason@zx2c4.com>
On 25 September 2018 at 16:56, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> These wire Andy Polyakov's implementations up to the kernel for ARMv7,8
> NEON, and introduce Eric Biggers' ultra-fast scalar implementation for
> CPUs without NEON or for CPUs with slow NEON (Cortex-A5,7).
>
> This commit does the following:
> - Adds the glue code for the assembly implementations.
> - Renames the ARMv8 code into place, since it can at this point be
> used wholesale.
> - Merges Andy Polyakov's ARMv7 NEON code with Eric Biggers' <=ARMv7
> scalar code.
>
> Commit note: Eric Biggers' scalar code is brand new, and quite possibly
> prematurely added to this commit, and so it may require a bit of revision.
>
Please put comments like this below the ---
> This commit delivers approximately the same or much better performance than
> the existing crypto API's code and has been measured to do as such on:
>
> - ARM1176JZF-S [ARMv6]
> - Cortex-A7 [ARMv7]
> - Cortex-A8 [ARMv7]
> - Cortex-A9 [ARMv7]
> - Cortex-A17 [ARMv7]
> - Cortex-A53 [ARMv8]
> - Cortex-A55 [ARMv8]
> - Cortex-A73 [ARMv8]
> - Cortex-A75 [ARMv8]
>
> Interestingly, Andy Polyakov's scalar code is slower than Eric Biggers',
> but is also significantly shorter. This has the advantage that it does
> not evict other code from L1 cache -- particularly on ARM11 chips -- and
> so in certain circumstances it can actually be faster. However, it wasn't
> found that this had an affect on any code existing in the kernel today.
>
> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
> Co-authored-by: Eric Biggers <ebiggers@google.com>
> Cc: Samuel Neves <sneves@dei.uc.pt>
> Cc: Andy Lutomirski <luto@kernel.org>
> Cc: Greg KH <gregkh@linuxfoundation.org>
> Cc: Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
> Cc: Russell King <linux@armlinux.org.uk>
> Cc: linux-arm-kernel@lists.infradead.org
> ---
> lib/zinc/Makefile | 2 +
> lib/zinc/chacha20/chacha20-arm-glue.h | 88 +++
> ...acha20-arm-cryptogams.S => chacha20-arm.S} | 502 ++++++++++++++++--
> ...20-arm64-cryptogams.S => chacha20-arm64.S} | 0
> lib/zinc/chacha20/chacha20.c | 2 +
> 5 files changed, 556 insertions(+), 38 deletions(-)
> create mode 100644 lib/zinc/chacha20/chacha20-arm-glue.h
> rename lib/zinc/chacha20/{chacha20-arm-cryptogams.S => chacha20-arm.S} (71%)
> rename lib/zinc/chacha20/{chacha20-arm64-cryptogams.S => chacha20-arm64.S} (100%)
>
> diff --git a/lib/zinc/Makefile b/lib/zinc/Makefile
> index 223a0816c918..e47f64e12bbd 100644
> --- a/lib/zinc/Makefile
> +++ b/lib/zinc/Makefile
> @@ -4,4 +4,6 @@ ccflags-$(CONFIG_ZINC_DEBUG) += -DDEBUG
>
> zinc_chacha20-y := chacha20/chacha20.o
> zinc_chacha20-$(CONFIG_ZINC_ARCH_X86_64) += chacha20/chacha20-x86_64.o
> +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM) += chacha20/chacha20-arm.o
> +zinc_chacha20-$(CONFIG_ZINC_ARCH_ARM64) += chacha20/chacha20-arm64.o
Are these CONFIG_ symbols defined anywhere at this point?
In any case, I don't think these is a reason for these, at least not
on ARM/arm64. The 64-bitness is implied in both cases, and the
dependency on !CPU_32v3 you introduce (looking at the version of
Kconfig at the end of the series) seems spurious to me. Was that added
because of some kbuild robot report? (we don't support ARMv3 in the
kernel but ARCH_RPC is built in v3 mode because of historical reasons
while the actual core is a v4)
> obj-$(CONFIG_ZINC_CHACHA20) += zinc_chacha20.o
> diff --git a/lib/zinc/chacha20/chacha20-arm-glue.h b/lib/zinc/chacha20/chacha20-arm-glue.h
> new file mode 100644
> index 000000000000..86cce851ed02
> --- /dev/null
> +++ b/lib/zinc/chacha20/chacha20-arm-glue.h
> @@ -0,0 +1,88 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
> +/*
> + * Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
> + */
> +
> +#include <asm/hwcap.h>
> +#include <asm/neon.h>
> +#if defined(CONFIG_ARM)
> +#include <asm/system_info.h>
> +#include <asm/cputype.h>
> +#endif
> +
> +asmlinkage void chacha20_arm(u8 *out, const u8 *in, const size_t len,
> + const u32 key[8], const u32 counter[4]);
> +#if defined(CONFIG_ARM)
> +asmlinkage void hchacha20_arm(const u32 state[16], u32 out[8]);
> +#endif
> +#if defined(CONFIG_KERNEL_MODE_NEON)
> +asmlinkage void chacha20_neon(u8 *out, const u8 *in, const size_t len,
> + const u32 key[8], const u32 counter[4]);
> +#endif
> +
No need to make asmlinkage declarations conditional
> +static bool chacha20_use_neon __ro_after_init;
> +
> +static void __init chacha20_fpu_init(void)
> +{
> +#if defined(CONFIG_ARM64)
> + chacha20_use_neon = elf_hwcap & HWCAP_ASIMD;
> +#elif defined(CONFIG_ARM)
> + switch (read_cpuid_part()) {
> + case ARM_CPU_PART_CORTEX_A7:
> + case ARM_CPU_PART_CORTEX_A5:
> + /* The Cortex-A7 and Cortex-A5 do not perform well with the NEON
> + * implementation but do incredibly with the scalar one and use
> + * less power.
> + */
> + break;
> + default:
> + chacha20_use_neon = elf_hwcap & HWCAP_NEON;
> + }
> +#endif
> +}
> +
> +static inline bool chacha20_arch(struct chacha20_ctx *state, u8 *dst,
> + const u8 *src, size_t len,
> + simd_context_t *simd_context)
> +{
> +#if defined(CONFIG_KERNEL_MODE_NEON)
if (IS_ENABLED())
> + if (chacha20_use_neon && len >= CHACHA20_BLOCK_SIZE * 3 &&
> + simd_use(simd_context))
> + chacha20_neon(dst, src, len, state->key, state->counter);
> + else
> +#endif
> + chacha20_arm(dst, src, len, state->key, state->counter);
> +
> + state->counter[0] += (len + 63) / 64;
> + return true;
> +}
> +
> +static inline bool hchacha20_arch(u32 derived_key[CHACHA20_KEY_WORDS],
> + const u8 nonce[HCHACHA20_NONCE_SIZE],
> + const u8 key[HCHACHA20_KEY_SIZE],
> + simd_context_t *simd_context)
> +{
> +#if defined(CONFIG_ARM)
> + u32 x[] = { CHACHA20_CONSTANT_EXPA,
> + CHACHA20_CONSTANT_ND_3,
> + CHACHA20_CONSTANT_2_BY,
> + CHACHA20_CONSTANT_TE_K,
> + get_unaligned_le32(key + 0),
> + get_unaligned_le32(key + 4),
> + get_unaligned_le32(key + 8),
> + get_unaligned_le32(key + 12),
> + get_unaligned_le32(key + 16),
> + get_unaligned_le32(key + 20),
> + get_unaligned_le32(key + 24),
> + get_unaligned_le32(key + 28),
> + get_unaligned_le32(nonce + 0),
> + get_unaligned_le32(nonce + 4),
> + get_unaligned_le32(nonce + 8),
> + get_unaligned_le32(nonce + 12)
> + };
> + hchacha20_arm(x, derived_key);
"""
if (!IS_ENABLED(CONFIG_ARM))
return false;
hchacha20_arm(x, derived_key);
return true;
"""
and drop the #ifdefs
> + return true;
> +#else
> + return false;
> +#endif
> +}
> diff --git a/lib/zinc/chacha20/chacha20-arm-cryptogams.S b/lib/zinc/chacha20/chacha20-arm.S
> similarity index 71%
> rename from lib/zinc/chacha20/chacha20-arm-cryptogams.S
> rename to lib/zinc/chacha20/chacha20-arm.S
> index 770bab469171..5abedafcf129 100644
> --- a/lib/zinc/chacha20/chacha20-arm-cryptogams.S
> +++ b/lib/zinc/chacha20/chacha20-arm.S
> @@ -1,13 +1,475 @@
> /* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
> /*
> + * Copyright (C) 2018 Google, Inc.
> * Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
> * Copyright (C) 2006-2017 CRYPTOGAMS by <appro@openssl.org>. All Rights Reserved.
> - *
> - * This is based in part on Andy Polyakov's implementation from CRYPTOGAMS.
> */
>
> #include <linux/linkage.h>
>
> +/*
> + * The following scalar routine was written by Eric Biggers.
> + *
> + * Design notes:
> + *
> + * 16 registers would be needed to hold the state matrix, but only 14 are
> + * available because 'sp' and 'pc' cannot be used. So we spill the elements
> + * (x8, x9) to the stack and swap them out with (x10, x11). This adds one
> + * 'ldrd' and one 'strd' instruction per round.
> + *
> + * All rotates are performed using the implicit rotate operand accepted by the
> + * 'add' and 'eor' instructions. This is faster than using explicit rotate
> + * instructions. To make this work, we allow the values in the second and last
> + * rows of the ChaCha state matrix (rows 'b' and 'd') to temporarily have the
> + * wrong rotation amount. The rotation amount is then fixed up just in time
> + * when the values are used. 'brot' is the number of bits the values in row 'b'
> + * need to be rotated right to arrive at the correct values, and 'drot'
> + * similarly for row 'd'. (brot, drot) start out as (0, 0) but we make it such
> + * that they end up as (25, 24) after every round.
> + */
> +
> + // ChaCha state registers
> + X0 .req r0
> + X1 .req r1
> + X2 .req r2
> + X3 .req r3
> + X4 .req r4
> + X5 .req r5
> + X6 .req r6
> + X7 .req r7
> + X8_X10 .req r8 // shared by x8 and x10
> + X9_X11 .req r9 // shared by x9 and x11
> + X12 .req r10
> + X13 .req r11
> + X14 .req r12
> + X15 .req r14
> +
> +.Lexpand_32byte_k:
> + // "expand 32-byte k"
> + .word 0x61707865, 0x3320646e, 0x79622d32, 0x6b206574
> +
> +#ifdef __thumb2__
> +# define adrl adr
> +#endif
> +
> +.macro __rev out, in, t0, t1, t2
> +.if __LINUX_ARM_ARCH__ >= 6
> + rev \out, \in
> +.else
> + lsl \t0, \in, #24
> + and \t1, \in, #0xff00
> + and \t2, \in, #0xff0000
> + orr \out, \t0, \in, lsr #24
> + orr \out, \out, \t1, lsl #8
> + orr \out, \out, \t2, lsr #8
> +.endif
> +.endm
> +
> +.macro _le32_bswap x, t0, t1, t2
> +#ifdef __ARMEB__
> + __rev \x, \x, \t0, \t1, \t2
> +#endif
> +.endm
> +
> +.macro _le32_bswap_4x a, b, c, d, t0, t1, t2
> + _le32_bswap \a, \t0, \t1, \t2
> + _le32_bswap \b, \t0, \t1, \t2
> + _le32_bswap \c, \t0, \t1, \t2
> + _le32_bswap \d, \t0, \t1, \t2
> +.endm
> +
> +.macro __ldrd a, b, src, offset
> +#if __LINUX_ARM_ARCH__ >= 6
> + ldrd \a, \b, [\src, #\offset]
> +#else
> + ldr \a, [\src, #\offset]
> + ldr \b, [\src, #\offset + 4]
> +#endif
> +.endm
> +
> +.macro __strd a, b, dst, offset
> +#if __LINUX_ARM_ARCH__ >= 6
> + strd \a, \b, [\dst, #\offset]
> +#else
> + str \a, [\dst, #\offset]
> + str \b, [\dst, #\offset + 4]
> +#endif
> +.endm
> +
> +.macro _halfround a1, b1, c1, d1, a2, b2, c2, d2
> +
> + // a += b; d ^= a; d = rol(d, 16);
> + add \a1, \a1, \b1, ror #brot
> + add \a2, \a2, \b2, ror #brot
> + eor \d1, \a1, \d1, ror #drot
> + eor \d2, \a2, \d2, ror #drot
> + // drot == 32 - 16 == 16
> +
> + // c += d; b ^= c; b = rol(b, 12);
> + add \c1, \c1, \d1, ror #16
> + add \c2, \c2, \d2, ror #16
> + eor \b1, \c1, \b1, ror #brot
> + eor \b2, \c2, \b2, ror #brot
> + // brot == 32 - 12 == 20
> +
> + // a += b; d ^= a; d = rol(d, 8);
> + add \a1, \a1, \b1, ror #20
> + add \a2, \a2, \b2, ror #20
> + eor \d1, \a1, \d1, ror #16
> + eor \d2, \a2, \d2, ror #16
> + // drot == 32 - 8 == 24
> +
> + // c += d; b ^= c; b = rol(b, 7);
> + add \c1, \c1, \d1, ror #24
> + add \c2, \c2, \d2, ror #24
> + eor \b1, \c1, \b1, ror #20
> + eor \b2, \c2, \b2, ror #20
> + // brot == 32 - 7 == 25
> +.endm
> +
> +.macro _doubleround
> +
> + // column round
> +
> + // quarterrounds: (x0, x4, x8, x12) and (x1, x5, x9, x13)
> + _halfround X0, X4, X8_X10, X12, X1, X5, X9_X11, X13
> +
> + // save (x8, x9); restore (x10, x11)
> + __strd X8_X10, X9_X11, sp, 0
> + __ldrd X8_X10, X9_X11, sp, 8
> +
> + // quarterrounds: (x2, x6, x10, x14) and (x3, x7, x11, x15)
> + _halfround X2, X6, X8_X10, X14, X3, X7, X9_X11, X15
> +
> + .set brot, 25
> + .set drot, 24
> +
> + // diagonal round
> +
> + // quarterrounds: (x0, x5, x10, x15) and (x1, x6, x11, x12)
> + _halfround X0, X5, X8_X10, X15, X1, X6, X9_X11, X12
> +
> + // save (x10, x11); restore (x8, x9)
> + __strd X8_X10, X9_X11, sp, 8
> + __ldrd X8_X10, X9_X11, sp, 0
> +
> + // quarterrounds: (x2, x7, x8, x13) and (x3, x4, x9, x14)
> + _halfround X2, X7, X8_X10, X13, X3, X4, X9_X11, X14
> +.endm
> +
> +.macro _chacha_permute nrounds
> + .set brot, 0
> + .set drot, 0
> + .rept \nrounds / 2
> + _doubleround
> + .endr
> +.endm
> +
> +.macro _chacha nrounds
> +
> +.Lnext_block\@:
> + // Stack: unused0-unused1 x10-x11 x0-x15 OUT IN LEN
> + // Registers contain x0-x9,x12-x15.
> +
> + // Do the core ChaCha permutation to update x0-x15.
> + _chacha_permute \nrounds
> +
> + add sp, #8
> + // Stack: x10-x11 orig_x0-orig_x15 OUT IN LEN
> + // Registers contain x0-x9,x12-x15.
> + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
> +
> + // Free up some registers (r8-r12,r14) by pushing (x8-x9,x12-x15).
> + push {X8_X10, X9_X11, X12, X13, X14, X15}
> +
> + // Load (OUT, IN, LEN).
> + ldr r14, [sp, #96]
> + ldr r12, [sp, #100]
> + ldr r11, [sp, #104]
> +
> + orr r10, r14, r12
> +
> + // Use slow path if fewer than 64 bytes remain.
> + cmp r11, #64
> + blt .Lxor_slowpath\@
> +
> + // Use slow path if IN and/or OUT isn't 4-byte aligned. Needed even on
> + // ARMv6+, since ldmia and stmia (used below) still require alignment.
> + tst r10, #3
> + bne .Lxor_slowpath\@
> +
> + // Fast path: XOR 64 bytes of aligned data.
> +
> + // Stack: x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN
> + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is OUT.
> + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
> +
> + // x0-x3
> + __ldrd r8, r9, sp, 32
> + __ldrd r10, r11, sp, 40
> + add X0, X0, r8
> + add X1, X1, r9
> + add X2, X2, r10
> + add X3, X3, r11
> + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10
> + ldmia r12!, {r8-r11}
> + eor X0, X0, r8
> + eor X1, X1, r9
> + eor X2, X2, r10
> + eor X3, X3, r11
> + stmia r14!, {X0-X3}
> +
> + // x4-x7
> + __ldrd r8, r9, sp, 48
> + __ldrd r10, r11, sp, 56
> + add X4, r8, X4, ror #brot
> + add X5, r9, X5, ror #brot
> + ldmia r12!, {X0-X3}
> + add X6, r10, X6, ror #brot
> + add X7, r11, X7, ror #brot
> + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10
> + eor X4, X4, X0
> + eor X5, X5, X1
> + eor X6, X6, X2
> + eor X7, X7, X3
> + stmia r14!, {X4-X7}
> +
> + // x8-x15
> + pop {r0-r7} // (x8-x9,x12-x15,x10-x11)
> + __ldrd r8, r9, sp, 32
> + __ldrd r10, r11, sp, 40
> + add r0, r0, r8 // x8
> + add r1, r1, r9 // x9
> + add r6, r6, r10 // x10
> + add r7, r7, r11 // x11
> + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10
> + ldmia r12!, {r8-r11}
> + eor r0, r0, r8 // x8
> + eor r1, r1, r9 // x9
> + eor r6, r6, r10 // x10
> + eor r7, r7, r11 // x11
> + stmia r14!, {r0,r1,r6,r7}
> + ldmia r12!, {r0,r1,r6,r7}
> + __ldrd r8, r9, sp, 48
> + __ldrd r10, r11, sp, 56
> + add r2, r8, r2, ror #drot // x12
> + add r3, r9, r3, ror #drot // x13
> + add r4, r10, r4, ror #drot // x14
> + add r5, r11, r5, ror #drot // x15
> + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11
> + ldr r9, [sp, #72] // load LEN
> + eor r2, r2, r0 // x12
> + eor r3, r3, r1 // x13
> + eor r4, r4, r6 // x14
> + eor r5, r5, r7 // x15
> + subs r9, #64 // decrement and check LEN
> + stmia r14!, {r2-r5}
> +
> + beq .Ldone\@
> +
> +.Lprepare_for_next_block\@:
> +
> + // Stack: x0-x15 OUT IN LEN
> +
> + // Increment block counter (x12)
> + add r8, #1
> +
> + // Store updated (OUT, IN, LEN)
> + str r14, [sp, #64]
> + str r12, [sp, #68]
> + str r9, [sp, #72]
> +
> + mov r14, sp
> +
> + // Store updated block counter (x12)
> + str r8, [sp, #48]
> +
> + sub sp, #16
> +
> + // Reload state and do next block
> + ldmia r14!, {r0-r11} // load x0-x11
> + __strd r10, r11, sp, 8 // store x10-x11 before state
> + ldmia r14, {r10-r12,r14} // load x12-x15
> + b .Lnext_block\@
> +
> +.Lxor_slowpath\@:
> + // Slow path: < 64 bytes remaining, or unaligned input or output buffer.
> + // We handle it by storing the 64 bytes of keystream to the stack, then
> + // XOR-ing the needed portion with the data.
> +
> + // Allocate keystream buffer
> + sub sp, #64
> + mov r14, sp
> +
> + // Stack: ks0-ks15 x8-x9 x12-x15 x10-x11 orig_x0-orig_x15 OUT IN LEN
> + // Registers: r0-r7 are x0-x7; r8-r11 are free; r12 is IN; r14 is &ks0.
> + // x4-x7 are rotated by 'brot'; x12-x15 are rotated by 'drot'.
> +
> + // Save keystream for x0-x3
> + __ldrd r8, r9, sp, 96
> + __ldrd r10, r11, sp, 104
> + add X0, X0, r8
> + add X1, X1, r9
> + add X2, X2, r10
> + add X3, X3, r11
> + _le32_bswap_4x X0, X1, X2, X3, r8, r9, r10
> + stmia r14!, {X0-X3}
> +
> + // Save keystream for x4-x7
> + __ldrd r8, r9, sp, 112
> + __ldrd r10, r11, sp, 120
> + add X4, r8, X4, ror #brot
> + add X5, r9, X5, ror #brot
> + add X6, r10, X6, ror #brot
> + add X7, r11, X7, ror #brot
> + _le32_bswap_4x X4, X5, X6, X7, r8, r9, r10
> + add r8, sp, #64
> + stmia r14!, {X4-X7}
> +
> + // Save keystream for x8-x15
> + ldm r8, {r0-r7} // (x8-x9,x12-x15,x10-x11)
> + __ldrd r8, r9, sp, 128
> + __ldrd r10, r11, sp, 136
> + add r0, r0, r8 // x8
> + add r1, r1, r9 // x9
> + add r6, r6, r10 // x10
> + add r7, r7, r11 // x11
> + _le32_bswap_4x r0, r1, r6, r7, r8, r9, r10
> + stmia r14!, {r0,r1,r6,r7}
> + __ldrd r8, r9, sp, 144
> + __ldrd r10, r11, sp, 152
> + add r2, r8, r2, ror #drot // x12
> + add r3, r9, r3, ror #drot // x13
> + add r4, r10, r4, ror #drot // x14
> + add r5, r11, r5, ror #drot // x15
> + _le32_bswap_4x r2, r3, r4, r5, r9, r10, r11
> + stmia r14, {r2-r5}
> +
> + // Stack: ks0-ks15 unused0-unused7 x0-x15 OUT IN LEN
> + // Registers: r8 is block counter, r12 is IN.
> +
> + ldr r9, [sp, #168] // LEN
> + ldr r14, [sp, #160] // OUT
> + cmp r9, #64
> + mov r0, sp
> + movle r1, r9
> + movgt r1, #64
> + // r1 is number of bytes to XOR, in range [1, 64]
> +
> +.if __LINUX_ARM_ARCH__ < 6
> + orr r2, r12, r14
> + tst r2, #3 // IN or OUT misaligned?
> + bne .Lxor_next_byte\@
> +.endif
> +
> + // XOR a word at a time
> +.rept 16
> + subs r1, #4
> + blt .Lxor_words_done\@
> + ldr r2, [r12], #4
> + ldr r3, [r0], #4
> + eor r2, r2, r3
> + str r2, [r14], #4
> +.endr
> + b .Lxor_slowpath_done\@
> +.Lxor_words_done\@:
> + ands r1, r1, #3
> + beq .Lxor_slowpath_done\@
> +
> + // XOR a byte at a time
> +.Lxor_next_byte\@:
> + ldrb r2, [r12], #1
> + ldrb r3, [r0], #1
> + eor r2, r2, r3
> + strb r2, [r14], #1
> + subs r1, #1
> + bne .Lxor_next_byte\@
> +
> +.Lxor_slowpath_done\@:
> + subs r9, #64
> + add sp, #96
> + bgt .Lprepare_for_next_block\@
> +
> +.Ldone\@:
> +.endm // _chacha
> +
> +/*
> + * void chacha20_arm(u8 *out, const u8 *in, size_t len, const u32 key[8],
> + * const u32 iv[4]);
> + */
> +ENTRY(chacha20_arm)
> + cmp r2, #0 // len == 0?
> + bxeq lr
> +
> + push {r0-r2,r4-r11,lr}
> +
> + // Push state x0-x15 onto stack.
> + // Also store an extra copy of x10-x11 just before the state.
> +
> + ldr r4, [sp, #48] // iv
> + mov r0, sp
> + sub sp, #80
> +
> + // iv: x12-x15
> + ldm r4, {X12,X13,X14,X15}
> + stmdb r0!, {X12,X13,X14,X15}
> +
> + // key: x4-x11
> + __ldrd X8_X10, X9_X11, r3, 24
> + __strd X8_X10, X9_X11, sp, 8
> + stmdb r0!, {X8_X10, X9_X11}
> + ldm r3, {X4-X9_X11}
> + stmdb r0!, {X4-X9_X11}
> +
> + // constants: x0-x3
> + adrl X3, .Lexpand_32byte_k
> + ldm X3, {X0-X3}
> + __strd X0, X1, sp, 16
> + __strd X2, X3, sp, 24
> +
> + _chacha 20
> +
> + add sp, #76
> + pop {r4-r11, pc}
> +ENDPROC(chacha20_arm)
> +
> +/*
> + * void hchacha20_arm(const u32 state[16], u32 out[8]);
> + */
> +ENTRY(hchacha20_arm)
> + push {r1,r4-r11,lr}
> +
> + mov r14, r0
> + ldmia r14!, {r0-r11} // load x0-x11
> + push {r10-r11} // store x10-x11 to stack
> + ldm r14, {r10-r12,r14} // load x12-x15
> + sub sp, #8
> +
> + _chacha_permute 20
> +
> + // Skip over (unused0-unused1, x10-x11)
> + add sp, #16
> +
> + // Fix up rotations of x12-x15
> + ror X12, X12, #drot
> + ror X13, X13, #drot
> + pop {r4} // load 'out'
> + ror X14, X14, #drot
> + ror X15, X15, #drot
> +
> + // Store (x0-x3,x12-x15) to 'out'
> + stm r4, {X0,X1,X2,X3,X12,X13,X14,X15}
> +
> + pop {r4-r11,pc}
> +ENDPROC(hchacha20_arm)
> +
> +#ifdef CONFIG_KERNEL_MODE_NEON
> +/*
> + * This following NEON routine was ported from Andy Polyakov's implementation
> + * from CRYPTOGAMS. It begins with parts of the CRYPTOGAMS scalar routine,
> + * since certain NEON code paths actually branch to it.
> + */
> +
> .text
> #if defined(__thumb2__) || defined(__clang__)
> .syntax unified
> @@ -22,39 +484,6 @@
> #define ldrhsb ldrbhs
> #endif
>
> -.align 5
> -.Lsigma:
> -.long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral
> -.Lone:
> -.long 1,0,0,0
> -.word -1
> -
> -.align 5
> -ENTRY(chacha20_arm)
> - ldr r12,[sp,#0] @ pull pointer to counter and nonce
> - stmdb sp!,{r0-r2,r4-r11,lr}
> - cmp r2,#0 @ len==0?
> -#ifdef __thumb2__
> - itt eq
> -#endif
> - addeq sp,sp,#4*3
> - beq .Lno_data_arm
> - ldmia r12,{r4-r7} @ load counter and nonce
> - sub sp,sp,#4*(16) @ off-load area
> -#if __LINUX_ARM_ARCH__ < 7 && !defined(__thumb2__)
> - sub r14,pc,#100 @ .Lsigma
> -#else
> - adr r14,.Lsigma @ .Lsigma
> -#endif
> - stmdb sp!,{r4-r7} @ copy counter and nonce
> - ldmia r3,{r4-r11} @ load key
> - ldmia r14,{r0-r3} @ load sigma
> - stmdb sp!,{r4-r11} @ copy key
> - stmdb sp!,{r0-r3} @ copy sigma
> - str r10,[sp,#4*(16+10)] @ off-load "rx"
> - str r11,[sp,#4*(16+11)] @ off-load "rx"
> - b .Loop_outer_enter
> -
> .align 4
> .Loop_outer:
> ldmia sp,{r0-r9} @ load key material
> @@ -748,11 +1177,8 @@ ENTRY(chacha20_arm)
>
> .Ldone:
> add sp,sp,#4*(32+3)
> -.Lno_data_arm:
> ldmia sp!,{r4-r11,pc}
> -ENDPROC(chacha20_arm)
>
> -#ifdef CONFIG_KERNEL_MODE_NEON
> .align 5
> .Lsigma2:
> .long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral
> diff --git a/lib/zinc/chacha20/chacha20-arm64-cryptogams.S b/lib/zinc/chacha20/chacha20-arm64.S
> similarity index 100%
> rename from lib/zinc/chacha20/chacha20-arm64-cryptogams.S
> rename to lib/zinc/chacha20/chacha20-arm64.S
> diff --git a/lib/zinc/chacha20/chacha20.c b/lib/zinc/chacha20/chacha20.c
> index 4354b874a6a5..fc4f74fca653 100644
> --- a/lib/zinc/chacha20/chacha20.c
> +++ b/lib/zinc/chacha20/chacha20.c
> @@ -16,6 +16,8 @@
>
> #if defined(CONFIG_ZINC_ARCH_X86_64)
> #include "chacha20-x86_64-glue.h"
> +#elif defined(CONFIG_ZINC_ARCH_ARM) || defined(CONFIG_ZINC_ARCH_ARM64)
As above, just use CONFIG_ARM / CONFIG_ARM64 directly
> +#include "chacha20-arm-glue.h"
> #else
> void __init chacha20_fpu_init(void)
> {
> --
> 2.19.0
>
^ permalink raw reply
* Re: [PATCH 00/15] octeontx2-af: Add RVU Admin Function driver
From: Sunil Kovvuri @ 2018-09-28 9:35 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Linux Netdev List, David S. Miller, linux-soc, Sunil Goutham
In-Reply-To: <CAK8P3a2051NXGnuOyOGuXTovxQ9UfiZPrgEwJLP4PrXO-9DBPw@mail.gmail.com>
On Fri, Sep 28, 2018 at 1:57 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Fri, Sep 28, 2018 at 8:08 AM <sunil.kovvuri@gmail.com> wrote:
> >
> > From: Sunil Goutham <sgoutham@marvell.com>
> >
> > Resource virtualization unit (RVU) on Marvell's OcteonTX2 SOC maps HW
> > resources from the network, crypto and other functional blocks into
> > PCI-compatible physical and virtual functions. Each functional block
> > again has multiple local functions (LFs) for provisioning to PCI devices.
> > RVU supports multiple PCIe SRIOV physical functions (PFs) and virtual
> > functions (VFs). PF0 is called the administrative / admin function (AF)
> > and has privileges to provision RVU functional block's LFs to each of the
> > PF/VF.
> >
> > RVU managed networking functional blocks
> > - Network pool allocator (NPA)
> > - Network interface controller (NIX)
> > - Network parser CAM (NPC)
> > - Schedule/Synchronize/Order unit (SSO)
> >
> > RVU managed non-networking functional blocks
> > - Crypto accelerator (CPT)
> > - Scheduled timers unit (TIM)
> > - Schedule/Synchronize/Order unit (SSO)
> > Used for both networking and non networking usecases
> > - Compression (upcoming in future variants of the silicons)
> >
> > Resource provisioning examples
> > - A PF/VF with NIX-LF & NPA-LF resources works as a pure network device
> > - A PF/VF with CPT-LF resource works as a pure cyrpto offload device.
> >
> > This admin function driver neither receives any data nor processes it i.e
> > no I/O, a configuration only driver.
> >
> > PF/VFs communicates with AF via a shared memory region (mailbox). Upon
> > receiving requests from PF/VF, AF does resource provisioning and other
> > HW configuration. AF is always attached to host, but PF/VFs may be used
> > by host kernel itself, or attached to VMs or to userspace applications
> > like DPDK etc. So AF has to handle provisioning/configuration requests
> > sent by any device from any domain.
> >
> > This patch series adds logic for the following
> > - RVU AF driver with functional blocks provisioning support.
> > - Mailbox infrastructure for communication between AF and PFs.
> > - CGX (MAC controller) driver which communicates with firmware for
> > managing physical ethernet interfaces. AF collects info from this
> > driver and forwards the same to the PF/VFs uaing these interfaces.
> >
> > This is the first set of patches out of 80+ patches.
>
> Hi Sunil,
>
> This looks good overall, thanks for moving it to drivers/net
> as I suggested. I found two more minor remaining issues here,
> it should not be a problem to change the code accordingly.
>
> One thing I was missing here is a revision history for the patch
> series, it would have been helpful to get a list of things you have
> changed compared to the previous submission, what triggered
> those changes, and which issues (if any) are still remaining.
> Best add the full revision history in the next (hopefully final)
> round for these patches, and use 'git format-patch --reroll-count'
> to add the patchset revision number.
Not much changes done to the patches since the v3 patchset i submitted
on drivers/soc. But yes, i will add the old revision history while re-submitting
patches after addressing your suggestions.
>
> I assume that the next set of patches will be more controversial
> as you get to the point that adds the user interface. I have
> not looked at those patches yet, so it may just be fine, assuming
> all the configuration is done using devlink. Please keep me
> on Cc for the future series, even if I don't participate in those
> reviews as much.
Sure I will keep you in CC.
>
> Thanks,
>
> Arnd
^ permalink raw reply
* Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations
From: Jason A. Donenfeld @ 2018-09-28 15:57 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: LKML, Netdev, Linux Crypto Mailing List, David Miller,
Greg Kroah-Hartman, Samuel Neves, Andrew Lutomirski,
Jean-Philippe Aumasson, Andy Polyakov, Russell King - ARM Linux,
linux-arm-kernel
In-Reply-To: <CAKv+Gu8-EwxFhQSUPxjEvTA5ZPz34RieMokM6CUqwURDr74jtg@mail.gmail.com>
Hi Ard,
On Fri, Sep 28, 2018 at 5:49 PM Ard Biesheuvel
<ard.biesheuvel@linaro.org> wrote:
> As I mentioned before, I'd prefer this to be based on the original .pl
> but if I am the only one objecting to this, I guess I can live with
> it.
We're working on that, actually. It's not obvious when it'll be ready
to ship -- perhaps after the initial merge, but perhaps way sooner --
but that is something we're trying to do for arm/arm64/mips64.
Jason
^ permalink raw reply
* Re: [PATCH 13/15] octeontx2-af: Add support for CGX link management
From: Sunil Kovvuri @ 2018-09-28 9:31 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Linux Netdev List, David S. Miller, linux-soc, Linu Cherian,
Nithya Mani
In-Reply-To: <CAK8P3a36irQQXSZSACiNDi04AzA2Fb+ZUkWj10mR7Xxu+b0DbA@mail.gmail.com>
On Fri, Sep 28, 2018 at 1:49 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Fri, Sep 28, 2018 at 8:09 AM <sunil.kovvuri@gmail.com> wrote:
> >
> > +/* scratchx(0) CSR used for ATF->non-secure SW communication.
> > + * This acts as the status register
> > + * Provides details on command ack/status, link status, error details
> > + */
> > +struct cgx_evt_sts {
> > + uint64_t ack:1;
> > + uint64_t evt_type:1; /* cgx_evt_type */
> > + uint64_t stat:1; /* cgx_stat */
> > + uint64_t id:6; /* cgx_evt_id/cgx_cmd_id */
> > + uint64_t reserved:55;
> > +};
> > +
> > +/* Response to command IDs with command status as CGX_STAT_FAIL
> > + *
> > + * Not applicable for commands :
> > + * CGX_CMD_LINK_BRING_UP/DOWN/CGX_EVT_LINK_CHANGE
> > + * check struct cgx_lnk_sts comments
> > + */
> > +struct cgx_err_sts_s {
> > + uint64_t reserved1:9;
> > + uint64_t type:10; /* cgx_error_type */
> > + uint64_t reserved2:35;
> > +};
> > +
> > +/* Response to cmd ID as CGX_CMD_GET_FW_VER with cmd status as
> > + * CGX_STAT_SUCCESS
> > + */
> > +struct cgx_ver_s {
> > + uint64_t reserved1:9;
> > + uint64_t major_ver:4;
> > + uint64_t minor_ver:4;
> > + uint64_t reserved2:47;
> > +};
>
> From what I can tell, you pass these structures to the device, so they
> are a binary interface. I don't think you can rely on bitfields to work
> correctly here, they are generally not portable and I wouldn't rely
> on them doing the right thing if you build a big-endian kernel.
>
> It's better to use bitmasks on a u64 value instead to make it obviously
> portable.
>
> Arnd
This communication between firmware and kernel driver is done using couple of
scratch registers. With limited space available we had to resort to bitfields.
Your point about endianness is correct. As you might be aware that the device to
which this driver registers to, is only found on OcteonTx2 SOC which operates
in a standalone mode. As of now we are not targeting to make these drivers
work in big-endian mode.
We would prefer to make big-endian related changes later on, test them
fully and
submit patches, would this be okay ?
If not we will define big endian bit fields in all command structures
and re-submit.
Sunil.
^ permalink raw reply
* Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations
From: Ard Biesheuvel @ 2018-09-28 15:51 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: Linux Kernel Mailing List, <netdev@vger.kernel.org>,
open list:HARDWARE RANDOM NUMBER GENERATOR CORE, David S. Miller,
Greg Kroah-Hartman, Samuel Neves, Andy Lutomirski,
Jean-Philippe Aumasson, Andy Polyakov, Russell King,
linux-arm-kernel
In-Reply-To: <CAKv+Gu8-EwxFhQSUPxjEvTA5ZPz34RieMokM6CUqwURDr74jtg@mail.gmail.com>
On 28 September 2018 at 17:49, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> On 25 September 2018 at 16:56, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>> These NEON and non-NEON implementations come from Andy Polyakov's
>> implementation, and are included here in raw form without modification,
>> so that subsequent commits that fix these up for the kernel can see how
>> it has changed. This awkward commit splitting has been requested for the
>> ARM[64] implementations in particular.
>>
>> While this is CRYPTOGAMS code, the originating code for this happens to
>> be the same as OpenSSL's commit 87cc649f30aaf69b351701875b9dac07c29ce8a2
>>
>> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
>> Based-on-code-from: Andy Polyakov <appro@openssl.org>
>> Cc: Samuel Neves <sneves@dei.uc.pt>
>> Cc: Andy Lutomirski <luto@kernel.org>
>> Cc: Greg KH <gregkh@linuxfoundation.org>
>> Cc: Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
>> Cc: Andy Polyakov <appro@openssl.org>
>> Cc: Russell King <linux@armlinux.org.uk>
>> Cc: linux-arm-kernel@lists.infradead.org
>
> As I mentioned before, I'd prefer this to be based on the original .pl
> but if I am the only one objecting to this, I guess I can live with
> it.
>
Note that I am getting bounces from LAKML because the patch is too big.
^ permalink raw reply
* Re: [PATCH net-next v6 05/23] zinc: import Andy Polyakov's ChaCha20 ARM and ARM64 implementations
From: Ard Biesheuvel @ 2018-09-28 15:49 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: Linux Kernel Mailing List, <netdev@vger.kernel.org>,
open list:HARDWARE RANDOM NUMBER GENERATOR CORE, David S. Miller,
Greg Kroah-Hartman, Samuel Neves, Andy Lutomirski,
Jean-Philippe Aumasson, Andy Polyakov, Russell King,
linux-arm-kernel
In-Reply-To: <20180925145622.29959-6-Jason@zx2c4.com>
On 25 September 2018 at 16:56, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> These NEON and non-NEON implementations come from Andy Polyakov's
> implementation, and are included here in raw form without modification,
> so that subsequent commits that fix these up for the kernel can see how
> it has changed. This awkward commit splitting has been requested for the
> ARM[64] implementations in particular.
>
> While this is CRYPTOGAMS code, the originating code for this happens to
> be the same as OpenSSL's commit 87cc649f30aaf69b351701875b9dac07c29ce8a2
>
> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
> Based-on-code-from: Andy Polyakov <appro@openssl.org>
> Cc: Samuel Neves <sneves@dei.uc.pt>
> Cc: Andy Lutomirski <luto@kernel.org>
> Cc: Greg KH <gregkh@linuxfoundation.org>
> Cc: Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
> Cc: Andy Polyakov <appro@openssl.org>
> Cc: Russell King <linux@armlinux.org.uk>
> Cc: linux-arm-kernel@lists.infradead.org
As I mentioned before, I'd prefer this to be based on the original .pl
but if I am the only one objecting to this, I guess I can live with
it.
> ---
> lib/zinc/chacha20/chacha20-arm-cryptogams.S | 1440 ++++++++++++
> lib/zinc/chacha20/chacha20-arm64-cryptogams.S | 1973 +++++++++++++++++
> 2 files changed, 3413 insertions(+)
> create mode 100644 lib/zinc/chacha20/chacha20-arm-cryptogams.S
> create mode 100644 lib/zinc/chacha20/chacha20-arm64-cryptogams.S
>
> diff --git a/lib/zinc/chacha20/chacha20-arm-cryptogams.S b/lib/zinc/chacha20/chacha20-arm-cryptogams.S
> new file mode 100644
> index 000000000000..05a3a9e6e93f
> --- /dev/null
> +++ b/lib/zinc/chacha20/chacha20-arm-cryptogams.S
> @@ -0,0 +1,1440 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
> +/*
> + * Copyright (C) 2006-2017 CRYPTOGAMS by <appro@openssl.org>. All Rights Reserved.
> + */
> +
> +#include "arm_arch.h"
> +
> +.text
> +#if defined(__thumb2__) || defined(__clang__)
> +.syntax unified
> +#endif
> +#if defined(__thumb2__)
> +.thumb
> +#else
> +.code 32
> +#endif
> +
> +#if defined(__thumb2__) || defined(__clang__)
> +#define ldrhsb ldrbhs
> +#endif
> +
> +.align 5
> +.Lsigma:
> +.long 0x61707865,0x3320646e,0x79622d32,0x6b206574 @ endian-neutral
> +.Lone:
> +.long 1,0,0,0
> +.Lrot8:
> +.long 0x02010003,0x06050407
> +#if __ARM_MAX_ARCH__>=7
> +.LOPENSSL_armcap:
> +.word OPENSSL_armcap_P-.LChaCha20_ctr32
> +#else
> +.word -1
> +#endif
> +
> +.globl ChaCha20_ctr32
> +.type ChaCha20_ctr32,%function
> +.align 5
> +ChaCha20_ctr32:
> +.LChaCha20_ctr32:
> + ldr r12,[sp,#0] @ pull pointer to counter and nonce
> + stmdb sp!,{r0-r2,r4-r11,lr}
> +#if __ARM_ARCH__<7 && !defined(__thumb2__)
> + sub r14,pc,#16 @ ChaCha20_ctr32
> +#else
> + adr r14,.LChaCha20_ctr32
> +#endif
> + cmp r2,#0 @ len==0?
> +#ifdef __thumb2__
> + itt eq
> +#endif
> + addeq sp,sp,#4*3
> + beq .Lno_data
> +#if __ARM_MAX_ARCH__>=7
> + cmp r2,#192 @ test len
> + bls .Lshort
> + ldr r4,[r14,#-24]
> + ldr r4,[r14,r4]
> +# ifdef __APPLE__
> + ldr r4,[r4]
> +# endif
> + tst r4,#ARMV7_NEON
> + bne .LChaCha20_neon
> +.Lshort:
> +#endif
> + ldmia r12,{r4-r7} @ load counter and nonce
> + sub sp,sp,#4*(16) @ off-load area
> + sub r14,r14,#64 @ .Lsigma
> + stmdb sp!,{r4-r7} @ copy counter and nonce
> + ldmia r3,{r4-r11} @ load key
> + ldmia r14,{r0-r3} @ load sigma
> + stmdb sp!,{r4-r11} @ copy key
> + stmdb sp!,{r0-r3} @ copy sigma
> + str r10,[sp,#4*(16+10)] @ off-load "rx"
> + str r11,[sp,#4*(16+11)] @ off-load "rx"
> + b .Loop_outer_enter
> +
> +.align 4
> +.Loop_outer:
> + ldmia sp,{r0-r9} @ load key material
> + str r11,[sp,#4*(32+2)] @ save len
> + str r12, [sp,#4*(32+1)] @ save inp
> + str r14, [sp,#4*(32+0)] @ save out
> +.Loop_outer_enter:
> + ldr r11, [sp,#4*(15)]
> + mov r4,r4,ror#19 @ twist b[0..3]
> + ldr r12,[sp,#4*(12)] @ modulo-scheduled load
> + mov r5,r5,ror#19
> + ldr r10, [sp,#4*(13)]
> + mov r6,r6,ror#19
> + ldr r14,[sp,#4*(14)]
> + mov r7,r7,ror#19
> + mov r11,r11,ror#8 @ twist d[0..3]
> + mov r12,r12,ror#8
> + mov r10,r10,ror#8
> + mov r14,r14,ror#8
> + str r11, [sp,#4*(16+15)]
> + mov r11,#10
> + b .Loop
> +
> +.align 4
> +.Loop:
> + subs r11,r11,#1
> + add r0,r0,r4,ror#13
> + add r1,r1,r5,ror#13
> + eor r12,r0,r12,ror#24
> + eor r10,r1,r10,ror#24
> + add r8,r8,r12,ror#16
> + add r9,r9,r10,ror#16
> + eor r4,r8,r4,ror#13
> + eor r5,r9,r5,ror#13
> + add r0,r0,r4,ror#20
> + add r1,r1,r5,ror#20
> + eor r12,r0,r12,ror#16
> + eor r10,r1,r10,ror#16
> + add r8,r8,r12,ror#24
> + str r10,[sp,#4*(16+13)]
> + add r9,r9,r10,ror#24
> + ldr r10,[sp,#4*(16+15)]
> + str r8,[sp,#4*(16+8)]
> + eor r4,r4,r8,ror#12
> + str r9,[sp,#4*(16+9)]
> + eor r5,r5,r9,ror#12
> + ldr r8,[sp,#4*(16+10)]
> + add r2,r2,r6,ror#13
> + ldr r9,[sp,#4*(16+11)]
> + add r3,r3,r7,ror#13
> + eor r14,r2,r14,ror#24
> + eor r10,r3,r10,ror#24
> + add r8,r8,r14,ror#16
> + add r9,r9,r10,ror#16
> + eor r6,r8,r6,ror#13
> + eor r7,r9,r7,ror#13
> + add r2,r2,r6,ror#20
> + add r3,r3,r7,ror#20
> + eor r14,r2,r14,ror#16
> + eor r10,r3,r10,ror#16
> + add r8,r8,r14,ror#24
> + add r9,r9,r10,ror#24
> + eor r6,r6,r8,ror#12
> + eor r7,r7,r9,ror#12
> + add r0,r0,r5,ror#13
> + add r1,r1,r6,ror#13
> + eor r10,r0,r10,ror#24
> + eor r12,r1,r12,ror#24
> + add r8,r8,r10,ror#16
> + add r9,r9,r12,ror#16
> + eor r5,r8,r5,ror#13
> + eor r6,r9,r6,ror#13
> + add r0,r0,r5,ror#20
> + add r1,r1,r6,ror#20
> + eor r10,r0,r10,ror#16
> + eor r12,r1,r12,ror#16
> + str r10,[sp,#4*(16+15)]
> + add r8,r8,r10,ror#24
> + ldr r10,[sp,#4*(16+13)]
> + add r9,r9,r12,ror#24
> + str r8,[sp,#4*(16+10)]
> + eor r5,r5,r8,ror#12
> + str r9,[sp,#4*(16+11)]
> + eor r6,r6,r9,ror#12
> + ldr r8,[sp,#4*(16+8)]
> + add r2,r2,r7,ror#13
> + ldr r9,[sp,#4*(16+9)]
> + add r3,r3,r4,ror#13
> + eor r10,r2,r10,ror#24
> + eor r14,r3,r14,ror#24
> + add r8,r8,r10,ror#16
> + add r9,r9,r14,ror#16
> + eor r7,r8,r7,ror#13
> + eor r4,r9,r4,ror#13
> + add r2,r2,r7,ror#20
> + add r3,r3,r4,ror#20
> + eor r10,r2,r10,ror#16
> + eor r14,r3,r14,ror#16
> + add r8,r8,r10,ror#24
> + add r9,r9,r14,ror#24
> + eor r7,r7,r8,ror#12
> + eor r4,r4,r9,ror#12
> + bne .Loop
> +
> + ldr r11,[sp,#4*(32+2)] @ load len
> +
> + str r8, [sp,#4*(16+8)] @ modulo-scheduled store
> + str r9, [sp,#4*(16+9)]
> + str r12,[sp,#4*(16+12)]
> + str r10, [sp,#4*(16+13)]
> + str r14,[sp,#4*(16+14)]
> +
> + @ at this point we have first half of 512-bit result in
> + @ rx and second half at sp+4*(16+8)
> +
> + cmp r11,#64 @ done yet?
> +#ifdef __thumb2__
> + itete lo
> +#endif
> + addlo r12,sp,#4*(0) @ shortcut or ...
> + ldrhs r12,[sp,#4*(32+1)] @ ... load inp
> + addlo r14,sp,#4*(0) @ shortcut or ...
> + ldrhs r14,[sp,#4*(32+0)] @ ... load out
> +
> + ldr r8,[sp,#4*(0)] @ load key material
> + ldr r9,[sp,#4*(1)]
> +
> +#if __ARM_ARCH__>=6 || !defined(__ARMEB__)
> +# if __ARM_ARCH__<7
> + orr r10,r12,r14
> + tst r10,#3 @ are input and output aligned?
> + ldr r10,[sp,#4*(2)]
> + bne .Lunaligned
> + cmp r11,#64 @ restore flags
> +# else
> + ldr r10,[sp,#4*(2)]
> +# endif
> + ldr r11,[sp,#4*(3)]
> +
> + add r0,r0,r8 @ accumulate key material
> + add r1,r1,r9
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r8,[r12],#16 @ load input
> + ldrhs r9,[r12,#-12]
> +
> + add r2,r2,r10
> + add r3,r3,r11
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r10,[r12,#-8]
> + ldrhs r11,[r12,#-4]
> +# if __ARM_ARCH__>=6 && defined(__ARMEB__)
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> +# endif
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r0,r0,r8 @ xor with input
> + eorhs r1,r1,r9
> + add r8,sp,#4*(4)
> + str r0,[r14],#16 @ store output
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r2,r2,r10
> + eorhs r3,r3,r11
> + ldmia r8,{r8-r11} @ load key material
> + str r1,[r14,#-12]
> + str r2,[r14,#-8]
> + str r3,[r14,#-4]
> +
> + add r4,r8,r4,ror#13 @ accumulate key material
> + add r5,r9,r5,ror#13
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r8,[r12],#16 @ load input
> + ldrhs r9,[r12,#-12]
> + add r6,r10,r6,ror#13
> + add r7,r11,r7,ror#13
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r10,[r12,#-8]
> + ldrhs r11,[r12,#-4]
> +# if __ARM_ARCH__>=6 && defined(__ARMEB__)
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r4,r4,r8
> + eorhs r5,r5,r9
> + add r8,sp,#4*(8)
> + str r4,[r14],#16 @ store output
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r6,r6,r10
> + eorhs r7,r7,r11
> + str r5,[r14,#-12]
> + ldmia r8,{r8-r11} @ load key material
> + str r6,[r14,#-8]
> + add r0,sp,#4*(16+8)
> + str r7,[r14,#-4]
> +
> + ldmia r0,{r0-r7} @ load second half
> +
> + add r0,r0,r8 @ accumulate key material
> + add r1,r1,r9
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r8,[r12],#16 @ load input
> + ldrhs r9,[r12,#-12]
> +# ifdef __thumb2__
> + itt hi
> +# endif
> + strhi r10,[sp,#4*(16+10)] @ copy "rx" while at it
> + strhi r11,[sp,#4*(16+11)] @ copy "rx" while at it
> + add r2,r2,r10
> + add r3,r3,r11
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r10,[r12,#-8]
> + ldrhs r11,[r12,#-4]
> +# if __ARM_ARCH__>=6 && defined(__ARMEB__)
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> +# endif
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r0,r0,r8
> + eorhs r1,r1,r9
> + add r8,sp,#4*(12)
> + str r0,[r14],#16 @ store output
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r2,r2,r10
> + eorhs r3,r3,r11
> + str r1,[r14,#-12]
> + ldmia r8,{r8-r11} @ load key material
> + str r2,[r14,#-8]
> + str r3,[r14,#-4]
> +
> + add r4,r8,r4,ror#24 @ accumulate key material
> + add r5,r9,r5,ror#24
> +# ifdef __thumb2__
> + itt hi
> +# endif
> + addhi r8,r8,#1 @ next counter value
> + strhi r8,[sp,#4*(12)] @ save next counter value
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r8,[r12],#16 @ load input
> + ldrhs r9,[r12,#-12]
> + add r6,r10,r6,ror#24
> + add r7,r11,r7,ror#24
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhs r10,[r12,#-8]
> + ldrhs r11,[r12,#-4]
> +# if __ARM_ARCH__>=6 && defined(__ARMEB__)
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r4,r4,r8
> + eorhs r5,r5,r9
> +# ifdef __thumb2__
> + it ne
> +# endif
> + ldrne r8,[sp,#4*(32+2)] @ re-load len
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + eorhs r6,r6,r10
> + eorhs r7,r7,r11
> + str r4,[r14],#16 @ store output
> + str r5,[r14,#-12]
> +# ifdef __thumb2__
> + it hs
> +# endif
> + subhs r11,r8,#64 @ len-=64
> + str r6,[r14,#-8]
> + str r7,[r14,#-4]
> + bhi .Loop_outer
> +
> + beq .Ldone
> +# if __ARM_ARCH__<7
> + b .Ltail
> +
> +.align 4
> +.Lunaligned: @ unaligned endian-neutral path
> + cmp r11,#64 @ restore flags
> +# endif
> +#endif
> +#if __ARM_ARCH__<7
> + ldr r11,[sp,#4*(3)]
> + add r0,r8,r0 @ accumulate key material
> + add r1,r9,r1
> + add r2,r10,r2
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r8,r8,r8 @ zero or ...
> + ldrhsb r8,[r12],#16 @ ... load input
> + eorlo r9,r9,r9
> + ldrhsb r9,[r12,#-12]
> +
> + add r3,r11,r3
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r10,r10,r10
> + ldrhsb r10,[r12,#-8]
> + eorlo r11,r11,r11
> + ldrhsb r11,[r12,#-4]
> +
> + eor r0,r8,r0 @ xor with input (or zero)
> + eor r1,r9,r1
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-15] @ load more input
> + ldrhsb r9,[r12,#-11]
> + eor r2,r10,r2
> + strb r0,[r14],#16 @ store output
> + eor r3,r11,r3
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-7]
> + ldrhsb r11,[r12,#-3]
> + strb r1,[r14,#-12]
> + eor r0,r8,r0,lsr#8
> + strb r2,[r14,#-8]
> + eor r1,r9,r1,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-14] @ load more input
> + ldrhsb r9,[r12,#-10]
> + strb r3,[r14,#-4]
> + eor r2,r10,r2,lsr#8
> + strb r0,[r14,#-15]
> + eor r3,r11,r3,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-6]
> + ldrhsb r11,[r12,#-2]
> + strb r1,[r14,#-11]
> + eor r0,r8,r0,lsr#8
> + strb r2,[r14,#-7]
> + eor r1,r9,r1,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-13] @ load more input
> + ldrhsb r9,[r12,#-9]
> + strb r3,[r14,#-3]
> + eor r2,r10,r2,lsr#8
> + strb r0,[r14,#-14]
> + eor r3,r11,r3,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-5]
> + ldrhsb r11,[r12,#-1]
> + strb r1,[r14,#-10]
> + strb r2,[r14,#-6]
> + eor r0,r8,r0,lsr#8
> + strb r3,[r14,#-2]
> + eor r1,r9,r1,lsr#8
> + strb r0,[r14,#-13]
> + eor r2,r10,r2,lsr#8
> + strb r1,[r14,#-9]
> + eor r3,r11,r3,lsr#8
> + strb r2,[r14,#-5]
> + strb r3,[r14,#-1]
> + add r8,sp,#4*(4+0)
> + ldmia r8,{r8-r11} @ load key material
> + add r0,sp,#4*(16+8)
> + add r4,r8,r4,ror#13 @ accumulate key material
> + add r5,r9,r5,ror#13
> + add r6,r10,r6,ror#13
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r8,r8,r8 @ zero or ...
> + ldrhsb r8,[r12],#16 @ ... load input
> + eorlo r9,r9,r9
> + ldrhsb r9,[r12,#-12]
> +
> + add r7,r11,r7,ror#13
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r10,r10,r10
> + ldrhsb r10,[r12,#-8]
> + eorlo r11,r11,r11
> + ldrhsb r11,[r12,#-4]
> +
> + eor r4,r8,r4 @ xor with input (or zero)
> + eor r5,r9,r5
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-15] @ load more input
> + ldrhsb r9,[r12,#-11]
> + eor r6,r10,r6
> + strb r4,[r14],#16 @ store output
> + eor r7,r11,r7
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-7]
> + ldrhsb r11,[r12,#-3]
> + strb r5,[r14,#-12]
> + eor r4,r8,r4,lsr#8
> + strb r6,[r14,#-8]
> + eor r5,r9,r5,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-14] @ load more input
> + ldrhsb r9,[r12,#-10]
> + strb r7,[r14,#-4]
> + eor r6,r10,r6,lsr#8
> + strb r4,[r14,#-15]
> + eor r7,r11,r7,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-6]
> + ldrhsb r11,[r12,#-2]
> + strb r5,[r14,#-11]
> + eor r4,r8,r4,lsr#8
> + strb r6,[r14,#-7]
> + eor r5,r9,r5,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-13] @ load more input
> + ldrhsb r9,[r12,#-9]
> + strb r7,[r14,#-3]
> + eor r6,r10,r6,lsr#8
> + strb r4,[r14,#-14]
> + eor r7,r11,r7,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-5]
> + ldrhsb r11,[r12,#-1]
> + strb r5,[r14,#-10]
> + strb r6,[r14,#-6]
> + eor r4,r8,r4,lsr#8
> + strb r7,[r14,#-2]
> + eor r5,r9,r5,lsr#8
> + strb r4,[r14,#-13]
> + eor r6,r10,r6,lsr#8
> + strb r5,[r14,#-9]
> + eor r7,r11,r7,lsr#8
> + strb r6,[r14,#-5]
> + strb r7,[r14,#-1]
> + add r8,sp,#4*(4+4)
> + ldmia r8,{r8-r11} @ load key material
> + ldmia r0,{r0-r7} @ load second half
> +# ifdef __thumb2__
> + itt hi
> +# endif
> + strhi r10,[sp,#4*(16+10)] @ copy "rx"
> + strhi r11,[sp,#4*(16+11)] @ copy "rx"
> + add r0,r8,r0 @ accumulate key material
> + add r1,r9,r1
> + add r2,r10,r2
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r8,r8,r8 @ zero or ...
> + ldrhsb r8,[r12],#16 @ ... load input
> + eorlo r9,r9,r9
> + ldrhsb r9,[r12,#-12]
> +
> + add r3,r11,r3
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r10,r10,r10
> + ldrhsb r10,[r12,#-8]
> + eorlo r11,r11,r11
> + ldrhsb r11,[r12,#-4]
> +
> + eor r0,r8,r0 @ xor with input (or zero)
> + eor r1,r9,r1
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-15] @ load more input
> + ldrhsb r9,[r12,#-11]
> + eor r2,r10,r2
> + strb r0,[r14],#16 @ store output
> + eor r3,r11,r3
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-7]
> + ldrhsb r11,[r12,#-3]
> + strb r1,[r14,#-12]
> + eor r0,r8,r0,lsr#8
> + strb r2,[r14,#-8]
> + eor r1,r9,r1,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-14] @ load more input
> + ldrhsb r9,[r12,#-10]
> + strb r3,[r14,#-4]
> + eor r2,r10,r2,lsr#8
> + strb r0,[r14,#-15]
> + eor r3,r11,r3,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-6]
> + ldrhsb r11,[r12,#-2]
> + strb r1,[r14,#-11]
> + eor r0,r8,r0,lsr#8
> + strb r2,[r14,#-7]
> + eor r1,r9,r1,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-13] @ load more input
> + ldrhsb r9,[r12,#-9]
> + strb r3,[r14,#-3]
> + eor r2,r10,r2,lsr#8
> + strb r0,[r14,#-14]
> + eor r3,r11,r3,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-5]
> + ldrhsb r11,[r12,#-1]
> + strb r1,[r14,#-10]
> + strb r2,[r14,#-6]
> + eor r0,r8,r0,lsr#8
> + strb r3,[r14,#-2]
> + eor r1,r9,r1,lsr#8
> + strb r0,[r14,#-13]
> + eor r2,r10,r2,lsr#8
> + strb r1,[r14,#-9]
> + eor r3,r11,r3,lsr#8
> + strb r2,[r14,#-5]
> + strb r3,[r14,#-1]
> + add r8,sp,#4*(4+8)
> + ldmia r8,{r8-r11} @ load key material
> + add r4,r8,r4,ror#24 @ accumulate key material
> +# ifdef __thumb2__
> + itt hi
> +# endif
> + addhi r8,r8,#1 @ next counter value
> + strhi r8,[sp,#4*(12)] @ save next counter value
> + add r5,r9,r5,ror#24
> + add r6,r10,r6,ror#24
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r8,r8,r8 @ zero or ...
> + ldrhsb r8,[r12],#16 @ ... load input
> + eorlo r9,r9,r9
> + ldrhsb r9,[r12,#-12]
> +
> + add r7,r11,r7,ror#24
> +# ifdef __thumb2__
> + itete lo
> +# endif
> + eorlo r10,r10,r10
> + ldrhsb r10,[r12,#-8]
> + eorlo r11,r11,r11
> + ldrhsb r11,[r12,#-4]
> +
> + eor r4,r8,r4 @ xor with input (or zero)
> + eor r5,r9,r5
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-15] @ load more input
> + ldrhsb r9,[r12,#-11]
> + eor r6,r10,r6
> + strb r4,[r14],#16 @ store output
> + eor r7,r11,r7
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-7]
> + ldrhsb r11,[r12,#-3]
> + strb r5,[r14,#-12]
> + eor r4,r8,r4,lsr#8
> + strb r6,[r14,#-8]
> + eor r5,r9,r5,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-14] @ load more input
> + ldrhsb r9,[r12,#-10]
> + strb r7,[r14,#-4]
> + eor r6,r10,r6,lsr#8
> + strb r4,[r14,#-15]
> + eor r7,r11,r7,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-6]
> + ldrhsb r11,[r12,#-2]
> + strb r5,[r14,#-11]
> + eor r4,r8,r4,lsr#8
> + strb r6,[r14,#-7]
> + eor r5,r9,r5,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r8,[r12,#-13] @ load more input
> + ldrhsb r9,[r12,#-9]
> + strb r7,[r14,#-3]
> + eor r6,r10,r6,lsr#8
> + strb r4,[r14,#-14]
> + eor r7,r11,r7,lsr#8
> +# ifdef __thumb2__
> + itt hs
> +# endif
> + ldrhsb r10,[r12,#-5]
> + ldrhsb r11,[r12,#-1]
> + strb r5,[r14,#-10]
> + strb r6,[r14,#-6]
> + eor r4,r8,r4,lsr#8
> + strb r7,[r14,#-2]
> + eor r5,r9,r5,lsr#8
> + strb r4,[r14,#-13]
> + eor r6,r10,r6,lsr#8
> + strb r5,[r14,#-9]
> + eor r7,r11,r7,lsr#8
> + strb r6,[r14,#-5]
> + strb r7,[r14,#-1]
> +# ifdef __thumb2__
> + it ne
> +# endif
> + ldrne r8,[sp,#4*(32+2)] @ re-load len
> +# ifdef __thumb2__
> + it hs
> +# endif
> + subhs r11,r8,#64 @ len-=64
> + bhi .Loop_outer
> +
> + beq .Ldone
> +#endif
> +
> +.Ltail:
> + ldr r12,[sp,#4*(32+1)] @ load inp
> + add r9,sp,#4*(0)
> + ldr r14,[sp,#4*(32+0)] @ load out
> +
> +.Loop_tail:
> + ldrb r10,[r9],#1 @ read buffer on stack
> + ldrb r11,[r12],#1 @ read input
> + subs r8,r8,#1
> + eor r11,r11,r10
> + strb r11,[r14],#1 @ store output
> + bne .Loop_tail
> +
> +.Ldone:
> + add sp,sp,#4*(32+3)
> +.Lno_data:
> + ldmia sp!,{r4-r11,pc}
> +.size ChaCha20_ctr32,.-ChaCha20_ctr32
> +#if __ARM_MAX_ARCH__>=7
> +.arch armv7-a
> +.fpu neon
> +
> +.type ChaCha20_neon,%function
> +.align 5
> +ChaCha20_neon:
> + ldr r12,[sp,#0] @ pull pointer to counter and nonce
> + stmdb sp!,{r0-r2,r4-r11,lr}
> +.LChaCha20_neon:
> + adr r14,.Lsigma
> + vstmdb sp!,{d8-d15} @ ABI spec says so
> + stmdb sp!,{r0-r3}
> +
> + vld1.32 {q1-q2},[r3] @ load key
> + ldmia r3,{r4-r11} @ load key
> +
> + sub sp,sp,#4*(16+16)
> + vld1.32 {q3},[r12] @ load counter and nonce
> + add r12,sp,#4*8
> + ldmia r14,{r0-r3} @ load sigma
> + vld1.32 {q0},[r14]! @ load sigma
> + vld1.32 {q12},[r14]! @ one
> + @ vld1.32 {d30},[r14] @ rot8
> + vst1.32 {q2-q3},[r12] @ copy 1/2key|counter|nonce
> + vst1.32 {q0-q1},[sp] @ copy sigma|1/2key
> +
> + str r10,[sp,#4*(16+10)] @ off-load "rx"
> + str r11,[sp,#4*(16+11)] @ off-load "rx"
> + vshl.i32 d26,d24,#1 @ two
> + vstr d24,[sp,#4*(16+0)]
> + vshl.i32 d28,d24,#2 @ four
> + vstr d26,[sp,#4*(16+2)]
> + vmov q4,q0
> + vstr d28,[sp,#4*(16+4)]
> + vmov q8,q0
> + @ vstr d30,[sp,#4*(16+6)]
> + vmov q5,q1
> + vmov q9,q1
> + b .Loop_neon_enter
> +
> +.align 4
> +.Loop_neon_outer:
> + ldmia sp,{r0-r9} @ load key material
> + cmp r11,#64*2 @ if len<=64*2
> + bls .Lbreak_neon @ switch to integer-only
> + @ vldr d30,[sp,#4*(16+6)] @ rot8
> + vmov q4,q0
> + str r11,[sp,#4*(32+2)] @ save len
> + vmov q8,q0
> + str r12, [sp,#4*(32+1)] @ save inp
> + vmov q5,q1
> + str r14, [sp,#4*(32+0)] @ save out
> + vmov q9,q1
> +.Loop_neon_enter:
> + ldr r11, [sp,#4*(15)]
> + mov r4,r4,ror#19 @ twist b[0..3]
> + vadd.i32 q7,q3,q12 @ counter+1
> + ldr r12,[sp,#4*(12)] @ modulo-scheduled load
> + mov r5,r5,ror#19
> + vmov q6,q2
> + ldr r10, [sp,#4*(13)]
> + mov r6,r6,ror#19
> + vmov q10,q2
> + ldr r14,[sp,#4*(14)]
> + mov r7,r7,ror#19
> + vadd.i32 q11,q7,q12 @ counter+2
> + add r12,r12,#3 @ counter+3
> + mov r11,r11,ror#8 @ twist d[0..3]
> + mov r12,r12,ror#8
> + mov r10,r10,ror#8
> + mov r14,r14,ror#8
> + str r11, [sp,#4*(16+15)]
> + mov r11,#10
> + b .Loop_neon
> +
> +.align 4
> +.Loop_neon:
> + subs r11,r11,#1
> + vadd.i32 q0,q0,q1
> + add r0,r0,r4,ror#13
> + vadd.i32 q4,q4,q5
> + add r1,r1,r5,ror#13
> + vadd.i32 q8,q8,q9
> + eor r12,r0,r12,ror#24
> + veor q3,q3,q0
> + eor r10,r1,r10,ror#24
> + veor q7,q7,q4
> + add r8,r8,r12,ror#16
> + veor q11,q11,q8
> + add r9,r9,r10,ror#16
> + vrev32.16 q3,q3
> + eor r4,r8,r4,ror#13
> + vrev32.16 q7,q7
> + eor r5,r9,r5,ror#13
> + vrev32.16 q11,q11
> + add r0,r0,r4,ror#20
> + vadd.i32 q2,q2,q3
> + add r1,r1,r5,ror#20
> + vadd.i32 q6,q6,q7
> + eor r12,r0,r12,ror#16
> + vadd.i32 q10,q10,q11
> + eor r10,r1,r10,ror#16
> + veor q12,q1,q2
> + add r8,r8,r12,ror#24
> + veor q13,q5,q6
> + str r10,[sp,#4*(16+13)]
> + veor q14,q9,q10
> + add r9,r9,r10,ror#24
> + vshr.u32 q1,q12,#20
> + ldr r10,[sp,#4*(16+15)]
> + vshr.u32 q5,q13,#20
> + str r8,[sp,#4*(16+8)]
> + vshr.u32 q9,q14,#20
> + eor r4,r4,r8,ror#12
> + vsli.32 q1,q12,#12
> + str r9,[sp,#4*(16+9)]
> + vsli.32 q5,q13,#12
> + eor r5,r5,r9,ror#12
> + vsli.32 q9,q14,#12
> + ldr r8,[sp,#4*(16+10)]
> + vadd.i32 q0,q0,q1
> + add r2,r2,r6,ror#13
> + vadd.i32 q4,q4,q5
> + ldr r9,[sp,#4*(16+11)]
> + vadd.i32 q8,q8,q9
> + add r3,r3,r7,ror#13
> + veor q12,q3,q0
> + eor r14,r2,r14,ror#24
> + veor q13,q7,q4
> + eor r10,r3,r10,ror#24
> + veor q14,q11,q8
> + add r8,r8,r14,ror#16
> + vshr.u32 q3,q12,#24
> + add r9,r9,r10,ror#16
> + vshr.u32 q7,q13,#24
> + eor r6,r8,r6,ror#13
> + vshr.u32 q11,q14,#24
> + eor r7,r9,r7,ror#13
> + vsli.32 q3,q12,#8
> + add r2,r2,r6,ror#20
> + vsli.32 q7,q13,#8
> + add r3,r3,r7,ror#20
> + vsli.32 q11,q14,#8
> + eor r14,r2,r14,ror#16
> + vadd.i32 q2,q2,q3
> + eor r10,r3,r10,ror#16
> + vadd.i32 q6,q6,q7
> + add r8,r8,r14,ror#24
> + vadd.i32 q10,q10,q11
> + add r9,r9,r10,ror#24
> + veor q12,q1,q2
> + eor r6,r6,r8,ror#12
> + veor q13,q5,q6
> + eor r7,r7,r9,ror#12
> + veor q14,q9,q10
> + vshr.u32 q1,q12,#25
> + vshr.u32 q5,q13,#25
> + vshr.u32 q9,q14,#25
> + vsli.32 q1,q12,#7
> + vsli.32 q5,q13,#7
> + vsli.32 q9,q14,#7
> + vext.8 q2,q2,q2,#8
> + vext.8 q6,q6,q6,#8
> + vext.8 q10,q10,q10,#8
> + vext.8 q1,q1,q1,#4
> + vext.8 q5,q5,q5,#4
> + vext.8 q9,q9,q9,#4
> + vext.8 q3,q3,q3,#12
> + vext.8 q7,q7,q7,#12
> + vext.8 q11,q11,q11,#12
> + vadd.i32 q0,q0,q1
> + add r0,r0,r5,ror#13
> + vadd.i32 q4,q4,q5
> + add r1,r1,r6,ror#13
> + vadd.i32 q8,q8,q9
> + eor r10,r0,r10,ror#24
> + veor q3,q3,q0
> + eor r12,r1,r12,ror#24
> + veor q7,q7,q4
> + add r8,r8,r10,ror#16
> + veor q11,q11,q8
> + add r9,r9,r12,ror#16
> + vrev32.16 q3,q3
> + eor r5,r8,r5,ror#13
> + vrev32.16 q7,q7
> + eor r6,r9,r6,ror#13
> + vrev32.16 q11,q11
> + add r0,r0,r5,ror#20
> + vadd.i32 q2,q2,q3
> + add r1,r1,r6,ror#20
> + vadd.i32 q6,q6,q7
> + eor r10,r0,r10,ror#16
> + vadd.i32 q10,q10,q11
> + eor r12,r1,r12,ror#16
> + veor q12,q1,q2
> + str r10,[sp,#4*(16+15)]
> + veor q13,q5,q6
> + add r8,r8,r10,ror#24
> + veor q14,q9,q10
> + ldr r10,[sp,#4*(16+13)]
> + vshr.u32 q1,q12,#20
> + add r9,r9,r12,ror#24
> + vshr.u32 q5,q13,#20
> + str r8,[sp,#4*(16+10)]
> + vshr.u32 q9,q14,#20
> + eor r5,r5,r8,ror#12
> + vsli.32 q1,q12,#12
> + str r9,[sp,#4*(16+11)]
> + vsli.32 q5,q13,#12
> + eor r6,r6,r9,ror#12
> + vsli.32 q9,q14,#12
> + ldr r8,[sp,#4*(16+8)]
> + vadd.i32 q0,q0,q1
> + add r2,r2,r7,ror#13
> + vadd.i32 q4,q4,q5
> + ldr r9,[sp,#4*(16+9)]
> + vadd.i32 q8,q8,q9
> + add r3,r3,r4,ror#13
> + veor q12,q3,q0
> + eor r10,r2,r10,ror#24
> + veor q13,q7,q4
> + eor r14,r3,r14,ror#24
> + veor q14,q11,q8
> + add r8,r8,r10,ror#16
> + vshr.u32 q3,q12,#24
> + add r9,r9,r14,ror#16
> + vshr.u32 q7,q13,#24
> + eor r7,r8,r7,ror#13
> + vshr.u32 q11,q14,#24
> + eor r4,r9,r4,ror#13
> + vsli.32 q3,q12,#8
> + add r2,r2,r7,ror#20
> + vsli.32 q7,q13,#8
> + add r3,r3,r4,ror#20
> + vsli.32 q11,q14,#8
> + eor r10,r2,r10,ror#16
> + vadd.i32 q2,q2,q3
> + eor r14,r3,r14,ror#16
> + vadd.i32 q6,q6,q7
> + add r8,r8,r10,ror#24
> + vadd.i32 q10,q10,q11
> + add r9,r9,r14,ror#24
> + veor q12,q1,q2
> + eor r7,r7,r8,ror#12
> + veor q13,q5,q6
> + eor r4,r4,r9,ror#12
> + veor q14,q9,q10
> + vshr.u32 q1,q12,#25
> + vshr.u32 q5,q13,#25
> + vshr.u32 q9,q14,#25
> + vsli.32 q1,q12,#7
> + vsli.32 q5,q13,#7
> + vsli.32 q9,q14,#7
> + vext.8 q2,q2,q2,#8
> + vext.8 q6,q6,q6,#8
> + vext.8 q10,q10,q10,#8
> + vext.8 q1,q1,q1,#12
> + vext.8 q5,q5,q5,#12
> + vext.8 q9,q9,q9,#12
> + vext.8 q3,q3,q3,#4
> + vext.8 q7,q7,q7,#4
> + vext.8 q11,q11,q11,#4
> + bne .Loop_neon
> +
> + add r11,sp,#32
> + vld1.32 {q12-q13},[sp] @ load key material
> + vld1.32 {q14-q15},[r11]
> +
> + ldr r11,[sp,#4*(32+2)] @ load len
> +
> + str r8, [sp,#4*(16+8)] @ modulo-scheduled store
> + str r9, [sp,#4*(16+9)]
> + str r12,[sp,#4*(16+12)]
> + str r10, [sp,#4*(16+13)]
> + str r14,[sp,#4*(16+14)]
> +
> + @ at this point we have first half of 512-bit result in
> + @ rx and second half at sp+4*(16+8)
> +
> + ldr r12,[sp,#4*(32+1)] @ load inp
> + ldr r14,[sp,#4*(32+0)] @ load out
> +
> + vadd.i32 q0,q0,q12 @ accumulate key material
> + vadd.i32 q4,q4,q12
> + vadd.i32 q8,q8,q12
> + vldr d24,[sp,#4*(16+0)] @ one
> +
> + vadd.i32 q1,q1,q13
> + vadd.i32 q5,q5,q13
> + vadd.i32 q9,q9,q13
> + vldr d26,[sp,#4*(16+2)] @ two
> +
> + vadd.i32 q2,q2,q14
> + vadd.i32 q6,q6,q14
> + vadd.i32 q10,q10,q14
> + vadd.i32 d14,d14,d24 @ counter+1
> + vadd.i32 d22,d22,d26 @ counter+2
> +
> + vadd.i32 q3,q3,q15
> + vadd.i32 q7,q7,q15
> + vadd.i32 q11,q11,q15
> +
> + cmp r11,#64*4
> + blo .Ltail_neon
> +
> + vld1.8 {q12-q13},[r12]! @ load input
> + mov r11,sp
> + vld1.8 {q14-q15},[r12]!
> + veor q0,q0,q12 @ xor with input
> + veor q1,q1,q13
> + vld1.8 {q12-q13},[r12]!
> + veor q2,q2,q14
> + veor q3,q3,q15
> + vld1.8 {q14-q15},[r12]!
> +
> + veor q4,q4,q12
> + vst1.8 {q0-q1},[r14]! @ store output
> + veor q5,q5,q13
> + vld1.8 {q12-q13},[r12]!
> + veor q6,q6,q14
> + vst1.8 {q2-q3},[r14]!
> + veor q7,q7,q15
> + vld1.8 {q14-q15},[r12]!
> +
> + veor q8,q8,q12
> + vld1.32 {q0-q1},[r11]! @ load for next iteration
> + veor d25,d25,d25
> + vldr d24,[sp,#4*(16+4)] @ four
> + veor q9,q9,q13
> + vld1.32 {q2-q3},[r11]
> + veor q10,q10,q14
> + vst1.8 {q4-q5},[r14]!
> + veor q11,q11,q15
> + vst1.8 {q6-q7},[r14]!
> +
> + vadd.i32 d6,d6,d24 @ next counter value
> + vldr d24,[sp,#4*(16+0)] @ one
> +
> + ldmia sp,{r8-r11} @ load key material
> + add r0,r0,r8 @ accumulate key material
> + ldr r8,[r12],#16 @ load input
> + vst1.8 {q8-q9},[r14]!
> + add r1,r1,r9
> + ldr r9,[r12,#-12]
> + vst1.8 {q10-q11},[r14]!
> + add r2,r2,r10
> + ldr r10,[r12,#-8]
> + add r3,r3,r11
> + ldr r11,[r12,#-4]
> +# ifdef __ARMEB__
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> +# endif
> + eor r0,r0,r8 @ xor with input
> + add r8,sp,#4*(4)
> + eor r1,r1,r9
> + str r0,[r14],#16 @ store output
> + eor r2,r2,r10
> + str r1,[r14,#-12]
> + eor r3,r3,r11
> + ldmia r8,{r8-r11} @ load key material
> + str r2,[r14,#-8]
> + str r3,[r14,#-4]
> +
> + add r4,r8,r4,ror#13 @ accumulate key material
> + ldr r8,[r12],#16 @ load input
> + add r5,r9,r5,ror#13
> + ldr r9,[r12,#-12]
> + add r6,r10,r6,ror#13
> + ldr r10,[r12,#-8]
> + add r7,r11,r7,ror#13
> + ldr r11,[r12,#-4]
> +# ifdef __ARMEB__
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> + eor r4,r4,r8
> + add r8,sp,#4*(8)
> + eor r5,r5,r9
> + str r4,[r14],#16 @ store output
> + eor r6,r6,r10
> + str r5,[r14,#-12]
> + eor r7,r7,r11
> + ldmia r8,{r8-r11} @ load key material
> + str r6,[r14,#-8]
> + add r0,sp,#4*(16+8)
> + str r7,[r14,#-4]
> +
> + ldmia r0,{r0-r7} @ load second half
> +
> + add r0,r0,r8 @ accumulate key material
> + ldr r8,[r12],#16 @ load input
> + add r1,r1,r9
> + ldr r9,[r12,#-12]
> +# ifdef __thumb2__
> + it hi
> +# endif
> + strhi r10,[sp,#4*(16+10)] @ copy "rx" while at it
> + add r2,r2,r10
> + ldr r10,[r12,#-8]
> +# ifdef __thumb2__
> + it hi
> +# endif
> + strhi r11,[sp,#4*(16+11)] @ copy "rx" while at it
> + add r3,r3,r11
> + ldr r11,[r12,#-4]
> +# ifdef __ARMEB__
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> +# endif
> + eor r0,r0,r8
> + add r8,sp,#4*(12)
> + eor r1,r1,r9
> + str r0,[r14],#16 @ store output
> + eor r2,r2,r10
> + str r1,[r14,#-12]
> + eor r3,r3,r11
> + ldmia r8,{r8-r11} @ load key material
> + str r2,[r14,#-8]
> + str r3,[r14,#-4]
> +
> + add r4,r8,r4,ror#24 @ accumulate key material
> + add r8,r8,#4 @ next counter value
> + add r5,r9,r5,ror#24
> + str r8,[sp,#4*(12)] @ save next counter value
> + ldr r8,[r12],#16 @ load input
> + add r6,r10,r6,ror#24
> + add r4,r4,#3 @ counter+3
> + ldr r9,[r12,#-12]
> + add r7,r11,r7,ror#24
> + ldr r10,[r12,#-8]
> + ldr r11,[r12,#-4]
> +# ifdef __ARMEB__
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> + eor r4,r4,r8
> +# ifdef __thumb2__
> + it hi
> +# endif
> + ldrhi r8,[sp,#4*(32+2)] @ re-load len
> + eor r5,r5,r9
> + eor r6,r6,r10
> + str r4,[r14],#16 @ store output
> + eor r7,r7,r11
> + str r5,[r14,#-12]
> + sub r11,r8,#64*4 @ len-=64*4
> + str r6,[r14,#-8]
> + str r7,[r14,#-4]
> + bhi .Loop_neon_outer
> +
> + b .Ldone_neon
> +
> +.align 4
> +.Lbreak_neon:
> + @ harmonize NEON and integer-only stack frames: load data
> + @ from NEON frame, but save to integer-only one; distance
> + @ between the two is 4*(32+4+16-32)=4*(20).
> +
> + str r11, [sp,#4*(20+32+2)] @ save len
> + add r11,sp,#4*(32+4)
> + str r12, [sp,#4*(20+32+1)] @ save inp
> + str r14, [sp,#4*(20+32+0)] @ save out
> +
> + ldr r12,[sp,#4*(16+10)]
> + ldr r14,[sp,#4*(16+11)]
> + vldmia r11,{d8-d15} @ fulfill ABI requirement
> + str r12,[sp,#4*(20+16+10)] @ copy "rx"
> + str r14,[sp,#4*(20+16+11)] @ copy "rx"
> +
> + ldr r11, [sp,#4*(15)]
> + mov r4,r4,ror#19 @ twist b[0..3]
> + ldr r12,[sp,#4*(12)] @ modulo-scheduled load
> + mov r5,r5,ror#19
> + ldr r10, [sp,#4*(13)]
> + mov r6,r6,ror#19
> + ldr r14,[sp,#4*(14)]
> + mov r7,r7,ror#19
> + mov r11,r11,ror#8 @ twist d[0..3]
> + mov r12,r12,ror#8
> + mov r10,r10,ror#8
> + mov r14,r14,ror#8
> + str r11, [sp,#4*(20+16+15)]
> + add r11,sp,#4*(20)
> + vst1.32 {q0-q1},[r11]! @ copy key
> + add sp,sp,#4*(20) @ switch frame
> + vst1.32 {q2-q3},[r11]
> + mov r11,#10
> + b .Loop @ go integer-only
> +
> +.align 4
> +.Ltail_neon:
> + cmp r11,#64*3
> + bhs .L192_or_more_neon
> + cmp r11,#64*2
> + bhs .L128_or_more_neon
> + cmp r11,#64*1
> + bhs .L64_or_more_neon
> +
> + add r8,sp,#4*(8)
> + vst1.8 {q0-q1},[sp]
> + add r10,sp,#4*(0)
> + vst1.8 {q2-q3},[r8]
> + b .Loop_tail_neon
> +
> +.align 4
> +.L64_or_more_neon:
> + vld1.8 {q12-q13},[r12]!
> + vld1.8 {q14-q15},[r12]!
> + veor q0,q0,q12
> + veor q1,q1,q13
> + veor q2,q2,q14
> + veor q3,q3,q15
> + vst1.8 {q0-q1},[r14]!
> + vst1.8 {q2-q3},[r14]!
> +
> + beq .Ldone_neon
> +
> + add r8,sp,#4*(8)
> + vst1.8 {q4-q5},[sp]
> + add r10,sp,#4*(0)
> + vst1.8 {q6-q7},[r8]
> + sub r11,r11,#64*1 @ len-=64*1
> + b .Loop_tail_neon
> +
> +.align 4
> +.L128_or_more_neon:
> + vld1.8 {q12-q13},[r12]!
> + vld1.8 {q14-q15},[r12]!
> + veor q0,q0,q12
> + veor q1,q1,q13
> + vld1.8 {q12-q13},[r12]!
> + veor q2,q2,q14
> + veor q3,q3,q15
> + vld1.8 {q14-q15},[r12]!
> +
> + veor q4,q4,q12
> + veor q5,q5,q13
> + vst1.8 {q0-q1},[r14]!
> + veor q6,q6,q14
> + vst1.8 {q2-q3},[r14]!
> + veor q7,q7,q15
> + vst1.8 {q4-q5},[r14]!
> + vst1.8 {q6-q7},[r14]!
> +
> + beq .Ldone_neon
> +
> + add r8,sp,#4*(8)
> + vst1.8 {q8-q9},[sp]
> + add r10,sp,#4*(0)
> + vst1.8 {q10-q11},[r8]
> + sub r11,r11,#64*2 @ len-=64*2
> + b .Loop_tail_neon
> +
> +.align 4
> +.L192_or_more_neon:
> + vld1.8 {q12-q13},[r12]!
> + vld1.8 {q14-q15},[r12]!
> + veor q0,q0,q12
> + veor q1,q1,q13
> + vld1.8 {q12-q13},[r12]!
> + veor q2,q2,q14
> + veor q3,q3,q15
> + vld1.8 {q14-q15},[r12]!
> +
> + veor q4,q4,q12
> + veor q5,q5,q13
> + vld1.8 {q12-q13},[r12]!
> + veor q6,q6,q14
> + vst1.8 {q0-q1},[r14]!
> + veor q7,q7,q15
> + vld1.8 {q14-q15},[r12]!
> +
> + veor q8,q8,q12
> + vst1.8 {q2-q3},[r14]!
> + veor q9,q9,q13
> + vst1.8 {q4-q5},[r14]!
> + veor q10,q10,q14
> + vst1.8 {q6-q7},[r14]!
> + veor q11,q11,q15
> + vst1.8 {q8-q9},[r14]!
> + vst1.8 {q10-q11},[r14]!
> +
> + beq .Ldone_neon
> +
> + ldmia sp,{r8-r11} @ load key material
> + add r0,r0,r8 @ accumulate key material
> + add r8,sp,#4*(4)
> + add r1,r1,r9
> + add r2,r2,r10
> + add r3,r3,r11
> + ldmia r8,{r8-r11} @ load key material
> +
> + add r4,r8,r4,ror#13 @ accumulate key material
> + add r8,sp,#4*(8)
> + add r5,r9,r5,ror#13
> + add r6,r10,r6,ror#13
> + add r7,r11,r7,ror#13
> + ldmia r8,{r8-r11} @ load key material
> +# ifdef __ARMEB__
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> + stmia sp,{r0-r7}
> + add r0,sp,#4*(16+8)
> +
> + ldmia r0,{r0-r7} @ load second half
> +
> + add r0,r0,r8 @ accumulate key material
> + add r8,sp,#4*(12)
> + add r1,r1,r9
> + add r2,r2,r10
> + add r3,r3,r11
> + ldmia r8,{r8-r11} @ load key material
> +
> + add r4,r8,r4,ror#24 @ accumulate key material
> + add r8,sp,#4*(8)
> + add r5,r9,r5,ror#24
> + add r4,r4,#3 @ counter+3
> + add r6,r10,r6,ror#24
> + add r7,r11,r7,ror#24
> + ldr r11,[sp,#4*(32+2)] @ re-load len
> +# ifdef __ARMEB__
> + rev r0,r0
> + rev r1,r1
> + rev r2,r2
> + rev r3,r3
> + rev r4,r4
> + rev r5,r5
> + rev r6,r6
> + rev r7,r7
> +# endif
> + stmia r8,{r0-r7}
> + add r10,sp,#4*(0)
> + sub r11,r11,#64*3 @ len-=64*3
> +
> +.Loop_tail_neon:
> + ldrb r8,[r10],#1 @ read buffer on stack
> + ldrb r9,[r12],#1 @ read input
> + subs r11,r11,#1
> + eor r8,r8,r9
> + strb r8,[r14],#1 @ store output
> + bne .Loop_tail_neon
> +
> +.Ldone_neon:
> + add sp,sp,#4*(32+4)
> + vldmia sp,{d8-d15}
> + add sp,sp,#4*(16+3)
> + ldmia sp!,{r4-r11,pc}
> +.size ChaCha20_neon,.-ChaCha20_neon
> +.comm OPENSSL_armcap_P,4,4
> +#endif
> diff --git a/lib/zinc/chacha20/chacha20-arm64-cryptogams.S b/lib/zinc/chacha20/chacha20-arm64-cryptogams.S
> new file mode 100644
> index 000000000000..4d029bfdad3a
> --- /dev/null
> +++ b/lib/zinc/chacha20/chacha20-arm64-cryptogams.S
> @@ -0,0 +1,1973 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
> +/*
> + * Copyright (C) 2006-2017 CRYPTOGAMS by <appro@openssl.org>. All Rights Reserved.
> + */
> +
> +#include "arm_arch.h"
> +
> +.text
> +
> +
> +
> +.align 5
> +.Lsigma:
> +.quad 0x3320646e61707865,0x6b20657479622d32 // endian-neutral
> +.Lone:
> +.long 1,0,0,0
> +.LOPENSSL_armcap_P:
> +#ifdef __ILP32__
> +.long OPENSSL_armcap_P-.
> +#else
> +.quad OPENSSL_armcap_P-.
> +#endif
> +.byte 67,104,97,67,104,97,50,48,32,102,111,114,32,65,82,77,118,56,44,32,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0
> +.align 2
> +
> +.globl ChaCha20_ctr32
> +.type ChaCha20_ctr32,%function
> +.align 5
> +ChaCha20_ctr32:
> + cbz x2,.Labort
> + adr x5,.LOPENSSL_armcap_P
> + cmp x2,#192
> + b.lo .Lshort
> +#ifdef __ILP32__
> + ldrsw x6,[x5]
> +#else
> + ldr x6,[x5]
> +#endif
> + ldr w17,[x6,x5]
> + tst w17,#ARMV7_NEON
> + b.ne ChaCha20_neon
> +
> +.Lshort:
> + stp x29,x30,[sp,#-96]!
> + add x29,sp,#0
> +
> + adr x5,.Lsigma
> + stp x19,x20,[sp,#16]
> + stp x21,x22,[sp,#32]
> + stp x23,x24,[sp,#48]
> + stp x25,x26,[sp,#64]
> + stp x27,x28,[sp,#80]
> + sub sp,sp,#64
> +
> + ldp x22,x23,[x5] // load sigma
> + ldp x24,x25,[x3] // load key
> + ldp x26,x27,[x3,#16]
> + ldp x28,x30,[x4] // load counter
> +#ifdef __ARMEB__
> + ror x24,x24,#32
> + ror x25,x25,#32
> + ror x26,x26,#32
> + ror x27,x27,#32
> + ror x28,x28,#32
> + ror x30,x30,#32
> +#endif
> +
> +.Loop_outer:
> + mov w5,w22 // unpack key block
> + lsr x6,x22,#32
> + mov w7,w23
> + lsr x8,x23,#32
> + mov w9,w24
> + lsr x10,x24,#32
> + mov w11,w25
> + lsr x12,x25,#32
> + mov w13,w26
> + lsr x14,x26,#32
> + mov w15,w27
> + lsr x16,x27,#32
> + mov w17,w28
> + lsr x19,x28,#32
> + mov w20,w30
> + lsr x21,x30,#32
> +
> + mov x4,#10
> + subs x2,x2,#64
> +.Loop:
> + sub x4,x4,#1
> + add w5,w5,w9
> + add w6,w6,w10
> + add w7,w7,w11
> + add w8,w8,w12
> + eor w17,w17,w5
> + eor w19,w19,w6
> + eor w20,w20,w7
> + eor w21,w21,w8
> + ror w17,w17,#16
> + ror w19,w19,#16
> + ror w20,w20,#16
> + ror w21,w21,#16
> + add w13,w13,w17
> + add w14,w14,w19
> + add w15,w15,w20
> + add w16,w16,w21
> + eor w9,w9,w13
> + eor w10,w10,w14
> + eor w11,w11,w15
> + eor w12,w12,w16
> + ror w9,w9,#20
> + ror w10,w10,#20
> + ror w11,w11,#20
> + ror w12,w12,#20
> + add w5,w5,w9
> + add w6,w6,w10
> + add w7,w7,w11
> + add w8,w8,w12
> + eor w17,w17,w5
> + eor w19,w19,w6
> + eor w20,w20,w7
> + eor w21,w21,w8
> + ror w17,w17,#24
> + ror w19,w19,#24
> + ror w20,w20,#24
> + ror w21,w21,#24
> + add w13,w13,w17
> + add w14,w14,w19
> + add w15,w15,w20
> + add w16,w16,w21
> + eor w9,w9,w13
> + eor w10,w10,w14
> + eor w11,w11,w15
> + eor w12,w12,w16
> + ror w9,w9,#25
> + ror w10,w10,#25
> + ror w11,w11,#25
> + ror w12,w12,#25
> + add w5,w5,w10
> + add w6,w6,w11
> + add w7,w7,w12
> + add w8,w8,w9
> + eor w21,w21,w5
> + eor w17,w17,w6
> + eor w19,w19,w7
> + eor w20,w20,w8
> + ror w21,w21,#16
> + ror w17,w17,#16
> + ror w19,w19,#16
> + ror w20,w20,#16
> + add w15,w15,w21
> + add w16,w16,w17
> + add w13,w13,w19
> + add w14,w14,w20
> + eor w10,w10,w15
> + eor w11,w11,w16
> + eor w12,w12,w13
> + eor w9,w9,w14
> + ror w10,w10,#20
> + ror w11,w11,#20
> + ror w12,w12,#20
> + ror w9,w9,#20
> + add w5,w5,w10
> + add w6,w6,w11
> + add w7,w7,w12
> + add w8,w8,w9
> + eor w21,w21,w5
> + eor w17,w17,w6
> + eor w19,w19,w7
> + eor w20,w20,w8
> + ror w21,w21,#24
> + ror w17,w17,#24
> + ror w19,w19,#24
> + ror w20,w20,#24
> + add w15,w15,w21
> + add w16,w16,w17
> + add w13,w13,w19
> + add w14,w14,w20
> + eor w10,w10,w15
> + eor w11,w11,w16
> + eor w12,w12,w13
> + eor w9,w9,w14
> + ror w10,w10,#25
> + ror w11,w11,#25
> + ror w12,w12,#25
> + ror w9,w9,#25
> + cbnz x4,.Loop
> +
> + add w5,w5,w22 // accumulate key block
> + add x6,x6,x22,lsr#32
> + add w7,w7,w23
> + add x8,x8,x23,lsr#32
> + add w9,w9,w24
> + add x10,x10,x24,lsr#32
> + add w11,w11,w25
> + add x12,x12,x25,lsr#32
> + add w13,w13,w26
> + add x14,x14,x26,lsr#32
> + add w15,w15,w27
> + add x16,x16,x27,lsr#32
> + add w17,w17,w28
> + add x19,x19,x28,lsr#32
> + add w20,w20,w30
> + add x21,x21,x30,lsr#32
> +
> + b.lo .Ltail
> +
> + add x5,x5,x6,lsl#32 // pack
> + add x7,x7,x8,lsl#32
> + ldp x6,x8,[x1,#0] // load input
> + add x9,x9,x10,lsl#32
> + add x11,x11,x12,lsl#32
> + ldp x10,x12,[x1,#16]
> + add x13,x13,x14,lsl#32
> + add x15,x15,x16,lsl#32
> + ldp x14,x16,[x1,#32]
> + add x17,x17,x19,lsl#32
> + add x20,x20,x21,lsl#32
> + ldp x19,x21,[x1,#48]
> + add x1,x1,#64
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + eor x5,x5,x6
> + eor x7,x7,x8
> + eor x9,x9,x10
> + eor x11,x11,x12
> + eor x13,x13,x14
> + eor x15,x15,x16
> + eor x17,x17,x19
> + eor x20,x20,x21
> +
> + stp x5,x7,[x0,#0] // store output
> + add x28,x28,#1 // increment counter
> + stp x9,x11,[x0,#16]
> + stp x13,x15,[x0,#32]
> + stp x17,x20,[x0,#48]
> + add x0,x0,#64
> +
> + b.hi .Loop_outer
> +
> + ldp x19,x20,[x29,#16]
> + add sp,sp,#64
> + ldp x21,x22,[x29,#32]
> + ldp x23,x24,[x29,#48]
> + ldp x25,x26,[x29,#64]
> + ldp x27,x28,[x29,#80]
> + ldp x29,x30,[sp],#96
> +.Labort:
> + ret
> +
> +.align 4
> +.Ltail:
> + add x2,x2,#64
> +.Less_than_64:
> + sub x0,x0,#1
> + add x1,x1,x2
> + add x0,x0,x2
> + add x4,sp,x2
> + neg x2,x2
> +
> + add x5,x5,x6,lsl#32 // pack
> + add x7,x7,x8,lsl#32
> + add x9,x9,x10,lsl#32
> + add x11,x11,x12,lsl#32
> + add x13,x13,x14,lsl#32
> + add x15,x15,x16,lsl#32
> + add x17,x17,x19,lsl#32
> + add x20,x20,x21,lsl#32
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + stp x5,x7,[sp,#0]
> + stp x9,x11,[sp,#16]
> + stp x13,x15,[sp,#32]
> + stp x17,x20,[sp,#48]
> +
> +.Loop_tail:
> + ldrb w10,[x1,x2]
> + ldrb w11,[x4,x2]
> + add x2,x2,#1
> + eor w10,w10,w11
> + strb w10,[x0,x2]
> + cbnz x2,.Loop_tail
> +
> + stp xzr,xzr,[sp,#0]
> + stp xzr,xzr,[sp,#16]
> + stp xzr,xzr,[sp,#32]
> + stp xzr,xzr,[sp,#48]
> +
> + ldp x19,x20,[x29,#16]
> + add sp,sp,#64
> + ldp x21,x22,[x29,#32]
> + ldp x23,x24,[x29,#48]
> + ldp x25,x26,[x29,#64]
> + ldp x27,x28,[x29,#80]
> + ldp x29,x30,[sp],#96
> + ret
> +.size ChaCha20_ctr32,.-ChaCha20_ctr32
> +
> +.type ChaCha20_neon,%function
> +.align 5
> +ChaCha20_neon:
> + stp x29,x30,[sp,#-96]!
> + add x29,sp,#0
> +
> + adr x5,.Lsigma
> + stp x19,x20,[sp,#16]
> + stp x21,x22,[sp,#32]
> + stp x23,x24,[sp,#48]
> + stp x25,x26,[sp,#64]
> + stp x27,x28,[sp,#80]
> + cmp x2,#512
> + b.hs .L512_or_more_neon
> +
> + sub sp,sp,#64
> +
> + ldp x22,x23,[x5] // load sigma
> + ld1 {v24.4s},[x5],#16
> + ldp x24,x25,[x3] // load key
> + ldp x26,x27,[x3,#16]
> + ld1 {v25.4s,v26.4s},[x3]
> + ldp x28,x30,[x4] // load counter
> + ld1 {v27.4s},[x4]
> + ld1 {v31.4s},[x5]
> +#ifdef __ARMEB__
> + rev64 v24.4s,v24.4s
> + ror x24,x24,#32
> + ror x25,x25,#32
> + ror x26,x26,#32
> + ror x27,x27,#32
> + ror x28,x28,#32
> + ror x30,x30,#32
> +#endif
> + add v27.4s,v27.4s,v31.4s // += 1
> + add v28.4s,v27.4s,v31.4s
> + add v29.4s,v28.4s,v31.4s
> + shl v31.4s,v31.4s,#2 // 1 -> 4
> +
> +.Loop_outer_neon:
> + mov w5,w22 // unpack key block
> + lsr x6,x22,#32
> + mov v0.16b,v24.16b
> + mov w7,w23
> + lsr x8,x23,#32
> + mov v4.16b,v24.16b
> + mov w9,w24
> + lsr x10,x24,#32
> + mov v16.16b,v24.16b
> + mov w11,w25
> + mov v1.16b,v25.16b
> + lsr x12,x25,#32
> + mov v5.16b,v25.16b
> + mov w13,w26
> + mov v17.16b,v25.16b
> + lsr x14,x26,#32
> + mov v3.16b,v27.16b
> + mov w15,w27
> + mov v7.16b,v28.16b
> + lsr x16,x27,#32
> + mov v19.16b,v29.16b
> + mov w17,w28
> + mov v2.16b,v26.16b
> + lsr x19,x28,#32
> + mov v6.16b,v26.16b
> + mov w20,w30
> + mov v18.16b,v26.16b
> + lsr x21,x30,#32
> +
> + mov x4,#10
> + subs x2,x2,#256
> +.Loop_neon:
> + sub x4,x4,#1
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w9
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w10
> + add v16.4s,v16.4s,v17.4s
> + add w7,w7,w11
> + eor v3.16b,v3.16b,v0.16b
> + add w8,w8,w12
> + eor v7.16b,v7.16b,v4.16b
> + eor w17,w17,w5
> + eor v19.16b,v19.16b,v16.16b
> + eor w19,w19,w6
> + rev32 v3.8h,v3.8h
> + eor w20,w20,w7
> + rev32 v7.8h,v7.8h
> + eor w21,w21,w8
> + rev32 v19.8h,v19.8h
> + ror w17,w17,#16
> + add v2.4s,v2.4s,v3.4s
> + ror w19,w19,#16
> + add v6.4s,v6.4s,v7.4s
> + ror w20,w20,#16
> + add v18.4s,v18.4s,v19.4s
> + ror w21,w21,#16
> + eor v20.16b,v1.16b,v2.16b
> + add w13,w13,w17
> + eor v21.16b,v5.16b,v6.16b
> + add w14,w14,w19
> + eor v22.16b,v17.16b,v18.16b
> + add w15,w15,w20
> + ushr v1.4s,v20.4s,#20
> + add w16,w16,w21
> + ushr v5.4s,v21.4s,#20
> + eor w9,w9,w13
> + ushr v17.4s,v22.4s,#20
> + eor w10,w10,w14
> + sli v1.4s,v20.4s,#12
> + eor w11,w11,w15
> + sli v5.4s,v21.4s,#12
> + eor w12,w12,w16
> + sli v17.4s,v22.4s,#12
> + ror w9,w9,#20
> + add v0.4s,v0.4s,v1.4s
> + ror w10,w10,#20
> + add v4.4s,v4.4s,v5.4s
> + ror w11,w11,#20
> + add v16.4s,v16.4s,v17.4s
> + ror w12,w12,#20
> + eor v20.16b,v3.16b,v0.16b
> + add w5,w5,w9
> + eor v21.16b,v7.16b,v4.16b
> + add w6,w6,w10
> + eor v22.16b,v19.16b,v16.16b
> + add w7,w7,w11
> + ushr v3.4s,v20.4s,#24
> + add w8,w8,w12
> + ushr v7.4s,v21.4s,#24
> + eor w17,w17,w5
> + ushr v19.4s,v22.4s,#24
> + eor w19,w19,w6
> + sli v3.4s,v20.4s,#8
> + eor w20,w20,w7
> + sli v7.4s,v21.4s,#8
> + eor w21,w21,w8
> + sli v19.4s,v22.4s,#8
> + ror w17,w17,#24
> + add v2.4s,v2.4s,v3.4s
> + ror w19,w19,#24
> + add v6.4s,v6.4s,v7.4s
> + ror w20,w20,#24
> + add v18.4s,v18.4s,v19.4s
> + ror w21,w21,#24
> + eor v20.16b,v1.16b,v2.16b
> + add w13,w13,w17
> + eor v21.16b,v5.16b,v6.16b
> + add w14,w14,w19
> + eor v22.16b,v17.16b,v18.16b
> + add w15,w15,w20
> + ushr v1.4s,v20.4s,#25
> + add w16,w16,w21
> + ushr v5.4s,v21.4s,#25
> + eor w9,w9,w13
> + ushr v17.4s,v22.4s,#25
> + eor w10,w10,w14
> + sli v1.4s,v20.4s,#7
> + eor w11,w11,w15
> + sli v5.4s,v21.4s,#7
> + eor w12,w12,w16
> + sli v17.4s,v22.4s,#7
> + ror w9,w9,#25
> + ext v2.16b,v2.16b,v2.16b,#8
> + ror w10,w10,#25
> + ext v6.16b,v6.16b,v6.16b,#8
> + ror w11,w11,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w12,w12,#25
> + ext v3.16b,v3.16b,v3.16b,#12
> + ext v7.16b,v7.16b,v7.16b,#12
> + ext v19.16b,v19.16b,v19.16b,#12
> + ext v1.16b,v1.16b,v1.16b,#4
> + ext v5.16b,v5.16b,v5.16b,#4
> + ext v17.16b,v17.16b,v17.16b,#4
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w10
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w11
> + add v16.4s,v16.4s,v17.4s
> + add w7,w7,w12
> + eor v3.16b,v3.16b,v0.16b
> + add w8,w8,w9
> + eor v7.16b,v7.16b,v4.16b
> + eor w21,w21,w5
> + eor v19.16b,v19.16b,v16.16b
> + eor w17,w17,w6
> + rev32 v3.8h,v3.8h
> + eor w19,w19,w7
> + rev32 v7.8h,v7.8h
> + eor w20,w20,w8
> + rev32 v19.8h,v19.8h
> + ror w21,w21,#16
> + add v2.4s,v2.4s,v3.4s
> + ror w17,w17,#16
> + add v6.4s,v6.4s,v7.4s
> + ror w19,w19,#16
> + add v18.4s,v18.4s,v19.4s
> + ror w20,w20,#16
> + eor v20.16b,v1.16b,v2.16b
> + add w15,w15,w21
> + eor v21.16b,v5.16b,v6.16b
> + add w16,w16,w17
> + eor v22.16b,v17.16b,v18.16b
> + add w13,w13,w19
> + ushr v1.4s,v20.4s,#20
> + add w14,w14,w20
> + ushr v5.4s,v21.4s,#20
> + eor w10,w10,w15
> + ushr v17.4s,v22.4s,#20
> + eor w11,w11,w16
> + sli v1.4s,v20.4s,#12
> + eor w12,w12,w13
> + sli v5.4s,v21.4s,#12
> + eor w9,w9,w14
> + sli v17.4s,v22.4s,#12
> + ror w10,w10,#20
> + add v0.4s,v0.4s,v1.4s
> + ror w11,w11,#20
> + add v4.4s,v4.4s,v5.4s
> + ror w12,w12,#20
> + add v16.4s,v16.4s,v17.4s
> + ror w9,w9,#20
> + eor v20.16b,v3.16b,v0.16b
> + add w5,w5,w10
> + eor v21.16b,v7.16b,v4.16b
> + add w6,w6,w11
> + eor v22.16b,v19.16b,v16.16b
> + add w7,w7,w12
> + ushr v3.4s,v20.4s,#24
> + add w8,w8,w9
> + ushr v7.4s,v21.4s,#24
> + eor w21,w21,w5
> + ushr v19.4s,v22.4s,#24
> + eor w17,w17,w6
> + sli v3.4s,v20.4s,#8
> + eor w19,w19,w7
> + sli v7.4s,v21.4s,#8
> + eor w20,w20,w8
> + sli v19.4s,v22.4s,#8
> + ror w21,w21,#24
> + add v2.4s,v2.4s,v3.4s
> + ror w17,w17,#24
> + add v6.4s,v6.4s,v7.4s
> + ror w19,w19,#24
> + add v18.4s,v18.4s,v19.4s
> + ror w20,w20,#24
> + eor v20.16b,v1.16b,v2.16b
> + add w15,w15,w21
> + eor v21.16b,v5.16b,v6.16b
> + add w16,w16,w17
> + eor v22.16b,v17.16b,v18.16b
> + add w13,w13,w19
> + ushr v1.4s,v20.4s,#25
> + add w14,w14,w20
> + ushr v5.4s,v21.4s,#25
> + eor w10,w10,w15
> + ushr v17.4s,v22.4s,#25
> + eor w11,w11,w16
> + sli v1.4s,v20.4s,#7
> + eor w12,w12,w13
> + sli v5.4s,v21.4s,#7
> + eor w9,w9,w14
> + sli v17.4s,v22.4s,#7
> + ror w10,w10,#25
> + ext v2.16b,v2.16b,v2.16b,#8
> + ror w11,w11,#25
> + ext v6.16b,v6.16b,v6.16b,#8
> + ror w12,w12,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w9,w9,#25
> + ext v3.16b,v3.16b,v3.16b,#4
> + ext v7.16b,v7.16b,v7.16b,#4
> + ext v19.16b,v19.16b,v19.16b,#4
> + ext v1.16b,v1.16b,v1.16b,#12
> + ext v5.16b,v5.16b,v5.16b,#12
> + ext v17.16b,v17.16b,v17.16b,#12
> + cbnz x4,.Loop_neon
> +
> + add w5,w5,w22 // accumulate key block
> + add v0.4s,v0.4s,v24.4s
> + add x6,x6,x22,lsr#32
> + add v4.4s,v4.4s,v24.4s
> + add w7,w7,w23
> + add v16.4s,v16.4s,v24.4s
> + add x8,x8,x23,lsr#32
> + add v2.4s,v2.4s,v26.4s
> + add w9,w9,w24
> + add v6.4s,v6.4s,v26.4s
> + add x10,x10,x24,lsr#32
> + add v18.4s,v18.4s,v26.4s
> + add w11,w11,w25
> + add v3.4s,v3.4s,v27.4s
> + add x12,x12,x25,lsr#32
> + add w13,w13,w26
> + add v7.4s,v7.4s,v28.4s
> + add x14,x14,x26,lsr#32
> + add w15,w15,w27
> + add v19.4s,v19.4s,v29.4s
> + add x16,x16,x27,lsr#32
> + add w17,w17,w28
> + add v1.4s,v1.4s,v25.4s
> + add x19,x19,x28,lsr#32
> + add w20,w20,w30
> + add v5.4s,v5.4s,v25.4s
> + add x21,x21,x30,lsr#32
> + add v17.4s,v17.4s,v25.4s
> +
> + b.lo .Ltail_neon
> +
> + add x5,x5,x6,lsl#32 // pack
> + add x7,x7,x8,lsl#32
> + ldp x6,x8,[x1,#0] // load input
> + add x9,x9,x10,lsl#32
> + add x11,x11,x12,lsl#32
> + ldp x10,x12,[x1,#16]
> + add x13,x13,x14,lsl#32
> + add x15,x15,x16,lsl#32
> + ldp x14,x16,[x1,#32]
> + add x17,x17,x19,lsl#32
> + add x20,x20,x21,lsl#32
> + ldp x19,x21,[x1,#48]
> + add x1,x1,#64
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
> + eor x5,x5,x6
> + eor x7,x7,x8
> + eor x9,x9,x10
> + eor x11,x11,x12
> + eor x13,x13,x14
> + eor v0.16b,v0.16b,v20.16b
> + eor x15,x15,x16
> + eor v1.16b,v1.16b,v21.16b
> + eor x17,x17,x19
> + eor v2.16b,v2.16b,v22.16b
> + eor x20,x20,x21
> + eor v3.16b,v3.16b,v23.16b
> + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
> +
> + stp x5,x7,[x0,#0] // store output
> + add x28,x28,#4 // increment counter
> + stp x9,x11,[x0,#16]
> + add v27.4s,v27.4s,v31.4s // += 4
> + stp x13,x15,[x0,#32]
> + add v28.4s,v28.4s,v31.4s
> + stp x17,x20,[x0,#48]
> + add v29.4s,v29.4s,v31.4s
> + add x0,x0,#64
> +
> + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[x0],#64
> + ld1 {v0.16b,v1.16b,v2.16b,v3.16b},[x1],#64
> +
> + eor v4.16b,v4.16b,v20.16b
> + eor v5.16b,v5.16b,v21.16b
> + eor v6.16b,v6.16b,v22.16b
> + eor v7.16b,v7.16b,v23.16b
> + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[x0],#64
> +
> + eor v16.16b,v16.16b,v0.16b
> + eor v17.16b,v17.16b,v1.16b
> + eor v18.16b,v18.16b,v2.16b
> + eor v19.16b,v19.16b,v3.16b
> + st1 {v16.16b,v17.16b,v18.16b,v19.16b},[x0],#64
> +
> + b.hi .Loop_outer_neon
> +
> + ldp x19,x20,[x29,#16]
> + add sp,sp,#64
> + ldp x21,x22,[x29,#32]
> + ldp x23,x24,[x29,#48]
> + ldp x25,x26,[x29,#64]
> + ldp x27,x28,[x29,#80]
> + ldp x29,x30,[sp],#96
> + ret
> +
> +.Ltail_neon:
> + add x2,x2,#256
> + cmp x2,#64
> + b.lo .Less_than_64
> +
> + add x5,x5,x6,lsl#32 // pack
> + add x7,x7,x8,lsl#32
> + ldp x6,x8,[x1,#0] // load input
> + add x9,x9,x10,lsl#32
> + add x11,x11,x12,lsl#32
> + ldp x10,x12,[x1,#16]
> + add x13,x13,x14,lsl#32
> + add x15,x15,x16,lsl#32
> + ldp x14,x16,[x1,#32]
> + add x17,x17,x19,lsl#32
> + add x20,x20,x21,lsl#32
> + ldp x19,x21,[x1,#48]
> + add x1,x1,#64
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + eor x5,x5,x6
> + eor x7,x7,x8
> + eor x9,x9,x10
> + eor x11,x11,x12
> + eor x13,x13,x14
> + eor x15,x15,x16
> + eor x17,x17,x19
> + eor x20,x20,x21
> +
> + stp x5,x7,[x0,#0] // store output
> + add x28,x28,#4 // increment counter
> + stp x9,x11,[x0,#16]
> + stp x13,x15,[x0,#32]
> + stp x17,x20,[x0,#48]
> + add x0,x0,#64
> + b.eq .Ldone_neon
> + sub x2,x2,#64
> + cmp x2,#64
> + b.lo .Less_than_128
> +
> + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
> + eor v0.16b,v0.16b,v20.16b
> + eor v1.16b,v1.16b,v21.16b
> + eor v2.16b,v2.16b,v22.16b
> + eor v3.16b,v3.16b,v23.16b
> + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[x0],#64
> + b.eq .Ldone_neon
> + sub x2,x2,#64
> + cmp x2,#64
> + b.lo .Less_than_192
> +
> + ld1 {v20.16b,v21.16b,v22.16b,v23.16b},[x1],#64
> + eor v4.16b,v4.16b,v20.16b
> + eor v5.16b,v5.16b,v21.16b
> + eor v6.16b,v6.16b,v22.16b
> + eor v7.16b,v7.16b,v23.16b
> + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[x0],#64
> + b.eq .Ldone_neon
> + sub x2,x2,#64
> +
> + st1 {v16.16b,v17.16b,v18.16b,v19.16b},[sp]
> + b .Last_neon
> +
> +.Less_than_128:
> + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[sp]
> + b .Last_neon
> +.Less_than_192:
> + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[sp]
> + b .Last_neon
> +
> +.align 4
> +.Last_neon:
> + sub x0,x0,#1
> + add x1,x1,x2
> + add x0,x0,x2
> + add x4,sp,x2
> + neg x2,x2
> +
> +.Loop_tail_neon:
> + ldrb w10,[x1,x2]
> + ldrb w11,[x4,x2]
> + add x2,x2,#1
> + eor w10,w10,w11
> + strb w10,[x0,x2]
> + cbnz x2,.Loop_tail_neon
> +
> + stp xzr,xzr,[sp,#0]
> + stp xzr,xzr,[sp,#16]
> + stp xzr,xzr,[sp,#32]
> + stp xzr,xzr,[sp,#48]
> +
> +.Ldone_neon:
> + ldp x19,x20,[x29,#16]
> + add sp,sp,#64
> + ldp x21,x22,[x29,#32]
> + ldp x23,x24,[x29,#48]
> + ldp x25,x26,[x29,#64]
> + ldp x27,x28,[x29,#80]
> + ldp x29,x30,[sp],#96
> + ret
> +.size ChaCha20_neon,.-ChaCha20_neon
> +.type ChaCha20_512_neon,%function
> +.align 5
> +ChaCha20_512_neon:
> + stp x29,x30,[sp,#-96]!
> + add x29,sp,#0
> +
> + adr x5,.Lsigma
> + stp x19,x20,[sp,#16]
> + stp x21,x22,[sp,#32]
> + stp x23,x24,[sp,#48]
> + stp x25,x26,[sp,#64]
> + stp x27,x28,[sp,#80]
> +
> +.L512_or_more_neon:
> + sub sp,sp,#128+64
> +
> + ldp x22,x23,[x5] // load sigma
> + ld1 {v24.4s},[x5],#16
> + ldp x24,x25,[x3] // load key
> + ldp x26,x27,[x3,#16]
> + ld1 {v25.4s,v26.4s},[x3]
> + ldp x28,x30,[x4] // load counter
> + ld1 {v27.4s},[x4]
> + ld1 {v31.4s},[x5]
> +#ifdef __ARMEB__
> + rev64 v24.4s,v24.4s
> + ror x24,x24,#32
> + ror x25,x25,#32
> + ror x26,x26,#32
> + ror x27,x27,#32
> + ror x28,x28,#32
> + ror x30,x30,#32
> +#endif
> + add v27.4s,v27.4s,v31.4s // += 1
> + stp q24,q25,[sp,#0] // off-load key block, invariant part
> + add v27.4s,v27.4s,v31.4s // not typo
> + str q26,[sp,#32]
> + add v28.4s,v27.4s,v31.4s
> + add v29.4s,v28.4s,v31.4s
> + add v30.4s,v29.4s,v31.4s
> + shl v31.4s,v31.4s,#2 // 1 -> 4
> +
> + stp d8,d9,[sp,#128+0] // meet ABI requirements
> + stp d10,d11,[sp,#128+16]
> + stp d12,d13,[sp,#128+32]
> + stp d14,d15,[sp,#128+48]
> +
> + sub x2,x2,#512 // not typo
> +
> +.Loop_outer_512_neon:
> + mov v0.16b,v24.16b
> + mov v4.16b,v24.16b
> + mov v8.16b,v24.16b
> + mov v12.16b,v24.16b
> + mov v16.16b,v24.16b
> + mov v20.16b,v24.16b
> + mov v1.16b,v25.16b
> + mov w5,w22 // unpack key block
> + mov v5.16b,v25.16b
> + lsr x6,x22,#32
> + mov v9.16b,v25.16b
> + mov w7,w23
> + mov v13.16b,v25.16b
> + lsr x8,x23,#32
> + mov v17.16b,v25.16b
> + mov w9,w24
> + mov v21.16b,v25.16b
> + lsr x10,x24,#32
> + mov v3.16b,v27.16b
> + mov w11,w25
> + mov v7.16b,v28.16b
> + lsr x12,x25,#32
> + mov v11.16b,v29.16b
> + mov w13,w26
> + mov v15.16b,v30.16b
> + lsr x14,x26,#32
> + mov v2.16b,v26.16b
> + mov w15,w27
> + mov v6.16b,v26.16b
> + lsr x16,x27,#32
> + add v19.4s,v3.4s,v31.4s // +4
> + mov w17,w28
> + add v23.4s,v7.4s,v31.4s // +4
> + lsr x19,x28,#32
> + mov v10.16b,v26.16b
> + mov w20,w30
> + mov v14.16b,v26.16b
> + lsr x21,x30,#32
> + mov v18.16b,v26.16b
> + stp q27,q28,[sp,#48] // off-load key block, variable part
> + mov v22.16b,v26.16b
> + str q29,[sp,#80]
> +
> + mov x4,#5
> + subs x2,x2,#512
> +.Loop_upper_neon:
> + sub x4,x4,#1
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w9
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w10
> + add v8.4s,v8.4s,v9.4s
> + add w7,w7,w11
> + add v12.4s,v12.4s,v13.4s
> + add w8,w8,w12
> + add v16.4s,v16.4s,v17.4s
> + eor w17,w17,w5
> + add v20.4s,v20.4s,v21.4s
> + eor w19,w19,w6
> + eor v3.16b,v3.16b,v0.16b
> + eor w20,w20,w7
> + eor v7.16b,v7.16b,v4.16b
> + eor w21,w21,w8
> + eor v11.16b,v11.16b,v8.16b
> + ror w17,w17,#16
> + eor v15.16b,v15.16b,v12.16b
> + ror w19,w19,#16
> + eor v19.16b,v19.16b,v16.16b
> + ror w20,w20,#16
> + eor v23.16b,v23.16b,v20.16b
> + ror w21,w21,#16
> + rev32 v3.8h,v3.8h
> + add w13,w13,w17
> + rev32 v7.8h,v7.8h
> + add w14,w14,w19
> + rev32 v11.8h,v11.8h
> + add w15,w15,w20
> + rev32 v15.8h,v15.8h
> + add w16,w16,w21
> + rev32 v19.8h,v19.8h
> + eor w9,w9,w13
> + rev32 v23.8h,v23.8h
> + eor w10,w10,w14
> + add v2.4s,v2.4s,v3.4s
> + eor w11,w11,w15
> + add v6.4s,v6.4s,v7.4s
> + eor w12,w12,w16
> + add v10.4s,v10.4s,v11.4s
> + ror w9,w9,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w10,w10,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w11,w11,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w12,w12,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w9
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w10
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w11
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w12
> + eor v28.16b,v17.16b,v18.16b
> + eor w17,w17,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w19,w19,w6
> + ushr v1.4s,v24.4s,#20
> + eor w20,w20,w7
> + ushr v5.4s,v25.4s,#20
> + eor w21,w21,w8
> + ushr v9.4s,v26.4s,#20
> + ror w17,w17,#24
> + ushr v13.4s,v27.4s,#20
> + ror w19,w19,#24
> + ushr v17.4s,v28.4s,#20
> + ror w20,w20,#24
> + ushr v21.4s,v29.4s,#20
> + ror w21,w21,#24
> + sli v1.4s,v24.4s,#12
> + add w13,w13,w17
> + sli v5.4s,v25.4s,#12
> + add w14,w14,w19
> + sli v9.4s,v26.4s,#12
> + add w15,w15,w20
> + sli v13.4s,v27.4s,#12
> + add w16,w16,w21
> + sli v17.4s,v28.4s,#12
> + eor w9,w9,w13
> + sli v21.4s,v29.4s,#12
> + eor w10,w10,w14
> + add v0.4s,v0.4s,v1.4s
> + eor w11,w11,w15
> + add v4.4s,v4.4s,v5.4s
> + eor w12,w12,w16
> + add v8.4s,v8.4s,v9.4s
> + ror w9,w9,#25
> + add v12.4s,v12.4s,v13.4s
> + ror w10,w10,#25
> + add v16.4s,v16.4s,v17.4s
> + ror w11,w11,#25
> + add v20.4s,v20.4s,v21.4s
> + ror w12,w12,#25
> + eor v24.16b,v3.16b,v0.16b
> + add w5,w5,w10
> + eor v25.16b,v7.16b,v4.16b
> + add w6,w6,w11
> + eor v26.16b,v11.16b,v8.16b
> + add w7,w7,w12
> + eor v27.16b,v15.16b,v12.16b
> + add w8,w8,w9
> + eor v28.16b,v19.16b,v16.16b
> + eor w21,w21,w5
> + eor v29.16b,v23.16b,v20.16b
> + eor w17,w17,w6
> + ushr v3.4s,v24.4s,#24
> + eor w19,w19,w7
> + ushr v7.4s,v25.4s,#24
> + eor w20,w20,w8
> + ushr v11.4s,v26.4s,#24
> + ror w21,w21,#16
> + ushr v15.4s,v27.4s,#24
> + ror w17,w17,#16
> + ushr v19.4s,v28.4s,#24
> + ror w19,w19,#16
> + ushr v23.4s,v29.4s,#24
> + ror w20,w20,#16
> + sli v3.4s,v24.4s,#8
> + add w15,w15,w21
> + sli v7.4s,v25.4s,#8
> + add w16,w16,w17
> + sli v11.4s,v26.4s,#8
> + add w13,w13,w19
> + sli v15.4s,v27.4s,#8
> + add w14,w14,w20
> + sli v19.4s,v28.4s,#8
> + eor w10,w10,w15
> + sli v23.4s,v29.4s,#8
> + eor w11,w11,w16
> + add v2.4s,v2.4s,v3.4s
> + eor w12,w12,w13
> + add v6.4s,v6.4s,v7.4s
> + eor w9,w9,w14
> + add v10.4s,v10.4s,v11.4s
> + ror w10,w10,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w11,w11,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w12,w12,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w9,w9,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w10
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w11
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w12
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w9
> + eor v28.16b,v17.16b,v18.16b
> + eor w21,w21,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w17,w17,w6
> + ushr v1.4s,v24.4s,#25
> + eor w19,w19,w7
> + ushr v5.4s,v25.4s,#25
> + eor w20,w20,w8
> + ushr v9.4s,v26.4s,#25
> + ror w21,w21,#24
> + ushr v13.4s,v27.4s,#25
> + ror w17,w17,#24
> + ushr v17.4s,v28.4s,#25
> + ror w19,w19,#24
> + ushr v21.4s,v29.4s,#25
> + ror w20,w20,#24
> + sli v1.4s,v24.4s,#7
> + add w15,w15,w21
> + sli v5.4s,v25.4s,#7
> + add w16,w16,w17
> + sli v9.4s,v26.4s,#7
> + add w13,w13,w19
> + sli v13.4s,v27.4s,#7
> + add w14,w14,w20
> + sli v17.4s,v28.4s,#7
> + eor w10,w10,w15
> + sli v21.4s,v29.4s,#7
> + eor w11,w11,w16
> + ext v2.16b,v2.16b,v2.16b,#8
> + eor w12,w12,w13
> + ext v6.16b,v6.16b,v6.16b,#8
> + eor w9,w9,w14
> + ext v10.16b,v10.16b,v10.16b,#8
> + ror w10,w10,#25
> + ext v14.16b,v14.16b,v14.16b,#8
> + ror w11,w11,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w12,w12,#25
> + ext v22.16b,v22.16b,v22.16b,#8
> + ror w9,w9,#25
> + ext v3.16b,v3.16b,v3.16b,#12
> + ext v7.16b,v7.16b,v7.16b,#12
> + ext v11.16b,v11.16b,v11.16b,#12
> + ext v15.16b,v15.16b,v15.16b,#12
> + ext v19.16b,v19.16b,v19.16b,#12
> + ext v23.16b,v23.16b,v23.16b,#12
> + ext v1.16b,v1.16b,v1.16b,#4
> + ext v5.16b,v5.16b,v5.16b,#4
> + ext v9.16b,v9.16b,v9.16b,#4
> + ext v13.16b,v13.16b,v13.16b,#4
> + ext v17.16b,v17.16b,v17.16b,#4
> + ext v21.16b,v21.16b,v21.16b,#4
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w9
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w10
> + add v8.4s,v8.4s,v9.4s
> + add w7,w7,w11
> + add v12.4s,v12.4s,v13.4s
> + add w8,w8,w12
> + add v16.4s,v16.4s,v17.4s
> + eor w17,w17,w5
> + add v20.4s,v20.4s,v21.4s
> + eor w19,w19,w6
> + eor v3.16b,v3.16b,v0.16b
> + eor w20,w20,w7
> + eor v7.16b,v7.16b,v4.16b
> + eor w21,w21,w8
> + eor v11.16b,v11.16b,v8.16b
> + ror w17,w17,#16
> + eor v15.16b,v15.16b,v12.16b
> + ror w19,w19,#16
> + eor v19.16b,v19.16b,v16.16b
> + ror w20,w20,#16
> + eor v23.16b,v23.16b,v20.16b
> + ror w21,w21,#16
> + rev32 v3.8h,v3.8h
> + add w13,w13,w17
> + rev32 v7.8h,v7.8h
> + add w14,w14,w19
> + rev32 v11.8h,v11.8h
> + add w15,w15,w20
> + rev32 v15.8h,v15.8h
> + add w16,w16,w21
> + rev32 v19.8h,v19.8h
> + eor w9,w9,w13
> + rev32 v23.8h,v23.8h
> + eor w10,w10,w14
> + add v2.4s,v2.4s,v3.4s
> + eor w11,w11,w15
> + add v6.4s,v6.4s,v7.4s
> + eor w12,w12,w16
> + add v10.4s,v10.4s,v11.4s
> + ror w9,w9,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w10,w10,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w11,w11,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w12,w12,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w9
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w10
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w11
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w12
> + eor v28.16b,v17.16b,v18.16b
> + eor w17,w17,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w19,w19,w6
> + ushr v1.4s,v24.4s,#20
> + eor w20,w20,w7
> + ushr v5.4s,v25.4s,#20
> + eor w21,w21,w8
> + ushr v9.4s,v26.4s,#20
> + ror w17,w17,#24
> + ushr v13.4s,v27.4s,#20
> + ror w19,w19,#24
> + ushr v17.4s,v28.4s,#20
> + ror w20,w20,#24
> + ushr v21.4s,v29.4s,#20
> + ror w21,w21,#24
> + sli v1.4s,v24.4s,#12
> + add w13,w13,w17
> + sli v5.4s,v25.4s,#12
> + add w14,w14,w19
> + sli v9.4s,v26.4s,#12
> + add w15,w15,w20
> + sli v13.4s,v27.4s,#12
> + add w16,w16,w21
> + sli v17.4s,v28.4s,#12
> + eor w9,w9,w13
> + sli v21.4s,v29.4s,#12
> + eor w10,w10,w14
> + add v0.4s,v0.4s,v1.4s
> + eor w11,w11,w15
> + add v4.4s,v4.4s,v5.4s
> + eor w12,w12,w16
> + add v8.4s,v8.4s,v9.4s
> + ror w9,w9,#25
> + add v12.4s,v12.4s,v13.4s
> + ror w10,w10,#25
> + add v16.4s,v16.4s,v17.4s
> + ror w11,w11,#25
> + add v20.4s,v20.4s,v21.4s
> + ror w12,w12,#25
> + eor v24.16b,v3.16b,v0.16b
> + add w5,w5,w10
> + eor v25.16b,v7.16b,v4.16b
> + add w6,w6,w11
> + eor v26.16b,v11.16b,v8.16b
> + add w7,w7,w12
> + eor v27.16b,v15.16b,v12.16b
> + add w8,w8,w9
> + eor v28.16b,v19.16b,v16.16b
> + eor w21,w21,w5
> + eor v29.16b,v23.16b,v20.16b
> + eor w17,w17,w6
> + ushr v3.4s,v24.4s,#24
> + eor w19,w19,w7
> + ushr v7.4s,v25.4s,#24
> + eor w20,w20,w8
> + ushr v11.4s,v26.4s,#24
> + ror w21,w21,#16
> + ushr v15.4s,v27.4s,#24
> + ror w17,w17,#16
> + ushr v19.4s,v28.4s,#24
> + ror w19,w19,#16
> + ushr v23.4s,v29.4s,#24
> + ror w20,w20,#16
> + sli v3.4s,v24.4s,#8
> + add w15,w15,w21
> + sli v7.4s,v25.4s,#8
> + add w16,w16,w17
> + sli v11.4s,v26.4s,#8
> + add w13,w13,w19
> + sli v15.4s,v27.4s,#8
> + add w14,w14,w20
> + sli v19.4s,v28.4s,#8
> + eor w10,w10,w15
> + sli v23.4s,v29.4s,#8
> + eor w11,w11,w16
> + add v2.4s,v2.4s,v3.4s
> + eor w12,w12,w13
> + add v6.4s,v6.4s,v7.4s
> + eor w9,w9,w14
> + add v10.4s,v10.4s,v11.4s
> + ror w10,w10,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w11,w11,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w12,w12,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w9,w9,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w10
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w11
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w12
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w9
> + eor v28.16b,v17.16b,v18.16b
> + eor w21,w21,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w17,w17,w6
> + ushr v1.4s,v24.4s,#25
> + eor w19,w19,w7
> + ushr v5.4s,v25.4s,#25
> + eor w20,w20,w8
> + ushr v9.4s,v26.4s,#25
> + ror w21,w21,#24
> + ushr v13.4s,v27.4s,#25
> + ror w17,w17,#24
> + ushr v17.4s,v28.4s,#25
> + ror w19,w19,#24
> + ushr v21.4s,v29.4s,#25
> + ror w20,w20,#24
> + sli v1.4s,v24.4s,#7
> + add w15,w15,w21
> + sli v5.4s,v25.4s,#7
> + add w16,w16,w17
> + sli v9.4s,v26.4s,#7
> + add w13,w13,w19
> + sli v13.4s,v27.4s,#7
> + add w14,w14,w20
> + sli v17.4s,v28.4s,#7
> + eor w10,w10,w15
> + sli v21.4s,v29.4s,#7
> + eor w11,w11,w16
> + ext v2.16b,v2.16b,v2.16b,#8
> + eor w12,w12,w13
> + ext v6.16b,v6.16b,v6.16b,#8
> + eor w9,w9,w14
> + ext v10.16b,v10.16b,v10.16b,#8
> + ror w10,w10,#25
> + ext v14.16b,v14.16b,v14.16b,#8
> + ror w11,w11,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w12,w12,#25
> + ext v22.16b,v22.16b,v22.16b,#8
> + ror w9,w9,#25
> + ext v3.16b,v3.16b,v3.16b,#4
> + ext v7.16b,v7.16b,v7.16b,#4
> + ext v11.16b,v11.16b,v11.16b,#4
> + ext v15.16b,v15.16b,v15.16b,#4
> + ext v19.16b,v19.16b,v19.16b,#4
> + ext v23.16b,v23.16b,v23.16b,#4
> + ext v1.16b,v1.16b,v1.16b,#12
> + ext v5.16b,v5.16b,v5.16b,#12
> + ext v9.16b,v9.16b,v9.16b,#12
> + ext v13.16b,v13.16b,v13.16b,#12
> + ext v17.16b,v17.16b,v17.16b,#12
> + ext v21.16b,v21.16b,v21.16b,#12
> + cbnz x4,.Loop_upper_neon
> +
> + add w5,w5,w22 // accumulate key block
> + add x6,x6,x22,lsr#32
> + add w7,w7,w23
> + add x8,x8,x23,lsr#32
> + add w9,w9,w24
> + add x10,x10,x24,lsr#32
> + add w11,w11,w25
> + add x12,x12,x25,lsr#32
> + add w13,w13,w26
> + add x14,x14,x26,lsr#32
> + add w15,w15,w27
> + add x16,x16,x27,lsr#32
> + add w17,w17,w28
> + add x19,x19,x28,lsr#32
> + add w20,w20,w30
> + add x21,x21,x30,lsr#32
> +
> + add x5,x5,x6,lsl#32 // pack
> + add x7,x7,x8,lsl#32
> + ldp x6,x8,[x1,#0] // load input
> + add x9,x9,x10,lsl#32
> + add x11,x11,x12,lsl#32
> + ldp x10,x12,[x1,#16]
> + add x13,x13,x14,lsl#32
> + add x15,x15,x16,lsl#32
> + ldp x14,x16,[x1,#32]
> + add x17,x17,x19,lsl#32
> + add x20,x20,x21,lsl#32
> + ldp x19,x21,[x1,#48]
> + add x1,x1,#64
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + eor x5,x5,x6
> + eor x7,x7,x8
> + eor x9,x9,x10
> + eor x11,x11,x12
> + eor x13,x13,x14
> + eor x15,x15,x16
> + eor x17,x17,x19
> + eor x20,x20,x21
> +
> + stp x5,x7,[x0,#0] // store output
> + add x28,x28,#1 // increment counter
> + mov w5,w22 // unpack key block
> + lsr x6,x22,#32
> + stp x9,x11,[x0,#16]
> + mov w7,w23
> + lsr x8,x23,#32
> + stp x13,x15,[x0,#32]
> + mov w9,w24
> + lsr x10,x24,#32
> + stp x17,x20,[x0,#48]
> + add x0,x0,#64
> + mov w11,w25
> + lsr x12,x25,#32
> + mov w13,w26
> + lsr x14,x26,#32
> + mov w15,w27
> + lsr x16,x27,#32
> + mov w17,w28
> + lsr x19,x28,#32
> + mov w20,w30
> + lsr x21,x30,#32
> +
> + mov x4,#5
> +.Loop_lower_neon:
> + sub x4,x4,#1
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w9
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w10
> + add v8.4s,v8.4s,v9.4s
> + add w7,w7,w11
> + add v12.4s,v12.4s,v13.4s
> + add w8,w8,w12
> + add v16.4s,v16.4s,v17.4s
> + eor w17,w17,w5
> + add v20.4s,v20.4s,v21.4s
> + eor w19,w19,w6
> + eor v3.16b,v3.16b,v0.16b
> + eor w20,w20,w7
> + eor v7.16b,v7.16b,v4.16b
> + eor w21,w21,w8
> + eor v11.16b,v11.16b,v8.16b
> + ror w17,w17,#16
> + eor v15.16b,v15.16b,v12.16b
> + ror w19,w19,#16
> + eor v19.16b,v19.16b,v16.16b
> + ror w20,w20,#16
> + eor v23.16b,v23.16b,v20.16b
> + ror w21,w21,#16
> + rev32 v3.8h,v3.8h
> + add w13,w13,w17
> + rev32 v7.8h,v7.8h
> + add w14,w14,w19
> + rev32 v11.8h,v11.8h
> + add w15,w15,w20
> + rev32 v15.8h,v15.8h
> + add w16,w16,w21
> + rev32 v19.8h,v19.8h
> + eor w9,w9,w13
> + rev32 v23.8h,v23.8h
> + eor w10,w10,w14
> + add v2.4s,v2.4s,v3.4s
> + eor w11,w11,w15
> + add v6.4s,v6.4s,v7.4s
> + eor w12,w12,w16
> + add v10.4s,v10.4s,v11.4s
> + ror w9,w9,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w10,w10,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w11,w11,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w12,w12,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w9
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w10
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w11
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w12
> + eor v28.16b,v17.16b,v18.16b
> + eor w17,w17,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w19,w19,w6
> + ushr v1.4s,v24.4s,#20
> + eor w20,w20,w7
> + ushr v5.4s,v25.4s,#20
> + eor w21,w21,w8
> + ushr v9.4s,v26.4s,#20
> + ror w17,w17,#24
> + ushr v13.4s,v27.4s,#20
> + ror w19,w19,#24
> + ushr v17.4s,v28.4s,#20
> + ror w20,w20,#24
> + ushr v21.4s,v29.4s,#20
> + ror w21,w21,#24
> + sli v1.4s,v24.4s,#12
> + add w13,w13,w17
> + sli v5.4s,v25.4s,#12
> + add w14,w14,w19
> + sli v9.4s,v26.4s,#12
> + add w15,w15,w20
> + sli v13.4s,v27.4s,#12
> + add w16,w16,w21
> + sli v17.4s,v28.4s,#12
> + eor w9,w9,w13
> + sli v21.4s,v29.4s,#12
> + eor w10,w10,w14
> + add v0.4s,v0.4s,v1.4s
> + eor w11,w11,w15
> + add v4.4s,v4.4s,v5.4s
> + eor w12,w12,w16
> + add v8.4s,v8.4s,v9.4s
> + ror w9,w9,#25
> + add v12.4s,v12.4s,v13.4s
> + ror w10,w10,#25
> + add v16.4s,v16.4s,v17.4s
> + ror w11,w11,#25
> + add v20.4s,v20.4s,v21.4s
> + ror w12,w12,#25
> + eor v24.16b,v3.16b,v0.16b
> + add w5,w5,w10
> + eor v25.16b,v7.16b,v4.16b
> + add w6,w6,w11
> + eor v26.16b,v11.16b,v8.16b
> + add w7,w7,w12
> + eor v27.16b,v15.16b,v12.16b
> + add w8,w8,w9
> + eor v28.16b,v19.16b,v16.16b
> + eor w21,w21,w5
> + eor v29.16b,v23.16b,v20.16b
> + eor w17,w17,w6
> + ushr v3.4s,v24.4s,#24
> + eor w19,w19,w7
> + ushr v7.4s,v25.4s,#24
> + eor w20,w20,w8
> + ushr v11.4s,v26.4s,#24
> + ror w21,w21,#16
> + ushr v15.4s,v27.4s,#24
> + ror w17,w17,#16
> + ushr v19.4s,v28.4s,#24
> + ror w19,w19,#16
> + ushr v23.4s,v29.4s,#24
> + ror w20,w20,#16
> + sli v3.4s,v24.4s,#8
> + add w15,w15,w21
> + sli v7.4s,v25.4s,#8
> + add w16,w16,w17
> + sli v11.4s,v26.4s,#8
> + add w13,w13,w19
> + sli v15.4s,v27.4s,#8
> + add w14,w14,w20
> + sli v19.4s,v28.4s,#8
> + eor w10,w10,w15
> + sli v23.4s,v29.4s,#8
> + eor w11,w11,w16
> + add v2.4s,v2.4s,v3.4s
> + eor w12,w12,w13
> + add v6.4s,v6.4s,v7.4s
> + eor w9,w9,w14
> + add v10.4s,v10.4s,v11.4s
> + ror w10,w10,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w11,w11,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w12,w12,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w9,w9,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w10
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w11
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w12
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w9
> + eor v28.16b,v17.16b,v18.16b
> + eor w21,w21,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w17,w17,w6
> + ushr v1.4s,v24.4s,#25
> + eor w19,w19,w7
> + ushr v5.4s,v25.4s,#25
> + eor w20,w20,w8
> + ushr v9.4s,v26.4s,#25
> + ror w21,w21,#24
> + ushr v13.4s,v27.4s,#25
> + ror w17,w17,#24
> + ushr v17.4s,v28.4s,#25
> + ror w19,w19,#24
> + ushr v21.4s,v29.4s,#25
> + ror w20,w20,#24
> + sli v1.4s,v24.4s,#7
> + add w15,w15,w21
> + sli v5.4s,v25.4s,#7
> + add w16,w16,w17
> + sli v9.4s,v26.4s,#7
> + add w13,w13,w19
> + sli v13.4s,v27.4s,#7
> + add w14,w14,w20
> + sli v17.4s,v28.4s,#7
> + eor w10,w10,w15
> + sli v21.4s,v29.4s,#7
> + eor w11,w11,w16
> + ext v2.16b,v2.16b,v2.16b,#8
> + eor w12,w12,w13
> + ext v6.16b,v6.16b,v6.16b,#8
> + eor w9,w9,w14
> + ext v10.16b,v10.16b,v10.16b,#8
> + ror w10,w10,#25
> + ext v14.16b,v14.16b,v14.16b,#8
> + ror w11,w11,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w12,w12,#25
> + ext v22.16b,v22.16b,v22.16b,#8
> + ror w9,w9,#25
> + ext v3.16b,v3.16b,v3.16b,#12
> + ext v7.16b,v7.16b,v7.16b,#12
> + ext v11.16b,v11.16b,v11.16b,#12
> + ext v15.16b,v15.16b,v15.16b,#12
> + ext v19.16b,v19.16b,v19.16b,#12
> + ext v23.16b,v23.16b,v23.16b,#12
> + ext v1.16b,v1.16b,v1.16b,#4
> + ext v5.16b,v5.16b,v5.16b,#4
> + ext v9.16b,v9.16b,v9.16b,#4
> + ext v13.16b,v13.16b,v13.16b,#4
> + ext v17.16b,v17.16b,v17.16b,#4
> + ext v21.16b,v21.16b,v21.16b,#4
> + add v0.4s,v0.4s,v1.4s
> + add w5,w5,w9
> + add v4.4s,v4.4s,v5.4s
> + add w6,w6,w10
> + add v8.4s,v8.4s,v9.4s
> + add w7,w7,w11
> + add v12.4s,v12.4s,v13.4s
> + add w8,w8,w12
> + add v16.4s,v16.4s,v17.4s
> + eor w17,w17,w5
> + add v20.4s,v20.4s,v21.4s
> + eor w19,w19,w6
> + eor v3.16b,v3.16b,v0.16b
> + eor w20,w20,w7
> + eor v7.16b,v7.16b,v4.16b
> + eor w21,w21,w8
> + eor v11.16b,v11.16b,v8.16b
> + ror w17,w17,#16
> + eor v15.16b,v15.16b,v12.16b
> + ror w19,w19,#16
> + eor v19.16b,v19.16b,v16.16b
> + ror w20,w20,#16
> + eor v23.16b,v23.16b,v20.16b
> + ror w21,w21,#16
> + rev32 v3.8h,v3.8h
> + add w13,w13,w17
> + rev32 v7.8h,v7.8h
> + add w14,w14,w19
> + rev32 v11.8h,v11.8h
> + add w15,w15,w20
> + rev32 v15.8h,v15.8h
> + add w16,w16,w21
> + rev32 v19.8h,v19.8h
> + eor w9,w9,w13
> + rev32 v23.8h,v23.8h
> + eor w10,w10,w14
> + add v2.4s,v2.4s,v3.4s
> + eor w11,w11,w15
> + add v6.4s,v6.4s,v7.4s
> + eor w12,w12,w16
> + add v10.4s,v10.4s,v11.4s
> + ror w9,w9,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w10,w10,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w11,w11,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w12,w12,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w9
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w10
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w11
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w12
> + eor v28.16b,v17.16b,v18.16b
> + eor w17,w17,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w19,w19,w6
> + ushr v1.4s,v24.4s,#20
> + eor w20,w20,w7
> + ushr v5.4s,v25.4s,#20
> + eor w21,w21,w8
> + ushr v9.4s,v26.4s,#20
> + ror w17,w17,#24
> + ushr v13.4s,v27.4s,#20
> + ror w19,w19,#24
> + ushr v17.4s,v28.4s,#20
> + ror w20,w20,#24
> + ushr v21.4s,v29.4s,#20
> + ror w21,w21,#24
> + sli v1.4s,v24.4s,#12
> + add w13,w13,w17
> + sli v5.4s,v25.4s,#12
> + add w14,w14,w19
> + sli v9.4s,v26.4s,#12
> + add w15,w15,w20
> + sli v13.4s,v27.4s,#12
> + add w16,w16,w21
> + sli v17.4s,v28.4s,#12
> + eor w9,w9,w13
> + sli v21.4s,v29.4s,#12
> + eor w10,w10,w14
> + add v0.4s,v0.4s,v1.4s
> + eor w11,w11,w15
> + add v4.4s,v4.4s,v5.4s
> + eor w12,w12,w16
> + add v8.4s,v8.4s,v9.4s
> + ror w9,w9,#25
> + add v12.4s,v12.4s,v13.4s
> + ror w10,w10,#25
> + add v16.4s,v16.4s,v17.4s
> + ror w11,w11,#25
> + add v20.4s,v20.4s,v21.4s
> + ror w12,w12,#25
> + eor v24.16b,v3.16b,v0.16b
> + add w5,w5,w10
> + eor v25.16b,v7.16b,v4.16b
> + add w6,w6,w11
> + eor v26.16b,v11.16b,v8.16b
> + add w7,w7,w12
> + eor v27.16b,v15.16b,v12.16b
> + add w8,w8,w9
> + eor v28.16b,v19.16b,v16.16b
> + eor w21,w21,w5
> + eor v29.16b,v23.16b,v20.16b
> + eor w17,w17,w6
> + ushr v3.4s,v24.4s,#24
> + eor w19,w19,w7
> + ushr v7.4s,v25.4s,#24
> + eor w20,w20,w8
> + ushr v11.4s,v26.4s,#24
> + ror w21,w21,#16
> + ushr v15.4s,v27.4s,#24
> + ror w17,w17,#16
> + ushr v19.4s,v28.4s,#24
> + ror w19,w19,#16
> + ushr v23.4s,v29.4s,#24
> + ror w20,w20,#16
> + sli v3.4s,v24.4s,#8
> + add w15,w15,w21
> + sli v7.4s,v25.4s,#8
> + add w16,w16,w17
> + sli v11.4s,v26.4s,#8
> + add w13,w13,w19
> + sli v15.4s,v27.4s,#8
> + add w14,w14,w20
> + sli v19.4s,v28.4s,#8
> + eor w10,w10,w15
> + sli v23.4s,v29.4s,#8
> + eor w11,w11,w16
> + add v2.4s,v2.4s,v3.4s
> + eor w12,w12,w13
> + add v6.4s,v6.4s,v7.4s
> + eor w9,w9,w14
> + add v10.4s,v10.4s,v11.4s
> + ror w10,w10,#20
> + add v14.4s,v14.4s,v15.4s
> + ror w11,w11,#20
> + add v18.4s,v18.4s,v19.4s
> + ror w12,w12,#20
> + add v22.4s,v22.4s,v23.4s
> + ror w9,w9,#20
> + eor v24.16b,v1.16b,v2.16b
> + add w5,w5,w10
> + eor v25.16b,v5.16b,v6.16b
> + add w6,w6,w11
> + eor v26.16b,v9.16b,v10.16b
> + add w7,w7,w12
> + eor v27.16b,v13.16b,v14.16b
> + add w8,w8,w9
> + eor v28.16b,v17.16b,v18.16b
> + eor w21,w21,w5
> + eor v29.16b,v21.16b,v22.16b
> + eor w17,w17,w6
> + ushr v1.4s,v24.4s,#25
> + eor w19,w19,w7
> + ushr v5.4s,v25.4s,#25
> + eor w20,w20,w8
> + ushr v9.4s,v26.4s,#25
> + ror w21,w21,#24
> + ushr v13.4s,v27.4s,#25
> + ror w17,w17,#24
> + ushr v17.4s,v28.4s,#25
> + ror w19,w19,#24
> + ushr v21.4s,v29.4s,#25
> + ror w20,w20,#24
> + sli v1.4s,v24.4s,#7
> + add w15,w15,w21
> + sli v5.4s,v25.4s,#7
> + add w16,w16,w17
> + sli v9.4s,v26.4s,#7
> + add w13,w13,w19
> + sli v13.4s,v27.4s,#7
> + add w14,w14,w20
> + sli v17.4s,v28.4s,#7
> + eor w10,w10,w15
> + sli v21.4s,v29.4s,#7
> + eor w11,w11,w16
> + ext v2.16b,v2.16b,v2.16b,#8
> + eor w12,w12,w13
> + ext v6.16b,v6.16b,v6.16b,#8
> + eor w9,w9,w14
> + ext v10.16b,v10.16b,v10.16b,#8
> + ror w10,w10,#25
> + ext v14.16b,v14.16b,v14.16b,#8
> + ror w11,w11,#25
> + ext v18.16b,v18.16b,v18.16b,#8
> + ror w12,w12,#25
> + ext v22.16b,v22.16b,v22.16b,#8
> + ror w9,w9,#25
> + ext v3.16b,v3.16b,v3.16b,#4
> + ext v7.16b,v7.16b,v7.16b,#4
> + ext v11.16b,v11.16b,v11.16b,#4
> + ext v15.16b,v15.16b,v15.16b,#4
> + ext v19.16b,v19.16b,v19.16b,#4
> + ext v23.16b,v23.16b,v23.16b,#4
> + ext v1.16b,v1.16b,v1.16b,#12
> + ext v5.16b,v5.16b,v5.16b,#12
> + ext v9.16b,v9.16b,v9.16b,#12
> + ext v13.16b,v13.16b,v13.16b,#12
> + ext v17.16b,v17.16b,v17.16b,#12
> + ext v21.16b,v21.16b,v21.16b,#12
> + cbnz x4,.Loop_lower_neon
> +
> + add w5,w5,w22 // accumulate key block
> + ldp q24,q25,[sp,#0]
> + add x6,x6,x22,lsr#32
> + ldp q26,q27,[sp,#32]
> + add w7,w7,w23
> + ldp q28,q29,[sp,#64]
> + add x8,x8,x23,lsr#32
> + add v0.4s,v0.4s,v24.4s
> + add w9,w9,w24
> + add v4.4s,v4.4s,v24.4s
> + add x10,x10,x24,lsr#32
> + add v8.4s,v8.4s,v24.4s
> + add w11,w11,w25
> + add v12.4s,v12.4s,v24.4s
> + add x12,x12,x25,lsr#32
> + add v16.4s,v16.4s,v24.4s
> + add w13,w13,w26
> + add v20.4s,v20.4s,v24.4s
> + add x14,x14,x26,lsr#32
> + add v2.4s,v2.4s,v26.4s
> + add w15,w15,w27
> + add v6.4s,v6.4s,v26.4s
> + add x16,x16,x27,lsr#32
> + add v10.4s,v10.4s,v26.4s
> + add w17,w17,w28
> + add v14.4s,v14.4s,v26.4s
> + add x19,x19,x28,lsr#32
> + add v18.4s,v18.4s,v26.4s
> + add w20,w20,w30
> + add v22.4s,v22.4s,v26.4s
> + add x21,x21,x30,lsr#32
> + add v19.4s,v19.4s,v31.4s // +4
> + add x5,x5,x6,lsl#32 // pack
> + add v23.4s,v23.4s,v31.4s // +4
> + add x7,x7,x8,lsl#32
> + add v3.4s,v3.4s,v27.4s
> + ldp x6,x8,[x1,#0] // load input
> + add v7.4s,v7.4s,v28.4s
> + add x9,x9,x10,lsl#32
> + add v11.4s,v11.4s,v29.4s
> + add x11,x11,x12,lsl#32
> + add v15.4s,v15.4s,v30.4s
> + ldp x10,x12,[x1,#16]
> + add v19.4s,v19.4s,v27.4s
> + add x13,x13,x14,lsl#32
> + add v23.4s,v23.4s,v28.4s
> + add x15,x15,x16,lsl#32
> + add v1.4s,v1.4s,v25.4s
> + ldp x14,x16,[x1,#32]
> + add v5.4s,v5.4s,v25.4s
> + add x17,x17,x19,lsl#32
> + add v9.4s,v9.4s,v25.4s
> + add x20,x20,x21,lsl#32
> + add v13.4s,v13.4s,v25.4s
> + ldp x19,x21,[x1,#48]
> + add v17.4s,v17.4s,v25.4s
> + add x1,x1,#64
> + add v21.4s,v21.4s,v25.4s
> +
> +#ifdef __ARMEB__
> + rev x5,x5
> + rev x7,x7
> + rev x9,x9
> + rev x11,x11
> + rev x13,x13
> + rev x15,x15
> + rev x17,x17
> + rev x20,x20
> +#endif
> + ld1 {v24.16b,v25.16b,v26.16b,v27.16b},[x1],#64
> + eor x5,x5,x6
> + eor x7,x7,x8
> + eor x9,x9,x10
> + eor x11,x11,x12
> + eor x13,x13,x14
> + eor v0.16b,v0.16b,v24.16b
> + eor x15,x15,x16
> + eor v1.16b,v1.16b,v25.16b
> + eor x17,x17,x19
> + eor v2.16b,v2.16b,v26.16b
> + eor x20,x20,x21
> + eor v3.16b,v3.16b,v27.16b
> + ld1 {v24.16b,v25.16b,v26.16b,v27.16b},[x1],#64
> +
> + stp x5,x7,[x0,#0] // store output
> + add x28,x28,#7 // increment counter
> + stp x9,x11,[x0,#16]
> + stp x13,x15,[x0,#32]
> + stp x17,x20,[x0,#48]
> + add x0,x0,#64
> + st1 {v0.16b,v1.16b,v2.16b,v3.16b},[x0],#64
> +
> + ld1 {v0.16b,v1.16b,v2.16b,v3.16b},[x1],#64
> + eor v4.16b,v4.16b,v24.16b
> + eor v5.16b,v5.16b,v25.16b
> + eor v6.16b,v6.16b,v26.16b
> + eor v7.16b,v7.16b,v27.16b
> + st1 {v4.16b,v5.16b,v6.16b,v7.16b},[x0],#64
> +
> + ld1 {v4.16b,v5.16b,v6.16b,v7.16b},[x1],#64
> + eor v8.16b,v8.16b,v0.16b
> + ldp q24,q25,[sp,#0]
> + eor v9.16b,v9.16b,v1.16b
> + ldp q26,q27,[sp,#32]
> + eor v10.16b,v10.16b,v2.16b
> + eor v11.16b,v11.16b,v3.16b
> + st1 {v8.16b,v9.16b,v10.16b,v11.16b},[x0],#64
> +
> + ld1 {v8.16b,v9.16b,v10.16b,v11.16b},[x1],#64
> + eor v12.16b,v12.16b,v4.16b
> + eor v13.16b,v13.16b,v5.16b
> + eor v14.16b,v14.16b,v6.16b
> + eor v15.16b,v15.16b,v7.16b
> + st1 {v12.16b,v13.16b,v14.16b,v15.16b},[x0],#64
> +
> + ld1 {v12.16b,v13.16b,v14.16b,v15.16b},[x1],#64
> + eor v16.16b,v16.16b,v8.16b
> + eor v17.16b,v17.16b,v9.16b
> + eor v18.16b,v18.16b,v10.16b
> + eor v19.16b,v19.16b,v11.16b
> + st1 {v16.16b,v17.16b,v18.16b,v19.16b},[x0],#64
> +
> + shl v0.4s,v31.4s,#1 // 4 -> 8
> + eor v20.16b,v20.16b,v12.16b
> + eor v21.16b,v21.16b,v13.16b
> + eor v22.16b,v22.16b,v14.16b
> + eor v23.16b,v23.16b,v15.16b
> + st1 {v20.16b,v21.16b,v22.16b,v23.16b},[x0],#64
> +
> + add v27.4s,v27.4s,v0.4s // += 8
> + add v28.4s,v28.4s,v0.4s
> + add v29.4s,v29.4s,v0.4s
> + add v30.4s,v30.4s,v0.4s
> +
> + b.hs .Loop_outer_512_neon
> +
> + adds x2,x2,#512
> + ushr v0.4s,v31.4s,#2 // 4 -> 1
> +
> + ldp d8,d9,[sp,#128+0] // meet ABI requirements
> + ldp d10,d11,[sp,#128+16]
> + ldp d12,d13,[sp,#128+32]
> + ldp d14,d15,[sp,#128+48]
> +
> + stp q24,q31,[sp,#0] // wipe off-load area
> + stp q24,q31,[sp,#32]
> + stp q24,q31,[sp,#64]
> +
> + b.eq .Ldone_512_neon
> +
> + cmp x2,#192
> + sub v27.4s,v27.4s,v0.4s // -= 1
> + sub v28.4s,v28.4s,v0.4s
> + sub v29.4s,v29.4s,v0.4s
> + add sp,sp,#128
> + b.hs .Loop_outer_neon
> +
> + eor v25.16b,v25.16b,v25.16b
> + eor v26.16b,v26.16b,v26.16b
> + eor v27.16b,v27.16b,v27.16b
> + eor v28.16b,v28.16b,v28.16b
> + eor v29.16b,v29.16b,v29.16b
> + eor v30.16b,v30.16b,v30.16b
> + b .Loop_outer
> +
> +.Ldone_512_neon:
> + ldp x19,x20,[x29,#16]
> + add sp,sp,#128+64
> + ldp x21,x22,[x29,#32]
> + ldp x23,x24,[x29,#48]
> + ldp x25,x26,[x29,#64]
> + ldp x27,x28,[x29,#80]
> + ldp x29,x30,[sp],#96
> + ret
> +.size ChaCha20_512_neon,.-ChaCha20_512_neon
> --
> 2.19.0
>
^ permalink raw reply
* Re: [PATCH 11/15] octeontx2-af: Add Marvell OcteonTX2 CGX driver
From: Sunil Kovvuri @ 2018-09-28 9:19 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Linux Netdev List, David S. Miller, linux-soc, Sunil Goutham
In-Reply-To: <CAK8P3a1-a_io1Ene58R6i4z99s_u0XTTkdrms9m0Z0q0Y3eb+g@mail.gmail.com>
On Fri, Sep 28, 2018 at 1:45 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Fri, Sep 28, 2018 at 8:09 AM <sunil.kovvuri@gmail.com> wrote:
> >
> > From: Sunil Goutham <sgoutham@marvell.com>
> >
> > This patch adds basic template for Marvell OcteonTX2's
> > CGX ethernet interface driver. Just the probe.
> > RVU AF driver will use APIs exported by this driver
> > for various things like PF to physical interface mapping,
> > loopback mode, interface stats etc. Hence marged both
> > drivers into a single module.
> >
> > Signed-off-by: Sunil Goutham <sgoutham@marvell.com>
> > ---
> > drivers/net/ethernet/marvell/octeontx2/af/Makefile | 2 +-
> > drivers/net/ethernet/marvell/octeontx2/af/cgx.c | 100 +++++++++++++++++++++
> > drivers/net/ethernet/marvell/octeontx2/af/cgx.h | 22 +++++
> > drivers/net/ethernet/marvell/octeontx2/af/rvu.c | 14 ++-
> > 4 files changed, 136 insertions(+), 2 deletions(-)
> > create mode 100644 drivers/net/ethernet/marvell/octeontx2/af/cgx.c
> > create mode 100644 drivers/net/ethernet/marvell/octeontx2/af/cgx.h
> >
> > diff --git a/drivers/net/ethernet/marvell/octeontx2/af/Makefile b/drivers/net/ethernet/marvell/octeontx2/af/Makefile
> > index ac17cb9..8646421 100644
> > --- a/drivers/net/ethernet/marvell/octeontx2/af/Makefile
> > +++ b/drivers/net/ethernet/marvell/octeontx2/af/Makefile
> > @@ -7,4 +7,4 @@ obj-$(CONFIG_OCTEONTX2_MBOX) += octeontx2_mbox.o
> > obj-$(CONFIG_OCTEONTX2_AF) += octeontx2_af.o
> >
> > octeontx2_mbox-y := mbox.o
> > -octeontx2_af-y := rvu.o
> > +octeontx2_af-y := cgx.o rvu.o
> > diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
> > new file mode 100644
> > index 0000000..cfd80d2
> > --- /dev/null
> > +++ b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
>
> > +MODULE_AUTHOR("Marvell International Ltd.");
> > +MODULE_DESCRIPTION(DRV_STRING);
> > +MODULE_LICENSE("GPL v2");
> > +MODULE_DEVICE_TABLE(pci, cgx_id_table);
>
> Since you now link both pci_driver instances in one module, I think
> you need to remove the duplicate author/description/license tags.
> Multiple device tables are fine I think.
>
> Arnd
Thanks, will remove these and re-submit.
Sunil.
^ permalink raw reply
* Re: [PATCH bpf-next] bpf: permit CGROUP_DEVICE programs accessing helper bpf_get_current_cgroup_id()
From: Alexei Starovoitov @ 2018-09-28 9:11 UTC (permalink / raw)
To: Yonghong Song; +Cc: ast, daniel, netdev, guro, kernel-team
In-Reply-To: <20180927213730.1224816-1-yhs@fb.com>
On Thu, Sep 27, 2018 at 02:37:30PM -0700, Yonghong Song wrote:
> Currently, helper bpf_get_current_cgroup_id() is not permitted
> for CGROUP_DEVICE type of programs. If the helper is used
> in such cases, the verifier will log the following error:
>
> 0: (bf) r6 = r1
> 1: (69) r7 = *(u16 *)(r6 +0)
> 2: (85) call bpf_get_current_cgroup_id#80
> unknown func bpf_get_current_cgroup_id#80
>
> The bpf_get_current_cgroup_id() is useful for CGROUP_DEVICE
> type of programs in order to customize action based on cgroup id.
> This patch added such a support.
>
> Cc: Roman Gushchin <guro@fb.com>
> Signed-off-by: Yonghong Song <yhs@fb.com>
Acked-by: Alexei Starovoitov <ast@kernel.org>
^ permalink raw reply
* Re: [PATCH net 0/2] s390/qeth: fixes 2019-09-26
From: Julian Wiedmann @ 2018-09-28 9:05 UTC (permalink / raw)
To: David Miller
Cc: netdev, linux-s390, schwidefsky, heiko.carstens, raspl, ubraun
In-Reply-To: <20180926.103948.934917523873266650.davem@davemloft.net>
On 26.09.2018 19:39, David Miller wrote:
> From: Julian Wiedmann <jwi@linux.ibm.com>
> Date: Wed, 26 Sep 2018 18:07:08 +0200
>
>> please apply two qeth patches for -net. The first is a trivial cleanup
>> required for patch #2 by Jean, which fixes a potential endless loop.
>
> Series applied, thank you.
>
Odd, doesn't look like they arrived in -net though. Forgot to push them out?
Thanks.
^ permalink raw reply
* Re: [net-next 1/8] ice: use [sr]q.count when checking if queue is initialized
From: Sergei Shtylyov @ 2018-09-28 9:01 UTC (permalink / raw)
To: Jeff Kirsher, davem
Cc: Jacob Keller, netdev, nhorman, sassmann, jogreene,
Anirudh Venkataramanan
In-Reply-To: <20180927162201.30900-2-jeffrey.t.kirsher@intel.com>
Hello!
On 9/27/2018 7:21 PM, Jeff Kirsher wrote:
> From: Jacob Keller <jacob.e.keller@intel.com>
>
> When shutting down the controlqs, we check if they are initialized
> before we shut them down and destroy the lock. This is important, as it
> prevents attempts to access the lock of an already shutdown queue.
>
> Unfortunately, we checked rq.head and sq.head as the value to determine
> if the queue was initialized. This doesn't work, because head is not
> reset when the queue is shutdown. In some flows, the adminq will have
> already been shut down prior to calling ice_shutdown_all_ctrlqs. This
> can result in a crash due to attempting to access the already destroyed
> mutex.
>
> Fix this by using rq.count and sq.count instead. Indeed, ice_shutdown_sq
> and ice_shutdown_rq already indicate that this is the value we should be
> using to determine of the queue was initialized.
s/of/if/?
> Signed-off-by: Jacob Keller <jacob.e.keller@intel.com>
> Signed-off-by: Anirudh Venkataramanan <anirudh.venkataramanan@intel.com>
> Tested-by: Andrew Bowers <andrewx.bowers@intel.com>
> Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
[...]
MBR, Sergei
^ permalink raw reply
* Re: [PATCH v3 2/2] netfilter: nf_tables: add requirements for connsecmark support
From: Pablo Neira Ayuso @ 2018-09-28 9:01 UTC (permalink / raw)
To: Christian Göttsche
Cc: kadlec, fw, davem, netfilter-devel, coreteam, netdev,
linux-kernel, paul, sds, eparis, jmorris, serge, selinux,
linux-security-module
In-Reply-To: <20180923182616.11398-2-cgzones@googlemail.com>
On Sun, Sep 23, 2018 at 08:26:16PM +0200, Christian Göttsche wrote:
> Add ability to set the connection tracking secmark value.
>
> Add ability to set the meta secmark value.
Also applied, thanks.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox