* Re: [PATCH net-next] netfilter: xt_quota: fix the behavior of xt_quota module
From: Pablo Neira Ayuso @ 2018-10-02 7:59 UTC (permalink / raw)
To: Chenbo Feng
Cc: netdev, netfilter-devel, kernel-team, Lorenzo Colitti, maze,
Chenbo Feng
In-Reply-To: <1538443388-6881-3-git-send-email-chenbofeng.kernel@gmail.com>
Hi,
On Mon, Oct 01, 2018 at 06:23:08PM -0700, Chenbo Feng wrote:
> From: Chenbo Feng <fengc@google.com>
>
> A major flaw of the current xt_quota module is that quota in a specific
> rule gets reset every time there is a rule change in the same table. It
> makes the xt_quota module not very useful in a table in which iptables
> rules are changed at run time. This fix introduces a new counter that is
> visible to userspace as the remaining quota of the current rule. When
> userspace restores the rules in a table, it can restore the counter to
> the remaining quota instead of resetting it to the full quota.
A few questions, see below.
First one is, don't we need a new match revision for this new option?
> Signed-off-by: Chenbo Feng <fengc@google.com>
> Suggested-by: Maciej Żenczykowski <maze@google.com>
> Reviewed-by: Maciej Żenczykowski <maze@google.com>
> ---
> include/uapi/linux/netfilter/xt_quota.h | 8 +++--
> net/netfilter/xt_quota.c | 55 +++++++++++++--------------------
> 2 files changed, 27 insertions(+), 36 deletions(-)
>
> diff --git a/include/uapi/linux/netfilter/xt_quota.h b/include/uapi/linux/netfilter/xt_quota.h
> index f3ba5d9..d72fd52 100644
> --- a/include/uapi/linux/netfilter/xt_quota.h
> +++ b/include/uapi/linux/netfilter/xt_quota.h
> @@ -15,9 +15,11 @@ struct xt_quota_info {
> __u32 flags;
> __u32 pad;
> __aligned_u64 quota;
> -
> - /* Used internally by the kernel */
> - struct xt_quota_priv *master;
> +#ifdef __KERNEL__
> + atomic64_t counter;
> +#else
> + __aligned_u64 remain;
> +#endif
> };
>
> #endif /* _XT_QUOTA_H */
> diff --git a/net/netfilter/xt_quota.c b/net/netfilter/xt_quota.c
> index 10d61a6..6afa7f4 100644
> --- a/net/netfilter/xt_quota.c
> +++ b/net/netfilter/xt_quota.c
> @@ -11,11 +11,6 @@
> #include <linux/netfilter/xt_quota.h>
> #include <linux/module.h>
>
> -struct xt_quota_priv {
> - spinlock_t lock;
> - uint64_t quota;
> -};
> -
> MODULE_LICENSE("GPL");
> MODULE_AUTHOR("Sam Johnston <samj@samj.net>");
> MODULE_DESCRIPTION("Xtables: countdown quota match");
> @@ -26,54 +21,48 @@ static bool
> quota_mt(const struct sk_buff *skb, struct xt_action_param *par)
> {
> struct xt_quota_info *q = (void *)par->matchinfo;
> - struct xt_quota_priv *priv = q->master;
> + u64 current_count = atomic64_read(&q->counter);
> bool ret = q->flags & XT_QUOTA_INVERT;
> -
> - spin_lock_bh(&priv->lock);
> - if (priv->quota >= skb->len) {
> - priv->quota -= skb->len;
> - ret = !ret;
> - } else {
> - /* we do not allow even small packets from now on */
> - priv->quota = 0;
> - }
> - spin_unlock_bh(&priv->lock);
> -
> - return ret;
> + u64 old_count, new_count;
> +
> + do {
> + if (current_count == 1)
> + return ret;
So 1 means, don't keep updating, quota is depleted?
This current_count = 1 would be exposed to userspace too, right?
Hm, this semantics are going to be a bit awkwards to users I think, I
would prefer to expose this in a different way.
> + if (current_count <= skb->len) {
> + atomic64_set(&q->counter, 1);
> + return ret;
> + }
> + old_count = current_count;
> + new_count = current_count - skb->len;
> + current_count = atomic64_cmpxchg(&q->counter, old_count,
> + new_count);
> + } while (current_count != old_count);
Probably we simplify this via atomic64_add_return()?
I guess problem is userspace may get a current counter that is larger
than the quota, but we could handle this from userspace iptables to
print a value that equals the quota, ie. from userspace, before
printing:
if (consumed > quota)
printf("--consumed %PRIu64 ", quota);
else
printf("--consumed %PRIu64 ", consumed);
> + return !ret;
> }
Thanks !
^ permalink raw reply
* [PATCH v2 0/5] Introducing ixgbe AF_XDP ZC support
From: Björn Töpel @ 2018-10-02 8:00 UTC (permalink / raw)
To: jeffrey.t.kirsher, intel-wired-lan
Cc: Björn Töpel, magnus.karlsson, magnus.karlsson, ast,
daniel, netdev, brouer, u9012063, tuc, jakub.kicinski
From: Björn Töpel <bjorn.topel@intel.com>
Jeff: Please remove the v1 patches from your dev-queue!
This patch set introduces zero-copy AF_XDP support for Intel's ixgbe
driver.
The ixgbe zero-copy code is located in its own file ixgbe_xsk.[ch],
analogous to the i40e ZC support. Again, as in i40e, code paths have
been copied from the XDP path to the zero-copy path. Going forward we
will try to generalize more code between the AF_XDP ZC drivers, and
also reduce the heavy C&P.
We have run some benchmarks on a dual socket system with two Broadwell
E5 2660 @ 2.0 GHz with hyperthreading turned off. Each socket has 14
cores which gives a total of 28, but only two cores are used in these
experiments. One for TR/RX and one for the user space application. The
memory is DDR4 @ 2133 MT/s (1067 MHz) and the size of each DIMM is
8192MB and with 8 of those DIMMs in the system we have 64 GB of total
memory. The compiler used is GCC 7.3.0. The NIC is Intel
82599ES/X520-2 10Gbit/s using the ixgbe driver.
Below are the results in Mpps of the 82599ES/X520-2 NIC benchmark runs
for 64B and 1500B packets, generated by a commercial packet generator
HW blasting packets at full 10Gbit/s line rate. The results are with
retpoline and all other spectre and meltdown fixes.
AF_XDP performance 64B packets:
Benchmark XDP_DRV with zerocopy
rxdrop 14.7
txpush 14.6
l2fwd 11.1
AF_XDP performance 1500B packets:
Benchmark XDP_DRV with zerocopy
rxdrop 0.8
l2fwd 0.8
XDP performance on our system as a base line.
64B packets:
XDP stats CPU Mpps issue-pps
XDP-RX CPU 16 14.7 0
1500B packets:
XDP stats CPU Mpps issue-pps
XDP-RX CPU 16 0.8 0
The structure of the patch set is as follows:
Patch 1: Introduce Rx/Tx ring enable/disable functionality
Patch 2: Preparatory patche to ixgbe driver code for RX
Patch 3: ixgbe zero-copy support for RX
Patch 4: Preparatory patch to ixgbe driver code for TX
Patch 5: ixgbe zero-copy support for TX
Changes since v1:
* Removed redundant AF_XDP precondition checks, pointed out by
Jakub. Now, the preconditions are only checked at XDP enable time.
* Fixed a crash in the egress path, due to incorrect usage of
ixgbe_ring queue_index member. In v2 a ring_idx back reference is
introduced, and used in favor of queue_index. William reported the
crash, and helped me smoke out the issue. Kudos!
* In ixgbe_xsk_async_xmit, validate qid against num_xdp_queues,
instead of num_rx_queues.
Cheers!
Björn
Björn Töpel (5):
ixgbe: added Rx/Tx ring disable/enable functions
ixgbe: move common Rx functions to ixgbe_txrx_common.h
ixgbe: add AF_XDP zero-copy Rx support
ixgbe: move common Tx functions to ixgbe_txrx_common.h
ixgbe: add AF_XDP zero-copy Tx support
drivers/net/ethernet/intel/ixgbe/Makefile | 3 +-
drivers/net/ethernet/intel/ixgbe/ixgbe.h | 28 +-
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 17 +-
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 291 ++++++-
.../ethernet/intel/ixgbe/ixgbe_txrx_common.h | 50 ++
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 803 ++++++++++++++++++
6 files changed, 1146 insertions(+), 46 deletions(-)
create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
--
2.17.1
^ permalink raw reply
* [PATCH v2 1/5] ixgbe: added Rx/Tx ring disable/enable functions
From: Björn Töpel @ 2018-10-02 8:00 UTC (permalink / raw)
To: jeffrey.t.kirsher, intel-wired-lan
Cc: Björn Töpel, magnus.karlsson, magnus.karlsson, ast,
daniel, netdev, brouer, u9012063, tuc, jakub.kicinski
In-Reply-To: <20181002080034.11754-1-bjorn.topel@gmail.com>
From: Björn Töpel <bjorn.topel@intel.com>
Add functions for Rx/Tx ring enable/disable. Instead of resetting the
whole device, only the affected ring is disabled or enabled.
This plumbing is used in later commits, when zero-copy AF_XDP support
is introduced.
Signed-off-by: Björn Töpel <bjorn.topel@intel.com>
---
drivers/net/ethernet/intel/ixgbe/ixgbe.h | 1 +
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 158 ++++++++++++++++++
2 files changed, 159 insertions(+)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe.h b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
index 5c6fd42e90ed..265db172042a 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
@@ -271,6 +271,7 @@ enum ixgbe_ring_state_t {
__IXGBE_TX_DETECT_HANG,
__IXGBE_HANG_CHECK_ARMED,
__IXGBE_TX_XDP_RING,
+ __IXGBE_TX_DISABLED,
};
#define ring_uses_build_skb(ring) \
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 187b78f950b5..6ff886498882 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -8694,6 +8694,8 @@ static netdev_tx_t __ixgbe_xmit_frame(struct sk_buff *skb,
return NETDEV_TX_OK;
tx_ring = ring ? ring : adapter->tx_ring[skb->queue_mapping];
+ if (unlikely(test_bit(__IXGBE_TX_DISABLED, &tx_ring->state)))
+ return NETDEV_TX_BUSY;
return ixgbe_xmit_frame_ring(skb, adapter, tx_ring);
}
@@ -10240,6 +10242,9 @@ static int ixgbe_xdp_xmit(struct net_device *dev, int n,
if (unlikely(!ring))
return -ENXIO;
+ if (unlikely(test_bit(__IXGBE_TX_DISABLED, &ring->state)))
+ return -ENXIO;
+
for (i = 0; i < n; i++) {
struct xdp_frame *xdpf = frames[i];
int err;
@@ -10303,6 +10308,159 @@ static const struct net_device_ops ixgbe_netdev_ops = {
.ndo_xdp_xmit = ixgbe_xdp_xmit,
};
+static void ixgbe_disable_txr_hw(struct ixgbe_adapter *adapter,
+ struct ixgbe_ring *tx_ring)
+{
+ unsigned long wait_delay, delay_interval;
+ struct ixgbe_hw *hw = &adapter->hw;
+ u8 reg_idx = tx_ring->reg_idx;
+ int wait_loop;
+ u32 txdctl;
+
+ IXGBE_WRITE_REG(hw, IXGBE_TXDCTL(reg_idx), IXGBE_TXDCTL_SWFLSH);
+
+ /* delay mechanism from ixgbe_disable_tx */
+ delay_interval = ixgbe_get_completion_timeout(adapter) / 100;
+
+ wait_loop = IXGBE_MAX_RX_DESC_POLL;
+ wait_delay = delay_interval;
+
+ while (wait_loop--) {
+ usleep_range(wait_delay, wait_delay + 10);
+ wait_delay += delay_interval * 2;
+ txdctl = IXGBE_READ_REG(hw, IXGBE_TXDCTL(reg_idx));
+
+ if (!(txdctl & IXGBE_TXDCTL_ENABLE))
+ return;
+ }
+
+ e_err(drv, "TXDCTL.ENABLE not cleared within the polling period\n");
+}
+
+static void ixgbe_disable_txr(struct ixgbe_adapter *adapter,
+ struct ixgbe_ring *tx_ring)
+{
+ set_bit(__IXGBE_TX_DISABLED, &tx_ring->state);
+ ixgbe_disable_txr_hw(adapter, tx_ring);
+}
+
+static void ixgbe_disable_rxr_hw(struct ixgbe_adapter *adapter,
+ struct ixgbe_ring *rx_ring)
+{
+ unsigned long wait_delay, delay_interval;
+ struct ixgbe_hw *hw = &adapter->hw;
+ u8 reg_idx = rx_ring->reg_idx;
+ int wait_loop;
+ u32 rxdctl;
+
+ rxdctl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(reg_idx));
+ rxdctl &= ~IXGBE_RXDCTL_ENABLE;
+ rxdctl |= IXGBE_RXDCTL_SWFLSH;
+
+ /* write value back with RXDCTL.ENABLE bit cleared */
+ IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(reg_idx), rxdctl);
+
+ /* RXDCTL.EN may not change on 82598 if link is down, so skip it */
+ if (hw->mac.type == ixgbe_mac_82598EB &&
+ !(IXGBE_READ_REG(hw, IXGBE_LINKS) & IXGBE_LINKS_UP))
+ return;
+
+ /* delay mechanism from ixgbe_disable_rx */
+ delay_interval = ixgbe_get_completion_timeout(adapter) / 100;
+
+ wait_loop = IXGBE_MAX_RX_DESC_POLL;
+ wait_delay = delay_interval;
+
+ while (wait_loop--) {
+ usleep_range(wait_delay, wait_delay + 10);
+ wait_delay += delay_interval * 2;
+ rxdctl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(reg_idx));
+
+ if (!(rxdctl & IXGBE_RXDCTL_ENABLE))
+ return;
+ }
+
+ e_err(drv, "RXDCTL.ENABLE not cleared within the polling period\n");
+}
+
+static void ixgbe_reset_txr_stats(struct ixgbe_ring *tx_ring)
+{
+ memset(&tx_ring->stats, 0, sizeof(tx_ring->stats));
+ memset(&tx_ring->tx_stats, 0, sizeof(tx_ring->tx_stats));
+}
+
+static void ixgbe_reset_rxr_stats(struct ixgbe_ring *rx_ring)
+{
+ memset(&rx_ring->stats, 0, sizeof(rx_ring->stats));
+ memset(&rx_ring->rx_stats, 0, sizeof(rx_ring->rx_stats));
+}
+
+/**
+ * ixgbe_txrx_ring_disable - Disable Rx/Tx/XDP Tx rings
+ * @adapter: adapter structure
+ * @ring: ring index
+ *
+ * This function disables a certain Rx/Tx/XDP Tx ring. The function
+ * assumes that the netdev is running.
+ **/
+void ixgbe_txrx_ring_disable(struct ixgbe_adapter *adapter, int ring)
+{
+ struct ixgbe_ring *rx_ring, *tx_ring, *xdp_ring;
+
+ rx_ring = adapter->rx_ring[ring];
+ tx_ring = adapter->tx_ring[ring];
+ xdp_ring = adapter->xdp_ring[ring];
+
+ ixgbe_disable_txr(adapter, tx_ring);
+ if (xdp_ring)
+ ixgbe_disable_txr(adapter, xdp_ring);
+ ixgbe_disable_rxr_hw(adapter, rx_ring);
+
+ if (xdp_ring)
+ synchronize_sched();
+
+ /* Rx/Tx/XDP Tx share the same napi context. */
+ napi_disable(&rx_ring->q_vector->napi);
+
+ ixgbe_clean_tx_ring(tx_ring);
+ if (xdp_ring)
+ ixgbe_clean_tx_ring(xdp_ring);
+ ixgbe_clean_rx_ring(rx_ring);
+
+ ixgbe_reset_txr_stats(tx_ring);
+ if (xdp_ring)
+ ixgbe_reset_txr_stats(xdp_ring);
+ ixgbe_reset_rxr_stats(rx_ring);
+}
+
+/**
+ * ixgbe_txrx_ring_enable - Enable Rx/Tx/XDP Tx rings
+ * @adapter: adapter structure
+ * @ring: ring index
+ *
+ * This function enables a certain Rx/Tx/XDP Tx ring. The function
+ * assumes that the netdev is running.
+ **/
+void ixgbe_txrx_ring_enable(struct ixgbe_adapter *adapter, int ring)
+{
+ struct ixgbe_ring *rx_ring, *tx_ring, *xdp_ring;
+
+ rx_ring = adapter->rx_ring[ring];
+ tx_ring = adapter->tx_ring[ring];
+ xdp_ring = adapter->xdp_ring[ring];
+
+ /* Rx/Tx/XDP Tx share the same napi context. */
+ napi_enable(&rx_ring->q_vector->napi);
+
+ ixgbe_configure_tx_ring(adapter, tx_ring);
+ if (xdp_ring)
+ ixgbe_configure_tx_ring(adapter, xdp_ring);
+ ixgbe_configure_rx_ring(adapter, rx_ring);
+
+ clear_bit(__IXGBE_TX_DISABLED, &tx_ring->state);
+ clear_bit(__IXGBE_TX_DISABLED, &xdp_ring->state);
+}
+
/**
* ixgbe_enumerate_functions - Get the number of ports this device has
* @adapter: adapter structure
--
2.17.1
^ permalink raw reply related
* [PATCH v2 2/5] ixgbe: move common Rx functions to ixgbe_txrx_common.h
From: Björn Töpel @ 2018-10-02 8:00 UTC (permalink / raw)
To: jeffrey.t.kirsher, intel-wired-lan
Cc: Björn Töpel, magnus.karlsson, magnus.karlsson, ast,
daniel, netdev, brouer, u9012063, tuc, jakub.kicinski
In-Reply-To: <20181002080034.11754-1-bjorn.topel@gmail.com>
From: Björn Töpel <bjorn.topel@intel.com>
This patch prepares for the upcoming zero-copy Rx functionality, by
moving/changing linkage of common functions, used both by the regular
path and zero-copy path.
Signed-off-by: Björn Töpel <bjorn.topel@intel.com>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 29 +++++++------------
.../ethernet/intel/ixgbe/ixgbe_txrx_common.h | 26 +++++++++++++++++
2 files changed, 37 insertions(+), 18 deletions(-)
create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 6ff886498882..cc655c4e24fd 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -40,6 +40,7 @@
#include "ixgbe_dcb_82599.h"
#include "ixgbe_sriov.h"
#include "ixgbe_model.h"
+#include "ixgbe_txrx_common.h"
char ixgbe_driver_name[] = "ixgbe";
static const char ixgbe_driver_string[] =
@@ -1673,9 +1674,9 @@ static void ixgbe_update_rsc_stats(struct ixgbe_ring *rx_ring,
* order to populate the hash, checksum, VLAN, timestamp, protocol, and
* other fields within the skb.
**/
-static void ixgbe_process_skb_fields(struct ixgbe_ring *rx_ring,
- union ixgbe_adv_rx_desc *rx_desc,
- struct sk_buff *skb)
+void ixgbe_process_skb_fields(struct ixgbe_ring *rx_ring,
+ union ixgbe_adv_rx_desc *rx_desc,
+ struct sk_buff *skb)
{
struct net_device *dev = rx_ring->netdev;
u32 flags = rx_ring->q_vector->adapter->flags;
@@ -1708,8 +1709,8 @@ static void ixgbe_process_skb_fields(struct ixgbe_ring *rx_ring,
skb->protocol = eth_type_trans(skb, dev);
}
-static void ixgbe_rx_skb(struct ixgbe_q_vector *q_vector,
- struct sk_buff *skb)
+void ixgbe_rx_skb(struct ixgbe_q_vector *q_vector,
+ struct sk_buff *skb)
{
napi_gro_receive(&q_vector->napi, skb);
}
@@ -1868,9 +1869,9 @@ static void ixgbe_dma_sync_frag(struct ixgbe_ring *rx_ring,
*
* Returns true if an error was encountered and skb was freed.
**/
-static bool ixgbe_cleanup_headers(struct ixgbe_ring *rx_ring,
- union ixgbe_adv_rx_desc *rx_desc,
- struct sk_buff *skb)
+bool ixgbe_cleanup_headers(struct ixgbe_ring *rx_ring,
+ union ixgbe_adv_rx_desc *rx_desc,
+ struct sk_buff *skb)
{
struct net_device *netdev = rx_ring->netdev;
@@ -2186,14 +2187,6 @@ static struct sk_buff *ixgbe_build_skb(struct ixgbe_ring *rx_ring,
return skb;
}
-#define IXGBE_XDP_PASS 0
-#define IXGBE_XDP_CONSUMED BIT(0)
-#define IXGBE_XDP_TX BIT(1)
-#define IXGBE_XDP_REDIR BIT(2)
-
-static int ixgbe_xmit_xdp_ring(struct ixgbe_adapter *adapter,
- struct xdp_frame *xdpf);
-
static struct sk_buff *ixgbe_run_xdp(struct ixgbe_adapter *adapter,
struct ixgbe_ring *rx_ring,
struct xdp_buff *xdp)
@@ -8471,8 +8464,8 @@ static u16 ixgbe_select_queue(struct net_device *dev, struct sk_buff *skb,
}
#endif
-static int ixgbe_xmit_xdp_ring(struct ixgbe_adapter *adapter,
- struct xdp_frame *xdpf)
+int ixgbe_xmit_xdp_ring(struct ixgbe_adapter *adapter,
+ struct xdp_frame *xdpf)
{
struct ixgbe_ring *ring = adapter->xdp_ring[smp_processor_id()];
struct ixgbe_tx_buffer *tx_buffer;
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
new file mode 100644
index 000000000000..3780d315b991
--- /dev/null
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright(c) 2018 Intel Corporation. */
+
+#ifndef _IXGBE_TXRX_COMMON_H_
+#define _IXGBE_TXRX_COMMON_H_
+
+#define IXGBE_XDP_PASS 0
+#define IXGBE_XDP_CONSUMED BIT(0)
+#define IXGBE_XDP_TX BIT(1)
+#define IXGBE_XDP_REDIR BIT(2)
+
+int ixgbe_xmit_xdp_ring(struct ixgbe_adapter *adapter,
+ struct xdp_frame *xdpf);
+bool ixgbe_cleanup_headers(struct ixgbe_ring *rx_ring,
+ union ixgbe_adv_rx_desc *rx_desc,
+ struct sk_buff *skb);
+void ixgbe_process_skb_fields(struct ixgbe_ring *rx_ring,
+ union ixgbe_adv_rx_desc *rx_desc,
+ struct sk_buff *skb);
+void ixgbe_rx_skb(struct ixgbe_q_vector *q_vector,
+ struct sk_buff *skb);
+
+void ixgbe_txrx_ring_disable(struct ixgbe_adapter *adapter, int ring);
+void ixgbe_txrx_ring_enable(struct ixgbe_adapter *adapter, int ring);
+
+#endif /* #define _IXGBE_TXRX_COMMON_H_ */
--
2.17.1
^ permalink raw reply related
* [PATCH v2 3/5] ixgbe: add AF_XDP zero-copy Rx support
From: Björn Töpel @ 2018-10-02 8:00 UTC (permalink / raw)
To: jeffrey.t.kirsher, intel-wired-lan
Cc: Björn Töpel, magnus.karlsson, magnus.karlsson, ast,
daniel, netdev, brouer, u9012063, tuc, jakub.kicinski
In-Reply-To: <20181002080034.11754-1-bjorn.topel@gmail.com>
From: Björn Töpel <bjorn.topel@intel.com>
This patch adds zero-copy Rx support for AF_XDP sockets. Instead of
allocating buffers of type MEM_TYPE_PAGE_SHARED, the Rx frames are
allocated as MEM_TYPE_ZERO_COPY when AF_XDP is enabled for a certain
queue.
All AF_XDP specific functions are added to a new file, ixgbe_xsk.c.
Note that when AF_XDP zero-copy is enabled, the XDP action XDP_PASS
will allocate a new buffer and copy the zero-copy frame prior passing
it to the kernel stack.
Signed-off-by: Björn Töpel <bjorn.topel@intel.com>
---
drivers/net/ethernet/intel/ixgbe/Makefile | 3 +-
drivers/net/ethernet/intel/ixgbe/ixgbe.h | 27 +-
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 17 +-
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 78 ++-
.../ethernet/intel/ixgbe/ixgbe_txrx_common.h | 15 +
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 628 ++++++++++++++++++
6 files changed, 747 insertions(+), 21 deletions(-)
create mode 100644 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
diff --git a/drivers/net/ethernet/intel/ixgbe/Makefile b/drivers/net/ethernet/intel/ixgbe/Makefile
index 5414685189ce..ca6b0c458e4a 100644
--- a/drivers/net/ethernet/intel/ixgbe/Makefile
+++ b/drivers/net/ethernet/intel/ixgbe/Makefile
@@ -8,7 +8,8 @@ obj-$(CONFIG_IXGBE) += ixgbe.o
ixgbe-objs := ixgbe_main.o ixgbe_common.o ixgbe_ethtool.o \
ixgbe_82599.o ixgbe_82598.o ixgbe_phy.o ixgbe_sriov.o \
- ixgbe_mbx.o ixgbe_x540.o ixgbe_x550.o ixgbe_lib.o ixgbe_ptp.o
+ ixgbe_mbx.o ixgbe_x540.o ixgbe_x550.o ixgbe_lib.o ixgbe_ptp.o \
+ ixgbe_xsk.o
ixgbe-$(CONFIG_IXGBE_DCB) += ixgbe_dcb.o ixgbe_dcb_82598.o \
ixgbe_dcb_82599.o ixgbe_dcb_nl.o
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe.h b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
index 265db172042a..7a7679e7be84 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
@@ -228,13 +228,17 @@ struct ixgbe_tx_buffer {
struct ixgbe_rx_buffer {
struct sk_buff *skb;
dma_addr_t dma;
- struct page *page;
-#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536)
- __u32 page_offset;
-#else
- __u16 page_offset;
-#endif
- __u16 pagecnt_bias;
+ union {
+ struct {
+ struct page *page;
+ __u32 page_offset;
+ __u16 pagecnt_bias;
+ };
+ struct {
+ void *addr;
+ u64 handle;
+ };
+ };
};
struct ixgbe_queue_stats {
@@ -348,6 +352,10 @@ struct ixgbe_ring {
struct ixgbe_rx_queue_stats rx_stats;
};
struct xdp_rxq_info xdp_rxq;
+ struct xdp_umem *xsk_umem;
+ struct zero_copy_allocator zca; /* ZC allocator anchor */
+ u16 ring_idx; /* {rx,tx,xdp}_ring back reference idx */
+ u16 rx_buf_len;
} ____cacheline_internodealigned_in_smp;
enum ixgbe_ring_f_enum {
@@ -765,6 +773,11 @@ struct ixgbe_adapter {
#ifdef CONFIG_XFRM_OFFLOAD
struct ixgbe_ipsec *ipsec;
#endif /* CONFIG_XFRM_OFFLOAD */
+
+ /* AF_XDP zero-copy */
+ struct xdp_umem **xsk_umems;
+ u16 num_xsk_umems_used;
+ u16 num_xsk_umems;
};
static inline u8 ixgbe_max_rss_indices(struct ixgbe_adapter *adapter)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
index d361f570ca37..62e6499e4146 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
@@ -1055,7 +1055,7 @@ static int ixgbe_alloc_q_vectors(struct ixgbe_adapter *adapter)
int txr_remaining = adapter->num_tx_queues;
int xdp_remaining = adapter->num_xdp_queues;
int rxr_idx = 0, txr_idx = 0, xdp_idx = 0, v_idx = 0;
- int err;
+ int err, i;
/* only one q_vector if MSI-X is disabled. */
if (!(adapter->flags & IXGBE_FLAG_MSIX_ENABLED))
@@ -1097,6 +1097,21 @@ static int ixgbe_alloc_q_vectors(struct ixgbe_adapter *adapter)
xdp_idx += xqpv;
}
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ if (adapter->rx_ring[i])
+ adapter->rx_ring[i]->ring_idx = i;
+ }
+
+ for (i = 0; i < adapter->num_tx_queues; i++) {
+ if (adapter->tx_ring[i])
+ adapter->tx_ring[i]->ring_idx = i;
+ }
+
+ for (i = 0; i < adapter->num_xdp_queues; i++) {
+ if (adapter->xdp_ring[i])
+ adapter->xdp_ring[i]->ring_idx = i;
+ }
+
return 0;
err_out:
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index cc655c4e24fd..547092b8fe54 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -34,6 +34,7 @@
#include <net/tc_act/tc_mirred.h>
#include <net/vxlan.h>
#include <net/mpls.h>
+#include <net/xdp_sock.h>
#include "ixgbe.h"
#include "ixgbe_common.h"
@@ -3176,7 +3177,10 @@ int ixgbe_poll(struct napi_struct *napi, int budget)
per_ring_budget = budget;
ixgbe_for_each_ring(ring, q_vector->rx) {
- int cleaned = ixgbe_clean_rx_irq(q_vector, ring,
+ int cleaned = ring->xsk_umem ?
+ ixgbe_clean_rx_irq_zc(q_vector, ring,
+ per_ring_budget) :
+ ixgbe_clean_rx_irq(q_vector, ring,
per_ring_budget);
work_done += cleaned;
@@ -3706,10 +3710,27 @@ static void ixgbe_configure_srrctl(struct ixgbe_adapter *adapter,
srrctl = IXGBE_RX_HDR_SIZE << IXGBE_SRRCTL_BSIZEHDRSIZE_SHIFT;
/* configure the packet buffer length */
- if (test_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state))
+ if (rx_ring->xsk_umem) {
+ u32 xsk_buf_len = rx_ring->xsk_umem->chunk_size_nohr -
+ XDP_PACKET_HEADROOM;
+
+ /* If the MAC support setting RXDCTL.RLPML, the
+ * SRRCTL[n].BSIZEPKT is set to PAGE_SIZE and
+ * RXDCTL.RLPML is set to the actual UMEM buffer
+ * size. If not, then we are stuck with a 1k buffer
+ * size resolution. In this case frames larger than
+ * the UMEM buffer size viewed in a 1k resolution will
+ * be dropped.
+ */
+ if (hw->mac.type != ixgbe_mac_82599EB)
+ srrctl |= PAGE_SIZE >> IXGBE_SRRCTL_BSIZEPKT_SHIFT;
+ else
+ srrctl |= xsk_buf_len >> IXGBE_SRRCTL_BSIZEPKT_SHIFT;
+ } else if (test_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state)) {
srrctl |= IXGBE_RXBUFFER_3K >> IXGBE_SRRCTL_BSIZEPKT_SHIFT;
- else
+ } else {
srrctl |= IXGBE_RXBUFFER_2K >> IXGBE_SRRCTL_BSIZEPKT_SHIFT;
+ }
/* configure descriptor type */
srrctl |= IXGBE_SRRCTL_DESCTYPE_ADV_ONEBUF;
@@ -4032,6 +4053,19 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter,
u32 rxdctl;
u8 reg_idx = ring->reg_idx;
+ xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq);
+ ring->xsk_umem = ixgbe_xsk_umem(adapter, ring);
+ if (ring->xsk_umem) {
+ ring->zca.free = ixgbe_zca_free;
+ WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
+ MEM_TYPE_ZERO_COPY,
+ &ring->zca));
+
+ } else {
+ WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
+ MEM_TYPE_PAGE_SHARED, NULL));
+ }
+
/* disable queue to avoid use of these values while updating state */
rxdctl = IXGBE_READ_REG(hw, IXGBE_RXDCTL(reg_idx));
rxdctl &= ~IXGBE_RXDCTL_ENABLE;
@@ -4081,6 +4115,17 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter,
#endif
}
+ if (ring->xsk_umem && hw->mac.type != ixgbe_mac_82599EB) {
+ u32 xsk_buf_len = ring->xsk_umem->chunk_size_nohr -
+ XDP_PACKET_HEADROOM;
+
+ rxdctl &= ~(IXGBE_RXDCTL_RLPMLMASK |
+ IXGBE_RXDCTL_RLPML_EN);
+ rxdctl |= xsk_buf_len | IXGBE_RXDCTL_RLPML_EN;
+
+ ring->rx_buf_len = xsk_buf_len;
+ }
+
/* initialize rx_buffer_info */
memset(ring->rx_buffer_info, 0,
sizeof(struct ixgbe_rx_buffer) * ring->count);
@@ -4094,7 +4139,10 @@ void ixgbe_configure_rx_ring(struct ixgbe_adapter *adapter,
IXGBE_WRITE_REG(hw, IXGBE_RXDCTL(reg_idx), rxdctl);
ixgbe_rx_desc_queue_enable(adapter, ring);
- ixgbe_alloc_rx_buffers(ring, ixgbe_desc_unused(ring));
+ if (ring->xsk_umem)
+ ixgbe_alloc_rx_buffers_zc(ring, ixgbe_desc_unused(ring));
+ else
+ ixgbe_alloc_rx_buffers(ring, ixgbe_desc_unused(ring));
}
static void ixgbe_setup_psrtype(struct ixgbe_adapter *adapter)
@@ -5208,6 +5256,11 @@ static void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring)
u16 i = rx_ring->next_to_clean;
struct ixgbe_rx_buffer *rx_buffer = &rx_ring->rx_buffer_info[i];
+ if (rx_ring->xsk_umem) {
+ ixgbe_xsk_clean_rx_ring(rx_ring);
+ goto skip_free;
+ }
+
/* Free all the Rx ring sk_buffs */
while (i != rx_ring->next_to_alloc) {
if (rx_buffer->skb) {
@@ -5246,6 +5299,7 @@ static void ixgbe_clean_rx_ring(struct ixgbe_ring *rx_ring)
}
}
+skip_free:
rx_ring->next_to_alloc = 0;
rx_ring->next_to_clean = 0;
rx_ring->next_to_use = 0;
@@ -6441,7 +6495,7 @@ int ixgbe_setup_rx_resources(struct ixgbe_adapter *adapter,
struct device *dev = rx_ring->dev;
int orig_node = dev_to_node(dev);
int ring_node = -1;
- int size, err;
+ int size;
size = sizeof(struct ixgbe_rx_buffer) * rx_ring->count;
@@ -6478,13 +6532,6 @@ int ixgbe_setup_rx_resources(struct ixgbe_adapter *adapter,
rx_ring->queue_index) < 0)
goto err;
- err = xdp_rxq_info_reg_mem_model(&rx_ring->xdp_rxq,
- MEM_TYPE_PAGE_SHARED, NULL);
- if (err) {
- xdp_rxq_info_unreg(&rx_ring->xdp_rxq);
- goto err;
- }
-
rx_ring->xdp_prog = adapter->xdp_prog;
return 0;
@@ -10200,6 +10247,13 @@ static int ixgbe_xdp(struct net_device *dev, struct netdev_bpf *xdp)
xdp->prog_id = adapter->xdp_prog ?
adapter->xdp_prog->aux->id : 0;
return 0;
+ case XDP_QUERY_XSK_UMEM:
+ return ixgbe_xsk_umem_query(adapter, &xdp->xsk.umem,
+ xdp->xsk.queue_id);
+ case XDP_SETUP_XSK_UMEM:
+ return ixgbe_xsk_umem_setup(adapter, xdp->xsk.umem,
+ xdp->xsk.queue_id);
+
default:
return -EINVAL;
}
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
index 3780d315b991..cf219f4e009d 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
@@ -23,4 +23,19 @@ void ixgbe_rx_skb(struct ixgbe_q_vector *q_vector,
void ixgbe_txrx_ring_disable(struct ixgbe_adapter *adapter, int ring);
void ixgbe_txrx_ring_enable(struct ixgbe_adapter *adapter, int ring);
+struct xdp_umem *ixgbe_xsk_umem(struct ixgbe_adapter *adapter,
+ struct ixgbe_ring *ring);
+int ixgbe_xsk_umem_query(struct ixgbe_adapter *adapter, struct xdp_umem **umem,
+ u16 qid);
+int ixgbe_xsk_umem_setup(struct ixgbe_adapter *adapter, struct xdp_umem *umem,
+ u16 qid);
+
+void ixgbe_zca_free(struct zero_copy_allocator *alloc, unsigned long handle);
+
+void ixgbe_alloc_rx_buffers_zc(struct ixgbe_ring *rx_ring, u16 cleaned_count);
+int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
+ struct ixgbe_ring *rx_ring,
+ const int budget);
+void ixgbe_xsk_clean_rx_ring(struct ixgbe_ring *rx_ring);
+
#endif /* #define _IXGBE_TXRX_COMMON_H_ */
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
new file mode 100644
index 000000000000..61259036ff4b
--- /dev/null
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
@@ -0,0 +1,628 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2018 Intel Corporation. */
+
+#include <linux/bpf_trace.h>
+#include <net/xdp_sock.h>
+#include <net/xdp.h>
+
+#include "ixgbe.h"
+#include "ixgbe_txrx_common.h"
+
+struct xdp_umem *ixgbe_xsk_umem(struct ixgbe_adapter *adapter,
+ struct ixgbe_ring *ring)
+{
+ bool xdp_on = READ_ONCE(adapter->xdp_prog);
+ int qid = ring->ring_idx;
+
+ if (!adapter->xsk_umems || !adapter->xsk_umems[qid] ||
+ qid >= adapter->num_xsk_umems || !xdp_on)
+ return NULL;
+
+ return adapter->xsk_umems[qid];
+}
+
+static int ixgbe_alloc_xsk_umems(struct ixgbe_adapter *adapter)
+{
+ if (adapter->xsk_umems)
+ return 0;
+
+ adapter->num_xsk_umems_used = 0;
+ adapter->num_xsk_umems = adapter->num_rx_queues;
+ adapter->xsk_umems = kcalloc(adapter->num_xsk_umems,
+ sizeof(*adapter->xsk_umems),
+ GFP_KERNEL);
+ if (!adapter->xsk_umems) {
+ adapter->num_xsk_umems = 0;
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static int ixgbe_add_xsk_umem(struct ixgbe_adapter *adapter,
+ struct xdp_umem *umem,
+ u16 qid)
+{
+ int err;
+
+ err = ixgbe_alloc_xsk_umems(adapter);
+ if (err)
+ return err;
+
+ adapter->xsk_umems[qid] = umem;
+ adapter->num_xsk_umems_used++;
+
+ return 0;
+}
+
+static void ixgbe_remove_xsk_umem(struct ixgbe_adapter *adapter, u16 qid)
+{
+ adapter->xsk_umems[qid] = NULL;
+ adapter->num_xsk_umems_used--;
+
+ if (adapter->num_xsk_umems == 0) {
+ kfree(adapter->xsk_umems);
+ adapter->xsk_umems = NULL;
+ adapter->num_xsk_umems = 0;
+ }
+}
+
+static int ixgbe_xsk_umem_dma_map(struct ixgbe_adapter *adapter,
+ struct xdp_umem *umem)
+{
+ struct device *dev = &adapter->pdev->dev;
+ unsigned int i, j;
+ dma_addr_t dma;
+
+ for (i = 0; i < umem->npgs; i++) {
+ dma = dma_map_page_attrs(dev, umem->pgs[i], 0, PAGE_SIZE,
+ DMA_BIDIRECTIONAL, IXGBE_RX_DMA_ATTR);
+ if (dma_mapping_error(dev, dma))
+ goto out_unmap;
+
+ umem->pages[i].dma = dma;
+ }
+
+ return 0;
+
+out_unmap:
+ for (j = 0; j < i; j++) {
+ dma_unmap_page_attrs(dev, umem->pages[i].dma, PAGE_SIZE,
+ DMA_BIDIRECTIONAL, IXGBE_RX_DMA_ATTR);
+ umem->pages[i].dma = 0;
+ }
+
+ return -1;
+}
+
+static void ixgbe_xsk_umem_dma_unmap(struct ixgbe_adapter *adapter,
+ struct xdp_umem *umem)
+{
+ struct device *dev = &adapter->pdev->dev;
+ unsigned int i;
+
+ for (i = 0; i < umem->npgs; i++) {
+ dma_unmap_page_attrs(dev, umem->pages[i].dma, PAGE_SIZE,
+ DMA_BIDIRECTIONAL, IXGBE_RX_DMA_ATTR);
+
+ umem->pages[i].dma = 0;
+ }
+}
+
+static int ixgbe_xsk_umem_enable(struct ixgbe_adapter *adapter,
+ struct xdp_umem *umem,
+ u16 qid)
+{
+ struct xdp_umem_fq_reuse *reuseq;
+ bool if_running;
+ int err;
+
+ if (qid >= adapter->num_rx_queues)
+ return -EINVAL;
+
+ if (adapter->xsk_umems) {
+ if (qid >= adapter->num_xsk_umems)
+ return -EINVAL;
+ if (adapter->xsk_umems[qid])
+ return -EBUSY;
+ }
+
+ reuseq = xsk_reuseq_prepare(adapter->rx_ring[0]->count);
+ if (!reuseq)
+ return -ENOMEM;
+
+ xsk_reuseq_free(xsk_reuseq_swap(umem, reuseq));
+
+ err = ixgbe_xsk_umem_dma_map(adapter, umem);
+ if (err)
+ return err;
+
+ if_running = netif_running(adapter->netdev) &&
+ READ_ONCE(adapter->xdp_prog);
+
+ if (if_running)
+ ixgbe_txrx_ring_disable(adapter, qid);
+
+ err = ixgbe_add_xsk_umem(adapter, umem, qid);
+
+ if (if_running)
+ ixgbe_txrx_ring_enable(adapter, qid);
+
+ return err;
+}
+
+static int ixgbe_xsk_umem_disable(struct ixgbe_adapter *adapter, u16 qid)
+{
+ bool if_running;
+
+ if (!adapter->xsk_umems || qid >= adapter->num_xsk_umems ||
+ !adapter->xsk_umems[qid])
+ return -EINVAL;
+
+ if_running = netif_running(adapter->netdev) &&
+ READ_ONCE(adapter->xdp_prog);
+
+ if (if_running)
+ ixgbe_txrx_ring_disable(adapter, qid);
+
+ ixgbe_xsk_umem_dma_unmap(adapter, adapter->xsk_umems[qid]);
+ ixgbe_remove_xsk_umem(adapter, qid);
+
+ if (if_running)
+ ixgbe_txrx_ring_enable(adapter, qid);
+
+ return 0;
+}
+
+int ixgbe_xsk_umem_query(struct ixgbe_adapter *adapter, struct xdp_umem **umem,
+ u16 qid)
+{
+ if (qid >= adapter->num_rx_queues)
+ return -EINVAL;
+
+ if (adapter->xsk_umems) {
+ if (qid >= adapter->num_xsk_umems)
+ return -EINVAL;
+ *umem = adapter->xsk_umems[qid];
+ return 0;
+ }
+
+ *umem = NULL;
+ return 0;
+}
+
+int ixgbe_xsk_umem_setup(struct ixgbe_adapter *adapter, struct xdp_umem *umem,
+ u16 qid)
+{
+ return umem ? ixgbe_xsk_umem_enable(adapter, umem, qid) :
+ ixgbe_xsk_umem_disable(adapter, qid);
+}
+
+static int ixgbe_run_xdp_zc(struct ixgbe_adapter *adapter,
+ struct ixgbe_ring *rx_ring,
+ struct xdp_buff *xdp)
+{
+ int err, result = IXGBE_XDP_PASS;
+ struct bpf_prog *xdp_prog;
+ struct xdp_frame *xdpf;
+ u32 act;
+
+ rcu_read_lock();
+ xdp_prog = READ_ONCE(rx_ring->xdp_prog);
+ act = bpf_prog_run_xdp(xdp_prog, xdp);
+ xdp->handle += xdp->data - xdp->data_hard_start;
+ switch (act) {
+ case XDP_PASS:
+ break;
+ case XDP_TX:
+ xdpf = convert_to_xdp_frame(xdp);
+ if (unlikely(!xdpf)) {
+ result = IXGBE_XDP_CONSUMED;
+ break;
+ }
+ result = ixgbe_xmit_xdp_ring(adapter, xdpf);
+ break;
+ case XDP_REDIRECT:
+ err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
+ result = !err ? IXGBE_XDP_REDIR : IXGBE_XDP_CONSUMED;
+ break;
+ default:
+ bpf_warn_invalid_xdp_action(act);
+ /* fallthrough */
+ case XDP_ABORTED:
+ trace_xdp_exception(rx_ring->netdev, xdp_prog, act);
+ /* fallthrough -- handle aborts by dropping packet */
+ case XDP_DROP:
+ result = IXGBE_XDP_CONSUMED;
+ break;
+ }
+ rcu_read_unlock();
+ return result;
+}
+
+static struct ixgbe_rx_buffer *ixgbe_get_rx_buffer_zc(
+ struct ixgbe_ring *rx_ring,
+ unsigned int size)
+{
+ struct ixgbe_rx_buffer *bi;
+
+ bi = &rx_ring->rx_buffer_info[rx_ring->next_to_clean];
+
+ /* we are reusing so sync this buffer for CPU use */
+ dma_sync_single_range_for_cpu(rx_ring->dev,
+ bi->dma, 0,
+ size,
+ DMA_BIDIRECTIONAL);
+
+ return bi;
+}
+
+static void ixgbe_reuse_rx_buffer_zc(struct ixgbe_ring *rx_ring,
+ struct ixgbe_rx_buffer *obi)
+{
+ unsigned long mask = (unsigned long)rx_ring->xsk_umem->chunk_mask;
+ u64 hr = rx_ring->xsk_umem->headroom + XDP_PACKET_HEADROOM;
+ u16 nta = rx_ring->next_to_alloc;
+ struct ixgbe_rx_buffer *nbi;
+
+ nbi = &rx_ring->rx_buffer_info[rx_ring->next_to_alloc];
+ /* update, and store next to alloc */
+ nta++;
+ rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
+
+ /* transfer page from old buffer to new buffer */
+ nbi->dma = obi->dma & mask;
+ nbi->dma += hr;
+
+ nbi->addr = (void *)((unsigned long)obi->addr & mask);
+ nbi->addr += hr;
+
+ nbi->handle = obi->handle & mask;
+ nbi->handle += rx_ring->xsk_umem->headroom;
+
+ obi->addr = NULL;
+ obi->skb = NULL;
+}
+
+void ixgbe_zca_free(struct zero_copy_allocator *alloc, unsigned long handle)
+{
+ struct ixgbe_rx_buffer *bi;
+ struct ixgbe_ring *rx_ring;
+ u64 hr, mask;
+ u16 nta;
+
+ rx_ring = container_of(alloc, struct ixgbe_ring, zca);
+ hr = rx_ring->xsk_umem->headroom + XDP_PACKET_HEADROOM;
+ mask = rx_ring->xsk_umem->chunk_mask;
+
+ nta = rx_ring->next_to_alloc;
+ bi = rx_ring->rx_buffer_info;
+
+ nta++;
+ rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
+
+ handle &= mask;
+
+ bi->dma = xdp_umem_get_dma(rx_ring->xsk_umem, handle);
+ bi->dma += hr;
+
+ bi->addr = xdp_umem_get_data(rx_ring->xsk_umem, handle);
+ bi->addr += hr;
+
+ bi->handle = (u64)handle + rx_ring->xsk_umem->headroom;
+}
+
+static bool ixgbe_alloc_buffer_zc(struct ixgbe_ring *rx_ring,
+ struct ixgbe_rx_buffer *bi)
+{
+ struct xdp_umem *umem = rx_ring->xsk_umem;
+ void *addr = bi->addr;
+ u64 handle, hr;
+
+ if (addr)
+ return true;
+
+ if (!xsk_umem_peek_addr(umem, &handle)) {
+ rx_ring->rx_stats.alloc_rx_page_failed++;
+ return false;
+ }
+
+ hr = umem->headroom + XDP_PACKET_HEADROOM;
+
+ bi->dma = xdp_umem_get_dma(umem, handle);
+ bi->dma += hr;
+
+ bi->addr = xdp_umem_get_data(umem, handle);
+ bi->addr += hr;
+
+ bi->handle = handle + umem->headroom;
+
+ xsk_umem_discard_addr(umem);
+ return true;
+}
+
+static bool ixgbe_alloc_buffer_slow_zc(struct ixgbe_ring *rx_ring,
+ struct ixgbe_rx_buffer *bi)
+{
+ struct xdp_umem *umem = rx_ring->xsk_umem;
+ u64 handle, hr;
+
+ if (!xsk_umem_peek_addr_rq(umem, &handle)) {
+ rx_ring->rx_stats.alloc_rx_page_failed++;
+ return false;
+ }
+
+ handle &= rx_ring->xsk_umem->chunk_mask;
+
+ hr = umem->headroom + XDP_PACKET_HEADROOM;
+
+ bi->dma = xdp_umem_get_dma(umem, handle);
+ bi->dma += hr;
+
+ bi->addr = xdp_umem_get_data(umem, handle);
+ bi->addr += hr;
+
+ bi->handle = handle + umem->headroom;
+
+ xsk_umem_discard_addr_rq(umem);
+ return true;
+}
+
+static __always_inline bool __ixgbe_alloc_rx_buffers_zc(
+ struct ixgbe_ring *rx_ring,
+ u16 cleaned_count,
+ bool alloc(struct ixgbe_ring *rx_ring,
+ struct ixgbe_rx_buffer *bi))
+{
+ union ixgbe_adv_rx_desc *rx_desc;
+ struct ixgbe_rx_buffer *bi;
+ u16 i = rx_ring->next_to_use;
+ bool ok = true;
+
+ /* nothing to do */
+ if (!cleaned_count)
+ return true;
+
+ rx_desc = IXGBE_RX_DESC(rx_ring, i);
+ bi = &rx_ring->rx_buffer_info[i];
+ i -= rx_ring->count;
+
+ do {
+ if (!alloc(rx_ring, bi)) {
+ ok = false;
+ break;
+ }
+
+ /* sync the buffer for use by the device */
+ dma_sync_single_range_for_device(rx_ring->dev, bi->dma,
+ bi->page_offset,
+ rx_ring->rx_buf_len,
+ DMA_BIDIRECTIONAL);
+
+ /* Refresh the desc even if buffer_addrs didn't change
+ * because each write-back erases this info.
+ */
+ rx_desc->read.pkt_addr = cpu_to_le64(bi->dma);
+
+ rx_desc++;
+ bi++;
+ i++;
+ if (unlikely(!i)) {
+ rx_desc = IXGBE_RX_DESC(rx_ring, 0);
+ bi = rx_ring->rx_buffer_info;
+ i -= rx_ring->count;
+ }
+
+ /* clear the length for the next_to_use descriptor */
+ rx_desc->wb.upper.length = 0;
+
+ cleaned_count--;
+ } while (cleaned_count);
+
+ i += rx_ring->count;
+
+ if (rx_ring->next_to_use != i) {
+ rx_ring->next_to_use = i;
+
+ /* update next to alloc since we have filled the ring */
+ rx_ring->next_to_alloc = i;
+
+ /* Force memory writes to complete before letting h/w
+ * know there are new descriptors to fetch. (Only
+ * applicable for weak-ordered memory model archs,
+ * such as IA-64).
+ */
+ wmb();
+ writel(i, rx_ring->tail);
+ }
+
+ return ok;
+}
+
+void ixgbe_alloc_rx_buffers_zc(struct ixgbe_ring *rx_ring, u16 count)
+{
+ __ixgbe_alloc_rx_buffers_zc(rx_ring, count,
+ ixgbe_alloc_buffer_slow_zc);
+}
+
+static bool ixgbe_alloc_rx_buffers_fast_zc(struct ixgbe_ring *rx_ring,
+ u16 count)
+{
+ return __ixgbe_alloc_rx_buffers_zc(rx_ring, count,
+ ixgbe_alloc_buffer_zc);
+}
+
+static struct sk_buff *ixgbe_construct_skb_zc(struct ixgbe_ring *rx_ring,
+ struct ixgbe_rx_buffer *bi,
+ struct xdp_buff *xdp)
+{
+ unsigned int metasize = xdp->data - xdp->data_meta;
+ unsigned int datasize = xdp->data_end - xdp->data;
+ struct sk_buff *skb;
+
+ /* allocate a skb to store the frags */
+ skb = __napi_alloc_skb(&rx_ring->q_vector->napi,
+ xdp->data_end - xdp->data_hard_start,
+ GFP_ATOMIC | __GFP_NOWARN);
+ if (unlikely(!skb))
+ return NULL;
+
+ skb_reserve(skb, xdp->data - xdp->data_hard_start);
+ memcpy(__skb_put(skb, datasize), xdp->data, datasize);
+ if (metasize)
+ skb_metadata_set(skb, metasize);
+
+ ixgbe_reuse_rx_buffer_zc(rx_ring, bi);
+ return skb;
+}
+
+static void ixgbe_inc_ntc(struct ixgbe_ring *rx_ring)
+{
+ u32 ntc = rx_ring->next_to_clean + 1;
+
+ ntc = (ntc < rx_ring->count) ? ntc : 0;
+ rx_ring->next_to_clean = ntc;
+ prefetch(IXGBE_RX_DESC(rx_ring, ntc));
+}
+
+int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
+ struct ixgbe_ring *rx_ring,
+ const int budget)
+{
+ unsigned int total_rx_bytes = 0, total_rx_packets = 0;
+ struct ixgbe_adapter *adapter = q_vector->adapter;
+ u16 cleaned_count = ixgbe_desc_unused(rx_ring);
+ unsigned int xdp_res, xdp_xmit = 0;
+ bool failure = false;
+ struct sk_buff *skb;
+ struct xdp_buff xdp;
+
+ xdp.rxq = &rx_ring->xdp_rxq;
+
+ while (likely(total_rx_packets < budget)) {
+ union ixgbe_adv_rx_desc *rx_desc;
+ struct ixgbe_rx_buffer *bi;
+ unsigned int size;
+
+ /* return some buffers to hardware, one at a time is too slow */
+ if (cleaned_count >= IXGBE_RX_BUFFER_WRITE) {
+ failure = failure ||
+ !ixgbe_alloc_rx_buffers_fast_zc(
+ rx_ring,
+ cleaned_count);
+ cleaned_count = 0;
+ }
+
+ rx_desc = IXGBE_RX_DESC(rx_ring, rx_ring->next_to_clean);
+ size = le16_to_cpu(rx_desc->wb.upper.length);
+ if (!size)
+ break;
+
+ /* This memory barrier is needed to keep us from reading
+ * any other fields out of the rx_desc until we know the
+ * descriptor has been written back
+ */
+ dma_rmb();
+
+ bi = ixgbe_get_rx_buffer_zc(rx_ring, size);
+
+ if (unlikely(!ixgbe_test_staterr(rx_desc,
+ IXGBE_RXD_STAT_EOP))) {
+ struct ixgbe_rx_buffer *next_bi;
+
+ ixgbe_reuse_rx_buffer_zc(rx_ring, bi);
+ ixgbe_inc_ntc(rx_ring);
+ next_bi = &rx_ring->rx_buffer_info[
+ rx_ring->next_to_clean];
+ next_bi->skb = ERR_PTR(-EINVAL);
+ continue;
+ }
+
+ if (unlikely(bi->skb)) {
+ ixgbe_reuse_rx_buffer_zc(rx_ring, bi);
+ ixgbe_inc_ntc(rx_ring);
+ continue;
+ }
+
+ xdp.data = bi->addr;
+ xdp.data_meta = xdp.data;
+ xdp.data_hard_start = xdp.data - XDP_PACKET_HEADROOM;
+ xdp.data_end = xdp.data + size;
+ xdp.handle = bi->handle;
+
+ xdp_res = ixgbe_run_xdp_zc(adapter, rx_ring, &xdp);
+
+ if (xdp_res) {
+ if (xdp_res & (IXGBE_XDP_TX | IXGBE_XDP_REDIR)) {
+ xdp_xmit |= xdp_res;
+ bi->addr = NULL;
+ bi->skb = NULL;
+ } else {
+ ixgbe_reuse_rx_buffer_zc(rx_ring, bi);
+ }
+ total_rx_packets++;
+ total_rx_bytes += size;
+
+ cleaned_count++;
+ ixgbe_inc_ntc(rx_ring);
+ continue;
+ }
+
+ /* XDP_PASS path */
+ skb = ixgbe_construct_skb_zc(rx_ring, bi, &xdp);
+ if (!skb) {
+ rx_ring->rx_stats.alloc_rx_buff_failed++;
+ break;
+ }
+
+ cleaned_count++;
+ ixgbe_inc_ntc(rx_ring);
+
+ if (eth_skb_pad(skb))
+ continue;
+
+ total_rx_bytes += skb->len;
+ total_rx_packets++;
+
+ ixgbe_process_skb_fields(rx_ring, rx_desc, skb);
+ ixgbe_rx_skb(q_vector, skb);
+ }
+
+ if (xdp_xmit & IXGBE_XDP_REDIR)
+ xdp_do_flush_map();
+
+ if (xdp_xmit & IXGBE_XDP_TX) {
+ struct ixgbe_ring *ring = adapter->xdp_ring[smp_processor_id()];
+
+ /* Force memory writes to complete before letting h/w
+ * know there are new descriptors to fetch.
+ */
+ wmb();
+ writel(ring->next_to_use, ring->tail);
+ }
+
+ u64_stats_update_begin(&rx_ring->syncp);
+ rx_ring->stats.packets += total_rx_packets;
+ rx_ring->stats.bytes += total_rx_bytes;
+ u64_stats_update_end(&rx_ring->syncp);
+ q_vector->rx.total_packets += total_rx_packets;
+ q_vector->rx.total_bytes += total_rx_bytes;
+
+ return failure ? budget : (int)total_rx_packets;
+}
+
+void ixgbe_xsk_clean_rx_ring(struct ixgbe_ring *rx_ring)
+{
+ u16 i = rx_ring->next_to_clean;
+ struct ixgbe_rx_buffer *bi = &rx_ring->rx_buffer_info[i];
+
+ while (i != rx_ring->next_to_alloc) {
+ xsk_umem_fq_reuse(rx_ring->xsk_umem, bi->handle);
+ i++;
+ bi++;
+ if (i == rx_ring->count) {
+ i = 0;
+ bi = rx_ring->rx_buffer_info;
+ }
+ }
+}
--
2.17.1
^ permalink raw reply related
* [PATCH v2 4/5] ixgbe: move common Tx functions to ixgbe_txrx_common.h
From: Björn Töpel @ 2018-10-02 8:00 UTC (permalink / raw)
To: jeffrey.t.kirsher, intel-wired-lan
Cc: Björn Töpel, magnus.karlsson, magnus.karlsson, ast,
daniel, netdev, brouer, u9012063, tuc, jakub.kicinski
In-Reply-To: <20181002080034.11754-1-bjorn.topel@gmail.com>
From: Björn Töpel <bjorn.topel@intel.com>
This patch prepares for the upcoming zero-copy Tx functionality by
moving common functions used both by the regular path and zero-copy
path.
Signed-off-by: Björn Töpel <bjorn.topel@intel.com>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 9 +++------
drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h | 5 +++++
2 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 547092b8fe54..b211032f8682 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -895,8 +895,8 @@ static void ixgbe_set_ivar(struct ixgbe_adapter *adapter, s8 direction,
}
}
-static inline void ixgbe_irq_rearm_queues(struct ixgbe_adapter *adapter,
- u64 qmask)
+void ixgbe_irq_rearm_queues(struct ixgbe_adapter *adapter,
+ u64 qmask)
{
u32 mask;
@@ -8156,9 +8156,6 @@ static inline int ixgbe_maybe_stop_tx(struct ixgbe_ring *tx_ring, u16 size)
return __ixgbe_maybe_stop_tx(tx_ring, size);
}
-#define IXGBE_TXD_CMD (IXGBE_TXD_CMD_EOP | \
- IXGBE_TXD_CMD_RS)
-
static int ixgbe_tx_map(struct ixgbe_ring *tx_ring,
struct ixgbe_tx_buffer *first,
const u8 hdr_len)
@@ -10259,7 +10256,7 @@ static int ixgbe_xdp(struct net_device *dev, struct netdev_bpf *xdp)
}
}
-static void ixgbe_xdp_ring_update_tail(struct ixgbe_ring *ring)
+void ixgbe_xdp_ring_update_tail(struct ixgbe_ring *ring)
{
/* Force memory writes to complete before letting h/w know there
* are new descriptors to fetch.
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
index cf219f4e009d..56afb685c648 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
@@ -9,6 +9,9 @@
#define IXGBE_XDP_TX BIT(1)
#define IXGBE_XDP_REDIR BIT(2)
+#define IXGBE_TXD_CMD (IXGBE_TXD_CMD_EOP | \
+ IXGBE_TXD_CMD_RS)
+
int ixgbe_xmit_xdp_ring(struct ixgbe_adapter *adapter,
struct xdp_frame *xdpf);
bool ixgbe_cleanup_headers(struct ixgbe_ring *rx_ring,
@@ -19,6 +22,8 @@ void ixgbe_process_skb_fields(struct ixgbe_ring *rx_ring,
struct sk_buff *skb);
void ixgbe_rx_skb(struct ixgbe_q_vector *q_vector,
struct sk_buff *skb);
+void ixgbe_xdp_ring_update_tail(struct ixgbe_ring *ring);
+void ixgbe_irq_rearm_queues(struct ixgbe_adapter *adapter, u64 qmask);
void ixgbe_txrx_ring_disable(struct ixgbe_adapter *adapter, int ring);
void ixgbe_txrx_ring_enable(struct ixgbe_adapter *adapter, int ring);
--
2.17.1
^ permalink raw reply related
* [PATCH v2 5/5] ixgbe: add AF_XDP zero-copy Tx support
From: Björn Töpel @ 2018-10-02 8:00 UTC (permalink / raw)
To: jeffrey.t.kirsher, intel-wired-lan
Cc: Björn Töpel, magnus.karlsson, magnus.karlsson, ast,
daniel, netdev, brouer, u9012063, tuc, jakub.kicinski
In-Reply-To: <20181002080034.11754-1-bjorn.topel@gmail.com>
From: Björn Töpel <bjorn.topel@intel.com>
This patch adds zero-copy Tx support for AF_XDP sockets. It implements
the ndo_xsk_async_xmit netdev ndo and performs all the Tx logic from a
NAPI context. This means pulling egress packets from the Tx ring,
placing the frames on the NIC HW descriptor ring and completing sent
frames back to the application via the completion ring.
The regular XDP Tx ring is used for AF_XDP as well. This rationale for
this is as follows: XDP_REDIRECT guarantees mutual exclusion between
different NAPI contexts based on CPU id. In other words, a netdev can
XDP_REDIRECT to another netdev with a different NAPI context, since
the operation is bound to a specific core and each core has its own
hardware ring.
As the AF_XDP Tx action is running in the same NAPI context and using
the same ring, it will also be protected from XDP_REDIRECT actions
with the exact same mechanism.
As with AF_XDP Rx, all AF_XDP Tx specific functions are added to
ixgbe_xsk.c.
Signed-off-by: Björn Töpel <bjorn.topel@intel.com>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 17 +-
.../ethernet/intel/ixgbe/ixgbe_txrx_common.h | 4 +
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 175 ++++++++++++++++++
3 files changed, 195 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index b211032f8682..ec31b32d6674 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -3161,7 +3161,11 @@ int ixgbe_poll(struct napi_struct *napi, int budget)
#endif
ixgbe_for_each_ring(ring, q_vector->tx) {
- if (!ixgbe_clean_tx_irq(q_vector, ring, budget))
+ bool wd = ring->xsk_umem ?
+ ixgbe_clean_xdp_tx_irq(q_vector, ring, budget) :
+ ixgbe_clean_tx_irq(q_vector, ring, budget);
+
+ if (!wd)
clean_complete = false;
}
@@ -3472,6 +3476,10 @@ void ixgbe_configure_tx_ring(struct ixgbe_adapter *adapter,
u32 txdctl = IXGBE_TXDCTL_ENABLE;
u8 reg_idx = ring->reg_idx;
+ ring->xsk_umem = NULL;
+ if (ring_is_xdp(ring))
+ ring->xsk_umem = ixgbe_xsk_umem(adapter, ring);
+
/* disable queue to avoid issues while updating state */
IXGBE_WRITE_REG(hw, IXGBE_TXDCTL(reg_idx), 0);
IXGBE_WRITE_FLUSH(hw);
@@ -5944,6 +5952,11 @@ static void ixgbe_clean_tx_ring(struct ixgbe_ring *tx_ring)
u16 i = tx_ring->next_to_clean;
struct ixgbe_tx_buffer *tx_buffer = &tx_ring->tx_buffer_info[i];
+ if (tx_ring->xsk_umem) {
+ ixgbe_xsk_clean_tx_ring(tx_ring);
+ goto out;
+ }
+
while (i != tx_ring->next_to_use) {
union ixgbe_adv_tx_desc *eop_desc, *tx_desc;
@@ -5995,6 +6008,7 @@ static void ixgbe_clean_tx_ring(struct ixgbe_ring *tx_ring)
if (!ring_is_xdp(tx_ring))
netdev_tx_reset_queue(txring_txq(tx_ring));
+out:
/* reset next_to_use and next_to_clean */
tx_ring->next_to_use = 0;
tx_ring->next_to_clean = 0;
@@ -10350,6 +10364,7 @@ static const struct net_device_ops ixgbe_netdev_ops = {
.ndo_features_check = ixgbe_features_check,
.ndo_bpf = ixgbe_xdp,
.ndo_xdp_xmit = ixgbe_xdp_xmit,
+ .ndo_xsk_async_xmit = ixgbe_xsk_async_xmit,
};
static void ixgbe_disable_txr_hw(struct ixgbe_adapter *adapter,
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
index 56afb685c648..53d4089f5644 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_txrx_common.h
@@ -42,5 +42,9 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
struct ixgbe_ring *rx_ring,
const int budget);
void ixgbe_xsk_clean_rx_ring(struct ixgbe_ring *rx_ring);
+bool ixgbe_clean_xdp_tx_irq(struct ixgbe_q_vector *q_vector,
+ struct ixgbe_ring *tx_ring, int napi_budget);
+int ixgbe_xsk_async_xmit(struct net_device *dev, u32 queue_id);
+void ixgbe_xsk_clean_tx_ring(struct ixgbe_ring *tx_ring);
#endif /* #define _IXGBE_TXRX_COMMON_H_ */
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
index 61259036ff4b..cf1c6f2d97e5 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
@@ -626,3 +626,178 @@ void ixgbe_xsk_clean_rx_ring(struct ixgbe_ring *rx_ring)
}
}
}
+
+static bool ixgbe_xmit_zc(struct ixgbe_ring *xdp_ring, unsigned int budget)
+{
+ union ixgbe_adv_tx_desc *tx_desc = NULL;
+ struct ixgbe_tx_buffer *tx_bi;
+ bool work_done = true;
+ u32 len, cmd_type;
+ dma_addr_t dma;
+
+ while (budget-- > 0) {
+ if (unlikely(!ixgbe_desc_unused(xdp_ring))) {
+ work_done = false;
+ break;
+ }
+
+ if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &dma, &len))
+ break;
+
+ dma_sync_single_for_device(xdp_ring->dev, dma, len,
+ DMA_BIDIRECTIONAL);
+
+ tx_bi = &xdp_ring->tx_buffer_info[xdp_ring->next_to_use];
+ tx_bi->bytecount = len;
+ tx_bi->xdpf = NULL;
+
+ tx_desc = IXGBE_TX_DESC(xdp_ring, xdp_ring->next_to_use);
+ tx_desc->read.buffer_addr = cpu_to_le64(dma);
+
+ /* put descriptor type bits */
+ cmd_type = IXGBE_ADVTXD_DTYP_DATA |
+ IXGBE_ADVTXD_DCMD_DEXT |
+ IXGBE_ADVTXD_DCMD_IFCS;
+ cmd_type |= len | IXGBE_TXD_CMD;
+ tx_desc->read.cmd_type_len = cpu_to_le32(cmd_type);
+ tx_desc->read.olinfo_status =
+ cpu_to_le32(len << IXGBE_ADVTXD_PAYLEN_SHIFT);
+
+ xdp_ring->next_to_use++;
+ if (xdp_ring->next_to_use == xdp_ring->count)
+ xdp_ring->next_to_use = 0;
+ }
+
+ if (tx_desc) {
+ ixgbe_xdp_ring_update_tail(xdp_ring);
+ xsk_umem_consume_tx_done(xdp_ring->xsk_umem);
+ }
+
+ return !!budget && work_done;
+}
+
+static void ixgbe_clean_xdp_tx_buffer(struct ixgbe_ring *tx_ring,
+ struct ixgbe_tx_buffer *tx_bi)
+{
+ xdp_return_frame(tx_bi->xdpf);
+ dma_unmap_single(tx_ring->dev,
+ dma_unmap_addr(tx_bi, dma),
+ dma_unmap_len(tx_bi, len), DMA_TO_DEVICE);
+ dma_unmap_len_set(tx_bi, len, 0);
+}
+
+bool ixgbe_clean_xdp_tx_irq(struct ixgbe_q_vector *q_vector,
+ struct ixgbe_ring *tx_ring, int napi_budget)
+{
+ unsigned int total_packets = 0, total_bytes = 0;
+ u32 i = tx_ring->next_to_clean, xsk_frames = 0;
+ unsigned int budget = q_vector->tx.work_limit;
+ struct xdp_umem *umem = tx_ring->xsk_umem;
+ union ixgbe_adv_tx_desc *tx_desc;
+ struct ixgbe_tx_buffer *tx_bi;
+ bool xmit_done;
+
+ tx_bi = &tx_ring->tx_buffer_info[i];
+ tx_desc = IXGBE_TX_DESC(tx_ring, i);
+ i -= tx_ring->count;
+
+ do {
+ if (!(tx_desc->wb.status & cpu_to_le32(IXGBE_TXD_STAT_DD)))
+ break;
+
+ total_bytes += tx_bi->bytecount;
+ total_packets += tx_bi->gso_segs;
+
+ if (tx_bi->xdpf)
+ ixgbe_clean_xdp_tx_buffer(tx_ring, tx_bi);
+ else
+ xsk_frames++;
+
+ tx_bi->xdpf = NULL;
+ total_bytes += tx_bi->bytecount;
+
+ tx_bi++;
+ tx_desc++;
+ i++;
+ if (unlikely(!i)) {
+ i -= tx_ring->count;
+ tx_bi = tx_ring->tx_buffer_info;
+ tx_desc = IXGBE_TX_DESC(tx_ring, 0);
+ }
+
+ /* issue prefetch for next Tx descriptor */
+ prefetch(tx_desc);
+
+ /* update budget accounting */
+ budget--;
+ } while (likely(budget));
+
+ i += tx_ring->count;
+ tx_ring->next_to_clean = i;
+
+ u64_stats_update_begin(&tx_ring->syncp);
+ tx_ring->stats.bytes += total_bytes;
+ tx_ring->stats.packets += total_packets;
+ u64_stats_update_end(&tx_ring->syncp);
+ q_vector->tx.total_bytes += total_bytes;
+ q_vector->tx.total_packets += total_packets;
+
+ if (xsk_frames)
+ xsk_umem_complete_tx(umem, xsk_frames);
+
+ xmit_done = ixgbe_xmit_zc(tx_ring, q_vector->tx.work_limit);
+ return budget > 0 && xmit_done;
+}
+
+int ixgbe_xsk_async_xmit(struct net_device *dev, u32 qid)
+{
+ struct ixgbe_adapter *adapter = netdev_priv(dev);
+ struct ixgbe_ring *ring;
+
+ if (test_bit(__IXGBE_DOWN, &adapter->state))
+ return -ENETDOWN;
+
+ if (!READ_ONCE(adapter->xdp_prog))
+ return -ENXIO;
+
+ if (qid >= adapter->num_xdp_queues)
+ return -ENXIO;
+
+ if (!adapter->xsk_umems || !adapter->xsk_umems[qid])
+ return -ENXIO;
+
+ ring = adapter->xdp_ring[qid];
+ if (!napi_if_scheduled_mark_missed(&ring->q_vector->napi)) {
+ u64 eics = BIT_ULL(ring->q_vector->v_idx);
+
+ ixgbe_irq_rearm_queues(adapter, eics);
+ }
+
+ return 0;
+}
+
+void ixgbe_xsk_clean_tx_ring(struct ixgbe_ring *tx_ring)
+{
+ u16 ntc = tx_ring->next_to_clean, ntu = tx_ring->next_to_use;
+ struct xdp_umem *umem = tx_ring->xsk_umem;
+ struct ixgbe_tx_buffer *tx_bi;
+ u32 xsk_frames = 0;
+
+ while (ntc != ntu) {
+ tx_bi = &tx_ring->tx_buffer_info[ntc];
+
+ if (tx_bi->xdpf)
+ ixgbe_clean_xdp_tx_buffer(tx_ring, tx_bi);
+ else
+ xsk_frames++;
+
+ tx_bi->xdpf = NULL;
+
+ ntc++;
+ if (ntc == tx_ring->count)
+ ntc = 0;
+ }
+
+ if (xsk_frames)
+ xsk_umem_complete_tx(umem, xsk_frames);
+}
--
2.17.1
^ permalink raw reply related
* Re: Regression: kernel 4.14 an later very slow with many ipsec tunnels
From: Wolfgang Walter @ 2018-10-02 14:45 UTC (permalink / raw)
To: Florian Westphal
Cc: Steffen Klassert, David Miller, netdev, linux-kernel, torvalds,
christophe.gouault
In-Reply-To: <20180914055437.77pffp2jrbfnykbp@breakpoint.cc>
Hello,
Am Freitag, 14. September 2018, 07:54:37 schrieb Florian Westphal:
> Steffen Klassert <steffen.klassert@secunet.com> wrote:
> > On Thu, Sep 13, 2018 at 11:03:25PM +0200, Florian Westphal wrote:
> > > David Miller <davem@davemloft.net> wrote:
> > > > From: Florian Westphal <fw@strlen.de>
> > > > Date: Thu, 13 Sep 2018 18:38:48 +0200
> > > >
> > > > > Wolfgang Walter <linux@stwm.de> wrote:
> > > > >> What I can say is that it depends mainly on number of policy rules
> > > > >> and SA.
> > > > >
> > > > > Thats already a good hint, I guess we're hitting long hash chains in
> > > > > xfrm_policy_lookup_bytype().
> > > >
> > > > I don't really see how recent changes can influence that.
> > >
> > > I don't think there is a recent change that did this.
> > >
> > > Walter says < 4.14 is ok, so this is likely related to flow cache
> > > removal.
> > >
> > > F.e. it looks like all prefixed policies end up in a linked list
> > > (net->xfrm.policy_inexact) and are not even in a hash table.
> > >
> > > I am staring at b58555f1767c9f4e330fcf168e4e753d2d9196e0
> > > but can't figure out how to configure that away from the
> > > 'no hashing for prefixed policies' default or why we even have
> > > policy_inexact in first place :/
> >
> > The hash threshold can be configured like this:
> >
> > ip x p set hthresh4 0 0
> >
> > This sets the hash threshold to local /0 and remote /0 netmasks.
> > With this configuration, all policies should go to the hashtable.
>
> Yes, but won't they all be hashed to same bucket?
>
> [ jhash(addr & 0, addr & 0) ] ?
>
> > Default hash thresholds are local /32 and remote /32 netmasks, so
> > all prefixed policies go to the inexact list.
>
> Yes.
>
> Wolfgang, before having to work on getting perf into your router image
> can you perhaps share a bit of info about the policies you're using?
>
> How many are there? Are they prefixed or not ("10.1.2.1")?
Since my last reply to this message I didn't get a reply: is there any
progress how to fix this performance regression I missed?
Or are we stuck here with longterm kernel 4.9 for a long time?
Regards,
--
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts
^ permalink raw reply
* Re: general protection fault in integrity_inode_free
From: Theodore Y. Ts'o @ 2018-10-02 14:50 UTC (permalink / raw)
To: syzbot; +Cc: davem, linux-kernel, netdev, syzkaller-bugs
In-Reply-To: <0000000000009581bf0576c16d3a@google.com>
On Wed, Sep 26, 2018 at 12:47:02AM -0700, syzbot wrote:
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit: 846e8dd47c26 Merge tag 'scsi-fixes' of git://git.kernel.or..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=12a59d56400000
> kernel config: https://syzkaller.appspot.com/x/.config?x=dfb440e26f0a6f6f
> dashboard link: https://syzkaller.appspot.com/bug?extid=4c0fa7d385ea5af34e91
> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
#syz dup: general protection fault in rb_erase
^ permalink raw reply
* Re: vlan missing with AF_PACKET and auxdata
From: Michael Walle @ 2018-10-02 8:13 UTC (permalink / raw)
To: netdev; +Cc: David S. Miller, Eric Dumazet, Steinar H. Gunderson, heiko.thiery
In-Reply-To: <9a284a4d29bb3301f50d7993c6bebb04@walle.cc>
Am 2018-09-24 14:01, schrieb Michael Walle:
> Hi,
>
> I'm using the AF_PACKET socket with setsockopt(PACKET_AUXDATA) to get
> the incoming VLAN tag. Correct me if I'm wrong, but as far as I see
> the first VLAN tag is always stripped - either in hardware or in
> net/core/dev.c in __netif_receive_skb_core() - and stored in
> skb->vlan_tci. Therefore, it won't be in the packet data anymore.
>
> If I use the socket with ETH_P_ALL as protocol, everything works as
> expected. But if I'm using an actual protocol number, instead of the
> catch all, the tp_vlan_tci field inside the auxdata will be zero. I've
> traced this to commit d4b812dea4a236f729526facf97df1a9d18e191c (vlan:
> mask vlan prio bits):
>
> + if (unlikely(vlan_tx_tag_present(skb))) {
> + if (vlan_tx_tag_get_id(skb))
> + skb->pkt_type = PACKET_OTHERHOST;
> + /* Note: we might in the future use prio bits
> + * and set skb->priority like in vlan_do_receive()
> + * For the time being, just ignore Priority Code Point
> + */
> + skb->vlan_tci = 0;
> + }
>
> The ptype_all callbacks (which are working as expected) are before
> this code and the ptype_specific callbacks (which are not working) are
> after this piece of code. I don't know it this is a bug or not, I
> guess it is one, at least from the AF_PACKET socket point of view. If
> not, how I'm supposed to get the original VLAN tag with
> socket(AF_PACKET, my_protocol)?
>
> I also don't understand the commit (message) as the subject suggesets
> only the prio bits should be masked, but with "skb->vlan_tci = 0"
> everything is masked. Therefore, I've put the original (hopefully the
> mail addresses are still valid) authors in CC.
>
ping? :)
-michael
^ permalink raw reply
* Re: Regression: kernel 4.14 an later very slow with many ipsec tunnels
From: Florian Westphal @ 2018-10-02 14:56 UTC (permalink / raw)
To: Wolfgang Walter
Cc: Florian Westphal, Steffen Klassert, David Miller, netdev,
linux-kernel, torvalds, christophe.gouault
In-Reply-To: <4708967.r5gU1pxIcW@stwm.de>
Wolfgang Walter <linux@stwm.de> wrote:
> Since my last reply to this message I didn't get a reply: is there any
> progress how to fix this performance regression I missed?
Did you test/experiment with hthresh config option?
> Or are we stuck here with longterm kernel 4.9 for a long time?
I'm experimenting with per-dst inexact lists in an rbtree but
this will take time.
^ permalink raw reply
* Re: [PATCH v2 net] inet: frags: rework rhashtable dismantle
From: Dmitry Vyukov @ 2018-10-02 8:19 UTC (permalink / raw)
To: Eric Dumazet
Cc: David S . Miller, netdev, Eric Dumazet, Thomas Graf, Herbert Xu
In-Reply-To: <20181002054917.37102-1-edumazet@google.com>
On Tue, Oct 2, 2018 at 7:49 AM, Eric Dumazet <edumazet@google.com> wrote:
> syszbot found an interesting use-after-free [1] happening
> while IPv4 fragment rhashtable was destroyed at netns dismantle.
>
> While no insertions can possibly happen at the time a dismantling
> netns is destroying this rhashtable, timers can still fire and
> attempt to remove elements from this rhashtable.
>
> This is forbidden, since rhashtable_free_and_destroy() has
> no synchronization against concurrent inserts and deletes.
>
> Add a new nf->dead flag so that timers do not attempt
> a rhashtable_remove_fast() operation.
>
> [1]
> BUG: KASAN: use-after-free in __read_once_size include/linux/compiler.h:188 [inline]
> BUG: KASAN: use-after-free in rhashtable_last_table+0x216/0x240 lib/rhashtable.c:217
> Read of size 8 at addr ffff88019a4c8840 by task kworker/0:4/8279
>
> CPU: 0 PID: 8279 Comm: kworker/0:4 Not tainted 4.19.0-rc5+ #61
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> Workqueue: events rht_deferred_worker
> Call Trace:
> __dump_stack lib/dump_stack.c:77 [inline]
> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
> kasan_report_error mm/kasan/report.c:354 [inline]
> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
> __read_once_size include/linux/compiler.h:188 [inline]
> rhashtable_last_table+0x216/0x240 lib/rhashtable.c:217
> rht_deferred_worker+0x157/0x1de0 lib/rhashtable.c:410
> process_one_work+0xc90/0x1b90 kernel/workqueue.c:2153
> worker_thread+0x17f/0x1390 kernel/workqueue.c:2296
> kthread+0x35a/0x420 kernel/kthread.c:246
> ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:413
>
> Allocated by task 5:
> save_stack+0x43/0xd0 mm/kasan/kasan.c:448
> set_track mm/kasan/kasan.c:460 [inline]
> kasan_kmalloc+0xc7/0xe0 mm/kasan/kasan.c:553
> __do_kmalloc_node mm/slab.c:3682 [inline]
> __kmalloc_node+0x47/0x70 mm/slab.c:3689
> kmalloc_node include/linux/slab.h:555 [inline]
> kvmalloc_node+0xb9/0xf0 mm/util.c:423
> kvmalloc include/linux/mm.h:577 [inline]
> kvzalloc include/linux/mm.h:585 [inline]
> bucket_table_alloc+0x9a/0x4e0 lib/rhashtable.c:176
> rhashtable_rehash_alloc+0x73/0x100 lib/rhashtable.c:353
> rht_deferred_worker+0x278/0x1de0 lib/rhashtable.c:413
> process_one_work+0xc90/0x1b90 kernel/workqueue.c:2153
> worker_thread+0x17f/0x1390 kernel/workqueue.c:2296
> kthread+0x35a/0x420 kernel/kthread.c:246
> ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:413
>
> Freed by task 8283:
> save_stack+0x43/0xd0 mm/kasan/kasan.c:448
> set_track mm/kasan/kasan.c:460 [inline]
> __kasan_slab_free+0x102/0x150 mm/kasan/kasan.c:521
> kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528
> __cache_free mm/slab.c:3498 [inline]
> kfree+0xcf/0x230 mm/slab.c:3813
> kvfree+0x61/0x70 mm/util.c:452
> bucket_table_free+0xda/0x250 lib/rhashtable.c:108
> rhashtable_free_and_destroy+0x152/0x900 lib/rhashtable.c:1163
> inet_frags_exit_net+0x3d/0x50 net/ipv4/inet_fragment.c:96
> ipv4_frags_exit_net+0x73/0x90 net/ipv4/ip_fragment.c:914
> ops_exit_list.isra.7+0xb0/0x160 net/core/net_namespace.c:153
> cleanup_net+0x555/0xb10 net/core/net_namespace.c:551
> process_one_work+0xc90/0x1b90 kernel/workqueue.c:2153
> worker_thread+0x17f/0x1390 kernel/workqueue.c:2296
> kthread+0x35a/0x420 kernel/kthread.c:246
> ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:413
>
> The buggy address belongs to the object at ffff88019a4c8800
> which belongs to the cache kmalloc-16384 of size 16384
> The buggy address is located 64 bytes inside of
> 16384-byte region [ffff88019a4c8800, ffff88019a4cc800)
> The buggy address belongs to the page:
> page:ffffea0006693200 count:1 mapcount:0 mapping:ffff8801da802200 index:0x0 compound_mapcount: 0
> flags: 0x2fffc0000008100(slab|head)
> raw: 02fffc0000008100 ffffea0006685608 ffffea0006617c08 ffff8801da802200
> raw: 0000000000000000 ffff88019a4c8800 0000000100000001 0000000000000000
> page dumped because: kasan: bad access detected
>
> Memory state around the buggy address:
> ffff88019a4c8700: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> ffff88019a4c8780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>>ffff88019a4c8800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ^
> ffff88019a4c8880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff88019a4c8900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>
> Fixes: 648700f76b03 ("inet: frags: use rhashtables for reassembly units")
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Reported-by: syzbot <syzkaller@googlegroups.com>
> Cc: Thomas Graf <tgraf@suug.ch>
> Cc: Herbert Xu <herbert@gondor.apana.org.au>
> ---
> include/net/inet_frag.h | 4 +++-
> net/ipv4/inet_fragment.c | 31 +++++++++++++++++++++----------
> 2 files changed, 24 insertions(+), 11 deletions(-)
>
> diff --git a/include/net/inet_frag.h b/include/net/inet_frag.h
> index 1662cbc0b46b45296a367ecbdaf03c68854fdce7..ffe5e1be40212fa63e360f3e29a56c1b2ce897ee 100644
> --- a/include/net/inet_frag.h
> +++ b/include/net/inet_frag.h
> @@ -11,7 +11,7 @@ struct netns_frags {
> int timeout;
> int max_dist;
> struct inet_frags *f;
> -
> + bool dead;
> struct rhashtable rhashtable ____cacheline_aligned_in_smp;
>
> /* Keep atomic mem on separate cachelines in structs that include it */
> @@ -24,11 +24,13 @@ struct netns_frags {
> * @INET_FRAG_FIRST_IN: first fragment has arrived
> * @INET_FRAG_LAST_IN: final fragment has arrived
> * @INET_FRAG_COMPLETE: frag queue has been processed and is due for destruction
> + * @INET_FRAG_HASH_DEAD: inet_frag_kill() has not removed fq from rhashtable
> */
> enum {
> INET_FRAG_FIRST_IN = BIT(0),
> INET_FRAG_LAST_IN = BIT(1),
> INET_FRAG_COMPLETE = BIT(2),
> + INET_FRAG_HASH_DEAD = BIT(3),
> };
>
> struct frag_v4_compare_key {
> diff --git a/net/ipv4/inet_fragment.c b/net/ipv4/inet_fragment.c
> index bcb11f3a27c0c34115af05034a5a20f57842eb0a..887134425350cc7c8ee081d8c83a037074d96d93 100644
> --- a/net/ipv4/inet_fragment.c
> +++ b/net/ipv4/inet_fragment.c
> @@ -71,28 +71,31 @@ EXPORT_SYMBOL(inet_frags_fini);
> static void inet_frags_free_cb(void *ptr, void *arg)
> {
> struct inet_frag_queue *fq = ptr;
> + int count = 0;
>
> - /* If we can not cancel the timer, it means this frag_queue
> - * is already disappearing, we have nothing to do.
> - * Otherwise, we own a refcount until the end of this function.
> - */
> - if (!del_timer(&fq->timer))
> - return;
> + if (del_timer_sync(&fq->timer))
> + count++;
>
> spin_lock_bh(&fq->lock);
> if (!(fq->flags & INET_FRAG_COMPLETE)) {
> fq->flags |= INET_FRAG_COMPLETE;
> - refcount_dec(&fq->refcnt);
> + count++;
> + } else if (fq->flags & INET_FRAG_HASH_DEAD) {
> + count++;
> }
> spin_unlock_bh(&fq->lock);
>
> - inet_frag_put(fq);
> + if (refcount_sub_and_test(count, &fq->refcnt))
> + inet_frag_destroy(fq);
> }
>
> void inet_frags_exit_net(struct netns_frags *nf)
> {
> nf->high_thresh = 0; /* prevent creation of new frags */
>
> + /* paired with READ_ONCE() in inet_frag_kill() */
> + smp_store_release(&nf->dead, true);
> +
> rhashtable_free_and_destroy(&nf->rhashtable, inet_frags_free_cb, NULL);
> }
> EXPORT_SYMBOL(inet_frags_exit_net);
> @@ -106,8 +109,16 @@ void inet_frag_kill(struct inet_frag_queue *fq)
> struct netns_frags *nf = fq->net;
>
> fq->flags |= INET_FRAG_COMPLETE;
> - rhashtable_remove_fast(&nf->rhashtable, &fq->node, nf->f->rhash_params);
> - refcount_dec(&fq->refcnt);
> + /* This READ_ONCE() is paired with smp_store_release()
> + * in inet_frags_exit_net().
> + */
> + if (!READ_ONCE(nf->dead)) {
Does inet_frag_kill() hold fq->lock? I am missing how inet_frag_kill()
and inet_frags_exit_net() are synchronized.
Since you use smp_store_release()/READ_ONCE() they seem to run in
parallel. But then isn't it possible that inet_frag_kill() reads
nf->dead == 0, then inet_frags_exit_net() sets nf->dead, and then we
have the same race on concurrent removal? Or, isn't it possible that
inet_frag_kill() reads nf->dead == 1, but does not set
INET_FRAG_HASH_DEAD yet, and then inet_frags_free_cb() misses the
INET_FRAG_HASH_DEAD flag?
> + rhashtable_remove_fast(&nf->rhashtable, &fq->node,
> + nf->f->rhash_params);
> + refcount_dec(&fq->refcnt);
> + } else {
> + fq->flags |= INET_FRAG_HASH_DEAD;
> + }
> }
> }
> EXPORT_SYMBOL(inet_frag_kill);
> --
> 2.19.0.605.g01d371f741-goog
>
^ permalink raw reply
* Re: [PATCH net-next] netfilter: xt_quota: fix the behavior of xt_quota module
From: Maciej Żenczykowski @ 2018-10-02 8:24 UTC (permalink / raw)
To: Pablo Neira Ayuso
Cc: Chenbo Feng, Linux NetDev, netfilter-devel, kernel-team,
Lorenzo Colitti, Chenbo Feng, Maciej Żenczykowski,
Maciej Zenczykowski
In-Reply-To: <20181002075903.3wpgej3j6dttbqck@salvia>
> A few questions, see below.
>
> First one is, don't we need a new match revision for this new option?
We were very careful to do this in a way that doesn't need a new revision.
That's what basically dictates most of the design.
If we bump the revision then you don't get fixed semantics unless
you update both kernel and userspace iptables versions... and additionally
we basically end up with two copies of xt_quota in the kernel source since
there's pretty much nothing that can be shared between the two.
> So 1 means, don't keep updating, quota is depleted?
The counter is always 1 higher then the remaining quota.
So 1 means 0.
And 0 means uninitialized and is only present on input from userspace.
> This current_count = 1 would be exposed to userspace too, right?
>
> Hm, this semantics are going to be a bit awkwards to users I think, I
> would prefer to expose this in a different way.
New userspace iptables hides this from users by adding/decrementing by
one as needed on ingress/egress from kernel.
Old iptables never looks at this field.
>> + if (current_count <= skb->len) {
>> + atomic64_set(&q->counter, 1);
>> + return ret;
>> + }
>> + old_count = current_count;
>> + new_count = current_count - skb->len;
>> + current_count = atomic64_cmpxchg(&q->counter, old_count,
>> + new_count);
>> + } while (current_count != old_count);
>
> Probably we simplify this via atomic64_add_return()?
Unfortunately that doesn't work because it's racy if current value is
2 and two (or three) threads both add -1 you end up at zero (or even
+lots).
> I guess problem is userspace may get a current counter that is larger
> than the quota, but we could handle this from userspace iptables to
> print a value that equals the quota, ie. from userspace, before
> printing:
I'm not sure what you mean.
>
> if (consumed > quota)
> printf("--consumed %PRIu64 ", quota);
> else
> printf("--consumed %PRIu64 ", consumed);
>
>> + return !ret;
>> }
>
> Thanks !
Maciej Żenczykowski, Kernel Networking Developer @ Google
^ permalink raw reply
* Re: [PATCH net-next] netfilter: xt_quota: fix the behavior of xt_quota module
From: Maciej Żenczykowski @ 2018-10-02 8:25 UTC (permalink / raw)
To: Pablo Neira Ayuso
Cc: Chenbo Feng, Linux NetDev, netfilter-devel, kernel-team,
Lorenzo Colitti, Chenbo Feng, Maciej Żenczykowski,
Maciej Zenczykowski
In-Reply-To: <CANP3RGeo71YYqPENVkEC_XzS+44ANQ9gLV2eX=aZbuc6TMuTAQ@mail.gmail.com>
(perhaps the field should be called remain_plus_one?)
^ permalink raw reply
* [PATCH net-next] net: drop unused skb_append_datato_frags()
From: Paolo Abeni @ 2018-10-02 9:03 UTC (permalink / raw)
To: netdev; +Cc: David S. Miller
This helper is unused since commit 988cf74deb45 ("inet:
Stop generating UFO packets.")
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
---
include/linux/skbuff.h | 5 ----
net/core/skbuff.c | 58 ------------------------------------------
2 files changed, 63 deletions(-)
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 87e29710373f..119d092c6b13 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -1082,11 +1082,6 @@ static inline int skb_pad(struct sk_buff *skb, int pad)
}
#define dev_kfree_skb(a) consume_skb(a)
-int skb_append_datato_frags(struct sock *sk, struct sk_buff *skb,
- int getfrag(void *from, char *to, int offset,
- int len, int odd, struct sk_buff *skb),
- void *from, int length);
-
int skb_append_pagefrags(struct sk_buff *skb, struct page *page,
int offset, size_t size);
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index b2c807f67aba..0e937d3d85b5 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -3381,64 +3381,6 @@ unsigned int skb_find_text(struct sk_buff *skb, unsigned int from,
}
EXPORT_SYMBOL(skb_find_text);
-/**
- * skb_append_datato_frags - append the user data to a skb
- * @sk: sock structure
- * @skb: skb structure to be appended with user data.
- * @getfrag: call back function to be used for getting the user data
- * @from: pointer to user message iov
- * @length: length of the iov message
- *
- * Description: This procedure append the user data in the fragment part
- * of the skb if any page alloc fails user this procedure returns -ENOMEM
- */
-int skb_append_datato_frags(struct sock *sk, struct sk_buff *skb,
- int (*getfrag)(void *from, char *to, int offset,
- int len, int odd, struct sk_buff *skb),
- void *from, int length)
-{
- int frg_cnt = skb_shinfo(skb)->nr_frags;
- int copy;
- int offset = 0;
- int ret;
- struct page_frag *pfrag = ¤t->task_frag;
-
- do {
- /* Return error if we don't have space for new frag */
- if (frg_cnt >= MAX_SKB_FRAGS)
- return -EMSGSIZE;
-
- if (!sk_page_frag_refill(sk, pfrag))
- return -ENOMEM;
-
- /* copy the user data to page */
- copy = min_t(int, length, pfrag->size - pfrag->offset);
-
- ret = getfrag(from, page_address(pfrag->page) + pfrag->offset,
- offset, copy, 0, skb);
- if (ret < 0)
- return -EFAULT;
-
- /* copy was successful so update the size parameters */
- skb_fill_page_desc(skb, frg_cnt, pfrag->page, pfrag->offset,
- copy);
- frg_cnt++;
- pfrag->offset += copy;
- get_page(pfrag->page);
-
- skb->truesize += copy;
- refcount_add(copy, &sk->sk_wmem_alloc);
- skb->len += copy;
- skb->data_len += copy;
- offset += copy;
- length -= copy;
-
- } while (length > 0);
-
- return 0;
-}
-EXPORT_SYMBOL(skb_append_datato_frags);
-
int skb_append_pagefrags(struct sk_buff *skb, struct page *page,
int offset, size_t size)
{
--
2.17.1
^ permalink raw reply related
* Re: [PATCH net] rtnetlink: Fail dump if target netnsid is invalid
From: Jiri Benc @ 2018-10-02 10:04 UTC (permalink / raw)
To: David Ahern; +Cc: netdev, davem, David Ahern
In-Reply-To: <20180928192841.20410-1-dsahern@kernel.org>
On Fri, 28 Sep 2018 12:28:41 -0700, David Ahern wrote:
> --- a/net/core/rtnetlink.c
> +++ b/net/core/rtnetlink.c
> @@ -1898,10 +1898,8 @@ static int rtnl_dump_ifinfo(struct sk_buff *skb, struct netlink_callback *cb)
> if (tb[IFLA_IF_NETNSID]) {
> netnsid = nla_get_s32(tb[IFLA_IF_NETNSID]);
> tgt_net = get_target_net(skb->sk, netnsid);
> - if (IS_ERR(tgt_net)) {
> - tgt_net = net;
> - netnsid = -1;
> - }
> + if (IS_ERR(tgt_net))
> + return PTR_ERR(tgt_net);
> }
>
> if (tb[IFLA_EXT_MASK])
Sorry for the late review, I see it has been applied.
I intentionally chose the behavior to preserve the behavior of the
older kernels: that attribute was silently ignored. Note that the
IFLA_IF_NETNSID is not returned in such case, thus it's easy to
distinguish that it was not applied. And the user space has to do such
check anyway to support old kernels.
But you're right that there was no way to distinguish "the kernel does
not support IFLA_IF_NETNSID" from "wrong IFLA_IF_NETNSID provided". I'm
okay with the patch, I just don't think the "Fixes" tag is justified but
whatever, can't be unapplied :-) (and it's my fault for not reviewing
the patches timely).
Thanks!
Jiri
^ permalink raw reply
* Re: [PATCH net-next] netfilter: xt_quota: fix the behavior of xt_quota module
From: Pablo Neira Ayuso @ 2018-10-02 10:11 UTC (permalink / raw)
To: Maciej Żenczykowski
Cc: Chenbo Feng, Linux NetDev, netfilter-devel, kernel-team,
Lorenzo Colitti, Chenbo Feng, Maciej Zenczykowski
In-Reply-To: <CANP3RGeo71YYqPENVkEC_XzS+44ANQ9gLV2eX=aZbuc6TMuTAQ@mail.gmail.com>
Hi Maciej!
On Tue, Oct 02, 2018 at 01:24:29AM -0700, Maciej Żenczykowski wrote:
> > A few questions, see below.
> >
> > First one is, don't we need a new match revision for this new option?
>
> We were very careful to do this in a way that doesn't need a new revision.
>
> That's what basically dictates most of the design.
>
> If we bump the revision then you don't get fixed semantics unless
> you update both kernel and userspace iptables versions...
Well, you will need a kernel + userspace update anyway, right?
> and additionally we basically end up with two copies of xt_quota in
> the kernel source since there's pretty much nothing that can be
> shared between the two.
OK. Well, there is still one downside though from user perspective: A
user with new iptables userspace that supports --remain and old kernel
will not get this to work, since this option will be silently accepted
by the kernel, but it will be simply ignored.
I agree having revision infrastructure is limited and results in more
duplicated code, but this is what we have in iptables.
> > So 1 means, don't keep updating, quota is depleted?
>
> The counter is always 1 higher then the remaining quota.
> So 1 means 0.
> And 0 means uninitialized and is only present on input from userspace.
OK.
> > This current_count = 1 would be exposed to userspace too, right?
> >
> > Hm, this semantics are going to be a bit awkwards to users I think, I
> > would prefer to expose this in a different way.
>
> New userspace iptables hides this from users by adding/decrementing by
> one as needed on ingress/egress from kernel.
> Old iptables never looks at this field.
I see, indeed.
> >> + if (current_count <= skb->len) {
> >> + atomic64_set(&q->counter, 1);
> >> + return ret;
> >> + }
> >> + old_count = current_count;
> >> + new_count = current_count - skb->len;
> >> + current_count = atomic64_cmpxchg(&q->counter, old_count,
> >> + new_count);
> >> + } while (current_count != old_count);
> >
> > Probably we simplify this via atomic64_add_return()?
>
> Unfortunately that doesn't work because it's racy if current value is
> 2 and two (or three) threads both add -1 you end up at zero (or even
> +lots).
Hm, not sure what you mean with adding -1. I mean replacing all this
code above with something like:
ret = (atomic64_add_return(skb->len, &consumed) >= quota) ^ XT_QUOTA_INVERT;
But I might be missing anything from your response on why this is
racy.
> > I guess problem is userspace may get a current counter that is larger
> > than the quota, but we could handle this from userspace iptables to
> > print a value that equals the quota, ie. from userspace, before
> > printing:
>
> I'm not sure what you mean.
I mean: Instead of using atomic64_set() to set the counter to 1 once
we went over quota,
Please, don't get this as some sort of push back / I'm saying no to
this as it is.
I just would like we are aware of possible downsides with this
approach. Having said this, I'll take it as is if you insist that this
is the right approach :-)
Thanks !
^ permalink raw reply
* [PATCH] usbnet: smsc95xx: simplify tx_fixup code
From: Ben Dooks @ 2018-10-02 16:56 UTC (permalink / raw)
To: netdev
Cc: David.Laight, oneukum, davem, linux-usb, linux-kernel,
linux-kernel, Ben Dooks
In-Reply-To: <59988ed22559410881addfecf58335eb@AcuMS.aculab.com>
The smsc95xx_tx_fixup is doing multiple calls to skb_push() to
put an 8-byte command header onto the packet. It would be easier
to do one skb_push() and then copy the data in once the push is
done.
Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
---
drivers/net/usb/smsc95xx.c | 25 +++++++++++++------------
1 file changed, 13 insertions(+), 12 deletions(-)
diff --git a/drivers/net/usb/smsc95xx.c b/drivers/net/usb/smsc95xx.c
index cb19aea139d3..813ab93ee2c3 100644
--- a/drivers/net/usb/smsc95xx.c
+++ b/drivers/net/usb/smsc95xx.c
@@ -2006,6 +2006,7 @@ static struct sk_buff *smsc95xx_tx_fixup(struct usbnet *dev,
bool csum = skb->ip_summed == CHECKSUM_PARTIAL;
int overhead = csum ? SMSC95XX_TX_OVERHEAD_CSUM : SMSC95XX_TX_OVERHEAD;
u32 tx_cmd_a, tx_cmd_b;
+ void *ptr;
/* We do not advertise SG, so skbs should be already linearized */
BUG_ON(skb_shinfo(skb)->nr_frags);
@@ -2019,6 +2020,9 @@ static struct sk_buff *smsc95xx_tx_fixup(struct usbnet *dev,
return NULL;
}
+ tx_cmd_b = (u32)skb->len;
+ tx_cmd_a = tx_cmd_b | TX_CMD_A_FIRST_SEG_ | TX_CMD_A_LAST_SEG_;
+
if (csum) {
if (skb->len <= 45) {
/* workaround - hardware tx checksum does not work
@@ -2035,21 +2039,18 @@ static struct sk_buff *smsc95xx_tx_fixup(struct usbnet *dev,
skb_push(skb, 4);
cpu_to_le32s(&csum_preamble);
memcpy(skb->data, &csum_preamble, 4);
+
+ tx_cmd_a += 4;
+ tx_cmd_b += 4;
+ tx_cmd_b |= TX_CMD_B_CSUM_ENABLE;
}
}
- skb_push(skb, 4);
- tx_cmd_b = (u32)(skb->len - 4);
- if (csum)
- tx_cmd_b |= TX_CMD_B_CSUM_ENABLE;
- cpu_to_le32s(&tx_cmd_b);
- memcpy(skb->data, &tx_cmd_b, 4);
-
- skb_push(skb, 4);
- tx_cmd_a = (u32)(skb->len - 8) | TX_CMD_A_FIRST_SEG_ |
- TX_CMD_A_LAST_SEG_;
- cpu_to_le32s(&tx_cmd_a);
- memcpy(skb->data, &tx_cmd_a, 4);
+ ptr = skb_push(skb, 8);
+ tx_cmd_a = cpu_to_le32(tx_cmd_a);
+ tx_cmd_b = cpu_to_le32(tx_cmd_b);
+ memcpy(ptr, &tx_cmd_a, 4);
+ memcpy(ptr+4, &tx_cmd_b, 4);
return skb;
}
--
2.19.0
^ permalink raw reply related
* Re: [PATCH net-next] netfilter: xt_quota: fix the behavior of xt_quota module
From: Pablo Neira Ayuso @ 2018-10-02 10:15 UTC (permalink / raw)
To: Maciej Żenczykowski
Cc: Chenbo Feng, Linux NetDev, netfilter-devel, kernel-team,
Lorenzo Colitti, Chenbo Feng, Maciej Zenczykowski
In-Reply-To: <20181002101119.tyljwzqpdj7qoe6f@salvia>
On Tue, Oct 02, 2018 at 12:11:19PM +0200, Pablo Neira Ayuso wrote:
> Hi Maciej!
>
[...]
> On Tue, Oct 02, 2018 at 01:24:29AM -0700, Maciej Żenczykowski wrote:
> > > I guess problem is userspace may get a current counter that is larger
> > > than the quota, but we could handle this from userspace iptables to
> > > print a value that equals the quota, ie. from userspace, before
> > > printing:
> >
> > I'm not sure what you mean.
>
> I mean: Instead of using atomic64_set() to set the counter to 1 once
> we went over quota,
incomplete sentence, sorry:
I mean: Instead of using atomic64_set() to set the counter to 1 once
we go overquota, we just keep updating 'consumed' bytes.
ie. we don't express things in 'remaining bytes' logic, but we account
for 'bytes we already consumed'. So we never go negative - I know
understand what you mean about -1... I think we are each other
thinking from our respective approach proposal.
:-)
Thanks!
^ permalink raw reply
* Re: [PATCH net-next v6 19/23] zinc: Curve25519 ARM implementation
From: Ard Biesheuvel @ 2018-10-02 16:59 UTC (permalink / raw)
To: Jason A. Donenfeld
Cc: Linux Kernel Mailing List, <netdev@vger.kernel.org>,
open list:HARDWARE RANDOM NUMBER GENERATOR CORE, David S. Miller,
Greg Kroah-Hartman, Samuel Neves, Andy Lutomirski,
Jean-Philippe Aumasson, Russell King, linux-arm-kernel
In-Reply-To: <20180925145622.29959-20-Jason@zx2c4.com>
Hi Jason,
On 25 September 2018 at 16:56, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> This comes from Dan Bernstein and Peter Schwabe's public domain NEON
> code, and has been modified to be friendly for kernel space, as well as
> removing some qhasm strangeness to be more idiomatic.
>
> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
> Cc: Samuel Neves <sneves@dei.uc.pt>
> Cc: Andy Lutomirski <luto@kernel.org>
> Cc: Greg KH <gregkh@linuxfoundation.org>
> Cc: Jean-Philippe Aumasson <jeanphilippe.aumasson@gmail.com>
> Cc: Russell King <linux@armlinux.org.uk>
> Cc: linux-arm-kernel@lists.infradead.org
> ---
> lib/zinc/Makefile | 1 +
> lib/zinc/curve25519/curve25519-arm-glue.h | 42 +
> lib/zinc/curve25519/curve25519-arm.S | 2095 +++++++++++++++++++++
> lib/zinc/curve25519/curve25519.c | 2 +
> 4 files changed, 2140 insertions(+)
> create mode 100644 lib/zinc/curve25519/curve25519-arm-glue.h
> create mode 100644 lib/zinc/curve25519/curve25519-arm.S
>
> diff --git a/lib/zinc/Makefile b/lib/zinc/Makefile
> index 65440438c6e5..be73c342f9ba 100644
> --- a/lib/zinc/Makefile
> +++ b/lib/zinc/Makefile
> @@ -27,4 +27,5 @@ zinc_blake2s-$(CONFIG_ZINC_ARCH_X86_64) += blake2s/blake2s-x86_64.o
> obj-$(CONFIG_ZINC_BLAKE2S) += zinc_blake2s.o
>
> zinc_curve25519-y := curve25519/curve25519.o
> +zinc_curve25519-$(CONFIG_ZINC_ARCH_ARM) += curve25519/curve25519-arm.o
> obj-$(CONFIG_ZINC_CURVE25519) += zinc_curve25519.o
> diff --git a/lib/zinc/curve25519/curve25519-arm-glue.h b/lib/zinc/curve25519/curve25519-arm-glue.h
> new file mode 100644
> index 000000000000..9211bcab5615
> --- /dev/null
> +++ b/lib/zinc/curve25519/curve25519-arm-glue.h
> @@ -0,0 +1,42 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
> +/*
> + * Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
> + */
> +
> +#include <asm/hwcap.h>
> +#include <asm/neon.h>
> +#include <asm/simd.h>
> +
> +#if defined(CONFIG_KERNEL_MODE_NEON)
> +asmlinkage void curve25519_neon(u8 mypublic[CURVE25519_KEY_SIZE],
> + const u8 secret[CURVE25519_KEY_SIZE],
> + const u8 basepoint[CURVE25519_KEY_SIZE]);
> +#endif
> +
> +static bool curve25519_use_neon __ro_after_init;
> +
> +static void __init curve25519_fpu_init(void)
> +{
> + curve25519_use_neon = elf_hwcap & HWCAP_NEON;
> +}
> +
> +static inline bool curve25519_arch(u8 mypublic[CURVE25519_KEY_SIZE],
> + const u8 secret[CURVE25519_KEY_SIZE],
> + const u8 basepoint[CURVE25519_KEY_SIZE])
> +{
> +#if defined(CONFIG_KERNEL_MODE_NEON)
> + if (curve25519_use_neon && may_use_simd()) {
> + kernel_neon_begin();
> + curve25519_neon(mypublic, secret, basepoint);
> + kernel_neon_end();
> + return true;
> + }
> +#endif
> + return false;
> +}
> +
> +static inline bool curve25519_base_arch(u8 pub[CURVE25519_KEY_SIZE],
> + const u8 secret[CURVE25519_KEY_SIZE])
> +{
> + return false;
> +}
Shouldn't this use the new simd abstraction as well?
> diff --git a/lib/zinc/curve25519/curve25519-arm.S b/lib/zinc/curve25519/curve25519-arm.S
> new file mode 100644
> index 000000000000..db6570c20fd1
> --- /dev/null
> +++ b/lib/zinc/curve25519/curve25519-arm.S
> @@ -0,0 +1,2095 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR MIT */
> +/*
> + * Copyright (C) 2015-2018 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
> + *
> + * Based on public domain code from Daniel J. Bernstein and Peter Schwabe. This
> + * has been built from SUPERCOP's curve25519/neon2/scalarmult.pq using qhasm,
> + * but has subsequently been manually reworked for use in kernel space.
> + */
> +
> +#ifdef CONFIG_KERNEL_MODE_NEON
> +#include <linux/linkage.h>
> +
> +.text
> +.fpu neon
> +.arch armv7-a
> +.align 4
> +
> +ENTRY(curve25519_neon)
> + push {r4-r11, lr}
> + mov ip, sp
> + sub r3, sp, #704
> + and r3, r3, #0xfffffff0
> + mov sp, r3
> + movw r4, #0
> + movw r5, #254
> + vmov.i32 q0, #1
> + vshr.u64 q1, q0, #7
> + vshr.u64 q0, q0, #8
> + vmov.i32 d4, #19
> + vmov.i32 d5, #38
> + add r6, sp, #480
> + vst1.8 {d2-d3}, [r6, : 128]
> + add r6, sp, #496
> + vst1.8 {d0-d1}, [r6, : 128]
> + add r6, sp, #512
> + vst1.8 {d4-d5}, [r6, : 128]
I guess qhasm means generated code, right?
Because many of these adds are completely redundant ...
> + add r6, r3, #0
> + vmov.i32 q2, #0
> + vst1.8 {d4-d5}, [r6, : 128]!
> + vst1.8 {d4-d5}, [r6, : 128]!
> + vst1.8 d4, [r6, : 64]
> + add r6, r3, #0
> + movw r7, #960
> + sub r7, r7, #2
> + neg r7, r7
> + sub r7, r7, r7, LSL #7
This looks odd as well.
Could you elaborate on what qhasm is exactly? And, as with the other
patches, I would prefer it if we could have your changes as a separate
patch (although having the qhasm base would be preferred)
> + str r7, [r6]
> + add r6, sp, #672
> + vld1.8 {d4-d5}, [r1]!
> + vld1.8 {d6-d7}, [r1]
> + vst1.8 {d4-d5}, [r6, : 128]!
> + vst1.8 {d6-d7}, [r6, : 128]
> + sub r1, r6, #16
> + ldrb r6, [r1]
> + and r6, r6, #248
> + strb r6, [r1]
> + ldrb r6, [r1, #31]
> + and r6, r6, #127
> + orr r6, r6, #64
> + strb r6, [r1, #31]
> + vmov.i64 q2, #0xffffffff
> + vshr.u64 q3, q2, #7
> + vshr.u64 q2, q2, #6
> + vld1.8 {d8}, [r2]
> + vld1.8 {d10}, [r2]
> + add r2, r2, #6
> + vld1.8 {d12}, [r2]
> + vld1.8 {d14}, [r2]
> + add r2, r2, #6
> + vld1.8 {d16}, [r2]
> + add r2, r2, #4
> + vld1.8 {d18}, [r2]
> + vld1.8 {d20}, [r2]
> + add r2, r2, #6
> + vld1.8 {d22}, [r2]
> + add r2, r2, #2
> + vld1.8 {d24}, [r2]
> + vld1.8 {d26}, [r2]
> + vshr.u64 q5, q5, #26
> + vshr.u64 q6, q6, #3
> + vshr.u64 q7, q7, #29
> + vshr.u64 q8, q8, #6
> + vshr.u64 q10, q10, #25
> + vshr.u64 q11, q11, #3
> + vshr.u64 q12, q12, #12
> + vshr.u64 q13, q13, #38
> + vand q4, q4, q2
> + vand q6, q6, q2
> + vand q8, q8, q2
> + vand q10, q10, q2
> + vand q2, q12, q2
> + vand q5, q5, q3
> + vand q7, q7, q3
> + vand q9, q9, q3
> + vand q11, q11, q3
> + vand q3, q13, q3
> + add r2, r3, #48
> + vadd.i64 q12, q4, q1
> + vadd.i64 q13, q10, q1
> + vshr.s64 q12, q12, #26
> + vshr.s64 q13, q13, #26
> + vadd.i64 q5, q5, q12
> + vshl.i64 q12, q12, #26
> + vadd.i64 q14, q5, q0
> + vadd.i64 q11, q11, q13
> + vshl.i64 q13, q13, #26
> + vadd.i64 q15, q11, q0
> + vsub.i64 q4, q4, q12
> + vshr.s64 q12, q14, #25
> + vsub.i64 q10, q10, q13
> + vshr.s64 q13, q15, #25
> + vadd.i64 q6, q6, q12
> + vshl.i64 q12, q12, #25
> + vadd.i64 q14, q6, q1
> + vadd.i64 q2, q2, q13
> + vsub.i64 q5, q5, q12
> + vshr.s64 q12, q14, #26
> + vshl.i64 q13, q13, #25
> + vadd.i64 q14, q2, q1
> + vadd.i64 q7, q7, q12
> + vshl.i64 q12, q12, #26
> + vadd.i64 q15, q7, q0
> + vsub.i64 q11, q11, q13
> + vshr.s64 q13, q14, #26
> + vsub.i64 q6, q6, q12
> + vshr.s64 q12, q15, #25
> + vadd.i64 q3, q3, q13
> + vshl.i64 q13, q13, #26
> + vadd.i64 q14, q3, q0
> + vadd.i64 q8, q8, q12
> + vshl.i64 q12, q12, #25
> + vadd.i64 q15, q8, q1
> + add r2, r2, #8
> + vsub.i64 q2, q2, q13
> + vshr.s64 q13, q14, #25
> + vsub.i64 q7, q7, q12
> + vshr.s64 q12, q15, #26
> + vadd.i64 q14, q13, q13
> + vadd.i64 q9, q9, q12
> + vtrn.32 d12, d14
> + vshl.i64 q12, q12, #26
> + vtrn.32 d13, d15
> + vadd.i64 q0, q9, q0
> + vadd.i64 q4, q4, q14
> + vst1.8 d12, [r2, : 64]!
> + vshl.i64 q6, q13, #4
> + vsub.i64 q7, q8, q12
> + vshr.s64 q0, q0, #25
> + vadd.i64 q4, q4, q6
> + vadd.i64 q6, q10, q0
> + vshl.i64 q0, q0, #25
> + vadd.i64 q8, q6, q1
> + vadd.i64 q4, q4, q13
> + vshl.i64 q10, q13, #25
> + vadd.i64 q1, q4, q1
> + vsub.i64 q0, q9, q0
> + vshr.s64 q8, q8, #26
> + vsub.i64 q3, q3, q10
> + vtrn.32 d14, d0
> + vshr.s64 q1, q1, #26
> + vtrn.32 d15, d1
> + vadd.i64 q0, q11, q8
> + vst1.8 d14, [r2, : 64]
> + vshl.i64 q7, q8, #26
> + vadd.i64 q5, q5, q1
> + vtrn.32 d4, d6
> + vshl.i64 q1, q1, #26
> + vtrn.32 d5, d7
> + vsub.i64 q3, q6, q7
> + add r2, r2, #16
> + vsub.i64 q1, q4, q1
> + vst1.8 d4, [r2, : 64]
> + vtrn.32 d6, d0
> + vtrn.32 d7, d1
> + sub r2, r2, #8
> + vtrn.32 d2, d10
> + vtrn.32 d3, d11
> + vst1.8 d6, [r2, : 64]
> + sub r2, r2, #24
> + vst1.8 d2, [r2, : 64]
> + add r2, r3, #96
> + vmov.i32 q0, #0
> + vmov.i64 d2, #0xff
> + vmov.i64 d3, #0
> + vshr.u32 q1, q1, #7
> + vst1.8 {d2-d3}, [r2, : 128]!
> + vst1.8 {d0-d1}, [r2, : 128]!
> + vst1.8 d0, [r2, : 64]
> + add r2, r3, #144
> + vmov.i32 q0, #0
> + vst1.8 {d0-d1}, [r2, : 128]!
> + vst1.8 {d0-d1}, [r2, : 128]!
> + vst1.8 d0, [r2, : 64]
> + add r2, r3, #240
> + vmov.i32 q0, #0
> + vmov.i64 d2, #0xff
> + vmov.i64 d3, #0
> + vshr.u32 q1, q1, #7
> + vst1.8 {d2-d3}, [r2, : 128]!
> + vst1.8 {d0-d1}, [r2, : 128]!
> + vst1.8 d0, [r2, : 64]
> + add r2, r3, #48
> + add r6, r3, #192
> + vld1.8 {d0-d1}, [r2, : 128]!
> + vld1.8 {d2-d3}, [r2, : 128]!
> + vld1.8 {d4}, [r2, : 64]
> + vst1.8 {d0-d1}, [r6, : 128]!
> + vst1.8 {d2-d3}, [r6, : 128]!
> + vst1.8 d4, [r6, : 64]
> +.Lmainloop:
> + mov r2, r5, LSR #3
> + and r6, r5, #7
> + ldrb r2, [r1, r2]
> + mov r2, r2, LSR r6
> + and r2, r2, #1
> + str r5, [sp, #456]
> + eor r4, r4, r2
> + str r2, [sp, #460]
> + neg r2, r4
> + add r4, r3, #96
> + add r5, r3, #192
> + add r6, r3, #144
> + vld1.8 {d8-d9}, [r4, : 128]!
> + add r7, r3, #240
> + vld1.8 {d10-d11}, [r5, : 128]!
> + veor q6, q4, q5
> + vld1.8 {d14-d15}, [r6, : 128]!
> + vdup.i32 q8, r2
> + vld1.8 {d18-d19}, [r7, : 128]!
> + veor q10, q7, q9
> + vld1.8 {d22-d23}, [r4, : 128]!
> + vand q6, q6, q8
> + vld1.8 {d24-d25}, [r5, : 128]!
> + vand q10, q10, q8
> + vld1.8 {d26-d27}, [r6, : 128]!
> + veor q4, q4, q6
> + vld1.8 {d28-d29}, [r7, : 128]!
> + veor q5, q5, q6
> + vld1.8 {d0}, [r4, : 64]
> + veor q6, q7, q10
> + vld1.8 {d2}, [r5, : 64]
> + veor q7, q9, q10
> + vld1.8 {d4}, [r6, : 64]
> + veor q9, q11, q12
> + vld1.8 {d6}, [r7, : 64]
> + veor q10, q0, q1
> + sub r2, r4, #32
> + vand q9, q9, q8
> + sub r4, r5, #32
> + vand q10, q10, q8
> + sub r5, r6, #32
> + veor q11, q11, q9
> + sub r6, r7, #32
> + veor q0, q0, q10
> + veor q9, q12, q9
> + veor q1, q1, q10
> + veor q10, q13, q14
> + veor q12, q2, q3
> + vand q10, q10, q8
> + vand q8, q12, q8
> + veor q12, q13, q10
> + veor q2, q2, q8
> + veor q10, q14, q10
> + veor q3, q3, q8
> + vadd.i32 q8, q4, q6
> + vsub.i32 q4, q4, q6
> + vst1.8 {d16-d17}, [r2, : 128]!
> + vadd.i32 q6, q11, q12
> + vst1.8 {d8-d9}, [r5, : 128]!
> + vsub.i32 q4, q11, q12
> + vst1.8 {d12-d13}, [r2, : 128]!
> + vadd.i32 q6, q0, q2
> + vst1.8 {d8-d9}, [r5, : 128]!
> + vsub.i32 q0, q0, q2
> + vst1.8 d12, [r2, : 64]
> + vadd.i32 q2, q5, q7
> + vst1.8 d0, [r5, : 64]
> + vsub.i32 q0, q5, q7
> + vst1.8 {d4-d5}, [r4, : 128]!
> + vadd.i32 q2, q9, q10
> + vst1.8 {d0-d1}, [r6, : 128]!
> + vsub.i32 q0, q9, q10
> + vst1.8 {d4-d5}, [r4, : 128]!
> + vadd.i32 q2, q1, q3
> + vst1.8 {d0-d1}, [r6, : 128]!
> + vsub.i32 q0, q1, q3
> + vst1.8 d4, [r4, : 64]
> + vst1.8 d0, [r6, : 64]
> + add r2, sp, #512
> + add r4, r3, #96
> + add r5, r3, #144
> + vld1.8 {d0-d1}, [r2, : 128]
> + vld1.8 {d2-d3}, [r4, : 128]!
> + vld1.8 {d4-d5}, [r5, : 128]!
> + vzip.i32 q1, q2
> + vld1.8 {d6-d7}, [r4, : 128]!
> + vld1.8 {d8-d9}, [r5, : 128]!
> + vshl.i32 q5, q1, #1
> + vzip.i32 q3, q4
> + vshl.i32 q6, q2, #1
> + vld1.8 {d14}, [r4, : 64]
> + vshl.i32 q8, q3, #1
> + vld1.8 {d15}, [r5, : 64]
> + vshl.i32 q9, q4, #1
> + vmul.i32 d21, d7, d1
> + vtrn.32 d14, d15
> + vmul.i32 q11, q4, q0
> + vmul.i32 q0, q7, q0
> + vmull.s32 q12, d2, d2
> + vmlal.s32 q12, d11, d1
> + vmlal.s32 q12, d12, d0
> + vmlal.s32 q12, d13, d23
> + vmlal.s32 q12, d16, d22
> + vmlal.s32 q12, d7, d21
> + vmull.s32 q10, d2, d11
> + vmlal.s32 q10, d4, d1
> + vmlal.s32 q10, d13, d0
> + vmlal.s32 q10, d6, d23
> + vmlal.s32 q10, d17, d22
> + vmull.s32 q13, d10, d4
> + vmlal.s32 q13, d11, d3
> + vmlal.s32 q13, d13, d1
> + vmlal.s32 q13, d16, d0
> + vmlal.s32 q13, d17, d23
> + vmlal.s32 q13, d8, d22
> + vmull.s32 q1, d10, d5
> + vmlal.s32 q1, d11, d4
> + vmlal.s32 q1, d6, d1
> + vmlal.s32 q1, d17, d0
> + vmlal.s32 q1, d8, d23
> + vmull.s32 q14, d10, d6
> + vmlal.s32 q14, d11, d13
> + vmlal.s32 q14, d4, d4
> + vmlal.s32 q14, d17, d1
> + vmlal.s32 q14, d18, d0
> + vmlal.s32 q14, d9, d23
> + vmull.s32 q11, d10, d7
> + vmlal.s32 q11, d11, d6
> + vmlal.s32 q11, d12, d5
> + vmlal.s32 q11, d8, d1
> + vmlal.s32 q11, d19, d0
> + vmull.s32 q15, d10, d8
> + vmlal.s32 q15, d11, d17
> + vmlal.s32 q15, d12, d6
> + vmlal.s32 q15, d13, d5
> + vmlal.s32 q15, d19, d1
> + vmlal.s32 q15, d14, d0
> + vmull.s32 q2, d10, d9
> + vmlal.s32 q2, d11, d8
> + vmlal.s32 q2, d12, d7
> + vmlal.s32 q2, d13, d6
> + vmlal.s32 q2, d14, d1
> + vmull.s32 q0, d15, d1
> + vmlal.s32 q0, d10, d14
> + vmlal.s32 q0, d11, d19
> + vmlal.s32 q0, d12, d8
> + vmlal.s32 q0, d13, d17
> + vmlal.s32 q0, d6, d6
> + add r2, sp, #480
> + vld1.8 {d18-d19}, [r2, : 128]
If you append a ! here ...
> + vmull.s32 q3, d16, d7
> + vmlal.s32 q3, d10, d15
> + vmlal.s32 q3, d11, d14
> + vmlal.s32 q3, d12, d9
> + vmlal.s32 q3, d13, d8
> + add r2, sp, #496
... you can drop this add
> + vld1.8 {d8-d9}, [r2, : 128]
> + vadd.i64 q5, q12, q9
> + vadd.i64 q6, q15, q9
> + vshr.s64 q5, q5, #26
> + vshr.s64 q6, q6, #26
> + vadd.i64 q7, q10, q5
> + vshl.i64 q5, q5, #26
> + vadd.i64 q8, q7, q4
> + vadd.i64 q2, q2, q6
> + vshl.i64 q6, q6, #26
> + vadd.i64 q10, q2, q4
> + vsub.i64 q5, q12, q5
> + vshr.s64 q8, q8, #25
> + vsub.i64 q6, q15, q6
> + vshr.s64 q10, q10, #25
> + vadd.i64 q12, q13, q8
> + vshl.i64 q8, q8, #25
> + vadd.i64 q13, q12, q9
> + vadd.i64 q0, q0, q10
> + vsub.i64 q7, q7, q8
> + vshr.s64 q8, q13, #26
> + vshl.i64 q10, q10, #25
> + vadd.i64 q13, q0, q9
> + vadd.i64 q1, q1, q8
> + vshl.i64 q8, q8, #26
> + vadd.i64 q15, q1, q4
> + vsub.i64 q2, q2, q10
> + vshr.s64 q10, q13, #26
> + vsub.i64 q8, q12, q8
> + vshr.s64 q12, q15, #25
> + vadd.i64 q3, q3, q10
> + vshl.i64 q10, q10, #26
> + vadd.i64 q13, q3, q4
> + vadd.i64 q14, q14, q12
> + add r2, r3, #288
> + vshl.i64 q12, q12, #25
> + add r4, r3, #336
> + vadd.i64 q15, q14, q9
> + add r2, r2, #8
> + vsub.i64 q0, q0, q10
> + add r4, r4, #8
> + vshr.s64 q10, q13, #25
> + vsub.i64 q1, q1, q12
> + vshr.s64 q12, q15, #26
> + vadd.i64 q13, q10, q10
> + vadd.i64 q11, q11, q12
> + vtrn.32 d16, d2
> + vshl.i64 q12, q12, #26
> + vtrn.32 d17, d3
> + vadd.i64 q1, q11, q4
> + vadd.i64 q4, q5, q13
> + vst1.8 d16, [r2, : 64]!
> + vshl.i64 q5, q10, #4
> + vst1.8 d17, [r4, : 64]!
> + vsub.i64 q8, q14, q12
> + vshr.s64 q1, q1, #25
> + vadd.i64 q4, q4, q5
> + vadd.i64 q5, q6, q1
> + vshl.i64 q1, q1, #25
> + vadd.i64 q6, q5, q9
> + vadd.i64 q4, q4, q10
> + vshl.i64 q10, q10, #25
> + vadd.i64 q9, q4, q9
> + vsub.i64 q1, q11, q1
> + vshr.s64 q6, q6, #26
> + vsub.i64 q3, q3, q10
> + vtrn.32 d16, d2
> + vshr.s64 q9, q9, #26
> + vtrn.32 d17, d3
> + vadd.i64 q1, q2, q6
> + vst1.8 d16, [r2, : 64]
> + vshl.i64 q2, q6, #26
> + vst1.8 d17, [r4, : 64]
> + vadd.i64 q6, q7, q9
> + vtrn.32 d0, d6
> + vshl.i64 q7, q9, #26
> + vtrn.32 d1, d7
> + vsub.i64 q2, q5, q2
> + add r2, r2, #16
> + vsub.i64 q3, q4, q7
> + vst1.8 d0, [r2, : 64]
> + add r4, r4, #16
> + vst1.8 d1, [r4, : 64]
> + vtrn.32 d4, d2
> + vtrn.32 d5, d3
> + sub r2, r2, #8
> + sub r4, r4, #8
> + vtrn.32 d6, d12
> + vtrn.32 d7, d13
> + vst1.8 d4, [r2, : 64]
> + vst1.8 d5, [r4, : 64]
> + sub r2, r2, #24
> + sub r4, r4, #24
> + vst1.8 d6, [r2, : 64]
> + vst1.8 d7, [r4, : 64]
> + add r2, r3, #240
> + add r4, r3, #96
> + vld1.8 {d0-d1}, [r4, : 128]!
> + vld1.8 {d2-d3}, [r4, : 128]!
> + vld1.8 {d4}, [r4, : 64]
> + add r4, r3, #144
> + vld1.8 {d6-d7}, [r4, : 128]!
> + vtrn.32 q0, q3
> + vld1.8 {d8-d9}, [r4, : 128]!
> + vshl.i32 q5, q0, #4
> + vtrn.32 q1, q4
> + vshl.i32 q6, q3, #4
> + vadd.i32 q5, q5, q0
> + vadd.i32 q6, q6, q3
> + vshl.i32 q7, q1, #4
> + vld1.8 {d5}, [r4, : 64]
> + vshl.i32 q8, q4, #4
> + vtrn.32 d4, d5
> + vadd.i32 q7, q7, q1
> + vadd.i32 q8, q8, q4
> + vld1.8 {d18-d19}, [r2, : 128]!
> + vshl.i32 q10, q2, #4
> + vld1.8 {d22-d23}, [r2, : 128]!
> + vadd.i32 q10, q10, q2
> + vld1.8 {d24}, [r2, : 64]
> + vadd.i32 q5, q5, q0
> + add r2, r3, #192
> + vld1.8 {d26-d27}, [r2, : 128]!
> + vadd.i32 q6, q6, q3
> + vld1.8 {d28-d29}, [r2, : 128]!
> + vadd.i32 q8, q8, q4
> + vld1.8 {d25}, [r2, : 64]
> + vadd.i32 q10, q10, q2
> + vtrn.32 q9, q13
> + vadd.i32 q7, q7, q1
> + vadd.i32 q5, q5, q0
> + vtrn.32 q11, q14
> + vadd.i32 q6, q6, q3
> + add r2, sp, #528
> + vadd.i32 q10, q10, q2
> + vtrn.32 d24, d25
> + vst1.8 {d12-d13}, [r2, : 128]
same here
> + vshl.i32 q6, q13, #1
> + add r2, sp, #544
> + vst1.8 {d20-d21}, [r2, : 128]
and here
> + vshl.i32 q10, q14, #1
> + add r2, sp, #560
> + vst1.8 {d12-d13}, [r2, : 128]
and here
> + vshl.i32 q15, q12, #1
> + vadd.i32 q8, q8, q4
> + vext.32 d10, d31, d30, #0
> + vadd.i32 q7, q7, q1
> + add r2, sp, #576
> + vst1.8 {d16-d17}, [r2, : 128]
and here
> + vmull.s32 q8, d18, d5
> + vmlal.s32 q8, d26, d4
> + vmlal.s32 q8, d19, d9
> + vmlal.s32 q8, d27, d3
> + vmlal.s32 q8, d22, d8
> + vmlal.s32 q8, d28, d2
> + vmlal.s32 q8, d23, d7
> + vmlal.s32 q8, d29, d1
> + vmlal.s32 q8, d24, d6
> + vmlal.s32 q8, d25, d0
> + add r2, sp, #592
> + vst1.8 {d14-d15}, [r2, : 128]
and here
> + vmull.s32 q2, d18, d4
> + vmlal.s32 q2, d12, d9
> + vmlal.s32 q2, d13, d8
> + vmlal.s32 q2, d19, d3
> + vmlal.s32 q2, d22, d2
> + vmlal.s32 q2, d23, d1
> + vmlal.s32 q2, d24, d0
> + add r2, sp, #608
> + vst1.8 {d20-d21}, [r2, : 128]
and here
> + vmull.s32 q7, d18, d9
> + vmlal.s32 q7, d26, d3
> + vmlal.s32 q7, d19, d8
> + vmlal.s32 q7, d27, d2
> + vmlal.s32 q7, d22, d7
> + vmlal.s32 q7, d28, d1
> + vmlal.s32 q7, d23, d6
> + vmlal.s32 q7, d29, d0
> + add r2, sp, #624
> + vst1.8 {d10-d11}, [r2, : 128]
and here
> + vmull.s32 q5, d18, d3
> + vmlal.s32 q5, d19, d2
> + vmlal.s32 q5, d22, d1
> + vmlal.s32 q5, d23, d0
> + vmlal.s32 q5, d12, d8
> + add r2, sp, #640
> + vst1.8 {d16-d17}, [r2, : 128]
> + vmull.s32 q4, d18, d8
> + vmlal.s32 q4, d26, d2
> + vmlal.s32 q4, d19, d7
> + vmlal.s32 q4, d27, d1
> + vmlal.s32 q4, d22, d6
> + vmlal.s32 q4, d28, d0
> + vmull.s32 q8, d18, d7
> + vmlal.s32 q8, d26, d1
> + vmlal.s32 q8, d19, d6
> + vmlal.s32 q8, d27, d0
> + add r2, sp, #544
> + vld1.8 {d20-d21}, [r2, : 128]
> + vmlal.s32 q7, d24, d21
> + vmlal.s32 q7, d25, d20
> + vmlal.s32 q4, d23, d21
> + vmlal.s32 q4, d29, d20
> + vmlal.s32 q8, d22, d21
> + vmlal.s32 q8, d28, d20
> + vmlal.s32 q5, d24, d20
> + add r2, sp, #544
redundant add
I'll stop here - let me just note that this code does not strike me as
particularly well optimized for in-order cores (such as A7).
For instance, the sequence
vmlal.s32 q2, d18, d7
vmlal.s32 q2, d19, d6
vmlal.s32 q5, d18, d6
vmlal.s32 q5, d19, d21
vmlal.s32 q1, d18, d21
vmlal.s32 q1, d19, d29
vmlal.s32 q0, d18, d28
vmlal.s32 q0, d19, d9
vmlal.s32 q6, d18, d29
vmlal.s32 q6, d19, d28
can be reordered as
vmlal.s32 q2, d18, d7
vmlal.s32 q5, d18, d6
vmlal.s32 q1, d18, d21
vmlal.s32 q0, d18, d28
vmlal.s32 q6, d18, d29
vmlal.s32 q2, d19, d6
vmlal.s32 q5, d19, d21
vmlal.s32 q1, d19, d29
vmlal.s32 q0, d19, d9
vmlal.s32 q6, d19, d28
and not have every other instruction depend on the output of the previous one.
Obviously, the ultimate truth is in the benchmark numbers, but I'd
thought I'd mention it anyway.
> + vst1.8 {d14-d15}, [r2, : 128]
> + vmull.s32 q7, d18, d6
> + vmlal.s32 q7, d26, d0
> + add r2, sp, #624
> + vld1.8 {d30-d31}, [r2, : 128]
> + vmlal.s32 q2, d30, d21
> + vmlal.s32 q7, d19, d21
> + vmlal.s32 q7, d27, d20
> + add r2, sp, #592
> + vld1.8 {d26-d27}, [r2, : 128]
> + vmlal.s32 q4, d25, d27
> + vmlal.s32 q8, d29, d27
> + vmlal.s32 q8, d25, d26
> + vmlal.s32 q7, d28, d27
> + vmlal.s32 q7, d29, d26
> + add r2, sp, #576
> + vld1.8 {d28-d29}, [r2, : 128]
> + vmlal.s32 q4, d24, d29
> + vmlal.s32 q8, d23, d29
> + vmlal.s32 q8, d24, d28
> + vmlal.s32 q7, d22, d29
> + vmlal.s32 q7, d23, d28
> + add r2, sp, #576
> + vst1.8 {d8-d9}, [r2, : 128]
> + add r2, sp, #528
> + vld1.8 {d8-d9}, [r2, : 128]
> + vmlal.s32 q7, d24, d9
> + vmlal.s32 q7, d25, d31
> + vmull.s32 q1, d18, d2
> + vmlal.s32 q1, d19, d1
> + vmlal.s32 q1, d22, d0
> + vmlal.s32 q1, d24, d27
> + vmlal.s32 q1, d23, d20
> + vmlal.s32 q1, d12, d7
> + vmlal.s32 q1, d13, d6
> + vmull.s32 q6, d18, d1
> + vmlal.s32 q6, d19, d0
> + vmlal.s32 q6, d23, d27
> + vmlal.s32 q6, d22, d20
> + vmlal.s32 q6, d24, d26
> + vmull.s32 q0, d18, d0
> + vmlal.s32 q0, d22, d27
> + vmlal.s32 q0, d23, d26
> + vmlal.s32 q0, d24, d31
> + vmlal.s32 q0, d19, d20
> + add r2, sp, #608
> + vld1.8 {d18-d19}, [r2, : 128]
> + vmlal.s32 q2, d18, d7
> + vmlal.s32 q2, d19, d6
> + vmlal.s32 q5, d18, d6
> + vmlal.s32 q5, d19, d21
> + vmlal.s32 q1, d18, d21
> + vmlal.s32 q1, d19, d29
> + vmlal.s32 q0, d18, d28
> + vmlal.s32 q0, d19, d9
> + vmlal.s32 q6, d18, d29
> + vmlal.s32 q6, d19, d28
> + add r2, sp, #560
> + vld1.8 {d18-d19}, [r2, : 128]
> + add r2, sp, #480
> + vld1.8 {d22-d23}, [r2, : 128]
> + vmlal.s32 q5, d19, d7
> + vmlal.s32 q0, d18, d21
> + vmlal.s32 q0, d19, d29
> + vmlal.s32 q6, d18, d6
> + add r2, sp, #496
> + vld1.8 {d6-d7}, [r2, : 128]
> + vmlal.s32 q6, d19, d21
> + add r2, sp, #544
> + vld1.8 {d18-d19}, [r2, : 128]
> + vmlal.s32 q0, d30, d8
> + add r2, sp, #640
> + vld1.8 {d20-d21}, [r2, : 128]
> + vmlal.s32 q5, d30, d29
> + add r2, sp, #576
> + vld1.8 {d24-d25}, [r2, : 128]
> + vmlal.s32 q1, d30, d28
> + vadd.i64 q13, q0, q11
> + vadd.i64 q14, q5, q11
> + vmlal.s32 q6, d30, d9
> + vshr.s64 q4, q13, #26
> + vshr.s64 q13, q14, #26
> + vadd.i64 q7, q7, q4
> + vshl.i64 q4, q4, #26
> + vadd.i64 q14, q7, q3
> + vadd.i64 q9, q9, q13
> + vshl.i64 q13, q13, #26
> + vadd.i64 q15, q9, q3
> + vsub.i64 q0, q0, q4
> + vshr.s64 q4, q14, #25
> + vsub.i64 q5, q5, q13
> + vshr.s64 q13, q15, #25
> + vadd.i64 q6, q6, q4
> + vshl.i64 q4, q4, #25
> + vadd.i64 q14, q6, q11
> + vadd.i64 q2, q2, q13
> + vsub.i64 q4, q7, q4
> + vshr.s64 q7, q14, #26
> + vshl.i64 q13, q13, #25
> + vadd.i64 q14, q2, q11
> + vadd.i64 q8, q8, q7
> + vshl.i64 q7, q7, #26
> + vadd.i64 q15, q8, q3
> + vsub.i64 q9, q9, q13
> + vshr.s64 q13, q14, #26
> + vsub.i64 q6, q6, q7
> + vshr.s64 q7, q15, #25
> + vadd.i64 q10, q10, q13
> + vshl.i64 q13, q13, #26
> + vadd.i64 q14, q10, q3
> + vadd.i64 q1, q1, q7
> + add r2, r3, #144
> + vshl.i64 q7, q7, #25
> + add r4, r3, #96
> + vadd.i64 q15, q1, q11
> + add r2, r2, #8
> + vsub.i64 q2, q2, q13
> + add r4, r4, #8
> + vshr.s64 q13, q14, #25
> + vsub.i64 q7, q8, q7
> + vshr.s64 q8, q15, #26
> + vadd.i64 q14, q13, q13
> + vadd.i64 q12, q12, q8
> + vtrn.32 d12, d14
> + vshl.i64 q8, q8, #26
> + vtrn.32 d13, d15
> + vadd.i64 q3, q12, q3
> + vadd.i64 q0, q0, q14
> + vst1.8 d12, [r2, : 64]!
> + vshl.i64 q7, q13, #4
> + vst1.8 d13, [r4, : 64]!
> + vsub.i64 q1, q1, q8
> + vshr.s64 q3, q3, #25
> + vadd.i64 q0, q0, q7
> + vadd.i64 q5, q5, q3
> + vshl.i64 q3, q3, #25
> + vadd.i64 q6, q5, q11
> + vadd.i64 q0, q0, q13
> + vshl.i64 q7, q13, #25
> + vadd.i64 q8, q0, q11
> + vsub.i64 q3, q12, q3
> + vshr.s64 q6, q6, #26
> + vsub.i64 q7, q10, q7
> + vtrn.32 d2, d6
> + vshr.s64 q8, q8, #26
> + vtrn.32 d3, d7
> + vadd.i64 q3, q9, q6
> + vst1.8 d2, [r2, : 64]
> + vshl.i64 q6, q6, #26
> + vst1.8 d3, [r4, : 64]
> + vadd.i64 q1, q4, q8
> + vtrn.32 d4, d14
> + vshl.i64 q4, q8, #26
> + vtrn.32 d5, d15
> + vsub.i64 q5, q5, q6
> + add r2, r2, #16
> + vsub.i64 q0, q0, q4
> + vst1.8 d4, [r2, : 64]
> + add r4, r4, #16
> + vst1.8 d5, [r4, : 64]
> + vtrn.32 d10, d6
> + vtrn.32 d11, d7
> + sub r2, r2, #8
> + sub r4, r4, #8
> + vtrn.32 d0, d2
> + vtrn.32 d1, d3
> + vst1.8 d10, [r2, : 64]
> + vst1.8 d11, [r4, : 64]
> + sub r2, r2, #24
> + sub r4, r4, #24
> + vst1.8 d0, [r2, : 64]
> + vst1.8 d1, [r4, : 64]
> + add r2, r3, #288
> + add r4, r3, #336
> + vld1.8 {d0-d1}, [r2, : 128]!
> + vld1.8 {d2-d3}, [r4, : 128]!
> + vsub.i32 q0, q0, q1
> + vld1.8 {d2-d3}, [r2, : 128]!
> + vld1.8 {d4-d5}, [r4, : 128]!
> + vsub.i32 q1, q1, q2
> + add r5, r3, #240
> + vld1.8 {d4}, [r2, : 64]
> + vld1.8 {d6}, [r4, : 64]
> + vsub.i32 q2, q2, q3
> + vst1.8 {d0-d1}, [r5, : 128]!
> + vst1.8 {d2-d3}, [r5, : 128]!
> + vst1.8 d4, [r5, : 64]
> + add r2, r3, #144
> + add r4, r3, #96
> + add r5, r3, #144
> + add r6, r3, #192
> + vld1.8 {d0-d1}, [r2, : 128]!
> + vld1.8 {d2-d3}, [r4, : 128]!
> + vsub.i32 q2, q0, q1
> + vadd.i32 q0, q0, q1
> + vld1.8 {d2-d3}, [r2, : 128]!
> + vld1.8 {d6-d7}, [r4, : 128]!
> + vsub.i32 q4, q1, q3
> + vadd.i32 q1, q1, q3
> + vld1.8 {d6}, [r2, : 64]
> + vld1.8 {d10}, [r4, : 64]
> + vsub.i32 q6, q3, q5
> + vadd.i32 q3, q3, q5
> + vst1.8 {d4-d5}, [r5, : 128]!
> + vst1.8 {d0-d1}, [r6, : 128]!
> + vst1.8 {d8-d9}, [r5, : 128]!
> + vst1.8 {d2-d3}, [r6, : 128]!
> + vst1.8 d12, [r5, : 64]
> + vst1.8 d6, [r6, : 64]
> + add r2, r3, #0
> + add r4, r3, #240
> + vld1.8 {d0-d1}, [r4, : 128]!
> + vld1.8 {d2-d3}, [r4, : 128]!
> + vld1.8 {d4}, [r4, : 64]
> + add r4, r3, #336
> + vld1.8 {d6-d7}, [r4, : 128]!
> + vtrn.32 q0, q3
> + vld1.8 {d8-d9}, [r4, : 128]!
> + vshl.i32 q5, q0, #4
> + vtrn.32 q1, q4
> + vshl.i32 q6, q3, #4
> + vadd.i32 q5, q5, q0
> + vadd.i32 q6, q6, q3
> + vshl.i32 q7, q1, #4
> + vld1.8 {d5}, [r4, : 64]
> + vshl.i32 q8, q4, #4
> + vtrn.32 d4, d5
> + vadd.i32 q7, q7, q1
> + vadd.i32 q8, q8, q4
> + vld1.8 {d18-d19}, [r2, : 128]!
> + vshl.i32 q10, q2, #4
> + vld1.8 {d22-d23}, [r2, : 128]!
> + vadd.i32 q10, q10, q2
> + vld1.8 {d24}, [r2, : 64]
> + vadd.i32 q5, q5, q0
> + add r2, r3, #288
> + vld1.8 {d26-d27}, [r2, : 128]!
> + vadd.i32 q6, q6, q3
> + vld1.8 {d28-d29}, [r2, : 128]!
> + vadd.i32 q8, q8, q4
> + vld1.8 {d25}, [r2, : 64]
> + vadd.i32 q10, q10, q2
> + vtrn.32 q9, q13
> + vadd.i32 q7, q7, q1
> + vadd.i32 q5, q5, q0
> + vtrn.32 q11, q14
> + vadd.i32 q6, q6, q3
> + add r2, sp, #528
> + vadd.i32 q10, q10, q2
> + vtrn.32 d24, d25
> + vst1.8 {d12-d13}, [r2, : 128]
> + vshl.i32 q6, q13, #1
> + add r2, sp, #544
> + vst1.8 {d20-d21}, [r2, : 128]
> + vshl.i32 q10, q14, #1
> + add r2, sp, #560
> + vst1.8 {d12-d13}, [r2, : 128]
> + vshl.i32 q15, q12, #1
> + vadd.i32 q8, q8, q4
> + vext.32 d10, d31, d30, #0
> + vadd.i32 q7, q7, q1
> + add r2, sp, #576
> + vst1.8 {d16-d17}, [r2, : 128]
> + vmull.s32 q8, d18, d5
> + vmlal.s32 q8, d26, d4
> + vmlal.s32 q8, d19, d9
> + vmlal.s32 q8, d27, d3
> + vmlal.s32 q8, d22, d8
> + vmlal.s32 q8, d28, d2
> + vmlal.s32 q8, d23, d7
> + vmlal.s32 q8, d29, d1
> + vmlal.s32 q8, d24, d6
> + vmlal.s32 q8, d25, d0
> + add r2, sp, #592
> + vst1.8 {d14-d15}, [r2, : 128]
> + vmull.s32 q2, d18, d4
> + vmlal.s32 q2, d12, d9
> + vmlal.s32 q2, d13, d8
> + vmlal.s32 q2, d19, d3
> + vmlal.s32 q2, d22, d2
> + vmlal.s32 q2, d23, d1
> + vmlal.s32 q2, d24, d0
> + add r2, sp, #608
> + vst1.8 {d20-d21}, [r2, : 128]
> + vmull.s32 q7, d18, d9
> + vmlal.s32 q7, d26, d3
> + vmlal.s32 q7, d19, d8
> + vmlal.s32 q7, d27, d2
> + vmlal.s32 q7, d22, d7
> + vmlal.s32 q7, d28, d1
> + vmlal.s32 q7, d23, d6
> + vmlal.s32 q7, d29, d0
> + add r2, sp, #624
> + vst1.8 {d10-d11}, [r2, : 128]
> + vmull.s32 q5, d18, d3
> + vmlal.s32 q5, d19, d2
> + vmlal.s32 q5, d22, d1
> + vmlal.s32 q5, d23, d0
> + vmlal.s32 q5, d12, d8
> + add r2, sp, #640
> + vst1.8 {d16-d17}, [r2, : 128]
> + vmull.s32 q4, d18, d8
> + vmlal.s32 q4, d26, d2
> + vmlal.s32 q4, d19, d7
> + vmlal.s32 q4, d27, d1
> + vmlal.s32 q4, d22, d6
> + vmlal.s32 q4, d28, d0
> + vmull.s32 q8, d18, d7
> + vmlal.s32 q8, d26, d1
> + vmlal.s32 q8, d19, d6
> + vmlal.s32 q8, d27, d0
> + add r2, sp, #544
> + vld1.8 {d20-d21}, [r2, : 128]
> + vmlal.s32 q7, d24, d21
> + vmlal.s32 q7, d25, d20
> + vmlal.s32 q4, d23, d21
> + vmlal.s32 q4, d29, d20
> + vmlal.s32 q8, d22, d21
> + vmlal.s32 q8, d28, d20
> + vmlal.s32 q5, d24, d20
> + add r2, sp, #544
> + vst1.8 {d14-d15}, [r2, : 128]
> + vmull.s32 q7, d18, d6
> + vmlal.s32 q7, d26, d0
> + add r2, sp, #624
> + vld1.8 {d30-d31}, [r2, : 128]
> + vmlal.s32 q2, d30, d21
> + vmlal.s32 q7, d19, d21
> + vmlal.s32 q7, d27, d20
> + add r2, sp, #592
> + vld1.8 {d26-d27}, [r2, : 128]
> + vmlal.s32 q4, d25, d27
> + vmlal.s32 q8, d29, d27
> + vmlal.s32 q8, d25, d26
> + vmlal.s32 q7, d28, d27
> + vmlal.s32 q7, d29, d26
> + add r2, sp, #576
> + vld1.8 {d28-d29}, [r2, : 128]
> + vmlal.s32 q4, d24, d29
> + vmlal.s32 q8, d23, d29
> + vmlal.s32 q8, d24, d28
> + vmlal.s32 q7, d22, d29
> + vmlal.s32 q7, d23, d28
> + add r2, sp, #576
> + vst1.8 {d8-d9}, [r2, : 128]
> + add r2, sp, #528
> + vld1.8 {d8-d9}, [r2, : 128]
> + vmlal.s32 q7, d24, d9
> + vmlal.s32 q7, d25, d31
> + vmull.s32 q1, d18, d2
> + vmlal.s32 q1, d19, d1
> + vmlal.s32 q1, d22, d0
> + vmlal.s32 q1, d24, d27
> + vmlal.s32 q1, d23, d20
> + vmlal.s32 q1, d12, d7
> + vmlal.s32 q1, d13, d6
> + vmull.s32 q6, d18, d1
> + vmlal.s32 q6, d19, d0
> + vmlal.s32 q6, d23, d27
> + vmlal.s32 q6, d22, d20
> + vmlal.s32 q6, d24, d26
> + vmull.s32 q0, d18, d0
> + vmlal.s32 q0, d22, d27
> + vmlal.s32 q0, d23, d26
> + vmlal.s32 q0, d24, d31
> + vmlal.s32 q0, d19, d20
> + add r2, sp, #608
> + vld1.8 {d18-d19}, [r2, : 128]
> + vmlal.s32 q2, d18, d7
> + vmlal.s32 q2, d19, d6
> + vmlal.s32 q5, d18, d6
> + vmlal.s32 q5, d19, d21
> + vmlal.s32 q1, d18, d21
> + vmlal.s32 q1, d19, d29
> + vmlal.s32 q0, d18, d28
> + vmlal.s32 q0, d19, d9
> + vmlal.s32 q6, d18, d29
> + vmlal.s32 q6, d19, d28
> + add r2, sp, #560
> + vld1.8 {d18-d19}, [r2, : 128]
> + add r2, sp, #480
> + vld1.8 {d22-d23}, [r2, : 128]
> + vmlal.s32 q5, d19, d7
> + vmlal.s32 q0, d18, d21
> + vmlal.s32 q0, d19, d29
> + vmlal.s32 q6, d18, d6
> + add r2, sp, #496
> + vld1.8 {d6-d7}, [r2, : 128]
> + vmlal.s32 q6, d19, d21
> + add r2, sp, #544
> + vld1.8 {d18-d19}, [r2, : 128]
> + vmlal.s32 q0, d30, d8
> + add r2, sp, #640
> + vld1.8 {d20-d21}, [r2, : 128]
> + vmlal.s32 q5, d30, d29
> + add r2, sp, #576
> + vld1.8 {d24-d25}, [r2, : 128]
> + vmlal.s32 q1, d30, d28
> + vadd.i64 q13, q0, q11
> + vadd.i64 q14, q5, q11
> + vmlal.s32 q6, d30, d9
> + vshr.s64 q4, q13, #26
> + vshr.s64 q13, q14, #26
> + vadd.i64 q7, q7, q4
> + vshl.i64 q4, q4, #26
> + vadd.i64 q14, q7, q3
> + vadd.i64 q9, q9, q13
> + vshl.i64 q13, q13, #26
> + vadd.i64 q15, q9, q3
> + vsub.i64 q0, q0, q4
> + vshr.s64 q4, q14, #25
> + vsub.i64 q5, q5, q13
> + vshr.s64 q13, q15, #25
> + vadd.i64 q6, q6, q4
> + vshl.i64 q4, q4, #25
> + vadd.i64 q14, q6, q11
> + vadd.i64 q2, q2, q13
> + vsub.i64 q4, q7, q4
> + vshr.s64 q7, q14, #26
> + vshl.i64 q13, q13, #25
> + vadd.i64 q14, q2, q11
> + vadd.i64 q8, q8, q7
> + vshl.i64 q7, q7, #26
> + vadd.i64 q15, q8, q3
> + vsub.i64 q9, q9, q13
> + vshr.s64 q13, q14, #26
> + vsub.i64 q6, q6, q7
> + vshr.s64 q7, q15, #25
> + vadd.i64 q10, q10, q13
> + vshl.i64 q13, q13, #26
> + vadd.i64 q14, q10, q3
> + vadd.i64 q1, q1, q7
> + add r2, r3, #288
> + vshl.i64 q7, q7, #25
> + add r4, r3, #96
> + vadd.i64 q15, q1, q11
> + add r2, r2, #8
> + vsub.i64 q2, q2, q13
> + add r4, r4, #8
> + vshr.s64 q13, q14, #25
> + vsub.i64 q7, q8, q7
> + vshr.s64 q8, q15, #26
> + vadd.i64 q14, q13, q13
> + vadd.i64 q12, q12, q8
> + vtrn.32 d12, d14
> + vshl.i64 q8, q8, #26
> + vtrn.32 d13, d15
> + vadd.i64 q3, q12, q3
> + vadd.i64 q0, q0, q14
> + vst1.8 d12, [r2, : 64]!
> + vshl.i64 q7, q13, #4
> + vst1.8 d13, [r4, : 64]!
> + vsub.i64 q1, q1, q8
> + vshr.s64 q3, q3, #25
> + vadd.i64 q0, q0, q7
> + vadd.i64 q5, q5, q3
> + vshl.i64 q3, q3, #25
> + vadd.i64 q6, q5, q11
> + vadd.i64 q0, q0, q13
> + vshl.i64 q7, q13, #25
> + vadd.i64 q8, q0, q11
> + vsub.i64 q3, q12, q3
> + vshr.s64 q6, q6, #26
> + vsub.i64 q7, q10, q7
> + vtrn.32 d2, d6
> + vshr.s64 q8, q8, #26
> + vtrn.32 d3, d7
> + vadd.i64 q3, q9, q6
> + vst1.8 d2, [r2, : 64]
> + vshl.i64 q6, q6, #26
> + vst1.8 d3, [r4, : 64]
> + vadd.i64 q1, q4, q8
> + vtrn.32 d4, d14
> + vshl.i64 q4, q8, #26
> + vtrn.32 d5, d15
> + vsub.i64 q5, q5, q6
> + add r2, r2, #16
> + vsub.i64 q0, q0, q4
> + vst1.8 d4, [r2, : 64]
> + add r4, r4, #16
> + vst1.8 d5, [r4, : 64]
> + vtrn.32 d10, d6
> + vtrn.32 d11, d7
> + sub r2, r2, #8
> + sub r4, r4, #8
> + vtrn.32 d0, d2
> + vtrn.32 d1, d3
> + vst1.8 d10, [r2, : 64]
> + vst1.8 d11, [r4, : 64]
> + sub r2, r2, #24
> + sub r4, r4, #24
> + vst1.8 d0, [r2, : 64]
> + vst1.8 d1, [r4, : 64]
> + add r2, sp, #512
> + add r4, r3, #144
> + add r5, r3, #192
> + vld1.8 {d0-d1}, [r2, : 128]
> + vld1.8 {d2-d3}, [r4, : 128]!
> + vld1.8 {d4-d5}, [r5, : 128]!
> + vzip.i32 q1, q2
> + vld1.8 {d6-d7}, [r4, : 128]!
> + vld1.8 {d8-d9}, [r5, : 128]!
> + vshl.i32 q5, q1, #1
> + vzip.i32 q3, q4
> + vshl.i32 q6, q2, #1
> + vld1.8 {d14}, [r4, : 64]
> + vshl.i32 q8, q3, #1
> + vld1.8 {d15}, [r5, : 64]
> + vshl.i32 q9, q4, #1
> + vmul.i32 d21, d7, d1
> + vtrn.32 d14, d15
> + vmul.i32 q11, q4, q0
> + vmul.i32 q0, q7, q0
> + vmull.s32 q12, d2, d2
> + vmlal.s32 q12, d11, d1
> + vmlal.s32 q12, d12, d0
> + vmlal.s32 q12, d13, d23
> + vmlal.s32 q12, d16, d22
> + vmlal.s32 q12, d7, d21
> + vmull.s32 q10, d2, d11
> + vmlal.s32 q10, d4, d1
> + vmlal.s32 q10, d13, d0
> + vmlal.s32 q10, d6, d23
> + vmlal.s32 q10, d17, d22
> + vmull.s32 q13, d10, d4
> + vmlal.s32 q13, d11, d3
> + vmlal.s32 q13, d13, d1
> + vmlal.s32 q13, d16, d0
> + vmlal.s32 q13, d17, d23
> + vmlal.s32 q13, d8, d22
> + vmull.s32 q1, d10, d5
> + vmlal.s32 q1, d11, d4
> + vmlal.s32 q1, d6, d1
> + vmlal.s32 q1, d17, d0
> + vmlal.s32 q1, d8, d23
> + vmull.s32 q14, d10, d6
> + vmlal.s32 q14, d11, d13
> + vmlal.s32 q14, d4, d4
> + vmlal.s32 q14, d17, d1
> + vmlal.s32 q14, d18, d0
> + vmlal.s32 q14, d9, d23
> + vmull.s32 q11, d10, d7
> + vmlal.s32 q11, d11, d6
> + vmlal.s32 q11, d12, d5
> + vmlal.s32 q11, d8, d1
> + vmlal.s32 q11, d19, d0
> + vmull.s32 q15, d10, d8
> + vmlal.s32 q15, d11, d17
> + vmlal.s32 q15, d12, d6
> + vmlal.s32 q15, d13, d5
> + vmlal.s32 q15, d19, d1
> + vmlal.s32 q15, d14, d0
> + vmull.s32 q2, d10, d9
> + vmlal.s32 q2, d11, d8
> + vmlal.s32 q2, d12, d7
> + vmlal.s32 q2, d13, d6
> + vmlal.s32 q2, d14, d1
> + vmull.s32 q0, d15, d1
> + vmlal.s32 q0, d10, d14
> + vmlal.s32 q0, d11, d19
> + vmlal.s32 q0, d12, d8
> + vmlal.s32 q0, d13, d17
> + vmlal.s32 q0, d6, d6
> + add r2, sp, #480
> + vld1.8 {d18-d19}, [r2, : 128]
> + vmull.s32 q3, d16, d7
> + vmlal.s32 q3, d10, d15
> + vmlal.s32 q3, d11, d14
> + vmlal.s32 q3, d12, d9
> + vmlal.s32 q3, d13, d8
> + add r2, sp, #496
> + vld1.8 {d8-d9}, [r2, : 128]
> + vadd.i64 q5, q12, q9
> + vadd.i64 q6, q15, q9
> + vshr.s64 q5, q5, #26
> + vshr.s64 q6, q6, #26
> + vadd.i64 q7, q10, q5
> + vshl.i64 q5, q5, #26
> + vadd.i64 q8, q7, q4
> + vadd.i64 q2, q2, q6
> + vshl.i64 q6, q6, #26
> + vadd.i64 q10, q2, q4
> + vsub.i64 q5, q12, q5
> + vshr.s64 q8, q8, #25
> + vsub.i64 q6, q15, q6
> + vshr.s64 q10, q10, #25
> + vadd.i64 q12, q13, q8
> + vshl.i64 q8, q8, #25
> + vadd.i64 q13, q12, q9
> + vadd.i64 q0, q0, q10
> + vsub.i64 q7, q7, q8
> + vshr.s64 q8, q13, #26
> + vshl.i64 q10, q10, #25
> + vadd.i64 q13, q0, q9
> + vadd.i64 q1, q1, q8
> + vshl.i64 q8, q8, #26
> + vadd.i64 q15, q1, q4
> + vsub.i64 q2, q2, q10
> + vshr.s64 q10, q13, #26
> + vsub.i64 q8, q12, q8
> + vshr.s64 q12, q15, #25
> + vadd.i64 q3, q3, q10
> + vshl.i64 q10, q10, #26
> + vadd.i64 q13, q3, q4
> + vadd.i64 q14, q14, q12
> + add r2, r3, #144
> + vshl.i64 q12, q12, #25
> + add r4, r3, #192
> + vadd.i64 q15, q14, q9
> + add r2, r2, #8
> + vsub.i64 q0, q0, q10
> + add r4, r4, #8
> + vshr.s64 q10, q13, #25
> + vsub.i64 q1, q1, q12
> + vshr.s64 q12, q15, #26
> + vadd.i64 q13, q10, q10
> + vadd.i64 q11, q11, q12
> + vtrn.32 d16, d2
> + vshl.i64 q12, q12, #26
> + vtrn.32 d17, d3
> + vadd.i64 q1, q11, q4
> + vadd.i64 q4, q5, q13
> + vst1.8 d16, [r2, : 64]!
> + vshl.i64 q5, q10, #4
> + vst1.8 d17, [r4, : 64]!
> + vsub.i64 q8, q14, q12
> + vshr.s64 q1, q1, #25
> + vadd.i64 q4, q4, q5
> + vadd.i64 q5, q6, q1
> + vshl.i64 q1, q1, #25
> + vadd.i64 q6, q5, q9
> + vadd.i64 q4, q4, q10
> + vshl.i64 q10, q10, #25
> + vadd.i64 q9, q4, q9
> + vsub.i64 q1, q11, q1
> + vshr.s64 q6, q6, #26
> + vsub.i64 q3, q3, q10
> + vtrn.32 d16, d2
> + vshr.s64 q9, q9, #26
> + vtrn.32 d17, d3
> + vadd.i64 q1, q2, q6
> + vst1.8 d16, [r2, : 64]
> + vshl.i64 q2, q6, #26
> + vst1.8 d17, [r4, : 64]
> + vadd.i64 q6, q7, q9
> + vtrn.32 d0, d6
> + vshl.i64 q7, q9, #26
> + vtrn.32 d1, d7
> + vsub.i64 q2, q5, q2
> + add r2, r2, #16
> + vsub.i64 q3, q4, q7
> + vst1.8 d0, [r2, : 64]
> + add r4, r4, #16
> + vst1.8 d1, [r4, : 64]
> + vtrn.32 d4, d2
> + vtrn.32 d5, d3
> + sub r2, r2, #8
> + sub r4, r4, #8
> + vtrn.32 d6, d12
> + vtrn.32 d7, d13
> + vst1.8 d4, [r2, : 64]
> + vst1.8 d5, [r4, : 64]
> + sub r2, r2, #24
> + sub r4, r4, #24
> + vst1.8 d6, [r2, : 64]
> + vst1.8 d7, [r4, : 64]
> + add r2, r3, #336
> + add r4, r3, #288
> + vld1.8 {d0-d1}, [r2, : 128]!
> + vld1.8 {d2-d3}, [r4, : 128]!
> + vadd.i32 q0, q0, q1
> + vld1.8 {d2-d3}, [r2, : 128]!
> + vld1.8 {d4-d5}, [r4, : 128]!
> + vadd.i32 q1, q1, q2
> + add r5, r3, #288
> + vld1.8 {d4}, [r2, : 64]
> + vld1.8 {d6}, [r4, : 64]
> + vadd.i32 q2, q2, q3
> + vst1.8 {d0-d1}, [r5, : 128]!
> + vst1.8 {d2-d3}, [r5, : 128]!
> + vst1.8 d4, [r5, : 64]
> + add r2, r3, #48
> + add r4, r3, #144
> + vld1.8 {d0-d1}, [r4, : 128]!
> + vld1.8 {d2-d3}, [r4, : 128]!
> + vld1.8 {d4}, [r4, : 64]
> + add r4, r3, #288
> + vld1.8 {d6-d7}, [r4, : 128]!
> + vtrn.32 q0, q3
> + vld1.8 {d8-d9}, [r4, : 128]!
> + vshl.i32 q5, q0, #4
> + vtrn.32 q1, q4
> + vshl.i32 q6, q3, #4
> + vadd.i32 q5, q5, q0
> + vadd.i32 q6, q6, q3
> + vshl.i32 q7, q1, #4
> + vld1.8 {d5}, [r4, : 64]
> + vshl.i32 q8, q4, #4
> + vtrn.32 d4, d5
> + vadd.i32 q7, q7, q1
> + vadd.i32 q8, q8, q4
> + vld1.8 {d18-d19}, [r2, : 128]!
> + vshl.i32 q10, q2, #4
> + vld1.8 {d22-d23}, [r2, : 128]!
> + vadd.i32 q10, q10, q2
> + vld1.8 {d24}, [r2, : 64]
> + vadd.i32 q5, q5, q0
> + add r2, r3, #240
> + vld1.8 {d26-d27}, [r2, : 128]!
> + vadd.i32 q6, q6, q3
> + vld1.8 {d28-d29}, [r2, : 128]!
> + vadd.i32 q8, q8, q4
> + vld1.8 {d25}, [r2, : 64]
> + vadd.i32 q10, q10, q2
> + vtrn.32 q9, q13
> + vadd.i32 q7, q7, q1
> + vadd.i32 q5, q5, q0
> + vtrn.32 q11, q14
> + vadd.i32 q6, q6, q3
> + add r2, sp, #528
> + vadd.i32 q10, q10, q2
> + vtrn.32 d24, d25
> + vst1.8 {d12-d13}, [r2, : 128]
> + vshl.i32 q6, q13, #1
> + add r2, sp, #544
> + vst1.8 {d20-d21}, [r2, : 128]
> + vshl.i32 q10, q14, #1
> + add r2, sp, #560
> + vst1.8 {d12-d13}, [r2, : 128]
> + vshl.i32 q15, q12, #1
> + vadd.i32 q8, q8, q4
> + vext.32 d10, d31, d30, #0
> + vadd.i32 q7, q7, q1
> + add r2, sp, #576
> + vst1.8 {d16-d17}, [r2, : 128]
> + vmull.s32 q8, d18, d5
> + vmlal.s32 q8, d26, d4
> + vmlal.s32 q8, d19, d9
> + vmlal.s32 q8, d27, d3
> + vmlal.s32 q8, d22, d8
> + vmlal.s32 q8, d28, d2
> + vmlal.s32 q8, d23, d7
> + vmlal.s32 q8, d29, d1
> + vmlal.s32 q8, d24, d6
> + vmlal.s32 q8, d25, d0
> + add r2, sp, #592
> + vst1.8 {d14-d15}, [r2, : 128]
> + vmull.s32 q2, d18, d4
> + vmlal.s32 q2, d12, d9
> + vmlal.s32 q2, d13, d8
> + vmlal.s32 q2, d19, d3
> + vmlal.s32 q2, d22, d2
> + vmlal.s32 q2, d23, d1
> + vmlal.s32 q2, d24, d0
> + add r2, sp, #608
> + vst1.8 {d20-d21}, [r2, : 128]
> + vmull.s32 q7, d18, d9
> + vmlal.s32 q7, d26, d3
> + vmlal.s32 q7, d19, d8
> + vmlal.s32 q7, d27, d2
> + vmlal.s32 q7, d22, d7
> + vmlal.s32 q7, d28, d1
> + vmlal.s32 q7, d23, d6
> + vmlal.s32 q7, d29, d0
> + add r2, sp, #624
> + vst1.8 {d10-d11}, [r2, : 128]
> + vmull.s32 q5, d18, d3
> + vmlal.s32 q5, d19, d2
> + vmlal.s32 q5, d22, d1
> + vmlal.s32 q5, d23, d0
> + vmlal.s32 q5, d12, d8
> + add r2, sp, #640
> + vst1.8 {d16-d17}, [r2, : 128]
> + vmull.s32 q4, d18, d8
> + vmlal.s32 q4, d26, d2
> + vmlal.s32 q4, d19, d7
> + vmlal.s32 q4, d27, d1
> + vmlal.s32 q4, d22, d6
> + vmlal.s32 q4, d28, d0
> + vmull.s32 q8, d18, d7
> + vmlal.s32 q8, d26, d1
> + vmlal.s32 q8, d19, d6
> + vmlal.s32 q8, d27, d0
> + add r2, sp, #544
> + vld1.8 {d20-d21}, [r2, : 128]
> + vmlal.s32 q7, d24, d21
> + vmlal.s32 q7, d25, d20
> + vmlal.s32 q4, d23, d21
> + vmlal.s32 q4, d29, d20
> + vmlal.s32 q8, d22, d21
> + vmlal.s32 q8, d28, d20
> + vmlal.s32 q5, d24, d20
> + add r2, sp, #544
> + vst1.8 {d14-d15}, [r2, : 128]
> + vmull.s32 q7, d18, d6
> + vmlal.s32 q7, d26, d0
> + add r2, sp, #624
> + vld1.8 {d30-d31}, [r2, : 128]
> + vmlal.s32 q2, d30, d21
> + vmlal.s32 q7, d19, d21
> + vmlal.s32 q7, d27, d20
> + add r2, sp, #592
> + vld1.8 {d26-d27}, [r2, : 128]
> + vmlal.s32 q4, d25, d27
> + vmlal.s32 q8, d29, d27
> + vmlal.s32 q8, d25, d26
> + vmlal.s32 q7, d28, d27
> + vmlal.s32 q7, d29, d26
> + add r2, sp, #576
> + vld1.8 {d28-d29}, [r2, : 128]
> + vmlal.s32 q4, d24, d29
> + vmlal.s32 q8, d23, d29
> + vmlal.s32 q8, d24, d28
> + vmlal.s32 q7, d22, d29
> + vmlal.s32 q7, d23, d28
> + add r2, sp, #576
> + vst1.8 {d8-d9}, [r2, : 128]
> + add r2, sp, #528
> + vld1.8 {d8-d9}, [r2, : 128]
> + vmlal.s32 q7, d24, d9
> + vmlal.s32 q7, d25, d31
> + vmull.s32 q1, d18, d2
> + vmlal.s32 q1, d19, d1
> + vmlal.s32 q1, d22, d0
> + vmlal.s32 q1, d24, d27
> + vmlal.s32 q1, d23, d20
> + vmlal.s32 q1, d12, d7
> + vmlal.s32 q1, d13, d6
> + vmull.s32 q6, d18, d1
> + vmlal.s32 q6, d19, d0
> + vmlal.s32 q6, d23, d27
> + vmlal.s32 q6, d22, d20
> + vmlal.s32 q6, d24, d26
> + vmull.s32 q0, d18, d0
> + vmlal.s32 q0, d22, d27
> + vmlal.s32 q0, d23, d26
> + vmlal.s32 q0, d24, d31
> + vmlal.s32 q0, d19, d20
> + add r2, sp, #608
> + vld1.8 {d18-d19}, [r2, : 128]
> + vmlal.s32 q2, d18, d7
> + vmlal.s32 q2, d19, d6
> + vmlal.s32 q5, d18, d6
> + vmlal.s32 q5, d19, d21
> + vmlal.s32 q1, d18, d21
> + vmlal.s32 q1, d19, d29
> + vmlal.s32 q0, d18, d28
> + vmlal.s32 q0, d19, d9
> + vmlal.s32 q6, d18, d29
> + vmlal.s32 q6, d19, d28
> + add r2, sp, #560
> + vld1.8 {d18-d19}, [r2, : 128]
> + add r2, sp, #480
> + vld1.8 {d22-d23}, [r2, : 128]
> + vmlal.s32 q5, d19, d7
> + vmlal.s32 q0, d18, d21
> + vmlal.s32 q0, d19, d29
> + vmlal.s32 q6, d18, d6
> + add r2, sp, #496
> + vld1.8 {d6-d7}, [r2, : 128]
> + vmlal.s32 q6, d19, d21
> + add r2, sp, #544
> + vld1.8 {d18-d19}, [r2, : 128]
> + vmlal.s32 q0, d30, d8
> + add r2, sp, #640
> + vld1.8 {d20-d21}, [r2, : 128]
> + vmlal.s32 q5, d30, d29
> + add r2, sp, #576
> + vld1.8 {d24-d25}, [r2, : 128]
> + vmlal.s32 q1, d30, d28
> + vadd.i64 q13, q0, q11
> + vadd.i64 q14, q5, q11
> + vmlal.s32 q6, d30, d9
> + vshr.s64 q4, q13, #26
> + vshr.s64 q13, q14, #26
> + vadd.i64 q7, q7, q4
> + vshl.i64 q4, q4, #26
> + vadd.i64 q14, q7, q3
> + vadd.i64 q9, q9, q13
> + vshl.i64 q13, q13, #26
> + vadd.i64 q15, q9, q3
> + vsub.i64 q0, q0, q4
> + vshr.s64 q4, q14, #25
> + vsub.i64 q5, q5, q13
> + vshr.s64 q13, q15, #25
> + vadd.i64 q6, q6, q4
> + vshl.i64 q4, q4, #25
> + vadd.i64 q14, q6, q11
> + vadd.i64 q2, q2, q13
> + vsub.i64 q4, q7, q4
> + vshr.s64 q7, q14, #26
> + vshl.i64 q13, q13, #25
> + vadd.i64 q14, q2, q11
> + vadd.i64 q8, q8, q7
> + vshl.i64 q7, q7, #26
> + vadd.i64 q15, q8, q3
> + vsub.i64 q9, q9, q13
> + vshr.s64 q13, q14, #26
> + vsub.i64 q6, q6, q7
> + vshr.s64 q7, q15, #25
> + vadd.i64 q10, q10, q13
> + vshl.i64 q13, q13, #26
> + vadd.i64 q14, q10, q3
> + vadd.i64 q1, q1, q7
> + add r2, r3, #240
> + vshl.i64 q7, q7, #25
> + add r4, r3, #144
> + vadd.i64 q15, q1, q11
> + add r2, r2, #8
> + vsub.i64 q2, q2, q13
> + add r4, r4, #8
> + vshr.s64 q13, q14, #25
> + vsub.i64 q7, q8, q7
> + vshr.s64 q8, q15, #26
> + vadd.i64 q14, q13, q13
> + vadd.i64 q12, q12, q8
> + vtrn.32 d12, d14
> + vshl.i64 q8, q8, #26
> + vtrn.32 d13, d15
> + vadd.i64 q3, q12, q3
> + vadd.i64 q0, q0, q14
> + vst1.8 d12, [r2, : 64]!
> + vshl.i64 q7, q13, #4
> + vst1.8 d13, [r4, : 64]!
> + vsub.i64 q1, q1, q8
> + vshr.s64 q3, q3, #25
> + vadd.i64 q0, q0, q7
> + vadd.i64 q5, q5, q3
> + vshl.i64 q3, q3, #25
> + vadd.i64 q6, q5, q11
> + vadd.i64 q0, q0, q13
> + vshl.i64 q7, q13, #25
> + vadd.i64 q8, q0, q11
> + vsub.i64 q3, q12, q3
> + vshr.s64 q6, q6, #26
> + vsub.i64 q7, q10, q7
> + vtrn.32 d2, d6
> + vshr.s64 q8, q8, #26
> + vtrn.32 d3, d7
> + vadd.i64 q3, q9, q6
> + vst1.8 d2, [r2, : 64]
> + vshl.i64 q6, q6, #26
> + vst1.8 d3, [r4, : 64]
> + vadd.i64 q1, q4, q8
> + vtrn.32 d4, d14
> + vshl.i64 q4, q8, #26
> + vtrn.32 d5, d15
> + vsub.i64 q5, q5, q6
> + add r2, r2, #16
> + vsub.i64 q0, q0, q4
> + vst1.8 d4, [r2, : 64]
> + add r4, r4, #16
> + vst1.8 d5, [r4, : 64]
> + vtrn.32 d10, d6
> + vtrn.32 d11, d7
> + sub r2, r2, #8
> + sub r4, r4, #8
> + vtrn.32 d0, d2
> + vtrn.32 d1, d3
> + vst1.8 d10, [r2, : 64]
> + vst1.8 d11, [r4, : 64]
> + sub r2, r2, #24
> + sub r4, r4, #24
> + vst1.8 d0, [r2, : 64]
> + vst1.8 d1, [r4, : 64]
> + ldr r2, [sp, #456]
> + ldr r4, [sp, #460]
> + subs r5, r2, #1
> + bge .Lmainloop
> + add r1, r3, #144
> + add r2, r3, #336
> + vld1.8 {d0-d1}, [r1, : 128]!
> + vld1.8 {d2-d3}, [r1, : 128]!
> + vld1.8 {d4}, [r1, : 64]
> + vst1.8 {d0-d1}, [r2, : 128]!
> + vst1.8 {d2-d3}, [r2, : 128]!
> + vst1.8 d4, [r2, : 64]
> + movw r1, #0
> +.Linvertloop:
> + add r2, r3, #144
> + movw r4, #0
> + movw r5, #2
> + cmp r1, #1
> + moveq r5, #1
> + addeq r2, r3, #336
> + addeq r4, r3, #48
> + cmp r1, #2
> + moveq r5, #1
> + addeq r2, r3, #48
> + cmp r1, #3
> + moveq r5, #5
> + addeq r4, r3, #336
> + cmp r1, #4
> + moveq r5, #10
> + cmp r1, #5
> + moveq r5, #20
> + cmp r1, #6
> + moveq r5, #10
> + addeq r2, r3, #336
> + addeq r4, r3, #336
> + cmp r1, #7
> + moveq r5, #50
> + cmp r1, #8
> + moveq r5, #100
> + cmp r1, #9
> + moveq r5, #50
> + addeq r2, r3, #336
> + cmp r1, #10
> + moveq r5, #5
> + addeq r2, r3, #48
> + cmp r1, #11
> + moveq r5, #0
> + addeq r2, r3, #96
> + add r6, r3, #144
> + add r7, r3, #288
> + vld1.8 {d0-d1}, [r6, : 128]!
> + vld1.8 {d2-d3}, [r6, : 128]!
> + vld1.8 {d4}, [r6, : 64]
> + vst1.8 {d0-d1}, [r7, : 128]!
> + vst1.8 {d2-d3}, [r7, : 128]!
> + vst1.8 d4, [r7, : 64]
> + cmp r5, #0
> + beq .Lskipsquaringloop
> +.Lsquaringloop:
> + add r6, r3, #288
> + add r7, r3, #288
> + add r8, r3, #288
> + vmov.i32 q0, #19
> + vmov.i32 q1, #0
> + vmov.i32 q2, #1
> + vzip.i32 q1, q2
> + vld1.8 {d4-d5}, [r7, : 128]!
> + vld1.8 {d6-d7}, [r7, : 128]!
> + vld1.8 {d9}, [r7, : 64]
> + vld1.8 {d10-d11}, [r6, : 128]!
> + add r7, sp, #384
> + vld1.8 {d12-d13}, [r6, : 128]!
> + vmul.i32 q7, q2, q0
> + vld1.8 {d8}, [r6, : 64]
> + vext.32 d17, d11, d10, #1
> + vmul.i32 q9, q3, q0
> + vext.32 d16, d10, d8, #1
> + vshl.u32 q10, q5, q1
> + vext.32 d22, d14, d4, #1
> + vext.32 d24, d18, d6, #1
> + vshl.u32 q13, q6, q1
> + vshl.u32 d28, d8, d2
> + vrev64.i32 d22, d22
> + vmul.i32 d1, d9, d1
> + vrev64.i32 d24, d24
> + vext.32 d29, d8, d13, #1
> + vext.32 d0, d1, d9, #1
> + vrev64.i32 d0, d0
> + vext.32 d2, d9, d1, #1
> + vext.32 d23, d15, d5, #1
> + vmull.s32 q4, d20, d4
> + vrev64.i32 d23, d23
> + vmlal.s32 q4, d21, d1
> + vrev64.i32 d2, d2
> + vmlal.s32 q4, d26, d19
> + vext.32 d3, d5, d15, #1
> + vmlal.s32 q4, d27, d18
> + vrev64.i32 d3, d3
> + vmlal.s32 q4, d28, d15
> + vext.32 d14, d12, d11, #1
> + vmull.s32 q5, d16, d23
> + vext.32 d15, d13, d12, #1
> + vmlal.s32 q5, d17, d4
> + vst1.8 d8, [r7, : 64]!
> + vmlal.s32 q5, d14, d1
> + vext.32 d12, d9, d8, #0
> + vmlal.s32 q5, d15, d19
> + vmov.i64 d13, #0
> + vmlal.s32 q5, d29, d18
> + vext.32 d25, d19, d7, #1
> + vmlal.s32 q6, d20, d5
> + vrev64.i32 d25, d25
> + vmlal.s32 q6, d21, d4
> + vst1.8 d11, [r7, : 64]!
> + vmlal.s32 q6, d26, d1
> + vext.32 d9, d10, d10, #0
> + vmlal.s32 q6, d27, d19
> + vmov.i64 d8, #0
> + vmlal.s32 q6, d28, d18
> + vmlal.s32 q4, d16, d24
> + vmlal.s32 q4, d17, d5
> + vmlal.s32 q4, d14, d4
> + vst1.8 d12, [r7, : 64]!
> + vmlal.s32 q4, d15, d1
> + vext.32 d10, d13, d12, #0
> + vmlal.s32 q4, d29, d19
> + vmov.i64 d11, #0
> + vmlal.s32 q5, d20, d6
> + vmlal.s32 q5, d21, d5
> + vmlal.s32 q5, d26, d4
> + vext.32 d13, d8, d8, #0
> + vmlal.s32 q5, d27, d1
> + vmov.i64 d12, #0
> + vmlal.s32 q5, d28, d19
> + vst1.8 d9, [r7, : 64]!
> + vmlal.s32 q6, d16, d25
> + vmlal.s32 q6, d17, d6
> + vst1.8 d10, [r7, : 64]
> + vmlal.s32 q6, d14, d5
> + vext.32 d8, d11, d10, #0
> + vmlal.s32 q6, d15, d4
> + vmov.i64 d9, #0
> + vmlal.s32 q6, d29, d1
> + vmlal.s32 q4, d20, d7
> + vmlal.s32 q4, d21, d6
> + vmlal.s32 q4, d26, d5
> + vext.32 d11, d12, d12, #0
> + vmlal.s32 q4, d27, d4
> + vmov.i64 d10, #0
> + vmlal.s32 q4, d28, d1
> + vmlal.s32 q5, d16, d0
> + sub r6, r7, #32
> + vmlal.s32 q5, d17, d7
> + vmlal.s32 q5, d14, d6
> + vext.32 d30, d9, d8, #0
> + vmlal.s32 q5, d15, d5
> + vld1.8 {d31}, [r6, : 64]!
> + vmlal.s32 q5, d29, d4
> + vmlal.s32 q15, d20, d0
> + vext.32 d0, d6, d18, #1
> + vmlal.s32 q15, d21, d25
> + vrev64.i32 d0, d0
> + vmlal.s32 q15, d26, d24
> + vext.32 d1, d7, d19, #1
> + vext.32 d7, d10, d10, #0
> + vmlal.s32 q15, d27, d23
> + vrev64.i32 d1, d1
> + vld1.8 {d6}, [r6, : 64]
> + vmlal.s32 q15, d28, d22
> + vmlal.s32 q3, d16, d4
> + add r6, r6, #24
> + vmlal.s32 q3, d17, d2
> + vext.32 d4, d31, d30, #0
> + vmov d17, d11
> + vmlal.s32 q3, d14, d1
> + vext.32 d11, d13, d13, #0
> + vext.32 d13, d30, d30, #0
> + vmlal.s32 q3, d15, d0
> + vext.32 d1, d8, d8, #0
> + vmlal.s32 q3, d29, d3
> + vld1.8 {d5}, [r6, : 64]
> + sub r6, r6, #16
> + vext.32 d10, d6, d6, #0
> + vmov.i32 q1, #0xffffffff
> + vshl.i64 q4, q1, #25
> + add r7, sp, #480
> + vld1.8 {d14-d15}, [r7, : 128]
> + vadd.i64 q9, q2, q7
> + vshl.i64 q1, q1, #26
> + vshr.s64 q10, q9, #26
> + vld1.8 {d0}, [r6, : 64]!
> + vadd.i64 q5, q5, q10
> + vand q9, q9, q1
> + vld1.8 {d16}, [r6, : 64]!
> + add r6, sp, #496
> + vld1.8 {d20-d21}, [r6, : 128]
> + vadd.i64 q11, q5, q10
> + vsub.i64 q2, q2, q9
> + vshr.s64 q9, q11, #25
> + vext.32 d12, d5, d4, #0
> + vand q11, q11, q4
> + vadd.i64 q0, q0, q9
> + vmov d19, d7
> + vadd.i64 q3, q0, q7
> + vsub.i64 q5, q5, q11
> + vshr.s64 q11, q3, #26
> + vext.32 d18, d11, d10, #0
> + vand q3, q3, q1
> + vadd.i64 q8, q8, q11
> + vadd.i64 q11, q8, q10
> + vsub.i64 q0, q0, q3
> + vshr.s64 q3, q11, #25
> + vand q11, q11, q4
> + vadd.i64 q3, q6, q3
> + vadd.i64 q6, q3, q7
> + vsub.i64 q8, q8, q11
> + vshr.s64 q11, q6, #26
> + vand q6, q6, q1
> + vadd.i64 q9, q9, q11
> + vadd.i64 d25, d19, d21
> + vsub.i64 q3, q3, q6
> + vshr.s64 d23, d25, #25
> + vand q4, q12, q4
> + vadd.i64 d21, d23, d23
> + vshl.i64 d25, d23, #4
> + vadd.i64 d21, d21, d23
> + vadd.i64 d25, d25, d21
> + vadd.i64 d4, d4, d25
> + vzip.i32 q0, q8
> + vadd.i64 d12, d4, d14
> + add r6, r8, #8
> + vst1.8 d0, [r6, : 64]
> + vsub.i64 d19, d19, d9
> + add r6, r6, #16
> + vst1.8 d16, [r6, : 64]
> + vshr.s64 d22, d12, #26
> + vand q0, q6, q1
> + vadd.i64 d10, d10, d22
> + vzip.i32 q3, q9
> + vsub.i64 d4, d4, d0
> + sub r6, r6, #8
> + vst1.8 d6, [r6, : 64]
> + add r6, r6, #16
> + vst1.8 d18, [r6, : 64]
> + vzip.i32 q2, q5
> + sub r6, r6, #32
> + vst1.8 d4, [r6, : 64]
> + subs r5, r5, #1
> + bhi .Lsquaringloop
> +.Lskipsquaringloop:
> + mov r2, r2
> + add r5, r3, #288
> + add r6, r3, #144
> + vmov.i32 q0, #19
> + vmov.i32 q1, #0
> + vmov.i32 q2, #1
> + vzip.i32 q1, q2
> + vld1.8 {d4-d5}, [r5, : 128]!
> + vld1.8 {d6-d7}, [r5, : 128]!
> + vld1.8 {d9}, [r5, : 64]
> + vld1.8 {d10-d11}, [r2, : 128]!
> + add r5, sp, #384
> + vld1.8 {d12-d13}, [r2, : 128]!
> + vmul.i32 q7, q2, q0
> + vld1.8 {d8}, [r2, : 64]
> + vext.32 d17, d11, d10, #1
> + vmul.i32 q9, q3, q0
> + vext.32 d16, d10, d8, #1
> + vshl.u32 q10, q5, q1
> + vext.32 d22, d14, d4, #1
> + vext.32 d24, d18, d6, #1
> + vshl.u32 q13, q6, q1
> + vshl.u32 d28, d8, d2
> + vrev64.i32 d22, d22
> + vmul.i32 d1, d9, d1
> + vrev64.i32 d24, d24
> + vext.32 d29, d8, d13, #1
> + vext.32 d0, d1, d9, #1
> + vrev64.i32 d0, d0
> + vext.32 d2, d9, d1, #1
> + vext.32 d23, d15, d5, #1
> + vmull.s32 q4, d20, d4
> + vrev64.i32 d23, d23
> + vmlal.s32 q4, d21, d1
> + vrev64.i32 d2, d2
> + vmlal.s32 q4, d26, d19
> + vext.32 d3, d5, d15, #1
> + vmlal.s32 q4, d27, d18
> + vrev64.i32 d3, d3
> + vmlal.s32 q4, d28, d15
> + vext.32 d14, d12, d11, #1
> + vmull.s32 q5, d16, d23
> + vext.32 d15, d13, d12, #1
> + vmlal.s32 q5, d17, d4
> + vst1.8 d8, [r5, : 64]!
> + vmlal.s32 q5, d14, d1
> + vext.32 d12, d9, d8, #0
> + vmlal.s32 q5, d15, d19
> + vmov.i64 d13, #0
> + vmlal.s32 q5, d29, d18
> + vext.32 d25, d19, d7, #1
> + vmlal.s32 q6, d20, d5
> + vrev64.i32 d25, d25
> + vmlal.s32 q6, d21, d4
> + vst1.8 d11, [r5, : 64]!
> + vmlal.s32 q6, d26, d1
> + vext.32 d9, d10, d10, #0
> + vmlal.s32 q6, d27, d19
> + vmov.i64 d8, #0
> + vmlal.s32 q6, d28, d18
> + vmlal.s32 q4, d16, d24
> + vmlal.s32 q4, d17, d5
> + vmlal.s32 q4, d14, d4
> + vst1.8 d12, [r5, : 64]!
> + vmlal.s32 q4, d15, d1
> + vext.32 d10, d13, d12, #0
> + vmlal.s32 q4, d29, d19
> + vmov.i64 d11, #0
> + vmlal.s32 q5, d20, d6
> + vmlal.s32 q5, d21, d5
> + vmlal.s32 q5, d26, d4
> + vext.32 d13, d8, d8, #0
> + vmlal.s32 q5, d27, d1
> + vmov.i64 d12, #0
> + vmlal.s32 q5, d28, d19
> + vst1.8 d9, [r5, : 64]!
> + vmlal.s32 q6, d16, d25
> + vmlal.s32 q6, d17, d6
> + vst1.8 d10, [r5, : 64]
> + vmlal.s32 q6, d14, d5
> + vext.32 d8, d11, d10, #0
> + vmlal.s32 q6, d15, d4
> + vmov.i64 d9, #0
> + vmlal.s32 q6, d29, d1
> + vmlal.s32 q4, d20, d7
> + vmlal.s32 q4, d21, d6
> + vmlal.s32 q4, d26, d5
> + vext.32 d11, d12, d12, #0
> + vmlal.s32 q4, d27, d4
> + vmov.i64 d10, #0
> + vmlal.s32 q4, d28, d1
> + vmlal.s32 q5, d16, d0
> + sub r2, r5, #32
> + vmlal.s32 q5, d17, d7
> + vmlal.s32 q5, d14, d6
> + vext.32 d30, d9, d8, #0
> + vmlal.s32 q5, d15, d5
> + vld1.8 {d31}, [r2, : 64]!
> + vmlal.s32 q5, d29, d4
> + vmlal.s32 q15, d20, d0
> + vext.32 d0, d6, d18, #1
> + vmlal.s32 q15, d21, d25
> + vrev64.i32 d0, d0
> + vmlal.s32 q15, d26, d24
> + vext.32 d1, d7, d19, #1
> + vext.32 d7, d10, d10, #0
> + vmlal.s32 q15, d27, d23
> + vrev64.i32 d1, d1
> + vld1.8 {d6}, [r2, : 64]
> + vmlal.s32 q15, d28, d22
> + vmlal.s32 q3, d16, d4
> + add r2, r2, #24
> + vmlal.s32 q3, d17, d2
> + vext.32 d4, d31, d30, #0
> + vmov d17, d11
> + vmlal.s32 q3, d14, d1
> + vext.32 d11, d13, d13, #0
> + vext.32 d13, d30, d30, #0
> + vmlal.s32 q3, d15, d0
> + vext.32 d1, d8, d8, #0
> + vmlal.s32 q3, d29, d3
> + vld1.8 {d5}, [r2, : 64]
> + sub r2, r2, #16
> + vext.32 d10, d6, d6, #0
> + vmov.i32 q1, #0xffffffff
> + vshl.i64 q4, q1, #25
> + add r5, sp, #480
> + vld1.8 {d14-d15}, [r5, : 128]
> + vadd.i64 q9, q2, q7
> + vshl.i64 q1, q1, #26
> + vshr.s64 q10, q9, #26
> + vld1.8 {d0}, [r2, : 64]!
> + vadd.i64 q5, q5, q10
> + vand q9, q9, q1
> + vld1.8 {d16}, [r2, : 64]!
> + add r2, sp, #496
> + vld1.8 {d20-d21}, [r2, : 128]
> + vadd.i64 q11, q5, q10
> + vsub.i64 q2, q2, q9
> + vshr.s64 q9, q11, #25
> + vext.32 d12, d5, d4, #0
> + vand q11, q11, q4
> + vadd.i64 q0, q0, q9
> + vmov d19, d7
> + vadd.i64 q3, q0, q7
> + vsub.i64 q5, q5, q11
> + vshr.s64 q11, q3, #26
> + vext.32 d18, d11, d10, #0
> + vand q3, q3, q1
> + vadd.i64 q8, q8, q11
> + vadd.i64 q11, q8, q10
> + vsub.i64 q0, q0, q3
> + vshr.s64 q3, q11, #25
> + vand q11, q11, q4
> + vadd.i64 q3, q6, q3
> + vadd.i64 q6, q3, q7
> + vsub.i64 q8, q8, q11
> + vshr.s64 q11, q6, #26
> + vand q6, q6, q1
> + vadd.i64 q9, q9, q11
> + vadd.i64 d25, d19, d21
> + vsub.i64 q3, q3, q6
> + vshr.s64 d23, d25, #25
> + vand q4, q12, q4
> + vadd.i64 d21, d23, d23
> + vshl.i64 d25, d23, #4
> + vadd.i64 d21, d21, d23
> + vadd.i64 d25, d25, d21
> + vadd.i64 d4, d4, d25
> + vzip.i32 q0, q8
> + vadd.i64 d12, d4, d14
> + add r2, r6, #8
> + vst1.8 d0, [r2, : 64]
> + vsub.i64 d19, d19, d9
> + add r2, r2, #16
> + vst1.8 d16, [r2, : 64]
> + vshr.s64 d22, d12, #26
> + vand q0, q6, q1
> + vadd.i64 d10, d10, d22
> + vzip.i32 q3, q9
> + vsub.i64 d4, d4, d0
> + sub r2, r2, #8
> + vst1.8 d6, [r2, : 64]
> + add r2, r2, #16
> + vst1.8 d18, [r2, : 64]
> + vzip.i32 q2, q5
> + sub r2, r2, #32
> + vst1.8 d4, [r2, : 64]
> + cmp r4, #0
> + beq .Lskippostcopy
> + add r2, r3, #144
> + mov r4, r4
> + vld1.8 {d0-d1}, [r2, : 128]!
> + vld1.8 {d2-d3}, [r2, : 128]!
> + vld1.8 {d4}, [r2, : 64]
> + vst1.8 {d0-d1}, [r4, : 128]!
> + vst1.8 {d2-d3}, [r4, : 128]!
> + vst1.8 d4, [r4, : 64]
> +.Lskippostcopy:
> + cmp r1, #1
> + bne .Lskipfinalcopy
> + add r2, r3, #288
> + add r4, r3, #144
> + vld1.8 {d0-d1}, [r2, : 128]!
> + vld1.8 {d2-d3}, [r2, : 128]!
> + vld1.8 {d4}, [r2, : 64]
> + vst1.8 {d0-d1}, [r4, : 128]!
> + vst1.8 {d2-d3}, [r4, : 128]!
> + vst1.8 d4, [r4, : 64]
> +.Lskipfinalcopy:
> + add r1, r1, #1
> + cmp r1, #12
> + blo .Linvertloop
> + add r1, r3, #144
> + ldr r2, [r1], #4
> + ldr r3, [r1], #4
> + ldr r4, [r1], #4
> + ldr r5, [r1], #4
> + ldr r6, [r1], #4
> + ldr r7, [r1], #4
> + ldr r8, [r1], #4
> + ldr r9, [r1], #4
> + ldr r10, [r1], #4
> + ldr r1, [r1]
> + add r11, r1, r1, LSL #4
> + add r11, r11, r1, LSL #1
> + add r11, r11, #16777216
> + mov r11, r11, ASR #25
> + add r11, r11, r2
> + mov r11, r11, ASR #26
> + add r11, r11, r3
> + mov r11, r11, ASR #25
> + add r11, r11, r4
> + mov r11, r11, ASR #26
> + add r11, r11, r5
> + mov r11, r11, ASR #25
> + add r11, r11, r6
> + mov r11, r11, ASR #26
> + add r11, r11, r7
> + mov r11, r11, ASR #25
> + add r11, r11, r8
> + mov r11, r11, ASR #26
> + add r11, r11, r9
> + mov r11, r11, ASR #25
> + add r11, r11, r10
> + mov r11, r11, ASR #26
> + add r11, r11, r1
> + mov r11, r11, ASR #25
> + add r2, r2, r11
> + add r2, r2, r11, LSL #1
> + add r2, r2, r11, LSL #4
> + mov r11, r2, ASR #26
> + add r3, r3, r11
> + sub r2, r2, r11, LSL #26
> + mov r11, r3, ASR #25
> + add r4, r4, r11
> + sub r3, r3, r11, LSL #25
> + mov r11, r4, ASR #26
> + add r5, r5, r11
> + sub r4, r4, r11, LSL #26
> + mov r11, r5, ASR #25
> + add r6, r6, r11
> + sub r5, r5, r11, LSL #25
> + mov r11, r6, ASR #26
> + add r7, r7, r11
> + sub r6, r6, r11, LSL #26
> + mov r11, r7, ASR #25
> + add r8, r8, r11
> + sub r7, r7, r11, LSL #25
> + mov r11, r8, ASR #26
> + add r9, r9, r11
> + sub r8, r8, r11, LSL #26
> + mov r11, r9, ASR #25
> + add r10, r10, r11
> + sub r9, r9, r11, LSL #25
> + mov r11, r10, ASR #26
> + add r1, r1, r11
> + sub r10, r10, r11, LSL #26
> + mov r11, r1, ASR #25
> + sub r1, r1, r11, LSL #25
> + add r2, r2, r3, LSL #26
> + mov r3, r3, LSR #6
> + add r3, r3, r4, LSL #19
> + mov r4, r4, LSR #13
> + add r4, r4, r5, LSL #13
> + mov r5, r5, LSR #19
> + add r5, r5, r6, LSL #6
> + add r6, r7, r8, LSL #25
> + mov r7, r8, LSR #7
> + add r7, r7, r9, LSL #19
> + mov r8, r9, LSR #13
> + add r8, r8, r10, LSL #12
> + mov r9, r10, LSR #20
> + add r1, r9, r1, LSL #6
> + str r2, [r0]
> + str r3, [r0, #4]
> + str r4, [r0, #8]
> + str r5, [r0, #12]
> + str r6, [r0, #16]
> + str r7, [r0, #20]
> + str r8, [r0, #24]
> + str r1, [r0, #28]
> + movw r0, #0
> + mov sp, ip
> + pop {r4-r11, pc}
> +ENDPROC(curve25519_neon)
> +#endif
> diff --git a/lib/zinc/curve25519/curve25519.c b/lib/zinc/curve25519/curve25519.c
> index 32536340d39d..0d5ea97762d4 100644
> --- a/lib/zinc/curve25519/curve25519.c
> +++ b/lib/zinc/curve25519/curve25519.c
> @@ -21,6 +21,8 @@
>
> #if defined(CONFIG_ZINC_ARCH_X86_64)
> #include "curve25519-x86_64-glue.h"
> +#elif defined(CONFIG_ZINC_ARCH_ARM)
> +#include "curve25519-arm-glue.h"
> #else
> void __init curve25519_fpu_init(void)
> {
> --
> 2.19.0
>
^ permalink raw reply
* pull-request: wireless-drivers-next 2018-10-02
From: Kalle Valo @ 2018-10-02 17:06 UTC (permalink / raw)
To: David Miller; +Cc: linux-wireless, netdev, linux-kernel
Hi Dave,
here's the first pull request to net-next for 4.20. This is rather large
due to mt76 refactoring and me not being able to send the pull request
earlier, sorry about that.
Most of this has been in linux-next for some time now so I'm hoping
there should not be any nasty surprises, but please let me know if you
have any problems.
Kalle
The following changes since commit 050cdc6c9501abcd64720b8cc3e7941efee9547d:
Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net (2018-08-27 11:59:39 -0700)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next.git tags/wireless-drivers-next-for-davem-2018-10-02
for you to fetch changes up to 08b0109eea36d71f87b863045d91bbcee98758bd:
Merge tag 'iwlwifi-next-for-kalle-2018-09-28' of git://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/iwlwifi-next (2018-10-01 18:49:48 +0300)
----------------------------------------------------------------
wireless-drivers-next patches for 4.20
First set of new features for 4.20. mt76 driver is going through major
refactoring and that's why there are so many mt76 patches. iwlwifi is
also under heavy development and smaller changes to other drivers.
Also wireless-drivers was merged to fix a conflict between the two trees.
Major changes:
ath10k
* limit available channels via DT ieee80211-freq-limit
wil6210
* add 802.11r Fast Roaming support for AP and station modes
* add support for channel 4
iwlwifi
* new FW API handling
* some improvements in the PCI recovery mechanism
* enable a new scanning feature;
* continued work on HE (mostly radiotap)
* TKIP implementation in new devices
* work continues for new 22560 hardware
mt76
* add support for Alfa AWUS036ACM
* lots of refactoring to make it easier to add new hardware support
* prepare for adding mt76x0e (pci-e variant) support
* add CONFIG_MT76x0E kconfig symbol
brcmfmac
* add support CYW89342 mini-PCIe device
* add 4-way handshake offload detection for FT-802.1X
* enable NL80211_EXT_FEATURE_CQM_RSSI_LIST
* fix for proper support of 160MHz bandwidth
rtl8xxxu
* add rtl8188ctv support
----------------------------------------------------------------
Ahmad Masri (1):
wil6210: add FT roam support for AP and station
Aleksei Mamlin (1):
rtl8xxxu: Add rtl8188ctv support
Alexei Avshalom Lazar (1):
wil6210: add support for channel 4
Arend van Spriel (2):
brcmfmac: fix for proper support of 160MHz bandwidth
brcmfmac: increase buffer for obtaining firmware capabilities
Avraham Stern (3):
iwlwifi: set the tid for non-QOS frames to zero
iwlwifi: mvm: Send LQ command as async when necessary
iwlwifi: mvm: set wep key for all stations in soft ap mode
Ayala Beker (2):
iwlwifi: mvm: skip EBS in low latency mode while fragmented scan isn't supported
iwlwifi: mvm: activate fragmented EBS in case of fragmented scan
Chung-Hsien Hsu (2):
brcmfmac: add FT-based AKMs in brcmf_set_key_mgmt() for FT support
brcmfmac: add 4-way handshake offload detection for FT-802.1X
Colin Ian King (3):
ath9k: remove unused array firstep_table
orinoco: remove unused array encaps_hdr and macro ENCAPS_OVERHEAD
b43: fix spelling mistake "hw_registred" -> "hw_registered"
Dan Carpenter (2):
libertas_tf: prevent underflow in process_cmdrequest()
rt2x00: use simple_read_from_buffer()
David Spinadel (1):
iwlwifi: mvm: Support TKIP on gen2 data path
Dedy Lansky (2):
wil6210: drop Rx multicast packets that are looped-back to STA
wil6210: fix invalid memory access for rx_buff_mgmt debugfs
Dreyfuss, Haim (1):
iwlwifi: mvm: cleanup dead code on resume flow for non unified image.
Emmanuel Grumbach (5):
iwlwifi: improve the flow when a NIC is disconnected
iwlwifi: mvm: send BCAST management frames to the right station
iwlwifi: mvm: fix a comment about the SP length
iwlwifi: dbg: don't crash if the firmware crashes in the middle of a debug dump
iwlwifi: mvm: remove support for adjacent channel compensation
Erel Geron (3):
iwlwifi: mvm: support Coex Schema 2
iwlwifi: fix non_shared_ant for 22000 devices
iwlwifi: mvm: TLC support for Coex Schema 2
Felix Fietkau (2):
ath9k: fix tx99 with monitor mode interface
mt76: use a per rx queue page fragment cache
Ganapathi Bhat (2):
mwifiex: do no submit URB in suspended state
mwifex: free rx_cmd skb in suspended state
Geert Uytterhoeven (1):
mt76: Fix comparisons with invalid hardware key index
Golan Ben Ami (9):
iwlwifi: add required include to iwl-fh.h
iwlwifi: pcie: allow using tx init for other queues but the command queue
iwlwifi: pcie: make non-static hcmd and rx code
iwlwifi: remove FSF's address from the license notice
iwlwifi: pcie: store the default rxq number
iwlwifi: pcie: make gen2 of apm_init non-static
iwlwifi: refactor txq_alloc for supporting more command type
iwlwifi: pcie: fit reclaim msg to MAX_MSG_LEN
iwlwifi: configure power scheme to balanced for 22560 devices
Gregory Greenman (1):
iwlwifi: mvm: add NOA and CSA to a probe response
Gustavo A. R. Silva (2):
ath10k: use struct_size() in kzalloc()
orinoco_usb: fix spelling mistake in fall-through annotation
Haim Dreyfuss (1):
iwlwifi: mvm: support new reduce tx power FW API.
Ido Yariv (1):
iwlwifi: Add missing 11n disable module parameter check
Igor Mitsyanko (11):
qtnfmac_pcie: do not store FW name in driver state structure
qtnfmac_pcie: move Pearl pcie sources to pcie-specific directory
qtnfmac_pcie: rename private Pearl PCIe state structure
qtnfmac_pcie: indicate pearl-specific structures by their names
qtnfmac_pcie: pearl: rename spinlock tx0_lock to tx_lock
qtnfmac_pcie: separate platform-independent PCIe structure
qtnfmac_pcie: rename platform-specific functions
qtnfmac: add missing header includes to bus.h
qtnfmac_pcie: extract platform-independent PCIe code
qtnfmac: wait for FW load work to finish at PCIe remove
qtnfmac_pcie: check for correct CHIP ID at pcie probe
Igor Stoppa (1):
wireless: remove unnecessary unlikely()
Ilan Peer (1):
iwlwifi: mvm: Allow TKIP for AP mode
Jia-Shyr Chuang (1):
brcmfmac: add CYW89342 mini-PCIe device
Johannes Berg (19):
iwlwifi: remove dump_regs() from transport ops
iwlwifi: don't WARN on trying to dump dead firmware
iwlwifi: mvm: implement extended HE-MU sniffer API
iwlwifi: mvm: put LTF symbol size into HE radiotap
iwlwifi: mvm: properly decode HE GI duration
iwlwifi: mvm: report # of LTF symbols for extended range SU PPDUs
iwlwifi: mvm: remove channel 2 from HE radiotap if not applicable
iwlwifi: mvm: decode HE TB PPDU data
Revert "iwlwifi: allow memory debug TLV to specify the memory type"
iwlwifi: RX API: remove unnecessary anonymous struct
iwlwifi: mvm: report RU offset is known
iwlwifi: remove ucode error tracepoint
iwlwifi: api: annotate compressed BA notif array sizes
iwlwifi: pcie: gen2: pull adding frags to helper routine
iwlwifi: pcie: gen2: build A-MSDU only for GSO
iwlwifi: pcie: tx: unify TFD unmapping
iwlwifi: pcie: tx: pull tracing out of iwl_fill_data_tbs()
iwlwifi: pcie: support transmitting SKBs with fraglist
iwlwifi: fix LED command capability bit
Kalle Valo (5):
Merge ath-next from git://git.kernel.org/.../kvalo/ath.git
Merge tag 'iwlwifi-next-for-kalle-2018-08-31' of git://git.kernel.org/.../iwlwifi/iwlwifi-next
Merge wireless-drivers into wireless-drivers-next
Merge tag 'mt76-for-kvalo-2018-09-19' of https://github.com/nbd168/wireless
Merge tag 'iwlwifi-next-for-kalle-2018-09-28' of git://git.kernel.org/.../iwlwifi/iwlwifi-next
Karthick Gopalasubramanian (1):
wil6210: remove reset file from debugfs
Kevin Lo (1):
rtlwifi: remove set but unused variables
Kristian Evensen (1):
mt76: Enable NL80211_EXT_FEATURE_CQM_RSSI_LIST
Larry Finger (1):
rtl8187: Fix warning generated when strncpy() destination length matches the sixe argument
Liad Kaufman (2):
iwlwifi: mvm: add bss color to radiotap
iwlwifi: mvm: add support for RX_AMPDU_EOF bit for HE SU
Lorenzo Bianconi (76):
mt76x2u: run device cleanup routine if resume fails
mt76: verify evt type in usb mcu response
mt76: move mt76_reg_pair definition in mt76.h
mt76: split __mt76u_mcu_send_msg and mt76u_mcu_send_msg routines
mt76: move mt76x2u_mcu_deinit routine in mt76-usb module
mt76: add multiple regs read support to usb_mcu layer
mt76x0: use mt76u_init for bus initialization
mt76x0: remove mt76x0_vendor_reset routine
mt76x0: remove mt76x0_vendor_single_wr routine
mt76x0: remove mt76x0_mcu_msg_alloc routine
mt76x0: use shared mt76 usb mcu completion
mt76x0: remove mt76x0_complete_urb routine
mt76x0: remove mt76x0_vendor_request routine
mt76x0: remove unused mt76x0_wait_asic_ready routine
mt76x0: use mt76u_mcu_send_msg utility routine to send fw cmds
mt76x0: remove unused mt76x0_mcu structure
mt76x0: use mt76u_mcu_fw_send_data for fw uploading
mt76x0: remove unused routines for usb_buff alloc/free
mt76x2: change mt76x2_tx_complete routine signature
mt76: move mt76x2_tx_complete routine in mt76x02-lib module
mt76: move mt76x2u_remove_dma_hdr in mt76x02-lib module
mt76: move mt76x2u_tx_complete_skb in mt76x02-lib moudule
mt76: move mt76_qsel definition in dma.h
mt76: move mt76x2u_set_txinfo in mt76x02-lib module
mt76x0: introduce mt76x0_tx_prepare_skb routine
mt76: move mt76x2u_tx_status_data in mt76x02-lib module
mt76x0: disable usb rx bulk aggregation
mt76x0: mark device as running in mt76x0_start
mt76x0: simplify mt76_mac_process_rx signature
mt76x0: add mt76x0_queue_rx_skb routine
mt76x0: init mt76_driver_ops callbacks
mt76x0: use mt76_alloc_device for device allocation
mt76x0: unify tx/rx datapath with mt76x2u driver
mt76x0: stop stat workqueue at hw stop
mt76x0: set max fragments size
mt76x0: remove unused dma.c source file
mt76x0: remove unused stat work_queue
mt76x0: remove unused {tx/rx}_queue definitions
mt76x0: remove unused mt76x0_tx_status routine
mt76x0: remove unused endpoint definitions
mt76x0: remove unused stat_work
mt76x0: enable per-sta tx queueing
mt76x0: init hw capabilities
mt76: remove unused MT76_MORE_STATS state
mt76x0: remove mt76x0_stop_hardware routine
mt76: move mt76 rate definitions in mt76x02-lib module
mt76x0: alloc mcu buffers first in mt76x0_mcu_cmd_init
mt76x0: fix memory leak during hw probe
mt76x0: move stop related routines in mt76x0_mac_stop
mt76x0: move mt76x0_init_hardware in mt76x0_register_device
mt76x0: do not free/alloc buffers during suspend/resume
mt76x0: remove has_{2,5}ghz fields of mt76x0_eeprom_params
mt76x0: use mt76_register_device for device registration
mt76x0: run vco calibration for each channel configuration
mt76: move mt76x0 and mt76x2 mcu shared defs in mt76x02_mcu.h
mt76: add mt76_mcu_ops data structure for mcu related pointers
mt76: usb: use common helpers for mcu_alloc_msg()/mcu_send_msg()
mt76: usb: move mt76x02 mcu code in mt76x02-usb module
mt76: usb: move mt76u_skb_dma_info in mt76x02_usb_core.c
mt76x02: move TXD/RXD/MCU definitions in mt76x02_dma.h
mt76x02: add static qualifier to mt76x02_remove_dma_hdr
mt76: usb: remove skb check in mt76x{0,2}u mcu routines
mt76x2: use mt76_dev instead of mt76x2_dev in mt76x2_tx_queue_mcu
mt76x2: remove leftover mt76u_buf data structure in mt76x2_mcu
mt76: introduce mmio data structure in mt76_dev
mt76: move __iomem regs in mt76_mmio
mt76x2: use mt76_dev instead of mt76x2_dev in mt76x2_mcu_msg_send
mt76x2: use common helpers for mcu_alloc_msg()/mcu_send_msg()
mt76: unify firmware header between mt76x0 and mt76x2
mt76: move mt76{0,2} mcu shared code in mt76x02_mcu.c
mt76x2: move mt76x2 mcu shared code in mt76x2_mcu_common.c
mt76: move shared mcu_calibrate routine in mt76x02-lib module
mt76x2: move mt76x2_phy_tssi_compensate in mt76x2-common module
mt76x0: remove mcu source file
mt76x0: remove unused usb header file
mt76x0: usb: remove mt76_fw definition
Luca Coelho (6):
iwlwifi: remove unused TLC debugging commands
iwlwifi: mvm: remove duplicate if in iwl_mvm_setup_connection_keep()
iwlwifi: mvm: protect D0i3 code behind CONFIG_PM
iwlwifi: mvm: support new WoWLAN status FW API
iwlwifi: remove all occurrences of the FSF address paragraph
iwlwifi: fix devices with PCI Device ID 0x34F0 and 11ac RF modules
Martin Willi (1):
ath10k: schedule hardware restart if WMI command times out
Matt Chen (1):
iwlwifi: pcie: avoid unnecessary work if NIC is disconnected
Maya Erez (4):
wil6210: set edma variables only for Talyn-MB devices
wil6210: allocate rx reorder buffer only if rx reorder is enabled
wil6210: prevent usage of tx ring 0 for eDMA
wil6210: fix RX buffers release and unmap
Mordechay Goodstein (3):
iwlwifi: turn timestamp marker cmd off by default
iwlwifi: enable reading the value of delay in timestamp_marker cmd
iwlwifi: add 80211 hdr offset to trace data
Naftali Goldstein (4):
iwlwifi: runtime: avoid calling debugfs read functions more than once
iwlwifi: mvm: always init rs_fw with 20MHz bandwidth rates
iwlwifi: rs-fw: enable STBC in he correctly
iwlwifi: rs-fw: support dcm
Nathan Chancellor (4):
ipw2x00: Remove unnecessary parentheses
rsi: Remove unnecessary boolean condition
ath5k: Remove unused BUG_ON
rtlwifi: btcoex: Use proper enumerated types for Wi-Fi only interface
Rajat Jain (1):
iwlwifi: pcie: Fail fast if HW is inaccessible at probe
Rakesh Pillai (1):
ath10k: skip resetting rx filter for WCN3990
Rasmus Villemoes (1):
brcmfmac: fix wrong strnchr usage
Rosen Penev (1):
mt76x2u: Add support for Alfa AWUS036ACM
Sara Sharon (11):
iwlwifi: mvm: move he RX handling to a separate function
iwlwifi: mvm: move he RX handling to a separate function
iwlwifi: drop packets with bad status in CD
iwlwifi: mvm: fix BAR seq ctrl reporting
iwlwifi: mvm: avoid sending too many BARs
iwlwifi: pcie: set interrupt coalescing also for gen2
iwlwifi: fw: add a restart FW debug function
iwlwifi: fw: stop and start debugging using host command
iwlwifi: pcie: read correct prph address for newer devices
iwlwifi: mvm: use correct FIFO length
iwlwifi: pcie: add infrastructure for multiple debug buffers
Shahar S Matityahu (4):
iwlwifi: add d3 debug data support
iwlwifi: change monitor DMA to be coherent
iwlwifi: avoid code duplication in stopping fw debug data recording
iwlwifi: debug flow cleanup
Shaul Triebitz (4):
iwlwifi: iwlmvm: fix typo when checking for TX Beamforming
iwlwifi: mvm: enable sending HE_AIR_SNIFFER command via debugfs
iwlwifi: mvm: do not override amsdu size user settings
iwlwifi: pcie: set RB size according to user settings
Siva Rebbagondla (2):
rsi: fix memory alignment issue in ARM32 platforms
rsi: improve kernel thread handling to fix kernel panic
Stanislaw Gruszka (52):
mt76: unify wait_for_mac
mt76: rename mt76x2_regs.h
mt76: merge mt76x0/regs.h into mt76x02_regs.h
mt76: create new mt76x02-lib module for common mt76x{0,2} code
mt76: unify mac_get_key_info
mt76: add helpers for register access with mt76_dev struct
mt76: unify mac_shared_key_setup
mt76: unify mac_wcid_set_key
mt76: unify mac_wcid_setup
mt76: use mac_wcid_set_drop in mt76x0
mt76x0: use mt76_wcid_free in mt76x0
mt76: unify mt76x02_vif struct
mt76: unify sta structure part 1
mt76: unify sta structure part 2
mt76x0: initalize custom tx queues
mt76x0: use mt76x02_sta and mt76x02_tx_status
mt76x0: fix remove_interface
mt76: move wcid fields to common mt76_dev struct
mt76: unify sta_add / sta_remove
mt76: pratially unify add_interface
mt76: unify ampdu_action
mt76: unify set_key
mt76x0: remove empty sta_notify
mt76: unify AC to hw queue mapping
mt76: unify conf_tx
mt76x0: remove vif_mask
mt76: unify remove_interface
mt76: unify add_interface
mt76: unify sta_rate_tbl_update and related helpers
mt76: unify txwi and rxwi structures
mt76: unify load_tx_status
mt76: unify send_tx_status and related helpers
mt76: use mt76_rx_status in mt76x0
mt76: unify mac_process_rate
mt76x0: reserve enough space in mac80211
mt76: unify {insert/remove}_hdr_pad
mt76: partially unify filling txwi fields
mt76x0: trim rx skb to proper length
mt76x0: inital split between pci and usb
mt76x0: remove unused mt76x0_wcid
mt76x0: remove some usb specific code from mt76x0_register_device
mt76x0: make device allocation bus neutral
mt76: add usb implementation of {wr,rd}_rp
mt76: add rd_rp and wr_rp to bus_ops/mcu_ops
mt76x0: remove unused mt76x0_burst_read_regs
mt76x0: remove mt76x0_burst_write_regs()
mt76x0: usb: move firmware loading to usb.c
rt2800: move usb specific txdone/txstatus routines to rt2800lib
rt2800mmio: use txdone/txstatus routines from lib
rt2x00: do not check for txstatus timeout every time on tasklet
rt2x00: use different txstatus timeouts when flushing
rt2800: flush and txstatus rework for rt2800mmio
Sven Eckelmann (1):
ath10k: limit available channels via DT ieee80211-freq-limit
Takashi Iwai (1):
brcmsmac: Use kvmalloc() for ucode allocations
Tamizh chelvam (1):
ath10k: fix kernel panic by moving pci flush after napi_disable
Varsha Rao (4):
ath9k: Remove unnecessary parentheses
ath6kl: Remove unnecessary parentheses
brcmsmac: Remove extra parentheses
cw1200: Remove extra parentheses
YueHaibing (5):
wcn36xx: Use kmemdup instead of duplicating it in wcn36xx_smd_process_ptt_msg_rsp
wcn36xx: use dma_zalloc_coherent instead of allocator/memset
rsi: remove set but not used variables 'header_size' and 'tx_params'
brcmfmac: remove set but not used variables 'sfdoff' and 'pad_size'
qtnfmac: remove set but not used variable 'vif'
zhong jiang (1):
brcm80211: remove redundant condition check before debugfs_remove_recursive
drivers/net/wireless/ath/ath10k/ahb.c | 4 +-
drivers/net/wireless/ath/ath10k/ce.c | 24 +-
drivers/net/wireless/ath/ath10k/core.c | 17 +-
drivers/net/wireless/ath/ath10k/htt_rx.c | 4 +-
drivers/net/wireless/ath/ath10k/hw.h | 5 +
drivers/net/wireless/ath/ath10k/mac.c | 2 +
drivers/net/wireless/ath/ath10k/pci.c | 2 +-
drivers/net/wireless/ath/ath10k/wmi.c | 6 +
drivers/net/wireless/ath/ath5k/debug.c | 2 -
drivers/net/wireless/ath/ath6kl/main.c | 2 +-
drivers/net/wireless/ath/ath9k/ar5008_phy.c | 4 -
drivers/net/wireless/ath/ath9k/ath9k.h | 1 -
drivers/net/wireless/ath/ath9k/debug_sta.c | 2 +-
drivers/net/wireless/ath/ath9k/main.c | 12 +-
drivers/net/wireless/ath/ath9k/tx99.c | 9 -
drivers/net/wireless/ath/ath9k/xmit.c | 2 +-
drivers/net/wireless/ath/carl9170/tx.c | 4 +-
drivers/net/wireless/ath/wcn36xx/dxe.c | 19 +-
drivers/net/wireless/ath/wcn36xx/smd.c | 4 +-
drivers/net/wireless/ath/wil6210/cfg80211.c | 333 ++++-
drivers/net/wireless/ath/wil6210/debugfs.c | 47 +-
drivers/net/wireless/ath/wil6210/main.c | 13 +-
drivers/net/wireless/ath/wil6210/pcie_bus.c | 1 +
drivers/net/wireless/ath/wil6210/rx_reorder.c | 12 +-
drivers/net/wireless/ath/wil6210/txrx.c | 98 +-
drivers/net/wireless/ath/wil6210/txrx_edma.c | 26 +-
drivers/net/wireless/ath/wil6210/wil6210.h | 19 +
drivers/net/wireless/ath/wil6210/wmi.c | 350 ++++-
drivers/net/wireless/ath/wil6210/wmi.h | 2 +
drivers/net/wireless/broadcom/b43/b43.h | 2 +-
drivers/net/wireless/broadcom/b43/dma.c | 2 +-
drivers/net/wireless/broadcom/b43/main.c | 8 +-
drivers/net/wireless/broadcom/b43legacy/dma.c | 2 +-
.../broadcom/brcm80211/brcmfmac/cfg80211.c | 8 +
.../wireless/broadcom/brcm80211/brcmfmac/common.c | 4 +-
.../wireless/broadcom/brcm80211/brcmfmac/feature.c | 2 +-
.../wireless/broadcom/brcm80211/brcmfmac/pcie.c | 1 +
.../wireless/broadcom/brcm80211/brcmfmac/sdio.c | 5 +-
.../wireless/broadcom/brcm80211/brcmsmac/debug.c | 3 +-
.../broadcom/brcm80211/brcmsmac/mac80211_if.c | 6 +-
.../broadcom/brcm80211/brcmsmac/phy/phy_n.c | 4 +-
.../net/wireless/broadcom/brcm80211/brcmutil/d11.c | 34 +-
.../broadcom/brcm80211/include/brcmu_wifi.h | 3 +
drivers/net/wireless/intel/ipw2x00/ipw2200.c | 2 +-
drivers/net/wireless/intel/iwlwifi/cfg/1000.c | 4 -
drivers/net/wireless/intel/iwlwifi/cfg/2000.c | 4 -
drivers/net/wireless/intel/iwlwifi/cfg/22000.c | 71 +-
drivers/net/wireless/intel/iwlwifi/cfg/5000.c | 4 -
drivers/net/wireless/intel/iwlwifi/cfg/6000.c | 4 -
drivers/net/wireless/intel/iwlwifi/cfg/7000.c | 5 -
drivers/net/wireless/intel/iwlwifi/cfg/8000.c | 5 -
drivers/net/wireless/intel/iwlwifi/cfg/9000.c | 4 +-
drivers/net/wireless/intel/iwlwifi/dvm/agn.h | 5 -
drivers/net/wireless/intel/iwlwifi/dvm/calib.c | 5 -
drivers/net/wireless/intel/iwlwifi/dvm/calib.h | 5 -
drivers/net/wireless/intel/iwlwifi/dvm/commands.h | 5 -
drivers/net/wireless/intel/iwlwifi/dvm/debugfs.c | 5 -
drivers/net/wireless/intel/iwlwifi/dvm/dev.h | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/devices.c | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/led.c | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/led.h | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/lib.c | 5 -
drivers/net/wireless/intel/iwlwifi/dvm/mac80211.c | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/main.c | 6 +-
drivers/net/wireless/intel/iwlwifi/dvm/power.c | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/power.h | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/rs.c | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/rs.h | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/rx.c | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/rxon.c | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/scan.c | 5 -
drivers/net/wireless/intel/iwlwifi/dvm/sta.c | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/tt.c | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/tt.h | 4 -
drivers/net/wireless/intel/iwlwifi/dvm/tx.c | 5 -
drivers/net/wireless/intel/iwlwifi/dvm/ucode.c | 5 -
drivers/net/wireless/intel/iwlwifi/fw/acpi.c | 3 -
drivers/net/wireless/intel/iwlwifi/fw/acpi.h | 3 -
drivers/net/wireless/intel/iwlwifi/fw/api/coex.h | 3 +
.../net/wireless/intel/iwlwifi/fw/api/commands.h | 3 +-
drivers/net/wireless/intel/iwlwifi/fw/api/d3.h | 89 +-
.../net/wireless/intel/iwlwifi/fw/api/datapath.h | 5 +
drivers/net/wireless/intel/iwlwifi/fw/api/debug.h | 32 +
.../net/wireless/intel/iwlwifi/fw/api/mac-cfg.h | 49 +
drivers/net/wireless/intel/iwlwifi/fw/api/mac.h | 14 +
drivers/net/wireless/intel/iwlwifi/fw/api/power.h | 30 +-
drivers/net/wireless/intel/iwlwifi/fw/api/rs.h | 78 +-
drivers/net/wireless/intel/iwlwifi/fw/api/rx.h | 189 ++-
drivers/net/wireless/intel/iwlwifi/fw/api/scan.h | 1 +
drivers/net/wireless/intel/iwlwifi/fw/api/sta.h | 4 +-
drivers/net/wireless/intel/iwlwifi/fw/api/tx.h | 8 +-
drivers/net/wireless/intel/iwlwifi/fw/dbg.c | 337 ++---
drivers/net/wireless/intel/iwlwifi/fw/dbg.h | 97 +-
drivers/net/wireless/intel/iwlwifi/fw/debugfs.c | 124 +-
drivers/net/wireless/intel/iwlwifi/fw/debugfs.h | 3 -
drivers/net/wireless/intel/iwlwifi/fw/error-dump.h | 8 +-
drivers/net/wireless/intel/iwlwifi/fw/file.h | 32 +-
drivers/net/wireless/intel/iwlwifi/fw/img.h | 5 -
drivers/net/wireless/intel/iwlwifi/fw/notif-wait.c | 5 -
drivers/net/wireless/intel/iwlwifi/fw/notif-wait.h | 5 -
drivers/net/wireless/intel/iwlwifi/fw/runtime.h | 1 +
drivers/net/wireless/intel/iwlwifi/iwl-agn-hw.h | 5 -
drivers/net/wireless/intel/iwlwifi/iwl-config.h | 18 +-
.../net/wireless/intel/iwlwifi/iwl-context-info.h | 25 +-
drivers/net/wireless/intel/iwlwifi/iwl-csr.h | 6 +-
drivers/net/wireless/intel/iwlwifi/iwl-debug.c | 5 -
drivers/net/wireless/intel/iwlwifi/iwl-debug.h | 4 -
.../net/wireless/intel/iwlwifi/iwl-devtrace-data.h | 10 +-
.../net/wireless/intel/iwlwifi/iwl-devtrace-io.h | 4 -
.../wireless/intel/iwlwifi/iwl-devtrace-iwlwifi.h | 69 +-
.../net/wireless/intel/iwlwifi/iwl-devtrace-msg.h | 4 -
.../wireless/intel/iwlwifi/iwl-devtrace-ucode.h | 4 -
drivers/net/wireless/intel/iwlwifi/iwl-devtrace.c | 7 +-
drivers/net/wireless/intel/iwlwifi/iwl-devtrace.h | 22 +-
drivers/net/wireless/intel/iwlwifi/iwl-drv.c | 24 +-
drivers/net/wireless/intel/iwlwifi/iwl-drv.h | 5 -
.../net/wireless/intel/iwlwifi/iwl-eeprom-parse.c | 7 +-
.../net/wireless/intel/iwlwifi/iwl-eeprom-parse.h | 5 -
.../net/wireless/intel/iwlwifi/iwl-eeprom-read.c | 5 -
.../net/wireless/intel/iwlwifi/iwl-eeprom-read.h | 5 -
drivers/net/wireless/intel/iwlwifi/iwl-fh.h | 5 +-
drivers/net/wireless/intel/iwlwifi/iwl-io.c | 4 -
drivers/net/wireless/intel/iwlwifi/iwl-io.h | 4 -
drivers/net/wireless/intel/iwlwifi/iwl-modparams.h | 3 -
drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c | 5 -
drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.h | 5 -
drivers/net/wireless/intel/iwlwifi/iwl-op-mode.h | 7 +-
drivers/net/wireless/intel/iwlwifi/iwl-phy-db.c | 5 -
drivers/net/wireless/intel/iwlwifi/iwl-phy-db.h | 5 -
drivers/net/wireless/intel/iwlwifi/iwl-prph.h | 5 -
drivers/net/wireless/intel/iwlwifi/iwl-scd.h | 5 -
drivers/net/wireless/intel/iwlwifi/iwl-trans.c | 5 -
drivers/net/wireless/intel/iwlwifi/iwl-trans.h | 41 +-
drivers/net/wireless/intel/iwlwifi/mvm/binding.c | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/coex.c | 23 +-
drivers/net/wireless/intel/iwlwifi/mvm/constants.h | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/d3.c | 183 ++-
.../net/wireless/intel/iwlwifi/mvm/debugfs-vif.c | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c | 37 +-
drivers/net/wireless/intel/iwlwifi/mvm/debugfs.h | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/fw-api.h | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/fw.c | 28 +-
drivers/net/wireless/intel/iwlwifi/mvm/led.c | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c | 70 +-
drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c | 91 +-
drivers/net/wireless/intel/iwlwifi/mvm/mvm.h | 36 +-
drivers/net/wireless/intel/iwlwifi/mvm/nvm.c | 5 -
.../net/wireless/intel/iwlwifi/mvm/offloading.c | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/ops.c | 69 +-
drivers/net/wireless/intel/iwlwifi/mvm/power.c | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/quota.c | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/rs-fw.c | 35 +-
drivers/net/wireless/intel/iwlwifi/mvm/rs.c | 25 +-
drivers/net/wireless/intel/iwlwifi/mvm/rs.h | 4 +-
drivers/net/wireless/intel/iwlwifi/mvm/rx.c | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c | 721 ++++++----
drivers/net/wireless/intel/iwlwifi/mvm/scan.c | 26 +-
drivers/net/wireless/intel/iwlwifi/mvm/sf.c | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/sta.c | 30 +-
drivers/net/wireless/intel/iwlwifi/mvm/tdls.c | 3 -
drivers/net/wireless/intel/iwlwifi/mvm/testmode.h | 5 -
.../net/wireless/intel/iwlwifi/mvm/time-event.h | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/tof.c | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/tof.h | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/tt.c | 5 -
drivers/net/wireless/intel/iwlwifi/mvm/tx.c | 162 ++-
drivers/net/wireless/intel/iwlwifi/mvm/utils.c | 27 +-
.../wireless/intel/iwlwifi/pcie/ctxt-info-gen3.c | 4 +-
.../net/wireless/intel/iwlwifi/pcie/ctxt-info.c | 28 +-
drivers/net/wireless/intel/iwlwifi/pcie/drv.c | 60 +-
drivers/net/wireless/intel/iwlwifi/pcie/internal.h | 109 +-
drivers/net/wireless/intel/iwlwifi/pcie/rx.c | 20 +-
.../net/wireless/intel/iwlwifi/pcie/trans-gen2.c | 9 +-
drivers/net/wireless/intel/iwlwifi/pcie/trans.c | 267 ++--
drivers/net/wireless/intel/iwlwifi/pcie/tx-gen2.c | 182 ++-
drivers/net/wireless/intel/iwlwifi/pcie/tx.c | 93 +-
.../net/wireless/intersil/orinoco/orinoco_usb.c | 6 +-
drivers/net/wireless/marvell/libertas_tf/if_usb.c | 5 +-
drivers/net/wireless/marvell/mwifiex/usb.c | 13 +
drivers/net/wireless/mediatek/mt76/Kconfig | 29 +-
drivers/net/wireless/mediatek/mt76/Makefile | 10 +-
drivers/net/wireless/mediatek/mt76/dma.c | 16 +-
drivers/net/wireless/mediatek/mt76/dma.h | 51 +-
drivers/net/wireless/mediatek/mt76/mac80211.c | 6 +-
drivers/net/wireless/mediatek/mt76/mmio.c | 12 +-
drivers/net/wireless/mediatek/mt76/mt76.h | 94 +-
drivers/net/wireless/mediatek/mt76/mt76x0/Makefile | 13 +-
drivers/net/wireless/mediatek/mt76/mt76x0/core.c | 34 -
drivers/net/wireless/mediatek/mt76/mt76x0/dma.c | 522 -------
drivers/net/wireless/mediatek/mt76/mt76x0/eeprom.c | 26 +-
drivers/net/wireless/mediatek/mt76/mt76x0/eeprom.h | 3 -
drivers/net/wireless/mediatek/mt76/mt76x0/init.c | 400 ++----
drivers/net/wireless/mediatek/mt76/mt76x0/mac.c | 437 +-----
drivers/net/wireless/mediatek/mt76/mt76x0/mac.h | 136 +-
drivers/net/wireless/mediatek/mt76/mt76x0/main.c | 272 +---
drivers/net/wireless/mediatek/mt76/mt76x0/mcu.c | 656 ---------
drivers/net/wireless/mediatek/mt76/mt76x0/mcu.h | 61 +-
drivers/net/wireless/mediatek/mt76/mt76x0/mt76x0.h | 181 +--
drivers/net/wireless/mediatek/mt76/mt76x0/pci.c | 81 ++
drivers/net/wireless/mediatek/mt76/mt76x0/phy.c | 37 +-
drivers/net/wireless/mediatek/mt76/mt76x0/regs.h | 651 ---------
drivers/net/wireless/mediatek/mt76/mt76x0/trace.h | 14 +-
drivers/net/wireless/mediatek/mt76/mt76x0/tx.c | 245 +---
drivers/net/wireless/mediatek/mt76/mt76x0/usb.c | 382 +++--
drivers/net/wireless/mediatek/mt76/mt76x0/usb.h | 61 -
drivers/net/wireless/mediatek/mt76/mt76x0/util.c | 42 -
drivers/net/wireless/mediatek/mt76/mt76x02_dma.h | 60 +
drivers/net/wireless/mediatek/mt76/mt76x02_mac.c | 504 +++++++
drivers/net/wireless/mediatek/mt76/mt76x02_mac.h | 205 +++
drivers/net/wireless/mediatek/mt76/mt76x02_mcu.c | 213 +++
drivers/net/wireless/mediatek/mt76/mt76x02_mcu.h | 100 ++
.../mt76/{mt76x2_regs.h => mt76x02_regs.h} | 78 +-
.../mediatek/mt76/{mt76x2_dma.h => mt76x02_usb.h} | 22 +-
.../net/wireless/mediatek/mt76/mt76x02_usb_core.c | 72 +
.../net/wireless/mediatek/mt76/mt76x02_usb_mcu.c | 359 +++++
drivers/net/wireless/mediatek/mt76/mt76x02_util.c | 456 ++++++
drivers/net/wireless/mediatek/mt76/mt76x02_util.h | 54 +
drivers/net/wireless/mediatek/mt76/mt76x2.h | 88 +-
drivers/net/wireless/mediatek/mt76/mt76x2_common.c | 318 +----
drivers/net/wireless/mediatek/mt76/mt76x2_dma.c | 55 +-
drivers/net/wireless/mediatek/mt76/mt76x2_init.c | 27 +-
.../wireless/mediatek/mt76/mt76x2_init_common.c | 48 +-
drivers/net/wireless/mediatek/mt76/mt76x2_mac.c | 19 +-
drivers/net/wireless/mediatek/mt76/mt76x2_mac.h | 131 +-
.../net/wireless/mediatek/mt76/mt76x2_mac_common.c | 488 +------
drivers/net/wireless/mediatek/mt76/mt76x2_main.c | 93 +-
drivers/net/wireless/mediatek/mt76/mt76x2_mcu.c | 261 +---
drivers/net/wireless/mediatek/mt76/mt76x2_mcu.h | 68 +-
.../net/wireless/mediatek/mt76/mt76x2_mcu_common.c | 124 ++
drivers/net/wireless/mediatek/mt76/mt76x2_phy.c | 65 +-
.../net/wireless/mediatek/mt76/mt76x2_phy_common.c | 53 +-
drivers/net/wireless/mediatek/mt76/mt76x2_trace.h | 2 +-
drivers/net/wireless/mediatek/mt76/mt76x2_tx.c | 13 +-
.../net/wireless/mediatek/mt76/mt76x2_tx_common.c | 49 +-
drivers/net/wireless/mediatek/mt76/mt76x2_usb.c | 17 +-
drivers/net/wireless/mediatek/mt76/mt76x2u.h | 17 +-
drivers/net/wireless/mediatek/mt76/mt76x2u_core.c | 69 +-
drivers/net/wireless/mediatek/mt76/mt76x2u_init.c | 23 +-
drivers/net/wireless/mediatek/mt76/mt76x2u_mac.c | 2 +-
drivers/net/wireless/mediatek/mt76/mt76x2u_main.c | 47 +-
drivers/net/wireless/mediatek/mt76/mt76x2u_mcu.c | 218 +--
drivers/net/wireless/mediatek/mt76/mt76x2u_phy.c | 67 +-
drivers/net/wireless/mediatek/mt76/tx.c | 16 +
drivers/net/wireless/mediatek/mt76/usb.c | 105 +-
drivers/net/wireless/mediatek/mt76/usb_mcu.c | 204 +--
drivers/net/wireless/quantenna/qtnfmac/Makefile | 3 +-
drivers/net/wireless/quantenna/qtnfmac/bus.h | 5 +-
drivers/net/wireless/quantenna/qtnfmac/cfg80211.c | 3 -
drivers/net/wireless/quantenna/qtnfmac/pcie/pcie.c | 391 +++++
.../wireless/quantenna/qtnfmac/pcie/pcie_priv.h | 85 ++
.../wireless/quantenna/qtnfmac/pcie/pearl_pcie.c | 1262 +++++++++++++++++
.../{pearl/pcie_ipc.h => pcie/pearl_pcie_ipc.h} | 58 -
.../pcie_regs_pearl.h => pcie/pearl_pcie_regs.h} | 0
.../net/wireless/quantenna/qtnfmac/pearl/pcie.c | 1494 --------------------
.../quantenna/qtnfmac/pearl/pcie_bus_priv.h | 91 --
.../net/wireless/quantenna/qtnfmac/qtn_hw_ids.h | 14 +
drivers/net/wireless/ralink/rt2x00/rt2800lib.c | 154 ++
drivers/net/wireless/ralink/rt2x00/rt2800lib.h | 3 +
drivers/net/wireless/ralink/rt2x00/rt2800mmio.c | 277 ++--
drivers/net/wireless/ralink/rt2x00/rt2800mmio.h | 1 +
drivers/net/wireless/ralink/rt2x00/rt2800pci.c | 2 +-
drivers/net/wireless/ralink/rt2x00/rt2800usb.c | 143 +-
drivers/net/wireless/ralink/rt2x00/rt2x00.h | 3 +
drivers/net/wireless/ralink/rt2x00/rt2x00debug.c | 18 +-
drivers/net/wireless/ralink/rt2x00/rt2x00mac.c | 4 +
drivers/net/wireless/ralink/rt2x00/rt2x00queue.c | 2 +
.../net/wireless/realtek/rtl818x/rtl8187/leds.c | 2 +-
.../net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c | 2 +
.../realtek/rtlwifi/btcoexist/halbtcoutsrc.c | 6 +-
.../net/wireless/realtek/rtlwifi/rtl8188ee/hw.c | 6 +-
.../net/wireless/realtek/rtlwifi/rtl8723ae/hw.c | 4 +-
drivers/net/wireless/rsi/rsi_91x_hal.c | 4 -
drivers/net/wireless/rsi/rsi_91x_mac80211.c | 2 +-
drivers/net/wireless/rsi/rsi_91x_usb.c | 11 +-
drivers/net/wireless/rsi/rsi_common.h | 1 -
drivers/net/wireless/st/cw1200/txrx.c | 4 +-
276 files changed, 8620 insertions(+), 9949 deletions(-)
delete mode 100644 drivers/net/wireless/mediatek/mt76/mt76x0/core.c
delete mode 100644 drivers/net/wireless/mediatek/mt76/mt76x0/dma.c
delete mode 100644 drivers/net/wireless/mediatek/mt76/mt76x0/mcu.c
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x0/pci.c
delete mode 100644 drivers/net/wireless/mediatek/mt76/mt76x0/regs.h
delete mode 100644 drivers/net/wireless/mediatek/mt76/mt76x0/usb.h
delete mode 100644 drivers/net/wireless/mediatek/mt76/mt76x0/util.c
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x02_dma.h
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x02_mac.c
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x02_mac.h
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x02_mcu.c
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x02_mcu.h
rename drivers/net/wireless/mediatek/mt76/{mt76x2_regs.h => mt76x02_regs.h} (89%)
rename drivers/net/wireless/mediatek/mt76/{mt76x2_dma.h => mt76x02_usb.h} (58%)
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x02_usb_core.c
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x02_usb_mcu.c
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x02_util.c
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x02_util.h
create mode 100644 drivers/net/wireless/mediatek/mt76/mt76x2_mcu_common.c
create mode 100644 drivers/net/wireless/quantenna/qtnfmac/pcie/pcie.c
create mode 100644 drivers/net/wireless/quantenna/qtnfmac/pcie/pcie_priv.h
create mode 100644 drivers/net/wireless/quantenna/qtnfmac/pcie/pearl_pcie.c
rename drivers/net/wireless/quantenna/qtnfmac/{pearl/pcie_ipc.h => pcie/pearl_pcie_ipc.h} (68%)
rename drivers/net/wireless/quantenna/qtnfmac/{pearl/pcie_regs_pearl.h => pcie/pearl_pcie_regs.h} (100%)
delete mode 100644 drivers/net/wireless/quantenna/qtnfmac/pearl/pcie.c
delete mode 100644 drivers/net/wireless/quantenna/qtnfmac/pearl/pcie_bus_priv.h
^ permalink raw reply
* Re: [PATCH net-next] netfilter: xt_quota: fix the behavior of xt_quota module
From: Maciej Żenczykowski @ 2018-10-02 10:38 UTC (permalink / raw)
To: Pablo Neira Ayuso
Cc: Chenbo Feng, Linux NetDev, netfilter-devel, kernel-team,
Lorenzo Colitti, Chenbo Feng
In-Reply-To: <20181002101556.lpvn4kz7xgv2at3f@salvia>
> Well, you will need a kernel + userspace update anyway, right?
It's true you need new iptables userspace to *see* during dump and/or
manually *set* during restore the remain counter.
However, (and I believe Chenbo tested this) just a new kernel is
enough to fix the problem of modifications within the table resetting
the counter.
This is because the data gets copied out of kernel and back into
kernel by old iptables without any further modifications.
ie. the new kernel not clearing the field on copy to userspace and
honouring it on copy to kernel is sufficient.
So iptables-save | iptables-restore doesn't work, but iptables -A foo does.
(currently iptables -t X -{A,D} foo clears all xt_quota counters in
table X even when foo is utterly unrelated)
>> I mean: Instead of using atomic64_set() to set the counter to 1 once
>> we went over quota,
>
> incomplete sentence, sorry:
>
> I mean: Instead of using atomic64_set() to set the counter to 1 once
> we go overquota, we just keep updating 'consumed' bytes.
I guess it's a fair point that with a u64 we won't ever realistically
overflow the number of sent bytes, so this could be a running counter
of matched bytes...
and we don't even need to update it if it was over the quota when we
first looked at it, so we'll go over by at most # of cpus * max size
of gso packet bytes.
> ie. we don't express things in 'remaining bytes' logic, but we account
> for 'bytes we already consumed'. So we never go negative - I know
> understand what you mean about -1... I think we are each other
> thinking from our respective approach proposal.
I guess our decision was probably driven by xt_quota2 use on android
where infinite quota is often used as a temporary placeholder.
^ permalink raw reply
* Re: Regression: kernel 4.14 an later very slow with many ipsec tunnels
From: Wolfgang Walter @ 2018-10-02 17:34 UTC (permalink / raw)
To: Florian Westphal
Cc: Steffen Klassert, David Miller, netdev, linux-kernel, torvalds,
christophe.gouault
In-Reply-To: <20181002145616.pwdhbmafgsihbxvm@breakpoint.cc>
Am Dienstag, 2. Oktober 2018, 16:56:16 schrieb Florian Westphal:
> Wolfgang Walter <linux@stwm.de> wrote:
> > Since my last reply to this message I didn't get a reply: is there any
> > progress how to fix this performance regression I missed?
>
> Did you test/experiment with hthresh config option?
I did. It did not improve the situation.
I suppose that is because our masks range from /16 to /30 and excpecially have
for example /16 <=> /8 and vice versa.
When forwarding, every policy A => B also implies that you add a policy B =>
A.
I'm not familiar when the policy database is consulted, but I think it now has
to for every not encrypted paket, and for those all rules have to be
consulted. And unencrypted traffic is a large part of the traffic on that
router.
That is: for unencrypted traffic neither the buckets of the hash nor the
inexact list may be large.
>
> > Or are we stuck here with longterm kernel 4.9 for a long time?
>
> I'm experimenting with per-dst inexact lists in an rbtree but
> this will take time.
Hmm, I doubt that this is worth the effort. And certainly not that easy
correctly done, as it still would have to obey the original order of the rules
(their priority).
You may have a lot of rules of the form say
10.0.0.0/16 <=> 10.1.0.0/29 encrypt ....
10.0.0.0/16 <=> 10.1.0.8/29 encrypt ....
....
And things like that.
Also, you get something like that
10.0.1.0/24 <=> 10.0.2.0/29 allow
10.0.0.0/16 <=> 10.0.2.0/24 encrypt
0.0.0.0 <=> 10.0.2.0/16 block
And people may use source port and/or destination port or protocol
(tcp/udp/imcp) to further tailor there ruleset.
Here is the approach HiPAC took for packet classification
https://pdfs.semanticscholar.org/a0bb/9d31e2499fb659c9e0d9544072d2f3c25079.pdf
https://pdfs.semanticscholar.org/0dea/8ee87f596f200de2722cbe9480610dd1a0db.pdf
Regards,
--
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts
^ permalink raw reply
* Re: [PATCH net-next] netfilter: xt_quota: fix the behavior of xt_quota module
From: Pablo Neira Ayuso @ 2018-10-02 10:51 UTC (permalink / raw)
To: Maciej Żenczykowski
Cc: Chenbo Feng, Linux NetDev, netfilter-devel, kernel-team,
Lorenzo Colitti, Chenbo Feng
In-Reply-To: <CANP3RGf4pnPYiLoLNqULz-ELUT18qDctmi3kUw6qpkppwtcXmg@mail.gmail.com>
On Tue, Oct 02, 2018 at 03:38:24AM -0700, Maciej Żenczykowski wrote:
> > Well, you will need a kernel + userspace update anyway, right?
>
> It's true you need new iptables userspace to *see* during dump and/or
> manually *set* during restore the remain counter.
>
> However, (and I believe Chenbo tested this) just a new kernel is
> enough to fix the problem of modifications within the table resetting
> the counter.
> This is because the data gets copied out of kernel and back into
> kernel by old iptables without any further modifications.
> ie. the new kernel not clearing the field on copy to userspace and
> honouring it on copy to kernel is sufficient.
I see, Willem removed this behaviour in newer kernels. The private
area is now zeroed, is that what you mean right? So I guess this
cannot be done transparently.
Anyway, I think the --remain approach to fix this longstanding
problem from iptables :-).
> So iptables-save | iptables-restore doesn't work, but iptables -A foo does.
>
> (currently iptables -t X -{A,D} foo clears all xt_quota counters in
> table X even when foo is utterly unrelated)
>
> >> I mean: Instead of using atomic64_set() to set the counter to 1 once
> >> we went over quota,
> >
> > incomplete sentence, sorry:
> >
> > I mean: Instead of using atomic64_set() to set the counter to 1 once
> > we go overquota, we just keep updating 'consumed' bytes.
>
> I guess it's a fair point that with a u64 we won't ever realistically
> overflow the number of sent bytes, so this could be a running counter
> of matched bytes...
>
> and we don't even need to update it if it was over the quota when we
> first looked at it, so we'll go over by at most # of cpus * max size
> of gso packet bytes.
>
> > ie. we don't express things in 'remaining bytes' logic, but we account
> > for 'bytes we already consumed'. So we never go negative - I know
> > understand what you mean about -1... I think we are each other
> > thinking from our respective approach proposal.
>
> I guess our decision was probably driven by xt_quota2 use on android
> where infinite quota is often used as a temporary placeholder.
I see, thanks for explaining.
Thanks.
^ permalink raw reply
* Re: [PATCH net-next] netfilter: xt_quota: fix the behavior of xt_quota module
From: Pablo Neira Ayuso @ 2018-10-02 10:52 UTC (permalink / raw)
To: Maciej Żenczykowski
Cc: Chenbo Feng, Linux NetDev, netfilter-devel, kernel-team,
Lorenzo Colitti, Chenbo Feng
In-Reply-To: <20181002105125.uv7mcitvaalpjueo@salvia>
On Tue, Oct 02, 2018 at 12:51:25PM +0200, Pablo Neira Ayuso wrote:
> On Tue, Oct 02, 2018 at 03:38:24AM -0700, Maciej Żenczykowski wrote:
> > > Well, you will need a kernel + userspace update anyway, right?
> >
> > It's true you need new iptables userspace to *see* during dump and/or
> > manually *set* during restore the remain counter.
> >
> > However, (and I believe Chenbo tested this) just a new kernel is
> > enough to fix the problem of modifications within the table resetting
> > the counter.
> > This is because the data gets copied out of kernel and back into
> > kernel by old iptables without any further modifications.
> > ie. the new kernel not clearing the field on copy to userspace and
> > honouring it on copy to kernel is sufficient.
>
> I see, Willem removed this behaviour in newer kernels. The private
> area is now zeroed, is that what you mean right? So I guess this
> cannot be done transparently.
>
> Anyway, I think the --remain approach to fix this longstanding
> problem from iptables :-).
Argh, broken sentence: I mean, I think it's the way to go for
iptables.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox