* [PATCH bpf-next 1/3] bpf: Allow narrow loads with offset > 0
From: Andrey Ignatov @ 2018-11-07 1:23 UTC (permalink / raw)
To: netdev; +Cc: Andrey Ignatov, ast, daniel, yhs, kernel-team
In-Reply-To: <cover.1541553332.git.rdna@fb.com>
Currently BPF verifier allows narrow loads for a context field only with
offset zero. E.g. if there is a __u32 field then only the following
loads are permitted:
* off=0, size=1 (narrow);
* off=0, size=2 (narrow);
* off=0, size=4 (full).
On the other hand LLVM can generate a load with offset different than
zero that make sense from program logic point of view, but verifier
doesn't accept it.
E.g. tools/testing/selftests/bpf/sendmsg4_prog.c has code:
#define DST_IP4 0xC0A801FEU /* 192.168.1.254 */
...
if ((ctx->user_ip4 >> 24) == (bpf_htonl(DST_IP4) >> 24) &&
where ctx is struct bpf_sock_addr.
Some versions of LLVM can produce the following byte code for it:
8: 71 12 07 00 00 00 00 00 r2 = *(u8 *)(r1 + 7)
9: 67 02 00 00 18 00 00 00 r2 <<= 24
10: 18 03 00 00 00 00 00 fe 00 00 00 00 00 00 00 00 r3 = 4261412864 ll
12: 5d 32 07 00 00 00 00 00 if r2 != r3 goto +7 <LBB0_6>
where `*(u8 *)(r1 + 7)` means narrow load for ctx->user_ip4 with size=1
and offset=3 (7 - sizeof(ctx->user_family) = 3). This load is currently
rejected by verifier.
Verifier code that rejects such loads is in bpf_ctx_narrow_access_ok()
what means any is_valid_access implementation, that uses the function,
works this way, e.g. bpf_skb_is_valid_access() for __sk_buff or
sock_addr_is_valid_access() for bpf_sock_addr.
The patch makes such loads supported. Offset can be in [0; size_default)
but has to be multiple of load size. E.g. for __u32 field the following
loads are supported now:
* off=0, size=1 (narrow);
* off=1, size=1 (narrow);
* off=2, size=1 (narrow);
* off=3, size=1 (narrow);
* off=0, size=2 (narrow);
* off=2, size=2 (narrow);
* off=0, size=4 (full).
Reported-by: Yonghong Song <yhs@fb.com>
Signed-off-by: Andrey Ignatov <rdna@fb.com>
---
include/linux/filter.h | 16 +---------------
kernel/bpf/verifier.c | 19 +++++++++++++++----
2 files changed, 16 insertions(+), 19 deletions(-)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index de629b706d1d..cc17f5f32fbb 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -668,24 +668,10 @@ static inline u32 bpf_ctx_off_adjust_machine(u32 size)
return size;
}
-static inline bool bpf_ctx_narrow_align_ok(u32 off, u32 size_access,
- u32 size_default)
-{
- size_default = bpf_ctx_off_adjust_machine(size_default);
- size_access = bpf_ctx_off_adjust_machine(size_access);
-
-#ifdef __LITTLE_ENDIAN
- return (off & (size_default - 1)) == 0;
-#else
- return (off & (size_default - 1)) + size_access == size_default;
-#endif
-}
-
static inline bool
bpf_ctx_narrow_access_ok(u32 off, u32 size, u32 size_default)
{
- return bpf_ctx_narrow_align_ok(off, size, size_default) &&
- size <= size_default && (size & (size - 1)) == 0;
+ return size <= size_default && (size & (size - 1)) == 0;
}
#define bpf_classic_proglen(fprog) (fprog->len * sizeof(fprog->filter[0]))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 1971ca325fb4..fa592502568e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -5803,9 +5803,9 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
* we will apply proper mask to the result.
*/
is_narrower_load = size < ctx_field_size;
+ u32 size_default = bpf_ctx_off_adjust_machine(ctx_field_size);
+ u32 off = insn->off;
if (is_narrower_load) {
- u32 size_default = bpf_ctx_off_adjust_machine(ctx_field_size);
- u32 off = insn->off;
u8 size_code;
if (type == BPF_WRITE) {
@@ -5833,12 +5833,23 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
}
if (is_narrower_load && size < target_size) {
- if (ctx_field_size <= 4)
+ u8 shift = (off & (size_default - 1)) * 8;
+
+ if (ctx_field_size <= 4) {
+ if (shift)
+ insn_buf[cnt++] = BPF_ALU32_IMM(BPF_RSH,
+ insn->dst_reg,
+ shift);
insn_buf[cnt++] = BPF_ALU32_IMM(BPF_AND, insn->dst_reg,
(1 << size * 8) - 1);
- else
+ } else {
+ if (shift)
+ insn_buf[cnt++] = BPF_ALU64_IMM(BPF_RSH,
+ insn->dst_reg,
+ shift);
insn_buf[cnt++] = BPF_ALU64_IMM(BPF_AND, insn->dst_reg,
(1 << size * 8) - 1);
+ }
}
new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
--
2.17.1
^ permalink raw reply related
* [PATCH bpf-next 0/3] bpf: Allow narrow loads with offset > 0
From: Andrey Ignatov @ 2018-11-07 1:23 UTC (permalink / raw)
To: netdev; +Cc: Andrey Ignatov, ast, daniel, yhs, kernel-team
This patch set adds support for narrow loads with offset > 0 to BPF
verifier.
Patch 1 provides more details and is the main patch in the set.
Patches 2 and 3 add new test cases to test_verifier and test_sock_addr
selftests.
Andrey Ignatov (3):
bpf: Allow narrow loads with offset > 0
selftests/bpf: Test narrow loads with off > 0 in test_verifier
selftests/bpf: Test narrow loads with off > 0 for bpf_sock_addr
include/linux/filter.h | 16 +------
kernel/bpf/verifier.c | 19 ++++++--
tools/testing/selftests/bpf/test_sock_addr.c | 28 ++++++++++--
tools/testing/selftests/bpf/test_verifier.c | 48 ++++++++++++++++----
4 files changed, 78 insertions(+), 33 deletions(-)
--
2.17.1
^ permalink raw reply
* [PATCH net-next 7/7] nfp: flower: use the common netdev notifier
From: Jakub Kicinski @ 2018-11-07 1:07 UTC (permalink / raw)
To: davem; +Cc: netdev, oss-drivers, Jakub Kicinski
In-Reply-To: <20181107010734.29935-1-jakub.kicinski@netronome.com>
Use driver's common notifier for LAG and tunnel configuration.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: John Hurley <john.hurley@netronome.com>
---
.../ethernet/netronome/nfp/flower/lag_conf.c | 14 ++-----
.../net/ethernet/netronome/nfp/flower/main.c | 23 +++++++----
.../net/ethernet/netronome/nfp/flower/main.h | 10 +++--
.../netronome/nfp/flower/tunnel_conf.c | 38 ++-----------------
4 files changed, 30 insertions(+), 55 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c b/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c
index 22b75a519269..5db838f45694 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c
@@ -626,17 +626,13 @@ nfp_fl_lag_changels_event(struct nfp_fl_lag *lag, struct net_device *netdev,
schedule_delayed_work(&lag->work, NFP_FL_LAG_DELAY);
}
-static int
-nfp_fl_lag_netdev_event(struct notifier_block *nb, unsigned long event,
- void *ptr)
+int nfp_flower_lag_netdev_event(struct nfp_flower_priv *priv,
+ struct net_device *netdev,
+ unsigned long event, void *ptr)
{
- struct net_device *netdev;
- struct nfp_fl_lag *lag;
+ struct nfp_fl_lag *lag = &priv->nfp_lag;
int err;
- netdev = netdev_notifier_info_to_dev(ptr);
- lag = container_of(nb, struct nfp_fl_lag, lag_nb);
-
switch (event) {
case NETDEV_CHANGEUPPER:
err = nfp_fl_lag_changeupper_event(lag, ptr);
@@ -673,8 +669,6 @@ void nfp_flower_lag_init(struct nfp_fl_lag *lag)
/* 0 is a reserved batch version so increment to first valid value. */
nfp_fl_increment_version(lag);
-
- lag->lag_nb.notifier_call = nfp_fl_lag_netdev_event;
}
void nfp_flower_lag_cleanup(struct nfp_fl_lag *lag)
diff --git a/drivers/net/ethernet/netronome/nfp/flower/main.c b/drivers/net/ethernet/netronome/nfp/flower/main.c
index 3a54728d2ea6..2ad00773750f 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/main.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/main.c
@@ -661,23 +661,30 @@ static int nfp_flower_start(struct nfp_app *app)
err = nfp_flower_lag_reset(&app_priv->nfp_lag);
if (err)
return err;
-
- err = register_netdevice_notifier(&app_priv->nfp_lag.lag_nb);
- if (err)
- return err;
}
return nfp_tunnel_config_start(app);
}
static void nfp_flower_stop(struct nfp_app *app)
+{
+ nfp_tunnel_config_stop(app);
+}
+
+static int
+nfp_flower_netdev_event(struct nfp_app *app, struct net_device *netdev,
+ unsigned long event, void *ptr)
{
struct nfp_flower_priv *app_priv = app->priv;
+ int ret;
- if (app_priv->flower_ext_feats & NFP_FL_FEATS_LAG)
- unregister_netdevice_notifier(&app_priv->nfp_lag.lag_nb);
+ if (app_priv->flower_ext_feats & NFP_FL_FEATS_LAG) {
+ ret = nfp_flower_lag_netdev_event(app_priv, netdev, event, ptr);
+ if (ret & NOTIFY_STOP_MASK)
+ return ret;
+ }
- nfp_tunnel_config_stop(app);
+ return nfp_tunnel_mac_event_handler(app, netdev, event, ptr);
}
const struct nfp_app_type app_flower = {
@@ -708,6 +715,8 @@ const struct nfp_app_type app_flower = {
.start = nfp_flower_start,
.stop = nfp_flower_stop,
+ .netdev_event = nfp_flower_netdev_event,
+
.ctrl_msg_rx = nfp_flower_cmsg_rx,
.sriov_enable = nfp_flower_sriov_enable,
diff --git a/drivers/net/ethernet/netronome/nfp/flower/main.h b/drivers/net/ethernet/netronome/nfp/flower/main.h
index 90045bab95bf..0f6f1675f6f1 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/main.h
+++ b/drivers/net/ethernet/netronome/nfp/flower/main.h
@@ -72,7 +72,6 @@ struct nfp_mtu_conf {
/**
* struct nfp_fl_lag - Flower APP priv data for link aggregation
- * @lag_nb: Notifier to track master/slave events
* @work: Work queue for writing configs to the HW
* @lock: Lock to protect lag_group_list
* @group_list: List of all master/slave groups offloaded
@@ -85,7 +84,6 @@ struct nfp_mtu_conf {
* retransmission
*/
struct nfp_fl_lag {
- struct notifier_block lag_nb;
struct delayed_work work;
struct mutex lock;
struct list_head group_list;
@@ -126,7 +124,6 @@ struct nfp_fl_lag {
* @nfp_neigh_off_lock: Lock for the neighbour address list
* @nfp_mac_off_ids: IDA to manage id assignment for offloaded macs
* @nfp_mac_off_count: Number of MACs in address list
- * @nfp_tun_mac_nb: Notifier to monitor link state
* @nfp_tun_neigh_nb: Notifier to monitor neighbour state
* @reify_replies: atomically stores the number of replies received
* from firmware for repr reify
@@ -160,7 +157,6 @@ struct nfp_flower_priv {
spinlock_t nfp_neigh_off_lock;
struct ida nfp_mac_off_ids;
int nfp_mac_off_count;
- struct notifier_block nfp_tun_mac_nb;
struct notifier_block nfp_tun_neigh_nb;
atomic_t reify_replies;
wait_queue_head_t reify_wait_queue;
@@ -252,6 +248,9 @@ void nfp_flower_rx_flow_stats(struct nfp_app *app, struct sk_buff *skb);
int nfp_tunnel_config_start(struct nfp_app *app);
void nfp_tunnel_config_stop(struct nfp_app *app);
+int nfp_tunnel_mac_event_handler(struct nfp_app *app,
+ struct net_device *netdev,
+ unsigned long event, void *ptr);
void nfp_tunnel_write_macs(struct nfp_app *app);
void nfp_tunnel_del_ipv4_off(struct nfp_app *app, __be32 ipv4);
void nfp_tunnel_add_ipv4_off(struct nfp_app *app, __be32 ipv4);
@@ -262,6 +261,9 @@ int nfp_flower_setup_tc_egress_cb(enum tc_setup_type type, void *type_data,
void nfp_flower_lag_init(struct nfp_fl_lag *lag);
void nfp_flower_lag_cleanup(struct nfp_fl_lag *lag);
int nfp_flower_lag_reset(struct nfp_fl_lag *lag);
+int nfp_flower_lag_netdev_event(struct nfp_flower_priv *priv,
+ struct net_device *netdev,
+ unsigned long event, void *ptr);
bool nfp_flower_lag_unprocessed_msg(struct nfp_app *app, struct sk_buff *skb);
int nfp_flower_lag_populate_pre_action(struct nfp_app *app,
struct net_device *master,
diff --git a/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c b/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c
index a3a44f1187d3..fb6442d820b5 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c
@@ -652,29 +652,16 @@ static void nfp_tun_add_to_mac_offload_list(struct net_device *netdev,
mutex_unlock(&priv->nfp_mac_off_lock);
}
-static int nfp_tun_mac_event_handler(struct notifier_block *nb,
- unsigned long event, void *ptr)
+int nfp_tunnel_mac_event_handler(struct nfp_app *app,
+ struct net_device *netdev,
+ unsigned long event, void *ptr)
{
- struct nfp_flower_priv *app_priv;
- struct net_device *netdev;
- struct nfp_app *app;
-
if (event == NETDEV_DOWN || event == NETDEV_UNREGISTER) {
- app_priv = container_of(nb, struct nfp_flower_priv,
- nfp_tun_mac_nb);
- app = app_priv->app;
- netdev = netdev_notifier_info_to_dev(ptr);
-
/* If non-nfp netdev then free its offload index. */
if (nfp_tun_is_netdev_to_offload(netdev))
nfp_tun_del_mac_idx(app, netdev->ifindex);
} else if (event == NETDEV_UP || event == NETDEV_CHANGEADDR ||
event == NETDEV_REGISTER) {
- app_priv = container_of(nb, struct nfp_flower_priv,
- nfp_tun_mac_nb);
- app = app_priv->app;
- netdev = netdev_notifier_info_to_dev(ptr);
-
nfp_tun_add_to_mac_offload_list(netdev, app);
/* Force a list write to keep NFP up to date. */
@@ -686,13 +673,11 @@ static int nfp_tun_mac_event_handler(struct notifier_block *nb,
int nfp_tunnel_config_start(struct nfp_app *app)
{
struct nfp_flower_priv *priv = app->priv;
- int err;
/* Initialise priv data for MAC offloading. */
priv->nfp_mac_off_count = 0;
mutex_init(&priv->nfp_mac_off_lock);
INIT_LIST_HEAD(&priv->nfp_mac_off_list);
- priv->nfp_tun_mac_nb.notifier_call = nfp_tun_mac_event_handler;
mutex_init(&priv->nfp_mac_index_lock);
INIT_LIST_HEAD(&priv->nfp_mac_index_list);
ida_init(&priv->nfp_mac_off_ids);
@@ -706,21 +691,7 @@ int nfp_tunnel_config_start(struct nfp_app *app)
INIT_LIST_HEAD(&priv->nfp_neigh_off_list);
priv->nfp_tun_neigh_nb.notifier_call = nfp_tun_neigh_event_handler;
- err = register_netdevice_notifier(&priv->nfp_tun_mac_nb);
- if (err)
- goto err_free_mac_ida;
-
- err = register_netevent_notifier(&priv->nfp_tun_neigh_nb);
- if (err)
- goto err_unreg_mac_nb;
-
- return 0;
-
-err_unreg_mac_nb:
- unregister_netdevice_notifier(&priv->nfp_tun_mac_nb);
-err_free_mac_ida:
- ida_destroy(&priv->nfp_mac_off_ids);
- return err;
+ return register_netevent_notifier(&priv->nfp_tun_neigh_nb);
}
void nfp_tunnel_config_stop(struct nfp_app *app)
@@ -732,7 +703,6 @@ void nfp_tunnel_config_stop(struct nfp_app *app)
struct nfp_ipv4_addr_entry *ip_entry;
struct list_head *ptr, *storage;
- unregister_netdevice_notifier(&priv->nfp_tun_mac_nb);
unregister_netevent_notifier(&priv->nfp_tun_neigh_nb);
/* Free any memory that may be occupied by MAC list. */
--
2.17.1
^ permalink raw reply related
* [PATCH net-next 6/7] nfp: register a notifier handler in a central location for the device
From: Jakub Kicinski @ 2018-11-07 1:07 UTC (permalink / raw)
To: davem; +Cc: netdev, oss-drivers, Jakub Kicinski
In-Reply-To: <20181107010734.29935-1-jakub.kicinski@netronome.com>
Code interested in networking events registers its own notifier
handlers. Create one device-wide notifier instance.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: John Hurley <john.hurley@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_app.c | 47 ++++++++++++++++++++
drivers/net/ethernet/netronome/nfp/nfp_app.h | 25 +++++------
2 files changed, 57 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_app.c b/drivers/net/ethernet/netronome/nfp/nfp_app.c
index 68a0991aac22..4a1b8f79e731 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_app.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_app.c
@@ -136,6 +136,53 @@ nfp_app_reprs_set(struct nfp_app *app, enum nfp_repr_type type,
return old;
}
+static int
+nfp_app_netdev_event(struct notifier_block *nb, unsigned long event, void *ptr)
+{
+ struct net_device *netdev;
+ struct nfp_app *app;
+
+ netdev = netdev_notifier_info_to_dev(ptr);
+ app = container_of(nb, struct nfp_app, netdev_nb);
+
+ if (app->type->netdev_event)
+ return app->type->netdev_event(app, netdev, event, ptr);
+ return NOTIFY_DONE;
+}
+
+int nfp_app_start(struct nfp_app *app, struct nfp_net *ctrl)
+{
+ int err;
+
+ app->ctrl = ctrl;
+
+ if (app->type->start) {
+ err = app->type->start(app);
+ if (err)
+ return err;
+ }
+
+ app->netdev_nb.notifier_call = nfp_app_netdev_event;
+ err = register_netdevice_notifier(&app->netdev_nb);
+ if (err)
+ goto err_app_stop;
+
+ return 0;
+
+err_app_stop:
+ if (app->type->stop)
+ app->type->stop(app);
+ return err;
+}
+
+void nfp_app_stop(struct nfp_app *app)
+{
+ unregister_netdevice_notifier(&app->netdev_nb);
+
+ if (app->type->stop)
+ app->type->stop(app);
+}
+
struct nfp_app *nfp_app_alloc(struct nfp_pf *pf, enum nfp_app_id id)
{
struct nfp_app *app;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_app.h b/drivers/net/ethernet/netronome/nfp/nfp_app.h
index 4d6ecf99b1cc..d578d856a009 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_app.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_app.h
@@ -69,6 +69,7 @@ extern const struct nfp_app_type app_abm;
* @port_get_stats_strings: get strings for extra statistics
* @start: start application logic
* @stop: stop application logic
+ * @netdev_event: Netdevice notifier event
* @ctrl_msg_rx: control message handler
* @ctrl_msg_rx_raw: handler for control messages from data queues
* @setup_tc: setup TC ndo
@@ -122,6 +123,9 @@ struct nfp_app_type {
int (*start)(struct nfp_app *app);
void (*stop)(struct nfp_app *app);
+ int (*netdev_event)(struct nfp_app *app, struct net_device *netdev,
+ unsigned long event, void *ptr);
+
void (*ctrl_msg_rx)(struct nfp_app *app, struct sk_buff *skb);
void (*ctrl_msg_rx_raw)(struct nfp_app *app, const void *data,
unsigned int len);
@@ -151,6 +155,7 @@ struct nfp_app_type {
* @reprs: array of pointers to representors
* @type: pointer to const application ops and info
* @ctrl_mtu: MTU to set on the control vNIC (set in .init())
+ * @netdev_nb: Netdevice notifier block
* @priv: app-specific priv data
*/
struct nfp_app {
@@ -163,6 +168,9 @@ struct nfp_app {
const struct nfp_app_type *type;
unsigned int ctrl_mtu;
+
+ struct notifier_block netdev_nb;
+
void *priv;
};
@@ -264,21 +272,6 @@ nfp_app_repr_change_mtu(struct nfp_app *app, struct net_device *netdev,
return app->type->repr_change_mtu(app, netdev, new_mtu);
}
-static inline int nfp_app_start(struct nfp_app *app, struct nfp_net *ctrl)
-{
- app->ctrl = ctrl;
- if (!app->type->start)
- return 0;
- return app->type->start(app);
-}
-
-static inline void nfp_app_stop(struct nfp_app *app)
-{
- if (!app->type->stop)
- return;
- app->type->stop(app);
-}
-
static inline const char *nfp_app_name(struct nfp_app *app)
{
if (!app)
@@ -430,6 +423,8 @@ nfp_app_ctrl_msg_alloc(struct nfp_app *app, unsigned int size, gfp_t priority);
struct nfp_app *nfp_app_alloc(struct nfp_pf *pf, enum nfp_app_id id);
void nfp_app_free(struct nfp_app *app);
+int nfp_app_start(struct nfp_app *app, struct nfp_net *ctrl);
+void nfp_app_stop(struct nfp_app *app);
/* Callbacks shared between apps */
--
2.17.1
^ permalink raw reply related
* [PATCH net-next 5/7] nfp: flower: make nfp_fl_lag_changels_event() void
From: Jakub Kicinski @ 2018-11-07 1:07 UTC (permalink / raw)
To: davem; +Cc: netdev, oss-drivers, Jakub Kicinski
In-Reply-To: <20181107010734.29935-1-jakub.kicinski@netronome.com>
nfp_fl_lag_changels_event() never fails, and therefore we would
never return NOTIFY_BAD for NETDEV_CHANGELOWERSTATE. Make this
clearer by changing nfp_fl_lag_changels_event()'s return type
to void.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: John Hurley <john.hurley@netronome.com>
---
.../net/ethernet/netronome/nfp/flower/lag_conf.c | 13 +++++--------
1 file changed, 5 insertions(+), 8 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c b/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c
index dc060748b33b..22b75a519269 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c
@@ -582,7 +582,7 @@ nfp_fl_lag_changeupper_event(struct nfp_fl_lag *lag,
return 0;
}
-static int
+static void
nfp_fl_lag_changels_event(struct nfp_fl_lag *lag, struct net_device *netdev,
struct netdev_notifier_changelowerstate_info *info)
{
@@ -593,18 +593,18 @@ nfp_fl_lag_changels_event(struct nfp_fl_lag *lag, struct net_device *netdev,
unsigned long *flags;
if (!netif_is_lag_port(netdev) || !nfp_netdev_is_nfp_repr(netdev))
- return 0;
+ return;
lag_lower_info = info->lower_state_info;
if (!lag_lower_info)
- return 0;
+ return;
priv = container_of(lag, struct nfp_flower_priv, nfp_lag);
repr = netdev_priv(netdev);
/* Verify that the repr is associated with this app. */
if (repr->app != priv->app)
- return 0;
+ return;
repr_priv = repr->app_priv;
flags = &repr_priv->lag_port_flags;
@@ -624,7 +624,6 @@ nfp_fl_lag_changels_event(struct nfp_fl_lag *lag, struct net_device *netdev,
mutex_unlock(&lag->lock);
schedule_delayed_work(&lag->work, NFP_FL_LAG_DELAY);
- return 0;
}
static int
@@ -645,9 +644,7 @@ nfp_fl_lag_netdev_event(struct notifier_block *nb, unsigned long event,
return NOTIFY_BAD;
return NOTIFY_OK;
case NETDEV_CHANGELOWERSTATE:
- err = nfp_fl_lag_changels_event(lag, netdev, ptr);
- if (err)
- return NOTIFY_BAD;
+ nfp_fl_lag_changels_event(lag, netdev, ptr);
return NOTIFY_OK;
case NETDEV_UNREGISTER:
nfp_fl_lag_schedule_group_delete(lag, netdev);
--
2.17.1
^ permalink raw reply related
* [PATCH net-next 4/7] nfp: flower: don't try to nack device unregister events
From: Jakub Kicinski @ 2018-11-07 1:07 UTC (permalink / raw)
To: davem; +Cc: netdev, oss-drivers, Jakub Kicinski
In-Reply-To: <20181107010734.29935-1-jakub.kicinski@netronome.com>
Returning an error from a notifier means we want to veto the change.
We shouldn't veto NETDEV_UNREGISTER just because we couldn't find
the tracking info for given master.
I can't seem to find a way to trigger this unless we have some
other bug, so it's probably not fix-worthy.
While at it move the checking if the netdev really is of interest
into the handling functions, like we do for other events.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: John Hurley <john.hurley@netronome.com>
---
.../ethernet/netronome/nfp/flower/lag_conf.c | 21 +++++++++++--------
1 file changed, 12 insertions(+), 9 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c b/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c
index 81dcf5b318ba..dc060748b33b 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/lag_conf.c
@@ -472,17 +472,25 @@ nfp_fl_lag_schedule_group_remove(struct nfp_fl_lag *lag,
schedule_delayed_work(&lag->work, NFP_FL_LAG_DELAY);
}
-static int
+static void
nfp_fl_lag_schedule_group_delete(struct nfp_fl_lag *lag,
struct net_device *master)
{
struct nfp_fl_lag_group *group;
+ struct nfp_flower_priv *priv;
+
+ priv = container_of(lag, struct nfp_flower_priv, nfp_lag);
+
+ if (!netif_is_bond_master(master))
+ return;
mutex_lock(&lag->lock);
group = nfp_fl_lag_find_group_for_master_with_lag(lag, master);
if (!group) {
mutex_unlock(&lag->lock);
- return -ENOENT;
+ nfp_warn(priv->app->cpp, "untracked bond got unregistered %s\n",
+ netdev_name(master));
+ return;
}
group->to_remove = true;
@@ -490,7 +498,6 @@ nfp_fl_lag_schedule_group_delete(struct nfp_fl_lag *lag,
mutex_unlock(&lag->lock);
schedule_delayed_work(&lag->work, NFP_FL_LAG_DELAY);
- return 0;
}
static int
@@ -643,12 +650,8 @@ nfp_fl_lag_netdev_event(struct notifier_block *nb, unsigned long event,
return NOTIFY_BAD;
return NOTIFY_OK;
case NETDEV_UNREGISTER:
- if (netif_is_bond_master(netdev)) {
- err = nfp_fl_lag_schedule_group_delete(lag, netdev);
- if (err)
- return NOTIFY_BAD;
- return NOTIFY_OK;
- }
+ nfp_fl_lag_schedule_group_delete(lag, netdev);
+ return NOTIFY_OK;
}
return NOTIFY_DONE;
--
2.17.1
^ permalink raw reply related
* [PATCH net-next 3/7] nfp: flower: remove unnecessary iteration over devices
From: Jakub Kicinski @ 2018-11-07 1:07 UTC (permalink / raw)
To: davem; +Cc: netdev, oss-drivers, Jakub Kicinski
In-Reply-To: <20181107010734.29935-1-jakub.kicinski@netronome.com>
For flower tunnel offloads FW has to be informed about MAC addresses
of tunnel devices. We use a netdev notifier to keep track of these
addresses.
Remove unnecessary loop over netdevices after notifier is registered.
The intention of the loop was to catch devices which already existed
on the system before nfp driver got loaded, but netdev notifier will
replay NETDEV_REGISTER events.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: John Hurley <john.hurley@netronome.com>
---
drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c | 7 -------
1 file changed, 7 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c b/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c
index 8e5bec04d1f9..a3a44f1187d3 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c
@@ -686,7 +686,6 @@ static int nfp_tun_mac_event_handler(struct notifier_block *nb,
int nfp_tunnel_config_start(struct nfp_app *app)
{
struct nfp_flower_priv *priv = app->priv;
- struct net_device *netdev;
int err;
/* Initialise priv data for MAC offloading. */
@@ -715,12 +714,6 @@ int nfp_tunnel_config_start(struct nfp_app *app)
if (err)
goto err_unreg_mac_nb;
- /* Parse netdevs already registered for MACs that need offloaded. */
- rtnl_lock();
- for_each_netdev(&init_net, netdev)
- nfp_tun_add_to_mac_offload_list(netdev, app);
- rtnl_unlock();
-
return 0;
err_unreg_mac_nb:
--
2.17.1
^ permalink raw reply related
* [PATCH net-next 2/7] nfp: flower: add ipv6 set flow label and hop limit offload
From: Jakub Kicinski @ 2018-11-07 1:07 UTC (permalink / raw)
To: davem; +Cc: netdev, oss-drivers, Pieter Jansen van Vuuren
In-Reply-To: <20181107010734.29935-1-jakub.kicinski@netronome.com>
From: Pieter Jansen van Vuuren <pieter.jansenvanvuuren@netronome.com>
Add ipv6 set flow label and hop limit action offload. Since pedit sets
headers per 4 byte word, we need to ensure that setting either version,
priority, payload_len or nexthdr does not get offloaded.
Signed-off-by: Pieter Jansen van Vuuren <pieter.jansenvanvuuren@netronome.com>
Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
.../ethernet/netronome/nfp/flower/action.c | 65 +++++++++++++++++--
.../net/ethernet/netronome/nfp/flower/cmsg.h | 14 ++++
2 files changed, 75 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/flower/action.c b/drivers/net/ethernet/netronome/nfp/flower/action.c
index b79b924ef56d..cfea8f790f95 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/action.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/action.c
@@ -476,12 +476,57 @@ nfp_fl_set_ip6_helper(int opcode_tag, u8 word, __be32 exact, __be32 mask,
ip6->head.len_lw = sizeof(*ip6) >> NFP_FL_LW_SIZ;
}
+struct ipv6_hop_limit_word {
+ __be16 payload_len;
+ u8 nexthdr;
+ u8 hop_limit;
+};
+
+static int
+nfp_fl_set_ip6_hop_limit_flow_label(u32 off, __be32 exact, __be32 mask,
+ struct nfp_fl_set_ipv6_tc_hl_fl *ip_hl_fl)
+{
+ struct ipv6_hop_limit_word *fl_hl_mask;
+ struct ipv6_hop_limit_word *fl_hl;
+
+ switch (off) {
+ case offsetof(struct ipv6hdr, payload_len):
+ fl_hl_mask = (struct ipv6_hop_limit_word *)&mask;
+ fl_hl = (struct ipv6_hop_limit_word *)&exact;
+
+ if (fl_hl_mask->nexthdr || fl_hl_mask->payload_len)
+ return -EOPNOTSUPP;
+
+ ip_hl_fl->ipv6_hop_limit_mask |= fl_hl_mask->hop_limit;
+ ip_hl_fl->ipv6_hop_limit &= ~fl_hl_mask->hop_limit;
+ ip_hl_fl->ipv6_hop_limit |= fl_hl->hop_limit &
+ fl_hl_mask->hop_limit;
+ break;
+ case round_down(offsetof(struct ipv6hdr, flow_lbl), 4):
+ if (mask & ~IPV6_FLOW_LABEL_MASK ||
+ exact & ~IPV6_FLOW_LABEL_MASK)
+ return -EOPNOTSUPP;
+
+ ip_hl_fl->ipv6_label_mask |= mask;
+ ip_hl_fl->ipv6_label &= ~mask;
+ ip_hl_fl->ipv6_label |= exact & mask;
+ break;
+ }
+
+ ip_hl_fl->head.jump_id = NFP_FL_ACTION_OPCODE_SET_IPV6_TC_HL_FL;
+ ip_hl_fl->head.len_lw = sizeof(*ip_hl_fl) >> NFP_FL_LW_SIZ;
+
+ return 0;
+}
+
static int
nfp_fl_set_ip6(const struct tc_action *action, int idx, u32 off,
struct nfp_fl_set_ipv6_addr *ip_dst,
- struct nfp_fl_set_ipv6_addr *ip_src)
+ struct nfp_fl_set_ipv6_addr *ip_src,
+ struct nfp_fl_set_ipv6_tc_hl_fl *ip_hl_fl)
{
__be32 exact, mask;
+ int err = 0;
u8 word;
/* We are expecting tcf_pedit to return a big endian value */
@@ -492,7 +537,8 @@ nfp_fl_set_ip6(const struct tc_action *action, int idx, u32 off,
return -EOPNOTSUPP;
if (off < offsetof(struct ipv6hdr, saddr)) {
- return -EOPNOTSUPP;
+ err = nfp_fl_set_ip6_hop_limit_flow_label(off, exact, mask,
+ ip_hl_fl);
} else if (off < offsetof(struct ipv6hdr, daddr)) {
word = (off - offsetof(struct ipv6hdr, saddr)) / sizeof(exact);
nfp_fl_set_ip6_helper(NFP_FL_ACTION_OPCODE_SET_IPV6_SRC, word,
@@ -506,7 +552,7 @@ nfp_fl_set_ip6(const struct tc_action *action, int idx, u32 off,
return -EOPNOTSUPP;
}
- return 0;
+ return err;
}
static int
@@ -557,6 +603,7 @@ nfp_fl_pedit(const struct tc_action *action, struct tc_cls_flower_offload *flow,
char *nfp_action, int *a_len, u32 *csum_updated)
{
struct nfp_fl_set_ipv6_addr set_ip6_dst, set_ip6_src;
+ struct nfp_fl_set_ipv6_tc_hl_fl set_ip6_tc_hl_fl;
struct nfp_fl_set_ip4_ttl_tos set_ip_ttl_tos;
struct nfp_fl_set_ip4_addrs set_ip_addr;
struct nfp_fl_set_tport set_tport;
@@ -567,6 +614,7 @@ nfp_fl_pedit(const struct tc_action *action, struct tc_cls_flower_offload *flow,
u32 offset, cmd;
u8 ip_proto = 0;
+ memset(&set_ip6_tc_hl_fl, 0, sizeof(set_ip6_tc_hl_fl));
memset(&set_ip_ttl_tos, 0, sizeof(set_ip_ttl_tos));
memset(&set_ip6_dst, 0, sizeof(set_ip6_dst));
memset(&set_ip6_src, 0, sizeof(set_ip6_src));
@@ -593,7 +641,7 @@ nfp_fl_pedit(const struct tc_action *action, struct tc_cls_flower_offload *flow,
break;
case TCA_PEDIT_KEY_EX_HDR_TYPE_IP6:
err = nfp_fl_set_ip6(action, idx, offset, &set_ip6_dst,
- &set_ip6_src);
+ &set_ip6_src, &set_ip6_tc_hl_fl);
break;
case TCA_PEDIT_KEY_EX_HDR_TYPE_TCP:
err = nfp_fl_set_tport(action, idx, offset, &set_tport,
@@ -644,6 +692,15 @@ nfp_fl_pedit(const struct tc_action *action, struct tc_cls_flower_offload *flow,
*csum_updated |= TCA_CSUM_UPDATE_FLAG_IPV4HDR |
nfp_fl_csum_l4_to_flag(ip_proto);
}
+ if (set_ip6_tc_hl_fl.head.len_lw) {
+ nfp_action += act_size;
+ act_size = sizeof(set_ip6_tc_hl_fl);
+ memcpy(nfp_action, &set_ip6_tc_hl_fl, act_size);
+ *a_len += act_size;
+
+ /* Hardware will automatically fix TCP/UDP checksum. */
+ *csum_updated |= nfp_fl_csum_l4_to_flag(ip_proto);
+ }
if (set_ip6_dst.head.len_lw && set_ip6_src.head.len_lw) {
/* TC compiles set src and dst IPv6 address as a single action,
* the hardware requires this to be 2 separate actions.
diff --git a/drivers/net/ethernet/netronome/nfp/flower/cmsg.h b/drivers/net/ethernet/netronome/nfp/flower/cmsg.h
index a00f45b5e16c..3e391555e191 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/cmsg.h
+++ b/drivers/net/ethernet/netronome/nfp/flower/cmsg.h
@@ -68,6 +68,7 @@
#define NFP_FL_ACTION_OPCODE_SET_IPV4_TTL_TOS 10
#define NFP_FL_ACTION_OPCODE_SET_IPV6_SRC 11
#define NFP_FL_ACTION_OPCODE_SET_IPV6_DST 12
+#define NFP_FL_ACTION_OPCODE_SET_IPV6_TC_HL_FL 13
#define NFP_FL_ACTION_OPCODE_SET_UDP 14
#define NFP_FL_ACTION_OPCODE_SET_TCP 15
#define NFP_FL_ACTION_OPCODE_PRE_LAG 16
@@ -83,6 +84,8 @@
#define NFP_FL_PUSH_VLAN_CFI BIT(12)
#define NFP_FL_PUSH_VLAN_VID GENMASK(11, 0)
+#define IPV6_FLOW_LABEL_MASK cpu_to_be32(0x000fffff)
+
/* LAG ports */
#define NFP_FL_LAG_OUT 0xC0DE0000
@@ -135,6 +138,17 @@ struct nfp_fl_set_ip4_ttl_tos {
__be16 reserved;
};
+struct nfp_fl_set_ipv6_tc_hl_fl {
+ struct nfp_fl_act_head head;
+ u8 ipv6_tc_mask;
+ u8 ipv6_hop_limit_mask;
+ __be16 reserved;
+ u8 ipv6_tc;
+ u8 ipv6_hop_limit;
+ __be32 ipv6_label_mask;
+ __be32 ipv6_label;
+};
+
struct nfp_fl_set_ipv6_addr {
struct nfp_fl_act_head head;
__be16 reserved;
--
2.17.1
^ permalink raw reply related
* [PATCH net-next 1/7] nfp: flower: add ipv4 set ttl and tos offload
From: Jakub Kicinski @ 2018-11-07 1:07 UTC (permalink / raw)
To: davem; +Cc: netdev, oss-drivers, Pieter Jansen van Vuuren
In-Reply-To: <20181107010734.29935-1-jakub.kicinski@netronome.com>
From: Pieter Jansen van Vuuren <pieter.jansenvanvuuren@netronome.com>
Add ipv4 set ttl and tos action offload. Since pedit sets headers per 4
byte word, we need to ensure that setting either version, ihl, protocol,
total length or checksum does not get offloaded.
Signed-off-by: Pieter Jansen van Vuuren <pieter.jansenvanvuuren@netronome.com>
Reviewed-by: John Hurley <john.hurley@netronome.com>
Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
.../ethernet/netronome/nfp/flower/action.c | 69 +++++++++++++++++--
.../net/ethernet/netronome/nfp/flower/cmsg.h | 10 +++
2 files changed, 73 insertions(+), 6 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/flower/action.c b/drivers/net/ethernet/netronome/nfp/flower/action.c
index 244dc261006e..b79b924ef56d 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/action.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/action.c
@@ -384,10 +384,21 @@ nfp_fl_set_eth(const struct tc_action *action, int idx, u32 off,
return 0;
}
+struct ipv4_ttl_word {
+ __u8 ttl;
+ __u8 protocol;
+ __sum16 check;
+};
+
static int
nfp_fl_set_ip4(const struct tc_action *action, int idx, u32 off,
- struct nfp_fl_set_ip4_addrs *set_ip_addr)
+ struct nfp_fl_set_ip4_addrs *set_ip_addr,
+ struct nfp_fl_set_ip4_ttl_tos *set_ip_ttl_tos)
{
+ struct ipv4_ttl_word *ttl_word_mask;
+ struct ipv4_ttl_word *ttl_word;
+ struct iphdr *tos_word_mask;
+ struct iphdr *tos_word;
__be32 exact, mask;
/* We are expecting tcf_pedit to return a big endian value */
@@ -402,20 +413,53 @@ nfp_fl_set_ip4(const struct tc_action *action, int idx, u32 off,
set_ip_addr->ipv4_dst_mask |= mask;
set_ip_addr->ipv4_dst &= ~mask;
set_ip_addr->ipv4_dst |= exact & mask;
+ set_ip_addr->head.jump_id = NFP_FL_ACTION_OPCODE_SET_IPV4_ADDRS;
+ set_ip_addr->head.len_lw = sizeof(*set_ip_addr) >>
+ NFP_FL_LW_SIZ;
break;
case offsetof(struct iphdr, saddr):
set_ip_addr->ipv4_src_mask |= mask;
set_ip_addr->ipv4_src &= ~mask;
set_ip_addr->ipv4_src |= exact & mask;
+ set_ip_addr->head.jump_id = NFP_FL_ACTION_OPCODE_SET_IPV4_ADDRS;
+ set_ip_addr->head.len_lw = sizeof(*set_ip_addr) >>
+ NFP_FL_LW_SIZ;
+ break;
+ case offsetof(struct iphdr, ttl):
+ ttl_word_mask = (struct ipv4_ttl_word *)&mask;
+ ttl_word = (struct ipv4_ttl_word *)&exact;
+
+ if (ttl_word_mask->protocol || ttl_word_mask->check)
+ return -EOPNOTSUPP;
+
+ set_ip_ttl_tos->ipv4_ttl_mask |= ttl_word_mask->ttl;
+ set_ip_ttl_tos->ipv4_ttl &= ~ttl_word_mask->ttl;
+ set_ip_ttl_tos->ipv4_ttl |= ttl_word->ttl & ttl_word_mask->ttl;
+ set_ip_ttl_tos->head.jump_id =
+ NFP_FL_ACTION_OPCODE_SET_IPV4_TTL_TOS;
+ set_ip_ttl_tos->head.len_lw = sizeof(*set_ip_ttl_tos) >>
+ NFP_FL_LW_SIZ;
+ break;
+ case round_down(offsetof(struct iphdr, tos), 4):
+ tos_word_mask = (struct iphdr *)&mask;
+ tos_word = (struct iphdr *)&exact;
+
+ if (tos_word_mask->version || tos_word_mask->ihl ||
+ tos_word_mask->tot_len)
+ return -EOPNOTSUPP;
+
+ set_ip_ttl_tos->ipv4_tos_mask |= tos_word_mask->tos;
+ set_ip_ttl_tos->ipv4_tos &= ~tos_word_mask->tos;
+ set_ip_ttl_tos->ipv4_tos |= tos_word->tos & tos_word_mask->tos;
+ set_ip_ttl_tos->head.jump_id =
+ NFP_FL_ACTION_OPCODE_SET_IPV4_TTL_TOS;
+ set_ip_ttl_tos->head.len_lw = sizeof(*set_ip_ttl_tos) >>
+ NFP_FL_LW_SIZ;
break;
default:
return -EOPNOTSUPP;
}
- set_ip_addr->reserved = cpu_to_be16(0);
- set_ip_addr->head.jump_id = NFP_FL_ACTION_OPCODE_SET_IPV4_ADDRS;
- set_ip_addr->head.len_lw = sizeof(*set_ip_addr) >> NFP_FL_LW_SIZ;
-
return 0;
}
@@ -513,6 +557,7 @@ nfp_fl_pedit(const struct tc_action *action, struct tc_cls_flower_offload *flow,
char *nfp_action, int *a_len, u32 *csum_updated)
{
struct nfp_fl_set_ipv6_addr set_ip6_dst, set_ip6_src;
+ struct nfp_fl_set_ip4_ttl_tos set_ip_ttl_tos;
struct nfp_fl_set_ip4_addrs set_ip_addr;
struct nfp_fl_set_tport set_tport;
struct nfp_fl_set_eth set_eth;
@@ -522,6 +567,7 @@ nfp_fl_pedit(const struct tc_action *action, struct tc_cls_flower_offload *flow,
u32 offset, cmd;
u8 ip_proto = 0;
+ memset(&set_ip_ttl_tos, 0, sizeof(set_ip_ttl_tos));
memset(&set_ip6_dst, 0, sizeof(set_ip6_dst));
memset(&set_ip6_src, 0, sizeof(set_ip6_src));
memset(&set_ip_addr, 0, sizeof(set_ip_addr));
@@ -542,7 +588,8 @@ nfp_fl_pedit(const struct tc_action *action, struct tc_cls_flower_offload *flow,
err = nfp_fl_set_eth(action, idx, offset, &set_eth);
break;
case TCA_PEDIT_KEY_EX_HDR_TYPE_IP4:
- err = nfp_fl_set_ip4(action, idx, offset, &set_ip_addr);
+ err = nfp_fl_set_ip4(action, idx, offset, &set_ip_addr,
+ &set_ip_ttl_tos);
break;
case TCA_PEDIT_KEY_EX_HDR_TYPE_IP6:
err = nfp_fl_set_ip6(action, idx, offset, &set_ip6_dst,
@@ -577,6 +624,16 @@ nfp_fl_pedit(const struct tc_action *action, struct tc_cls_flower_offload *flow,
memcpy(nfp_action, &set_eth, act_size);
*a_len += act_size;
}
+ if (set_ip_ttl_tos.head.len_lw) {
+ nfp_action += act_size;
+ act_size = sizeof(set_ip_ttl_tos);
+ memcpy(nfp_action, &set_ip_ttl_tos, act_size);
+ *a_len += act_size;
+
+ /* Hardware will automatically fix IPv4 and TCP/UDP checksum. */
+ *csum_updated |= TCA_CSUM_UPDATE_FLAG_IPV4HDR |
+ nfp_fl_csum_l4_to_flag(ip_proto);
+ }
if (set_ip_addr.head.len_lw) {
nfp_action += act_size;
act_size = sizeof(set_ip_addr);
diff --git a/drivers/net/ethernet/netronome/nfp/flower/cmsg.h b/drivers/net/ethernet/netronome/nfp/flower/cmsg.h
index 29d673aa5277..a00f45b5e16c 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/cmsg.h
+++ b/drivers/net/ethernet/netronome/nfp/flower/cmsg.h
@@ -65,6 +65,7 @@
#define NFP_FL_ACTION_OPCODE_SET_IPV4_TUNNEL 6
#define NFP_FL_ACTION_OPCODE_SET_ETHERNET 7
#define NFP_FL_ACTION_OPCODE_SET_IPV4_ADDRS 9
+#define NFP_FL_ACTION_OPCODE_SET_IPV4_TTL_TOS 10
#define NFP_FL_ACTION_OPCODE_SET_IPV6_SRC 11
#define NFP_FL_ACTION_OPCODE_SET_IPV6_DST 12
#define NFP_FL_ACTION_OPCODE_SET_UDP 14
@@ -125,6 +126,15 @@ struct nfp_fl_set_ip4_addrs {
__be32 ipv4_dst;
};
+struct nfp_fl_set_ip4_ttl_tos {
+ struct nfp_fl_act_head head;
+ u8 ipv4_ttl_mask;
+ u8 ipv4_tos_mask;
+ u8 ipv4_ttl;
+ u8 ipv4_tos;
+ __be16 reserved;
+};
+
struct nfp_fl_set_ipv6_addr {
struct nfp_fl_act_head head;
__be16 reserved;
--
2.17.1
^ permalink raw reply related
* [PATCH net-next 0/7] nfp: more set actions and notifier refactor
From: Jakub Kicinski @ 2018-11-07 1:07 UTC (permalink / raw)
To: davem; +Cc: netdev, oss-drivers, Jakub Kicinski
Hi!
This series brings updates to flower offload code. First Pieter adds
support for setting TTL, ToS, Flow Label and Hop Limit fields in IPv4
and IPv6 headers.
Remaining 5 patches deal with factoring out netdev notifiers from flower
code. We already have two instances, and more is coming, so it's time
to move to one central notifier which then feeds individual feature
handlers.
I start that part by cleaning up the existing notifiers. Next a central
notifier is added, and used by flower offloads.
Jakub Kicinski (5):
nfp: flower: remove unnecessary iteration over devices
nfp: flower: don't try to nack device unregister events
nfp: flower: make nfp_fl_lag_changels_event() void
nfp: register a notifier handler in a central location for the device
nfp: flower: use the common netdev notifier
Pieter Jansen van Vuuren (2):
nfp: flower: add ipv4 set ttl and tos offload
nfp: flower: add ipv6 set flow label and hop limit offload
.../ethernet/netronome/nfp/flower/action.c | 143 ++++++++++++++++--
.../net/ethernet/netronome/nfp/flower/cmsg.h | 24 +++
.../ethernet/netronome/nfp/flower/lag_conf.c | 48 +++---
.../net/ethernet/netronome/nfp/flower/main.c | 23 ++-
.../net/ethernet/netronome/nfp/flower/main.h | 10 +-
.../netronome/nfp/flower/tunnel_conf.c | 45 +-----
drivers/net/ethernet/netronome/nfp/nfp_app.c | 47 ++++++
drivers/net/ethernet/netronome/nfp/nfp_app.h | 25 ++-
8 files changed, 261 insertions(+), 104 deletions(-)
--
2.17.1
^ permalink raw reply
* Re: [PATCH bpf-next v2 02/13] bpf: btf: Add BTF_KIND_FUNC and BTF_KIND_FUNC_PROTO
From: Alexei Starovoitov @ 2018-11-07 0:59 UTC (permalink / raw)
To: Edward Cree
Cc: Martin Lau, Yonghong Song, Alexei Starovoitov,
daniel@iogearbox.net, netdev@vger.kernel.org, Kernel Team
In-Reply-To: <a84d7f90-f93f-d2f3-f403-d506d37b5152@solarflare.com>
On Tue, Nov 06, 2018 at 10:58:42PM +0000, Edward Cree wrote:
> share its type record with 'foo'. And partly just because the
> name of the function itself is no more part of its type than the
> name of an integer variable is part of the integer's type.
correct. function name is not part of its type.
function name is part of BTF that provide debug info about the function.
Function name and function argument names are part of the same debug info.
Splitting them makes no sense.
> (Whereas names of parameters are like names of struct members:
> while they are not part of the 'pure type' from a language
> perspective, they are part of the type from the perspective of
> debugging, which is why they belong in the BTF type record.)
struct name and struct field names live in the same BTF record.
Similarly function name and function argument names should be
in the same BTF record, so we can reuse most of the BTF validation
and BTF parsing logic by doing so.
The minor difference between KIND_STRUCT and KIND_FUNC is
an addition of return type_id.
Everything else is common.
imo that speaks for itself that it's a correct path forward.
> > There are C, bpftrace, p4 and python frontends. These languages
> > should be free to put into BTF KIND_FUNC name that makes sense
> > from the language point of view.
> I'm paying attention to BTF because I'm adding support for it into
> my ebpf_asm. Don't you think I *know* that frontends for BPF are
> more than just C?
assembler is not a high level language.
I believe it's a proper trade-off to make C easier to use
in expense of some ugliness in your ebpf_asm.
> > The global variables for given .c file will look like single KIND_STRUCT
> That's exactly the kind of superficially-clever but nasty hack
> that results from the continued insistence on conflating types
> and instances (objects). In the long run it will make
> maintenance harder, and frustrate new features owing to the need
> to find new hacks to shoehorn them into the same model.
Let's keep 'nasty hack' claims out of this discussion.
I find the current BTF design and KIND_FUNC addition to be elegant
and appropriate.
> Instead there should be entries for the globals in something like
> the variables table I mentioned,
> 2 "fred" type=1 where=global func=0 offset=8
> in which 'func' is unused and 'offset' gives offset in .bss.
> 'where' might also include indication of whether it's static.
'static' like boolean flag? That won't help introspection.
To properly describe 'static' functions more information is necessary.
I don't like to invent new formats. BTF is extensible description
of any debug info. I prefer to keep all debug info in one place
and in one common format.
> I'm saying that the *function* is entirely different to its
> *type*. It's a category error to conflate them:
> f: x -> x + 1
> is a function.
BTF does not describe function. BTF describes debug info about function.
BPF program is the function.
BTF is not *type* only format. It's debug info format.
Trying to make BTF into type only is not going to work.
It's already more than type only as I showed earlier.
^ permalink raw reply
* Re: [PATCH RFC net-next 0/3] net: phy: sfp: Warn when using generic PHY driver
From: Andrew Lunn @ 2018-11-07 0:59 UTC (permalink / raw)
To: Florian Fainelli; +Cc: Russell King - ARM Linux, David Miller, netdev
In-Reply-To: <38f03d3d-d25d-6ff3-2f44-baa01e060746@gmail.com>
> Another approach could be to maintain a list of modules that do not work
> with the generic PHY driver and therefore require a specialized driver,
> in that case we could even go as far as not letting sfp_sm_probe_phy()
> return success. Not sure how well things would scale, probably not too
> bad given there are only a handful of users of the SFP framework thus far...
Hi Florian
Blacklisting modules with known issues with the generic driver does
not sound too bad. This is just a warning, a helpful hint, and it is
not going to work anyway. And i don't see scaling problems, Copper
SFPs seems quite odd to start with...
Andrew
^ permalink raw reply
* [net-next:master 8/13] drivers/net/dsa/bcm_sf2_cfp.c:532:2-3: Unneeded semicolon
From: kbuild test robot @ 2018-11-07 0:50 UTC (permalink / raw)
To: Florian Fainelli; +Cc: kbuild-all, netdev
tree: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git master
head: 5882d526d887e42ead4014d79620e5a8aa741151
commit: ae7a5aff783c79d5ca87867df84b08c43447159b [8/13] net: dsa: bcm_sf2: Keep copy of inserted rules
coccinelle warnings: (new ones prefixed by >>)
>> drivers/net/dsa/bcm_sf2_cfp.c:532:2-3: Unneeded semicolon
Please review and possibly fold the followup patch.
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
^ permalink raw reply
* Re: [PATCH RFC net-next 0/3] net: phy: sfp: Warn when using generic PHY driver
From: Florian Fainelli @ 2018-11-07 0:51 UTC (permalink / raw)
To: Russell King - ARM Linux; +Cc: David Miller, netdev, andrew
In-Reply-To: <20181107003448.GQ30658@n2100.armlinux.org.uk>
On 11/6/18 4:34 PM, Russell King - ARM Linux wrote:
> On Tue, Nov 06, 2018 at 04:09:35PM -0800, Florian Fainelli wrote:
>> On 11/6/18 4:03 PM, Russell King - ARM Linux wrote:
>>> On Tue, Nov 06, 2018 at 03:38:44PM -0800, David Miller wrote:
>>>> From: Florian Fainelli <f.fainelli@gmail.com>
>>>> Date: Tue, 6 Nov 2018 15:29:10 -0800
>>>>
>>>>> This patch series allows warning an user that the generic PHY driver(s)
>>>>> are used when a SFP incorporates a PHY (e.g: 1000BaseT SFP) which is
>>>>> likely not going to work at all.
>>>>>
>>>>> Let me know if you would want to do that differently.
>>>>
>>>> Is there ever a possibility that the generic PHY driver could work
>>>> in an SFP situation?
>>>
>>> I don't yet see the reason for Florian's patch series - all the Marvell
>>> 88e1111 based modules I have, or have come across in information from
>>> manufacturers self-configure themselves and don't really need the
>>> Marvell 1G PHY driver.
>>>
>>> For example, the Source Photonics were offering a range of 1GbaseT
>>> modules with the 88e1111 programmed in different modes, but published
>>> instructions for the register accesses to configure them differently
>>> (eg, SGMII vs 1000base-X interface facing the MAC). Depending on
>>> the module part number determines which mode the PHY has been
>>> programmed to come up in.
>>>
>>> So in theory, you don't need any PHY driver for these modules - but
>>> it's useful to have a functional PHY driver to be able to read out
>>> the negotiated flow control results.
>>>
>>> I'd like more information from Florian about the reasoning behind
>>> this patch series before it's merged.
>>>
>>
>> The module that I am using [1] would not work, as in , no link up being
>> reported without turning on the Marvell PHY driver:
>>
>> https://www.amazon.com/dp/B01LW2P72V/ref=twister_B07F3WQJQX?_encoding=UTF8&psc=1
>>
>> this module uses a 88E1111 PHY as well (OUI: 0x01410cc2).
>
> From the above URL:
>
> * This is 1000M SFP-T Transceiver, not 10/100/1000M Multi-Rate SFP-T. If
> you want to buy 10/100/1000M Multi-Rate SFP-T, pls contact us.10Gtek
> offer more compatible options, if your brands not listed above, pls
> contact us.
>
> I wonder if this is like the Source Photonics situation, where the
> 1000base-T only variant of their module uses 1000base-X on the MAC
> side, whereas their 10/100/1000base-T variant uses SGMII. The only
> difference between these are the part numbers and the programming
> of the 88E1111 to tell it which mode to default to for the host
> side. (There's no true way to know from the EEPROM whether a module
> wants SGMII or 1000base-X.)
>
> What I also gather is that this is a 10Gtek-manufactured version of
> the Ubiquiti UF-RJ45-1G - the original Ubiquiti version supports
> 10/100/1G speeds which would require the 88e1111 to configure for
> a SGMII host interface by default.
>
> Now, the reason that modules with an 88E1111 configured to default to
> 1000base-X will work when the marvell PHY driver is present, but not
> with the generic driver is that the marvell PHY driver will see that
> SFP/phylink is wanting to use SGMII mode, and the Marvell PHY driver
> reprograms the PHY to use SGMII. This is only a problem for these
> modules.
>
> So, in so far as your patch 3 goes to give a hint that the Marvell
> driver should be selected, that's correct.
>
> However, where the 88e1111 is configured for SGMII by default, the
> Marvell driver shouldn't be required, and I wonder whether we ought
> to be issuing a warning in that case. The problem, however, is there
> is no way to know for certain.
>
> We could have modules that do not use the Marvell PHY, and if we don't
> have a PHY driver for their particular PHY, do we want a warning to be
> issued?
Another approach could be to maintain a list of modules that do not work
with the generic PHY driver and therefore require a specialized driver,
in that case we could even go as far as not letting sfp_sm_probe_phy()
return success. Not sure how well things would scale, probably not too
bad given there are only a handful of users of the SFP framework thus far...
>
> The whole 1000base-X vs SGMII with SFP modules is all very icky. :(
>
--
Florian
^ permalink raw reply
* Re: [PATCH v2 1/3] bpf: allow zero-initializing hash map seed
From: Song Liu @ 2018-11-07 0:39 UTC (permalink / raw)
To: lmb; +Cc: Alexei Starovoitov, Daniel Borkmann, Networking, linux-api
In-Reply-To: <CACAyw98RnRZtqMYcS5GZxhf-F-z=sW+Ki+_PBADG+V_qZiuA1Q@mail.gmail.com>
On Thu, Oct 25, 2018 at 8:12 AM Lorenz Bauer <lmb@cloudflare.com> wrote:
>
> On Tue, 9 Oct 2018 at 01:08, Song Liu <liu.song.a23@gmail.com> wrote:
> >
> > > --- a/include/uapi/linux/bpf.h
> > > +++ b/include/uapi/linux/bpf.h
> > > @@ -253,6 +253,8 @@ enum bpf_attach_type {
> > > #define BPF_F_NO_COMMON_LRU (1U << 1)
> > > /* Specify numa node during map creation */
> > > #define BPF_F_NUMA_NODE (1U << 2)
> > > +/* Zero-initialize hash function seed. This should only be used for testing. */
> > > +#define BPF_F_ZERO_SEED (1U << 6)
> >
> > Please add this line after
> > #define BPF_F_STACK_BUILD_ID (1U << 5)
>
> I wanted to keep the flags for BPF_MAP_CREATE grouped together.
> Maybe the correct value is (1U << 3)? It seemed like the other flags
> were allocated to avoid
> overlap between different BPF commands, however, so I tried to follow suit.
I think it should be (1U << 6). We probably should move BPF_F_QUERY_EFFECTIVE
to after BPF_F_STACK_BUILD_ID (and BPF_F_ZERO_SEED).
Also, please rebase against the latest bpf-next tree and resubmit the set.
Thanks,
Song
^ permalink raw reply
* [PATCH net-next] net: phy: bcm7xxx: Add entry for BCM7255
From: Florian Fainelli @ 2018-11-07 0:37 UTC (permalink / raw)
To: netdev; +Cc: davem, andrew, Justin Chen, Florian Fainelli
From: Justin Chen <justinpopo6@gmail.com>
Add support for BCM7255 EPHY.
Signed-off-by: Justin Chen <justinpopo6@gmail.com>
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
drivers/net/phy/bcm7xxx.c | 2 ++
include/linux/brcmphy.h | 1 +
2 files changed, 3 insertions(+)
diff --git a/drivers/net/phy/bcm7xxx.c b/drivers/net/phy/bcm7xxx.c
index b2b6307d64a4..712224cc442d 100644
--- a/drivers/net/phy/bcm7xxx.c
+++ b/drivers/net/phy/bcm7xxx.c
@@ -650,6 +650,7 @@ static int bcm7xxx_28nm_probe(struct phy_device *phydev)
static struct phy_driver bcm7xxx_driver[] = {
BCM7XXX_28NM_GPHY(PHY_ID_BCM7250, "Broadcom BCM7250"),
+ BCM7XXX_28NM_EPHY(PHY_ID_BCM7255, "Broadcom BCM7255"),
BCM7XXX_28NM_EPHY(PHY_ID_BCM7260, "Broadcom BCM7260"),
BCM7XXX_28NM_EPHY(PHY_ID_BCM7268, "Broadcom BCM7268"),
BCM7XXX_28NM_EPHY(PHY_ID_BCM7271, "Broadcom BCM7271"),
@@ -670,6 +671,7 @@ static struct phy_driver bcm7xxx_driver[] = {
static struct mdio_device_id __maybe_unused bcm7xxx_tbl[] = {
{ PHY_ID_BCM7250, 0xfffffff0, },
+ { PHY_ID_BCM7255, 0xfffffff0, },
{ PHY_ID_BCM7260, 0xfffffff0, },
{ PHY_ID_BCM7268, 0xfffffff0, },
{ PHY_ID_BCM7271, 0xfffffff0, },
diff --git a/include/linux/brcmphy.h b/include/linux/brcmphy.h
index 949e9af8d9d6..9cd00a37b8d3 100644
--- a/include/linux/brcmphy.h
+++ b/include/linux/brcmphy.h
@@ -28,6 +28,7 @@
#define PHY_ID_BCM89610 0x03625cd0
#define PHY_ID_BCM7250 0xae025280
+#define PHY_ID_BCM7255 0xae025120
#define PHY_ID_BCM7260 0xae025190
#define PHY_ID_BCM7268 0xae025090
#define PHY_ID_BCM7271 0xae0253b0
--
2.17.1
^ permalink raw reply related
* Re: [PATCH RFC net-next 0/3] net: phy: sfp: Warn when using generic PHY driver
From: Russell King - ARM Linux @ 2018-11-07 0:34 UTC (permalink / raw)
To: Florian Fainelli; +Cc: David Miller, netdev, andrew
In-Reply-To: <cf5eb510-6130-c417-a618-74c34dafa72e@gmail.com>
On Tue, Nov 06, 2018 at 04:09:35PM -0800, Florian Fainelli wrote:
> On 11/6/18 4:03 PM, Russell King - ARM Linux wrote:
> > On Tue, Nov 06, 2018 at 03:38:44PM -0800, David Miller wrote:
> >> From: Florian Fainelli <f.fainelli@gmail.com>
> >> Date: Tue, 6 Nov 2018 15:29:10 -0800
> >>
> >>> This patch series allows warning an user that the generic PHY driver(s)
> >>> are used when a SFP incorporates a PHY (e.g: 1000BaseT SFP) which is
> >>> likely not going to work at all.
> >>>
> >>> Let me know if you would want to do that differently.
> >>
> >> Is there ever a possibility that the generic PHY driver could work
> >> in an SFP situation?
> >
> > I don't yet see the reason for Florian's patch series - all the Marvell
> > 88e1111 based modules I have, or have come across in information from
> > manufacturers self-configure themselves and don't really need the
> > Marvell 1G PHY driver.
> >
> > For example, the Source Photonics were offering a range of 1GbaseT
> > modules with the 88e1111 programmed in different modes, but published
> > instructions for the register accesses to configure them differently
> > (eg, SGMII vs 1000base-X interface facing the MAC). Depending on
> > the module part number determines which mode the PHY has been
> > programmed to come up in.
> >
> > So in theory, you don't need any PHY driver for these modules - but
> > it's useful to have a functional PHY driver to be able to read out
> > the negotiated flow control results.
> >
> > I'd like more information from Florian about the reasoning behind
> > this patch series before it's merged.
> >
>
> The module that I am using [1] would not work, as in , no link up being
> reported without turning on the Marvell PHY driver:
>
> https://www.amazon.com/dp/B01LW2P72V/ref=twister_B07F3WQJQX?_encoding=UTF8&psc=1
>
> this module uses a 88E1111 PHY as well (OUI: 0x01410cc2).
>From the above URL:
* This is 1000M SFP-T Transceiver, not 10/100/1000M Multi-Rate SFP-T. If
you want to buy 10/100/1000M Multi-Rate SFP-T, pls contact us.10Gtek
offer more compatible options, if your brands not listed above, pls
contact us.
I wonder if this is like the Source Photonics situation, where the
1000base-T only variant of their module uses 1000base-X on the MAC
side, whereas their 10/100/1000base-T variant uses SGMII. The only
difference between these are the part numbers and the programming
of the 88E1111 to tell it which mode to default to for the host
side. (There's no true way to know from the EEPROM whether a module
wants SGMII or 1000base-X.)
What I also gather is that this is a 10Gtek-manufactured version of
the Ubiquiti UF-RJ45-1G - the original Ubiquiti version supports
10/100/1G speeds which would require the 88e1111 to configure for
a SGMII host interface by default.
Now, the reason that modules with an 88E1111 configured to default to
1000base-X will work when the marvell PHY driver is present, but not
with the generic driver is that the marvell PHY driver will see that
SFP/phylink is wanting to use SGMII mode, and the Marvell PHY driver
reprograms the PHY to use SGMII. This is only a problem for these
modules.
So, in so far as your patch 3 goes to give a hint that the Marvell
driver should be selected, that's correct.
However, where the 88e1111 is configured for SGMII by default, the
Marvell driver shouldn't be required, and I wonder whether we ought
to be issuing a warning in that case. The problem, however, is there
is no way to know for certain.
We could have modules that do not use the Marvell PHY, and if we don't
have a PHY driver for their particular PHY, do we want a warning to be
issued?
The whole 1000base-X vs SGMII with SFP modules is all very icky. :(
--
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
^ permalink raw reply
* [RFC PATCH 10/12] soc: qcom: ipa: data path
From: Alex Elder @ 2018-11-07 0:32 UTC (permalink / raw)
To: davem, arnd, bjorn.andersson, ilias.apalodimas
Cc: netdev, devicetree, linux-arm-msm, linux-soc, linux-arm-kernel,
linux-kernel, syadagir, mjavid, robh+dt, mark.rutland
In-Reply-To: <20181107003250.5832-1-elder@linaro.org>
This patch contains "ipa_dp.c", which includes the bulk of the data
path code. There is an overview in the code of how things operate,
but there are already plans to rework this portion of the driver.
In particular:
- Interrupt handling will be replaced with a threaded interrupt
handler. Currently handling occurs in a combination of
interrupt and workqueue context, and this requires locking
and atomic operations for proper synchronization.
- Currently, only receive endpoints use NAPI. Transmit
completion interrupts are disabled, and are handled in batches
by periodically scheduling an interrupting no-op request.
The plan is to arrange for transmit requests to generate
interrupts, and their completion will be processed with other
completions in the NAPI poll function. This will also allow
accurate feedback about packet sojourn time to be provided to
queue limiting mechanisms.
- Not all receive endpoints use NAPI. The plan is for *all*
endpoints to use NAPI. And because all endpoints share a
common GSI interrupt, a single NAPI structure will used to
managing the processing for all completions on all endpoints.
- Receive buffers are posted to the hardware by a workqueue
function. Instead, the plan is to have this done by the
NAPI poll routine.
Signed-off-by: Alex Elder <elder@linaro.org>
---
drivers/net/ipa/ipa_dp.c | 1994 ++++++++++++++++++++++++++++++++++++++
1 file changed, 1994 insertions(+)
create mode 100644 drivers/net/ipa/ipa_dp.c
diff --git a/drivers/net/ipa/ipa_dp.c b/drivers/net/ipa/ipa_dp.c
new file mode 100644
index 000000000000..c16ac74765b8
--- /dev/null
+++ b/drivers/net/ipa/ipa_dp.c
@@ -0,0 +1,1994 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <linux/netdevice.h>
+
+#include "ipa_i.h" /* ipa_err() */
+#include "ipahal.h"
+#include "ipa_dma.h"
+
+/**
+ * DOC: The IPA Data Path
+ *
+ * The IPA is used to transmit data between execution environments.
+ * The data path code uses functions and structures supplied by the
+ * GSI to interact with the IPA hardware. A packet to be transmitted
+ * or received is held in a socket buffer. Each has a "wrapper"
+ * structure associated with it. A GSI transfer request refers to
+ * the packet wrapper, and when queued to the hardware the packet
+ * wrapper is added to a list of outstanding requests for an endpoint
+ * (maintained in the head_desc_list in the endpoint's system context).
+ * When the GSI transfer completes, a callback function is provided
+ * the packet wrapper pointer, allowing it to be released after the
+ * received socket buffer has been passed up the stack, or a buffer
+ * whose data has been transmitted has been freed.
+ *
+ * Producer (PROD) endpoints are used to send data from the AP toward
+ * the IPA. The common function for sending data on producer endpoints
+ * is ipa_send(). It takes a system context and an array of IPA
+ * descriptors as arguments. Each descriptor is given a TX packet
+ * wrapper, and its content is translated into an equivalent GSI
+ * transfer element structure after its memory address is mapped for
+ * DMA. The GSI transfer element array is finally passed to the GSI
+ * layer using gsi_channel_queue().
+ *
+ * The code provides a "no_intr" feature, allowing endpoints to have
+ * their transmit completions not produce an interrupt. (This
+ * behavior is used only for the modem producer.) In this case, a
+ * no-op request is generated every 200 milliseconds while transmit
+ * requests are outstanding. The no-op will generate an interrupt
+ * when it's complete, and its completion implies the completion of
+ * all transmit requests issued before it. The GSI will call
+ * ipa_gsi_irq_tx_notify_cb() in response to interrupts on a producer
+ * endpoint.
+ *
+ * Receive buffers are passed to consumer (CONS) channels to be
+ * available to hold incoming data. Arriving data is placed
+ * in these buffers, leading to events being generated on the event
+ * ring assciated with a channel. When an interrupt occurs on a
+ * consumer endpoint, the GSI layer calls ipa_gsi_irq_rx_notify_cb().
+ * This causes the endpoint to switch to polling mode. The
+ * completion of a receive also leads to ipa_replenish_rx_cache()
+ * being called, to replace the consumed buffer.
+ *
+ * Consumer enpoints optionally use NAPI (only the modem consumer,
+ * WWAN_CONS, does currently). An atomic variable records whether
+ * the endpoint is in polling mode or not. This is needed because
+ * switching to polling mode is currently done in a workqueue. Once
+ * NAPI polling completes, and endpoint switches back to interrupt
+ * mode.
+ */
+
+/**
+ * struct ipa_tx_pkt_wrapper - IPA transmit packet wrapper
+ * @type: type of descriptor
+ * @sys: Corresponding IPA sys context
+ * @mem: Memory buffer used by this packet
+ * @callback: IPA client provided callback
+ * @user1: Cookie1 for above callback
+ * @user2: Cookie2 for above callback
+ * @link: Links for the endpoint's sys->head_desc_list
+ * @cnt: Number of descriptors in request
+ * @done_work: Work structure used when complete
+ */
+struct ipa_tx_pkt_wrapper {
+ enum ipa_desc_type type;
+ struct ipa_sys_context *sys;
+ struct ipa_dma_mem mem;
+ void (*callback)(void *user1, int user2);
+ void *user1;
+ int user2;
+ struct list_head link;
+ u32 cnt;
+ struct work_struct done_work;
+};
+
+/** struct ipa_rx_pkt_wrapper - IPA Rx packet wrapper
+ * @link: Links for the endpoint's sys->head_desc_list
+ * @skb: Socket buffer containing the received packet
+ * @len: How many bytes are copied into skb's buffer
+ */
+struct ipa_rx_pkt_wrapper {
+ struct list_head link;
+ struct sk_buff *skb;
+ dma_addr_t dma_addr;
+};
+
+/** struct ipa_sys_context - IPA GPI endpoint context
+ * @len: The number of entries in @head_desc_list
+ * @tx: Details related to AP->IPA endpoints
+ * @rx: Details related to IPA->AP endpoints
+ * @ep: Associated endpoint
+ * @head_desc_list: List of packets
+ * @spinlock: Lock protecting the descriptor list
+ * @workqueue: Workqueue used for this endpoint
+ */
+struct ipa_sys_context {
+ u32 len;
+ union {
+ struct { /* Consumer endpoints only */
+ u32 len_pending_xfer;
+ atomic_t curr_polling_state;
+ struct delayed_work switch_to_intr_work; /* sys->wq */
+ void (*pyld_hdlr)(struct sk_buff *,
+ struct ipa_sys_context *);
+ u32 buff_sz;
+ u32 pool_sz;
+ struct sk_buff *prev_skb;
+ unsigned int len_rem;
+ unsigned int len_pad; /* APPS_LAN only */
+ unsigned int len_partial; /* APPS_LAN only */
+ bool drop_packet; /* APPS_LAN only */
+
+ struct work_struct work; /* sys->wq */
+ struct delayed_work replenish_work; /* sys->wq */
+ } rx;
+ struct { /* Producer endpoints only */
+ /* no_intr/nop is APPS_WAN_PROD only */
+ bool no_intr;
+ atomic_t nop_pending;
+ struct hrtimer nop_timer;
+ struct work_struct nop_work; /* sys->wq */
+ } tx;
+ };
+
+ /* ordering is important - mutable fields go above */
+ struct ipa_ep_context *ep;
+ struct list_head head_desc_list; /* contains len entries */
+ spinlock_t spinlock; /* protects head_desc list */
+ struct workqueue_struct *wq;
+ /* ordering is important - other immutable fields go below */
+};
+
+/**
+ * struct ipa_dp - IPA data path information
+ * @tx_pkt_wrapper_cache: Tx packets cache
+ * @rx_pkt_wrapper_cache: Rx packets cache
+ */
+struct ipa_dp {
+ struct kmem_cache *tx_pkt_wrapper_cache;
+ struct kmem_cache *rx_pkt_wrapper_cache;
+};
+
+/**
+ * struct ipa_tag_completion - Reference counted completion object
+ * @comp: Completion when last reference is dropped
+ * @cnt: Reference count
+ */
+struct ipa_tag_completion {
+ struct completion comp;
+ atomic_t cnt;
+};
+
+#define CHANNEL_RESET_AGGR_RETRY_COUNT 3
+#define CHANNEL_RESET_DELAY 1 /* milliseconds */
+
+#define IPA_QMAP_HEADER_LENGTH 4
+
+#define IPA_WAN_AGGR_PKT_CNT 5
+#define POLLING_INACTIVITY_RX 40
+#define POLLING_MIN_SLEEP_RX 1010 /* microseconds */
+#define POLLING_MAX_SLEEP_RX 1050 /* microseconds */
+
+#define IPA_RX_BUFFER_ORDER 1 /* Default RX buffer is 2^1 pages */
+#define IPA_RX_BUFFER_SIZE (1 << (IPA_RX_BUFFER_ORDER + PAGE_SHIFT))
+
+/* The amount of RX buffer space consumed by standard skb overhead */
+#define IPA_RX_BUFFER_RESERVED \
+ (IPA_RX_BUFFER_SIZE - SKB_MAX_ORDER(NET_SKB_PAD, IPA_RX_BUFFER_ORDER))
+
+/* RX buffer space remaining after standard overhead is consumed */
+#define IPA_RX_BUFFER_AVAILABLE(X) ((X) - IPA_RX_BUFFER_RESERVED)
+
+#define IPA_RX_BUFF_CLIENT_HEADROOM 256
+
+#define IPA_SIZE_DL_CSUM_META_TRAILER 8
+
+#define IPA_REPL_XFER_THRESH 10
+
+/* How long before sending an interrupting no-op to handle TX completions */
+#define IPA_TX_NOP_DELAY_NS (2 * 1000 * 1000) /* 2 msec */
+
+static void ipa_rx_switch_to_intr_mode(struct ipa_sys_context *sys);
+
+static void ipa_replenish_rx_cache(struct ipa_sys_context *sys);
+static void ipa_replenish_rx_work_func(struct work_struct *work);
+static void ipa_wq_handle_rx(struct work_struct *work);
+static void ipa_rx_common(struct ipa_sys_context *sys, u32 size);
+static void ipa_cleanup_rx(struct ipa_sys_context *sys);
+static int ipa_poll_gsi_pkt(struct ipa_sys_context *sys);
+
+static void ipa_tx_complete(struct ipa_tx_pkt_wrapper *tx_pkt)
+{
+ struct device *dev = ipa_ctx->dev;
+
+ /* If DMA memory was mapped, unmap it */
+ if (tx_pkt->mem.virt) {
+ if (tx_pkt->type == IPA_DATA_DESC_SKB_PAGED)
+ dma_unmap_page(dev, tx_pkt->mem.phys,
+ tx_pkt->mem.size, DMA_TO_DEVICE);
+ else
+ dma_unmap_single(dev, tx_pkt->mem.phys,
+ tx_pkt->mem.size, DMA_TO_DEVICE);
+ }
+
+ if (tx_pkt->callback)
+ tx_pkt->callback(tx_pkt->user1, tx_pkt->user2);
+
+ kmem_cache_free(ipa_ctx->dp->tx_pkt_wrapper_cache, tx_pkt);
+}
+
+static void
+ipa_wq_write_done_common(struct ipa_sys_context *sys,
+ struct ipa_tx_pkt_wrapper *tx_pkt)
+{
+ struct ipa_tx_pkt_wrapper *next_pkt;
+ int cnt;
+ int i;
+
+ cnt = tx_pkt->cnt;
+ for (i = 0; i < cnt; i++) {
+ ipa_assert(!list_empty(&sys->head_desc_list));
+
+ spin_lock_bh(&sys->spinlock);
+
+ next_pkt = list_next_entry(tx_pkt, link);
+ list_del(&tx_pkt->link);
+ sys->len--;
+
+ spin_unlock_bh(&sys->spinlock);
+
+ ipa_tx_complete(tx_pkt);
+
+ tx_pkt = next_pkt;
+ }
+}
+
+/**
+ * ipa_wq_write_done() - Work function executed when TX completes
+ * * @done_work: work_struct used by the work queue
+ */
+static void ipa_wq_write_done(struct work_struct *done_work)
+{
+ struct ipa_tx_pkt_wrapper *this_pkt;
+ struct ipa_tx_pkt_wrapper *tx_pkt;
+ struct ipa_sys_context *sys;
+
+ tx_pkt = container_of(done_work, struct ipa_tx_pkt_wrapper, done_work);
+ sys = tx_pkt->sys;
+ spin_lock_bh(&sys->spinlock);
+ this_pkt = list_first_entry(&sys->head_desc_list,
+ struct ipa_tx_pkt_wrapper, link);
+ while (tx_pkt != this_pkt) {
+ spin_unlock_bh(&sys->spinlock);
+ ipa_wq_write_done_common(sys, this_pkt);
+ spin_lock_bh(&sys->spinlock);
+ this_pkt = list_first_entry(&sys->head_desc_list,
+ struct ipa_tx_pkt_wrapper, link);
+ }
+ spin_unlock_bh(&sys->spinlock);
+ ipa_wq_write_done_common(sys, tx_pkt);
+}
+
+/**
+ * ipa_rx_poll() - Poll the rx packets from IPA hardware
+ * @ep_id: Endpoint to poll
+ * @weight: NAPI poll weight
+ *
+ * Return: The number of received packets.
+ */
+int ipa_rx_poll(u32 ep_id, int weight)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+ static int total_cnt;
+ int cnt = 0;
+
+ while (cnt < weight && ipa_ep_polling(ep)) {
+ int ret;
+
+ ret = ipa_poll_gsi_pkt(ep->sys);
+ if (ret < 0)
+ break;
+
+ ipa_rx_common(ep->sys, (u32)ret);
+ cnt += IPA_WAN_AGGR_PKT_CNT;
+ total_cnt++;
+
+ /* Force switch back to interrupt mode if no more packets */
+ if (!ep->sys->len || total_cnt >= ep->sys->rx.pool_sz) {
+ total_cnt = 0;
+ cnt--;
+ break;
+ }
+ }
+
+ if (cnt < weight) {
+ ep->client_notify(ep->priv, IPA_CLIENT_COMP_NAPI, 0);
+ ipa_rx_switch_to_intr_mode(ep->sys);
+
+ /* Matching enable is in ipa_gsi_irq_rx_notify_cb() */
+ ipa_client_remove();
+ }
+
+ return cnt;
+}
+
+/**
+ * ipa_send_nop() - Send an interrupting no-op request to a producer endpoint.
+ * @sys: System context for the endpoint
+ *
+ * Normally an interrupt is generated upon completion of every transfer
+ * performed by an endpoint, but a producer endpoint can be configured
+ * to avoid getting these interrupts. Instead, once a transfer has been
+ * initiated, a no-op is scheduled to be sent after a short delay. This
+ * no-op request will interrupt when it is complete, and in handling that
+ * interrupt, previously-completed transfers will be handled as well. If
+ * a no-op is already scheduled, another is not initiated (there's only
+ * one pending at a time).
+ */
+static bool ipa_send_nop(struct ipa_sys_context *sys)
+{
+ struct gsi_xfer_elem nop_xfer = { };
+ struct ipa_tx_pkt_wrapper *nop_pkt;
+ u32 channel_id;
+
+ nop_pkt = kmem_cache_zalloc(ipa_ctx->dp->tx_pkt_wrapper_cache,
+ GFP_KERNEL);
+ if (!nop_pkt)
+ return false;
+
+ nop_pkt->type = IPA_DATA_DESC;
+ /* No-op packet uses no memory for data */
+ INIT_WORK(&nop_pkt->done_work, ipa_wq_write_done);
+ nop_pkt->sys = sys;
+ nop_pkt->cnt = 1;
+
+ nop_xfer.type = GSI_XFER_ELEM_NOP;
+ nop_xfer.flags = GSI_XFER_FLAG_EOT;
+ nop_xfer.user_data = nop_pkt;
+
+ spin_lock_bh(&sys->spinlock);
+ list_add_tail(&nop_pkt->link, &sys->head_desc_list);
+ spin_unlock_bh(&sys->spinlock);
+
+ channel_id = sys->ep->channel_id;
+ if (!gsi_channel_queue(ipa_ctx->gsi, channel_id, 1, &nop_xfer, true))
+ return true; /* Success */
+
+ spin_lock_bh(&sys->spinlock);
+ list_del(&nop_pkt->link);
+ spin_unlock_bh(&sys->spinlock);
+
+ kmem_cache_free(ipa_ctx->dp->tx_pkt_wrapper_cache, nop_pkt);
+
+ return false;
+}
+
+/**
+ * ipa_send_nop_work() - Work function for sending a no-op request
+ * nop_work: Work structure for the request
+ *
+ * Try to send the no-op request. If it fails, arrange to try again.
+ */
+static void ipa_send_nop_work(struct work_struct *nop_work)
+{
+ struct ipa_sys_context *sys;
+
+ sys = container_of(nop_work, struct ipa_sys_context, tx.nop_work);
+
+ /* If sending a no-op request fails, schedule another try */
+ if (!ipa_send_nop(sys))
+ queue_work(sys->wq, nop_work);
+}
+
+/**
+ * ipa_nop_timer_expiry() - Timer function to schedule a no-op request
+ * @timer: High-resolution timer structure
+ *
+ * The delay before sending the no-op request is implemented by a
+ * high resolution timer, which will call this in interrupt context.
+ * Arrange to send the no-op in workqueue context when it expires.
+ */
+static enum hrtimer_restart ipa_nop_timer_expiry(struct hrtimer *timer)
+{
+ struct ipa_sys_context *sys;
+
+ sys = container_of(timer, struct ipa_sys_context, tx.nop_timer);
+ atomic_set(&sys->tx.nop_pending, 0);
+ queue_work(sys->wq, &sys->tx.nop_work);
+
+ return HRTIMER_NORESTART;
+}
+
+static void ipa_nop_timer_schedule(struct ipa_sys_context *sys)
+{
+ ktime_t time;
+
+ if (atomic_xchg(&sys->tx.nop_pending, 1))
+ return;
+
+ time = ktime_set(0, IPA_TX_NOP_DELAY_NS);
+ hrtimer_start(&sys->tx.nop_timer, time, HRTIMER_MODE_REL);
+}
+
+/**
+ * ipa_no_intr_init() - Configure endpoint point for no-op requests
+ * @prod_ep_id: Endpoint that will use interrupting no-ops
+ *
+ * For some producer endpoints we don't interrupt on completions.
+ * Instead we schedule an interrupting NOP command to be issued on
+ * the endpoint after a short delay (if one is not already scheduled).
+ * When the NOP completes it signals all preceding transfers have
+ * completed also.
+ */
+void ipa_no_intr_init(u32 prod_ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[prod_ep_id];
+
+ INIT_WORK(&ep->sys->tx.nop_work, ipa_send_nop_work);
+ atomic_set(&ep->sys->tx.nop_pending, 0);
+ hrtimer_init(&ep->sys->tx.nop_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+ ep->sys->tx.nop_timer.function = ipa_nop_timer_expiry;
+ ep->sys->tx.no_intr = true;
+}
+
+/**
+ * ipa_send() - Send descriptors to hardware as a single transaction
+ * @sys: System context for endpoint
+ * @num_desc: Number of descriptors
+ * @desc: Transfer descriptors to send
+ *
+ * Return: 0 iff successful, or a negative error code.
+ */
+static int
+ipa_send(struct ipa_sys_context *sys, u32 num_desc, struct ipa_desc *desc)
+{
+ struct ipa_tx_pkt_wrapper *tx_pkt;
+ struct ipa_tx_pkt_wrapper *first;
+ struct ipa_tx_pkt_wrapper *next;
+ struct gsi_xfer_elem *xfer_elem;
+ LIST_HEAD(pkt_list);
+ int ret;
+ int i;
+
+ ipa_assert(num_desc);
+ ipa_assert(num_desc <= ipa_client_tlv_count(sys->ep->client));
+
+ xfer_elem = kcalloc(num_desc, sizeof(*xfer_elem), GFP_ATOMIC);
+ if (!xfer_elem)
+ return -ENOMEM;
+
+ /* Within loop, all errors are allocation or DMA mapping */
+ ret = -ENOMEM;
+ first = NULL;
+ for (i = 0; i < num_desc; i++) {
+ dma_addr_t phys;
+
+ tx_pkt = kmem_cache_zalloc(ipa_ctx->dp->tx_pkt_wrapper_cache,
+ GFP_ATOMIC);
+ if (!tx_pkt)
+ goto err_unwind;
+
+ if (!first)
+ first = tx_pkt;
+
+ if (desc[i].type == IPA_DATA_DESC_SKB_PAGED)
+ phys = skb_frag_dma_map(ipa_ctx->dev, desc[i].payload,
+ 0, desc[i].len_opcode,
+ DMA_TO_DEVICE);
+ else
+ phys = dma_map_single(ipa_ctx->dev, desc[i].payload,
+ desc[i].len_opcode,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(ipa_ctx->dev, phys)) {
+ ipa_err("dma mapping error on descriptor\n");
+ kmem_cache_free(ipa_ctx->dp->tx_pkt_wrapper_cache,
+ tx_pkt);
+ goto err_unwind;
+ }
+
+ tx_pkt->type = desc[i].type;
+ tx_pkt->sys = sys;
+ tx_pkt->mem.virt = desc[i].payload;
+ tx_pkt->mem.phys = phys;
+ tx_pkt->mem.size = desc[i].len_opcode;
+ tx_pkt->callback = desc[i].callback;
+ tx_pkt->user1 = desc[i].user1;
+ tx_pkt->user2 = desc[i].user2;
+ list_add_tail(&tx_pkt->link, &pkt_list);
+
+ xfer_elem[i].addr = tx_pkt->mem.phys;
+ if (desc[i].type == IPA_IMM_CMD_DESC)
+ xfer_elem[i].type = GSI_XFER_ELEM_IMME_CMD;
+ else
+ xfer_elem[i].type = GSI_XFER_ELEM_DATA;
+ xfer_elem[i].len_opcode = desc[i].len_opcode;
+ if (i < num_desc - 1)
+ xfer_elem[i].flags = GSI_XFER_FLAG_CHAIN;
+ }
+
+ /* Fill in extra fields in the first TX packet */
+ first->cnt = num_desc;
+ INIT_WORK(&first->done_work, ipa_wq_write_done);
+
+ /* Fill in extra fields in the last transfer element */
+ if (!sys->tx.no_intr) {
+ xfer_elem[num_desc - 1].flags = GSI_XFER_FLAG_EOT;
+ xfer_elem[num_desc - 1].flags |= GSI_XFER_FLAG_BEI;
+ }
+ xfer_elem[num_desc - 1].user_data = first;
+
+ spin_lock_bh(&sys->spinlock);
+
+ list_splice_tail_init(&pkt_list, &sys->head_desc_list);
+ ret = gsi_channel_queue(ipa_ctx->gsi, sys->ep->channel_id, num_desc,
+ xfer_elem, true);
+ if (ret)
+ list_cut_end(&pkt_list, &sys->head_desc_list, &first->link);
+
+ spin_unlock_bh(&sys->spinlock);
+
+ kfree(xfer_elem);
+
+ if (!ret) {
+ if (sys->tx.no_intr)
+ ipa_nop_timer_schedule(sys);
+ return 0;
+ }
+err_unwind:
+ list_for_each_entry_safe(tx_pkt, next, &pkt_list, link) {
+ list_del(&tx_pkt->link);
+ tx_pkt->callback = NULL; /* Avoid doing the callback */
+ ipa_tx_complete(tx_pkt);
+ }
+
+ return ret;
+}
+
+/**
+ * ipa_send_cmd_timeout_complete() - Command completion callback
+ * @user1: Opaque value carried by the command
+ * @ignored: Second opaque value (ignored)
+ *
+ * Schedule a completion to signal that a command is done. Free the
+ * tag_completion structure if its reference count reaches zero.
+ */
+static void ipa_send_cmd_timeout_complete(void *user1, int ignored)
+{
+ struct ipa_tag_completion *comp = user1;
+
+ complete(&comp->comp);
+ if (!atomic_dec_return(&comp->cnt))
+ kfree(comp);
+}
+
+/**
+ * ipa_send_cmd_timeout() - Send an immediate command with timeout
+ * @desc: descriptor structure
+ * @timeout: milliseconds to wait (or 0 to wait indefinitely)
+ *
+ * Send an immediate command, and wait for it to complete. If
+ * timeout is non-zero it indicates the number of milliseconds to
+ * wait to receive the acknowledgment from the hardware before
+ * timing out. If 0 is supplied, wait will not time out.
+ *
+ * Return: 0 if successful, or a negative error code
+ */
+int ipa_send_cmd_timeout(struct ipa_desc *desc, u32 timeout)
+{
+ struct ipa_tag_completion *comp;
+ unsigned long timeout_jiffies;
+ struct ipa_ep_context *ep;
+ int ret;
+
+ comp = kzalloc(sizeof(*comp), GFP_KERNEL);
+ if (!comp)
+ return -ENOMEM;
+
+ /* The reference count is decremented both here and in ack
+ * callback. Whichever reaches 0 frees the structure.
+ */
+ atomic_set(&comp->cnt, 2);
+ init_completion(&comp->comp);
+
+ /* Fill in the callback info (the sole descriptor is the last) */
+ desc->callback = ipa_send_cmd_timeout_complete;
+ desc->user1 = comp;
+
+ ep = &ipa_ctx->ep[ipa_client_ep_id(IPA_CLIENT_APPS_CMD_PROD)];
+ ret = ipa_send(ep->sys, 1, desc);
+ if (ret) {
+ /* Callback won't run; drop reference on its behalf */
+ atomic_dec(&comp->cnt);
+ goto out;
+ }
+
+ timeout_jiffies = msecs_to_jiffies(timeout);
+ if (!timeout_jiffies) {
+ wait_for_completion(&comp->comp);
+ } else if (!wait_for_completion_timeout(&comp->comp, timeout_jiffies)) {
+ ret = -ETIMEDOUT;
+ ipa_err("command timed out\n");
+ }
+out:
+ if (!atomic_dec_return(&comp->cnt))
+ kfree(comp);
+
+ return ret;
+}
+
+/**
+ * ipa_handle_rx_core() - Core packet reception handling
+ * @sys: System context for endpoint receiving packets
+ *
+ * Return: The number of packets processed, or a negative error code
+ */
+static int ipa_handle_rx_core(struct ipa_sys_context *sys)
+{
+ int cnt;
+
+ /* Stop if the endpoint leaves polling state */
+ cnt = 0;
+ while (ipa_ep_polling(sys->ep)) {
+ int ret = ipa_poll_gsi_pkt(sys);
+
+ if (ret < 0)
+ break;
+
+ ipa_rx_common(sys, (u32)ret);
+
+ cnt++;
+ }
+
+ return cnt;
+}
+
+/**
+ * ipa_rx_switch_to_intr_mode() - Switch from polling to interrupt mode
+ * @sys: System context for endpoint switching mode
+ */
+static void ipa_rx_switch_to_intr_mode(struct ipa_sys_context *sys)
+{
+ if (!atomic_xchg(&sys->rx.curr_polling_state, 0)) {
+ ipa_err("already in intr mode\n");
+ queue_delayed_work(sys->wq, &sys->rx.switch_to_intr_work,
+ msecs_to_jiffies(1));
+ return;
+ }
+ ipa_dec_release_wakelock();
+ gsi_channel_intr_enable(ipa_ctx->gsi, sys->ep->channel_id);
+}
+
+void ipa_rx_switch_to_poll_mode(struct ipa_sys_context *sys)
+{
+ if (atomic_xchg(&sys->rx.curr_polling_state, 1))
+ return;
+ gsi_channel_intr_disable(ipa_ctx->gsi, sys->ep->channel_id);
+ ipa_inc_acquire_wakelock();
+ queue_work(sys->wq, &sys->rx.work);
+}
+
+/**
+ * ipa_handle_rx() - Handle packet reception.
+ * @sys: System context for endpoint receiving packets
+ */
+static void ipa_handle_rx(struct ipa_sys_context *sys)
+{
+ int inactive_cycles = 0;
+ int cnt;
+
+ ipa_client_add();
+ do {
+ cnt = ipa_handle_rx_core(sys);
+ if (cnt == 0)
+ inactive_cycles++;
+ else
+ inactive_cycles = 0;
+
+ usleep_range(POLLING_MIN_SLEEP_RX, POLLING_MAX_SLEEP_RX);
+
+ /* if endpoint is out of buffers there is no point polling for
+ * completed descs; release the worker so delayed work can
+ * run in a timely manner
+ */
+ if (sys->len - sys->rx.len_pending_xfer == 0)
+ break;
+
+ } while (inactive_cycles <= POLLING_INACTIVITY_RX);
+
+ ipa_rx_switch_to_intr_mode(sys);
+ ipa_client_remove();
+}
+
+static void ipa_switch_to_intr_rx_work_func(struct work_struct *work)
+{
+ struct delayed_work *dwork = to_delayed_work(work);
+ struct ipa_sys_context *sys;
+
+ sys = container_of(dwork, struct ipa_sys_context,
+ rx.switch_to_intr_work);
+
+ /* For NAPI, interrupt mode is done in ipa_rx_poll context */
+ ipa_assert(!sys->ep->napi_enabled);
+
+ ipa_handle_rx(sys);
+}
+
+static struct ipa_sys_context *ipa_ep_sys_create(enum ipa_client_type client)
+{
+ const unsigned int wq_flags = WQ_MEM_RECLAIM | WQ_UNBOUND;
+ struct ipa_sys_context *sys;
+
+ /* Caller will zero all "mutable" fields; we fill in the rest */
+ sys = kmalloc(sizeof(*sys), GFP_KERNEL);
+ if (!sys)
+ return NULL;
+
+ sys->wq = alloc_workqueue("ipawq%u", wq_flags, 1, (u32)client);
+ if (!sys->wq) {
+ kfree(sys);
+ return NULL;
+ }
+
+ /* Caller assigns sys->ep = ep */
+ INIT_LIST_HEAD(&sys->head_desc_list);
+ spin_lock_init(&sys->spinlock);
+
+ return sys;
+}
+
+/**
+ * ipa_tx_dp_complete() - Transmit complete callback
+ * @user1: Caller-supplied pointer value
+ * @user2: Caller-supplied integer value
+ *
+ * Calls the endpoint's client_notify function if it exists;
+ * otherwise just frees the socket buffer (supplied in user1).
+ */
+static void ipa_tx_dp_complete(void *user1, int user2)
+{
+ struct sk_buff *skb = user1;
+ int ep_id = user2;
+
+ if (ipa_ctx->ep[ep_id].client_notify) {
+ unsigned long data;
+ void *priv;
+
+ priv = ipa_ctx->ep[ep_id].priv;
+ data = (unsigned long)skb;
+ ipa_ctx->ep[ep_id].client_notify(priv, IPA_WRITE_DONE, data);
+ } else {
+ dev_kfree_skb_any(skb);
+ }
+}
+
+/**
+ * ipa_tx_dp() - Transmit a socket buffer for APPS_WAN_PROD
+ * @client: IPA client that is sending packets (WAN producer)
+ * @skb: The socket buffer to send
+ *
+ * Returns: 0 if successful, or a negative error code
+ */
+int ipa_tx_dp(enum ipa_client_type client, struct sk_buff *skb)
+{
+ struct ipa_desc _desc = { }; /* Used for common case */
+ struct ipa_desc *desc;
+ u32 tlv_count;
+ int data_idx;
+ u32 nr_frags;
+ u32 ep_id;
+ int ret;
+ u32 f;
+
+ if (!skb->len)
+ return -EINVAL;
+
+ ep_id = ipa_client_ep_id(client);
+
+ /* Make sure source endpoint's TLV FIFO has enough entries to
+ * hold the linear portion of the skb and all its frags.
+ * If not, see if we can linearize it before giving up.
+ */
+ nr_frags = skb_shinfo(skb)->nr_frags;
+ tlv_count = ipa_client_tlv_count(client);
+ if (1 + nr_frags > tlv_count) {
+ if (skb_linearize(skb))
+ return -ENOMEM;
+ nr_frags = 0;
+ }
+ if (nr_frags) {
+ desc = kcalloc(1 + nr_frags, sizeof(*desc), GFP_ATOMIC);
+ if (!desc)
+ return -ENOMEM;
+ } else {
+ desc = &_desc; /* Default, linear case */
+ }
+
+ /* Fill in the IPA request descriptors--one for the linear
+ * data in the skb, one each for each of its fragments.
+ */
+ data_idx = 0;
+ desc[data_idx].payload = skb->data;
+ desc[data_idx].len_opcode = skb_headlen(skb);
+ desc[data_idx].type = IPA_DATA_DESC_SKB;
+ for (f = 0; f < nr_frags; f++) {
+ data_idx++;
+ desc[data_idx].payload = &skb_shinfo(skb)->frags[f];
+ desc[data_idx].type = IPA_DATA_DESC_SKB_PAGED;
+ desc[data_idx].len_opcode =
+ skb_frag_size(desc[data_idx].payload);
+ }
+
+ /* Have the skb be freed after the last descriptor completes. */
+ desc[data_idx].callback = ipa_tx_dp_complete;
+ desc[data_idx].user1 = skb;
+ desc[data_idx].user2 = ep_id;
+
+ ret = ipa_send(ipa_ctx->ep[ep_id].sys, data_idx + 1, desc);
+
+ if (nr_frags)
+ kfree(desc);
+
+ return ret;
+}
+
+static void ipa_wq_handle_rx(struct work_struct *work)
+{
+ struct ipa_sys_context *sys;
+
+ sys = container_of(work, struct ipa_sys_context, rx.work);
+
+ if (sys->ep->napi_enabled) {
+ ipa_client_add();
+ sys->ep->client_notify(sys->ep->priv, IPA_CLIENT_START_POLL, 0);
+ } else {
+ ipa_handle_rx(sys);
+ }
+}
+
+static int
+queue_rx_cache(struct ipa_sys_context *sys, struct ipa_rx_pkt_wrapper *rx_pkt)
+{
+ struct gsi_xfer_elem gsi_xfer_elem;
+ bool ring_doorbell;
+ int ret;
+
+ /* Don't bother zeroing this; we fill all fields */
+ gsi_xfer_elem.addr = rx_pkt->dma_addr;
+ gsi_xfer_elem.len_opcode = sys->rx.buff_sz;
+ gsi_xfer_elem.flags = GSI_XFER_FLAG_EOT;
+ gsi_xfer_elem.flags |= GSI_XFER_FLAG_EOB;
+ gsi_xfer_elem.type = GSI_XFER_ELEM_DATA;
+ gsi_xfer_elem.user_data = rx_pkt;
+
+ /* Doorbell is expensive; only ring it when a batch is queued */
+ ring_doorbell = sys->rx.len_pending_xfer++ >= IPA_REPL_XFER_THRESH;
+
+ ret = gsi_channel_queue(ipa_ctx->gsi, sys->ep->channel_id,
+ 1, &gsi_xfer_elem, ring_doorbell);
+ if (ret)
+ return ret;
+
+ if (ring_doorbell)
+ sys->rx.len_pending_xfer = 0;
+
+ return 0;
+}
+
+/**
+ * ipa_replenish_rx_cache() - Replenish the Rx packets cache.
+ * @sys: System context for IPA->AP endpoint
+ *
+ * Allocate RX packet wrapper structures with maximal socket buffers
+ * for an endpoint. These are supplied to the hardware, which fills
+ * them with incoming data.
+ */
+static void ipa_replenish_rx_cache(struct ipa_sys_context *sys)
+{
+ struct ipa_rx_pkt_wrapper *rx_pkt;
+ struct device *dev = ipa_ctx->dev;
+ u32 rx_len_cached = sys->len;
+
+ while (rx_len_cached < sys->rx.pool_sz) {
+ gfp_t flag = GFP_NOWAIT | __GFP_NOWARN;
+ void *ptr;
+ int ret;
+
+ rx_pkt = kmem_cache_zalloc(ipa_ctx->dp->rx_pkt_wrapper_cache,
+ flag);
+ if (!rx_pkt)
+ goto fail_kmem_cache_alloc;
+
+ INIT_LIST_HEAD(&rx_pkt->link);
+
+ rx_pkt->skb = __dev_alloc_skb(sys->rx.buff_sz, flag);
+ if (!rx_pkt->skb) {
+ ipa_err("failed to alloc skb\n");
+ goto fail_skb_alloc;
+ }
+ ptr = skb_put(rx_pkt->skb, sys->rx.buff_sz);
+ rx_pkt->dma_addr = dma_map_single(dev, ptr, sys->rx.buff_sz,
+ DMA_FROM_DEVICE);
+ if (dma_mapping_error(dev, rx_pkt->dma_addr)) {
+ ipa_err("dma_map_single failure %p for %p\n",
+ (void *)rx_pkt->dma_addr, ptr);
+ goto fail_dma_mapping;
+ }
+
+ list_add_tail(&rx_pkt->link, &sys->head_desc_list);
+ rx_len_cached = ++sys->len;
+
+ ret = queue_rx_cache(sys, rx_pkt);
+ if (ret)
+ goto fail_provide_rx_buffer;
+ }
+
+ return;
+
+fail_provide_rx_buffer:
+ list_del(&rx_pkt->link);
+ rx_len_cached = --sys->len;
+ dma_unmap_single(dev, rx_pkt->dma_addr, sys->rx.buff_sz,
+ DMA_FROM_DEVICE);
+fail_dma_mapping:
+ dev_kfree_skb_any(rx_pkt->skb);
+fail_skb_alloc:
+ kmem_cache_free(ipa_ctx->dp->rx_pkt_wrapper_cache, rx_pkt);
+fail_kmem_cache_alloc:
+ if (rx_len_cached - sys->rx.len_pending_xfer == 0)
+ queue_delayed_work(sys->wq, &sys->rx.replenish_work,
+ msecs_to_jiffies(1));
+}
+
+static void ipa_replenish_rx_work_func(struct work_struct *work)
+{
+ struct delayed_work *dwork = to_delayed_work(work);
+ struct ipa_sys_context *sys;
+
+ sys = container_of(dwork, struct ipa_sys_context, rx.replenish_work);
+ ipa_client_add();
+ ipa_replenish_rx_cache(sys);
+ ipa_client_remove();
+}
+
+/** ipa_cleanup_rx() - release RX queue resources */
+static void ipa_cleanup_rx(struct ipa_sys_context *sys)
+{
+ struct ipa_rx_pkt_wrapper *rx_pkt;
+ struct ipa_rx_pkt_wrapper *r;
+
+ list_for_each_entry_safe(rx_pkt, r, &sys->head_desc_list, link) {
+ list_del(&rx_pkt->link);
+ dma_unmap_single(ipa_ctx->dev, rx_pkt->dma_addr,
+ sys->rx.buff_sz, DMA_FROM_DEVICE);
+ dev_kfree_skb_any(rx_pkt->skb);
+ kmem_cache_free(ipa_ctx->dp->rx_pkt_wrapper_cache, rx_pkt);
+ }
+}
+
+static struct sk_buff *ipa_skb_copy_for_client(struct sk_buff *skb, int len)
+{
+ struct sk_buff *skb2;
+
+ skb2 = __dev_alloc_skb(len + IPA_RX_BUFF_CLIENT_HEADROOM, GFP_KERNEL);
+ if (likely(skb2)) {
+ /* Set the data pointer */
+ skb_reserve(skb2, IPA_RX_BUFF_CLIENT_HEADROOM);
+ memcpy(skb2->data, skb->data, len);
+ skb2->len = len;
+ skb_set_tail_pointer(skb2, len);
+ }
+
+ return skb2;
+}
+
+static struct sk_buff *ipa_join_prev_skb(struct sk_buff *prev_skb,
+ struct sk_buff *skb, unsigned int len)
+{
+ struct sk_buff *skb2;
+
+ skb2 = skb_copy_expand(prev_skb, 0, len, GFP_KERNEL);
+ if (likely(skb2))
+ memcpy(skb_put(skb2, len), skb->data, len);
+ else
+ ipa_err("copy expand failed\n");
+ dev_kfree_skb_any(prev_skb);
+
+ return skb2;
+}
+
+static bool ipa_status_opcode_supported(enum ipahal_pkt_status_opcode opcode)
+{
+ return opcode == IPAHAL_PKT_STATUS_OPCODE_PACKET ||
+ opcode == IPAHAL_PKT_STATUS_OPCODE_DROPPED_PACKET ||
+ opcode == IPAHAL_PKT_STATUS_OPCODE_SUSPENDED_PACKET ||
+ opcode == IPAHAL_PKT_STATUS_OPCODE_PACKET_2ND_PASS;
+}
+
+static void
+ipa_lan_rx_pyld_hdlr(struct sk_buff *skb, struct ipa_sys_context *sys)
+{
+ struct ipahal_pkt_status status;
+ struct sk_buff *skb2;
+ unsigned long unused;
+ unsigned int align;
+ unsigned int used;
+ unsigned char *buf;
+ u32 pkt_status_sz;
+ int pad_len_byte;
+ u32 ep_id;
+ int len;
+ int len2;
+
+ pkt_status_sz = ipahal_pkt_status_get_size();
+ used = *(unsigned int *)skb->cb;
+ align = ALIGN(used, 32);
+ unused = IPA_RX_BUFFER_SIZE - used;
+
+ ipa_assert(skb->len);
+
+ if (sys->rx.len_partial) {
+ buf = skb_push(skb, sys->rx.len_partial);
+ memcpy(buf, sys->rx.prev_skb->data, sys->rx.len_partial);
+ sys->rx.len_partial = 0;
+ dev_kfree_skb_any(sys->rx.prev_skb);
+ sys->rx.prev_skb = NULL;
+ goto begin;
+ }
+
+ /* this endpoint has TX comp (status only) + mux-ed LAN RX data
+ * (status+data)
+ */
+ if (sys->rx.len_rem) {
+ if (sys->rx.len_rem <= skb->len) {
+ if (sys->rx.prev_skb) {
+ skb2 = skb_copy_expand(sys->rx.prev_skb, 0,
+ sys->rx.len_rem,
+ GFP_KERNEL);
+ if (likely(skb2)) {
+ memcpy(skb_put(skb2, sys->rx.len_rem),
+ skb->data, sys->rx.len_rem);
+ skb_trim(skb2,
+ skb2->len - sys->rx.len_pad);
+ skb2->truesize = skb2->len +
+ sizeof(struct sk_buff);
+ if (sys->rx.drop_packet)
+ dev_kfree_skb_any(skb2);
+ else
+ sys->ep->client_notify(
+ sys->ep->priv,
+ IPA_RECEIVE,
+ (unsigned long)(skb2));
+ } else {
+ ipa_err("copy expand failed\n");
+ }
+ dev_kfree_skb_any(sys->rx.prev_skb);
+ }
+ skb_pull(skb, sys->rx.len_rem);
+ sys->rx.prev_skb = NULL;
+ sys->rx.len_rem = 0;
+ sys->rx.len_pad = 0;
+ } else {
+ if (sys->rx.prev_skb) {
+ skb2 = ipa_join_prev_skb(sys->rx.prev_skb, skb,
+ skb->len);
+ dev_kfree_skb_any(sys->rx.prev_skb);
+ sys->rx.prev_skb = skb2;
+ }
+ sys->rx.len_rem -= skb->len;
+ return;
+ }
+ }
+
+begin:
+ while (skb->len) {
+ sys->rx.drop_packet = false;
+
+ if (skb->len < pkt_status_sz) {
+ WARN_ON(sys->rx.prev_skb);
+ sys->rx.prev_skb = skb_copy(skb, GFP_KERNEL);
+ sys->rx.len_partial = skb->len;
+ return;
+ }
+
+ ipahal_pkt_status_parse(skb->data, &status);
+
+ if (!ipa_status_opcode_supported(status.status_opcode)) {
+ ipa_err("unsupported opcode(%d)\n",
+ status.status_opcode);
+ skb_pull(skb, pkt_status_sz);
+ continue;
+ }
+
+ if (status.pkt_len == 0) {
+ skb_pull(skb, pkt_status_sz);
+ continue;
+ }
+
+ if (status.endp_dest_idx == (sys->ep - ipa_ctx->ep)) {
+ /* RX data */
+ ep_id = status.endp_src_idx;
+
+ /* A packet which is received back to the AP after
+ * there was no route match.
+ */
+
+ if (status.exception ==
+ IPAHAL_PKT_STATUS_EXCEPTION_NONE &&
+ status.rt_miss)
+ sys->rx.drop_packet = true;
+ if (skb->len == pkt_status_sz &&
+ status.exception ==
+ IPAHAL_PKT_STATUS_EXCEPTION_NONE) {
+ WARN_ON(sys->rx.prev_skb);
+ sys->rx.prev_skb = skb_copy(skb, GFP_KERNEL);
+ sys->rx.len_partial = skb->len;
+ return;
+ }
+
+ pad_len_byte = ((status.pkt_len + 3) & ~3) -
+ status.pkt_len;
+
+ len = status.pkt_len + pad_len_byte +
+ IPA_SIZE_DL_CSUM_META_TRAILER;
+
+ if (status.exception ==
+ IPAHAL_PKT_STATUS_EXCEPTION_DEAGGR) {
+ sys->rx.drop_packet = true;
+ }
+
+ len2 = min(status.pkt_len + pkt_status_sz, skb->len);
+ skb2 = ipa_skb_copy_for_client(skb, len2);
+ if (likely(skb2)) {
+ if (skb->len < len + pkt_status_sz) {
+ sys->rx.prev_skb = skb2;
+ sys->rx.len_rem = len - skb->len +
+ pkt_status_sz;
+ sys->rx.len_pad = pad_len_byte;
+ skb_pull(skb, skb->len);
+ } else {
+ skb_trim(skb2, status.pkt_len +
+ pkt_status_sz);
+ if (sys->rx.drop_packet) {
+ dev_kfree_skb_any(skb2);
+ } else {
+ skb2->truesize =
+ skb2->len +
+ sizeof(struct sk_buff) +
+ (ALIGN(len +
+ pkt_status_sz, 32) *
+ unused / align);
+ sys->ep->client_notify(
+ sys->ep->priv,
+ IPA_RECEIVE,
+ (unsigned long)(skb2));
+ }
+ skb_pull(skb, len + pkt_status_sz);
+ }
+ } else {
+ ipa_err("fail to alloc skb\n");
+ if (skb->len < len) {
+ sys->rx.prev_skb = NULL;
+ sys->rx.len_rem = len - skb->len +
+ pkt_status_sz;
+ sys->rx.len_pad = pad_len_byte;
+ skb_pull(skb, skb->len);
+ } else {
+ skb_pull(skb, len + pkt_status_sz);
+ }
+ }
+ } else {
+ skb_pull(skb, pkt_status_sz);
+ }
+ }
+}
+
+static void
+ipa_wan_rx_handle_splt_pyld(struct sk_buff *skb, struct ipa_sys_context *sys)
+{
+ struct sk_buff *skb2;
+
+ if (sys->rx.len_rem <= skb->len) {
+ if (sys->rx.prev_skb) {
+ skb2 = ipa_join_prev_skb(sys->rx.prev_skb, skb,
+ sys->rx.len_rem);
+ if (likely(skb2)) {
+ skb_pull(skb2, ipahal_pkt_status_get_size());
+ skb2->truesize = skb2->len +
+ sizeof(struct sk_buff);
+ sys->ep->client_notify(sys->ep->priv,
+ IPA_RECEIVE,
+ (unsigned long)skb2);
+ }
+ }
+ skb_pull(skb, sys->rx.len_rem);
+ sys->rx.prev_skb = NULL;
+ sys->rx.len_rem = 0;
+ } else {
+ if (sys->rx.prev_skb) {
+ skb2 = ipa_join_prev_skb(sys->rx.prev_skb, skb,
+ skb->len);
+ sys->rx.prev_skb = skb2;
+ }
+ sys->rx.len_rem -= skb->len;
+ skb_pull(skb, skb->len);
+ }
+}
+
+static void
+ipa_wan_rx_pyld_hdlr(struct sk_buff *skb, struct ipa_sys_context *sys)
+{
+ struct ipahal_pkt_status status;
+ unsigned char *skb_data;
+ struct sk_buff *skb2;
+ u16 pkt_len_with_pad;
+ unsigned long unused;
+ unsigned int align;
+ unsigned int used;
+ u32 pkt_status_sz;
+ int frame_len;
+ u32 qmap_hdr;
+ int checksum;
+ int ep_id;
+
+ used = *(unsigned int *)skb->cb;
+ align = ALIGN(used, 32);
+ unused = IPA_RX_BUFFER_SIZE - used;
+
+ ipa_assert(skb->len);
+
+ if (ipa_ctx->ipa_client_apps_wan_cons_agg_gro) {
+ sys->ep->client_notify(sys->ep->priv, IPA_RECEIVE,
+ (unsigned long)(skb));
+ return;
+ }
+
+ /* payload splits across 2 buff or more,
+ * take the start of the payload from rx.prev_skb
+ */
+ if (sys->rx.len_rem)
+ ipa_wan_rx_handle_splt_pyld(skb, sys);
+
+ pkt_status_sz = ipahal_pkt_status_get_size();
+ while (skb->len) {
+ u32 status_mask;
+
+ if (skb->len < pkt_status_sz) {
+ ipa_err("status straddles buffer\n");
+ WARN_ON(1);
+ goto bail;
+ }
+ ipahal_pkt_status_parse(skb->data, &status);
+ skb_data = skb->data;
+
+ if (!ipa_status_opcode_supported(status.status_opcode) ||
+ status.status_opcode ==
+ IPAHAL_PKT_STATUS_OPCODE_SUSPENDED_PACKET) {
+ ipa_err("unsupported opcode(%d)\n",
+ status.status_opcode);
+ skb_pull(skb, pkt_status_sz);
+ continue;
+ }
+
+ if (status.endp_dest_idx >= ipa_ctx->ep_count ||
+ status.endp_src_idx >= ipa_ctx->ep_count ||
+ status.pkt_len > IPA_GENERIC_AGGR_BYTE_LIMIT) {
+ ipa_err("status fields invalid\n");
+ WARN_ON(1);
+ goto bail;
+ }
+ if (status.pkt_len == 0) {
+ skb_pull(skb, pkt_status_sz);
+ continue;
+ }
+ ep_id = ipa_client_ep_id(IPA_CLIENT_APPS_WAN_CONS);
+ if (status.endp_dest_idx != ep_id) {
+ ipa_err("expected endp_dest_idx %d received %d\n",
+ ep_id, status.endp_dest_idx);
+ WARN_ON(1);
+ goto bail;
+ }
+ /* RX data */
+ if (skb->len == pkt_status_sz) {
+ ipa_err("Ins header in next buffer\n");
+ WARN_ON(1);
+ goto bail;
+ }
+ qmap_hdr = *(u32 *)(skb_data + pkt_status_sz);
+
+ /* Take the pkt_len_with_pad from the last 2 bytes of the QMAP
+ * header
+ */
+ /*QMAP is BE: convert the pkt_len field from BE to LE*/
+ pkt_len_with_pad = ntohs((qmap_hdr >> 16) & 0xffff);
+ /*get the CHECKSUM_PROCESS bit*/
+ status_mask = status.status_mask;
+ checksum = status_mask & IPAHAL_PKT_STATUS_MASK_CKSUM_PROCESS;
+
+ frame_len = pkt_status_sz + IPA_QMAP_HEADER_LENGTH +
+ pkt_len_with_pad;
+ if (checksum)
+ frame_len += IPA_DL_CHECKSUM_LENGTH;
+
+ skb2 = skb_clone(skb, GFP_ATOMIC);
+ if (likely(skb2)) {
+ /* the len of actual data is smaller than expected
+ * payload split across 2 buff
+ */
+ if (skb->len < frame_len) {
+ sys->rx.prev_skb = skb2;
+ sys->rx.len_rem = frame_len - skb->len;
+ skb_pull(skb, skb->len);
+ } else {
+ skb_trim(skb2, frame_len);
+ skb_pull(skb2, pkt_status_sz);
+ skb2->truesize = skb2->len +
+ sizeof(struct sk_buff) +
+ (ALIGN(frame_len, 32) *
+ unused / align);
+ sys->ep->client_notify(sys->ep->priv,
+ IPA_RECEIVE,
+ (unsigned long)(skb2));
+ skb_pull(skb, frame_len);
+ }
+ } else {
+ ipa_err("fail to clone\n");
+ if (skb->len < frame_len) {
+ sys->rx.prev_skb = NULL;
+ sys->rx.len_rem = frame_len - skb->len;
+ skb_pull(skb, skb->len);
+ } else {
+ skb_pull(skb, frame_len);
+ }
+ }
+ }
+bail:
+ dev_kfree_skb_any(skb);
+}
+
+void ipa_lan_rx_cb(void *priv, enum ipa_dp_evt_type evt, unsigned long data)
+{
+ struct sk_buff *rx_skb = (struct sk_buff *)data;
+ struct ipahal_pkt_status status;
+ struct ipa_ep_context *ep;
+ u32 pkt_status_size;
+ u32 metadata;
+ u32 ep_id;
+
+ pkt_status_size = ipahal_pkt_status_get_size();
+
+ ipa_assert(rx_skb->len >= pkt_status_size);
+
+ ipahal_pkt_status_parse(rx_skb->data, &status);
+ ep_id = status.endp_src_idx;
+ metadata = status.metadata;
+ ep = &ipa_ctx->ep[ep_id];
+ if (ep_id >= ipa_ctx->ep_count || !ep->allocated ||
+ !ep->client_notify) {
+ ipa_err("drop endpoint=%u allocated=%s client_notify=%p\n",
+ ep_id, ep->allocated ? "true" : "false",
+ ep->client_notify);
+ dev_kfree_skb_any(rx_skb);
+ return;
+ }
+
+ /* Consume the status packet, and if no exception, the header */
+ skb_pull(rx_skb, pkt_status_size);
+ if (status.exception == IPAHAL_PKT_STATUS_EXCEPTION_NONE)
+ skb_pull(rx_skb, IPA_LAN_RX_HEADER_LENGTH);
+
+ /* Metadata Info
+ * ------------------------------------------
+ * | 3 | 2 | 1 | 0 |
+ * | fw_desc | vdev_id | qmap mux id | Resv |
+ * ------------------------------------------
+ */
+ *(u16 *)rx_skb->cb = ((metadata >> 16) & 0xffff);
+
+ ep->client_notify(ep->priv, IPA_RECEIVE, (unsigned long)rx_skb);
+}
+
+static void ipa_rx_common(struct ipa_sys_context *sys, u32 size)
+{
+ struct ipa_rx_pkt_wrapper *rx_pkt;
+ struct sk_buff *rx_skb;
+
+ ipa_assert(!list_empty(&sys->head_desc_list));
+
+ spin_lock_bh(&sys->spinlock);
+
+ rx_pkt = list_first_entry(&sys->head_desc_list,
+ struct ipa_rx_pkt_wrapper, link);
+ list_del(&rx_pkt->link);
+ sys->len--;
+
+ spin_unlock_bh(&sys->spinlock);
+
+ rx_skb = rx_pkt->skb;
+ dma_unmap_single(ipa_ctx->dev, rx_pkt->dma_addr, sys->rx.buff_sz,
+ DMA_FROM_DEVICE);
+
+ skb_trim(rx_skb, size);
+
+ *(unsigned int *)rx_skb->cb = rx_skb->len;
+ rx_skb->truesize = size + sizeof(struct sk_buff);
+
+ sys->rx.pyld_hdlr(rx_skb, sys);
+ kmem_cache_free(ipa_ctx->dp->rx_pkt_wrapper_cache, rx_pkt);
+ ipa_replenish_rx_cache(sys);
+}
+
+/**
+ * ipa_aggr_byte_limit_buf_size()
+ * @byte_limit: Desired limit (in bytes) for aggregation
+ *
+ * Compute the buffer size required to support a requested aggregation
+ * byte limit. Aggregration will close when *more* than the configured
+ * number of bytes have been added to an aggregation frame. Our
+ * buffers therefore need to to be big enough to receive one complete
+ * packet once the configured byte limit has been consumed.
+ *
+ * An incoming packet can have as much as IPA_MTU of data in it, but
+ * the buffer also needs to be large enough to accomodate the standard
+ * socket buffer overhead (NET_SKB_PAD of headroom, plus an implied
+ * skb_shared_info structure at the end).
+ *
+ * So we compute the required buffer size by adding the standard
+ * socket buffer overhead and MTU to the requested size. We round
+ * that down to a power of 2 in an effort to avoid fragmentation due
+ * to unaligned buffer sizes.
+ *
+ * After accounting for all of this, we return the number of bytes
+ * of buffer space the IPA hardware will know is available to hold
+ * received data (without any overhead).
+ *
+ * Return: The computes size of buffer space available
+ */
+u32 ipa_aggr_byte_limit_buf_size(u32 byte_limit)
+{
+ /* Account for one additional packet, including overhead */
+ byte_limit += IPA_RX_BUFFER_RESERVED;
+ byte_limit += IPA_MTU;
+
+ /* Convert this size to a nearby power-of-2. We choose one
+ * that's *less than* the limit we seek--so we start by
+ * subracting 1. The highest set bit in that is used to
+ * compute the power of 2.
+ *
+ * XXX Why is this *less than* and not possibly equal?
+ */
+ byte_limit = 1 << __fls(byte_limit - 1);
+
+ /* Given that size, figure out how much buffer space that
+ * leaves us for received data.
+ */
+ return IPA_RX_BUFFER_AVAILABLE(byte_limit);
+}
+
+void ipa_gsi_irq_tx_notify_cb(void *xfer_data)
+{
+ struct ipa_tx_pkt_wrapper *tx_pkt = xfer_data;
+
+ queue_work(tx_pkt->sys->wq, &tx_pkt->done_work);
+}
+
+void ipa_gsi_irq_rx_notify_cb(void *chan_data, u16 count)
+{
+ struct ipa_sys_context *sys = chan_data;
+
+ sys->ep->bytes_xfered_valid = true;
+ sys->ep->bytes_xfered = count;
+
+ ipa_rx_switch_to_poll_mode(sys);
+}
+
+static int ipa_gsi_setup_channel(struct ipa_ep_context *ep, u32 channel_count,
+ u32 evt_ring_mult)
+{
+ u32 channel_id = ipa_client_channel_id(ep->client);
+ u32 tlv_count = ipa_client_tlv_count(ep->client);
+ bool from_ipa = ipa_consumer(ep->client);
+ bool moderation;
+ bool priority;
+ int ret;
+
+ priority = ep->client == IPA_CLIENT_APPS_CMD_PROD;
+ moderation = !ep->sys->tx.no_intr;
+
+ ret = gsi_channel_alloc(ipa_ctx->gsi, channel_id, channel_count,
+ from_ipa, priority, evt_ring_mult, moderation,
+ ep->sys);
+ if (ret)
+ return ret;
+ ep->channel_id = channel_id;
+
+ gsi_channel_scratch_write(ipa_ctx->gsi, ep->channel_id, tlv_count);
+
+ ret = gsi_channel_start(ipa_ctx->gsi, ep->channel_id);
+ if (ret)
+ gsi_channel_free(ipa_ctx->gsi, ep->channel_id);
+
+ return ret;
+}
+
+void ipa_endp_init_hdr_cons(u32 ep_id, u32 header_size,
+ u32 metadata_offset, u32 length_offset)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_hdr_cons(&ep->init_hdr, header_size, metadata_offset,
+ length_offset);
+}
+
+void ipa_endp_init_hdr_prod(u32 ep_id, u32 header_size,
+ u32 metadata_offset, u32 length_offset)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_hdr_prod(&ep->init_hdr, header_size, metadata_offset,
+ length_offset);
+}
+
+void
+ipa_endp_init_hdr_ext_cons(u32 ep_id, u32 pad_align, bool pad_included)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_hdr_ext_cons(&ep->hdr_ext, pad_align, pad_included);
+}
+
+void ipa_endp_init_hdr_ext_prod(u32 ep_id, u32 pad_align)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_hdr_ext_prod(&ep->hdr_ext, pad_align);
+}
+
+void
+ipa_endp_init_aggr_cons(u32 ep_id, u32 size, u32 count, bool close_on_eof)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_aggr_cons(&ep->init_aggr, size, count, close_on_eof);
+}
+
+void ipa_endp_init_aggr_prod(u32 ep_id, enum ipa_aggr_en aggr_en,
+ enum ipa_aggr_type aggr_type)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_aggr_prod(&ep->init_aggr, aggr_en, aggr_type);
+}
+
+void ipa_endp_init_cfg_cons(u32 ep_id, enum ipa_cs_offload_en offload_type)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_cfg_cons(&ep->init_cfg, offload_type);
+}
+
+void ipa_endp_init_cfg_prod(u32 ep_id, enum ipa_cs_offload_en offload_type,
+ u32 metadata_offset)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_cfg_prod(&ep->init_cfg, offload_type,
+ metadata_offset);
+}
+
+void ipa_endp_init_hdr_metadata_mask_cons(u32 ep_id, u32 mask)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_hdr_metadata_mask_cons(&ep->metadata_mask, mask);
+}
+
+void ipa_endp_init_hdr_metadata_mask_prod(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_hdr_metadata_mask_prod(&ep->metadata_mask);
+}
+
+void ipa_endp_status_cons(u32 ep_id, bool enable)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_status_cons(&ep->status, enable);
+}
+
+void ipa_endp_status_prod(u32 ep_id, bool enable,
+ enum ipa_client_type status_client)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+ u32 status_ep_id;
+
+ status_ep_id = ipa_client_ep_id(status_client);
+
+ ipa_reg_endp_status_prod(&ep->status, enable, status_ep_id);
+}
+
+
+/* Note that the mode setting is not valid for consumer endpoints */
+void ipa_endp_init_mode_cons(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_mode_cons(&ep->init_mode);
+}
+
+void ipa_endp_init_mode_prod(u32 ep_id, enum ipa_mode mode,
+ enum ipa_client_type dst_client)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+ u32 dst_ep_id;
+
+ dst_ep_id = ipa_client_ep_id(dst_client);
+
+ ipa_reg_endp_init_mode_prod(&ep->init_mode, mode, dst_ep_id);
+}
+
+/* XXX The sequencer setting seems not to be valid for consumer endpoints */
+void ipa_endp_init_seq_cons(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_seq_cons(&ep->init_seq);
+}
+
+void ipa_endp_init_seq_prod(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+ u32 seq_type;
+
+ seq_type = (u32)ipa_endp_seq_type(ep_id);
+
+ ipa_reg_endp_init_seq_prod(&ep->init_seq, seq_type);
+}
+
+/* XXX The deaggr setting seems not to be valid for consumer endpoints */
+void ipa_endp_init_deaggr_cons(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_deaggr_cons(&ep->init_deaggr);
+}
+
+void ipa_endp_init_deaggr_prod(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_deaggr_prod(&ep->init_deaggr);
+}
+
+int ipa_ep_alloc(enum ipa_client_type client)
+{
+ u32 ep_id = ipa_client_ep_id(client);
+ struct ipa_sys_context *sys;
+ struct ipa_ep_context *ep;
+
+ ep = &ipa_ctx->ep[ep_id];
+
+ ipa_assert(!ep->allocated);
+
+ /* Reuse the endpoint's sys pointer if it is initialized */
+ sys = ep->sys;
+ if (!sys) {
+ sys = ipa_ep_sys_create(client);
+ if (!sys)
+ return -ENOMEM;
+ sys->ep = ep;
+ }
+
+ /* Zero the "mutable" part of the system context */
+ memset(sys, 0, offsetof(struct ipa_sys_context, ep));
+
+ /* Initialize the endpoint context */
+ memset(ep, 0, sizeof(*ep));
+ ep->sys = sys;
+ ep->client = client;
+ ep->allocated = true;
+
+ return ep_id;
+}
+
+void ipa_ep_free(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_assert(ep->allocated);
+
+ ep->allocated = false;
+}
+
+/**
+ * ipa_ep_setup() - Set up an IPA endpoint
+ * @ep_id: Endpoint to set up
+ * @channel_count: Number of transfer elements in the channel
+ * @evt_ring_mult: Used to determine number of elements in event ring
+ * @rx_buffer_size: Receive buffer size to use (or 0 for TX endpoitns)
+ * @client_notify: Notify function to call on completion
+ * @priv: Value supplied to the notify function
+ *
+ * Returns: 0 if successful, or a negative error code
+ */
+int ipa_ep_setup(u32 ep_id, u32 channel_count, u32 evt_ring_mult,
+ u32 rx_buffer_size,
+ void (*client_notify)(void *priv, enum ipa_dp_evt_type type,
+ unsigned long data),
+ void *priv)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+ int ret;
+
+ if (ipa_consumer(ep->client)) {
+ atomic_set(&ep->sys->rx.curr_polling_state, 0);
+ INIT_DELAYED_WORK(&ep->sys->rx.switch_to_intr_work,
+ ipa_switch_to_intr_rx_work_func);
+ if (ep->client == IPA_CLIENT_APPS_LAN_CONS)
+ ep->sys->rx.pyld_hdlr = ipa_lan_rx_pyld_hdlr;
+ else
+ ep->sys->rx.pyld_hdlr = ipa_wan_rx_pyld_hdlr;
+ ep->sys->rx.buff_sz = rx_buffer_size;
+ ep->sys->rx.pool_sz = IPA_GENERIC_RX_POOL_SZ;
+ INIT_WORK(&ep->sys->rx.work, ipa_wq_handle_rx);
+ INIT_DELAYED_WORK(&ep->sys->rx.replenish_work,
+ ipa_replenish_rx_work_func);
+ }
+
+ ep->client_notify = client_notify;
+ ep->priv = priv;
+ ep->napi_enabled = ep->client == IPA_CLIENT_APPS_WAN_CONS;
+
+ ipa_client_add();
+
+ ipa_cfg_ep(ep_id);
+
+ ret = ipa_gsi_setup_channel(ep, channel_count, evt_ring_mult);
+ if (ret)
+ goto err_client_remove;
+
+ if (ipa_consumer(ep->client))
+ ipa_replenish_rx_cache(ep->sys);
+err_client_remove:
+ ipa_client_remove();
+
+ return ret;
+}
+
+/**
+ * ipa_channel_reset_aggr() - Reset with aggregation active
+ * @ep_id: Endpoint on which reset is performed
+ *
+ * If aggregation is active on a channel when a reset is performed,
+ * a special sequence of actions must be taken. This is a workaround
+ * for a hardware limitation.
+ *
+ * Return: 0 if successful, or a negative error code.
+ */
+static int ipa_channel_reset_aggr(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+ struct ipa_reg_aggr_force_close force_close;
+ struct ipa_reg_endp_init_ctrl init_ctrl;
+ struct gsi_xfer_elem xfer_elem = { };
+ struct ipa_dma_mem dma_byte;
+ int aggr_active_bitmap = 0;
+ bool ep_suspended = false;
+ int ret;
+ int i;
+
+ ipa_reg_aggr_force_close(&force_close, BIT(ep_id));
+ ipa_write_reg_fields(IPA_AGGR_FORCE_CLOSE, &force_close);
+
+ /* Reset channel */
+ ret = gsi_channel_reset(ipa_ctx->gsi, ep->channel_id);
+ if (ret)
+ return ret;
+
+ /* Turn off the doorbell engine. We're going to poll until
+ * we know aggregation isn't active.
+ */
+ gsi_channel_config(ipa_ctx->gsi, ep->channel_id, false);
+
+ ipa_read_reg_n_fields(IPA_ENDP_INIT_CTRL_N, ep_id, &init_ctrl);
+ if (init_ctrl.endp_suspend) {
+ ep_suspended = true;
+ ipa_reg_endp_init_ctrl(&init_ctrl, false);
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_CTRL_N, ep_id, &init_ctrl);
+ }
+
+ /* Start channel and put 1 Byte descriptor on it */
+ ret = gsi_channel_start(ipa_ctx->gsi, ep->channel_id);
+ if (ret)
+ goto out_suspend_again;
+
+ if (ipa_dma_alloc(&dma_byte, 1, GFP_KERNEL)) {
+ ret = -ENOMEM;
+ goto err_stop_channel;
+ }
+
+ xfer_elem.addr = dma_byte.phys;
+ xfer_elem.len_opcode = 1; /* = dma_byte.size; */
+ xfer_elem.flags = GSI_XFER_FLAG_EOT;
+ xfer_elem.type = GSI_XFER_ELEM_DATA;
+
+ ret = gsi_channel_queue(ipa_ctx->gsi, ep->channel_id, 1, &xfer_elem,
+ true);
+ if (ret)
+ goto err_dma_free;
+
+ /* Wait for aggregation frame to be closed */
+ for (i = 0; i < CHANNEL_RESET_AGGR_RETRY_COUNT; i++) {
+ aggr_active_bitmap = ipa_read_reg(IPA_STATE_AGGR_ACTIVE);
+ if (!(aggr_active_bitmap & BIT(ep_id)))
+ break;
+ msleep(CHANNEL_RESET_DELAY);
+ }
+ ipa_bug_on(aggr_active_bitmap & BIT(ep_id));
+
+ ipa_dma_free(&dma_byte);
+
+ ret = ipa_stop_gsi_channel(ep_id);
+ if (ret)
+ goto out_suspend_again;
+
+ /* Reset the channel. If successful we need to sleep for 1
+ * msec to complete the GSI channel reset sequence. Either
+ * way we finish by suspending the channel again (if necessary)
+ * and re-enabling its doorbell engine.
+ */
+ ret = gsi_channel_reset(ipa_ctx->gsi, ep->channel_id);
+ if (!ret)
+ msleep(CHANNEL_RESET_DELAY);
+ goto out_suspend_again;
+
+err_dma_free:
+ ipa_dma_free(&dma_byte);
+err_stop_channel:
+ ipa_stop_gsi_channel(ep_id);
+out_suspend_again:
+ if (ep_suspended) {
+ ipa_reg_endp_init_ctrl(&init_ctrl, true);
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_CTRL_N, ep_id, &init_ctrl);
+ }
+ /* Turn on the doorbell engine again */
+ gsi_channel_config(ipa_ctx->gsi, ep->channel_id, true);
+
+ return ret;
+}
+
+static void ipa_reset_gsi_channel(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+ u32 aggr_active_bitmap = 0;
+
+ /* For consumer endpoints, a hardware limitation prevents us
+ * from issuing a channel reset if aggregation is active.
+ * Check for this case, and if detected, perform a special
+ * reset sequence. Otherwise just do a "normal" reset.
+ */
+ if (ipa_consumer(ep->client))
+ aggr_active_bitmap = ipa_read_reg(IPA_STATE_AGGR_ACTIVE);
+
+ if (aggr_active_bitmap & BIT(ep_id)) {
+ ipa_bug_on(ipa_channel_reset_aggr(ep_id));
+ } else {
+ /* In case the reset follows stop, need to wait 1 msec */
+ msleep(CHANNEL_RESET_DELAY);
+ ipa_bug_on(gsi_channel_reset(ipa_ctx->gsi, ep->channel_id));
+ }
+}
+
+/**
+ * ipa_ep_teardown() - Tear down an endpoint
+ * @ep_id: The endpoint to tear down
+ */
+void ipa_ep_teardown(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+ int empty;
+ int ret;
+ int i;
+
+ if (ep->napi_enabled) {
+ do {
+ usleep_range(95, 105);
+ } while (ipa_ep_polling(ep));
+ }
+
+ if (ipa_producer(ep->client)) {
+ do {
+ spin_lock_bh(&ep->sys->spinlock);
+ empty = list_empty(&ep->sys->head_desc_list);
+ spin_unlock_bh(&ep->sys->spinlock);
+ if (!empty)
+ usleep_range(95, 105);
+ else
+ break;
+ } while (1);
+ }
+
+ if (ipa_consumer(ep->client))
+ cancel_delayed_work_sync(&ep->sys->rx.replenish_work);
+ flush_workqueue(ep->sys->wq);
+ /* channel stop might fail on timeout if IPA is busy */
+ for (i = 0; i < IPA_GSI_CHANNEL_STOP_MAX_RETRY; i++) {
+ ret = ipa_stop_gsi_channel(ep_id);
+ if (!ret)
+ break;
+ ipa_bug_on(ret != -EAGAIN && ret != -ETIMEDOUT);
+ }
+
+ ipa_reset_gsi_channel(ep_id);
+ gsi_channel_free(ipa_ctx->gsi, ep->channel_id);
+
+ if (ipa_consumer(ep->client))
+ ipa_cleanup_rx(ep->sys);
+
+ ipa_ep_free(ep_id);
+}
+
+static int ipa_poll_gsi_pkt(struct ipa_sys_context *sys)
+{
+ if (sys->ep->bytes_xfered_valid) {
+ sys->ep->bytes_xfered_valid = false;
+
+ return (int)sys->ep->bytes_xfered;
+ }
+
+ return gsi_channel_poll(ipa_ctx->gsi, sys->ep->channel_id);
+}
+
+bool ipa_ep_polling(struct ipa_ep_context *ep)
+{
+ ipa_assert(ipa_consumer(ep->client));
+
+ return !!atomic_read(&ep->sys->rx.curr_polling_state);
+}
+
+struct ipa_dp *ipa_dp_init(void)
+{
+ struct kmem_cache *cache;
+ struct ipa_dp *dp;
+
+ dp = kzalloc(sizeof(*dp), GFP_KERNEL);
+ if (!dp)
+ return NULL;
+
+ cache = kmem_cache_create("IPA_TX_PKT_WRAPPER",
+ sizeof(struct ipa_tx_pkt_wrapper),
+ 0, 0, NULL);
+ if (!cache) {
+ kfree(dp);
+ return NULL;
+ }
+ dp->tx_pkt_wrapper_cache = cache;
+
+ cache = kmem_cache_create("IPA_RX_PKT_WRAPPER",
+ sizeof(struct ipa_rx_pkt_wrapper),
+ 0, 0, NULL);
+ if (!cache) {
+ kmem_cache_destroy(dp->tx_pkt_wrapper_cache);
+ kfree(dp);
+ return NULL;
+ }
+ dp->rx_pkt_wrapper_cache = cache;
+
+ return dp;
+}
+
+void ipa_dp_exit(struct ipa_dp *dp)
+{
+ kmem_cache_destroy(dp->rx_pkt_wrapper_cache);
+ kmem_cache_destroy(dp->tx_pkt_wrapper_cache);
+ kfree(dp);
+}
--
2.17.1
^ permalink raw reply related
* [RFC PATCH 09/12] soc: qcom: ipa: main IPA source file
From: Alex Elder @ 2018-11-07 0:32 UTC (permalink / raw)
To: davem, arnd, bjorn.andersson, ilias.apalodimas
Cc: netdev, devicetree, linux-arm-msm, linux-soc, linux-arm-kernel,
linux-kernel, syadagir, mjavid, robh+dt, mark.rutland
In-Reply-To: <20181107003250.5832-1-elder@linaro.org>
This patch includes "ipa_main.c", which consists mostly of the
initialization code.
The IPA is a hardware resource shared by multiple independent
execution environments (currently, the AP and the modem). In some
cases, initialization must be performed by only one of these. As an
example, the AP must initialize some filter table data structures
that are only used by the modem. (And in general, some initialization
of IPA hardware is required regardless of whether it will be used.)
There are two phases of IPA initialization. The first phase is
triggered by the probe of the driver. It involves setting up
operating system resources, and doing some basic initialization
of IPA memory resources using register and DMA access.
The second phase involves configuration of enpoints used, and this
phase requires access to the GSI layer. However the GSI layer is
requires some firmware to be loaded before it can be used. So
the second stage (in ipa_post_init()) only occurs after it is known
firmware is loaded.
The GSI firmware can be loaded in two ways: the modem can load it;
or Trust Zone code running on the AP can load it. If the modem
loads the firmware, it will send an SMP2P interrupt to the AP to
signal that GSI firmware is loaded and the AP can proceed with its
second stage IPA initialization. If Trust Zone is responsible for
loading the firmware, the IPA driver requests the firmware blob
from the file system and passes the result via an SMC to Trust Zone
to load and activate the GSI firmware. When that has completed
successfully, the second stage of initialization can proceed.
Signed-off-by: Alex Elder <elder@linaro.org>
---
drivers/net/ipa/ipa_main.c | 1400 ++++++++++++++++++++++++++++++++++++
1 file changed, 1400 insertions(+)
create mode 100644 drivers/net/ipa/ipa_main.c
diff --git a/drivers/net/ipa/ipa_main.c b/drivers/net/ipa/ipa_main.c
new file mode 100644
index 000000000000..3d7c59177388
--- /dev/null
+++ b/drivers/net/ipa/ipa_main.c
@@ -0,0 +1,1400 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/atomic.h>
+#include <linux/spinlock.h>
+#include <linux/mutex.h>
+#include <linux/device.h>
+#include <linux/firmware.h>
+#include <linux/workqueue.h>
+#include <linux/bug.h>
+#include <linux/io.h>
+#include <linux/clk.h>
+#include <linux/of.h>
+#include <linux/of_irq.h>
+#include <linux/of_address.h>
+#include <linux/of_platform.h>
+#include <linux/interrupt.h>
+#include <linux/notifier.h>
+#include <linux/remoteproc.h>
+#include <linux/pm_wakeup.h>
+#include <linux/kconfig.h>
+#include <linux/qcom_scm.h>
+#include <linux/soc/qcom/mdt_loader.h>
+#include <linux/soc/qcom/smem.h>
+#include <linux/soc/qcom/smem_state.h>
+#include <linux/module.h>
+
+#include "ipa_i.h"
+#include "ipa_dma.h"
+#include "ipahal.h"
+
+#define IPA_CORE_CLOCK_RATE (75UL * 1000 * 1000)
+
+/* The name of the main firmware file relative to /lib/firmware */
+#define IPA_FWS_PATH "ipa_fws.mdt"
+#define IPA_PAS_ID 15
+
+#define IPA_APPS_CMD_PROD_RING_COUNT 256
+#define IPA_APPS_LAN_CONS_RING_COUNT 256
+
+/* Details of the initialization sequence are determined by who is
+ * responsible for doing some early IPA hardware initialization.
+ * The Device Tree compatible string defines what to expect.
+ */
+enum ipa_init_type {
+ ipa_undefined_init = 0,
+ ipa_tz_init,
+ ipa_modem_init,
+};
+
+struct ipa_match_data {
+ enum ipa_init_type init_type;
+};
+
+static void ipa_client_remove_deferred(struct work_struct *work);
+static DECLARE_WORK(ipa_client_remove_work, ipa_client_remove_deferred);
+
+static struct ipa_context ipa_ctx_struct;
+struct ipa_context *ipa_ctx = &ipa_ctx_struct;
+
+static int hdr_init_local_cmd(u32 offset, u32 size)
+{
+ struct ipa_desc desc = { };
+ struct ipa_dma_mem mem;
+ void *payload;
+ int ret;
+
+ if (ipa_dma_alloc(&mem, size, GFP_KERNEL))
+ return -ENOMEM;
+
+ offset += ipa_ctx->smem_offset;
+
+ payload = ipahal_hdr_init_local_pyld(&mem, offset);
+ if (!payload) {
+ ret = -ENOMEM;
+ goto err_dma_free;
+ }
+
+ desc.type = IPA_IMM_CMD_DESC;
+ desc.len_opcode = IPA_IMM_CMD_HDR_INIT_LOCAL;
+ desc.payload = payload;
+
+ ret = ipa_send_cmd(&desc);
+
+ ipahal_payload_free(payload);
+err_dma_free:
+ ipa_dma_free(&mem);
+
+ return ret;
+}
+
+static int dma_shared_mem_zero_cmd(u32 offset, u32 size)
+{
+ struct ipa_desc desc = { };
+ struct ipa_dma_mem mem;
+ void *payload;
+ int ret;
+
+ ipa_assert(size > 0);
+
+ if (ipa_dma_alloc(&mem, size, GFP_KERNEL))
+ return -ENOMEM;
+
+ offset += ipa_ctx->smem_offset;
+
+ payload = ipahal_dma_shared_mem_write_pyld(&mem, offset);
+ if (!payload) {
+ ret = -ENOMEM;
+ goto err_dma_free;
+ }
+
+ desc.type = IPA_IMM_CMD_DESC;
+ desc.len_opcode = IPA_IMM_CMD_DMA_SHARED_MEM;
+ desc.payload = payload;
+
+ ret = ipa_send_cmd(&desc);
+
+ ipahal_payload_free(payload);
+err_dma_free:
+ ipa_dma_free(&mem);
+
+ return ret;
+}
+
+/**
+ * ipa_modem_smem_init() - Initialize modem general memory and header memory
+ */
+int ipa_modem_smem_init(void)
+{
+ int ret;
+
+ ret = dma_shared_mem_zero_cmd(IPA_MEM_MODEM_OFST, IPA_MEM_MODEM_SIZE);
+ if (ret)
+ return ret;
+
+ ret = dma_shared_mem_zero_cmd(IPA_MEM_MODEM_HDR_OFST,
+ IPA_MEM_MODEM_HDR_SIZE);
+ if (ret)
+ return ret;
+
+ return dma_shared_mem_zero_cmd(IPA_MEM_MODEM_HDR_PROC_CTX_OFST,
+ IPA_MEM_MODEM_HDR_PROC_CTX_SIZE);
+}
+
+static int ipa_ep_apps_cmd_prod_setup(void)
+{
+ enum ipa_client_type dst_client;
+ enum ipa_client_type client;
+ u32 channel_count;
+ u32 ep_id;
+ int ret;
+
+ if (ipa_ctx->cmd_prod_ep_id != IPA_EP_ID_BAD)
+ ret = -EBUSY;
+
+ client = IPA_CLIENT_APPS_CMD_PROD;
+ dst_client = IPA_CLIENT_APPS_LAN_CONS;
+ channel_count = IPA_APPS_CMD_PROD_RING_COUNT;
+
+ ret = ipa_ep_alloc(client);
+ if (ret < 0)
+ return ret;
+ ep_id = ret;
+
+
+ ipa_endp_init_mode_prod(ep_id, IPA_DMA, dst_client);
+ ipa_endp_init_seq_prod(ep_id);
+ ipa_endp_init_deaggr_prod(ep_id);
+
+ ret = ipa_ep_setup(ep_id, channel_count, 2, 0, NULL, NULL);
+ if (ret)
+ ipa_ep_free(ep_id);
+ else
+ ipa_ctx->cmd_prod_ep_id = ep_id;
+
+ return ret;
+}
+
+/* Only used for IPA_MEM_UC_EVENT_RING_OFST, which must be 1KB aligned */
+static __always_inline void sram_set_canary(u32 *sram_mmio, u32 offset)
+{
+ BUILD_BUG_ON(offset < sizeof(*sram_mmio));
+ BUILD_BUG_ON(offset % 1024);
+
+ sram_mmio += offset / sizeof(*sram_mmio);
+ *--sram_mmio = IPA_MEM_CANARY_VAL;
+}
+
+static __always_inline void sram_set_canaries(u32 *sram_mmio, u32 offset)
+{
+ BUILD_BUG_ON(offset < 2 * sizeof(*sram_mmio));
+ BUILD_BUG_ON(offset % 8);
+
+ sram_mmio += offset / sizeof(*sram_mmio);
+ *--sram_mmio = IPA_MEM_CANARY_VAL;
+ *--sram_mmio = IPA_MEM_CANARY_VAL;
+}
+
+/**
+ * ipa_init_sram() - Initialize IPA local SRAM.
+ *
+ * Return: 0 if successful, or a negative error code
+ */
+static int ipa_init_sram(void)
+{
+ phys_addr_t phys_addr;
+ u32 *ipa_sram_mmio;
+
+ phys_addr = ipa_ctx->ipa_phys;
+ phys_addr += ipa_reg_n_offset(IPA_SRAM_DIRECT_ACCESS_N, 0);
+ phys_addr += ipa_ctx->smem_offset;
+
+ ipa_sram_mmio = ioremap(phys_addr, ipa_ctx->smem_size);
+ if (!ipa_sram_mmio) {
+ ipa_err("fail to ioremap IPA SRAM\n");
+ return -ENOMEM;
+ }
+
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_V4_FLT_HASH_OFST);
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_V4_FLT_NHASH_OFST);
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_V6_FLT_HASH_OFST);
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_V6_FLT_NHASH_OFST);
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_V4_RT_HASH_OFST);
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_V4_RT_NHASH_OFST);
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_V6_RT_HASH_OFST);
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_V6_RT_NHASH_OFST);
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_MODEM_HDR_OFST);
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_MODEM_HDR_PROC_CTX_OFST);
+ sram_set_canaries(ipa_sram_mmio, IPA_MEM_MODEM_OFST);
+
+ /* Only one canary precedes the microcontroller ring */
+ sram_set_canary(ipa_sram_mmio, IPA_MEM_UC_EVENT_RING_OFST);
+
+ iounmap(ipa_sram_mmio);
+
+ return 0;
+}
+
+/**
+ * ipa_init_hdr() - Initialize IPA header block.
+ *
+ * Return: 0 if successful, or a negative error code
+ */
+static int ipa_init_hdr(void)
+{
+ int ret;
+
+ if (IPA_MEM_MODEM_HDR_SIZE) {
+ ret = hdr_init_local_cmd(IPA_MEM_MODEM_HDR_OFST,
+ IPA_MEM_MODEM_HDR_SIZE);
+ if (ret)
+ return ret;
+ }
+
+ if (IPA_MEM_APPS_HDR_SIZE) {
+ BUILD_BUG_ON(IPA_MEM_APPS_HDR_OFST % 8);
+ ret = hdr_init_local_cmd(IPA_MEM_APPS_HDR_OFST,
+ IPA_MEM_APPS_HDR_SIZE);
+ if (ret)
+ return ret;
+ }
+
+ if (IPA_MEM_MODEM_HDR_PROC_CTX_SIZE) {
+ ret = dma_shared_mem_zero_cmd(IPA_MEM_MODEM_HDR_PROC_CTX_OFST,
+ IPA_MEM_MODEM_HDR_PROC_CTX_SIZE);
+ if (ret)
+ return ret;
+ }
+
+ if (IPA_MEM_APPS_HDR_PROC_CTX_SIZE) {
+ BUILD_BUG_ON(IPA_MEM_APPS_HDR_PROC_CTX_OFST % 8);
+ ret = dma_shared_mem_zero_cmd(IPA_MEM_APPS_HDR_PROC_CTX_OFST,
+ IPA_MEM_APPS_HDR_PROC_CTX_SIZE);
+ if (ret)
+ return ret;
+ }
+
+ ipa_write_reg(IPA_LOCAL_PKT_PROC_CNTXT_BASE,
+ ipa_ctx->smem_offset + IPA_MEM_MODEM_HDR_PROC_CTX_OFST);
+
+ return 0;
+}
+
+/**
+ * ipa_init_rt4() - Initialize IPA routing block for IPv4.
+ *
+ * Return: 0 if successful, or a negative error code
+ */
+static int ipa_init_rt4(struct ipa_dma_mem *mem)
+{
+ struct ipa_desc desc = { };
+ u32 nhash_offset;
+ u32 hash_offset;
+ void *payload;
+ int ret;
+
+ hash_offset = ipa_ctx->smem_offset + IPA_MEM_V4_RT_HASH_OFST;
+ nhash_offset = ipa_ctx->smem_offset + IPA_MEM_V4_RT_NHASH_OFST;
+ payload = ipahal_ip_v4_routing_init_pyld(mem, hash_offset,
+ nhash_offset);
+ if (!payload)
+ return -ENOMEM;
+
+ desc.type = IPA_IMM_CMD_DESC;
+ desc.len_opcode = IPA_IMM_CMD_IP_V4_ROUTING_INIT;
+ desc.payload = payload;
+
+ ret = ipa_send_cmd(&desc);
+
+ ipahal_payload_free(payload);
+
+ return ret;
+}
+
+/**
+ * ipa_init_rt6() - Initialize IPA routing block for IPv6.
+ *
+ * Return: 0 if successful, or a negative error code
+ */
+static int ipa_init_rt6(struct ipa_dma_mem *mem)
+{
+ struct ipa_desc desc = { };
+ u32 nhash_offset;
+ u32 hash_offset;
+ void *payload;
+ int ret;
+
+ hash_offset = ipa_ctx->smem_offset + IPA_MEM_V6_RT_HASH_OFST;
+ nhash_offset = ipa_ctx->smem_offset + IPA_MEM_V6_RT_NHASH_OFST;
+ payload = ipahal_ip_v6_routing_init_pyld(mem, hash_offset,
+ nhash_offset);
+ if (!payload)
+ return -ENOMEM;
+
+ desc.type = IPA_IMM_CMD_DESC;
+ desc.len_opcode = IPA_IMM_CMD_IP_V6_ROUTING_INIT;
+ desc.payload = payload;
+
+ ret = ipa_send_cmd(&desc);
+
+ ipahal_payload_free(payload);
+
+ return ret;
+}
+
+/**
+ * ipa_init_flt4() - Initialize IPA filtering block for IPv4.
+ *
+ * Return: 0 if successful, or a negative error code
+ */
+static int ipa_init_flt4(struct ipa_dma_mem *mem)
+{
+ struct ipa_desc desc = { };
+ u32 nhash_offset;
+ u32 hash_offset;
+ void *payload;
+ int ret;
+
+ hash_offset = ipa_ctx->smem_offset + IPA_MEM_V4_FLT_HASH_OFST;
+ nhash_offset = ipa_ctx->smem_offset + IPA_MEM_V4_FLT_NHASH_OFST;
+ payload = ipahal_ip_v4_filter_init_pyld(mem, hash_offset,
+ nhash_offset);
+ if (!payload)
+ return -ENOMEM;
+
+ desc.type = IPA_IMM_CMD_DESC;
+ desc.len_opcode = IPA_IMM_CMD_IP_V4_FILTER_INIT;
+ desc.payload = payload;
+
+ ret = ipa_send_cmd(&desc);
+
+ ipahal_payload_free(payload);
+
+ return ret;
+}
+
+/**
+ * ipa_init_flt6() - Initialize IPA filtering block for IPv6.
+ *
+ * Return: 0 if successful, or a negative error code
+ */
+static int ipa_init_flt6(struct ipa_dma_mem *mem)
+{
+ struct ipa_desc desc = { };
+ u32 nhash_offset;
+ u32 hash_offset;
+ void *payload;
+ int ret;
+
+ hash_offset = ipa_ctx->smem_offset + IPA_MEM_V6_FLT_HASH_OFST;
+ nhash_offset = ipa_ctx->smem_offset + IPA_MEM_V6_FLT_NHASH_OFST;
+ payload = ipahal_ip_v6_filter_init_pyld(mem, hash_offset,
+ nhash_offset);
+ if (!payload)
+ return -ENOMEM;
+
+ desc.type = IPA_IMM_CMD_DESC;
+ desc.len_opcode = IPA_IMM_CMD_IP_V6_FILTER_INIT;
+ desc.payload = payload;
+
+ ret = ipa_send_cmd(&desc);
+
+ ipahal_payload_free(payload);
+
+ return ret;
+}
+
+static void ipa_setup_flt_hash_tuple(void)
+{
+ u32 ep_mask = ipa_ctx->filter_bitmap;
+
+ while (ep_mask) {
+ u32 i = __ffs(ep_mask);
+
+ ep_mask ^= BIT(i);
+ if (!ipa_is_modem_ep(i))
+ ipa_set_flt_tuple_mask(i);
+ }
+}
+
+static void ipa_setup_rt_hash_tuple(void)
+{
+ u32 route_mask;
+ u32 modem_mask;
+
+ BUILD_BUG_ON(!IPA_MEM_MODEM_RT_COUNT);
+ BUILD_BUG_ON(IPA_MEM_RT_COUNT < IPA_MEM_MODEM_RT_COUNT);
+
+ /* Compute a mask representing non-modem route table entries */
+ route_mask = GENMASK(IPA_MEM_RT_COUNT - 1, 0);
+ modem_mask = GENMASK(IPA_MEM_MODEM_RT_INDEX_MAX,
+ IPA_MEM_MODEM_RT_INDEX_MIN);
+ route_mask &= ~modem_mask;
+
+ while (route_mask) {
+ u32 i = __ffs(route_mask);
+
+ route_mask ^= BIT(i);
+ ipa_set_rt_tuple_mask(i);
+ }
+}
+
+static int ipa_ep_apps_lan_cons_setup(void)
+{
+ enum ipa_client_type client;
+ u32 rx_buffer_size;
+ u32 channel_count;
+ u32 aggr_count;
+ u32 aggr_bytes;
+ u32 aggr_size;
+ u32 ep_id;
+ int ret;
+
+ client = IPA_CLIENT_APPS_LAN_CONS;
+ channel_count = IPA_APPS_LAN_CONS_RING_COUNT;
+ aggr_count = IPA_GENERIC_AGGR_PKT_LIMIT;
+ aggr_bytes = IPA_GENERIC_AGGR_BYTE_LIMIT;
+
+ if (aggr_bytes > ipa_reg_aggr_max_byte_limit())
+ return -EINVAL;
+
+ if (aggr_count > ipa_reg_aggr_max_packet_limit())
+ return -EINVAL;
+
+ if (ipa_ctx->lan_cons_ep_id != IPA_EP_ID_BAD)
+ return -EBUSY;
+
+ /* Compute the buffer size required to handle the requested
+ * aggregation byte limit. The aggr_byte_limit value is
+ * expressed as a number of KB, but we derive that value
+ * after computing the buffer size to use (in bytes). The
+ * buffer must be sufficient to hold one IPA_MTU-sized
+ * packet *after* the limit is reached.
+ *
+ * (Note that the rx_buffer_size value reflects only the
+ * space for data, not any standard metadata or headers.)
+ */
+ rx_buffer_size = ipa_aggr_byte_limit_buf_size(aggr_bytes);
+
+ /* Account for the extra IPA_MTU past the limit in the
+ * buffer, and convert the result to the KB units the
+ * aggr_byte_limit uses.
+ */
+ aggr_size = (rx_buffer_size - IPA_MTU) / SZ_1K;
+
+ ret = ipa_ep_alloc(client);
+ if (ret < 0)
+ return ret;
+ ep_id = ret;
+
+ ipa_endp_init_hdr_cons(ep_id, IPA_LAN_RX_HEADER_LENGTH, 0, 0);
+ ipa_endp_init_hdr_ext_cons(ep_id, ilog2(sizeof(u32)), false);
+ ipa_endp_init_aggr_cons(ep_id, aggr_size, aggr_count, false);
+ ipa_endp_init_cfg_cons(ep_id, IPA_CS_OFFLOAD_DL);
+ ipa_endp_init_hdr_metadata_mask_cons(ep_id, 0x0);
+ ipa_endp_status_cons(ep_id, true);
+
+ ret = ipa_ep_setup(ep_id, channel_count, 1, rx_buffer_size,
+ ipa_lan_rx_cb, NULL);
+ if (ret)
+ ipa_ep_free(ep_id);
+ else
+ ipa_ctx->lan_cons_ep_id = ep_id;
+
+ return ret;
+}
+
+static int ipa_ep_apps_setup(void)
+{
+ struct ipa_dma_mem mem; /* Empty table */
+ int ret;
+
+ /* CMD OUT (AP->IPA) */
+ ret = ipa_ep_apps_cmd_prod_setup();
+ if (ret < 0)
+ return ret;
+
+ ipa_init_sram();
+ ipa_init_hdr();
+
+ ret = ipahal_rt_generate_empty_img(IPA_MEM_RT_COUNT, &mem);
+ ipa_assert(!ret);
+ ipa_init_rt4(&mem);
+ ipa_init_rt6(&mem);
+ ipahal_free_empty_img(&mem);
+
+ ret = ipahal_flt_generate_empty_img(ipa_ctx->filter_bitmap, &mem);
+ ipa_assert(!ret);
+ ipa_init_flt4(&mem);
+ ipa_init_flt6(&mem);
+ ipahal_free_empty_img(&mem);
+
+ ipa_setup_flt_hash_tuple();
+ ipa_setup_rt_hash_tuple();
+
+ /* LAN IN (IPA->AP)
+ *
+ * Even without supporting LAN traffic, we use the LAN consumer
+ * endpoint for receiving some information from the IPA. If we issue
+ * a tagged command, we arrange to be notified of its completion
+ * through this endpoint. In addition, we arrange for this endpoint
+ * to be used as the IPA's default route; the IPA will notify the AP
+ * of exceptions (unroutable packets, but other events as well)
+ * through this endpoint.
+ */
+ ret = ipa_ep_apps_lan_cons_setup();
+ if (ret < 0)
+ goto fail_flt_hash_tuple;
+
+ ipa_cfg_default_route(IPA_CLIENT_APPS_LAN_CONS);
+
+ return 0;
+
+fail_flt_hash_tuple:
+ ipa_ep_teardown(ipa_ctx->cmd_prod_ep_id);
+ ipa_ctx->cmd_prod_ep_id = IPA_EP_ID_BAD;
+
+ return ret;
+}
+
+static int ipa_clock_init(struct device *dev)
+{
+ struct clk *clk;
+ int ret;
+
+ clk = clk_get(dev, "core");
+ if (IS_ERR(clk))
+ return PTR_ERR(clk);
+
+ ret = clk_set_rate(clk, IPA_CORE_CLOCK_RATE);
+ if (ret) {
+ clk_put(clk);
+ return ret;
+ }
+
+ ipa_ctx->core_clock = clk;
+
+ return 0;
+}
+
+static void ipa_clock_exit(void)
+{
+ clk_put(ipa_ctx->core_clock);
+ ipa_ctx->core_clock = NULL;
+}
+
+/**
+ * ipa_enable_clks() - Turn on IPA clocks
+ */
+static void ipa_enable_clks(void)
+{
+ if (WARN_ON(ipa_interconnect_enable()))
+ return;
+
+ if (WARN_ON(clk_prepare_enable(ipa_ctx->core_clock)))
+ ipa_interconnect_disable();
+}
+
+/**
+ * ipa_disable_clks() - Turn off IPA clocks
+ */
+static void ipa_disable_clks(void)
+{
+ clk_disable_unprepare(ipa_ctx->core_clock);
+ WARN_ON(ipa_interconnect_disable());
+}
+
+/* Add an IPA client under protection of the mutex. This is called
+ * for the first client, but a race could mean another caller gets
+ * the first reference. When the first reference is taken, IPA
+ * clocks are enabled endpoints are resumed. A positive reference count
+ * means the endpoints are active; this doesn't set the first reference
+ * until after this is complete (and the mutex, not the atomic
+ * count, is what protects this).
+ */
+static void ipa_client_add_first(void)
+{
+ mutex_lock(&ipa_ctx->active_clients_mutex);
+
+ /* A reference might have been added while awaiting the mutex. */
+ if (!atomic_inc_not_zero(&ipa_ctx->active_clients_count)) {
+ ipa_enable_clks();
+ ipa_ep_resume_all();
+ atomic_inc(&ipa_ctx->active_clients_count);
+ } else {
+ ipa_assert(atomic_read(&ipa_ctx->active_clients_count) > 1);
+ }
+
+ mutex_unlock(&ipa_ctx->active_clients_mutex);
+}
+
+/* Attempt to add an IPA client reference, but only if this does not
+ * represent the initiaal reference. Returns true if the reference
+ * was taken, false otherwise.
+ */
+static bool ipa_client_add_not_first(void)
+{
+ return !!atomic_inc_not_zero(&ipa_ctx->active_clients_count);
+}
+
+/* Add an IPA client, but only if the reference count is already
+ * non-zero. (This is used to avoid blocking.) Returns true if the
+ * additional reference was added successfully, or false otherwise.
+ */
+bool ipa_client_add_additional(void)
+{
+ return ipa_client_add_not_first();
+}
+
+/* Add an IPA client. If this is not the first client, the
+ * reference count is updated and return is immediate. Otherwise
+ * ipa_client_add_first() will safely add the first client, enabling
+ * clocks and setting up (resuming) endpoints before returning.
+ */
+void ipa_client_add(void)
+{
+ /* There's nothing more to do if this isn't the first reference */
+ if (!ipa_client_add_not_first())
+ ipa_client_add_first();
+}
+
+/* Remove an IPA client under protection of the mutex. This is
+ * called for the last remaining client, but a race could mean
+ * another caller gets an additional reference before the mutex
+ * is acquired. When the final reference is dropped, endpoints are
+ * suspended and IPA clocks disabled.
+ */
+static void ipa_client_remove_final(void)
+{
+ mutex_lock(&ipa_ctx->active_clients_mutex);
+
+ /* A reference might have been removed while awaiting the mutex. */
+ if (!atomic_dec_return(&ipa_ctx->active_clients_count)) {
+ ipa_ep_suspend_all();
+ ipa_disable_clks();
+ }
+
+ mutex_unlock(&ipa_ctx->active_clients_mutex);
+}
+
+/* Decrement the active clients reference count, and if the result
+ * is 0, suspend the endpoints and disable clocks.
+ *
+ * This function runs in work queue context, scheduled to run whenever
+ * the last reference would be dropped in ipa_client_remove().
+ */
+static void ipa_client_remove_deferred(struct work_struct *work)
+{
+ ipa_client_remove_final();
+}
+
+/* Attempt to remove a client reference, but only if this is not the
+ * only reference remaining. Returns true if the reference was
+ * removed, or false if doing so would produce a zero reference
+ * count.
+ */
+static bool ipa_client_remove_not_final(void)
+{
+ return !!atomic_add_unless(&ipa_ctx->active_clients_count, -1, 1);
+}
+
+/* Attempt to remove an IPA client reference. If this represents
+ * the last reference arrange for ipa_client_remove_final() to be
+ * called in workqueue context, dropping the last reference under
+ * protection of the mutex.
+ */
+void ipa_client_remove(void)
+{
+ if (!ipa_client_remove_not_final())
+ queue_work(ipa_ctx->power_mgmt_wq, &ipa_client_remove_work);
+}
+
+/** ipa_inc_acquire_wakelock() - Increase active clients counter, and
+ * acquire wakelock if necessary
+ */
+void ipa_inc_acquire_wakelock(void)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&ipa_ctx->wakeup_lock, flags);
+
+ ipa_ctx->wakeup_count++;
+ if (ipa_ctx->wakeup_count == 1)
+ __pm_stay_awake(&ipa_ctx->wakeup);
+
+ spin_unlock_irqrestore(&ipa_ctx->wakeup_lock, flags);
+}
+
+/** ipa_dec_release_wakelock() - Decrease active clients counter
+ *
+ * In case if the ref count is 0, release the wakelock.
+ */
+void ipa_dec_release_wakelock(void)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&ipa_ctx->wakeup_lock, flags);
+
+ ipa_ctx->wakeup_count--;
+ if (ipa_ctx->wakeup_count == 0)
+ __pm_relax(&ipa_ctx->wakeup);
+
+ spin_unlock_irqrestore(&ipa_ctx->wakeup_lock, flags);
+}
+
+/** ipa_suspend_handler() - Handle the suspend interrupt
+ * @interrupt: Interrupt type
+ * @endpoints: Interrupt specific information data
+ */
+static void ipa_suspend_handler(enum ipa_irq_type interrupt, u32 interrupt_data)
+{
+ u32 endpoints = interrupt_data;
+
+ while (endpoints) {
+ enum ipa_client_type client;
+ u32 i = __ffs(endpoints);
+
+ endpoints ^= BIT(i);
+
+ if (!ipa_ctx->ep[i].allocated)
+ continue;
+
+ client = ipa_ctx->ep[i].client;
+ if (!ipa_ap_consumer(client))
+ continue;
+
+ /* endpoint will be unsuspended by enabling IPA clocks */
+ mutex_lock(&ipa_ctx->transport_pm.transport_pm_mutex);
+ if (!atomic_read(&ipa_ctx->transport_pm.dec_clients)) {
+ ipa_client_add();
+
+ atomic_set(&ipa_ctx->transport_pm.dec_clients, 1);
+ }
+ mutex_unlock(&ipa_ctx->transport_pm.transport_pm_mutex);
+ }
+}
+
+/**
+ * ipa_init_interrupts() - Initialize IPA interrupts
+ */
+static int ipa_init_interrupts(void)
+{
+ int ret;
+
+ ret = ipa_interrupts_init();
+ if (!ret)
+ return ret;
+
+ ipa_add_interrupt_handler(IPA_TX_SUSPEND_IRQ, ipa_suspend_handler);
+
+ return 0;
+}
+
+static void ipa_freeze_clock_vote_and_notify_modem(void)
+{
+ u32 value;
+ u32 mask;
+
+ if (ipa_ctx->smp2p_info.res_sent)
+ return;
+
+ if (!ipa_ctx->smp2p_info.enabled_state) {
+ ipa_err("smp2p out gpio not assigned\n");
+ return;
+ }
+
+ ipa_ctx->smp2p_info.ipa_clk_on = ipa_client_add_additional();
+
+ /* Signal whether the clock is enabled */
+ mask = BIT(ipa_ctx->smp2p_info.enabled_bit);
+ value = ipa_ctx->smp2p_info.ipa_clk_on ? mask : 0;
+ qcom_smem_state_update_bits(ipa_ctx->smp2p_info.enabled_state, mask,
+ value);
+
+ /* Now indicate that the enabled flag is valid */
+ mask = BIT(ipa_ctx->smp2p_info.valid_bit);
+ value = mask;
+ qcom_smem_state_update_bits(ipa_ctx->smp2p_info.valid_state, mask,
+ value);
+
+ ipa_ctx->smp2p_info.res_sent = true;
+}
+
+void ipa_reset_freeze_vote(void)
+{
+ u32 mask;
+
+ if (!ipa_ctx->smp2p_info.res_sent)
+ return;
+
+ if (ipa_ctx->smp2p_info.ipa_clk_on)
+ ipa_client_remove();
+
+ /* Reset the clock enabled valid flag */
+ mask = BIT(ipa_ctx->smp2p_info.valid_bit);
+ qcom_smem_state_update_bits(ipa_ctx->smp2p_info.valid_state, mask, 0);
+
+ /* Mark the clock disabled for good measure... */
+ mask = BIT(ipa_ctx->smp2p_info.enabled_bit);
+ qcom_smem_state_update_bits(ipa_ctx->smp2p_info.enabled_state, mask, 0);
+
+ ipa_ctx->smp2p_info.res_sent = false;
+ ipa_ctx->smp2p_info.ipa_clk_on = false;
+}
+
+static int
+ipa_panic_notifier(struct notifier_block *this, unsigned long event, void *ptr)
+{
+ ipa_freeze_clock_vote_and_notify_modem();
+ ipa_uc_panic_notifier();
+
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block ipa_panic_blk = {
+ .notifier_call = ipa_panic_notifier,
+ /* IPA panic handler needs to run before modem shuts down */
+ .priority = INT_MAX,
+};
+
+static void ipa_register_panic_hdlr(void)
+{
+ atomic_notifier_chain_register(&panic_notifier_list, &ipa_panic_blk);
+}
+
+/* Remoteproc callbacks for SSR events: prepare, start, stop, unprepare */
+int ipa_ssr_prepare(struct rproc_subdev *subdev)
+{
+ printk("======== SSR prepare received ========\n");
+ return 0;
+}
+EXPORT_SYMBOL_GPL(ipa_ssr_prepare);
+
+int ipa_ssr_start(struct rproc_subdev *subdev)
+{
+ printk("======== SSR start received ========\n");
+ return 0;
+}
+EXPORT_SYMBOL_GPL(ipa_ssr_start);
+
+void ipa_ssr_stop(struct rproc_subdev *subdev, bool crashed)
+{
+ printk("======== SSR stop received ========\n");
+}
+EXPORT_SYMBOL_GPL(ipa_ssr_stop);
+
+void ipa_ssr_unprepare(struct rproc_subdev *subdev)
+{
+ printk("======== SSR unprepare received ========\n");
+}
+EXPORT_SYMBOL_GPL(ipa_ssr_unprepare);
+
+/**
+ * ipa_post_init() - Initialize the IPA Driver (Part II).
+ *
+ * Perform initialization that requires interaction with IPA hardware.
+ */
+static void ipa_post_init(void)
+{
+ int ret;
+
+ ipa_debug("ipa_post_init() started\n");
+
+ ret = gsi_device_init(ipa_ctx->gsi);
+ if (ret) {
+ ipa_err(":gsi register error - %d\n", ret);
+ return;
+ }
+
+ /* setup the AP-IPA endpoints */
+ if (ipa_ep_apps_setup()) {
+ ipa_err(":failed to setup IPA-Apps endpoints\n");
+ gsi_device_exit(ipa_ctx->gsi);
+
+ return;
+ }
+
+ ipa_ctx->uc_ctx = ipa_uc_init(ipa_ctx->ipa_phys);
+ if (!ipa_ctx->uc_ctx)
+ ipa_err("microcontroller init failed\n");
+
+ ipa_register_panic_hdlr();
+
+ ipa_ctx->modem_clk_vote_valid = true;
+
+ if (ipa_wwan_init())
+ ipa_err("WWAN init failed (ignoring)\n");
+
+ dev_info(ipa_ctx->dev, "IPA driver initialization was successful.\n");
+}
+
+/** ipa_pre_init() - Initialize the IPA Driver.
+ *
+ * Perform initialization which doesn't require access to IPA hardware.
+ */
+static int ipa_pre_init(void)
+{
+ int ret = 0;
+
+ /* enable IPA clocks explicitly to allow the initialization */
+ ipa_enable_clks();
+
+ ipa_init_hw();
+
+ ipa_ctx->ep_count = ipa_get_ep_count();
+ ipa_debug("ep_count %u\n", ipa_get_ep_count());
+ ipa_assert(ipa_ctx->ep_count <= IPA_EP_COUNT_MAX);
+
+ ipa_sram_settings_read();
+ if (ipa_ctx->smem_size < IPA_MEM_END_OFST) {
+ ipa_err("insufficient memory: %hu bytes available, need %u\n",
+ ipa_ctx->smem_size, IPA_MEM_END_OFST);
+ ret = -ENOMEM;
+ goto err_disable_clks;
+ }
+
+ mutex_init(&ipa_ctx->active_clients_mutex);
+ atomic_set(&ipa_ctx->active_clients_count, 1);
+
+ /* Create workqueues for power management */
+ ipa_ctx->power_mgmt_wq =
+ create_singlethread_workqueue("ipa_power_mgmt");
+ if (!ipa_ctx->power_mgmt_wq) {
+ ipa_err("failed to create power mgmt wq\n");
+ ret = -ENOMEM;
+ goto err_disable_clks;
+ }
+
+ mutex_init(&ipa_ctx->transport_pm.transport_pm_mutex);
+
+ /* init the lookaside cache */
+
+ ipa_ctx->dp = ipa_dp_init();
+ if (!ipa_ctx->dp)
+ goto err_destroy_pm_wq;
+
+ /* allocate memory for DMA_TASK workaround */
+ ret = ipa_gsi_dma_task_alloc();
+ if (ret)
+ goto err_dp_exit;
+
+ /* Create a wakeup source. */
+ wakeup_source_init(&ipa_ctx->wakeup, "IPA_WS");
+ spin_lock_init(&ipa_ctx->wakeup_lock);
+
+ /* Note enabling dynamic clock division must not be
+ * attempted for IPA hardware versions prior to 3.5.
+ */
+ ipa_enable_dcd();
+
+ /* Assign resource limitation to each group */
+ ipa_set_resource_groups_min_max_limits();
+
+ ret = ipa_init_interrupts();
+ if (!ret)
+ return 0; /* Success! */
+
+ ipa_err("ipa initialization of interrupts failed\n");
+err_dp_exit:
+ ipa_dp_exit(ipa_ctx->dp);
+ ipa_ctx->dp = NULL;
+err_destroy_pm_wq:
+ destroy_workqueue(ipa_ctx->power_mgmt_wq);
+err_disable_clks:
+ ipa_disable_clks();
+
+ return ret;
+}
+
+static int ipa_firmware_load(struct device *dev)
+{
+ const struct firmware *fw;
+ struct device_node *node;
+ struct resource res;
+ phys_addr_t phys;
+ ssize_t size;
+ void *virt;
+ int ret;
+
+ ret = request_firmware(&fw, IPA_FWS_PATH, dev);
+ if (ret)
+ return ret;
+
+ node = of_parse_phandle(dev->of_node, "memory-region", 0);
+ if (!node) {
+ dev_err(dev, "memory-region not specified\n");
+ ret = -EINVAL;
+ goto out_release_firmware;
+ }
+
+ ret = of_address_to_resource(node, 0, &res);
+ if (ret)
+ goto out_release_firmware;
+
+ phys = res.start,
+ size = (size_t)resource_size(&res);
+ virt = memremap(phys, size, MEMREMAP_WC);
+ if (!virt) {
+ ret = -ENOMEM;
+ goto out_release_firmware;
+ }
+
+ ret = qcom_mdt_load(dev, fw, IPA_FWS_PATH, IPA_PAS_ID,
+ virt, phys, size, NULL);
+ if (!ret)
+ ret = qcom_scm_pas_auth_and_reset(IPA_PAS_ID);
+
+ memunmap(virt);
+out_release_firmware:
+ release_firmware(fw);
+
+ return ret;
+}
+
+/* Threaded IRQ handler for modem "ipa-clock-query" SMP2P interrupt */
+static irqreturn_t ipa_smp2p_modem_clk_query_isr(int irq, void *ctxt)
+{
+ ipa_freeze_clock_vote_and_notify_modem();
+
+ return IRQ_HANDLED;
+}
+
+/* Threaded IRQ handler for modem "ipa-post-init" SMP2P interrupt */
+static irqreturn_t ipa_smp2p_modem_post_init_isr(int irq, void *ctxt)
+{
+ ipa_post_init();
+
+ return IRQ_HANDLED;
+}
+
+static int
+ipa_smp2p_irq_init(struct device *dev, const char *name, irq_handler_t handler)
+{
+ struct device_node *node = dev->of_node;
+ unsigned int irq;
+ int ret;
+
+ ret = of_irq_get_byname(node, name);
+ if (ret < 0)
+ return ret;
+ if (!ret)
+ return -EINVAL; /* IRQ mapping failure */
+ irq = ret;
+
+ ret = devm_request_threaded_irq(dev, irq, NULL, handler, 0, name, dev);
+ if (ret)
+ return ret;
+
+ return irq;
+}
+
+static void
+ipa_smp2p_irq_exit(struct device *dev, unsigned int irq)
+{
+ devm_free_irq(dev, irq, dev);
+}
+
+static int ipa_smp2p_init(struct device *dev, bool modem_init)
+{
+ struct qcom_smem_state *enabled_state;
+ struct qcom_smem_state *valid_state;
+ struct device_node *node;
+ unsigned int enabled_bit;
+ unsigned int valid_bit;
+ unsigned int clock_irq;
+ int ret;
+
+ node = dev->of_node;
+
+ valid_state = qcom_smem_state_get(dev, "ipa-clock-enabled-valid",
+ &valid_bit);
+ if (IS_ERR(valid_state))
+ return PTR_ERR(valid_state);
+
+ enabled_state = qcom_smem_state_get(dev, "ipa-clock-enabled",
+ &enabled_bit);
+ if (IS_ERR(enabled_state)) {
+ ret = PTR_ERR(enabled_state);
+ ipa_err("error %d getting ipa-clock-enabled state\n", ret);
+
+ return ret;
+ }
+
+ ret = ipa_smp2p_irq_init(dev, "ipa-clock-query",
+ ipa_smp2p_modem_clk_query_isr);
+ if (ret < 0)
+ return ret;
+ clock_irq = ret;
+
+ if (modem_init) {
+ /* Result will be non-zero (negative for error) */
+ ret = ipa_smp2p_irq_init(dev, "ipa-post-init",
+ ipa_smp2p_modem_post_init_isr);
+ if (ret < 0) {
+ ipa_smp2p_irq_exit(dev, clock_irq);
+
+ return ret;
+ }
+ }
+
+ /* Success. Record our smp2p information */
+ ipa_ctx->smp2p_info.valid_state = valid_state;
+ ipa_ctx->smp2p_info.valid_bit = valid_bit;
+ ipa_ctx->smp2p_info.enabled_state = enabled_state;
+ ipa_ctx->smp2p_info.enabled_bit = enabled_bit;
+ ipa_ctx->smp2p_info.clock_query_irq = clock_irq;
+ ipa_ctx->smp2p_info.post_init_irq = modem_init ? ret : 0;
+
+ return 0;
+}
+
+static void ipa_smp2p_exit(struct device *dev)
+{
+ if (ipa_ctx->smp2p_info.post_init_irq)
+ ipa_smp2p_irq_exit(dev, ipa_ctx->smp2p_info.post_init_irq);
+ ipa_smp2p_irq_exit(dev, ipa_ctx->smp2p_info.clock_query_irq);
+
+ memset(&ipa_ctx->smp2p_info, 0, sizeof(ipa_ctx->smp2p_info));
+}
+
+static const struct ipa_match_data tz_init = {
+ .init_type = ipa_tz_init,
+};
+
+static const struct ipa_match_data modem_init = {
+ .init_type = ipa_modem_init,
+};
+
+static const struct of_device_id ipa_plat_drv_match[] = {
+ {
+ .compatible = "qcom,ipa-sdm845-tz_init",
+ .data = &tz_init,
+ },
+ {
+ .compatible = "qcom,ipa-sdm845-modem_init",
+ .data = &modem_init,
+ },
+ {}
+};
+
+static int ipa_plat_drv_probe(struct platform_device *pdev)
+{
+ const struct ipa_match_data *match_data;
+ struct resource *res;
+ struct device *dev;
+ bool modem_init;
+ int ret;
+
+ /* We assume we're working on 64-bit hardware */
+ BUILD_BUG_ON(!IS_ENABLED(CONFIG_64BIT));
+
+ dev = &pdev->dev;
+
+ match_data = of_device_get_match_data(dev);
+ modem_init = match_data->init_type == ipa_modem_init;
+
+ /* If we need Trust Zone, make sure it's ready */
+ if (!modem_init)
+ if (!qcom_scm_is_available())
+ return -EPROBE_DEFER;
+
+ /* Initialize the smp2p driver early. It might not be ready
+ * when we're probed, so it might return -EPROBE_DEFER.
+ */
+ ret = ipa_smp2p_init(dev, modem_init);
+ if (ret)
+ return ret;
+
+ /* Initialize the interconnect driver early too. It might
+ * also return -EPROBE_DEFER.
+ */
+ ret = ipa_interconnect_init(dev);
+ if (ret)
+ goto out_smp2p_exit;
+
+ ret = ipa_clock_init(dev);
+ if (ret)
+ goto err_interconnect_exit;
+
+ ipa_ctx->dev = dev; /* Set early for ipa_err()/ipa_debug() */
+
+ /* Compute a bitmask representing which endpoints support filtering */
+ ipa_ctx->filter_bitmap = ipa_filter_bitmap_init();
+ ipa_debug("filter_bitmap 0x%08x\n", ipa_ctx->filter_bitmap);
+ if (!ipa_ctx->filter_bitmap)
+ goto err_clock_exit;
+
+ ret = platform_get_irq_byname(pdev, "ipa");
+ if (ret < 0)
+ goto err_clear_filter_bitmap;
+ ipa_ctx->ipa_irq = ret;
+
+ /* Get IPA memory range */
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "ipa");
+ if (!res) {
+ ret = -ENODEV;
+ goto err_clear_ipa_irq;
+ }
+
+ /* Setup IPA register access */
+ ret = ipa_reg_init(res->start, (size_t)resource_size(res));
+ if (ret)
+ goto err_clear_ipa_irq;
+ ipa_ctx->ipa_phys = res->start;
+
+ ipa_ctx->gsi = gsi_init(pdev);
+ if (IS_ERR(ipa_ctx->gsi)) {
+ ret = PTR_ERR(ipa_ctx->gsi);
+ goto err_clear_gsi;
+ }
+
+ ret = ipa_dma_init(dev, IPA_HW_TBL_SYSADDR_ALIGN);
+ if (ret)
+ goto err_clear_gsi;
+
+ ret = ipahal_init();
+ if (ret)
+ goto err_dma_exit;
+
+ ipa_ctx->cmd_prod_ep_id = IPA_EP_ID_BAD;
+ ipa_ctx->lan_cons_ep_id = IPA_EP_ID_BAD;
+
+ /* Proceed to real initialization */
+ ret = ipa_pre_init();
+ if (ret)
+ goto err_clear_dev;
+
+ /* If the modem is not verifying and loading firmware we need to
+ * get it loaded ourselves. Only then can we proceed with the
+ * second stage of IPA initialization. If the modem is doing it,
+ * it will send an SMP2P interrupt to signal this has been done,
+ * and that will trigger the "post init".
+ */
+ if (!modem_init) {
+ ret = ipa_firmware_load(dev);
+ if (ret)
+ goto err_clear_dev;
+
+ /* Now we can proceed to stage two initialization */
+ ipa_post_init();
+ }
+
+ return 0; /* Success */
+
+err_clear_dev:
+ ipa_ctx->lan_cons_ep_id = 0;
+ ipa_ctx->cmd_prod_ep_id = 0;
+ ipahal_exit();
+err_dma_exit:
+ ipa_dma_exit();
+err_clear_gsi:
+ ipa_ctx->gsi = NULL;
+ ipa_ctx->ipa_phys = 0;
+ ipa_reg_exit();
+err_clear_ipa_irq:
+ ipa_ctx->ipa_irq = 0;
+err_clear_filter_bitmap:
+ ipa_ctx->filter_bitmap = 0;
+err_interconnect_exit:
+ ipa_interconnect_exit();
+err_clock_exit:
+ ipa_clock_exit();
+ ipa_ctx->dev = NULL;
+out_smp2p_exit:
+ ipa_smp2p_exit(dev);
+
+ return ret;
+}
+
+static int ipa_plat_drv_remove(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+
+ ipa_ctx->dev = NULL;
+ ipahal_exit();
+ ipa_dma_exit();
+ ipa_ctx->gsi = NULL; /* XXX ipa_gsi_exit() */
+ ipa_reg_exit();
+
+ ipa_ctx->ipa_phys = 0;
+
+ if (ipa_ctx->lan_cons_ep_id != IPA_EP_ID_BAD) {
+ ipa_ep_free(ipa_ctx->lan_cons_ep_id);
+ ipa_ctx->lan_cons_ep_id = IPA_EP_ID_BAD;
+ }
+ if (ipa_ctx->cmd_prod_ep_id != IPA_EP_ID_BAD) {
+ ipa_ep_free(ipa_ctx->cmd_prod_ep_id);
+ ipa_ctx->cmd_prod_ep_id = IPA_EP_ID_BAD;
+ }
+ ipa_ctx->ipa_irq = 0; /* XXX Need to de-initialize? */
+ ipa_ctx->filter_bitmap = 0;
+ ipa_interconnect_exit();
+ ipa_smp2p_exit(dev);
+
+ return 0;
+}
+
+/**
+ * ipa_ap_suspend() - suspend callback for runtime_pm
+ * @dev: IPA device structure
+ *
+ * This callback will be invoked by the runtime_pm framework when an AP suspend
+ * operation is invoked, usually by pressing a suspend button.
+ *
+ * Return: 0 if successful, -EAGAIN if IPA is in use
+ */
+int ipa_ap_suspend(struct device *dev)
+{
+ u32 i;
+
+ /* In case there is a tx/rx handler in polling mode fail to suspend */
+ for (i = 0; i < ipa_ctx->ep_count; i++) {
+ if (ipa_ctx->ep[i].sys && ipa_ep_polling(&ipa_ctx->ep[i])) {
+ ipa_err("EP %d is in polling state, do not suspend\n",
+ i);
+ return -EAGAIN;
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * ipa_ap_resume() - resume callback for runtime_pm
+ * @dev: IPA device structure
+ *
+ * This callback will be invoked by the runtime_pm framework when an AP resume
+ * operation is invoked.
+ *
+ * Return: Zero
+ */
+int ipa_ap_resume(struct device *dev)
+{
+ return 0;
+}
+
+static const struct dev_pm_ops ipa_pm_ops = {
+ .suspend_noirq = ipa_ap_suspend,
+ .resume_noirq = ipa_ap_resume,
+};
+
+static struct platform_driver ipa_plat_drv = {
+ .probe = ipa_plat_drv_probe,
+ .remove = ipa_plat_drv_remove,
+ .driver = {
+ .name = "ipa",
+ .owner = THIS_MODULE,
+ .pm = &ipa_pm_ops,
+ .of_match_table = ipa_plat_drv_match,
+ },
+};
+
+builtin_platform_driver(ipa_plat_drv);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("IPA HW device driver");
--
2.17.1
^ permalink raw reply related
* [RFC PATCH 08/12] soc: qcom: ipa: utility functions
From: Alex Elder @ 2018-11-07 0:32 UTC (permalink / raw)
To: davem, arnd, bjorn.andersson, ilias.apalodimas
Cc: netdev, devicetree, linux-arm-msm, linux-soc, linux-arm-kernel,
linux-kernel, syadagir, mjavid, robh+dt, mark.rutland
In-Reply-To: <20181107003250.5832-1-elder@linaro.org>
This patch contains "ipa_utils.c", which provides some miscellaneous
supporting code. Included are:
- Endpoint configuration. The IPA hardware has a fixed number of
endpoints (pipes) available. In some cases the AP and modem
must agree on aspects of these. A key example is that they need
to agree what particular endpoints are used for (the modem can't
send messages to the AP command endpoint, for example). There
is a static array that defines these parameters, and this is
found in "ipa_utils.c".
- Resource group configuration. The IPA has a number of internal
resources it uses for its operation. These are configurable,
and it is up to the AP to set these configuration values at
initialization time. There are some static arrays that define
these configuration values. Functions are also defined here
to send those values to hardware.
- Shared memory. The IPA uses a region of shared memory to hold
various data structures. A function ipa_sram_settings_read()
fetches the location and size of this shared memory. (The
individual regions are currently initialized in "ipa_main.c".)
- Endpoint configuration. Each endpoint (or channel) has a number
of configurable properties. Functions found in this file are
used to configure these properties.
- Interconnect handling. The IPA driver depends on the
interconnect framework to request that buses it uses be enabled
when needed.
This is not a complete list, but it covers much of the functionality
found in "ipa_utils.c".
Signed-off-by: Alex Elder <elder@linaro.org>
---
drivers/net/ipa/ipa_utils.c | 1035 +++++++++++++++++++++++++++++++++++
1 file changed, 1035 insertions(+)
create mode 100644 drivers/net/ipa/ipa_utils.c
diff --git a/drivers/net/ipa/ipa_utils.c b/drivers/net/ipa/ipa_utils.c
new file mode 100644
index 000000000000..085b0218779b
--- /dev/null
+++ b/drivers/net/ipa/ipa_utils.c
@@ -0,0 +1,1035 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/interconnect.h>
+#include <linux/module.h>
+#include <linux/delay.h>
+
+#include "ipa_dma.h"
+#include "ipa_i.h"
+#include "ipahal.h"
+
+/* Interconnect path bandwidths (each times 1000 bytes per second) */
+#define IPA_MEMORY_AVG 80000
+#define IPA_MEMORY_PEAK 600000
+
+#define IPA_IMEM_AVG 80000
+#define IPA_IMEM_PEAK 350000
+
+#define IPA_CONFIG_AVG 40000
+#define IPA_CONFIG_PEAK 40000
+
+#define IPA_BCR_REG_VAL 0x0000003b
+
+#define IPA_GSI_DMA_TASK_TIMEOUT 15 /* milliseconds */
+
+#define IPA_GSI_CHANNEL_STOP_SLEEP_MIN 1000 /* microseconds */
+#define IPA_GSI_CHANNEL_STOP_SLEEP_MAX 2000 /* microseconds */
+
+#define QMB_MASTER_SELECT_DDR 0
+
+enum ipa_rsrc_group {
+ IPA_RSRC_GROUP_LWA_DL, /* currently not used */
+ IPA_RSRC_GROUP_UL_DL,
+ IPA_RSRC_GROUP_MAX,
+};
+
+enum ipa_rsrc_grp_type_src {
+ IPA_RSRC_GRP_TYPE_SRC_PKT_CONTEXTS,
+ IPA_RSRC_GRP_TYPE_SRS_DESCRIPTOR_LISTS,
+ IPA_RSRC_GRP_TYPE_SRC_DESCRIPTOR_BUFF,
+ IPA_RSRC_GRP_TYPE_SRC_HPS_DMARS,
+ IPA_RSRC_GRP_TYPE_SRC_ACK_ENTRIES,
+};
+
+enum ipa_rsrc_grp_type_dst {
+ IPA_RSRC_GRP_TYPE_DST_DATA_SECTORS,
+ IPA_RSRC_GRP_TYPE_DST_DPS_DMARS,
+};
+
+enum ipa_rsrc_grp_type_rx {
+ IPA_RSRC_GRP_TYPE_RX_HPS_CMDQ,
+ IPA_RSRC_GRP_TYPE_RX_MAX
+};
+
+struct rsrc_min_max {
+ u32 min;
+ u32 max;
+};
+
+/* IPA_HW_v3_5_1 */
+static const struct rsrc_min_max ipa_src_rsrc_grp[][IPA_RSRC_GROUP_MAX] = {
+ [IPA_RSRC_GRP_TYPE_SRC_PKT_CONTEXTS] = {
+ [IPA_RSRC_GROUP_LWA_DL] = { .min = 1, .max = 63, },
+ [IPA_RSRC_GROUP_UL_DL] = { .min = 1, .max = 63, },
+ },
+ [IPA_RSRC_GRP_TYPE_SRS_DESCRIPTOR_LISTS] = {
+ [IPA_RSRC_GROUP_LWA_DL] = { .min = 10, .max = 10, },
+ [IPA_RSRC_GROUP_UL_DL] = { .min = 10, .max = 10, },
+ },
+ [IPA_RSRC_GRP_TYPE_SRC_DESCRIPTOR_BUFF] = {
+ [IPA_RSRC_GROUP_LWA_DL] = { .min = 12, .max = 12, },
+ [IPA_RSRC_GROUP_UL_DL] = { .min = 14, .max = 14, },
+ },
+ [IPA_RSRC_GRP_TYPE_SRC_HPS_DMARS] = {
+ [IPA_RSRC_GROUP_LWA_DL] = { .min = 0, .max = 63, },
+ [IPA_RSRC_GROUP_UL_DL] = { .min = 0, .max = 63, },
+ },
+ [IPA_RSRC_GRP_TYPE_SRC_ACK_ENTRIES] = {
+ [IPA_RSRC_GROUP_LWA_DL] = { .min = 14, .max = 14, },
+ [IPA_RSRC_GROUP_UL_DL] = { .min = 20, .max = 20, },
+ },
+};
+
+/* IPA_HW_v3_5_1 */
+static const struct rsrc_min_max ipa_dst_rsrc_grp[][IPA_RSRC_GROUP_MAX] = {
+ [IPA_RSRC_GRP_TYPE_DST_DATA_SECTORS] = {
+ [IPA_RSRC_GROUP_LWA_DL] = { .min = 4, .max = 4, },
+ [IPA_RSRC_GROUP_UL_DL] = { .min = 4, .max = 4, },
+ },
+ [IPA_RSRC_GRP_TYPE_DST_DPS_DMARS] = {
+ [IPA_RSRC_GROUP_LWA_DL] = { .min = 2, .max = 63, },
+ [IPA_RSRC_GROUP_UL_DL] = { .min = 1, .max = 63, },
+ },
+};
+
+/**
+ * struct ipa_gsi_ep_config - GSI endpoint configuration.
+ * @ep_id: IPA endpoint identifier.
+ * @channel_id: GSI channel number used for this endpoint.
+ * @tlv_count: The number of TLV (type-length-value) entries for the channel.
+ * @ee: Execution environment endpoint is associated with.
+ *
+ * Each GSI endpoint has a set of configuration parameters defined within
+ * entries in the ipa_ep_configuration[] array. Its @ep_id field uniquely
+ * defines the endpoint, and @channel_id defines which data channel (ring
+ * buffer) is used for the endpoint.
+ * XXX TLV
+ * XXX ee is never used in the code
+ */
+struct ipa_gsi_ep_config {
+ u32 ep_id;
+ u32 channel_id;
+ u32 tlv_count;
+ u32 ee;
+};
+
+struct ipa_ep_configuration {
+ bool support_flt;
+ enum ipa_seq_type seq_type;
+ struct ipa_gsi_ep_config ipa_gsi_ep_info;
+};
+
+/* IPA_HW_v3_5_1 */
+/* clients not included in the list below are considered as invalid */
+static const struct ipa_ep_configuration ipa_ep_configuration[] = {
+ [IPA_CLIENT_WLAN1_PROD] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 7,
+ .channel_id = 1,
+ .tlv_count = 8,
+ .ee = IPA_EE_UC,
+ },
+ },
+ [IPA_CLIENT_USB_PROD] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 0,
+ .channel_id = 0,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_APPS_LAN_PROD] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_PKT_PROCESS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 8,
+ .channel_id = 7,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_APPS_WAN_PROD] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 2,
+ .channel_id = 3,
+ .tlv_count = 16,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_APPS_CMD_PROD] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_DMA_ONLY,
+ .ipa_gsi_ep_info = {
+ .ep_id = 5,
+ .channel_id = 4,
+ .tlv_count = 20,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_Q6_LAN_PROD] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_PKT_PROCESS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 3,
+ .channel_id = 0,
+ .tlv_count = 16,
+ .ee = IPA_EE_Q6,
+ },
+ },
+ [IPA_CLIENT_Q6_WAN_PROD] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_PKT_PROCESS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 6,
+ .channel_id = 4,
+ .tlv_count = 12,
+ .ee = IPA_EE_Q6,
+ },
+ },
+ [IPA_CLIENT_Q6_CMD_PROD] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_PKT_PROCESS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 4,
+ .channel_id = 1,
+ .tlv_count = 20,
+ .ee = IPA_EE_Q6,
+ },
+ },
+ [IPA_CLIENT_TEST_CONS] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 14,
+ .channel_id = 5,
+ .tlv_count = 8,
+ .ee = IPA_EE_Q6,
+ },
+ },
+ [IPA_CLIENT_TEST1_CONS] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 15,
+ .channel_id = 2,
+ .tlv_count = 8,
+ .ee = IPA_EE_UC,
+ },
+ },
+ /* Only for testing */
+ [IPA_CLIENT_TEST_PROD] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 0,
+ .channel_id = 0,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_TEST1_PROD] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 0,
+ .channel_id = 0,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_TEST2_PROD] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 2,
+ .channel_id = 3,
+ .tlv_count = 16,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_TEST3_PROD] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 4,
+ .channel_id = 1,
+ .tlv_count = 20,
+ .ee = IPA_EE_Q6,
+ },
+ },
+ [IPA_CLIENT_TEST4_PROD] = {
+ .support_flt = true,
+ .seq_type = IPA_SEQ_2ND_PKT_PROCESS_PASS_NO_DEC_UCP,
+ .ipa_gsi_ep_info = {
+ .ep_id = 1,
+ .channel_id = 0,
+ .tlv_count = 8,
+ .ee = IPA_EE_UC,
+ },
+ },
+ [IPA_CLIENT_WLAN1_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 16,
+ .channel_id = 3,
+ .tlv_count = 8,
+ .ee = IPA_EE_UC,
+ },
+ },
+ [IPA_CLIENT_WLAN2_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 18,
+ .channel_id = 9,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_WLAN3_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 19,
+ .channel_id = 10,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_USB_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 17,
+ .channel_id = 8,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_USB_DPL_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 11,
+ .channel_id = 2,
+ .tlv_count = 4,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_APPS_LAN_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 9,
+ .channel_id = 5,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_APPS_WAN_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 10,
+ .channel_id = 6,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_Q6_LAN_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 13,
+ .channel_id = 3,
+ .tlv_count = 8,
+ .ee = IPA_EE_Q6,
+ },
+ },
+ [IPA_CLIENT_Q6_WAN_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 12,
+ .channel_id = 2,
+ .tlv_count = 8,
+ .ee = IPA_EE_Q6,
+ },
+ },
+ /* Only for testing */
+ [IPA_CLIENT_TEST2_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 18,
+ .channel_id = 9,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_TEST3_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 19,
+ .channel_id = 10,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+ [IPA_CLIENT_TEST4_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 11,
+ .channel_id = 2,
+ .tlv_count = 4,
+ .ee = IPA_EE_AP,
+ },
+ },
+/* Dummy consumer (endpoint 31) is used in L2TP rt rule */
+ [IPA_CLIENT_DUMMY_CONS] = {
+ .support_flt = false,
+ .seq_type = IPA_SEQ_INVALID,
+ .ipa_gsi_ep_info = {
+ .ep_id = 31,
+ .channel_id = 31,
+ .tlv_count = 8,
+ .ee = IPA_EE_AP,
+ },
+ },
+};
+
+/** ipa_client_ep_id() - provide endpoint mapping
+ * @client: client type
+ *
+ * Return value: endpoint mapping
+ */
+u32 ipa_client_ep_id(enum ipa_client_type client)
+{
+ return ipa_ep_configuration[client].ipa_gsi_ep_info.ep_id;
+}
+
+u32 ipa_client_channel_id(enum ipa_client_type client)
+{
+ return ipa_ep_configuration[client].ipa_gsi_ep_info.channel_id;
+}
+
+u32 ipa_client_tlv_count(enum ipa_client_type client)
+{
+ return ipa_ep_configuration[client].ipa_gsi_ep_info.tlv_count;
+}
+
+enum ipa_seq_type ipa_endp_seq_type(u32 ep_id)
+{
+ return ipa_ep_configuration[ipa_ctx->ep[ep_id].client].seq_type;
+}
+
+/** ipa_sram_settings_read() - Read SRAM settings from HW
+ *
+ * Returns: None
+ */
+void ipa_sram_settings_read(void)
+{
+ struct ipa_reg_shared_mem_size mem_size;
+
+ ipa_read_reg_fields(IPA_SHARED_MEM_SIZE, &mem_size);
+
+ /* reg fields are in 8B units */
+ ipa_ctx->smem_offset = mem_size.shared_mem_baddr * 8;
+ ipa_ctx->smem_size = mem_size.shared_mem_size * 8;
+
+ ipa_debug("sram size 0x%x offset 0x%x\n", ipa_ctx->smem_size,
+ ipa_ctx->smem_offset);
+}
+
+/** ipa_init_hw() - initialize HW */
+void ipa_init_hw(void)
+{
+ struct ipa_reg_qsb_max_writes max_writes;
+ struct ipa_reg_qsb_max_reads max_reads;
+
+ /* SDM845 has IPA version 3.5.1 */
+ ipa_write_reg(IPA_BCR, IPA_BCR_REG_VAL);
+
+ ipa_reg_qsb_max_writes(&max_writes, 8, 4);
+ ipa_write_reg_fields(IPA_QSB_MAX_WRITES, &max_writes);
+
+ ipa_reg_qsb_max_reads(&max_reads, 8, 12);
+ ipa_write_reg_fields(IPA_QSB_MAX_READS, &max_reads);
+}
+
+/** ipa_filter_bitmap_init() - Initialize the bitmap
+ * that represents the End-points that supports filtering
+ */
+u32 ipa_filter_bitmap_init(void)
+{
+ enum ipa_client_type client;
+ u32 filter_bitmap = 0;
+ u32 count = 0;
+
+ for (client = 0; client < IPA_CLIENT_MAX ; client++) {
+ const struct ipa_ep_configuration *ep_config;
+
+ ep_config = &ipa_ep_configuration[client];
+ if (!ep_config->support_flt)
+ continue;
+ if (++count > IPA_MEM_FLT_COUNT)
+ return 0; /* Too many filtering endpoints */
+
+ filter_bitmap |= BIT(ep_config->ipa_gsi_ep_info.ep_id);
+ }
+
+ return filter_bitmap;
+}
+
+/* In IPAv3 only endpoints 0-3 can be configured to deaggregation */
+bool ipa_endp_aggr_support(u32 ep_id)
+{
+ return ep_id < 4;
+}
+
+/** ipa_endp_init_hdr_write()
+ *
+ * @ep_id: endpoint whose header config register should be written
+ */
+static void ipa_endp_init_hdr_write(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_HDR_N, ep_id, &ep->init_hdr);
+}
+
+/** ipa_endp_init_hdr_ext_write() - write endpoint extended header register
+ *
+ * @ep_id: endpoint whose register should be written
+ */
+static void
+ipa_endp_init_hdr_ext_write(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_HDR_EXT_N, ep_id,
+ &ep->hdr_ext);
+}
+
+/** ipa_endp_init_aggr_write() write endpoint aggregation register
+ *
+ * @ep_id: endpoint whose aggregation config register should be written
+ */
+static void ipa_endp_init_aggr_write(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_AGGR_N, ep_id, &ep->init_aggr);
+}
+
+/** ipa_endp_init_cfg_write() - write endpoint configuration register
+ *
+ * @ep_id: endpoint whose configuration register should be written
+ */
+static void ipa_endp_init_cfg_write(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_CFG_N, ep_id, &ep->init_cfg);
+}
+
+/** ipa_endp_init_mode_write() - write endpoint mode register
+ *
+ * @ep_id: endpoint whose register should be written
+ */
+static void ipa_endp_init_mode_write(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_MODE_N, ep_id,
+ &ep->init_mode);
+}
+
+/** ipa_endp_init_seq_write() - write endpoint sequencer register
+ *
+ * @ep_id: endpoint whose register should be written
+ */
+static void ipa_endp_init_seq_write(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_SEQ_N, ep_id, &ep->init_seq);
+}
+
+/** ipa_endp_init_deaggr_write() - write endpoint deaggregation register
+ *
+ * @ep_id: endpoint whose register should be written
+ */
+void ipa_endp_init_deaggr_write(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_DEAGGR_N, ep_id,
+ &ep->init_deaggr);
+}
+
+/** ipa_endp_init_hdr_metadata_mask_write() - endpoint metadata mask register
+ *
+ * @ep_id: endpoint whose register should be written
+ */
+static void ipa_endp_init_hdr_metadata_mask_write(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_HDR_METADATA_MASK_N, ep_id,
+ &ep->metadata_mask);
+}
+
+/** ipa_endp_init_hdr_metadata_mask_write() - endpoint metadata mask register
+ *
+ * @ep_id: endpoint whose register should be written
+ */
+static void ipa_endp_status_write(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+
+ ipa_write_reg_n_fields(IPA_ENDP_STATUS_N, ep_id, &ep->status);
+}
+
+/** ipa_cfg_ep - IPA end-point configuration
+ * @ep_id: [in] endpoint id assigned by IPA to client
+ * @dst: [in] destination client handle (ignored for consumer clients)
+ *
+ * This includes nat, IPv6CT, header, mode, aggregation and route settings and
+ * is a one shot API to configure the IPA end-point fully
+ *
+ * Returns: 0 on success, negative on failure
+ *
+ * Note: Should not be called from atomic context
+ */
+void ipa_cfg_ep(u32 ep_id)
+{
+ ipa_endp_init_hdr_write(ep_id);
+ ipa_endp_init_hdr_ext_write(ep_id);
+
+ ipa_endp_init_aggr_write(ep_id);
+ ipa_endp_init_cfg_write(ep_id);
+
+ if (ipa_producer(ipa_ctx->ep[ep_id].client)) {
+ ipa_endp_init_mode_write(ep_id);
+ ipa_endp_init_seq_write(ep_id);
+ ipa_endp_init_deaggr_write(ep_id);
+ } else {
+ ipa_endp_init_hdr_metadata_mask_write(ep_id);
+ }
+
+ ipa_endp_status_write(ep_id);
+}
+
+int ipa_interconnect_init(struct device *dev)
+{
+ struct icc_path *path;
+
+ path = of_icc_get(dev, "memory");
+ if (IS_ERR(path))
+ goto err_return;
+ ipa_ctx->memory_path = path;
+
+ path = of_icc_get(dev, "imem");
+ if (IS_ERR(path))
+ goto err_memory_path_put;
+ ipa_ctx->imem_path = path;
+
+ path = of_icc_get(dev, "config");
+ if (IS_ERR(path))
+ goto err_imem_path_put;
+ ipa_ctx->config_path = path;
+
+ return 0;
+
+err_imem_path_put:
+ icc_put(ipa_ctx->imem_path);
+ ipa_ctx->imem_path = NULL;
+err_memory_path_put:
+ icc_put(ipa_ctx->memory_path);
+ ipa_ctx->memory_path = NULL;
+err_return:
+
+ return PTR_ERR(path);
+}
+
+void ipa_interconnect_exit(void)
+{
+ icc_put(ipa_ctx->config_path);
+ ipa_ctx->config_path = NULL;
+
+ icc_put(ipa_ctx->imem_path);
+ ipa_ctx->imem_path = NULL;
+
+ icc_put(ipa_ctx->memory_path);
+ ipa_ctx->memory_path = NULL;
+}
+
+/* Currently we only use bandwidth level, so just "enable" interconnects */
+int ipa_interconnect_enable(void)
+{
+ int ret;
+
+ ret = icc_set(ipa_ctx->memory_path, IPA_MEMORY_AVG, IPA_MEMORY_PEAK);
+ if (ret)
+ return ret;
+
+ ret = icc_set(ipa_ctx->imem_path, IPA_IMEM_AVG, IPA_IMEM_PEAK);
+ if (ret)
+ goto err_disable_memory_path;
+
+ ret = icc_set(ipa_ctx->config_path, IPA_CONFIG_AVG, IPA_CONFIG_PEAK);
+ if (!ret)
+ return 0; /* Success */
+
+ (void)icc_set(ipa_ctx->imem_path, 0, 0);
+err_disable_memory_path:
+ (void)icc_set(ipa_ctx->memory_path, 0, 0);
+
+ return ret;
+}
+
+/* To disable an interconnect, we just its bandwidth to 0 */
+int ipa_interconnect_disable(void)
+{
+ int ret;
+
+ ret = icc_set(ipa_ctx->memory_path, 0, 0);
+ if (ret)
+ return ret;
+
+ ret = icc_set(ipa_ctx->imem_path, 0, 0);
+ if (ret)
+ goto err_reenable_memory_path;
+
+ ret = icc_set(ipa_ctx->config_path, 0, 0);
+ if (!ret)
+ return 0; /* Success */
+
+ /* Re-enable things in the event of an error */
+ (void)icc_set(ipa_ctx->imem_path, IPA_IMEM_AVG, IPA_IMEM_PEAK);
+err_reenable_memory_path:
+ (void)icc_set(ipa_ctx->memory_path, IPA_MEMORY_AVG, IPA_MEMORY_PEAK);
+
+ return ret;
+}
+
+/** ipa_proxy_clk_unvote() - called to remove IPA clock proxy vote
+ *
+ * Return value: none
+ */
+void ipa_proxy_clk_unvote(void)
+{
+ if (ipa_ctx->modem_clk_vote_valid) {
+ ipa_client_remove();
+ ipa_ctx->modem_clk_vote_valid = false;
+ }
+}
+
+/** ipa_proxy_clk_vote() - called to add IPA clock proxy vote
+ *
+ * Return value: none
+ */
+void ipa_proxy_clk_vote(void)
+{
+ if (!ipa_ctx->modem_clk_vote_valid) {
+ ipa_client_add();
+ ipa_ctx->modem_clk_vote_valid = true;
+ }
+}
+
+u32 ipa_get_ep_count(void)
+{
+ return ipa_read_reg(IPA_ENABLED_PIPES);
+}
+
+/** ipa_is_modem_ep()- Checks if endpoint is owned by the modem
+ *
+ * @ep_id: endpoint identifier
+ * Return value: true if owned by modem, false otherwize
+ */
+bool ipa_is_modem_ep(u32 ep_id)
+{
+ int client_idx;
+
+ for (client_idx = 0; client_idx < IPA_CLIENT_MAX; client_idx++) {
+ if (!ipa_modem_consumer(client_idx) &&
+ !ipa_modem_producer(client_idx))
+ continue;
+ if (ipa_client_ep_id(client_idx) == ep_id)
+ return true;
+ }
+
+ return false;
+}
+
+static void ipa_src_rsrc_grp_init(enum ipa_rsrc_grp_type_src n)
+{
+ struct ipa_reg_rsrc_grp_xy_rsrc_type_n limits;
+ const struct rsrc_min_max *x_limits;
+ const struct rsrc_min_max *y_limits;
+
+ x_limits = &ipa_src_rsrc_grp[n][IPA_RSRC_GROUP_LWA_DL];
+ y_limits = &ipa_src_rsrc_grp[n][IPA_RSRC_GROUP_UL_DL];
+ ipa_reg_rsrc_grp_xy_rsrc_type_n(&limits, x_limits->min, x_limits->max,
+ y_limits->min, y_limits->max);
+
+ ipa_write_reg_n_fields(IPA_SRC_RSRC_GRP_01_RSRC_TYPE_N, n, &limits);
+}
+
+static void ipa_dst_rsrc_grp_init(enum ipa_rsrc_grp_type_dst n)
+{
+ struct ipa_reg_rsrc_grp_xy_rsrc_type_n limits;
+ const struct rsrc_min_max *x_limits;
+ const struct rsrc_min_max *y_limits;
+
+ x_limits = &ipa_dst_rsrc_grp[n][IPA_RSRC_GROUP_LWA_DL];
+ y_limits = &ipa_dst_rsrc_grp[n][IPA_RSRC_GROUP_UL_DL];
+ ipa_reg_rsrc_grp_xy_rsrc_type_n(&limits, x_limits->min, x_limits->max,
+ y_limits->min, y_limits->max);
+
+ ipa_write_reg_n_fields(IPA_DST_RSRC_GRP_01_RSRC_TYPE_N, n, &limits);
+}
+
+void ipa_set_resource_groups_min_max_limits(void)
+{
+ ipa_src_rsrc_grp_init(IPA_RSRC_GRP_TYPE_SRC_PKT_CONTEXTS);
+ ipa_src_rsrc_grp_init(IPA_RSRC_GRP_TYPE_SRS_DESCRIPTOR_LISTS);
+ ipa_src_rsrc_grp_init(IPA_RSRC_GRP_TYPE_SRC_DESCRIPTOR_BUFF);
+ ipa_src_rsrc_grp_init(IPA_RSRC_GRP_TYPE_SRC_HPS_DMARS);
+ ipa_src_rsrc_grp_init(IPA_RSRC_GRP_TYPE_SRC_ACK_ENTRIES);
+
+ ipa_dst_rsrc_grp_init(IPA_RSRC_GRP_TYPE_DST_DATA_SECTORS);
+ ipa_dst_rsrc_grp_init(IPA_RSRC_GRP_TYPE_DST_DPS_DMARS);
+}
+
+static void ipa_gsi_poll_after_suspend(struct ipa_ep_context *ep)
+{
+ ipa_rx_switch_to_poll_mode(ep->sys);
+}
+
+/* Suspend a consumer endpoint */
+static void ipa_ep_cons_suspend(enum ipa_client_type client)
+{
+ struct ipa_reg_endp_init_ctrl init_ctrl;
+ u32 ep_id = ipa_client_ep_id(client);
+
+ ipa_reg_endp_init_ctrl(&init_ctrl, true);
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_CTRL_N, ep_id, &init_ctrl);
+
+ /* Due to a hardware bug, a client suspended with an open
+ * aggregation frame will not generate a SUSPEND IPA interrupt.
+ * We work around this by force-closing the aggregation frame,
+ * then simulating the arrival of such an interrupt.
+ */
+ ipa_suspend_active_aggr_wa(ep_id);
+
+ ipa_gsi_poll_after_suspend(&ipa_ctx->ep[ep_id]);
+}
+
+void ipa_ep_suspend_all(void)
+{
+ ipa_ep_cons_suspend(IPA_CLIENT_APPS_WAN_CONS);
+ ipa_ep_cons_suspend(IPA_CLIENT_APPS_LAN_CONS);
+}
+
+/* Resume a suspended consumer endpoint */
+static void ipa_ep_cons_resume(enum ipa_client_type client)
+{
+ struct ipa_reg_endp_init_ctrl init_ctrl;
+ struct ipa_ep_context *ep;
+ u32 ep_id;
+
+ ep_id = ipa_client_ep_id(client);
+ ep = &ipa_ctx->ep[ep_id];
+
+ ipa_reg_endp_init_ctrl(&init_ctrl, false);
+ ipa_write_reg_n_fields(IPA_ENDP_INIT_CTRL_N, ep_id, &init_ctrl);
+
+ if (!ipa_ep_polling(ep))
+ gsi_channel_intr_enable(ipa_ctx->gsi, ep->channel_id);
+}
+
+void ipa_ep_resume_all(void)
+{
+ ipa_ep_cons_resume(IPA_CLIENT_APPS_LAN_CONS);
+ ipa_ep_cons_resume(IPA_CLIENT_APPS_WAN_CONS);
+}
+
+/** ipa_cfg_route() - configure IPA route
+ * @route: IPA route
+ *
+ * Return codes:
+ * 0: success
+ */
+void ipa_cfg_default_route(enum ipa_client_type client)
+{
+ struct ipa_reg_route route;
+
+ ipa_reg_route(&route, ipa_client_ep_id(client));
+ ipa_write_reg_fields(IPA_ROUTE, &route);
+}
+
+/* In certain cases we need to issue a command to reliably clear the
+ * IPA pipeline. Sending a 1-byte DMA task is sufficient, and this
+ * function preallocates a command to do just that. There are
+ * conditions (process context in KILL state) where DMA allocations
+ * can fail, and we need to be able to issue this command to put the
+ * hardware in a known state. By preallocating the command here we
+ * guarantee it can't fail for that reason.
+ */
+int ipa_gsi_dma_task_alloc(void)
+{
+ struct ipa_dma_mem *mem = &ipa_ctx->dma_task_info.mem;
+
+ if (ipa_dma_alloc(mem, IPA_GSI_CHANNEL_STOP_PKT_SIZE, GFP_KERNEL))
+ return -ENOMEM;
+
+ /* IPA_IMM_CMD_DMA_TASK_32B_ADDR */
+ ipa_ctx->dma_task_info.payload = ipahal_dma_task_32b_addr_pyld(mem);
+ if (!ipa_ctx->dma_task_info.payload) {
+ ipa_dma_free(mem);
+
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+void ipa_gsi_dma_task_free(void)
+{
+ struct ipa_dma_mem *mem = &ipa_ctx->dma_task_info.mem;
+
+ ipahal_payload_free(ipa_ctx->dma_task_info.payload);
+ ipa_ctx->dma_task_info.payload = NULL;
+ ipa_dma_free(mem);
+}
+
+/** ipa_gsi_dma_task_inject()- Send DMA_TASK to IPA for GSI stop channel
+ *
+ * Send a DMA_TASK of 1B to IPA to unblock GSI channel in STOP_IN_PROG.
+ * Return value: 0 on success, negative otherwise
+ */
+static int ipa_gsi_dma_task_inject(void)
+{
+ struct ipa_desc desc = { };
+
+ desc.type = IPA_IMM_CMD_DESC;
+ desc.len_opcode = IPA_IMM_CMD_DMA_TASK_32B_ADDR;
+ desc.payload = ipa_ctx->dma_task_info.payload;
+
+ return ipa_send_cmd_timeout(&desc, IPA_GSI_DMA_TASK_TIMEOUT);
+}
+
+/** ipa_stop_gsi_channel()- Stops a GSI channel in IPA
+ * @chan_hdl: GSI channel handle
+ *
+ * This function implements the sequence to stop a GSI channel
+ * in IPA. This function returns when the channel is is STOP state.
+ *
+ * Return value: 0 on success, negative otherwise
+ */
+int ipa_stop_gsi_channel(u32 ep_id)
+{
+ struct ipa_ep_context *ep = &ipa_ctx->ep[ep_id];
+ int ret;
+ int i;
+
+ if (ipa_producer(ep->client))
+ return gsi_channel_stop(ipa_ctx->gsi, ep->channel_id);
+
+ for (i = 0; i < IPA_GSI_CHANNEL_STOP_MAX_RETRY; i++) {
+ ret = gsi_channel_stop(ipa_ctx->gsi, ep->channel_id);
+ if (ret != -EAGAIN && ret != -ETIMEDOUT)
+ return ret;
+
+ /* Send a 1B packet DMA_TASK to IPA and try again */
+ ret = ipa_gsi_dma_task_inject();
+ if (ret)
+ return ret;
+
+ /* sleep for short period to flush IPA */
+ usleep_range(IPA_GSI_CHANNEL_STOP_SLEEP_MIN,
+ IPA_GSI_CHANNEL_STOP_SLEEP_MAX);
+ }
+
+ ipa_err("Failed to stop GSI channel with retries\n");
+
+ return -EFAULT;
+}
+
+/** ipa_enable_dcd() - enable dynamic clock division on IPA
+ *
+ * Return value: Non applicable
+ *
+ */
+void ipa_enable_dcd(void)
+{
+ struct ipa_reg_idle_indication_cfg indication;
+
+ /* recommended values for IPA 3.5 according to IPA HPG */
+ ipa_reg_idle_indication_cfg(&indication, 256, 0);
+ ipa_write_reg_fields(IPA_IDLE_INDICATION_CFG, &indication);
+}
+
+/** ipa_set_flt_tuple_mask() - Sets the flt tuple masking for the given
+ * endpoint. Endpoint must be for AP (not modem) and support filtering.
+ * Updates the the filtering masking values without changing the rt ones.
+ *
+ * @ep_id: filter endpoint to configure the tuple masking
+ * @tuple: the tuple members masking
+ * Returns: 0 on success, negative on failure
+ *
+ */
+void ipa_set_flt_tuple_mask(u32 ep_id)
+{
+ struct ipa_ep_filter_router_hsh_cfg hsh_cfg;
+
+ ipa_read_reg_n_fields(IPA_ENDP_FILTER_ROUTER_HSH_CFG_N, ep_id,
+ &hsh_cfg);
+
+ ipa_reg_hash_tuple(&hsh_cfg.flt);
+
+ ipa_write_reg_n_fields(IPA_ENDP_FILTER_ROUTER_HSH_CFG_N, ep_id,
+ &hsh_cfg);
+}
+
+/** ipa_set_rt_tuple_mask() - Sets the rt tuple masking for the given tbl
+ * table index must be for AP EP (not modem)
+ * updates the the routing masking values without changing the flt ones.
+ *
+ * @tbl_idx: routing table index to configure the tuple masking
+ * @tuple: the tuple members masking
+ * Returns: 0 on success, negative on failure
+ *
+ */
+void ipa_set_rt_tuple_mask(int tbl_idx)
+{
+ struct ipa_ep_filter_router_hsh_cfg hsh_cfg;
+
+ ipa_read_reg_n_fields(IPA_ENDP_FILTER_ROUTER_HSH_CFG_N, tbl_idx,
+ &hsh_cfg);
+
+ ipa_reg_hash_tuple(&hsh_cfg.rt);
+
+ ipa_write_reg_n_fields(IPA_ENDP_FILTER_ROUTER_HSH_CFG_N, tbl_idx,
+ &hsh_cfg);
+}
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("IPA HW device driver");
--
2.17.1
^ permalink raw reply related
* [RFC PATCH 06/12] soc: qcom: ipa: QMI modem communication
From: Alex Elder @ 2018-11-07 0:32 UTC (permalink / raw)
To: davem, arnd, bjorn.andersson, ilias.apalodimas
Cc: netdev, devicetree, linux-arm-msm, linux-soc, linux-arm-kernel,
linux-kernel, syadagir, mjavid, robh+dt, mark.rutland
In-Reply-To: <20181107003250.5832-1-elder@linaro.org>
QMI is a mechanism that allows entities on the AP to communicate and
coordinate with peer entities on a modem. Each peer can create an
endpoint for communicating with the other. QMI defines a way for
the format of messages sent between endpoints to be described, and
uses a service to route and deliver these messages.
For IPA, QMI is used to synchronize the startup sequence of the IPA
drivers resident on the AP and modem. The documentation in
"ipa_qmi.c" below provides more detail about the message interchange
involved. The IPA QMI code is divided into the "main" code that
implements message handling and synchronization, and the message
code that defines the structure and layout of the messages used.
Signed-off-by: Alex Elder <elder@linaro.org>
---
drivers/net/ipa/ipa_qmi.c | 406 +++++++++++++++++++++++
drivers/net/ipa/ipa_qmi.h | 12 +
drivers/net/ipa/ipa_qmi_msg.c | 587 ++++++++++++++++++++++++++++++++++
drivers/net/ipa/ipa_qmi_msg.h | 233 ++++++++++++++
4 files changed, 1238 insertions(+)
create mode 100644 drivers/net/ipa/ipa_qmi.c
create mode 100644 drivers/net/ipa/ipa_qmi.h
create mode 100644 drivers/net/ipa/ipa_qmi_msg.c
create mode 100644 drivers/net/ipa/ipa_qmi_msg.h
diff --git a/drivers/net/ipa/ipa_qmi.c b/drivers/net/ipa/ipa_qmi.c
new file mode 100644
index 000000000000..a19cfc4043d3
--- /dev/null
+++ b/drivers/net/ipa/ipa_qmi.c
@@ -0,0 +1,406 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2013-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/string.h>
+#include <linux/slab.h>
+#include <linux/qrtr.h>
+#include <linux/soc/qcom/qmi.h>
+
+#include "ipa_qmi_msg.h"
+
+#include "ipa_i.h" /* ipa_err() */
+
+#define QMI_INIT_DRIVER_TIMEOUT 60000 /* A minute in milliseconds */
+
+static bool ipa_qmi_initialized;
+
+/* The AP and modem perform a "handshake" at initialization time to
+ * ensure each side knows the other side is ready. Two pairs of QMI
+ * handles (endpoints) are used for this; one provides service on
+ * the modem for AP requests, and the other is on the AP to service
+ * modem requests (and to supply an indication from the AP).
+ *
+ * The QMI service on the modem expects to receive an INIT_DRIVER
+ * request from the AP, which contains parameters used by the
+ * modem during initialization. The AP sends this request as soon
+ * as it is knows the service is available. The modem responds to
+ * this request immediately.
+ *
+ * When the modem learns the AP service is available, it is able
+ * to communicate its status to the AP. The modem uses this to
+ * tell the AP when it is ready to receive an indication, sending
+ * an INDICATION_REGISTER request to the handle served by the AP.
+ * This is separate from the modem driver initialization.
+ *
+ * When the modem has completed the driver initialization requested
+ * by the AP, it sends a DRIVER_INIT_COMPLETE request to the AP.
+ * This request could arrive at the AP either before or after the
+ * INDICATION_REGISTER request.
+ *
+ * The final step in the handshake occurs after the AP has received
+ * both requests from the modem. The AP completes the handshake by
+ * sending an INIT_COMPLETE_IND indication message to the modem.
+ */
+
+#define IPA_HOST_SERVICE_SVC_ID 0x31
+#define IPA_HOST_SVC_VERS 1
+#define IPA_HOST_SERVICE_INS_ID 1
+
+#define IPA_MODEM_SERVICE_SVC_ID 0x31
+#define IPA_MODEM_SERVICE_INS_ID 2
+#define IPA_MODEM_SVC_VERS 1
+
+/* Used to send an INIT_DRIVER request to the modem */
+static struct qmi_handle client_handle;
+
+/* Requests from the modem arrive on the server handle to tell us
+ * when it is prepared to receive an INIT_COMPLETE indication, and
+ * when its driver initialization is complete. The AP sends the
+ * indication after it has received and responded to both requests.
+ */
+static struct qmi_handle server_handle;
+
+/* These track state during the handshake */
+static bool indication_register_received;
+static bool init_driver_response_received;
+
+/* Send an INIT_COMPLETE_IND indication message to the modem */
+static int ipa_send_master_driver_init_complete_ind(struct qmi_handle *qmi,
+ struct sockaddr_qrtr *sq)
+{
+ struct ipa_init_complete_ind ind = { };
+
+ ind.status.result = QMI_RESULT_SUCCESS_V01;
+ ind.status.error = QMI_ERR_NONE_V01;
+
+ return qmi_send_indication(qmi, sq, IPA_QMI_INIT_COMPLETE_IND,
+ IPA_QMI_INIT_COMPLETE_IND_SZ,
+ ipa_init_complete_ind_ei, &ind);
+}
+
+/* This function is called to determine whether to complete the
+ * handshake by sending an INIT_COMPLETE_IND indication message to
+ * the modem. The "init_driver" parameter is false when we've
+ * received an INDICATION_REGISTER request message from the modem,
+ * or true when we've received the response from the INIT_DRIVER
+ * request message we send. If this function decides the message
+ * should be sent, it calls ipa_send_master_driver_init_complete_ind()
+ * to send it.
+ */
+static void ipa_handshake_complete(struct qmi_handle *qmi,
+ struct sockaddr_qrtr *sq, bool init_driver)
+{
+ bool send_it;
+ int ret;
+
+ if (init_driver) {
+ init_driver_response_received = true;
+ send_it = indication_register_received;
+ } else {
+ indication_register_received = true;
+ send_it = init_driver_response_received;
+ }
+ if (!send_it)
+ return;
+
+ ret = ipa_send_master_driver_init_complete_ind(qmi, sq);
+ if (ret)
+ ipa_err("error %d sending init complete indication\n", ret);
+}
+
+/* Callback function to handle an INDICATION_REGISTER request message
+ * from the modem. This informs the AP that the modem is now ready to
+ * receive the INIT_COMPLETE_IND indication message.
+ */
+static void ipa_indication_register_fn(struct qmi_handle *qmi,
+ struct sockaddr_qrtr *sq,
+ struct qmi_txn *txn,
+ const void *decoded)
+{
+ struct ipa_indication_register_rsp rsp = { };
+ int ret;
+
+ rsp.rsp.result = QMI_RESULT_SUCCESS_V01;
+ rsp.rsp.error = QMI_ERR_NONE_V01;
+
+ ret = qmi_send_response(qmi, sq, txn, IPA_QMI_INDICATION_REGISTER,
+ IPA_QMI_INDICATION_REGISTER_RSP_SZ,
+ ipa_indication_register_rsp_ei, &rsp);
+ if (ret)
+ ipa_err("error %d sending response\n", ret);
+ else
+ ipa_handshake_complete(qmi, sq, false);
+}
+
+/* Callback function to handle a DRIVER_INIT_COMPLETE request message
+ * from the modem. This informs the AP that the modem has completed
+ * the initializion of its driver.
+ */
+static void ipa_driver_init_complete_fn(struct qmi_handle *qmi,
+ struct sockaddr_qrtr *sq,
+ struct qmi_txn *txn,
+ const void *decoded)
+{
+ struct ipa_driver_init_complete_rsp rsp = { };
+ int ret;
+
+ rsp.rsp.result = QMI_RESULT_SUCCESS_V01;
+ rsp.rsp.error = QMI_ERR_NONE_V01;
+
+ ret = qmi_send_response(qmi, sq, txn, IPA_QMI_DRIVER_INIT_COMPLETE,
+ IPA_QMI_DRIVER_INIT_COMPLETE_RSP_SZ,
+ ipa_driver_init_complete_rsp_ei, &rsp);
+ if (ret)
+ ipa_err("error %d sending response\n", ret);
+}
+
+/* The server handles two request message types sent by the modem. */
+static struct qmi_msg_handler ipa_server_msg_handlers[] = {
+ {
+ .type = QMI_REQUEST,
+ .msg_id = IPA_QMI_INDICATION_REGISTER,
+ .ei = ipa_indication_register_req_ei,
+ .decoded_size = IPA_QMI_INDICATION_REGISTER_REQ_SZ,
+ .fn = ipa_indication_register_fn,
+ },
+ {
+ .type = QMI_REQUEST,
+ .msg_id = IPA_QMI_DRIVER_INIT_COMPLETE,
+ .ei = ipa_driver_init_complete_req_ei,
+ .decoded_size = IPA_QMI_DRIVER_INIT_COMPLETE_REQ_SZ,
+ .fn = ipa_driver_init_complete_fn,
+ },
+};
+
+/* Callback function to handle an IPA_QMI_INIT_DRIVER response message
+ * from the modem. This only acknowledges that the modem received the
+ * request. The modem will eventually report that it has completed its
+ * modem initialization by sending a IPA_QMI_DRIVER_INIT_COMPLETE request.
+ */
+static void ipa_init_driver_rsp_fn(struct qmi_handle *qmi,
+ struct sockaddr_qrtr *sq,
+ struct qmi_txn *txn,
+ const void *decoded)
+{
+ txn->result = 0; /* IPA_QMI_INIT_DRIVER request was successful */
+ complete(&txn->completion);
+
+ ipa_handshake_complete(qmi, sq, true);
+}
+
+/* The client handles one response message type sent by the modem. */
+static struct qmi_msg_handler ipa_client_msg_handlers[] = {
+ {
+ .type = QMI_RESPONSE,
+ .msg_id = IPA_QMI_INIT_DRIVER,
+ .ei = ipa_init_modem_driver_rsp_ei,
+ .decoded_size = IPA_QMI_INIT_DRIVER_RSP_SZ,
+ .fn = ipa_init_driver_rsp_fn,
+ },
+};
+
+/* Return a pointer to an init modem driver request structure, which
+ * contains configuration parameters for the modem. The modem may
+ * be started multiple times, but generally these parameters don't
+ * change so we can reuse the request structure once it's initialized.
+ * The only exception is the skip_uc_load field, which will be set
+ * only after the microcontroller has reported it has completed its
+ * initialization.
+ */
+static const struct ipa_init_modem_driver_req *init_modem_driver_req(void)
+{
+ static struct ipa_init_modem_driver_req req;
+ u32 base;
+
+ /* This is not the first boot if the microcontroller is loaded */
+ req.skip_uc_load = ipa_uc_loaded() ? 1 : 0;
+ req.skip_uc_load_valid = true;
+
+ /* We only have to initialize most of it once */
+ if (req.platform_type_valid)
+ return &req;
+
+ /* All offsets are relative to the start of IPA shared memory */
+ base = (u32)ipa_ctx->smem_offset;
+
+ req.platform_type_valid = true;
+ req.platform_type = IPA_QMI_PLATFORM_TYPE_MSM_ANDROID;
+
+ req.hdr_tbl_info_valid = IPA_MEM_MODEM_HDR_SIZE ? 1 : 0;
+ req.hdr_tbl_info.start = base + IPA_MEM_MODEM_HDR_OFST;
+ req.hdr_tbl_info.end = req.hdr_tbl_info.start +
+ IPA_MEM_MODEM_HDR_SIZE - 1;
+
+ req.v4_route_tbl_info_valid = true;
+ req.v4_route_tbl_info.start = base + IPA_MEM_V4_RT_NHASH_OFST;
+ req.v4_route_tbl_info.count = IPA_MEM_MODEM_RT_COUNT;
+
+ req.v6_route_tbl_info_valid = true;
+ req.v6_route_tbl_info.start = base + IPA_MEM_V6_RT_NHASH_OFST;
+ req.v6_route_tbl_info.count = IPA_MEM_MODEM_RT_COUNT;
+
+ req.v4_filter_tbl_start_valid = true;
+ req.v4_filter_tbl_start = base + IPA_MEM_V4_FLT_NHASH_OFST;
+
+ req.v6_filter_tbl_start_valid = true;
+ req.v6_filter_tbl_start = base + IPA_MEM_V6_FLT_NHASH_OFST;
+
+ req.modem_mem_info_valid = IPA_MEM_MODEM_SIZE ? 1 : 0;
+ req.modem_mem_info.start = base + IPA_MEM_MODEM_OFST;
+ req.modem_mem_info.size = IPA_MEM_MODEM_SIZE;
+
+ req.ctrl_comm_dest_end_pt_valid = true;
+ req.ctrl_comm_dest_end_pt = ipa_client_ep_id(IPA_CLIENT_APPS_WAN_CONS);
+
+ req.hdr_proc_ctx_tbl_info_valid =
+ IPA_MEM_MODEM_HDR_PROC_CTX_SIZE ? 1 : 0;
+ req.hdr_proc_ctx_tbl_info.start =
+ base + IPA_MEM_MODEM_HDR_PROC_CTX_OFST;
+ req.hdr_proc_ctx_tbl_info.end = req.hdr_proc_ctx_tbl_info.start +
+ IPA_MEM_MODEM_HDR_PROC_CTX_SIZE - 1;
+
+ req.v4_hash_route_tbl_info_valid = true;
+ req.v4_hash_route_tbl_info.start = base + IPA_MEM_V4_RT_HASH_OFST;
+ req.v4_hash_route_tbl_info.count = IPA_MEM_MODEM_RT_COUNT;
+
+ req.v6_hash_route_tbl_info_valid = true;
+ req.v6_hash_route_tbl_info.start = base + IPA_MEM_V6_RT_HASH_OFST;
+ req.v6_hash_route_tbl_info.count = IPA_MEM_MODEM_RT_COUNT;
+
+ req.v4_hash_filter_tbl_start_valid = true;
+ req.v4_hash_filter_tbl_start = base + IPA_MEM_V4_FLT_HASH_OFST;
+
+ req.v6_hash_filter_tbl_start_valid = true;
+ req.v6_hash_filter_tbl_start = base + IPA_MEM_V6_FLT_HASH_OFST;
+
+ return &req;
+}
+
+/* The modem service we requested is now available via the client
+ * handle. Send an INIT_DRIVER request to the modem.
+ */
+static int
+ipa_client_new_server(struct qmi_handle *qmi, struct qmi_service *svc)
+{
+ const struct ipa_init_modem_driver_req *req = init_modem_driver_req();
+ struct sockaddr_qrtr sq;
+ struct qmi_txn *txn;
+ int ret;
+
+ txn = kzalloc(sizeof(*txn), GFP_KERNEL);
+ if (!txn)
+ return -ENOMEM;
+
+ ret = qmi_txn_init(qmi, txn, NULL, NULL);
+ if (ret) {
+ kfree(txn);
+ return ret;
+ }
+
+ sq.sq_family = AF_QIPCRTR;
+ sq.sq_node = svc->node;
+ sq.sq_port = svc->port;
+
+ ret = qmi_send_request(qmi, &sq, txn, IPA_QMI_INIT_DRIVER,
+ IPA_QMI_INIT_DRIVER_REQ_SZ,
+ ipa_init_modem_driver_req_ei, req);
+ if (!ret)
+ ret = qmi_txn_wait(txn, MAX_SCHEDULE_TIMEOUT);
+ if (ret)
+ qmi_txn_cancel(txn);
+ kfree(txn);
+
+ return ret;
+}
+
+/* The only callback we supply for the client handle is notification
+ * that the service on the modem has become available.
+ */
+static struct qmi_ops ipa_client_ops = {
+ .new_server = ipa_client_new_server,
+};
+
+static int ipa_qmi_initialize(void)
+{
+ int ret;
+
+ /* The only handle operation that might be interesting for the
+ * server would be del_client, to find out when the modem side
+ * client has disappeared. But other than reporting the event,
+ * we wouldn't do anything about that. So we just pass a null
+ * pointer for its handle operations. All the real work is
+ * done by the message handlers.
+ */
+ ret = qmi_handle_init(&server_handle, IPA_QMI_SERVER_MAX_RCV_SZ,
+ NULL, ipa_server_msg_handlers);
+ if (ret < 0)
+ return ret;
+
+ ret = qmi_add_server(&server_handle, IPA_HOST_SERVICE_SVC_ID,
+ IPA_HOST_SVC_VERS, IPA_HOST_SERVICE_INS_ID);
+ if (ret < 0)
+ goto err_release_server_handle;
+
+ /* The client handle is only used for sending an INIT_DRIVER
+ * request to the modem, and receiving its response message.
+ */
+ ret = qmi_handle_init(&client_handle, IPA_QMI_CLIENT_MAX_RCV_SZ,
+ &ipa_client_ops, ipa_client_msg_handlers);
+ if (ret < 0)
+ goto err_release_server_handle;
+
+ ret = qmi_add_lookup(&client_handle, IPA_MODEM_SERVICE_SVC_ID,
+ IPA_MODEM_SVC_VERS, IPA_MODEM_SERVICE_INS_ID);
+ if (ret < 0)
+ goto err_release_client_handle;
+
+ ipa_qmi_initialized = true;
+
+ return 0;
+
+err_release_client_handle:
+ /* Releasing the handle also removes registered lookups */
+ qmi_handle_release(&client_handle);
+ memset(&client_handle, 0, sizeof(client_handle));
+err_release_server_handle:
+ /* Releasing the handle also removes registered services */
+ qmi_handle_release(&server_handle);
+ memset(&server_handle, 0, sizeof(server_handle));
+
+ return ret;
+}
+
+/* This is called by the rmnet probe routine. The rmnet driver can
+ * be unregistered after it has been initialized as a result of a
+ * subsystem shutdown; it can later be registered again if a
+ * subsystem restart occurs. This function can therefore be called
+ * more than once.
+ */
+int ipa_qmi_init(void)
+{
+ init_driver_response_received = false;
+ indication_register_received = false;
+
+ if (!ipa_qmi_initialized)
+ return ipa_qmi_initialize();
+
+ return 0;
+}
+
+void ipa_qmi_exit(void)
+{
+ if (!ipa_qmi_initialized)
+ return;
+
+ qmi_handle_release(&client_handle);
+ memset(&client_handle, 0, sizeof(client_handle));
+
+ qmi_handle_release(&server_handle);
+ memset(&server_handle, 0, sizeof(server_handle));
+
+ ipa_qmi_initialized = false;
+}
diff --git a/drivers/net/ipa/ipa_qmi.h b/drivers/net/ipa/ipa_qmi.h
new file mode 100644
index 000000000000..3c03ff5c0454
--- /dev/null
+++ b/drivers/net/ipa/ipa_qmi.h
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+#ifndef _IPA_QMI_H_
+#define _IPA_QMI_H_
+
+int ipa_qmi_init(void);
+void ipa_qmi_exit(void);
+
+#endif /* !_IPA_QMI_H_ */
diff --git a/drivers/net/ipa/ipa_qmi_msg.c b/drivers/net/ipa/ipa_qmi_msg.c
new file mode 100644
index 000000000000..c5347c63ef41
--- /dev/null
+++ b/drivers/net/ipa/ipa_qmi_msg.c
@@ -0,0 +1,587 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+#include <linux/stddef.h>
+#include <linux/soc/qcom/qmi.h>
+
+#include "ipa_qmi_msg.h"
+
+#ifndef sizeof_field
+#define sizeof_field(TYPE, MEMBER) sizeof(((TYPE *)0)->MEMBER)
+#endif /* !sizeof_field */
+
+/* QMI message structure definition for struct ipa_indication_register_req */
+struct qmi_elem_info ipa_indication_register_req_ei[] = {
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_indication_register_req,
+ master_driver_init_complete_valid),
+ .tlv_type = 0x10,
+ .offset = offsetof(struct ipa_indication_register_req,
+ master_driver_init_complete_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_1_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_indication_register_req,
+ master_driver_init_complete),
+ .tlv_type = 0x10,
+ .offset = offsetof(struct ipa_indication_register_req,
+ master_driver_init_complete),
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_indication_register_req,
+ data_usage_quota_reached_valid),
+ .tlv_type = 0x11,
+ .offset = offsetof(struct ipa_indication_register_req,
+ data_usage_quota_reached_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_1_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_indication_register_req,
+ data_usage_quota_reached),
+ .tlv_type = 0x11,
+ .offset = offsetof(struct ipa_indication_register_req,
+ data_usage_quota_reached),
+ },
+ {
+ .data_type = QMI_EOTI,
+ },
+};
+
+/* QMI message structure definition for struct ipa_indication_register_rsp */
+struct qmi_elem_info ipa_indication_register_rsp_ei[] = {
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_indication_register_rsp,
+ rsp),
+ .tlv_type = 0x02,
+ .offset = offsetof(struct ipa_indication_register_rsp,
+ rsp),
+ .ei_array = qmi_response_type_v01_ei,
+ },
+ {
+ .data_type = QMI_EOTI,
+ },
+};
+
+/* QMI message structure definition for struct ipa_driver_init_complete_req */
+struct qmi_elem_info ipa_driver_init_complete_req_ei[] = {
+ {
+ .data_type = QMI_UNSIGNED_1_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_driver_init_complete_req,
+ status),
+ .tlv_type = 0x01,
+ .offset = offsetof(struct ipa_driver_init_complete_req,
+ status),
+ },
+ {
+ .data_type = QMI_EOTI,
+ },
+};
+
+/* QMI message structure definition for struct ipa_driver_init_complete_rsp */
+struct qmi_elem_info ipa_driver_init_complete_rsp_ei[] = {
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_driver_init_complete_rsp,
+ rsp),
+ .tlv_type = 0x02,
+ .elem_size = offsetof(struct ipa_driver_init_complete_rsp,
+ rsp),
+ .ei_array = qmi_response_type_v01_ei,
+ },
+ {
+ .data_type = QMI_EOTI,
+ },
+};
+
+/* QMI message structure definition for struct ipa_init_complete_ind */
+struct qmi_elem_info ipa_init_complete_ind_ei[] = {
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_complete_ind,
+ status),
+ .tlv_type = 0x02,
+ .elem_size = offsetof(struct ipa_init_complete_ind,
+ status),
+ .ei_array = qmi_response_type_v01_ei,
+ },
+ {
+ .data_type = QMI_EOTI,
+ },
+};
+
+/* QMI message structure definition for struct ipa_mem_bounds */
+struct qmi_elem_info ipa_mem_bounds_ei[] = {
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_mem_bounds, start),
+ .offset = offsetof(struct ipa_mem_bounds, start),
+ },
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_mem_bounds, end),
+ .offset = offsetof(struct ipa_mem_bounds, end),
+ },
+ {
+ .data_type = QMI_EOTI,
+ },
+};
+
+/* QMI message structure definition for struct ipa_mem_array */
+struct qmi_elem_info ipa_mem_array_ei[] = {
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_mem_array, start),
+ .offset = offsetof(struct ipa_mem_array, start),
+ },
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_mem_array, count),
+ .offset = offsetof(struct ipa_mem_array, count),
+ },
+ {
+ .data_type = QMI_EOTI,
+ },
+};
+
+/* QMI message structure definition for struct ipa_mem_range */
+struct qmi_elem_info ipa_mem_range_ei[] = {
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_mem_range, start),
+ .offset = offsetof(struct ipa_mem_range, start),
+ },
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_mem_range, size),
+ .offset = offsetof(struct ipa_mem_range, size),
+ },
+ {
+ .data_type = QMI_EOTI,
+ },
+};
+
+/* QMI message structure definition for struct ipa_init_modem_driver_req */
+struct qmi_elem_info ipa_init_modem_driver_req_ei[] = {
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ platform_type_valid),
+ .tlv_type = 0x10,
+ .elem_size = offsetof(struct ipa_init_modem_driver_req,
+ platform_type_valid),
+ },
+ {
+ .data_type = QMI_SIGNED_4_BYTE_ENUM,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ platform_type),
+ .tlv_type = 0x10,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ platform_type),
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ hdr_tbl_info_valid),
+ .tlv_type = 0x11,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ hdr_tbl_info_valid),
+ },
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ hdr_tbl_info),
+ .tlv_type = 0x11,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ hdr_tbl_info),
+ .ei_array = ipa_mem_bounds_ei,
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v4_route_tbl_info_valid),
+ .tlv_type = 0x12,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v4_route_tbl_info_valid),
+ },
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v4_route_tbl_info),
+ .tlv_type = 0x12,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v4_route_tbl_info),
+ .ei_array = ipa_mem_array_ei,
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v6_route_tbl_info_valid),
+ .tlv_type = 0x13,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v6_route_tbl_info_valid),
+ },
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v6_route_tbl_info),
+ .tlv_type = 0x13,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v6_route_tbl_info),
+ .ei_array = ipa_mem_array_ei,
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v4_filter_tbl_start_valid),
+ .tlv_type = 0x14,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v4_filter_tbl_start_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v4_filter_tbl_start),
+ .tlv_type = 0x14,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v4_filter_tbl_start),
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v6_filter_tbl_start_valid),
+ .tlv_type = 0x15,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v6_filter_tbl_start_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v6_filter_tbl_start),
+ .tlv_type = 0x15,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v6_filter_tbl_start),
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ modem_mem_info_valid),
+ .tlv_type = 0x16,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ modem_mem_info_valid),
+ },
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ modem_mem_info),
+ .tlv_type = 0x16,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ modem_mem_info),
+ .ei_array = ipa_mem_range_ei,
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ ctrl_comm_dest_end_pt_valid),
+ .tlv_type = 0x17,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ ctrl_comm_dest_end_pt_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ ctrl_comm_dest_end_pt),
+ .tlv_type = 0x17,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ ctrl_comm_dest_end_pt),
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ skip_uc_load_valid),
+ .tlv_type = 0x18,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ skip_uc_load_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_1_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ skip_uc_load),
+ .tlv_type = 0x18,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ skip_uc_load),
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ hdr_proc_ctx_tbl_info_valid),
+ .tlv_type = 0x19,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ hdr_proc_ctx_tbl_info_valid),
+ },
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ hdr_proc_ctx_tbl_info),
+ .tlv_type = 0x19,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ hdr_proc_ctx_tbl_info),
+ .ei_array = ipa_mem_bounds_ei,
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ zip_tbl_info_valid),
+ .tlv_type = 0x1a,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ zip_tbl_info_valid),
+ },
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ zip_tbl_info),
+ .tlv_type = 0x1a,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ zip_tbl_info),
+ .ei_array = ipa_mem_bounds_ei,
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v4_hash_route_tbl_info_valid),
+ .tlv_type = 0x1b,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v4_hash_route_tbl_info_valid),
+ },
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v4_hash_route_tbl_info),
+ .tlv_type = 0x1b,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v4_hash_route_tbl_info),
+ .ei_array = ipa_mem_array_ei,
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v6_hash_route_tbl_info_valid),
+ .tlv_type = 0x1c,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v6_hash_route_tbl_info_valid),
+ },
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v6_hash_route_tbl_info),
+ .tlv_type = 0x1c,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v6_hash_route_tbl_info),
+ .ei_array = ipa_mem_array_ei,
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v4_hash_filter_tbl_start_valid),
+ .tlv_type = 0x1d,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v4_hash_filter_tbl_start_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v4_hash_filter_tbl_start),
+ .tlv_type = 0x1d,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v4_hash_filter_tbl_start),
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v6_hash_filter_tbl_start_valid),
+ .tlv_type = 0x1e,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v6_hash_filter_tbl_start_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_req,
+ v6_hash_filter_tbl_start),
+ .tlv_type = 0x1e,
+ .offset = offsetof(struct ipa_init_modem_driver_req,
+ v6_hash_filter_tbl_start),
+ },
+ {
+ .data_type = QMI_EOTI,
+ },
+};
+
+/* QMI message structure definition for struct ipa_init_modem_driver_rsp */
+struct qmi_elem_info ipa_init_modem_driver_rsp_ei[] = {
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_rsp,
+ rsp),
+ .tlv_type = 0x02,
+ .offset = offsetof(struct ipa_init_modem_driver_rsp,
+ rsp),
+ .ei_array = qmi_response_type_v01_ei,
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_rsp,
+ ctrl_comm_dest_end_pt_valid),
+ .tlv_type = 0x10,
+ .offset = offsetof(struct ipa_init_modem_driver_rsp,
+ ctrl_comm_dest_end_pt_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_rsp,
+ ctrl_comm_dest_end_pt),
+ .tlv_type = 0x10,
+ .offset = offsetof(struct ipa_init_modem_driver_rsp,
+ ctrl_comm_dest_end_pt),
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_rsp,
+ default_end_pt_valid),
+ .tlv_type = 0x11,
+ .offset = offsetof(struct ipa_init_modem_driver_rsp,
+ default_end_pt_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_4_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_rsp,
+ default_end_pt),
+ .tlv_type = 0x11,
+ .offset = offsetof(struct ipa_init_modem_driver_rsp,
+ default_end_pt),
+ },
+ {
+ .data_type = QMI_OPT_FLAG,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_rsp,
+ modem_driver_init_pending_valid),
+ .tlv_type = 0x12,
+ .offset = offsetof(struct ipa_init_modem_driver_rsp,
+ modem_driver_init_pending_valid),
+ },
+ {
+ .data_type = QMI_UNSIGNED_1_BYTE,
+ .elem_len = 1,
+ .elem_size =
+ sizeof_field(struct ipa_init_modem_driver_rsp,
+ modem_driver_init_pending),
+ .tlv_type = 0x12,
+ .offset = offsetof(struct ipa_init_modem_driver_rsp,
+ modem_driver_init_pending),
+ },
+ {
+ .data_type = QMI_EOTI,
+ },
+};
diff --git a/drivers/net/ipa/ipa_qmi_msg.h b/drivers/net/ipa/ipa_qmi_msg.h
new file mode 100644
index 000000000000..009ce65ea114
--- /dev/null
+++ b/drivers/net/ipa/ipa_qmi_msg.h
@@ -0,0 +1,233 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+#ifndef _IPA_QMI_MSG_H_
+#define _IPA_QMI_MSG_H_
+
+/* Request/response/indication QMI message ids used for IPA. Receiving
+ * end issues a response for requests; indications require no response.
+ */
+#define IPA_QMI_INDICATION_REGISTER 0x20 /* modem -> AP request */
+#define IPA_QMI_INIT_DRIVER 0x21 /* AP -> modem request */
+#define IPA_QMI_INIT_COMPLETE_IND 0x22 /* AP -> modem indication */
+#define IPA_QMI_DRIVER_INIT_COMPLETE 0x35 /* modem -> AP request */
+
+/* The maximum size required for message types. These sizes include
+ * the message data, along with type (1 byte) and length (2 byte)
+ * information for each field. The qmi_send_*() interfaces require
+ * the message size to be provided.
+ */
+#define IPA_QMI_INDICATION_REGISTER_REQ_SZ 8 /* -> server handle */
+#define IPA_QMI_INDICATION_REGISTER_RSP_SZ 7 /* <- server handle */
+#define IPA_QMI_INIT_DRIVER_REQ_SZ 134 /* client handle -> */
+#define IPA_QMI_INIT_DRIVER_RSP_SZ 25 /* client handle <- */
+#define IPA_QMI_INIT_COMPLETE_IND_SZ 7 /* server handle -> */
+#define IPA_QMI_DRIVER_INIT_COMPLETE_REQ_SZ 4 /* -> server handle */
+#define IPA_QMI_DRIVER_INIT_COMPLETE_RSP_SZ 7 /* <- server handle */
+
+/* Maximum size of messages we expect the AP to receive (max of above) */
+#define IPA_QMI_SERVER_MAX_RCV_SZ 8
+#define IPA_QMI_CLIENT_MAX_RCV_SZ 25
+
+/* Request message for the IPA_QMI_INDICATION_REGISTER request */
+struct ipa_indication_register_req {
+ u8 master_driver_init_complete_valid;
+ u8 master_driver_init_complete;
+ u8 data_usage_quota_reached_valid;
+ u8 data_usage_quota_reached;
+};
+
+/* The response to a IPA_QMI_INDICATION_REGISTER request consists only of
+ * a standard QMI response.
+ */
+struct ipa_indication_register_rsp {
+ struct qmi_response_type_v01 rsp;
+};
+
+/* Request message for the IPA_QMI_DRIVER_INIT_COMPLETE request */
+struct ipa_driver_init_complete_req {
+ u8 status;
+};
+
+/* The response to a IPA_QMI_DRIVER_INIT_COMPLETE request consists only
+ * of a standard QMI response.
+ */
+struct ipa_driver_init_complete_rsp {
+ struct qmi_response_type_v01 rsp;
+};
+
+/* The message for the IPA_QMI_INIT_COMPLETE_IND indication consists
+ * only of a standard QMI response.
+ */
+struct ipa_init_complete_ind {
+ struct qmi_response_type_v01 status;
+};
+
+/* The AP tells the modem its platform type. We assume Android. */
+enum ipa_platform_type {
+ IPA_QMI_PLATFORM_TYPE_INVALID = 0, /* Invalid */
+ IPA_QMI_PLATFORM_TYPE_TN = 1, /* Data card */
+ IPA_QMI_PLATFORM_TYPE_LE = 2, /* Data router */
+ IPA_QMI_PLATFORM_TYPE_MSM_ANDROID = 3, /* Android MSM */
+ IPA_QMI_PLATFORM_TYPE_MSM_WINDOWS = 4, /* Windows MSM */
+ IPA_QMI_PLATFORM_TYPE_MSM_QNX_V01 = 5, /* QNX MSM */
+};
+
+/* This defines the start and end offset of a range of memory. Both
+ * fields are offsets relative to the start of IPA shared memory.
+ * The end value is the last addressable byte *within* the range.
+ */
+struct ipa_mem_bounds {
+ u32 start;
+ u32 end;
+};
+
+/* This defines the location and size of an array. The start value
+ * is an offset relative to the start of IPA shared memory. The
+ * size of the array is implied by the number of entries (the entry
+ * size is assumed to be known).
+ */
+struct ipa_mem_array {
+ u32 start;
+ u32 count;
+};
+
+/* This defines the location and size of a range of memory. The
+ * start is an offset relative to the start of IPA shared memory.
+ * This differs from the ipa_mem_bounds structure in that the size
+ * (in bytes) of the memory region is specified rather than the
+ * offset of its last byte.
+ */
+struct ipa_mem_range {
+ u32 start;
+ u32 size;
+};
+
+/* The message for the IPA_QMI_INIT_DRIVER request contains information
+ * from the AP that affects modem initialization.
+ */
+struct ipa_init_modem_driver_req {
+ u8 platform_type_valid;
+ u32 platform_type; /* enum ipa_platform_type */
+
+ /* Modem header table information. This defines the IPA shared
+ * memory in which the modem may insert header table entries.
+ */
+ u8 hdr_tbl_info_valid;
+ struct ipa_mem_bounds hdr_tbl_info;
+
+ /* Routing table information. These define the location and size of
+ * non-hashable IPv4 and IPv6 filter tables. The start values are
+ * offsets relative to the start of IPA shared memory.
+ */
+ u8 v4_route_tbl_info_valid;
+ struct ipa_mem_array v4_route_tbl_info;
+ u8 v6_route_tbl_info_valid;
+ struct ipa_mem_array v6_route_tbl_info;
+
+ /* Filter table information. These define the location and size of
+ * non-hashable IPv4 and IPv6 filter tables. The start values are
+ * offsets relative to the start of IPA shared memory.
+ */
+ u8 v4_filter_tbl_start_valid;
+ u32 v4_filter_tbl_start;
+ u8 v6_filter_tbl_start_valid;
+ u32 v6_filter_tbl_start;
+
+ /* Modem memory information. This defines the location and
+ * size of memory available for the modem to use.
+ */
+ u8 modem_mem_info_valid;
+ struct ipa_mem_range modem_mem_info;
+
+ /* This defines the destination endpoint on the AP to which
+ * the modem driver can send control commands. IPA supports
+ * 20 endpoints, so this must be 19 or less.
+ */
+ u8 ctrl_comm_dest_end_pt_valid;
+ u32 ctrl_comm_dest_end_pt;
+
+ /* This defines whether the modem should load the microcontroller
+ * or not. It is unnecessary to reload it if the modem is being
+ * restarted.
+ *
+ * NOTE: this field is named "is_ssr_bootup" elsewhere.
+ */
+ u8 skip_uc_load_valid;
+ u8 skip_uc_load;
+
+ /* Processing context memory information. This defines the memory in
+ * which the modem may insert header processing context table entries.
+ */
+ u8 hdr_proc_ctx_tbl_info_valid;
+ struct ipa_mem_bounds hdr_proc_ctx_tbl_info;
+
+ /* Compression command memory information. This defines the memory
+ * in which the modem may insert compression/decompression commands.
+ */
+ u8 zip_tbl_info_valid;
+ struct ipa_mem_bounds zip_tbl_info;
+
+ /* Routing table information. These define the location and size
+ * of hashable IPv4 and IPv6 filter tables. The start values are
+ * offsets relative to the start of IPA shared memory.
+ */
+ u8 v4_hash_route_tbl_info_valid;
+ struct ipa_mem_array v4_hash_route_tbl_info;
+ u8 v6_hash_route_tbl_info_valid;
+ struct ipa_mem_array v6_hash_route_tbl_info;
+
+ /* Filter table information. These define the location and size
+ * of hashable IPv4 and IPv6 filter tables. The start values are
+ * offsets relative to the start of IPA shared memory.
+ */
+ u8 v4_hash_filter_tbl_start_valid;
+ u32 v4_hash_filter_tbl_start;
+ u8 v6_hash_filter_tbl_start_valid;
+ u32 v6_hash_filter_tbl_start;
+};
+
+/* The response to a IPA_QMI_INIT_DRIVER request begins with a standard
+ * QMI response, but contains other information as well. Currently we
+ * simply wait for the the INIT_DRIVER transaction to complete and
+ * ignore any other data that might be returned.
+ */
+struct ipa_init_modem_driver_rsp {
+ struct qmi_response_type_v01 rsp;
+
+ /* This defines the destination endpoint on the modem to which
+ * the AP driver can send control commands. IPA supports
+ * 20 endpoints, so this must be 19 or less.
+ */
+ u8 ctrl_comm_dest_end_pt_valid;
+ u32 ctrl_comm_dest_end_pt;
+
+ /* This defines the default endpoint. The AP driver is not
+ * required to configure the hardware with this value. IPA
+ * supports 20 endpoints, so this must be 19 or less.
+ */
+ u8 default_end_pt_valid;
+ u32 default_end_pt;
+
+ /* This defines whether a second handshake is required to complete
+ * initialization.
+ */
+ u8 modem_driver_init_pending_valid;
+ u8 modem_driver_init_pending;
+};
+
+/* Message structure definitions defined in "ipa_qmi_msg.c" */
+extern struct qmi_elem_info ipa_indication_register_req_ei[];
+extern struct qmi_elem_info ipa_indication_register_rsp_ei[];
+extern struct qmi_elem_info ipa_driver_init_complete_req_ei[];
+extern struct qmi_elem_info ipa_driver_init_complete_rsp_ei[];
+extern struct qmi_elem_info ipa_init_complete_ind_ei[];
+extern struct qmi_elem_info ipa_mem_bounds_ei[];
+extern struct qmi_elem_info ipa_mem_array_ei[];
+extern struct qmi_elem_info ipa_mem_range_ei[];
+extern struct qmi_elem_info ipa_init_modem_driver_req_ei[];
+extern struct qmi_elem_info ipa_init_modem_driver_rsp_ei[];
+
+#endif /* !_IPA_QMI_MSG_H_ */
--
2.17.1
^ permalink raw reply related
* [RFC PATCH 05/12] soc: qcom: ipa: IPA interrupts and the microcontroller
From: Alex Elder @ 2018-11-07 0:32 UTC (permalink / raw)
To: davem, arnd, bjorn.andersson, ilias.apalodimas
Cc: netdev, devicetree, linux-arm-msm, linux-soc, linux-arm-kernel,
linux-kernel, syadagir, mjavid, robh+dt, mark.rutland
In-Reply-To: <20181107003250.5832-1-elder@linaro.org>
The IPA has an interrupt line distinct from the interrupt used by
the GSI code. Whereas GSI interrupts are generally related to
channel events (like transfer completions), IPA interrupts are
related to other events related to the IPA. When the IPA IRQ fires,
an IPA interrupt status register indicates which IPA interrupt
events are being signaled. IPA interrupts can be masked
independently, and can also be endependently enabled or disabled.
The IPA has an embedded microcontroller that can be used for
additional processing of messages passing through the IPA. This
feature is generally not used by the current code.
Currently only three IPA interrupts are used: one to trigger a
resume when in a suspended state; and two that allow the embedded
microcontroller to signal events.
Signed-off-by: Alex Elder <elder@linaro.org>
---
drivers/net/ipa/ipa_interrupts.c | 307 ++++++++++++++++++++++++++++
drivers/net/ipa/ipa_uc.c | 336 +++++++++++++++++++++++++++++++
2 files changed, 643 insertions(+)
create mode 100644 drivers/net/ipa/ipa_interrupts.c
create mode 100644 drivers/net/ipa/ipa_uc.c
diff --git a/drivers/net/ipa/ipa_interrupts.c b/drivers/net/ipa/ipa_interrupts.c
new file mode 100644
index 000000000000..75cd81a1eab0
--- /dev/null
+++ b/drivers/net/ipa/ipa_interrupts.c
@@ -0,0 +1,307 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2014-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+
+/*
+ * DOC: IPA Interrupts
+ *
+ * The IPA has an interrupt line distinct from the interrupt used
+ * by the GSI code. Whereas GSI interrupts are generally related
+ * to channel events (like transfer completions), IPA interrupts are
+ * related to other events related to the IPA. Some of the IPA
+ * interrupts come from a microcontroller embedded in the IPA.
+ * Each IPA interrupt type can be both masked and acknowledged
+ * independent of the others,
+ *
+ * So two of the IPA interrupts are initiated by the microcontroller.
+ * A third can be generated to signal the need for a wakeup/resume
+ * when the IPA has been suspended. The modem can cause this event
+ * to occur (for example, for an incoming call). There are other IPA
+ * events defined, but at this time only these three are supported.
+ */
+
+#include <linux/types.h>
+#include <linux/interrupt.h>
+#include <linux/workqueue.h>
+
+#include "ipa_i.h"
+#include "ipa_reg.h"
+
+struct ipa_interrupt_info {
+ ipa_irq_handler_t handler;
+ enum ipa_irq_type interrupt;
+};
+
+#define IPA_IRQ_NUM_MAX 32 /* Number of IRQ bits in IPA interrupt mask */
+static struct ipa_interrupt_info ipa_interrupt_info[IPA_IRQ_NUM_MAX];
+
+static struct workqueue_struct *ipa_interrupt_wq;
+
+static void enable_tx_suspend_work_func(struct work_struct *work);
+static DECLARE_DELAYED_WORK(tx_suspend_work, enable_tx_suspend_work_func);
+
+static const int ipa_irq_mapping[] = {
+ [IPA_INVALID_IRQ] = -1,
+ [IPA_UC_IRQ_0] = 2,
+ [IPA_UC_IRQ_1] = 3,
+ [IPA_TX_SUSPEND_IRQ] = 14,
+};
+
+/* IPA interrupt handlers are called in contexts that can block */
+static void ipa_interrupt_work_func(struct work_struct *work);
+static DECLARE_WORK(ipa_interrupt_work, ipa_interrupt_work_func);
+
+/* Workaround disables TX_SUSPEND interrupt for this long */
+#define DISABLE_TX_SUSPEND_INTR_DELAY msecs_to_jiffies(5)
+
+/* Disable the IPA TX_SUSPEND interrupt, and arrange for it to be
+ * re-enabled again in 5 milliseconds.
+ *
+ * This is part of a hardware bug workaround.
+ */
+static void ipa_tx_suspend_interrupt_wa(void)
+{
+ u32 val;
+
+ val = ipa_read_reg_n(IPA_IRQ_EN_EE_N, IPA_EE_AP);
+ val &= ~BIT(ipa_irq_mapping[IPA_TX_SUSPEND_IRQ]);
+ ipa_write_reg_n(IPA_IRQ_EN_EE_N, IPA_EE_AP, val);
+
+ queue_delayed_work(ipa_interrupt_wq, &tx_suspend_work,
+ DISABLE_TX_SUSPEND_INTR_DELAY);
+}
+
+static void ipa_handle_interrupt(int irq_num)
+{
+ struct ipa_interrupt_info *intr_info = &ipa_interrupt_info[irq_num];
+ u32 endpoints = 0; /* Only TX_SUSPEND uses its interrupt_data */
+
+ if (!intr_info->handler)
+ return;
+
+ if (intr_info->interrupt == IPA_TX_SUSPEND_IRQ) {
+ /* Disable the suspend interrupt temporarily */
+ ipa_tx_suspend_interrupt_wa();
+
+ /* Get and clear mask of endpoints signaling TX_SUSPEND */
+ endpoints = ipa_read_reg_n(IPA_IRQ_SUSPEND_INFO_EE_N,
+ IPA_EE_AP);
+ ipa_write_reg_n(IPA_SUSPEND_IRQ_CLR_EE_N, IPA_EE_AP, endpoints);
+ }
+
+ intr_info->handler(intr_info->interrupt, endpoints);
+}
+
+static inline bool is_uc_irq(int irq_num)
+{
+ enum ipa_irq_type interrupt = ipa_interrupt_info[irq_num].interrupt;
+
+ return interrupt != IPA_UC_IRQ_0 && interrupt != IPA_UC_IRQ_1;
+}
+
+static void ipa_process_interrupts(void)
+{
+ while (true) {
+ u32 ipa_intr_mask;
+ u32 imask; /* one set bit */
+
+ /* Determine which interrupts have fired, then examine only
+ * those that are enabled. Note that a suspend interrupt
+ * bug forces us to re-read the enabled mask every time to
+ * avoid an endless loop.
+ */
+ ipa_intr_mask = ipa_read_reg_n(IPA_IRQ_STTS_EE_N, IPA_EE_AP);
+ ipa_intr_mask &= ipa_read_reg_n(IPA_IRQ_EN_EE_N, IPA_EE_AP);
+
+ if (!ipa_intr_mask)
+ break;
+
+ do {
+ int i = __ffs(ipa_intr_mask);
+ bool uc_irq = is_uc_irq(i);
+
+ imask = BIT(i);
+
+ /* Clear uC interrupt before processing to avoid
+ * clearing unhandled interrupts
+ */
+ if (uc_irq)
+ ipa_write_reg_n(IPA_IRQ_CLR_EE_N, IPA_EE_AP,
+ imask);
+
+ ipa_handle_interrupt(i);
+
+ /* Clear non-uC interrupt after processing
+ * to avoid clearing interrupt data
+ */
+ if (!uc_irq)
+ ipa_write_reg_n(IPA_IRQ_CLR_EE_N, IPA_EE_AP,
+ imask);
+ } while ((ipa_intr_mask ^= imask));
+ }
+}
+
+static void ipa_interrupt_work_func(struct work_struct *work)
+{
+ ipa_client_add();
+
+ ipa_process_interrupts();
+
+ ipa_client_remove();
+}
+
+static irqreturn_t ipa_isr(int irq, void *ctxt)
+{
+ /* Schedule handling (if not already scheduled) */
+ queue_work(ipa_interrupt_wq, &ipa_interrupt_work);
+
+ return IRQ_HANDLED;
+}
+
+/* Re-enable the IPA TX_SUSPEND interrupt after having been disabled
+ * for a moment by ipa_tx_suspend_interrupt_wa(). This is part of a
+ * workaround for a hardware bug.
+ */
+static void enable_tx_suspend_work_func(struct work_struct *work)
+{
+ u32 val;
+
+ ipa_client_add();
+
+ val = ipa_read_reg_n(IPA_IRQ_EN_EE_N, IPA_EE_AP);
+ val |= BIT(ipa_irq_mapping[IPA_TX_SUSPEND_IRQ]);
+ ipa_write_reg_n(IPA_IRQ_EN_EE_N, IPA_EE_AP, val);
+
+ ipa_process_interrupts();
+
+ ipa_client_remove();
+}
+
+/* Register SUSPEND_IRQ_EN_EE_N_ADDR for L2 interrupt. */
+static void tx_suspend_enable(void)
+{
+ enum ipa_client_type client;
+ u32 val = ~0;
+
+ /* Compute the mask to use (bits set for all non-modem endpoints) */
+ for (client = 0; client < IPA_CLIENT_MAX; client++)
+ if (ipa_modem_consumer(client) || ipa_modem_producer(client))
+ val &= ~BIT(ipa_client_ep_id(client));
+
+ ipa_write_reg_n(IPA_SUSPEND_IRQ_EN_EE_N, IPA_EE_AP, val);
+}
+
+/* Unregister SUSPEND_IRQ_EN_EE_N_ADDR for L2 interrupt. */
+static void tx_suspend_disable(void)
+{
+ ipa_write_reg_n(IPA_SUSPEND_IRQ_EN_EE_N, IPA_EE_AP, 0);
+}
+
+/**
+ * ipa_add_interrupt_handler() - Adds handler for an IPA interrupt
+ * @interrupt: IPA interrupt type
+ * @handler: The handler for that interrupt
+ *
+ * Adds handler to an IPA interrupt type and enable it. IPA interrupt
+ * handlers are allowed to block (they aren't run in interrupt context).
+ */
+void ipa_add_interrupt_handler(enum ipa_irq_type interrupt,
+ ipa_irq_handler_t handler)
+{
+ int irq_num = ipa_irq_mapping[interrupt];
+ struct ipa_interrupt_info *intr_info;
+ u32 val;
+
+ intr_info = &ipa_interrupt_info[irq_num];
+ intr_info->handler = handler;
+ intr_info->interrupt = interrupt;
+
+ /* Enable the IPA interrupt */
+ val = ipa_read_reg_n(IPA_IRQ_EN_EE_N, IPA_EE_AP);
+ val |= BIT(irq_num);
+ ipa_write_reg_n(IPA_IRQ_EN_EE_N, IPA_EE_AP, val);
+
+ if (interrupt == IPA_TX_SUSPEND_IRQ)
+ tx_suspend_enable();
+}
+
+/**
+ * ipa_remove_interrupt_handler() - Removes handler for an IPA interrupt type
+ * @interrupt: IPA interrupt type
+ *
+ * Remove an IPA interrupt handler and disable it.
+ */
+void ipa_remove_interrupt_handler(enum ipa_irq_type interrupt)
+{
+ int irq_num = ipa_irq_mapping[interrupt];
+ struct ipa_interrupt_info *intr_info;
+ u32 val;
+
+ intr_info = &ipa_interrupt_info[irq_num];
+ intr_info->handler = NULL;
+ intr_info->interrupt = IPA_INVALID_IRQ;
+
+ if (interrupt == IPA_TX_SUSPEND_IRQ)
+ tx_suspend_disable();
+
+ /* Disable the interrupt */
+ val = ipa_read_reg_n(IPA_IRQ_EN_EE_N, IPA_EE_AP);
+ val &= ~BIT(irq_num);
+ ipa_write_reg_n(IPA_IRQ_EN_EE_N, IPA_EE_AP, val);
+}
+
+/**
+ * ipa_interrupts_init() - Initialize the IPA interrupts framework
+ */
+int ipa_interrupts_init(void)
+{
+ int ret;
+
+ ret = request_irq(ipa_ctx->ipa_irq, ipa_isr, IRQF_TRIGGER_RISING,
+ "ipa", ipa_ctx->dev);
+ if (ret)
+ return ret;
+
+ ipa_interrupt_wq = alloc_ordered_workqueue("ipa_interrupt_wq", 0);
+ if (ipa_interrupt_wq)
+ return 0;
+
+ free_irq(ipa_ctx->ipa_irq, ipa_ctx->dev);
+
+ return -ENOMEM;
+}
+
+/**
+ * ipa_suspend_active_aggr_wa() - Emulate suspend interrupt
+ * @ep_id: Endpoint on which to emulate a suspend
+ *
+ * Emulate suspend IRQ to unsuspend a client suspended with an open
+ * aggregation frame. This is to work around a hardware issue
+ * where an IRQ is not generated as it should be when this occurs.
+ */
+void ipa_suspend_active_aggr_wa(u32 ep_id)
+{
+ struct ipa_reg_aggr_force_close force_close;
+ struct ipa_interrupt_info *intr_info;
+ u32 clnt_mask;
+ int irq_num;
+
+ irq_num = ipa_irq_mapping[IPA_TX_SUSPEND_IRQ];
+ intr_info = &ipa_interrupt_info[irq_num];
+ clnt_mask = BIT(ep_id);
+
+ /* Nothing to do if the endpoint doesn't have aggregation open */
+ if (!(ipa_read_reg(IPA_STATE_AGGR_ACTIVE) & clnt_mask))
+ return;
+
+ /* Force close aggregation */
+ ipa_reg_aggr_force_close(&force_close, clnt_mask);
+ ipa_write_reg_fields(IPA_AGGR_FORCE_CLOSE, &force_close);
+
+ /* Simulate suspend IRQ */
+ ipa_assert(!in_interrupt());
+ if (intr_info->handler)
+ intr_info->handler(intr_info->interrupt, clnt_mask);
+}
diff --git a/drivers/net/ipa/ipa_uc.c b/drivers/net/ipa/ipa_uc.c
new file mode 100644
index 000000000000..2065e53f3601
--- /dev/null
+++ b/drivers/net/ipa/ipa_uc.c
@@ -0,0 +1,336 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/delay.h>
+
+#include "ipa_i.h"
+
+/**
+ * DOC: The IPA Embedded Microcontroller
+ *
+ * The IPA incorporates an embedded microcontroller that is able to
+ * do some additional handling/offloading of network activity. The
+ * current code makes essentially no use of the microcontroller.
+ * Despite not being used, the microcontroller still requires some
+ * initialization, and it needs to be notified in the event the AP
+ * crashes. The IPA embedded microcontroller represents another IPA
+ * execution environment (in addition to the AP subsystem and
+ * modem).
+ */
+
+/* Supports hardware interface version 0x2000 */
+
+#define IPA_RAM_UC_SMEM_SIZE 128 /* Size of shared memory area */
+
+/* Delay to allow a the microcontroller to save state when crashing */
+#define IPA_SEND_DELAY 100 /* microseconds */
+
+/*
+ * The IPA has an embedded microcontroller that is capable of doing
+ * more general-purpose processing, for example for handling certain
+ * exceptional conditions. When it has completed its boot sequence
+ * it signals the AP with an interrupt. At this time we don't use
+ * any of the microcontroller capabilities, but we do handle the
+ * "ready" interrupt. We also notify it (by sending it a special
+ * command) in the event of a crash.
+ *
+ * A 128 byte block of structured memory within the IPA SRAM is used
+ * to communicate between the AP and the microcontroller embedded in
+ * the IPA.
+ *
+ * To send a command to the microcontroller, the AP fills in the
+ * command opcode and command parameter fields in this area, then
+ * writes a register to signal to the microcontroller the command is
+ * available. When the microcontroller has executed the command, it
+ * writes response data to this shared area, then issues a response
+ * interrupt (micrcontroller IRQ 1) to the AP. The response
+ * includes a "response operation" that indicates the completion,
+ * along with a "response parameter" which encodes the original
+ * command and the command's status (result).
+ *
+ * The shared area is also used to communicate events asynchronously
+ * from the microcontroller to the AP. Events are signaled using
+ * the event interrupt (micrcontroller IRQ 0). The microcontroller
+ * fills in an "event operation" and "event parameter" before
+ * triggering the interrupt.
+ *
+ * Some additional information is also found in this shared area,
+ * but is currently unused by the IPA driver.
+ *
+ * All other space in the shared area is reserved, and must not be
+ * read or written by the AP.
+ */
+
+/** struct ipa_uc_shared_area - AP/microcontroller shared memory area
+ *
+ * @command: command code (AP->microcontroller)
+ * @command_param: low 32 bits of command parameter (AP->microcontroller)
+ * @command_param_hi: high 32 bits of command parameter (AP->microcontroller)
+ *
+ * @response: response code (microcontroller->AP)
+ * @response_param: response parameter (microcontroller->AP)
+ *
+ * @event: event code (microcontroller->AP)
+ * @event_param: event parameter (microcontroller->AP)
+ *
+ * @first_error_address: address of first error-source on SNOC
+ * @hw_state: state of hardware (including error type information)
+ * @warning_counter: counter of non-fatal hardware errors
+ * @interface_version: hardware-reported interface version
+ */
+struct ipa_uc_shared_area {
+ u32 command : 8; /* enum ipa_uc_command */
+ /* 3 reserved bytes */
+ u32 command_param;
+ u32 command_param_hi;
+
+ u32 response : 8; /* enum ipa_uc_response */
+ /* 3 reserved bytes */
+ u32 response_param;
+
+ u32 event : 8; /* enum ipa_uc_event */
+ /* 3 reserved bytes */
+ u32 event_param;
+
+ u32 first_error_address;
+ u32 hw_state : 8,
+ warning_counter : 8,
+ reserved : 16;
+ u32 interface_version : 16;
+ /* 2 reserved bytes */
+};
+
+/** struct ipa_uc_ctx - IPA microcontroller context
+ *
+ * @uc_loaded: whether microcontroller has been loaded
+ * @shared: pointer to AP/microcontroller shared memory area
+ */
+struct ipa_uc_ctx {
+ bool uc_loaded;
+ struct ipa_uc_shared_area *shared;
+} ipa_uc_ctx;
+
+/*
+ * Microcontroller event codes, error codes, commands, and responses
+ * to commands all encode both a "code" and a "feature" in their
+ * 8-bit numeric value. The top 3 bits represent the feature, and
+ * the bottom 5 bits represent the code. A "common" feature uses
+ * feature code 0, and at this time we only deal with common
+ * features. Because of this we can just ignore the feature bits
+ * and define the values of symbols in the following enumerated
+ * types by just their code values.
+ */
+
+/** enum ipa_uc_event - common cpu events (microcontroller->AP)
+ *
+ * @IPA_UC_EVENT_NO_OP: no event present
+ * @IPA_UC_EVENT_ERROR: system error has been detected
+ * @IPA_UC_EVENT_LOG_INFO: logging information available
+ */
+enum ipa_uc_event {
+ IPA_UC_EVENT_NO_OP = 0,
+ IPA_UC_EVENT_ERROR = 1,
+ IPA_UC_EVENT_LOG_INFO = 2,
+};
+
+/** enum ipa_uc_error - common error types (microcontroller->AP)
+ *
+ * @IPA_UC_ERROR_NONE: no error
+ * @IPA_UC_ERROR_INVALID_DOORBELL: invalid data read from doorbell
+ * @IPA_UC_ERROR_DMA: unexpected DMA error
+ * @IPA_UC_ERROR_FATAL_SYSTEM: microcontroller has crashed and requires reset
+ * @IPA_UC_ERROR_INVALID_OPCODE: invalid opcode sent
+ * @IPA_UC_ERROR_INVALID_PARAMS: invalid params for the requested command
+ * @IPA_UC_ERROR_CONS_DISABLE_CMD_GSI_STOP: consumer endpoint stop failure
+ * @IPA_UC_ERROR_PROD_DISABLE_CMD_GSI_STOP: producer endpoint stop failure
+ * @IPA_UC_ERROR_CH_NOT_EMPTY: micrcontroller GSI channel is not empty
+ */
+enum ipa_uc_error {
+ IPA_UC_ERROR_NONE = 0,
+ IPA_UC_ERROR_INVALID_DOORBELL = 1,
+ IPA_UC_ERROR_DMA = 2,
+ IPA_UC_ERROR_FATAL_SYSTEM = 3,
+ IPA_UC_ERROR_INVALID_OPCODE = 4,
+ IPA_UC_ERROR_INVALID_PARAMS = 5,
+ IPA_UC_ERROR_CONS_DISABLE_CMD_GSI_STOP = 6,
+ IPA_UC_ERROR_PROD_DISABLE_CMD_GSI_STOP = 7,
+ IPA_UC_ERROR_CH_NOT_EMPTY = 8,
+};
+
+/** enum ipa_uc_command - commands from the AP to the microcontroller
+ *
+ * @IPA_UC_COMMAND_NO_OP: no operation
+ * @IPA_UC_COMMAND_UPDATE_FLAGS: request to re-read configuration flags
+ * @IPA_UC_COMMAND_DEBUG_RUN_TEST: request to run hardware test
+ * @IPA_UC_COMMAND_DEBUG_GET_INFO: request to read internal debug information
+ * @IPA_UC_COMMAND_ERR_FATAL: AP system crash notification
+ * @IPA_UC_COMMAND_CLK_GATE: request hardware to enter clock gated state
+ * @IPA_UC_COMMAND_CLK_UNGATE: request hardware to enter clock ungated state
+ * @IPA_UC_COMMAND_MEMCPY: request hardware to perform memcpy
+ * @IPA_UC_COMMAND_RESET_PIPE: request endpoint reset
+ * @IPA_UC_COMMAND_REG_WRITE: request a register be written
+ * @IPA_UC_COMMAND_GSI_CH_EMPTY: request to determine whether channel is empty
+ */
+enum ipa_uc_command {
+ IPA_UC_COMMAND_NO_OP = 0,
+ IPA_UC_COMMAND_UPDATE_FLAGS = 1,
+ IPA_UC_COMMAND_DEBUG_RUN_TEST = 2,
+ IPA_UC_COMMAND_DEBUG_GET_INFO = 3,
+ IPA_UC_COMMAND_ERR_FATAL = 4,
+ IPA_UC_COMMAND_CLK_GATE = 5,
+ IPA_UC_COMMAND_CLK_UNGATE = 6,
+ IPA_UC_COMMAND_MEMCPY = 7,
+ IPA_UC_COMMAND_RESET_PIPE = 8,
+ IPA_UC_COMMAND_REG_WRITE = 9,
+ IPA_UC_COMMAND_GSI_CH_EMPTY = 10,
+};
+
+/** enum ipa_uc_response - common hardware response codes
+ *
+ * @IPA_UC_RESPONSE_NO_OP: no operation
+ * @IPA_UC_RESPONSE_INIT_COMPLETED: microcontroller ready
+ * @IPA_UC_RESPONSE_CMD_COMPLETED: AP-issued command has completed
+ * @IPA_UC_RESPONSE_DEBUG_GET_INFO: get debug info
+ */
+enum ipa_uc_response {
+ IPA_UC_RESPONSE_NO_OP = 0,
+ IPA_UC_RESPONSE_INIT_COMPLETED = 1,
+ IPA_UC_RESPONSE_CMD_COMPLETED = 2,
+ IPA_UC_RESPONSE_DEBUG_GET_INFO = 3,
+};
+
+/** union ipa_uc_event_data - microcontroller->AP event data
+ *
+ * @error_type: ipa_uc_error error type value
+ * @raw32b: 32-bit register value (used when reading)
+ */
+union ipa_uc_event_data {
+ u8 error_type; /* enum ipa_uc_error */
+ u32 raw32b;
+} __packed;
+
+/** union ipa_uc_response_data - response to AP command
+ *
+ * @command: the AP issued command this is responding to
+ * @status: 0 for success indication, otherwise failure
+ * @raw32b: 32-bit register value (used when reading)
+ */
+union ipa_uc_response_data {
+ struct ipa_uc_response_param {
+ u8 command; /* enum ipa_uc_command */
+ u8 status; /* enum ipa_uc_error */
+ } params;
+ u32 raw32b;
+} __packed;
+
+/** ipa_uc_loaded() - tell whether the microcontroller has been loaded
+ *
+ * Returns true if the microcontroller is loaded, false otherwise
+ */
+bool ipa_uc_loaded(void)
+{
+ return ipa_uc_ctx.uc_loaded;
+}
+
+static void
+ipa_uc_event_handler(enum ipa_irq_type interrupt, u32 interrupt_data)
+{
+ struct ipa_uc_shared_area *shared = ipa_uc_ctx.shared;
+ union ipa_uc_event_data event_param;
+ u8 event;
+
+ event = shared->event;
+ event_param.raw32b = shared->event_param;
+
+ /* General handling */
+ if (event == IPA_UC_EVENT_ERROR) {
+ ipa_err("uC error type 0x%02x timestamp 0x%08x\n",
+ event_param.error_type, ipa_read_reg(IPA_TAG_TIMER));
+ ipa_bug();
+ } else {
+ ipa_err("unsupported uC event opcode=%u\n", event);
+ }
+}
+
+static void
+ipa_uc_response_hdlr(enum ipa_irq_type interrupt, u32 interrupt_data)
+{
+ struct ipa_uc_shared_area *shared = ipa_uc_ctx.shared;
+ union ipa_uc_response_data response_data;
+ u8 response;
+
+ response = shared->response;
+
+ /* An INIT_COMPLETED response message is sent to the AP by
+ * the microcontroller when it is operational. Other than
+ * this, the AP should only receive responses from the
+ * microntroller when it has sent it a request message.
+ */
+ if (response == IPA_UC_RESPONSE_INIT_COMPLETED) {
+ /* The proxy vote is held until uC is loaded to ensure that
+ * IPA_HW_2_CPU_RESPONSE_INIT_COMPLETED is received.
+ */
+ ipa_proxy_clk_unvote();
+ ipa_uc_ctx.uc_loaded = true;
+ } else if (response == IPA_UC_RESPONSE_CMD_COMPLETED) {
+ response_data.raw32b = shared->response_param;
+ ipa_err("uC command response code %u status %u\n",
+ response_data.params.command,
+ response_data.params.status);
+ } else {
+ ipa_err("Unsupported uC rsp opcode = %u\n", response);
+ }
+}
+
+/** ipa_uc_init() - Initialize the microcontroller
+ *
+ * Returns pointer to microcontroller context on success, NULL otherwise
+ */
+struct ipa_uc_ctx *ipa_uc_init(phys_addr_t phys_addr)
+{
+ phys_addr += ipa_reg_n_offset(IPA_SRAM_DIRECT_ACCESS_N, 0);
+ ipa_uc_ctx.shared = ioremap(phys_addr, IPA_RAM_UC_SMEM_SIZE);
+ if (!ipa_uc_ctx.shared)
+ return NULL;
+
+ ipa_add_interrupt_handler(IPA_UC_IRQ_0, ipa_uc_event_handler);
+ ipa_add_interrupt_handler(IPA_UC_IRQ_1, ipa_uc_response_hdlr);
+
+ return &ipa_uc_ctx;
+}
+
+/* Send a command to the microcontroller */
+static void send_uc_command(u32 command, u32 command_param)
+{
+ struct ipa_uc_shared_area *shared = ipa_uc_ctx.shared;
+
+ shared->command = command;
+ shared->command_param = command_param;
+ shared->command_param_hi = 0;
+ shared->response = 0;
+ shared->response_param = 0;
+
+ wmb(); /* ensure write to shared memory is done before triggering uc */
+
+ ipa_write_reg_n(IPA_IRQ_EE_UC_N, IPA_EE_AP, 0x1);
+}
+
+void ipa_uc_panic_notifier(void)
+{
+ if (!ipa_uc_ctx.uc_loaded)
+ return;
+
+ if (!ipa_client_add_additional())
+ return;
+
+ send_uc_command(IPA_UC_COMMAND_ERR_FATAL, 0);
+
+ /* give uc enough time to save state */
+ udelay(IPA_SEND_DELAY);
+
+ ipa_client_remove();
+}
--
2.17.1
^ permalink raw reply related
* [RFC PATCH 04/12] soc: qcom: ipa: immediate commands
From: Alex Elder @ 2018-11-07 0:32 UTC (permalink / raw)
To: davem, arnd, bjorn.andersson, ilias.apalodimas
Cc: netdev, devicetree, linux-arm-msm, linux-soc, linux-arm-kernel,
linux-kernel, syadagir, mjavid, robh+dt, mark.rutland
In-Reply-To: <20181107003250.5832-1-elder@linaro.org>
This patch contains (mostly) code implementing "immediate commands."
(The source files are still named "ipahal" for historical reasons.)
One channel (APPS CMD_PROD) is used for sending commands *to* the
IPA itself, rather than passing data through it. These immediate
commands are issued to the IPA using the normal GSI queueing
mechanism. And each command's completion is handled using the
normal GSI transfer completion mechanisms.
In addition to immediate commands, the "IPA HAL" includes code for
interpreting status packets that are supplied to the IPA on consumer
channels.
Signed-off-by: Alex Elder <elder@linaro.org>
---
drivers/net/ipa/ipahal.c | 541 +++++++++++++++++++++++++++++++++++++++
drivers/net/ipa/ipahal.h | 253 ++++++++++++++++++
2 files changed, 794 insertions(+)
create mode 100644 drivers/net/ipa/ipahal.c
create mode 100644 drivers/net/ipa/ipahal.h
diff --git a/drivers/net/ipa/ipahal.c b/drivers/net/ipa/ipahal.c
new file mode 100644
index 000000000000..de00bcd54d4f
--- /dev/null
+++ b/drivers/net/ipa/ipahal.c
@@ -0,0 +1,541 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2016-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <asm/unaligned.h>
+
+#include "ipahal.h"
+#include "ipa_i.h" /* ipa_err() */
+#include "ipa_dma.h"
+
+/**
+ * DOC: IPA Immediate Commands
+ *
+ * The APPS_CMD_PROD channel is used to issue immediate commands to
+ * the IPA. An immediate command is generally used to request the
+ * IPA do something other than data transfer.
+ *
+ * An immediate command is represented by a GSI transfer element.
+ * Each immediate command has a well-defined format, with a known
+ * length. The transfer element's length field can therefore be
+ * used to hold a command's opcode. The "payload" of an immediate
+ * command contains additional information required for the command.
+ * It resides in DRAM and is referred to using the DMA memory data
+ * pointer (the same one used to refer to the data in a "normal"
+ * transfer).
+ *
+ * Immediate commands are issued to the IPA through the APPS_CMD_PROD
+ * channel using the normal GSI queueing mechanism. And each command's
+ * completion is handled using the normal GSI transfer completion
+ * mechanisms.
+ */
+
+/**
+ * struct ipahal_context - HAL global context data
+ * @empty_fltrt_tbl: Empty table to be used for table initialization
+ */
+static struct ipahal_context {
+ struct ipa_dma_mem empty_fltrt_tbl;
+} ipahal_ctx_struct;
+static struct ipahal_context *ipahal_ctx = &ipahal_ctx_struct;
+
+/* enum ipa_pipeline_clear_option - Values for pipeline clear waiting options
+ * @IPAHAL_HPS_CLEAR: Wait for HPS clear. All queues except high priority queue
+ * shall not be serviced until HPS is clear of packets or immediate commands.
+ * The high priority Rx queue / Q6ZIP group shall still be serviced normally.
+ *
+ * @IPAHAL_SRC_GRP_CLEAR: Wait for originating source group to be clear
+ * (for no packet contexts allocated to the originating source group).
+ * The source group / Rx queue shall not be serviced until all previously
+ * allocated packet contexts are released. All other source groups/queues shall
+ * be serviced normally.
+ *
+ * @IPAHAL_FULL_PIPELINE_CLEAR: Wait for full pipeline to be clear.
+ * All groups / Rx queues shall not be serviced until IPA pipeline is fully
+ * clear. This should be used for debug only.
+ *
+ * The values assigned to these are assumed by the REGISTER_WRITE
+ * (struct ipa_imm_cmd_hw_register_write) and the DMA_SHARED_MEM
+ * (struct ipa_imm_cmd_hw_dma_shared_mem) immediate commands for
+ * IPA version 3 hardware. They are also used to modify the opcode
+ * used to implement these commands for IPA version 4 hardware.
+ */
+enum ipahal_pipeline_clear_option {
+ IPAHAL_HPS_CLEAR = 0,
+ IPAHAL_SRC_GRP_CLEAR = 1,
+ IPAHAL_FULL_PIPELINE_CLEAR = 2,
+};
+
+/* Immediate commands H/W structures */
+
+/* struct ipa_imm_cmd_hw_ip_fltrt_init - IP_V*_FILTER_INIT/IP_V*_ROUTING_INIT
+ * command payload in H/W format.
+ * Inits IPv4/v6 routing or filter block.
+ * @hash_rules_addr: Addr in system mem where hashable flt/rt rules starts
+ * @hash_rules_size: Size in bytes of the hashable tbl to cpy to local mem
+ * @hash_local_addr: Addr in shared mem where hashable flt/rt tbl should
+ * be copied to
+ * @nhash_rules_size: Size in bytes of the non-hashable tbl to cpy to local mem
+ * @nhash_local_addr: Addr in shared mem where non-hashable flt/rt tbl should
+ * be copied to
+ * @rsvd: reserved
+ * @nhash_rules_addr: Addr in sys mem where non-hashable flt/rt tbl starts
+ */
+struct ipa_imm_cmd_hw_ip_fltrt_init {
+ u64 hash_rules_addr;
+ u64 hash_rules_size : 12,
+ hash_local_addr : 16,
+ nhash_rules_size : 12,
+ nhash_local_addr : 16,
+ rsvd : 8;
+ u64 nhash_rules_addr;
+};
+
+/* struct ipa_imm_cmd_hw_hdr_init_local - HDR_INIT_LOCAL command payload
+ * in H/W format.
+ * Inits hdr table within local mem with the hdrs and their length.
+ * @hdr_table_addr: Word address in sys mem where the table starts (SRC)
+ * @size_hdr_table: Size of the above (in bytes)
+ * @hdr_addr: header address in IPA sram (used as DST for memory copy)
+ * @rsvd: reserved
+ */
+struct ipa_imm_cmd_hw_hdr_init_local {
+ u64 hdr_table_addr;
+ u32 size_hdr_table : 12,
+ hdr_addr : 16,
+ rsvd : 4;
+};
+
+/* struct ipa_imm_cmd_hw_dma_shared_mem - DMA_SHARED_MEM command payload
+ * in H/W format.
+ * Perform mem copy into or out of the SW area of IPA local mem
+ * @sw_rsvd: Ignored by H/W. My be used by S/W
+ * @size: Size in bytes of data to copy. Expected size is up to 2K bytes
+ * @local_addr: Address in IPA local memory
+ * @direction: Read or write?
+ * 0: IPA write, Write to local address from system address
+ * 1: IPA read, Read from local address to system address
+ * @skip_pipeline_clear: 0 to wait until IPA pipeline is clear. 1 don't wait
+ * @pipeline_clear_options: options for pipeline to clear
+ * 0: HPS - no pkt inside HPS (not grp specific)
+ * 1: source group - The immediate cmd src grp does npt use any pkt ctxs
+ * 2: Wait until no pkt reside inside IPA pipeline
+ * 3: reserved
+ * @rsvd: reserved - should be set to zero
+ * @system_addr: Address in system memory
+ */
+struct ipa_imm_cmd_hw_dma_shared_mem {
+ u16 sw_rsvd;
+ u16 size;
+ u16 local_addr;
+ u16 direction : 1,
+ skip_pipeline_clear : 1,
+ pipeline_clear_options : 2,
+ rsvd : 12;
+ u64 system_addr;
+};
+
+/* struct ipa_imm_cmd_hw_dma_task_32b_addr -
+ * IPA_DMA_TASK_32B_ADDR command payload in H/W format.
+ * Used by clients using 32bit addresses. Used to perform DMA operation on
+ * multiple descriptors.
+ * The Opcode is dynamic, where it holds the number of buffer to process
+ * @sw_rsvd: Ignored by H/W. My be used by S/W
+ * @cmplt: Complete flag: When asserted IPA will interrupt SW when the entire
+ * DMA related data was completely xfered to its destination.
+ * @eof: Enf Of Frame flag: When asserted IPA will assert the EOT to the
+ * dest client. This is used used for aggr sequence
+ * @flsh: Flush flag: When asserted, pkt will go through the IPA blocks but
+ * will not be xfered to dest client but rather will be discarded
+ * @lock: Lock endpoint flag: When asserted, IPA will stop processing
+ * descriptors from other EPs in the same src grp (RX queue)
+ * @unlock: Unlock endpoint flag: When asserted, IPA will stop exclusively
+ * servicing current EP out of the src EPs of the grp (RX queue)
+ * @size1: Size of buffer1 data
+ * @addr1: Pointer to buffer1 data
+ * @packet_size: Total packet size. If a pkt send using multiple DMA_TASKs,
+ * only the first one needs to have this field set. It will be ignored
+ * in subsequent DMA_TASKs until the packet ends (EOT). First DMA_TASK
+ * must contain this field (2 or more buffers) or EOT.
+ */
+struct ipa_imm_cmd_hw_dma_task_32b_addr {
+ u16 sw_rsvd : 11,
+ cmplt : 1,
+ eof : 1,
+ flsh : 1,
+ lock : 1,
+ unlock : 1;
+ u16 size1;
+ u32 addr1;
+ u16 packet_size;
+ u16 rsvd1;
+ u32 rsvd2;
+};
+
+/* IPA Status packet H/W structures and info */
+
+/* struct ipa_status_pkt_hw - IPA status packet payload in H/W format.
+ * This structure describes the status packet H/W structure for the
+ * following statuses: IPA_STATUS_PACKET, IPA_STATUS_DROPPED_PACKET,
+ * IPA_STATUS_SUSPENDED_PACKET.
+ * Other statuses types has different status packet structure.
+ * @status_opcode: The Type of the status (Opcode).
+ * @exception: (not bitmask) - the first exception that took place.
+ * In case of exception, src endp and pkt len are always valid.
+ * @status_mask: Bit mask specifying on which H/W blocks the pkt was processed.
+ * @pkt_len: Pkt payload len including hdr, include retained hdr if used. Does
+ * not include padding or checksum trailer len.
+ * @endp_src_idx: Source end point index.
+ * @rsvd1: reserved
+ * @endp_dest_idx: Destination end point index.
+ * Not valid in case of exception
+ * @rsvd2: reserved
+ * @metadata: meta data value used by packet
+ * @flt_local: Filter table location flag: Does matching flt rule belongs to
+ * flt tbl that resides in lcl memory? (if not, then system mem)
+ * @flt_hash: Filter hash hit flag: Does matching flt rule was in hash tbl?
+ * @flt_global: Global filter rule flag: Does matching flt rule belongs to
+ * the global flt tbl? (if not, then the per endp tables)
+ * @flt_ret_hdr: Retain header in filter rule flag: Does matching flt rule
+ * specifies to retain header?
+ * @flt_rule_id: The ID of the matching filter rule. This info can be combined
+ * with endp_src_idx to locate the exact rule. ID=0x3ff reserved to specify
+ * flt miss. In case of miss, all flt info to be ignored
+ * @rt_local: Route table location flag: Does matching rt rule belongs to
+ * rt tbl that resides in lcl memory? (if not, then system mem)
+ * @rt_hash: Route hash hit flag: Does matching rt rule was in hash tbl?
+ * @ucp: UC Processing flag.
+ * @rt_tbl_idx: Index of rt tbl that contains the rule on which was a match
+ * @rt_rule_id: The ID of the matching rt rule. This info can be combined
+ * with rt_tbl_idx to locate the exact rule. ID=0x3ff reserved to specify
+ * rt miss. In case of miss, all rt info to be ignored
+ * @nat_hit: NAT hit flag: Was their NAT hit?
+ * @nat_entry_idx: Index of the NAT entry used of NAT processing
+ * @nat_type: Defines the type of the NAT operation (ignored for now)
+ * @tag_info: S/W defined value provided via immediate command
+ * @seq_num: Per source endp unique packet sequence number
+ * @time_of_day_ctr: running counter from IPA clock
+ * @hdr_local: Header table location flag: In header insertion, was the header
+ * taken from the table resides in local memory? (If no, then system mem)
+ * @hdr_offset: Offset of used header in the header table
+ * @frag_hit: Frag hit flag: Was their frag rule hit in H/W frag table?
+ * @frag_rule: Frag rule index in H/W frag table in case of frag hit
+ * @hw_specific: H/W specific reserved value
+ */
+#define IPA_RULE_ID_BITS 10 /* See ipahal_is_rule_miss_id() */
+struct ipa_pkt_status_hw {
+ u8 status_opcode;
+ u8 exception;
+ u16 status_mask;
+ u16 pkt_len;
+ u8 endp_src_idx : 5,
+ rsvd1 : 3;
+ u8 endp_dest_idx : 5,
+ rsvd2 : 3;
+ u32 metadata;
+ u16 flt_local : 1,
+ flt_hash : 1,
+ flt_global : 1,
+ flt_ret_hdr : 1,
+ flt_rule_id : IPA_RULE_ID_BITS,
+ rt_local : 1,
+ rt_hash : 1;
+ u16 ucp : 1,
+ rt_tbl_idx : 5,
+ rt_rule_id : IPA_RULE_ID_BITS;
+ u64 nat_hit : 1,
+ nat_entry_idx : 13,
+ nat_type : 2,
+ tag_info : 48;
+ u32 seq_num : 8,
+ time_of_day_ctr : 24;
+ u16 hdr_local : 1,
+ hdr_offset : 10,
+ frag_hit : 1,
+ frag_rule : 4;
+ u16 hw_specific;
+};
+
+void *ipahal_dma_shared_mem_write_pyld(struct ipa_dma_mem *mem, u32 offset)
+{
+ struct ipa_imm_cmd_hw_dma_shared_mem *data;
+
+ ipa_assert(mem->size < 1 << 16); /* size is 16 bits wide */
+ ipa_assert(offset < 1 << 16); /* local_addr is 16 bits wide */
+
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return NULL;
+
+ data->size = mem->size;
+ data->local_addr = offset;
+ data->direction = 0; /* 0 = write to IPA; 1 = read from IPA */
+ data->skip_pipeline_clear = 0;
+ data->pipeline_clear_options = IPAHAL_HPS_CLEAR;
+ data->system_addr = mem->phys;
+
+ return data;
+}
+
+void *ipahal_hdr_init_local_pyld(struct ipa_dma_mem *mem, u32 offset)
+{
+ struct ipa_imm_cmd_hw_hdr_init_local *data;
+
+ ipa_assert(mem->size < 1 << 12); /* size_hdr_table is 12 bits wide */
+ ipa_assert(offset < 1 << 16); /* hdr_addr is 16 bits wide */
+
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return NULL;
+
+ data->hdr_table_addr = mem->phys;
+ data->size_hdr_table = mem->size;
+ data->hdr_addr = offset;
+
+ return data;
+}
+
+static void *fltrt_init_common(struct ipa_dma_mem *mem, u32 hash_offset,
+ u32 nhash_offset)
+{
+ struct ipa_imm_cmd_hw_ip_fltrt_init *data;
+
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return NULL;
+
+ data->hash_rules_addr = (u64)mem->phys;
+ data->hash_rules_size = (u32)mem->size;
+ data->hash_local_addr = hash_offset;
+ data->nhash_rules_addr = (u64)mem->phys;
+ data->nhash_rules_size = (u32)mem->size;
+ data->nhash_local_addr = nhash_offset;
+
+ return data;
+}
+
+void *ipahal_ip_v4_routing_init_pyld(struct ipa_dma_mem *mem, u32 hash_offset,
+ u32 nhash_offset)
+{
+ return fltrt_init_common(mem, hash_offset, nhash_offset);
+}
+
+void *ipahal_ip_v6_routing_init_pyld(struct ipa_dma_mem *mem, u32 hash_offset,
+ u32 nhash_offset)
+{
+ return fltrt_init_common(mem, hash_offset, nhash_offset);
+}
+
+void *ipahal_ip_v4_filter_init_pyld(struct ipa_dma_mem *mem, u32 hash_offset,
+ u32 nhash_offset)
+{
+ return fltrt_init_common(mem, hash_offset, nhash_offset);
+}
+
+void *ipahal_ip_v6_filter_init_pyld(struct ipa_dma_mem *mem, u32 hash_offset,
+ u32 nhash_offset)
+{
+ return fltrt_init_common(mem, hash_offset, nhash_offset);
+}
+
+void *ipahal_dma_task_32b_addr_pyld(struct ipa_dma_mem *mem)
+{
+ struct ipa_imm_cmd_hw_dma_task_32b_addr *data;
+
+ /* size1 and packet_size are both 16 bits wide */
+ ipa_assert(mem->size < 1 << 16);
+
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return NULL;
+
+ data->cmplt = 0;
+ data->eof = 0;
+ data->flsh = 1;
+ data->lock = 0;
+ data->unlock = 0;
+ data->size1 = mem->size;
+ data->addr1 = mem->phys;
+ data->packet_size = mem->size;
+
+ return data;
+}
+
+void ipahal_payload_free(void *payload)
+{
+ kfree(payload);
+}
+
+/* IPA Packet Status Logic */
+
+/* Maps an exception type returned in a ipa_pkt_status_hw structure
+ * to the ipahal_pkt_status_exception value that represents it in
+ * the exception field of a ipahal_pkt_status structure. Returns
+ * IPAHAL_PKT_STATUS_EXCEPTION_MAX for an unrecognized value.
+ */
+static enum ipahal_pkt_status_exception
+exception_map(u8 exception, bool is_ipv6)
+{
+ switch (exception) {
+ case 0x00: return IPAHAL_PKT_STATUS_EXCEPTION_NONE;
+ case 0x01: return IPAHAL_PKT_STATUS_EXCEPTION_DEAGGR;
+ case 0x04: return IPAHAL_PKT_STATUS_EXCEPTION_IPTYPE;
+ case 0x08: return IPAHAL_PKT_STATUS_EXCEPTION_PACKET_LENGTH;
+ case 0x10: return IPAHAL_PKT_STATUS_EXCEPTION_FRAG_RULE_MISS;
+ case 0x20: return IPAHAL_PKT_STATUS_EXCEPTION_SW_FILT;
+ case 0x40: return is_ipv6 ? IPAHAL_PKT_STATUS_EXCEPTION_IPV6CT
+ : IPAHAL_PKT_STATUS_EXCEPTION_NAT;
+ default: return IPAHAL_PKT_STATUS_EXCEPTION_MAX;
+ }
+}
+
+/* ipahal_pkt_status_get_size() - Get H/W size of packet status */
+u32 ipahal_pkt_status_get_size(void)
+{
+ return sizeof(struct ipa_pkt_status_hw);
+}
+
+/* ipahal_pkt_status_parse() - Parse Packet Status payload to abstracted form
+ * @unparsed_status: Pointer to H/W format of the packet status as read from H/W
+ * @status: Pointer to pre-allocated buffer where the parsed info will be stored
+ */
+void ipahal_pkt_status_parse(const void *unparsed_status,
+ struct ipahal_pkt_status *status)
+{
+ const struct ipa_pkt_status_hw *hw_status = unparsed_status;
+ bool is_ipv6;
+
+ status->status_opcode =
+ (enum ipahal_pkt_status_opcode)hw_status->status_opcode;
+ is_ipv6 = hw_status->status_mask & BIT(7) ? false : true;
+ /* If hardware status values change we may have to re-map this */
+ status->status_mask =
+ (enum ipahal_pkt_status_mask)hw_status->status_mask;
+ status->exception = exception_map(hw_status->exception, is_ipv6);
+ status->pkt_len = hw_status->pkt_len;
+ status->endp_src_idx = hw_status->endp_src_idx;
+ status->endp_dest_idx = hw_status->endp_dest_idx;
+ status->metadata = hw_status->metadata;
+ status->rt_miss = ipahal_is_rule_miss_id(hw_status->rt_rule_id);
+}
+
+int ipahal_init(void)
+{
+ struct ipa_dma_mem *mem = &ipahal_ctx->empty_fltrt_tbl;
+
+ /* Set up an empty filter/route table entry in system
+ * memory. This will be used, for example, to delete a
+ * route safely.
+ */
+ if (ipa_dma_alloc(mem, IPA_HW_TBL_WIDTH, GFP_KERNEL)) {
+ ipa_err("error allocating empty filter/route table\n");
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+void ipahal_exit(void)
+{
+ ipa_dma_free(&ipahal_ctx->empty_fltrt_tbl);
+}
+
+/* Does the given rule ID represent a routing or filter rule miss?
+ * A rule miss is indicated as an all-1's value in the rt_rule_id
+ * or flt_rule_id field of the ipahal_pkt_status structure.
+ */
+bool ipahal_is_rule_miss_id(u32 id)
+{
+ BUILD_BUG_ON(IPA_RULE_ID_BITS < 2);
+
+ return id == (1U << IPA_RULE_ID_BITS) - 1;
+}
+
+/**
+ * ipahal_rt_generate_empty_img() - Generate empty route table header
+ * @route_count: Number of table entries
+ * @mem: DMA memory object representing the header structure
+ *
+ * Allocates and fills an "empty" route table header having the given
+ * number of entries. Each entry in the table contains the DMA address
+ * of a routing entry.
+ *
+ * This function initializes all entries to point at the preallocated
+ * empty routing entry in system RAM.
+ *
+ * Return: 0 if successful, or a negative error code otherwise
+ */
+int ipahal_rt_generate_empty_img(u32 route_count, struct ipa_dma_mem *mem)
+{
+ u64 addr;
+ int i;
+
+ BUILD_BUG_ON(!IPA_HW_TBL_HDR_WIDTH);
+
+ if (ipa_dma_alloc(mem, route_count * IPA_HW_TBL_HDR_WIDTH, GFP_KERNEL))
+ return -ENOMEM;
+
+ addr = (u64)ipahal_ctx->empty_fltrt_tbl.phys;
+ for (i = 0; i < route_count; i++)
+ put_unaligned(addr, mem->virt + i * IPA_HW_TBL_HDR_WIDTH);
+
+ return 0;
+}
+
+/**
+ * ipahal_flt_generate_empty_img() - Generate empty filter table header
+ * @filter_bitmap: Bitmap representing which endpoints support filtering
+ * @mem: DMA memory object representing the header structure
+ *
+ * Allocates and fills an "empty" filter table header based on the
+ * given filter bitmap.
+ *
+ * The first slot in a filter table header is a 64-bit bitmap whose
+ * set bits define which endpoints support filtering. Following
+ * this, each set bit in the mask has the DMA address of the filter
+ * used for the corresponding endpoint.
+ *
+ * This function initializes all endpoints that support filtering to
+ * point at the preallocated empty filter in system RAM.
+ *
+ * Note: the (software) bitmap here uses bit 0 to represent
+ * endpoint 0, bit 1 for endpoint 1, and so on. This is different
+ * from the hardware (which uses bit 1 to represent filter 0, etc.).
+ *
+ * Return: 0 if successful, or a negative error code
+ */
+int ipahal_flt_generate_empty_img(u64 filter_bitmap, struct ipa_dma_mem *mem)
+{
+ u32 filter_count = hweight32(filter_bitmap) + 1;
+ u64 addr;
+ int i;
+
+ ipa_assert(filter_bitmap);
+
+ if (ipa_dma_alloc(mem, filter_count * IPA_HW_TBL_HDR_WIDTH, GFP_KERNEL))
+ return -ENOMEM;
+
+ /* Save the endpoint bitmap in the first slot of the table.
+ * Convert it from software to hardware representation by
+ * shifting it left one position.
+ * XXX Does bit position 0 represent global? At IPA3, global
+ * XXX configuration is possible but not used.
+ */
+ put_unaligned(filter_bitmap << 1, mem->virt);
+
+ /* Point every entry in the table at the empty filter */
+ addr = (u64)ipahal_ctx->empty_fltrt_tbl.phys;
+ for (i = 1; i < filter_count; i++)
+ put_unaligned(addr, mem->virt + i * IPA_HW_TBL_HDR_WIDTH);
+
+ return 0;
+}
+
+void ipahal_free_empty_img(struct ipa_dma_mem *mem)
+{
+ ipa_dma_free(mem);
+}
diff --git a/drivers/net/ipa/ipahal.h b/drivers/net/ipa/ipahal.h
new file mode 100644
index 000000000000..940254940d90
--- /dev/null
+++ b/drivers/net/ipa/ipahal.h
@@ -0,0 +1,253 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright (c) 2016-2018, The Linux Foundation. All rights reserved.
+ * Copyright (C) 2018 Linaro Ltd.
+ */
+#ifndef _IPAHAL_H_
+#define _IPAHAL_H_
+
+#include <linux/types.h>
+
+#include "ipa_dma.h"
+
+/* The IPA implements offloaded packet filtering and routing
+ * capabilities. This is managed by programming IPA-resident
+ * tables of rules that define the processing that should be
+ * performed by the IPA and the conditions under which they
+ * should be applied. Aspects of these rules are constrained
+ * by things like table entry sizes and alignment requirements;
+ * all of these are in units of bytes. These definitions are
+ * subject to some constraints:
+ * - IPA_HW_TBL_WIDTH must be non-zero
+ * - IPA_HW_TBL_SYSADDR_ALIGN must be a non-zero power of 2
+ * - IPA_HW_TBL_HDR_WIDTH must be non-zero
+ *
+ * Values could differ for different versions of IPA hardware.
+ * These values are for v3.5.1, found in the SDM845.
+ */
+#define IPA_HW_TBL_WIDTH 8
+#define IPA_HW_TBL_SYSADDR_ALIGN 128
+#define IPA_HW_TBL_HDR_WIDTH 8
+
+/**
+ * ipahal_dma_shared_mem_write_pyld() - Write to shared memory command payload
+ *
+ * Return a pointer to the payload for a DMA shared memory write immediate
+ * command, or null if one can't be allocated. Result is dynamically
+ * allocated, and caller must ensure it gets released by providing it to
+ * ipahal_destroy_imm_cmd() when it is no longer needed.
+ *
+ * Return: Pointer to the immediate command payload, or NULL
+ */
+void *ipahal_dma_shared_mem_write_pyld(struct ipa_dma_mem *mem, u32 offset);
+
+/**
+ * ipahal_hdr_init_local_pyld() - Header initialization command payload
+ * mem: DMA buffer containing data for initialization
+ * offset: Where in location IPA local memory to write
+ *
+ * Return a pointer to the payload for a header init local immediate
+ * command, or null if one can't be allocated. Caller must ensure result
+ * gets released by providing it to ipahal_destroy_imm_cmd().
+ *
+ * Return: Pointer to the immediate command payload, or NULL
+ */
+void *ipahal_hdr_init_local_pyld(struct ipa_dma_mem *mem, u32 offset);
+
+/**
+ * ipahal_ip_v4_routing_init_pyld() - IPv4 routing table initialization payload
+ * mem: The IPv4 routing table data to be written
+ * hash_offset: The location in IPA memory for a hashed routing table
+ * nhash_offset: The location in IPA memory for a non-hashed routing table
+ *
+ * Return a pointer to the payload for an IPv4 routing init immediate
+ * command, or null if one can't be allocated. Caller must ensure result
+ * gets released by providing it to ipahal_destroy_imm_cmd().
+ *
+ * Return: Pointer to the immediate command payload, or NULL
+ */
+void *ipahal_ip_v4_routing_init_pyld(struct ipa_dma_mem *mem,
+ u32 hash_offset, u32 nhash_offset);
+
+/**
+ * ipahal_ip_v6_routing_init_pyld() - IPv6 routing table initialization payload
+ * mem: The IPv6 routing table data to be written
+ * hash_offset: The location in IPA memory for a hashed routing table
+ * nhash_offset: The location in IPA memory for a non-hashed routing table
+ *
+ * Return a pointer to the payload for an IPv4 routing init immediate
+ * command, or null if one can't be allocated. Caller must ensure result
+ * gets released by providing it to ipahal_destroy_imm_cmd().
+ *
+ * Return: Pointer to the immediate command payload, or NULL
+ */
+void *ipahal_ip_v6_routing_init_pyld(struct ipa_dma_mem *mem,
+ u32 hash_offset, u32 nhash_offset);
+
+/**
+ * ipahal_ip_v4_filter_init_pyld() - IPv4 filter table initialization payload
+ * mem: The IPv4 filter table data to be written
+ * hash_offset: The location in IPA memory for a hashed filter table
+ * nhash_offset: The location in IPA memory for a non-hashed filter table
+ *
+ * Return a pointer to the payload for an IPv4 filter init immediate
+ * command, or null if one can't be allocated. Caller must ensure result
+ * gets released by providing it to ipahal_destroy_imm_cmd().
+ *
+ * Return: Pointer to the immediate command payload, or NULL
+ */
+void *ipahal_ip_v4_filter_init_pyld(struct ipa_dma_mem *mem,
+ u32 hash_offset, u32 nhash_offset);
+
+/**
+ * ipahal_ip_v6_filter_init_pyld() - IPv6 filter table initialization payload
+ * mem: The IPv6 filter table data to be written
+ * hash_offset: The location in IPA memory for a hashed filter table
+ * nhash_offset: The location in IPA memory for a non-hashed filter table
+ *
+ * Return a pointer to the payload for an IPv4 filter init immediate
+ * command, or null if one can't be allocated. Caller must ensure result
+ * gets released by providing it to ipahal_destroy_imm_cmd().
+ *
+ * Return: Pointer to the immediate command payload, or NULL
+ */
+void *ipahal_ip_v6_filter_init_pyld(struct ipa_dma_mem *mem,
+ u32 hash_offset, u32 nhash_offset);
+
+/**
+ * ipahal_dma_task_32b_addr_pyld() - 32-bit DMA task command payload
+ * mem: DMA memory involved in the task
+ *
+ * Return a pointer to the payload for DMA task 32-bit address immediate
+ * command, or null if one can't be allocated. Caller must ensure result
+ * gets released by providing it to ipahal_destroy_imm_cmd().
+ */
+void *ipahal_dma_task_32b_addr_pyld(struct ipa_dma_mem *mem);
+
+/**
+ * ipahal_payload_free() - Release an allocated immediate command payload
+ * @payload: Payload to be released
+ */
+void ipahal_payload_free(void *payload);
+
+/**
+ * enum ipahal_pkt_status_opcode - Packet Status Opcode
+ * @IPAHAL_STATUS_OPCODE_PACKET_2ND_PASS: Packet Status generated as part of
+ * IPA second processing pass for a packet (i.e. IPA XLAT processing for
+ * the translated packet).
+ *
+ * The values assigned here are assumed by ipa_pkt_status_parse()
+ * to match values returned in the status_opcode field of a
+ * ipa_pkt_status_hw structure inserted by the IPA in received
+ * buffer.
+ */
+enum ipahal_pkt_status_opcode {
+ IPAHAL_PKT_STATUS_OPCODE_PACKET = 0x01,
+ IPAHAL_PKT_STATUS_OPCODE_NEW_FRAG_RULE = 0x02,
+ IPAHAL_PKT_STATUS_OPCODE_DROPPED_PACKET = 0x04,
+ IPAHAL_PKT_STATUS_OPCODE_SUSPENDED_PACKET = 0x08,
+ IPAHAL_PKT_STATUS_OPCODE_LOG = 0x10,
+ IPAHAL_PKT_STATUS_OPCODE_DCMP = 0x20,
+ IPAHAL_PKT_STATUS_OPCODE_PACKET_2ND_PASS = 0x40,
+};
+
+/**
+ * enum ipahal_pkt_status_exception - Packet Status exception type
+ * @IPAHAL_PKT_STATUS_EXCEPTION_PACKET_LENGTH: formerly IHL exception.
+ *
+ * Note: IPTYPE, PACKET_LENGTH and PACKET_THRESHOLD exceptions means that
+ * partial / no IP processing took place and corresponding Status Mask
+ * fields should be ignored. Flt and rt info is not valid.
+ *
+ * NOTE:: Any change to this enum, need to change to
+ * ipahal_pkt_status_exception_to_str array as well.
+ */
+enum ipahal_pkt_status_exception {
+ IPAHAL_PKT_STATUS_EXCEPTION_NONE = 0,
+ IPAHAL_PKT_STATUS_EXCEPTION_DEAGGR,
+ IPAHAL_PKT_STATUS_EXCEPTION_IPTYPE,
+ IPAHAL_PKT_STATUS_EXCEPTION_PACKET_LENGTH,
+ IPAHAL_PKT_STATUS_EXCEPTION_PACKET_THRESHOLD,
+ IPAHAL_PKT_STATUS_EXCEPTION_FRAG_RULE_MISS,
+ IPAHAL_PKT_STATUS_EXCEPTION_SW_FILT,
+ /* NAT and IPv6CT have the same value at HW.
+ * NAT for IPv4 and IPv6CT for IPv6 exceptions
+ */
+ IPAHAL_PKT_STATUS_EXCEPTION_NAT,
+ IPAHAL_PKT_STATUS_EXCEPTION_IPV6CT,
+ IPAHAL_PKT_STATUS_EXCEPTION_MAX,
+};
+
+/**
+ * enum ipahal_pkt_status_mask - Packet Status bitmask values of
+ * the contained flags. This bitmask indicates flags on the properties of
+ * the packet as well as IPA processing it may had.
+ * @TAG_VALID: Flag specifying if TAG and TAG info valid?
+ * @CKSUM_PROCESS: CSUM block processing flag: Was pkt processed by csum block?
+ * If so, csum trailer exists
+ */
+enum ipahal_pkt_status_mask {
+ /* Other values are defined but are not specifically handled yet. */
+ IPAHAL_PKT_STATUS_MASK_CKSUM_PROCESS = 0x0100,
+};
+
+/**
+ * struct ipahal_pkt_status - IPA status packet abstracted payload.
+ * @status_opcode: The type of status (Opcode).
+ * @exception: The first exception that took place.
+ * In case of exception, endp_src_idx and pkt_len are always valid.
+ * @status_mask: Bit mask for flags on several properties on the packet
+ * and processing it may passed at IPA.
+ * @pkt_len: Pkt pyld len including hdr and retained hdr if used. Does
+ * not include padding or checksum trailer len.
+ * @endp_src_idx: Source end point index.
+ * @endp_dest_idx: Destination end point index.
+ * Not valid in case of exception
+ * @metadata: meta data value used by packet
+ * @rt_miss: Routing miss flag: Was their a routing rule miss?
+ *
+ * This structure describes the status packet fields for the following
+ * status values: IPA_STATUS_PACKET, IPA_STATUS_DROPPED_PACKET,
+ * IPA_STATUS_SUSPENDED_PACKET. Other status types have different status
+ * packet structure. Note that the hardware supplies additional status
+ * information that is currently unused.
+ */
+struct ipahal_pkt_status {
+ enum ipahal_pkt_status_opcode status_opcode;
+ enum ipahal_pkt_status_exception exception;
+ enum ipahal_pkt_status_mask status_mask;
+ u32 pkt_len;
+ u8 endp_src_idx;
+ u8 endp_dest_idx;
+ u32 metadata;
+ bool rt_miss;
+};
+
+/**
+ * ipahal_pkt_status_get_size() - Get size of a hardware packet status
+ */
+u32 ipahal_pkt_status_get_size(void);
+
+/* ipahal_pkt_status_parse() - Parse packet status payload
+ * @unparsed_status: Packet status read from hardware
+ * @status: Buffer to hold parsed status information
+ */
+void ipahal_pkt_status_parse(const void *unparsed_status,
+ struct ipahal_pkt_status *status);
+
+int ipahal_init(void);
+void ipahal_exit(void);
+
+/* Does the given ID represent rule miss? */
+bool ipahal_is_rule_miss_id(u32 id);
+
+int ipahal_rt_generate_empty_img(u32 route_count, struct ipa_dma_mem *mem);
+int ipahal_flt_generate_empty_img(u64 ep_bitmap, struct ipa_dma_mem *mem);
+
+/**
+ * ipahal_free_empty_img() - Free empty filter or route image
+ * @mem: DMA memory containing filter/route data
+ */
+void ipahal_free_empty_img(struct ipa_dma_mem *mem);
+
+#endif /* _IPAHAL_H_ */
--
2.17.1
^ permalink raw reply related
* Re: [PATCH v2 1/2] mm/page_alloc: free order-0 pages through PCP in page_frag_free()
From: Tariq Toukan @ 2018-11-07 9:59 UTC (permalink / raw)
To: Aaron Lu, linux-mm@kvack.org, linux-kernel@vger.kernel.org,
netdev@vger.kernel.org
Cc: Andrew Morton, Paweł Staszewski, Jesper Dangaard Brouer,
Eric Dumazet, Tariq Toukan, Ilias Apalodimas, Yoel Caspersen,
Mel Gorman, Saeed Mahameed, Michal Hocko, Vlastimil Babka,
Dave Hansen, Alexander Duyck
In-Reply-To: <20181106052833.GC6203@intel.com>
On 06/11/2018 7:28 AM, Aaron Lu wrote:
> page_frag_free() calls __free_pages_ok() to free the page back to
> Buddy. This is OK for high order page, but for order-0 pages, it
> misses the optimization opportunity of using Per-Cpu-Pages and can
> cause zone lock contention when called frequently.
>
> Paweł Staszewski recently shared his result of 'how Linux kernel
> handles normal traffic'[1] and from perf data, Jesper Dangaard Brouer
> found the lock contention comes from page allocator:
>
> mlx5e_poll_tx_cq
> |
> --16.34%--napi_consume_skb
> |
> |--12.65%--__free_pages_ok
> | |
> | --11.86%--free_one_page
> | |
> | |--10.10%--queued_spin_lock_slowpath
> | |
> | --0.65%--_raw_spin_lock
> |
> |--1.55%--page_frag_free
> |
> --1.44%--skb_release_data
>
> Jesper explained how it happened: mlx5 driver RX-page recycle
> mechanism is not effective in this workload and pages have to go
> through the page allocator. The lock contention happens during
> mlx5 DMA TX completion cycle. And the page allocator cannot keep
> up at these speeds.[2]
>
> I thought that __free_pages_ok() are mostly freeing high order
> pages and thought this is an lock contention for high order pages
> but Jesper explained in detail that __free_pages_ok() here are
> actually freeing order-0 pages because mlx5 is using order-0 pages
> to satisfy its page pool allocation request.[3]
>
Thanks for your patch!
Acked-by: Tariq Toukan <tariqt@mellanox.com>
> The free path as pointed out by Jesper is:
> skb_free_head()
> -> skb_free_frag()
> -> page_frag_free()
> And the pages being freed on this path are order-0 pages.
>
> Fix this by doing similar things as in __page_frag_cache_drain() -
> send the being freed page to PCP if it's an order-0 page, or
> directly to Buddy if it is a high order page.
>
> With this change, Paweł hasn't noticed lock contention yet in
> his workload and Jesper has noticed a 7% performance improvement
> using a micro benchmark and lock contention is gone. Ilias' test
> on a 'low' speed 1Gbit interface on an cortex-a53 shows ~11%
> performance boost testing with 64byte packets and __free_pages_ok()
> disappeared from perf top.
>
> [1]: https://www.spinics.net/lists/netdev/msg531362.html
> [2]: https://www.spinics.net/lists/netdev/msg531421.html
> [3]: https://www.spinics.net/lists/netdev/msg531556.html
> Reported-by: Paweł Staszewski <pstaszewski@itcare.pl>
> Analysed-by: Jesper Dangaard Brouer <brouer@redhat.com>
> Acked-by: Vlastimil Babka <vbabka@suse.cz>
> Acked-by: Mel Gorman <mgorman@techsingularity.net>
> Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
> Acked-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> Tested-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> Acked-by: Alexander Duyck <alexander.h.duyck@linux.intel.com>
> Signed-off-by: Aaron Lu <aaron.lu@intel.com>
> ---
> v2: only changelog changes:
> - remove the duplicated skb_free_frag() as pointed by Jesper;
> - add Ilias' test result;
> - add people's ack/test tag.
>
> mm/page_alloc.c | 10 ++++++++--
> 1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index ae31839874b8..91a9a6af41a2 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -4555,8 +4555,14 @@ void page_frag_free(void *addr)
> {
> struct page *page = virt_to_head_page(addr);
>
> - if (unlikely(put_page_testzero(page)))
> - __free_pages_ok(page, compound_order(page));
> + if (unlikely(put_page_testzero(page))) {
> + unsigned int order = compound_order(page);
> +
> + if (order == 0)
> + free_unref_page(page);
> + else
> + __free_pages_ok(page, order);
> + }
> }
> EXPORT_SYMBOL(page_frag_free);
>
>
^ permalink raw reply
* Re: [PATCH bpf-next] bpf_load: add map name to load_maps error message
From: Song Liu @ 2018-11-07 0:26 UTC (permalink / raw)
To: John Fastabend
Cc: shannon.nelson, Alexei Starovoitov, Daniel Borkmann, Networking,
shannon.lee.nelson
In-Reply-To: <ad307fb7-0ea3-3929-1dfe-38dbf281e206@gmail.com>
On Mon, Oct 29, 2018 at 3:12 PM John Fastabend <john.fastabend@gmail.com> wrote:
>
> On 10/29/2018 02:14 PM, Shannon Nelson wrote:
> > To help when debugging bpf/xdp load issues, have the load_map()
> > error message include the number and name of the map that
> > failed.
> >
> > Signed-off-by: Shannon Nelson <shannon.nelson@oracle.com>
> > ---
> > samples/bpf/bpf_load.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/samples/bpf/bpf_load.c b/samples/bpf/bpf_load.c
> > index 89161c9..5de0357 100644
> > --- a/samples/bpf/bpf_load.c
> > +++ b/samples/bpf/bpf_load.c
> > @@ -282,8 +282,8 @@ static int load_maps(struct bpf_map_data *maps, int nr_maps,
> > numa_node);
> > }
> > if (map_fd[i] < 0) {
> > - printf("failed to create a map: %d %s\n",
> > - errno, strerror(errno));
> > + printf("failed to create map %d (%s): %d %s\n",
> > + i, maps[i].name, errno, strerror(errno));
> > return 1;
> > }
> > maps[i].fd = map_fd[i];
> >
>
> LGTM
>
> Acked-by: John Fastabend <john.fastabend@gmail.com>
Acked-by: Song Liu <songliubraving@fb.com>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox