* Re: Fwd: [PATCH] bonding/802.3ad: fix slave initialization states race
From: Aleksei Zakharov @ 2019-09-26 14:25 UTC (permalink / raw)
To: Jay Vosburgh; +Cc: netdev, zhangsha (A)
In-Reply-To: <15507.1569472734@nyx>
чт, 26 сент. 2019 г. в 07:38, Jay Vosburgh <jay.vosburgh@canonical.com>:
>
> Aleksei Zakharov <zaharov@selectel.ru> wrote:
>
> >ср, 25 сент. 2019 г. в 03:31, Jay Vosburgh <jay.vosburgh@canonical.com>:
> >>
> >> Алексей Захаров wrote:
> >> [...]
> >> >Right after reboot one of the slaves hangs with actor port state 71
> >> >and partner port state 1.
> >> >It doesn't send lacpdu and seems to be broken.
> >> >Setting link down and up again fixes slave state.
> >> [...]
> >>
> >> I think I see what failed in the first patch, could you test the
> >> following patch? This one is for net-next, so you'd need to again swap
> >> slave_err / netdev_err for the Ubuntu 4.15 kernel.
> >>
> >I've tested new patch. It seems to work. I can't reproduce the bug
> >with this patch.
> >There are two types of messages when link becomes up:
> >First:
> >bond-san: EVENT 1 llu 4294895911 slave eth2
> >8021q: adding VLAN 0 to HW filter on device eth2
> >bond-san: link status definitely down for interface eth2, disabling it
> >mlx4_en: eth2: Link Up
> >bond-san: EVENT 4 llu 4294895911 slave eth2
> >bond-san: link status up for interface eth2, enabling it in 500 ms
> >bond-san: invalid new link 3 on slave eth2
> >bond-san: link status definitely up for interface eth2, 10000 Mbps full duplex
> >Second:
> >bond-san: EVENT 1 llu 4295147594 slave eth2
> >8021q: adding VLAN 0 to HW filter on device eth2
> >mlx4_en: eth2: Link Up
> >bond-san: EVENT 4 llu 4295147594 slave eth2
> >bond-san: link status up again after 0 ms for interface eth2
> >bond-san: link status definitely up for interface eth2, 10000 Mbps full duplex
> > [...]
>
> The "invalid new link" is appearing because bond_miimon_commit
> is being asked to commit a new state that isn't UP or DOWN (3 is
> BOND_LINK_BACK). I looked through the patched code today, and I don't
> see a way to get to that message with the new link set to 3, so I'll add
> some instrumentation and send out another patch to figure out what's
> going on, as that shouldn't happen.
>
> I don't see the "invalid" message testing locally, I think
> because my network device doesn't transition to carrier up as quickly as
> yours. I thought you were getting BOND_LINK_BACK passed through from
> bond_enslave (which calls bond_set_slave_link_state, which will set
> link_new_link to BOND_LINK_BACK and leave it there), but the
> link_new_link is reset first thing in bond_miimon_inspect, so I'm not
> sure how it gets into bond_miimon_commit (I'm thinking perhaps a
> concurrent commit triggered by another slave, which then picks up this
> proposed link state change by happenstance).
I assume that "invalid new link" happens in this way:
Interface goes up
NETDEV_CHANGE event occurs
bond_update_speed_duplex fails
and slave->last_link_up returns true
slave->link becomes BOND_LINK_FAIL
bond_check_dev_link returns 0
miimon proposes slave->link_new_state BOND_LINK_DOWN
NETDEV_UP event occurs
miimon sets commit++
miimon proposes slave->link_new_state BOND_LINK_BACK
miimon sets slave->link to BOND_LINK_BACK
we have updelay configured, so it doesn't set BOND_LINK_UP in the next
case section
miimon says "Invalid new link" and sets link state UP during next
inspection(after updelay, i suppose)
For the second type of messages it looks like this:
Interface goes up
NETDEV_CHANGE event occurs
bond_update_speed_duplex fails
and slave->last_link_up returns true
slave->link becomes BOND_LINK_FAIL
NETDEV_UP event occurs
bond_check_dev_link returns 1
miimon proposes slave->link_new_state BOND_LINK_UP and says "link
status up again"
My first patch changed slave->last_link_up check to (slave->link ==
BOND_LINK_UP).
This check looks more consistent for me, but I might be wrong here.
As a result if link was in BOND_LINK_FAIL or BOND_LINK_BACK when
CHANGE or UP event,
it became BOND_LINK_DOWN.
But if it was initially UP and bond_update_speed_duplex was unable to
get speed/duplex,
link became BOND_LINK_FAIL.
I don't understand a few things here:
How could a link be in a different state from time to time during the
first NETDEV_* event?
And why slave->last_link_up is set when the first NETDEV event occurs?
I hope I didn't messed things up too much here.
--
Best Regards,
Aleksei Zakharov
^ permalink raw reply
* Re: CONFIG_NET_TC_SKB_EXT
From: Edward Cree @ 2019-09-26 14:26 UTC (permalink / raw)
To: Paul Blakey, Jakub Kicinski
Cc: Pravin Shelar, Daniel Borkmann, Vlad Buslov, David Miller,
netdev@vger.kernel.org, Jiri Pirko, Cong Wang, Jamal Hadi Salim,
Simon Horman, Or Gerlitz
In-Reply-To: <ecfb7918-7660-91f0-035e-56f58a41dc17@mellanox.com>
On 26/09/2019 14:56, Paul Blakey wrote:
>>> In nat scenarios the packet will be modified, and then there can be a miss:
>>>
>>> -trk .... CT(zone X, Restore NAT),goto chain 1
>>>
>>> +trk+est, match on ipv4, CT(zone Y), goto chain 2
>>>
>>> +trk+est, output..
>> I'm confused, I thought the usual nat scenario looked more like
>> 0: -trk ... action ct(zone x), goto chain 1
>> 1: +trk+new ... action ct(commit, nat=foo) # sw only
>> 1: +trk+est ... action ct(nat), mirred eth1
>> i.e. the NAT only happens after conntrack has matched (and thus provided
>> the saved NAT metadata), at the end of the pipe. I don't see how you
>> can NAT a -trk packet.
> Both are valid, Nat in the first hop, executes the nat stored on the
> connection if available (configured by commit).
This still isn't making sense to me.
Until you've done a conntrack lookup and found the connection, you can't
use NAT information that's stored in the connection.
So the NAT can only happen after a conntrack match is found.
And all the rest of your stuff (like doing conntrack twice, in different
zones X and Y) is 'weird' inasmuch as it's beyond the basic minimum
functionality for a useful offload, and inherently doesn't map to a
fixed-layout (non-loopy) HW pipeline. You may want to support it in
your driver, you may be able to support it in your hardware, but it's
not true that "even nat needs that" (the nat scenario I described above
is entirely reasonable and is perfectly workable in an all-or-nothing
offload world), so if your changes are causing problems, they should be
reverted for this cycle.
>> AFAICT only 'deliverish' actions (i.e. mirred and drop) in TC have stats.
>> So stats are unlikely to be a problem unless you've got (say) a mirred
>> mirror before you send to ct and goto chain, in which case the extra
>> copy of the packet is a rather bigger problem for idempotency than mere
>> stats ;-)
> All tc actions have software stats, and at least one (goto, mirred,
> drop) per OvS generated rule will have hardware stats.
Ooh, goto has hardware stats? That's something I hadn't spotted *re-
draws hardware design slightly*
> All OvS datapath rules have stats, and in turn the translated TC rules
> all have stats. OvS ages each rule independently.
TC rules do not have stats. Only TC actions have stats. (The offload
API currently gets confused about this and it's really annoyed me...)
^ permalink raw reply
* [PATCH net] tcp_bbr: fix quantization code to not raise cwnd if not probing bandwidth
From: Kevin(Yudong) Yang @ 2019-09-26 14:30 UTC (permalink / raw)
To: David Miller
Cc: netdev, Kevin(Yudong) Yang, Neal Cardwell, Yuchung Cheng,
Soheil Hassas Yeganeh, Priyaranjan Jha
There was a bug in the previous logic that attempted to ensure gain cycling
gets inflight above BDP even for small BDPs. This code correctly raised and
lowered target inflight values during the gain cycle. And this code
correctly ensured that cwnd was raised when probing bandwidth. However, it
did not correspondingly ensure that cwnd was *not* raised in this way when
*not* probing for bandwidth. The result was that small-BDP flows that were
always cwnd-bound could go for many cycles with a fixed cwnd, and not probe
or yield bandwidth at all. This meant that multiple small-BDP flows could
fail to converge in their bandwidth allocations.
Fixes: 383d470 ("tcp_bbr: fix bw probing to raise in-flight data for very small BDPs")
Signed-off-by: Kevin(Yudong) Yang <yyd@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Acked-by: Priyaranjan Jha <priyarjha@google.com>
---
net/ipv4/tcp_bbr.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/net/ipv4/tcp_bbr.c b/net/ipv4/tcp_bbr.c
index 95b59540eee1..32772d6ded4e 100644
--- a/net/ipv4/tcp_bbr.c
+++ b/net/ipv4/tcp_bbr.c
@@ -388,7 +388,7 @@ static u32 bbr_bdp(struct sock *sk, u32 bw, int gain)
* which allows 2 outstanding 2-packet sequences, to try to keep pipe
* full even with ACK-every-other-packet delayed ACKs.
*/
-static u32 bbr_quantization_budget(struct sock *sk, u32 cwnd, int gain)
+static u32 bbr_quantization_budget(struct sock *sk, u32 cwnd)
{
struct bbr *bbr = inet_csk_ca(sk);
@@ -399,7 +399,7 @@ static u32 bbr_quantization_budget(struct sock *sk, u32 cwnd, int gain)
cwnd = (cwnd + 1) & ~1U;
/* Ensure gain cycling gets inflight above BDP even for small BDPs. */
- if (bbr->mode == BBR_PROBE_BW && gain > BBR_UNIT)
+ if (bbr->mode == BBR_PROBE_BW && bbr->cycle_idx == 0)
cwnd += 2;
return cwnd;
@@ -411,7 +411,7 @@ static u32 bbr_inflight(struct sock *sk, u32 bw, int gain)
u32 inflight;
inflight = bbr_bdp(sk, bw, gain);
- inflight = bbr_quantization_budget(sk, inflight, gain);
+ inflight = bbr_quantization_budget(sk, inflight);
return inflight;
}
@@ -531,7 +531,7 @@ static void bbr_set_cwnd(struct sock *sk, const struct rate_sample *rs,
* due to aggregation (of data and/or ACKs) visible in the ACK stream.
*/
target_cwnd += bbr_ack_aggregation_cwnd(sk);
- target_cwnd = bbr_quantization_budget(sk, target_cwnd, gain);
+ target_cwnd = bbr_quantization_budget(sk, target_cwnd);
/* If we're below target cwnd, slow start cwnd toward target cwnd. */
if (bbr_full_bw_reached(sk)) /* only cut cwnd if we filled the pipe */
--
2.23.0.444.g18eeb5a265-goog
^ permalink raw reply related
* Re: [PATCH v2] ipv6: do not free rt if FIB_LOOKUP_NOREF is set on suppress rule
From: David Ahern @ 2019-09-26 14:43 UTC (permalink / raw)
To: Jason A. Donenfeld, netdev, linux-kernel, davem, weiwan; +Cc: stable
In-Reply-To: <20190924140128.19394-1-Jason@zx2c4.com>
On 9/24/19 8:01 AM, Jason A. Donenfeld wrote:
> Commit 7d9e5f422150 removed references from certain dsts, but accounting
> for this never translated down into the fib6 suppression code. This bug
> was triggered by WireGuard users who use wg-quick(8), which uses the
> "suppress-prefix" directive to ip-rule(8) for routing all of their
> internet traffic without routing loops. The test case added here
> causes the reference underflow by causing packets to evaluate a suppress
> rule.
>
> Cc: stable@vger.kernel.org
> Fixes: 7d9e5f422150 ("ipv6: convert major tx path to use RT6_LOOKUP_F_DST_NOREF")
> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
> ---
> net/ipv6/fib6_rules.c | 3 ++-
> tools/testing/selftests/net/fib_tests.sh | 17 ++++++++++++++++-
> 2 files changed, 18 insertions(+), 2 deletions(-)
Thanks for adding the test case.
Reviewed-by: David Ahern <dsahern@gmail.com>
^ permalink raw reply
* Re: [PATCH ghak90 V7 04/21] audit: convert to contid list to check for orch/engine ownership
From: Neil Horman @ 2019-09-26 14:46 UTC (permalink / raw)
To: Richard Guy Briggs
Cc: containers, linux-api, Linux-Audit Mailing List, linux-fsdevel,
LKML, netdev, netfilter-devel, Paul Moore, sgrubb, omosnace,
dhowells, simo, eparis, serge, ebiederm, dwalsh, mpatel
In-Reply-To: <6fb4e270bfafef3d0477a06b0365fdcc5a5305b5.1568834524.git.rgb@redhat.com>
On Wed, Sep 18, 2019 at 09:22:21PM -0400, Richard Guy Briggs wrote:
> Store the audit container identifier in a refcounted kernel object that
> is added to the master list of audit container identifiers. This will
> allow multiple container orchestrators/engines to work on the same
> machine without danger of inadvertantly re-using an existing identifier.
> It will also allow an orchestrator to inject a process into an existing
> container by checking if the original container owner is the one
> injecting the task. A hash table list is used to optimize searches.
>
> Signed-off-by: Richard Guy Briggs <rgb@redhat.com>
> ---
> include/linux/audit.h | 26 ++++++++++++++--
> kernel/audit.c | 86 ++++++++++++++++++++++++++++++++++++++++++++++++---
> kernel/audit.h | 8 +++++
> 3 files changed, 112 insertions(+), 8 deletions(-)
>
> diff --git a/include/linux/audit.h b/include/linux/audit.h
> index f2e3b81f2942..e317807cdd3e 100644
> --- a/include/linux/audit.h
> +++ b/include/linux/audit.h
> @@ -95,10 +95,18 @@ struct audit_ntp_data {
> struct audit_ntp_data {};
> #endif
>
> +struct audit_cont {
> + struct list_head list;
> + u64 id;
> + struct task_struct *owner;
> + refcount_t refcount;
> + struct rcu_head rcu;
> +};
> +
> struct audit_task_info {
> kuid_t loginuid;
> unsigned int sessionid;
> - u64 contid;
> + struct audit_cont *cont;
> #ifdef CONFIG_AUDITSYSCALL
> struct audit_context *ctx;
> #endif
> @@ -203,11 +211,15 @@ static inline unsigned int audit_get_sessionid(struct task_struct *tsk)
>
> static inline u64 audit_get_contid(struct task_struct *tsk)
> {
> - if (!tsk->audit)
> + if (!tsk->audit || !tsk->audit->cont)
> return AUDIT_CID_UNSET;
> - return tsk->audit->contid;
> + return tsk->audit->cont->id;
> }
>
> +extern struct audit_cont *audit_cont(struct task_struct *tsk);
> +
> +extern void audit_cont_put(struct audit_cont *cont);
> +
I see that you manual increment this refcount at various call sites, why
no corresponding audit_contid_hold function?
Neil
> extern u32 audit_enabled;
>
> extern int audit_signal_info(int sig, struct task_struct *t);
> @@ -277,6 +289,14 @@ static inline u64 audit_get_contid(struct task_struct *tsk)
> return AUDIT_CID_UNSET;
> }
>
> +static inline struct audit_cont *audit_cont(struct task_struct *tsk)
> +{
> + return NULL;
> +}
> +
> +static inline void audit_cont_put(struct audit_cont *cont)
> +{ }
> +
> #define audit_enabled AUDIT_OFF
>
> static inline int audit_signal_info(int sig, struct task_struct *t)
> diff --git a/kernel/audit.c b/kernel/audit.c
> index a36ea57cbb61..ea0899130cc1 100644
> --- a/kernel/audit.c
> +++ b/kernel/audit.c
> @@ -137,6 +137,8 @@ struct audit_net {
>
> /* Hash for inode-based rules */
> struct list_head audit_inode_hash[AUDIT_INODE_BUCKETS];
> +/* Hash for contid-based rules */
> +struct list_head audit_contid_hash[AUDIT_CONTID_BUCKETS];
>
> static struct kmem_cache *audit_buffer_cache;
>
> @@ -204,6 +206,8 @@ struct audit_reply {
>
> static struct kmem_cache *audit_task_cache;
>
> +static DEFINE_SPINLOCK(audit_contid_list_lock);
> +
> void __init audit_task_init(void)
> {
> audit_task_cache = kmem_cache_create("audit_task",
> @@ -231,7 +235,9 @@ int audit_alloc(struct task_struct *tsk)
> }
> info->loginuid = audit_get_loginuid(current);
> info->sessionid = audit_get_sessionid(current);
> - info->contid = audit_get_contid(current);
> + info->cont = audit_cont(current);
> + if (info->cont)
> + refcount_inc(&info->cont->refcount);
> tsk->audit = info;
>
> ret = audit_alloc_syscall(tsk);
> @@ -246,7 +252,7 @@ int audit_alloc(struct task_struct *tsk)
> struct audit_task_info init_struct_audit = {
> .loginuid = INVALID_UID,
> .sessionid = AUDIT_SID_UNSET,
> - .contid = AUDIT_CID_UNSET,
> + .cont = NULL,
> #ifdef CONFIG_AUDITSYSCALL
> .ctx = NULL,
> #endif
> @@ -266,6 +272,9 @@ void audit_free(struct task_struct *tsk)
> /* Freeing the audit_task_info struct must be performed after
> * audit_log_exit() due to need for loginuid and sessionid.
> */
> + spin_lock(&audit_contid_list_lock);
> + audit_cont_put(tsk->audit->cont);
> + spin_unlock(&audit_contid_list_lock);
> info = tsk->audit;
> tsk->audit = NULL;
> kmem_cache_free(audit_task_cache, info);
> @@ -1657,6 +1666,9 @@ static int __init audit_init(void)
> for (i = 0; i < AUDIT_INODE_BUCKETS; i++)
> INIT_LIST_HEAD(&audit_inode_hash[i]);
>
> + for (i = 0; i < AUDIT_CONTID_BUCKETS; i++)
> + INIT_LIST_HEAD(&audit_contid_hash[i]);
> +
> mutex_init(&audit_cmd_mutex.lock);
> audit_cmd_mutex.owner = NULL;
>
> @@ -2356,6 +2368,32 @@ int audit_signal_info(int sig, struct task_struct *t)
> return audit_signal_info_syscall(t);
> }
>
> +struct audit_cont *audit_cont(struct task_struct *tsk)
> +{
> + if (!tsk->audit || !tsk->audit->cont)
> + return NULL;
> + return tsk->audit->cont;
> +}
> +
> +/* audit_contid_list_lock must be held by caller */
> +void audit_cont_put(struct audit_cont *cont)
> +{
> + if (!cont)
> + return;
> + if (refcount_dec_and_test(&cont->refcount)) {
> + put_task_struct(cont->owner);
> + list_del_rcu(&cont->list);
> + kfree_rcu(cont, rcu);
> + }
> +}
> +
> +static struct task_struct *audit_cont_owner(struct task_struct *tsk)
> +{
> + if (tsk->audit && tsk->audit->cont)
> + return tsk->audit->cont->owner;
> + return NULL;
> +}
> +
> /*
> * audit_set_contid - set current task's audit contid
> * @task: target task
> @@ -2382,9 +2420,12 @@ int audit_set_contid(struct task_struct *task, u64 contid)
> }
> oldcontid = audit_get_contid(task);
> read_lock(&tasklist_lock);
> - /* Don't allow the audit containerid to be unset */
> + /* Don't allow the contid to be unset */
> if (!audit_contid_valid(contid))
> rc = -EINVAL;
> + /* Don't allow the contid to be set to the same value again */
> + else if (contid == oldcontid) {
> + rc = -EADDRINUSE;
> /* if we don't have caps, reject */
> else if (!capable(CAP_AUDIT_CONTROL))
> rc = -EPERM;
> @@ -2397,8 +2438,43 @@ int audit_set_contid(struct task_struct *task, u64 contid)
> else if (audit_contid_set(task))
> rc = -ECHILD;
> read_unlock(&tasklist_lock);
> - if (!rc)
> - task->audit->contid = contid;
> + if (!rc) {
> + struct audit_cont *oldcont = audit_cont(task);
> + struct audit_cont *cont = NULL;
> + struct audit_cont *newcont = NULL;
> + int h = audit_hash_contid(contid);
> +
> + spin_lock(&audit_contid_list_lock);
> + list_for_each_entry_rcu(cont, &audit_contid_hash[h], list)
> + if (cont->id == contid) {
> + /* task injection to existing container */
> + if (current == cont->owner) {
> + refcount_inc(&cont->refcount);
> + newcont = cont;
> + } else {
> + rc = -ENOTUNIQ;
> + goto conterror;
> + }
> + }
> + if (!newcont) {
> + newcont = kmalloc(sizeof(struct audit_cont), GFP_ATOMIC);
> + if (newcont) {
> + INIT_LIST_HEAD(&newcont->list);
> + newcont->id = contid;
> + get_task_struct(current);
> + newcont->owner = current;
> + refcount_set(&newcont->refcount, 1);
> + list_add_rcu(&newcont->list, &audit_contid_hash[h]);
> + } else {
> + rc = -ENOMEM;
> + goto conterror;
> + }
> + }
> + task->audit->cont = newcont;
> + audit_cont_put(oldcont);
> +conterror:
> + spin_unlock(&audit_contid_list_lock);
> + }
> task_unlock(task);
>
> if (!audit_enabled)
> diff --git a/kernel/audit.h b/kernel/audit.h
> index 16bd03b88e0d..e4a31aa92dfe 100644
> --- a/kernel/audit.h
> +++ b/kernel/audit.h
> @@ -211,6 +211,14 @@ static inline int audit_hash_ino(u32 ino)
> return (ino & (AUDIT_INODE_BUCKETS-1));
> }
>
> +#define AUDIT_CONTID_BUCKETS 32
> +extern struct list_head audit_contid_hash[AUDIT_CONTID_BUCKETS];
> +
> +static inline int audit_hash_contid(u64 contid)
> +{
> + return (contid & (AUDIT_CONTID_BUCKETS-1));
> +}
> +
> /* Indicates that audit should log the full pathname. */
> #define AUDIT_NAME_FULL -1
>
> --
> 1.8.3.1
>
>
^ permalink raw reply
* Re: [PATCH iproute2(-next) 1/1] ip: fix ip route show json output for multipath nexthops
From: David Ahern @ 2019-09-26 14:58 UTC (permalink / raw)
To: Julien Fortin, netdev; +Cc: roopa
In-Reply-To: <20190924223256.74017-1-julien@cumulusnetworks.com>
On 9/24/19 4:32 PM, Julien Fortin wrote:
> diff --git a/ip/iproute.c b/ip/iproute.c
> index a4533851..5d5f1551 100644
> --- a/ip/iproute.c
> +++ b/ip/iproute.c
> @@ -649,23 +649,27 @@ static void print_rta_multipath(FILE *fp, const struct rtmsg *r,
> int len = RTA_PAYLOAD(rta);
> int first = 1;
>
> + open_json_array(PRINT_JSON, "nexthops");
> +
> while (len >= sizeof(*nh)) {
> struct rtattr *tb[RTA_MAX + 1];
>
> if (nh->rtnh_len > len)
> break;
>
> + open_json_object(NULL);
> +
> if (!is_json_context()) {
With the fprintf removed in favor of print_string(PRINT_FP), you should
be able to remove this is_json_context() check and remove a level of
indentation for this section.
> if ((r->rtm_flags & RTM_F_CLONED) &&
> r->rtm_type == RTN_MULTICAST) {
> if (first) {
> - fprintf(fp, "Oifs: ");
> + print_string(PRINT_FP, NULL, "Oifs: ", NULL);
> first = 0;
> } else {
> - fprintf(fp, " ");
> + print_string(PRINT_FP, NULL, " ", NULL);
> }
> } else
> - fprintf(fp, "%s\tnexthop ", _SL_);
> + print_string(PRINT_FP, NULL, "%s\tnexthop ", _SL_);
> }
>
> if (nh->rtnh_len > sizeof(*nh)) {
^ permalink raw reply
* Re: TCP_USER_TIMEOUT, SYN-SENT and tcp_syn_retries
From: Eric Dumazet @ 2019-09-26 15:05 UTC (permalink / raw)
To: Marek Majkowski, netdev
In-Reply-To: <CAJPywTL0PiesEwiRWHdJr0Te_rqZ62TXbgOtuz7NTYmQksE_7w@mail.gmail.com>
On 9/25/19 1:46 AM, Marek Majkowski wrote:
> Hello my favorite mailing list!
>
> Recently I've been looking into TCP_USER_TIMEOUT and noticed some
> strange behaviour on fresh sockets in SYN-SENT state. Full writeup:
> https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/
>
> Here's a reproducer. It does a simple thing: sets TCP_USER_TIMEOUT and
> does connect() to a blackholed IP:
>
> $ wget https://gist.githubusercontent.com/majek/b4ad53c5795b226d62fad1fa4a87151a/raw/cbb928cb99cd6c5aa9f73ba2d3bc0aef22fbc2bf/user-timeout-and-syn.py
>
> $ sudo python3 user-timeout-and-syn.py
> 00:00.000000 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S]
> 00:01.007053 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S]
> 00:03.023051 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S]
> 00:05.007096 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S]
> 00:05.015037 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S]
> 00:05.023020 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S]
> 00:05.034983 IP 192.1.1.1.52974 > 244.0.0.1.1234: Flags [S]
>
> The connect() times out with ETIMEDOUT after 5 seconds - as intended.
> But Linux (5.3.0-rc3) does something weird on the network - it sends
> remaining tcp_syn_retries packets aligned to the 5s mark.
>
> In other words: with TCP_USER_TIMEOUT we are sending spurious SYN
> packets on a timeout.
>
> For the record, the man page doesn't define what TCP_USER_TIMEOUT does
> on SYN-SENT state.
>
Exactly, so far this option has only be used on established flows.
Feel free to send patches if you need to override the stack behavior
for connection establishment (Same remark for passive side...)
Thanks.
^ permalink raw reply
* Re: CONFIG_NET_TC_SKB_EXT
From: Paul Blakey @ 2019-09-26 15:14 UTC (permalink / raw)
To: Edward Cree, Jakub Kicinski
Cc: Pravin Shelar, Daniel Borkmann, Vlad Buslov, David Miller,
netdev@vger.kernel.org, Jiri Pirko, Cong Wang, Jamal Hadi Salim,
Simon Horman, Or Gerlitz
In-Reply-To: <3aff5059-c28c-f194-72f0-69edddf89f84@solarflare.com>
On 9/26/2019 5:26 PM, Edward Cree wrote:
> On 26/09/2019 14:56, Paul Blakey wrote:
>>>> In nat scenarios the packet will be modified, and then there can be a miss:
>>>>
>>>> -trk .... CT(zone X, Restore NAT),goto chain 1
>>>>
>>>> +trk+est, match on ipv4, CT(zone Y), goto chain 2
>>>>
>>>> +trk+est, output..
>>> I'm confused, I thought the usual nat scenario looked more like
>>> 0: -trk ... action ct(zone x), goto chain 1
>>> 1: +trk+new ... action ct(commit, nat=foo) # sw only
>>> 1: +trk+est ... action ct(nat), mirred eth1
>>> i.e. the NAT only happens after conntrack has matched (and thus provided
>>> the saved NAT metadata), at the end of the pipe. I don't see how you
>>> can NAT a -trk packet.
>> Both are valid, Nat in the first hop, executes the nat stored on the
>> connection if available (configured by commit).
> This still isn't making sense to me.
> Until you've done a conntrack lookup and found the connection, you can't
> use NAT information that's stored in the connection.
> So the NAT can only happen after a conntrack match is found.
That's how it works, CT action restores the metadata (nf_conn on the
SKB), you can then, if needed, execute the nat,
That is stored implicitly on this metadata (by using the reverse of the
reply tuple). It's why nf_conn status has two status bits
(IPS_SRC_NAT_DONE_BIT and IPS_SRC_NAT).
After you execute it, IPS_SRC_NAT_DONE_BIT is set, instead of just
IPS_SRC_NAT (nat needed bit).
The usage is straight from ovs testsuite, please see it in ovs git.
>
> And all the rest of your stuff (like doing conntrack twice, in different
> zones X and Y) is 'weird' inasmuch as it's beyond the basic minimum
> functionality for a useful offload, and inherently doesn't map to a
> fixed-layout (non-loopy) HW pipeline. You may want to support it in
> your driver, you may be able to support it in your hardware, but it's
> not true that "even nat needs that" (the nat scenario I described above
> is entirely reasonable and is perfectly workable in an all-or-nothing
> offload world), so if your changes are causing problems, they should be
> reverted for this cycle.
The change didn't cause any problem, and this doesn't contradict any
other vendor, doing what you suggest - offloading just a subset of the
rules.
As we are discussing the config default, I've sent a patch to set the
config to N.
^ permalink raw reply
* Re: [1/2] net/phy/mdio-mscc-miim: Use devm_platform_ioremap_resource() in mscc_miim_probe()
From: Markus Elfring @ 2019-09-26 15:29 UTC (permalink / raw)
To: Andrew Lunn, netdev
Cc: Alexandre Belloni, Florian Fainelli, David S. Miller,
Heiner Kallweit, LKML, kernel-janitors
In-Reply-To: <20190920190908.GH3530@lunn.ch>
>> Simplify this function implementation by using a known wrapper function.
…
> Reviewed-by: Andrew Lunn <andrew@lunn.ch>
Does this feedback indicate also an agreement for the detail
if the mapping of internal phy registers would be a required operation?
(Would such a resource allocation eventually be optional?)
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/phy/mdio-mscc-miim.c?id=f41def397161053eb0d3ed6861ef65985efbf293#n145
https://elixir.bootlin.com/linux/v5.3.1/source/drivers/net/phy/mdio-mscc-miim.c#L145
Regards,
Markus
^ permalink raw reply
* [PATCH iproute2(-next) v2 1/1] ip: fix ip route show json output for multipath nexthops
From: Julien Fortin @ 2019-09-26 15:29 UTC (permalink / raw)
To: netdev; +Cc: roopa, dsahern, Julien Fortin
From: Julien Fortin <julien@cumulusnetworks.com>
print_rta_multipath doesn't support JSON output:
{
"dst":"27.0.0.13",
"protocol":"bgp",
"metric":20,
"flags":[],
"gateway":"169.254.0.1"dev uplink-1 weight 1 ,
"flags":["onlink"],
"gateway":"169.254.0.1"dev uplink-2 weight 1 ,
"flags":["onlink"]
},
since RTA_MULTIPATH has nested objects we should print them
in a json array.
With the path we have the following output:
{
"flags": [],
"dst": "36.0.0.13",
"protocol": "bgp",
"metric": 20,
"nexthops": [
{
"weight": 1,
"flags": [
"onlink"
],
"gateway": "169.254.0.1",
"dev": "uplink-1"
},
{
"weight": 1,
"flags": [
"onlink"
],
"gateway": "169.254.0.1",
"dev": "uplink-2"
}
]
}
Fixes: 663c3cb23103f4 ("iproute: implement JSON and color output")
Signed-off-by: Julien Fortin <julien@cumulusnetworks.com>
---
ip/iproute.c | 46 ++++++++++++++++++++++++++++------------------
1 file changed, 28 insertions(+), 18 deletions(-)
diff --git a/ip/iproute.c b/ip/iproute.c
index a4533851..32bb52df 100644
--- a/ip/iproute.c
+++ b/ip/iproute.c
@@ -649,24 +649,26 @@ static void print_rta_multipath(FILE *fp, const struct rtmsg *r,
int len = RTA_PAYLOAD(rta);
int first = 1;
+ open_json_array(PRINT_JSON, "nexthops");
+
while (len >= sizeof(*nh)) {
struct rtattr *tb[RTA_MAX + 1];
if (nh->rtnh_len > len)
break;
- if (!is_json_context()) {
- if ((r->rtm_flags & RTM_F_CLONED) &&
- r->rtm_type == RTN_MULTICAST) {
- if (first) {
- fprintf(fp, "Oifs: ");
- first = 0;
- } else {
- fprintf(fp, " ");
- }
- } else
- fprintf(fp, "%s\tnexthop ", _SL_);
- }
+ open_json_object(NULL);
+
+ if ((r->rtm_flags & RTM_F_CLONED) &&
+ r->rtm_type == RTN_MULTICAST) {
+ if (first) {
+ print_string(PRINT_FP, NULL, "Oifs: ", NULL);
+ first = 0;
+ } else {
+ print_string(PRINT_FP, NULL, " ", NULL);
+ }
+ } else
+ print_string(PRINT_FP, NULL, "%s\tnexthop ", _SL_);
if (nh->rtnh_len > sizeof(*nh)) {
parse_rtattr(tb, RTA_MAX, RTNH_DATA(nh),
@@ -689,22 +691,30 @@ static void print_rta_multipath(FILE *fp, const struct rtmsg *r,
if ((r->rtm_flags & RTM_F_CLONED) &&
r->rtm_type == RTN_MULTICAST) {
- fprintf(fp, "%s", ll_index_to_name(nh->rtnh_ifindex));
+ print_string(PRINT_ANY, "dev",
+ "%s", ll_index_to_name(nh->rtnh_ifindex));
+
if (nh->rtnh_hops != 1)
- fprintf(fp, "(ttl>%d)", nh->rtnh_hops);
- fprintf(fp, " ");
+ print_int(PRINT_ANY, "ttl", "(ttl>%d)", nh->rtnh_hops);
+
+ print_string(PRINT_FP, NULL, " ", NULL);
} else {
- fprintf(fp, "dev %s ", ll_index_to_name(nh->rtnh_ifindex));
+ print_string(PRINT_ANY, "dev",
+ "dev %s ", ll_index_to_name(nh->rtnh_ifindex));
+
if (r->rtm_family != AF_MPLS)
- fprintf(fp, "weight %d ",
- nh->rtnh_hops+1);
+ print_int(PRINT_ANY, "weight",
+ "weight %d ", nh->rtnh_hops + 1);
}
print_rt_flags(fp, nh->rtnh_flags);
len -= NLMSG_ALIGN(nh->rtnh_len);
nh = RTNH_NEXT(nh);
+
+ close_json_object();
}
+ close_json_array(PRINT_JSON, NULL);
}
int print_route(struct nlmsghdr *n, void *arg)
--
2.23.0
^ permalink raw reply related
* Re: [PATCH V11 0/4] BPF: New helper to obtain namespace data from current task
From: Yonghong Song @ 2019-09-26 15:51 UTC (permalink / raw)
To: Eric W. Biederman, Carlos Neira
Cc: netdev@vger.kernel.org, brouer@redhat.com, bpf@vger.kernel.org
In-Reply-To: <87ef033maf.fsf@x220.int.ebiederm.org>
On 9/25/19 5:59 PM, Eric W. Biederman wrote:
> Carlos Neira <cneirabustos@gmail.com> writes:
>
>> Currently bpf_get_current_pid_tgid(), is used to do pid filtering in bcc's
>> scripts but this helper returns the pid as seen by the root namespace which is
>> fine when a bcc script is not executed inside a container.
>> When the process of interest is inside a container, pid filtering will not work
>> if bpf_get_current_pid_tgid() is used.
>> This helper addresses this limitation returning the pid as it's seen by the current
>> namespace where the script is executing.
>>
>> In the future different pid_ns files may belong to different devices, according to the
>> discussion between Eric Biederman and Yonghong in 2017 Linux plumbers conference.
>> To address that situation the helper requires inum and dev_t from /proc/self/ns/pid.
>> This helper has the same use cases as bpf_get_current_pid_tgid() as it can be
>> used to do pid filtering even inside a container.
>
> I think I may have asked this before. If I am repeating old gound
> please excuse me.
>
> Am I correct in understanding these new helpers are designed to be used
> when programs running in ``conainers'' call it inside pid namespaces
> register bpf programs for tracing?
Right.
>
> If so would it be possible to change how the existing bpf opcodes
> operate when they are used in the context of a pid namespace?
Today, typical bpf program getting pid like:
uint64_t pid_tgid = bpf_get_current_pid_tgid();
pid_t pid = pid_tgid >> 32;
pid_t tid = pid_tgid;
/* possible filtering ... */
if (pid == <user_provided pid>) ....
...
/* record pid in some places */
map_val->pid = pid;
...
The bpf_get_current_pid_tgid() is a kernel helper
BPF_CALL_0(bpf_get_current_pid_tgid)
{
struct task_struct *task = current;
if (unlikely(!task))
return -EINVAL;
return (u64) task->tgid << 32 | task->pid;
}
So the bpf_get_current_pid_tgid() gets the tgid/pid outside any
pid namespaces.
To make the program work inside the container, just get namespace
pid/tgid not enough. You need to make sure the namespace you are
tracking is the one you are in. That is what the new proposed
helper to do.
Do you suggest we change
bpf_get_current_pid_tgid()
to return namespaced tgid/pid?
First, this will break user API (kernel helper is an API) and second,
even if we do get pid/tgid, we still not sure whether
this is for my namespace or not.
Do you have something in mind to address this issue?
>
> That later would seem to allow just moving an existing application into
> a pid namespace with no modifications. If we can do this with trivial
> cost at bpf compile time and with no userspace changes that would seem
> a better approach.
>
> If not can someone point me to why we can't do that? What am I missing?
>
> Eric
>
>> Signed-off-by: Carlos Neira <cneirabustos@gmail.com>
>>
>> Carlos Neira (4):
>> fs/nsfs.c: added ns_match
>> bpf: added new helper bpf_get_ns_current_pid_tgid
>> tools: Added bpf_get_ns_current_pid_tgid helper
>> tools/testing/selftests/bpf: Add self-tests for new helper. self tests
>> added for new helper
>>
>> fs/nsfs.c | 8 +
>> include/linux/bpf.h | 1 +
>> include/linux/proc_ns.h | 2 +
>> include/uapi/linux/bpf.h | 18 ++-
>> kernel/bpf/core.c | 1 +
>> kernel/bpf/helpers.c | 32 ++++
>> kernel/trace/bpf_trace.c | 2 +
>> tools/include/uapi/linux/bpf.h | 18 ++-
>> tools/testing/selftests/bpf/Makefile | 2 +-
>> tools/testing/selftests/bpf/bpf_helpers.h | 3 +
>> .../selftests/bpf/progs/test_pidns_kern.c | 71 ++++++++
>> tools/testing/selftests/bpf/test_pidns.c | 152 ++++++++++++++++++
>> 12 files changed, 307 insertions(+), 3 deletions(-)
>> create mode 100644 tools/testing/selftests/bpf/progs/test_pidns_kern.c
>> create mode 100644 tools/testing/selftests/bpf/test_pidns.c
^ permalink raw reply
* Re: [PATCH iproute2(-next) v2 1/1] ip: fix ip route show json output for multipath nexthops
From: Stephen Hemminger @ 2019-09-26 16:07 UTC (permalink / raw)
To: Julien Fortin; +Cc: netdev, roopa, dsahern
In-Reply-To: <20190926152934.9121-1-julien@cumulusnetworks.com>
On Thu, 26 Sep 2019 17:29:34 +0200
Julien Fortin <julien@cumulusnetworks.com> wrote:
> + print_string(PRINT_ANY, "dev",
> + "%s", ll_index_to_name(nh->rtnh_ifindex))
you might want to use interface color for this?
^ permalink raw reply
* Re: [PATCH V11 0/4] BPF: New helper to obtain namespace data from current task
From: John Fastabend @ 2019-09-26 16:16 UTC (permalink / raw)
To: Eric W. Biederman, Carlos Neira; +Cc: netdev, yhs, brouer, bpf
In-Reply-To: <87ef033maf.fsf@x220.int.ebiederm.org>
Eric W. Biederman wrote:
> Carlos Neira <cneirabustos@gmail.com> writes:
>
> > Currently bpf_get_current_pid_tgid(), is used to do pid filtering in bcc's
> > scripts but this helper returns the pid as seen by the root namespace which is
> > fine when a bcc script is not executed inside a container.
> > When the process of interest is inside a container, pid filtering will not work
> > if bpf_get_current_pid_tgid() is used.
> > This helper addresses this limitation returning the pid as it's seen by the current
> > namespace where the script is executing.
> >
> > In the future different pid_ns files may belong to different devices, according to the
> > discussion between Eric Biederman and Yonghong in 2017 Linux plumbers conference.
> > To address that situation the helper requires inum and dev_t from /proc/self/ns/pid.
> > This helper has the same use cases as bpf_get_current_pid_tgid() as it can be
> > used to do pid filtering even inside a container.
>
> I think I may have asked this before. If I am repeating old gound
> please excuse me.
>
> Am I correct in understanding these new helpers are designed to be used
> when programs running in ``conainers'' call it inside pid namespaces
> register bpf programs for tracing?
>
> If so would it be possible to change how the existing bpf opcodes
> operate when they are used in the context of a pid namespace?
>
> That later would seem to allow just moving an existing application into
> a pid namespace with no modifications. If we can do this with trivial
> cost at bpf compile time and with no userspace changes that would seem
> a better approach.
>
> If not can someone point me to why we can't do that? What am I missing?
We have some management/observabiliity bpf programs loaded from privileged
containers that end up getting triggered in multiple container context. Here
we want the root namespace pid otherwise there would be collisions (same pid
in multiple containers) when its used as a key and we would have difficulty
finding the pid from the root namespace.
I guess at load time if its an unprivileged program we could convert it to
use the pid of the current namespace?
Or if the application is moved into a unprivileged container?
Our code is outside bcc so not sure exactly how the bcc case works. Just
wanted to point out we use the root namespace pid for various things
so I think it might need to be a bit smarter than just the moving an
existing application into a pid namespace.
.John
^ permalink raw reply
* Re: [1/2] net/phy/mdio-mscc-miim: Use devm_platform_ioremap_resource() in mscc_miim_probe()
From: Alexandre Belloni @ 2019-09-26 16:18 UTC (permalink / raw)
To: Markus Elfring
Cc: Andrew Lunn, netdev, Florian Fainelli, David S. Miller,
Heiner Kallweit, LKML, kernel-janitors
In-Reply-To: <121e75c5-4d45-9df2-a471-6997a1fb3218@web.de>
On 26/09/2019 17:29:02+0200, Markus Elfring wrote:
> >> Simplify this function implementation by using a known wrapper function.
> …
> > Reviewed-by: Andrew Lunn <andrew@lunn.ch>
>
> Does this feedback indicate also an agreement for the detail
> if the mapping of internal phy registers would be a required operation?
> (Would such a resource allocation eventually be optional?)
It is optional.
--
Alexandre Belloni, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply
* RE: [PATCH 1/1] bpf: Fix bpf_event_output re-entry issue
From: John Fastabend @ 2019-09-26 16:20 UTC (permalink / raw)
To: Allan Zhang, daniel, songliubraving, netdev, bpf
Cc: linux-kernel, Allan Zhang, Stanislav Fomichev, Eric Dumazet
In-Reply-To: <20190925234312.94063-2-allanzhang@google.com>
Allan Zhang wrote:
> BPF_PROG_TYPE_SOCK_OPS program can reenter bpf_event_output because it can
> be called from atomic and non-atomic contexts since we don't have
> bpf_prog_active to prevent it happen.
>
> This patch enables 3 level of nesting to support normal, irq and nmi
> context.
>
> We can easily reproduce the issue by running neper crr mode with 100 flows
> and 10 threads from neper client side.
>
> Here is the whole stack dump:
>
> [ 515.228898] WARNING: CPU: 20 PID: 14686 at kernel/trace/bpf_trace.c:549 bpf_event_output+0x1f9/0x220
> [ 515.228903] CPU: 20 PID: 14686 Comm: tcp_crr Tainted: G W 4.15.0-smp-fixpanic #44
> [ 515.228904] Hardware name: Intel TBG,ICH10/Ikaria_QC_1b, BIOS 1.22.0 06/04/2018
> [ 515.228905] RIP: 0010:bpf_event_output+0x1f9/0x220
[...]
> Fixes: a5a3a828cd00 ("bpf: add perf event notificaton support for sock_ops")
>
> Effort: BPF
> Signed-off-by: Allan Zhang <allanzhang@google.com>
> Reviewed-by: Stanislav Fomichev <sdf@google.com>
> Reviewed-by: Eric Dumazet <edumazet@google.com>
> ---
LGTM thanks.
Acked-by: John Fastabend <john.fastabend@gmail.com>
^ permalink raw reply
* Re: [PATCH 1/3] docs: fix some broken references
From: Shannon Nelson @ 2019-09-26 16:29 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Linux Doc Mailing List, corbet
Cc: Mauro Carvalho Chehab, linux-kernel, Rob Herring, Mark Rutland,
Daniel Lezcano, Thomas Gleixner, Linus Walleij,
Bartosz Golaszewski, Jean Delvare, Guenter Roeck, Ralf Baechle,
Paul Burton, James Hogan, Saeed Mahameed, Leon Romanovsky,
David S. Miller, Pensando Drivers, Steve French, Paul Walmsley,
Palmer Dabbelt, Albert Ou, devicetree, linux-gpio, linux-hwmon,
linux-mips, netdev, linux-rdma, linux-cifs, samba-technical,
linux-riscv
In-Reply-To: <b87385b2ac6ce6c75df82062fce2976149bbaa6b.1569330078.git.mchehab+samsung@kernel.org>
On 9/24/19 6:01 AM, Mauro Carvalho Chehab wrote:
> There are a number of documentation files that got moved or
> renamed. update their references.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> drivers/net/ethernet/pensando/ionic/ionic_if.h | 4 ++--
Acked-by: Shannon Nelson <snelson@pensando.io>
^ permalink raw reply
* [RFC 00/20] Intel RDMA/IDC Driver series
From: Jeff Kirsher @ 2019-09-26 16:44 UTC (permalink / raw)
To: dledford, jgg, gregkh; +Cc: Jeff Kirsher, netdev, linux-rdma
This series is sent out as an RFC to verify that our implementation of
the MFD subsystem is correct to facilitate inner driver communication
(IDC) between the new "irdma" driver to support Intel's ice and i40e
drivers.
The changes contain the modified ice and i40e driver changes using the
MFD subsystem. It also contains the new irdma driver which is replacing
the i40iw driver and supports both the i40e and ice drivers.
Michael J. Ruhl (1):
RDMA/irdma: Add dynamic tracing for CM
Mustafa Ismail (14):
i40e: Register multi-function device to provide RDMA
RDMA/irdma: Add driver framework definitions
RDMA/irdma: Implement device initialization definitions
RDMA/irdma: Implement HW Admin Queue OPs
RDMA/irdma: Add HMC backing store setup functions
RDMA/irdma: Add privileged UDA queue implementation
RDMA/irdma: Add QoS definitions
RDMA/irdma: Add connection manager
RDMA/irdma: Add PBLE resource manager
RDMA/irdma: Implement device supported verb APIs
RDMA/irdma: Add RoCEv2 UD OP support
RDMA/irdma: Add user/kernel shared libraries
RDMA/irdma: Add miscellaneous utility definitions
RDMA/irdma: Add ABI definitions
Shiraz Saleem (3):
RDMA/irdma: Update MAINTAINERS file
RDMA/irdma: Add Kconfig and Makefile
RDMA/i40iw: Mark i40iw as deprecated
Tony Nguyen (2):
ice: Initialize and register multi-function device to provide RDMA
ice: Implement peer communications
MAINTAINERS | 11 +-
drivers/infiniband/Kconfig | 1 +
drivers/infiniband/hw/Makefile | 1 +
drivers/infiniband/hw/i40iw/Kconfig | 4 +-
drivers/infiniband/hw/i40iw/Makefile | 1 -
drivers/infiniband/hw/i40iw/i40iw.h | 2 +-
drivers/infiniband/hw/irdma/Kconfig | 11 +
drivers/infiniband/hw/irdma/Makefile | 28 +
drivers/infiniband/hw/irdma/cm.c | 4511 +++++++++++++
drivers/infiniband/hw/irdma/cm.h | 415 ++
drivers/infiniband/hw/irdma/ctrl.c | 5958 +++++++++++++++++
drivers/infiniband/hw/irdma/defs.h | 2126 ++++++
drivers/infiniband/hw/irdma/hmc.c | 706 ++
drivers/infiniband/hw/irdma/hmc.h | 219 +
drivers/infiniband/hw/irdma/hw.c | 2564 +++++++
drivers/infiniband/hw/irdma/i40iw_hw.c | 210 +
drivers/infiniband/hw/irdma/i40iw_hw.h | 163 +
drivers/infiniband/hw/irdma/i40iw_if.c | 270 +
drivers/infiniband/hw/irdma/icrdma_hw.c | 75 +
drivers/infiniband/hw/irdma/icrdma_hw.h | 63 +
drivers/infiniband/hw/irdma/irdma.h | 191 +
drivers/infiniband/hw/irdma/irdma_if.c | 436 ++
drivers/infiniband/hw/irdma/main.c | 531 ++
drivers/infiniband/hw/irdma/main.h | 639 ++
drivers/infiniband/hw/irdma/osdep.h | 108 +
drivers/infiniband/hw/irdma/pble.c | 511 ++
drivers/infiniband/hw/irdma/pble.h | 136 +
drivers/infiniband/hw/irdma/protos.h | 96 +
drivers/infiniband/hw/irdma/puda.c | 1693 +++++
drivers/infiniband/hw/irdma/puda.h | 187 +
drivers/infiniband/hw/irdma/status.h | 70 +
drivers/infiniband/hw/irdma/trace.c | 113 +
drivers/infiniband/hw/irdma/trace.h | 4 +
drivers/infiniband/hw/irdma/trace_cm.h | 459 ++
drivers/infiniband/hw/irdma/type.h | 1701 +++++
drivers/infiniband/hw/irdma/uda.c | 391 ++
drivers/infiniband/hw/irdma/uda.h | 65 +
drivers/infiniband/hw/irdma/uda_d.h | 383 ++
drivers/infiniband/hw/irdma/uk.c | 1739 +++++
drivers/infiniband/hw/irdma/user.h | 449 ++
drivers/infiniband/hw/irdma/utils.c | 2333 +++++++
drivers/infiniband/hw/irdma/verbs.c | 4346 ++++++++++++
drivers/infiniband/hw/irdma/verbs.h | 199 +
drivers/infiniband/hw/irdma/ws.c | 396 ++
drivers/infiniband/hw/irdma/ws.h | 40 +
drivers/net/ethernet/intel/Kconfig | 2 +
drivers/net/ethernet/intel/i40e/i40e.h | 3 +-
drivers/net/ethernet/intel/i40e/i40e_client.c | 149 +-
drivers/net/ethernet/intel/ice/Makefile | 1 +
drivers/net/ethernet/intel/ice/ice.h | 18 +
.../net/ethernet/intel/ice/ice_adminq_cmd.h | 33 +
drivers/net/ethernet/intel/ice/ice_common.c | 194 +
drivers/net/ethernet/intel/ice/ice_common.h | 9 +
drivers/net/ethernet/intel/ice/ice_dcb_lib.c | 65 +
drivers/net/ethernet/intel/ice/ice_dcb_lib.h | 3 +
.../net/ethernet/intel/ice/ice_hw_autogen.h | 1 +
drivers/net/ethernet/intel/ice/ice_idc.c | 1326 ++++
drivers/net/ethernet/intel/ice/ice_idc_int.h | 119 +
drivers/net/ethernet/intel/ice/ice_lib.c | 46 +-
drivers/net/ethernet/intel/ice/ice_lib.h | 3 +
drivers/net/ethernet/intel/ice/ice_main.c | 131 +-
drivers/net/ethernet/intel/ice/ice_sched.c | 69 +-
drivers/net/ethernet/intel/ice/ice_switch.c | 27 +
drivers/net/ethernet/intel/ice/ice_switch.h | 4 +
drivers/net/ethernet/intel/ice/ice_type.h | 4 +
.../net/ethernet/intel/ice/ice_virtchnl_pf.c | 25 -
.../linux/net/intel}/i40e_client.h | 21 +
include/linux/net/intel/iidc.h | 355 +
include/uapi/rdma/irdma-abi.h | 159 +
include/uapi/rdma/rdma_user_ioctl_cmds.h | 1 +
70 files changed, 37268 insertions(+), 55 deletions(-)
create mode 100644 drivers/infiniband/hw/irdma/Kconfig
create mode 100644 drivers/infiniband/hw/irdma/Makefile
create mode 100644 drivers/infiniband/hw/irdma/cm.c
create mode 100644 drivers/infiniband/hw/irdma/cm.h
create mode 100644 drivers/infiniband/hw/irdma/ctrl.c
create mode 100644 drivers/infiniband/hw/irdma/defs.h
create mode 100644 drivers/infiniband/hw/irdma/hmc.c
create mode 100644 drivers/infiniband/hw/irdma/hmc.h
create mode 100644 drivers/infiniband/hw/irdma/hw.c
create mode 100644 drivers/infiniband/hw/irdma/i40iw_hw.c
create mode 100644 drivers/infiniband/hw/irdma/i40iw_hw.h
create mode 100644 drivers/infiniband/hw/irdma/i40iw_if.c
create mode 100644 drivers/infiniband/hw/irdma/icrdma_hw.c
create mode 100644 drivers/infiniband/hw/irdma/icrdma_hw.h
create mode 100644 drivers/infiniband/hw/irdma/irdma.h
create mode 100644 drivers/infiniband/hw/irdma/irdma_if.c
create mode 100644 drivers/infiniband/hw/irdma/main.c
create mode 100644 drivers/infiniband/hw/irdma/main.h
create mode 100644 drivers/infiniband/hw/irdma/osdep.h
create mode 100644 drivers/infiniband/hw/irdma/pble.c
create mode 100644 drivers/infiniband/hw/irdma/pble.h
create mode 100644 drivers/infiniband/hw/irdma/protos.h
create mode 100644 drivers/infiniband/hw/irdma/puda.c
create mode 100644 drivers/infiniband/hw/irdma/puda.h
create mode 100644 drivers/infiniband/hw/irdma/status.h
create mode 100644 drivers/infiniband/hw/irdma/trace.c
create mode 100644 drivers/infiniband/hw/irdma/trace.h
create mode 100644 drivers/infiniband/hw/irdma/trace_cm.h
create mode 100644 drivers/infiniband/hw/irdma/type.h
create mode 100644 drivers/infiniband/hw/irdma/uda.c
create mode 100644 drivers/infiniband/hw/irdma/uda.h
create mode 100644 drivers/infiniband/hw/irdma/uda_d.h
create mode 100644 drivers/infiniband/hw/irdma/uk.c
create mode 100644 drivers/infiniband/hw/irdma/user.h
create mode 100644 drivers/infiniband/hw/irdma/utils.c
create mode 100644 drivers/infiniband/hw/irdma/verbs.c
create mode 100644 drivers/infiniband/hw/irdma/verbs.h
create mode 100644 drivers/infiniband/hw/irdma/ws.c
create mode 100644 drivers/infiniband/hw/irdma/ws.h
create mode 100644 drivers/net/ethernet/intel/ice/ice_idc.c
create mode 100644 drivers/net/ethernet/intel/ice/ice_idc_int.h
rename {drivers/net/ethernet/intel/i40e => include/linux/net/intel}/i40e_client.h (92%)
create mode 100644 include/linux/net/intel/iidc.h
create mode 100644 include/uapi/rdma/irdma-abi.h
--
2.21.0
^ permalink raw reply
* [RFC 03/20] i40e: Register multi-function device to provide RDMA
From: Jeff Kirsher @ 2019-09-26 16:45 UTC (permalink / raw)
To: dledford, jgg, gregkh
Cc: Mustafa Ismail, netdev, linux-rdma, Shiraz Saleem, Jeff Kirsher
In-Reply-To: <20190926164519.10471-1-jeffrey.t.kirsher@intel.com>
From: Mustafa Ismail <mustafa.ismail@intel.com>
Register multi-function devices (MFD) for the RDMA platform
function (irdma) driver to bind to. It realizes a single RDMA
driver capable of working with multiple LAN drivers over
multi-generation Intel HW supporting RDMA. There is also no load
ordering dependencies between i40e and irdma.
Summary of changes:
* Support to add/remove MFD devices
* Add 2 new client ops.
* i40e_client_device_register() which is called during RDMA
probe() per PF. Validate client drv OPs and schedule service
task to call open()
* i40e_client_device_unregister() called during RDMA remove()
per PF. Call client close() and release_qvlist.
* The global register/unregister calls exported for i40iw are retained
until i40iw is removed from the kernel.
Signed-off-by: Mustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
drivers/infiniband/hw/i40iw/Makefile | 1 -
drivers/infiniband/hw/i40iw/i40iw.h | 2 +-
drivers/net/ethernet/intel/Kconfig | 1 +
drivers/net/ethernet/intel/i40e/i40e.h | 3 +-
drivers/net/ethernet/intel/i40e/i40e_client.c | 149 ++++++++++++++++--
.../linux/net/intel}/i40e_client.h | 21 +++
6 files changed, 158 insertions(+), 19 deletions(-)
rename {drivers/net/ethernet/intel/i40e => include/linux/net/intel}/i40e_client.h (92%)
diff --git a/drivers/infiniband/hw/i40iw/Makefile b/drivers/infiniband/hw/i40iw/Makefile
index 8942f8229945..34da9eba8a7c 100644
--- a/drivers/infiniband/hw/i40iw/Makefile
+++ b/drivers/infiniband/hw/i40iw/Makefile
@@ -1,5 +1,4 @@
# SPDX-License-Identifier: GPL-2.0
-ccflags-y := -I $(srctree)/drivers/net/ethernet/intel/i40e
obj-$(CONFIG_INFINIBAND_I40IW) += i40iw.o
diff --git a/drivers/infiniband/hw/i40iw/i40iw.h b/drivers/infiniband/hw/i40iw/i40iw.h
index 8feec35f95a7..3197e3536d5c 100644
--- a/drivers/infiniband/hw/i40iw/i40iw.h
+++ b/drivers/infiniband/hw/i40iw/i40iw.h
@@ -57,7 +57,7 @@
#include "i40iw_d.h"
#include "i40iw_hmc.h"
-#include <i40e_client.h>
+#include <linux/net/intel/i40e_client.h>
#include "i40iw_type.h"
#include "i40iw_p.h"
#include <rdma/i40iw-abi.h>
diff --git a/drivers/net/ethernet/intel/Kconfig b/drivers/net/ethernet/intel/Kconfig
index 48ec63f27869..6be57a86fe17 100644
--- a/drivers/net/ethernet/intel/Kconfig
+++ b/drivers/net/ethernet/intel/Kconfig
@@ -241,6 +241,7 @@ config I40E
tristate "Intel(R) Ethernet Controller XL710 Family support"
imply PTP_1588_CLOCK
depends on PCI
+ select MFD_CORE
---help---
This driver supports Intel(R) Ethernet Controller XL710 Family of
devices. For more information on how to identify your adapter, go
diff --git a/drivers/net/ethernet/intel/i40e/i40e.h b/drivers/net/ethernet/intel/i40e/i40e.h
index 2af9f6308f84..ed7c721fdcd4 100644
--- a/drivers/net/ethernet/intel/i40e/i40e.h
+++ b/drivers/net/ethernet/intel/i40e/i40e.h
@@ -38,7 +38,7 @@
#include <net/xdp_sock.h>
#include "i40e_type.h"
#include "i40e_prototype.h"
-#include "i40e_client.h"
+#include <linux/net/intel/i40e_client.h>
#include <linux/avf/virtchnl.h>
#include "i40e_virtchnl_pf.h"
#include "i40e_txrx.h"
@@ -655,6 +655,7 @@ struct i40e_pf {
u16 last_sw_conf_valid_flags;
/* List to keep previous DDP profiles to be rolled back in the future */
struct list_head ddp_old_prof;
+ int peer_idx;
};
/**
diff --git a/drivers/net/ethernet/intel/i40e/i40e_client.c b/drivers/net/ethernet/intel/i40e/i40e_client.c
index e81530ca08d0..13edd8fa9bec 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_client.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_client.c
@@ -6,8 +6,9 @@
#include "i40e.h"
#include "i40e_prototype.h"
-#include "i40e_client.h"
+#include <linux/net/intel/i40e_client.h>
+static struct mfd_cell i40e_mfd_cells[] = ASSIGN_PEER_INFO;
static const char i40e_client_interface_version_str[] = I40E_CLIENT_VERSION_STR;
static struct i40e_client *registered_client;
static LIST_HEAD(i40e_devices);
@@ -30,11 +31,17 @@ static int i40e_client_update_vsi_ctxt(struct i40e_info *ldev,
bool is_vf, u32 vf_id,
u32 flag, u32 valid_flag);
+static int i40e_client_device_register(struct i40e_info *ldev);
+
+static void i40e_client_device_unregister(struct i40e_info *ldev);
+
static struct i40e_ops i40e_lan_ops = {
.virtchnl_send = i40e_client_virtchnl_send,
.setup_qvlist = i40e_client_setup_qvlist,
.request_reset = i40e_client_request_reset,
.update_vsi_ctxt = i40e_client_update_vsi_ctxt,
+ .client_device_register = i40e_client_device_register,
+ .client_device_unregister = i40e_client_device_unregister,
};
/**
@@ -275,6 +282,55 @@ void i40e_client_update_msix_info(struct i40e_pf *pf)
cdev->lan_info.msix_entries = &pf->msix_entries[pf->iwarp_base_vector];
}
+DEFINE_IDA(i40e_peer_index_ida);
+
+int i40e_init_peer_devices(struct i40e_pf *pf)
+{
+ struct i40e_peer_dev_platform_data *platform_data;
+ struct pci_dev *pdev = pf->pdev;
+ int status = 0;
+ int i;
+
+ platform_data = kcalloc(ARRAY_SIZE(i40e_mfd_cells),
+ sizeof(*platform_data), GFP_KERNEL);
+ if (!platform_data)
+ return -ENOMEM;
+
+ for (i = 0; i < ARRAY_SIZE(i40e_mfd_cells); i++) {
+ /* don't create an RDMA MFD device if NIC does not
+ * support RDMA functionality
+ */
+ if (i40e_mfd_cells[i].id == I40E_PEER_RDMA_ID &&
+ !(I40E_FLAG_IWARP_ENABLED & pf->flags)) {
+ dev_warn(&pf->pdev->dev,
+ "RDMA not supported with this config\n");
+ continue;
+ }
+ platform_data[i].ldev = &pf->cinst->lan_info;
+ i40e_mfd_cells[i].platform_data = &platform_data[i];
+ i40e_mfd_cells[i].pdata_size = sizeof(platform_data);
+ }
+
+ status = ida_simple_get(&i40e_peer_index_ida, 0, 0, GFP_KERNEL);
+ if (status < 0) {
+ dev_err(&pdev->dev,
+ "failed to get unique index for device\n");
+ return status;
+ }
+
+ pf->peer_idx = status;
+ status = mfd_add_devices(&pf->pdev->dev, pf->peer_idx,
+ i40e_mfd_cells, ARRAY_SIZE(i40e_mfd_cells),
+ NULL, 0, NULL);
+
+ if (status)
+ dev_err(&pf->pdev->dev,
+ "Failure adding MFD devs for peers: %d\n", status);
+
+ kfree(platform_data);
+ return status;
+}
+
/**
* i40e_client_add_instance - add a client instance struct to the instance list
* @pf: pointer to the board struct
@@ -288,9 +344,6 @@ static void i40e_client_add_instance(struct i40e_pf *pf)
struct netdev_hw_addr *mac = NULL;
struct i40e_vsi *vsi = pf->vsi[pf->lan_vsi];
- if (!registered_client || pf->cinst)
- return;
-
cdev = kzalloc(sizeof(*cdev), GFP_KERNEL);
if (!cdev)
return;
@@ -326,7 +379,11 @@ static void i40e_client_add_instance(struct i40e_pf *pf)
cdev->client = registered_client;
pf->cinst = cdev;
- i40e_client_update_msix_info(pf);
+ cdev->lan_info.msix_count = pf->num_iwarp_msix;
+ cdev->lan_info.msix_entries = &pf->msix_entries[pf->iwarp_base_vector];
+
+ i40e_init_peer_devices(pf);
+ set_bit(__I40E_CLIENT_INSTANCE_NONE, &cdev->state);
}
/**
@@ -347,7 +404,7 @@ void i40e_client_del_instance(struct i40e_pf *pf)
**/
void i40e_client_subtask(struct i40e_pf *pf)
{
- struct i40e_client *client = registered_client;
+ struct i40e_client *client;
struct i40e_client_instance *cdev;
struct i40e_vsi *vsi = pf->vsi[pf->lan_vsi];
int ret = 0;
@@ -361,9 +418,11 @@ void i40e_client_subtask(struct i40e_pf *pf)
test_bit(__I40E_CONFIG_BUSY, pf->state))
return;
- if (!client || !cdev)
+ if (!cdev || !cdev->client)
return;
+ client = cdev->client;
+
/* Here we handle client opens. If the client is down, and
* the netdev is registered, then open the client.
*/
@@ -424,16 +483,8 @@ int i40e_lan_add_device(struct i40e_pf *pf)
pf->hw.pf_id, pf->hw.bus.bus_id,
pf->hw.bus.device, pf->hw.bus.func);
- /* If a client has already been registered, we need to add an instance
- * of it to our new LAN device.
- */
- if (registered_client)
- i40e_client_add_instance(pf);
+ i40e_client_add_instance(pf);
- /* Since in some cases register may have happened before a device gets
- * added, we can schedule a subtask to go initiate the clients if
- * they can be launched at probe time.
- */
set_bit(__I40E_CLIENT_SERVICE_REQUESTED, pf->state);
i40e_service_event_schedule(pf);
@@ -453,6 +504,8 @@ int i40e_lan_del_device(struct i40e_pf *pf)
struct i40e_device *ldev, *tmp;
int ret = -ENODEV;
+ mfd_remove_devices(&pf->pdev->dev);
+
/* First, remove any client instance. */
i40e_client_del_instance(pf);
@@ -733,6 +786,70 @@ static int i40e_client_update_vsi_ctxt(struct i40e_info *ldev,
return err;
}
+static int i40e_client_device_register(struct i40e_info *ldev)
+{
+ struct i40e_client *client;
+ struct i40e_pf *pf;
+
+ if (!ldev) {
+ pr_err("Failed to reg client dev: ldev ptr NULL\n");
+ return -EINVAL;
+ }
+
+ client = ldev->client;
+ pf = ldev->pf;
+ if (!client) {
+ pr_err("Failed to reg client dev: client ptr NULL\n");
+ return -EINVAL;
+ }
+
+ if (!ldev->ops || !client->ops) {
+ pr_err("Failed to reg client dev: client dev peer_ops/ops NULL\n");
+ return -EINVAL;
+ }
+
+ if (client->version.major != I40E_CLIENT_VERSION_MAJOR ||
+ client->version.minor != I40E_CLIENT_VERSION_MINOR) {
+ pr_err("i40e: Failed to register client %s due to mismatched client interface version\n",
+ client->name);
+ pr_err("Client is using version: %02d.%02d.%02d while LAN driver supports %s\n",
+ client->version.major, client->version.minor,
+ client->version.build,
+ i40e_client_interface_version_str);
+ return -EINVAL;
+ }
+
+ pf->cinst->client = ldev->client;
+ set_bit(__I40E_CLIENT_SERVICE_REQUESTED, pf->state);
+ i40e_service_event_schedule(pf);
+
+ return 0;
+}
+
+static void i40e_client_device_unregister(struct i40e_info *ldev)
+{
+ struct i40e_pf *pf = ldev->pf;
+ struct i40e_client_instance *cdev = pf->cinst;
+
+ while (test_and_set_bit(__I40E_SERVICE_SCHED, pf->state))
+ usleep_range(500, 1000);
+
+ if (!cdev || !cdev->client || !cdev->client->ops ||
+ !cdev->client->ops->close) {
+ dev_err(&pf->pdev->dev, "Cannot close client device\n");
+ return;
+ }
+ cdev->client->ops->close(&cdev->lan_info, cdev->client, false);
+ clear_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state);
+ i40e_client_release_qvlist(&cdev->lan_info);
+ pf->cinst->client = NULL;
+ clear_bit(__I40E_SERVICE_SCHED, pf->state);
+}
+
+/* Retain legacy global registration/unregistration calls till i40iw is
+ * deprecated from the kernel. The irdma unified driver does not use these
+ * exported symbols.
+ */
/**
* i40e_register_client - Register a i40e client driver with the L2 driver
* @client: pointer to the i40e_client struct
diff --git a/drivers/net/ethernet/intel/i40e/i40e_client.h b/include/linux/net/intel/i40e_client.h
similarity index 92%
rename from drivers/net/ethernet/intel/i40e/i40e_client.h
rename to include/linux/net/intel/i40e_client.h
index 72994baf4941..916794653991 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_client.h
+++ b/include/linux/net/intel/i40e_client.h
@@ -4,6 +4,11 @@
#ifndef _I40E_CLIENT_H_
#define _I40E_CLIENT_H_
+#include <linux/mfd/core.h>
+
+#define I40E_PEER_RDMA_NAME "i40e_rdma"
+#define I40E_PEER_RDMA_ID PLATFORM_DEVID_AUTO
+
#define I40E_CLIENT_STR_LENGTH 10
/* Client interface version should be updated anytime there is a change in the
@@ -80,6 +85,7 @@ struct i40e_params {
/* Structure to hold Lan device info for a client device */
struct i40e_info {
+ struct platform_device *platform_dev;
struct i40e_client_version version;
u8 lanmac[6];
struct net_device *netdev;
@@ -97,6 +103,7 @@ struct i40e_info {
struct i40e_qvlist_info *qvlist_info;
struct i40e_params params;
struct i40e_ops *ops;
+ struct i40e_client *client;
u16 msix_count; /* number of msix vectors*/
/* Array down below will be dynamically allocated based on msix_count */
@@ -132,6 +139,11 @@ struct i40e_ops {
struct i40e_client *client,
bool is_vf, u32 vf_id,
u32 flag, u32 valid_flag);
+
+ int (*client_device_register)(struct i40e_info *ldev);
+
+ void (*client_device_unregister)(struct i40e_info *ldev);
+
};
struct i40e_client_ops {
@@ -200,4 +212,13 @@ static inline bool i40e_client_is_registered(struct i40e_client *client)
int i40e_register_client(struct i40e_client *client);
int i40e_unregister_client(struct i40e_client *client);
+#define ASSIGN_PEER_INFO \
+{ \
+ { .name = I40E_PEER_RDMA_NAME, .id = I40E_PEER_RDMA_ID }, \
+}
+
+struct i40e_peer_dev_platform_data {
+ struct i40e_info *ldev;
+};
+
#endif /* _I40E_CLIENT_H_ */
--
2.21.0
^ permalink raw reply related
* [RFC 01/20] ice: Initialize and register multi-function device to provide RDMA
From: Jeff Kirsher @ 2019-09-26 16:45 UTC (permalink / raw)
To: dledford, jgg, gregkh
Cc: Tony Nguyen, netdev, linux-rdma, Dave Ertman, Jeff Kirsher
In-Reply-To: <20190926164519.10471-1-jeffrey.t.kirsher@intel.com>
From: Tony Nguyen <anthony.l.nguyen@intel.com>
The RDMA block does not advertise on the PCI bus or any other bus.
Thus the ice driver needs to provide access to the RDMA hardware block
via a virtual bus; utilize a multi-function device to provide this access.
This patch initializes the driver to support RDMA as well as creates
and registers a multi-function device for the RDMA driver to register to.
At this point the driver is fully initialized to register a platform
driver, however, can not yet register as the ops have not been
implemented.
We refer to the interaction of this platform device as Inter-Driver
Communication (IDC); where the platform device is referred to as the peer
device and the platform driver is referred to as the peer driver.
Note that the header file iidc.h has been located under
include/linux/net/intel as this file is a unified header file to be used by
the ice and irdma driver.
Signed-off-by: Dave Ertman <david.m.ertman@intel.com>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
MAINTAINERS | 1 +
drivers/net/ethernet/intel/Kconfig | 1 +
drivers/net/ethernet/intel/ice/Makefile | 1 +
drivers/net/ethernet/intel/ice/ice.h | 16 +
.../net/ethernet/intel/ice/ice_adminq_cmd.h | 1 +
drivers/net/ethernet/intel/ice/ice_common.c | 5 +
drivers/net/ethernet/intel/ice/ice_dcb_lib.c | 31 ++
drivers/net/ethernet/intel/ice/ice_dcb_lib.h | 3 +
.../net/ethernet/intel/ice/ice_hw_autogen.h | 1 +
drivers/net/ethernet/intel/ice/ice_idc.c | 442 ++++++++++++++++++
drivers/net/ethernet/intel/ice/ice_idc_int.h | 81 ++++
drivers/net/ethernet/intel/ice/ice_lib.c | 11 +
drivers/net/ethernet/intel/ice/ice_lib.h | 1 +
drivers/net/ethernet/intel/ice/ice_main.c | 70 ++-
drivers/net/ethernet/intel/ice/ice_type.h | 1 +
include/linux/net/intel/iidc.h | 355 ++++++++++++++
16 files changed, 1019 insertions(+), 2 deletions(-)
create mode 100644 drivers/net/ethernet/intel/ice/ice_idc.c
create mode 100644 drivers/net/ethernet/intel/ice/ice_idc_int.h
create mode 100644 include/linux/net/intel/iidc.h
diff --git a/MAINTAINERS b/MAINTAINERS
index b2326dece28e..07c374fa1975 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8205,6 +8205,7 @@ F: Documentation/networking/device_drivers/intel/ice.rst
F: drivers/net/ethernet/intel/
F: drivers/net/ethernet/intel/*/
F: include/linux/avf/virtchnl.h
+F: include/linux/net/intel/iidc.h
INTEL FRAMEBUFFER DRIVER (excluding 810 and 815)
M: Maik Broemme <mbroemme@libmpq.org>
diff --git a/drivers/net/ethernet/intel/Kconfig b/drivers/net/ethernet/intel/Kconfig
index 154e2e818ec6..48ec63f27869 100644
--- a/drivers/net/ethernet/intel/Kconfig
+++ b/drivers/net/ethernet/intel/Kconfig
@@ -294,6 +294,7 @@ config ICE
tristate "Intel(R) Ethernet Connection E800 Series Support"
default n
depends on PCI_MSI
+ select MFD_CORE
---help---
This driver supports Intel(R) Ethernet Connection E800 Series of
devices. For more information on how to identify your adapter, go
diff --git a/drivers/net/ethernet/intel/ice/Makefile b/drivers/net/ethernet/intel/ice/Makefile
index 9edde960b4f2..2f0ba4aa4957 100644
--- a/drivers/net/ethernet/intel/ice/Makefile
+++ b/drivers/net/ethernet/intel/ice/Makefile
@@ -16,6 +16,7 @@ ice-y := ice_main.o \
ice_lib.o \
ice_txrx.o \
ice_flex_pipe.o \
+ ice_idc.o \
ice_ethtool.o
ice-$(CONFIG_PCI_IOV) += ice_virtchnl_pf.o ice_sriov.o
ice-$(CONFIG_DCB) += ice_dcb.o ice_dcb_lib.o
diff --git a/drivers/net/ethernet/intel/ice/ice.h b/drivers/net/ethernet/intel/ice/ice.h
index 45e100666049..7160556ec55e 100644
--- a/drivers/net/ethernet/intel/ice/ice.h
+++ b/drivers/net/ethernet/intel/ice/ice.h
@@ -32,6 +32,7 @@
#include <linux/if_bridge.h>
#include <linux/ctype.h>
#include <linux/avf/virtchnl.h>
+#include <linux/mfd/core.h>
#include <net/ipv6.h>
#include "ice_devids.h"
#include "ice_type.h"
@@ -40,6 +41,7 @@
#include "ice_switch.h"
#include "ice_common.h"
#include "ice_sched.h"
+#include "ice_idc_int.h"
#include "ice_virtchnl_pf.h"
#include "ice_sriov.h"
@@ -69,6 +71,7 @@ extern const char ice_drv_ver[];
#define ICE_MAX_SMALL_RSS_QS 8
#define ICE_RES_VALID_BIT 0x8000
#define ICE_RES_MISC_VEC_ID (ICE_RES_VALID_BIT - 1)
+#define ICE_RES_RDMA_VEC_ID (ICE_RES_MISC_VEC_ID - 1)
#define ICE_INVAL_Q_INDEX 0xffff
#define ICE_INVAL_VFID 256
@@ -303,11 +306,13 @@ struct ice_q_vector {
enum ice_pf_flags {
ICE_FLAG_FLTR_SYNC,
+ ICE_FLAG_IWARP_ENA,
ICE_FLAG_RSS_ENA,
ICE_FLAG_SRIOV_ENA,
ICE_FLAG_SRIOV_CAPABLE,
ICE_FLAG_DCB_CAPABLE,
ICE_FLAG_DCB_ENA,
+ ICE_FLAG_PEER_ENA,
ICE_FLAG_ADV_FEATURES,
ICE_FLAG_LINK_DOWN_ON_CLOSE_ENA,
ICE_FLAG_NO_MEDIA,
@@ -347,6 +352,9 @@ struct ice_pf {
struct mutex avail_q_mutex; /* protects access to avail_[rx|tx]qs */
struct mutex sw_mutex; /* lock for protecting VSI alloc flow */
u32 msg_enable;
+ /* Total number of MSIX vectors reserved for base driver */
+ u32 num_rdma_msix;
+ u32 rdma_base_vector;
u32 hw_csum_rx_error;
u32 oicr_idx; /* Other interrupt cause MSIX vector index */
u32 num_avail_sw_msix; /* remaining MSIX SW vectors left unclaimed */
@@ -373,6 +381,8 @@ struct ice_pf {
unsigned long tx_timeout_last_recovery;
u32 tx_timeout_recovery_level;
char int_name[ICE_INT_NAME_STR_LEN];
+ struct ice_peer_dev_int **peers;
+ int peer_idx;
u32 sw_int_count;
};
@@ -380,6 +390,8 @@ struct ice_netdev_priv {
struct ice_vsi *vsi;
};
+extern struct ida ice_peer_index_ida;
+
/**
* ice_irq_dynamic_ena - Enable default interrupt generation settings
* @hw: pointer to HW struct
@@ -447,6 +459,10 @@ int ice_set_rss(struct ice_vsi *vsi, u8 *seed, u8 *lut, u16 lut_size);
int ice_get_rss(struct ice_vsi *vsi, u8 *seed, u8 *lut, u16 lut_size);
void ice_fill_rss_lut(u8 *lut, u16 rss_table_size, u16 rss_size);
void ice_print_link_msg(struct ice_vsi *vsi, bool isup);
+int ice_init_peer_devices(struct ice_pf *pf);
+int
+ice_for_each_peer(struct ice_pf *pf, void *data,
+ int (*fn)(struct ice_peer_dev_int *, void *));
#ifdef CONFIG_DCB
int ice_pf_ena_all_vsi(struct ice_pf *pf, bool locked);
void ice_pf_dis_all_vsi(struct ice_pf *pf, bool locked);
diff --git a/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h b/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h
index 023e3d2fee5f..c54e78492395 100644
--- a/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h
+++ b/drivers/net/ethernet/intel/ice/ice_adminq_cmd.h
@@ -108,6 +108,7 @@ struct ice_aqc_list_caps_elem {
#define ICE_AQC_CAPS_TXQS 0x0042
#define ICE_AQC_CAPS_MSIX 0x0043
#define ICE_AQC_CAPS_MAX_MTU 0x0047
+#define ICE_AQC_CAPS_IWARP 0x0051
u8 major_ver;
u8 minor_ver;
diff --git a/drivers/net/ethernet/intel/ice/ice_common.c b/drivers/net/ethernet/intel/ice/ice_common.c
index 3a6b3950eb0e..ed59eec57a52 100644
--- a/drivers/net/ethernet/intel/ice/ice_common.c
+++ b/drivers/net/ethernet/intel/ice/ice_common.c
@@ -1748,6 +1748,11 @@ ice_parse_caps(struct ice_hw *hw, void *buf, u32 cap_count,
"%s: msix_vector_first_id = %d\n", prefix,
caps->msix_vector_first_id);
break;
+ case ICE_AQC_CAPS_IWARP:
+ caps->iwarp = (number == 1);
+ ice_debug(hw, ICE_DBG_INIT,
+ "%s: iwarp = %d\n", prefix, caps->iwarp);
+ break;
case ICE_AQC_CAPS_MAX_MTU:
caps->max_mtu = number;
ice_debug(hw, ICE_DBG_INIT, "%s: max_mtu = %d\n",
diff --git a/drivers/net/ethernet/intel/ice/ice_dcb_lib.c b/drivers/net/ethernet/intel/ice/ice_dcb_lib.c
index dd47869c4ad4..ed639ef5da42 100644
--- a/drivers/net/ethernet/intel/ice/ice_dcb_lib.c
+++ b/drivers/net/ethernet/intel/ice/ice_dcb_lib.c
@@ -613,6 +613,37 @@ ice_tx_prepare_vlan_flags_dcb(struct ice_ring *tx_ring,
return 0;
}
+/**
+ * ice_setup_dcb_qos_info - Setup DCB QoS information
+ * @pf: ptr to ice_pf
+ * @qos_info: QoS param instance
+ */
+void ice_setup_dcb_qos_info(struct ice_pf *pf, struct iidc_qos_params *qos_info)
+{
+ struct ice_dcbx_cfg *dcbx_cfg;
+ u32 up2tc;
+ int i;
+
+ dcbx_cfg = &pf->hw.port_info->local_dcbx_cfg;
+ up2tc = rd32(&pf->hw, PRTDCB_TUP2TC);
+ qos_info->num_apps = dcbx_cfg->numapps;
+
+ qos_info->num_tc = ice_dcb_get_num_tc(dcbx_cfg);
+
+ for (i = 0; i < IIDC_MAX_USER_PRIORITY; i++)
+ qos_info->up2tc[i] = (up2tc >> (i * 3)) & 0x7;
+
+ for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++)
+ qos_info->tc_info[i].rel_bw =
+ dcbx_cfg->etscfg.tcbwtable[i];
+
+ for (i = 0; i < qos_info->num_apps; i++) {
+ qos_info->apps[i].priority = dcbx_cfg->app[i].priority;
+ qos_info->apps[i].prot_id = dcbx_cfg->app[i].prot_id;
+ qos_info->apps[i].selector = dcbx_cfg->app[i].selector;
+ }
+}
+
/**
* ice_dcb_process_lldp_set_mib_change - Process MIB change
* @pf: ptr to ice_pf
diff --git a/drivers/net/ethernet/intel/ice/ice_dcb_lib.h b/drivers/net/ethernet/intel/ice/ice_dcb_lib.h
index 661a6f7bca64..6c0585d1bc97 100644
--- a/drivers/net/ethernet/intel/ice/ice_dcb_lib.h
+++ b/drivers/net/ethernet/intel/ice/ice_dcb_lib.h
@@ -20,6 +20,8 @@ int
ice_tx_prepare_vlan_flags_dcb(struct ice_ring *tx_ring,
struct ice_tx_buf *first);
void
+ice_setup_dcb_qos_info(struct ice_pf *pf, struct iidc_qos_params *qos_info);
+void
ice_dcb_process_lldp_set_mib_change(struct ice_pf *pf,
struct ice_rq_event_info *event);
void ice_vsi_cfg_netdev_tc(struct ice_vsi *vsi, u8 ena_tc);
@@ -57,6 +59,7 @@ ice_tx_prepare_vlan_flags_dcb(struct ice_ring __always_unused *tx_ring,
#define ice_update_dcb_stats(pf) do {} while (0)
#define ice_vsi_cfg_dcb_rings(vsi) do {} while (0)
+#define ice_setup_dcb_qos_info(pf, qos_info) do {} while (0)
#define ice_dcb_process_lldp_set_mib_change(pf, event) do {} while (0)
#define ice_set_cgd_num(tlan_ctx, ring) do {} while (0)
#define ice_vsi_cfg_netdev_tc(vsi, ena_tc) do {} while (0)
diff --git a/drivers/net/ethernet/intel/ice/ice_hw_autogen.h b/drivers/net/ethernet/intel/ice/ice_hw_autogen.h
index 152fbd556e9b..05a71f223c5d 100644
--- a/drivers/net/ethernet/intel/ice/ice_hw_autogen.h
+++ b/drivers/net/ethernet/intel/ice/ice_hw_autogen.h
@@ -55,6 +55,7 @@
#define PRTDCB_GENS 0x00083020
#define PRTDCB_GENS_DCBX_STATUS_S 0
#define PRTDCB_GENS_DCBX_STATUS_M ICE_M(0x7, 0)
+#define PRTDCB_TUP2TC 0x001D26C0
#define GL_PREEXT_L2_PMASK0(_i) (0x0020F0FC + ((_i) * 4))
#define GL_PREEXT_L2_PMASK1(_i) (0x0020F108 + ((_i) * 4))
#define GLFLXP_RXDID_FLAGS(_i, _j) (0x0045D000 + ((_i) * 4 + (_j) * 256))
diff --git a/drivers/net/ethernet/intel/ice/ice_idc.c b/drivers/net/ethernet/intel/ice/ice_idc.c
new file mode 100644
index 000000000000..0850773ee679
--- /dev/null
+++ b/drivers/net/ethernet/intel/ice/ice_idc.c
@@ -0,0 +1,442 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2019, Intel Corporation. */
+
+/* Inter-Driver Communication */
+#include "ice.h"
+#include "ice_lib.h"
+#include "ice_dcb_lib.h"
+
+DEFINE_IDA(ice_peer_index_ida);
+
+static struct mfd_cell ice_mfd_cells[] = ASSIGN_PEER_INFO;
+
+/**
+ * ice_peer_state_change - manage state machine for peer
+ * @peer_dev: pointer to peer's configuration
+ * @new_state: the state requested to transition into
+ * @locked: boolean to determine if call made with mutex held
+ *
+ * This function handles all state transitions for peer devices.
+ * The state machine is as follows:
+ *
+ * +<-----------------------+<-----------------------------+
+ * |<-------+<----------+ +
+ * \/ + + +
+ * INIT --------------> PROBED --> OPENING CLOSED --> REMOVED
+ * + +
+ * OPENED --> CLOSING
+ * + +
+ * PREP_RST +
+ * + +
+ * PREPPED +
+ * +---------->+
+ */
+static void
+ice_peer_state_change(struct ice_peer_dev_int *peer_dev, long new_state,
+ bool locked)
+{
+ struct device *dev;
+
+ dev = bus_find_device_by_name(&platform_bus_type, NULL,
+ peer_dev->plat_name);
+
+ if (!locked)
+ mutex_lock(&peer_dev->peer_dev_state_mutex);
+
+ switch (new_state) {
+ case ICE_PEER_DEV_STATE_INIT:
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_REMOVED,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_INIT, peer_dev->state);
+ dev_dbg(dev,
+ "state transition from _REMOVED to _INIT\n");
+ } else {
+ set_bit(ICE_PEER_DEV_STATE_INIT, peer_dev->state);
+ if (dev)
+ dev_dbg(dev, "state set to _INIT\n");
+ }
+ break;
+ case ICE_PEER_DEV_STATE_PROBED:
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_INIT,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_PROBED, peer_dev->state);
+ dev_dbg(dev,
+ "state transition from _INIT to _PROBED\n");
+ } else if (test_and_clear_bit(ICE_PEER_DEV_STATE_REMOVED,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_PROBED, peer_dev->state);
+ dev_dbg(dev,
+ "state transition from _REMOVED to _PROBED\n");
+ } else if (test_and_clear_bit(ICE_PEER_DEV_STATE_OPENING,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_PROBED, peer_dev->state);
+ dev_dbg(dev,
+ "state transition from _OPENING to _PROBED\n");
+ }
+ break;
+ case ICE_PEER_DEV_STATE_OPENING:
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_PROBED,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_OPENING, peer_dev->state);
+ dev_dbg(dev,
+ "state transition from _PROBED to _OPENING\n");
+ } else if (test_and_clear_bit(ICE_PEER_DEV_STATE_CLOSED,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_OPENING, peer_dev->state);
+ dev_dbg(dev,
+ "state transition from _CLOSED to _OPENING\n");
+ }
+ break;
+ case ICE_PEER_DEV_STATE_OPENED:
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_OPENING,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_OPENED, peer_dev->state);
+ dev_dbg(dev,
+ "state transition from _OPENING to _OPENED\n");
+ }
+ break;
+ case ICE_PEER_DEV_STATE_PREP_RST:
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_OPENED,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_PREP_RST, peer_dev->state);
+ dev_dbg(dev,
+ "state transition from _OPENED to _PREP_RST\n");
+ }
+ break;
+ case ICE_PEER_DEV_STATE_PREPPED:
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_PREP_RST,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_PREPPED, peer_dev->state);
+ dev_dbg(dev,
+ "state transition _PREP_RST to _PREPPED\n");
+ }
+ break;
+ case ICE_PEER_DEV_STATE_CLOSING:
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_OPENED,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_CLOSING, peer_dev->state);
+ dev_dbg(dev,
+ "state transition from _OPENED to _CLOSING\n");
+ }
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_PREPPED,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_CLOSING, peer_dev->state);
+ dev_dbg(dev, "state transition _PREPPED to _CLOSING\n");
+ }
+ /* NOTE - up to peer to handle this situation correctly */
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_PREP_RST,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_CLOSING, peer_dev->state);
+ dev_warn(dev,
+ "WARN: Peer state PREP_RST to _CLOSING\n");
+ }
+ break;
+ case ICE_PEER_DEV_STATE_CLOSED:
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_CLOSING,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_CLOSED, peer_dev->state);
+ dev_dbg(dev,
+ "state transition from _CLOSING to _CLOSED\n");
+ }
+ break;
+ case ICE_PEER_DEV_STATE_REMOVED:
+ if (test_and_clear_bit(ICE_PEER_DEV_STATE_OPENED,
+ peer_dev->state) ||
+ test_and_clear_bit(ICE_PEER_DEV_STATE_CLOSED,
+ peer_dev->state)) {
+ set_bit(ICE_PEER_DEV_STATE_REMOVED, peer_dev->state);
+ dev_dbg(dev,
+ "state from _OPENED/_CLOSED to _REMOVED\n");
+ /* Clear registration for events when peer removed */
+ bitmap_zero(peer_dev->events, ICE_PEER_DEV_STATE_NBITS);
+ }
+ break;
+ default:
+ break;
+ }
+
+ if (!locked)
+ mutex_unlock(&peer_dev->peer_dev_state_mutex);
+}
+
+/**
+ * ice_peer_update_vsi - update the pf_vsi info in peer_dev struct
+ * @peer_dev_int: pointer to peer dev internal struct
+ * @data: opaque pointer - VSI to be updated
+ */
+int ice_peer_update_vsi(struct ice_peer_dev_int *peer_dev_int, void *data)
+{
+ struct ice_vsi *vsi = (struct ice_vsi *)data;
+ struct iidc_peer_dev *peer_dev;
+
+ peer_dev = ice_get_peer_dev(peer_dev_int);
+ if (!peer_dev)
+ return 0;
+
+ peer_dev->pf_vsi_num = vsi->vsi_num;
+ return 0;
+}
+
+/**
+ * ice_for_each_peer - iterate across and call function for each peer dev
+ * @pf: pointer to private board struct
+ * @data: data to pass to function on each call
+ * @fn: pointer to function to call for each peer
+ */
+int
+ice_for_each_peer(struct ice_pf *pf, void *data,
+ int (*fn)(struct ice_peer_dev_int *, void *))
+{
+ int i;
+
+ if (!pf->peers)
+ return 0;
+
+ for (i = 0; i < ARRAY_SIZE(ice_mfd_cells); i++) {
+ struct ice_peer_dev_int *peer_dev_int;
+
+ peer_dev_int = pf->peers[i];
+ if (peer_dev_int) {
+ int ret = fn(peer_dev_int, data);
+
+ if (ret)
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * ice_unreg_peer_device - unregister specified device
+ * @peer_dev_int: ptr to peer device internal
+ * @data: ptr to opaque data
+ *
+ * This function invokes device unregistration, removes ID associated with
+ * the specified device.
+ */
+int
+ice_unreg_peer_device(struct ice_peer_dev_int *peer_dev_int,
+ void __always_unused *data)
+{
+ struct ice_peer_drv_int *peer_drv_int;
+ struct iidc_peer_dev *peer_dev;
+ struct pci_dev *pdev;
+ struct ice_pf *pf;
+
+ if (!peer_dev_int)
+ return 0;
+
+ peer_dev = ice_get_peer_dev(peer_dev_int);
+ pdev = peer_dev->pdev;
+ if (!pdev)
+ return 0;
+
+ pf = pci_get_drvdata(pdev);
+ if (!pf)
+ return 0;
+
+ mfd_remove_devices(&pdev->dev);
+
+ peer_drv_int = peer_dev_int->peer_drv_int;
+
+ if (peer_dev_int->ice_peer_wq) {
+ if (peer_dev_int->peer_prep_task.func)
+ cancel_work_sync(&peer_dev_int->peer_prep_task);
+
+ if (peer_dev_int->peer_close_task.func)
+ cancel_work_sync(&peer_dev_int->peer_close_task);
+ destroy_workqueue(peer_dev_int->ice_peer_wq);
+ }
+
+ devm_kfree(&pf->pdev->dev, peer_drv_int);
+
+ devm_kfree(&pf->pdev->dev, peer_dev_int);
+
+ return 0;
+}
+
+/**
+ * ice_unroll_peer - destroy peers and peer_wq in case of error
+ * @peer_dev_int: ptr to peer device internal struct
+ * @data: ptr to opaque data
+ *
+ * This function releases resources in the event of a failure in creating
+ * peer devices or their individual work_queues. Meant to be called from
+ * a ice_for_each_peer invocation
+ */
+int
+ice_unroll_peer(struct ice_peer_dev_int *peer_dev_int,
+ void __always_unused *data)
+{
+ struct iidc_peer_dev *peer_dev;
+ struct ice_pf *pf;
+
+ peer_dev = ice_get_peer_dev(peer_dev_int);
+ if (!peer_dev)
+ return 0;
+
+ pf = pci_get_drvdata(peer_dev->pdev);
+ if (!pf)
+ return 0;
+
+ if (peer_dev_int->ice_peer_wq)
+ destroy_workqueue(peer_dev_int->ice_peer_wq);
+ devm_kfree(&pf->pdev->dev, peer_dev_int);
+
+ return 0;
+}
+
+/**
+ * ice_reserve_peer_qvector - Reserve vector resources for peer drivers
+ * @pf: board private structure to initialize
+ */
+static int ice_reserve_peer_qvector(struct ice_pf *pf)
+{
+ if (test_bit(ICE_FLAG_IWARP_ENA, pf->flags)) {
+ int index;
+
+ index = ice_get_res(pf, pf->irq_tracker, pf->num_rdma_msix,
+ ICE_RES_RDMA_VEC_ID);
+ if (index < 0)
+ return index;
+ pf->num_avail_sw_msix -= pf->num_rdma_msix;
+ pf->rdma_base_vector = index;
+ }
+ return 0;
+}
+
+/**
+ * ice_init_peer_devices - initializes peer devices
+ * @pf: ptr to ice_pf
+ *
+ * This function initializes peer devices and associates them with specified
+ * pci_dev as their parent.
+ */
+int ice_init_peer_devices(struct ice_pf *pf)
+{
+ struct ice_vsi *vsi = pf->vsi[0];
+ struct pci_dev *pdev = pf->pdev;
+ struct device *dev = &pdev->dev;
+ int status = 0;
+ int i;
+
+ /* Reserve vector resources */
+ status = ice_reserve_peer_qvector(pf);
+ if (status < 0) {
+ dev_err(dev, "failed to reserve vectors for peer drivers\n");
+ return status;
+ }
+ for (i = 0; i < ARRAY_SIZE(ice_mfd_cells); i++) {
+ struct iidc_peer_dev_platform_data *platform_data;
+ struct ice_peer_dev_int *peer_dev_int;
+ struct ice_peer_drv_int *peer_drv_int;
+ struct iidc_qos_params *qos_info;
+ struct msix_entry *entry = NULL;
+ struct iidc_peer_dev *peer_dev;
+ int j;
+
+ peer_dev_int = devm_kzalloc(dev, sizeof(*peer_dev_int),
+ GFP_KERNEL);
+ if (!peer_dev_int)
+ return -ENOMEM;
+ pf->peers[i] = peer_dev_int;
+
+ peer_drv_int = devm_kzalloc(dev, sizeof(*peer_drv_int),
+ GFP_KERNEL);
+ if (!peer_drv_int) {
+ devm_kfree(&pf->pdev->dev, peer_dev_int);
+ return -ENOMEM;
+ }
+
+ peer_dev_int->peer_drv_int = peer_drv_int;
+
+ /* Initialize driver values */
+ for (j = 0; j < IIDC_EVENT_NBITS; j++)
+ bitmap_zero(peer_drv_int->current_events[j].type,
+ IIDC_EVENT_NBITS);
+
+ mutex_init(&peer_dev_int->peer_dev_state_mutex);
+
+ peer_dev = ice_get_peer_dev(peer_dev_int);
+ peer_dev_int->plat_data.peer_dev = peer_dev;
+ platform_data = &peer_dev_int->plat_data;
+ peer_dev->peer_ops = NULL;
+ peer_dev->hw_addr = (u8 __iomem *)pf->hw.hw_addr;
+ peer_dev->ver.major = IIDC_PEER_MAJOR_VER;
+ peer_dev->ver.minor = IIDC_PEER_MINOR_VER;
+ peer_dev->peer_dev_id = ice_mfd_cells[i].id;
+ peer_dev->pf_vsi_num = vsi->vsi_num;
+ peer_dev->netdev = vsi->netdev;
+
+ ice_mfd_cells[i].platform_data = platform_data;
+ ice_mfd_cells[i].pdata_size = sizeof(*platform_data);
+
+ peer_dev_int->ice_peer_wq =
+ alloc_ordered_workqueue("ice_peer_wq_%d", WQ_UNBOUND,
+ i);
+ if (!peer_dev_int->ice_peer_wq)
+ return -ENOMEM;
+
+ peer_dev->pdev = pdev;
+ qos_info = &peer_dev->initial_qos_info;
+
+ /* setup qos_info fields with defaults */
+ qos_info->num_apps = 0;
+ qos_info->num_tc = 1;
+
+ for (j = 0; j < IIDC_MAX_USER_PRIORITY; j++)
+ qos_info->up2tc[j] = 0;
+
+ qos_info->tc_info[0].rel_bw = 100;
+ for (j = 1; j < IEEE_8021QAZ_MAX_TCS; j++)
+ qos_info->tc_info[j].rel_bw = 0;
+
+ /* for DCB, override the qos_info defaults. */
+ ice_setup_dcb_qos_info(pf, qos_info);
+
+ /* make sure peer specific resources such as msix_count and
+ * msix_entries are initialized
+ */
+ switch (ice_mfd_cells[i].id) {
+ case IIDC_PEER_RDMA_ID:
+ if (test_bit(ICE_FLAG_IWARP_ENA, pf->flags)) {
+ peer_dev->msix_count = pf->num_rdma_msix;
+ entry = &pf->msix_entries[pf->rdma_base_vector];
+ }
+ break;
+ default:
+ break;
+ }
+
+ peer_dev->msix_entries = entry;
+ ice_peer_state_change(peer_dev_int, ICE_PEER_DEV_STATE_INIT,
+ false);
+ }
+
+ status = ida_simple_get(&ice_peer_index_ida, 0, 0, GFP_KERNEL);
+ if (status < 0) {
+ dev_err(&pdev->dev,
+ "failed to get unique index for device\n");
+ return status;
+ }
+
+ pf->peer_idx = status;
+
+ status = mfd_add_devices(dev, pf->peer_idx, ice_mfd_cells,
+ ARRAY_SIZE(ice_mfd_cells), NULL, 0, NULL);
+ if (status)
+ dev_err(dev, "Failure adding MFD devs for peers: %d\n", status);
+
+ for (i = 0; i < ARRAY_SIZE(ice_mfd_cells); i++) {
+ snprintf(pf->peers[i]->plat_name, ICE_MAX_PEER_NAME, "%s.%d",
+ ice_mfd_cells[i].name,
+ pf->peer_idx + ice_mfd_cells[i].id);
+ dev = bus_find_device_by_name(&platform_bus_type, NULL,
+ pf->peers[i]->plat_name);
+ dev_dbg(dev, "Peer Created: %s %d\n", pf->peers[i]->plat_name,
+ pf->peer_idx);
+ }
+
+ return status;
+}
diff --git a/drivers/net/ethernet/intel/ice/ice_idc_int.h b/drivers/net/ethernet/intel/ice/ice_idc_int.h
new file mode 100644
index 000000000000..26ecd45faf16
--- /dev/null
+++ b/drivers/net/ethernet/intel/ice/ice_idc_int.h
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef _ICE_IDC_INT_H_
+#define _ICE_IDC_INT_H_
+
+#include <linux/net/intel/iidc.h>
+#include "ice.h"
+
+enum ice_peer_dev_state {
+ ICE_PEER_DEV_STATE_INIT,
+ ICE_PEER_DEV_STATE_PROBED,
+ ICE_PEER_DEV_STATE_OPENING,
+ ICE_PEER_DEV_STATE_OPENED,
+ ICE_PEER_DEV_STATE_PREP_RST,
+ ICE_PEER_DEV_STATE_PREPPED,
+ ICE_PEER_DEV_STATE_CLOSING,
+ ICE_PEER_DEV_STATE_CLOSED,
+ ICE_PEER_DEV_STATE_REMOVED,
+ ICE_PEER_DEV_STATE_API_RDY,
+ ICE_PEER_DEV_STATE_NBITS, /* must be last */
+};
+
+enum ice_peer_drv_state {
+ ICE_PEER_DRV_STATE_MBX_RDY,
+ ICE_PEER_DRV_STATE_NBITS, /* must be last */
+};
+
+struct ice_peer_drv_int {
+ struct iidc_peer_drv *peer_drv;
+
+ /* States associated with peer driver */
+ DECLARE_BITMAP(state, ICE_PEER_DRV_STATE_NBITS);
+
+ /* if this peer_dev is the originator of an event, these are the
+ * most recent events of each type
+ */
+ struct iidc_event current_events[IIDC_EVENT_NBITS];
+};
+
+#define ICE_MAX_PEER_NAME 64
+
+struct ice_peer_dev_int {
+ struct iidc_peer_dev peer_dev;
+ struct ice_peer_drv_int *peer_drv_int; /* driver private structure */
+ char plat_name[ICE_MAX_PEER_NAME];
+ struct iidc_peer_dev_platform_data plat_data;
+
+ /* if this peer_dev is the originator of an event, these are the
+ * most recent events of each type
+ */
+ struct iidc_event current_events[IIDC_EVENT_NBITS];
+ /* Events a peer has registered to be notified about */
+ DECLARE_BITMAP(events, IIDC_EVENT_NBITS);
+
+ /* States associated with peer device */
+ DECLARE_BITMAP(state, ICE_PEER_DEV_STATE_NBITS);
+ struct mutex peer_dev_state_mutex; /* peer_dev state mutex */
+
+ /* per peer workqueue */
+ struct workqueue_struct *ice_peer_wq;
+
+ struct work_struct peer_prep_task;
+ struct work_struct peer_close_task;
+
+ enum iidc_close_reason rst_type;
+};
+
+int ice_peer_update_vsi(struct ice_peer_dev_int *peer_dev_int, void *data);
+int ice_unroll_peer(struct ice_peer_dev_int *peer_dev_int, void *data);
+int ice_unreg_peer_device(struct ice_peer_dev_int *peer_dev_int, void *data);
+
+static inline struct
+iidc_peer_dev *ice_get_peer_dev(struct ice_peer_dev_int *peer_dev_int)
+{
+ if (peer_dev_int)
+ return &peer_dev_int->peer_dev;
+ else
+ return NULL;
+}
+#endif /* !_ICE_IDC_INT_H_ */
diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c
index cc755382df25..5b95efab5f5c 100644
--- a/drivers/net/ethernet/intel/ice/ice_lib.c
+++ b/drivers/net/ethernet/intel/ice/ice_lib.c
@@ -763,6 +763,17 @@ bool ice_is_safe_mode(struct ice_pf *pf)
return !test_bit(ICE_FLAG_ADV_FEATURES, pf->flags);
}
+/*
+ * ice_is_peer_ena
+ * @pf: pointer to the PF struct
+ *
+ * returns true if peer devices/drivers are supported, false otherwise
+ */
+bool ice_is_peer_ena(struct ice_pf *pf)
+{
+ return test_bit(ICE_FLAG_PEER_ENA, pf->flags);
+}
+
/**
* ice_rss_clean - Delete RSS related VSI structures that hold user inputs
* @vsi: the VSI being removed
diff --git a/drivers/net/ethernet/intel/ice/ice_lib.h b/drivers/net/ethernet/intel/ice/ice_lib.h
index 47bc033fff20..578de33493b6 100644
--- a/drivers/net/ethernet/intel/ice/ice_lib.h
+++ b/drivers/net/ethernet/intel/ice/ice_lib.h
@@ -126,4 +126,5 @@ enum ice_status
ice_vsi_cfg_mac_fltr(struct ice_vsi *vsi, const u8 *macaddr, bool set);
bool ice_is_safe_mode(struct ice_pf *pf);
+bool ice_is_peer_ena(struct ice_pf *pf);
#endif /* !_ICE_LIB_H_ */
diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
index 214cd6eca405..706e5f5cadfc 100644
--- a/drivers/net/ethernet/intel/ice/ice_main.c
+++ b/drivers/net/ethernet/intel/ice/ice_main.c
@@ -2321,6 +2321,12 @@ static void ice_set_pf_caps(struct ice_pf *pf)
{
struct ice_hw_func_caps *func_caps = &pf->hw.func_caps;
+ clear_bit(ICE_FLAG_IWARP_ENA, pf->flags);
+ clear_bit(ICE_FLAG_PEER_ENA, pf->flags);
+ if (func_caps->common_cap.iwarp) {
+ set_bit(ICE_FLAG_IWARP_ENA, pf->flags);
+ set_bit(ICE_FLAG_PEER_ENA, pf->flags);
+ }
clear_bit(ICE_FLAG_DCB_CAPABLE, pf->flags);
if (func_caps->common_cap.dcb)
set_bit(ICE_FLAG_DCB_CAPABLE, pf->flags);
@@ -2400,6 +2406,17 @@ static int ice_ena_msix_range(struct ice_pf *pf)
v_budget += needed;
v_left -= needed;
+ /* reserve vectors for RDMA peer driver */
+ if (test_bit(ICE_FLAG_IWARP_ENA, pf->flags)) {
+ /* RDMA peer driver needs one extra to handle misc causes */
+ needed = min_t(int, num_online_cpus(), v_left) + 1;
+ if (v_left < needed)
+ goto no_hw_vecs_left_err;
+ pf->num_rdma_msix = needed;
+ v_budget += needed;
+ v_left -= needed;
+ }
+
pf->msix_entries = devm_kcalloc(&pf->pdev->dev, v_budget,
sizeof(*pf->msix_entries), GFP_KERNEL);
@@ -2425,16 +2442,19 @@ static int ice_ena_msix_range(struct ice_pf *pf)
dev_warn(&pf->pdev->dev,
"not enough OS MSI-X vectors. requested = %d, obtained = %d\n",
v_budget, v_actual);
-/* 2 vectors for LAN (traffic + OICR) */
+/* 2 vectors for LAN and RDMA (traffic + OICR) */
#define ICE_MIN_LAN_VECS 2
+#define ICE_MIN_RDMA_VECS 2
+#define ICE_MIN_VECS (ICE_MIN_LAN_VECS + ICE_MIN_RDMA_VECS)
- if (v_actual < ICE_MIN_LAN_VECS) {
+ if (v_actual < ICE_MIN_VECS) {
/* error if we can't get minimum vectors */
pci_disable_msix(pf->pdev);
err = -ERANGE;
goto msix_err;
} else {
pf->num_lan_msix = ICE_MIN_LAN_VECS;
+ pf->num_rdma_msix = ICE_MIN_RDMA_VECS;
}
}
@@ -2451,6 +2471,7 @@ static int ice_ena_msix_range(struct ice_pf *pf)
err = -ERANGE;
exit_err:
pf->num_lan_msix = 0;
+ pf->num_rdma_msix = 0;
return err;
}
@@ -2960,6 +2981,26 @@ ice_probe(struct pci_dev *pdev, const struct pci_device_id __always_unused *ent)
goto err_alloc_sw_unroll;
}
+ /* init peers only if supported */
+ if (ice_is_peer_ena(pf)) {
+ pf->peers = devm_kcalloc(dev, IIDC_MAX_NUM_PEERS,
+ sizeof(*pf->peers), GFP_KERNEL);
+ if (!pf->peers) {
+ err = -ENOMEM;
+ goto err_init_peer_unroll;
+ }
+
+ err = ice_init_peer_devices(pf);
+ if (err) {
+ dev_err(dev, "Failed to initialize peer devices: 0x%x\n",
+ err);
+ err = -EIO;
+ goto err_init_peer_unroll;
+ }
+ } else {
+ dev_warn(dev, "RDMA is not supported on this device\n");
+ }
+
ice_verify_cacheline_size(pf);
/* If no DDP driven features have to be setup, return here */
@@ -2978,6 +3019,15 @@ ice_probe(struct pci_dev *pdev, const struct pci_device_id __always_unused *ent)
return 0;
+ /* Unwind non-managed device resources, etc. if something failed */
+err_init_peer_unroll:
+ if (ice_is_peer_ena(pf)) {
+ ice_for_each_peer(pf, NULL, ice_unroll_peer);
+ if (pf->peers) {
+ devm_kfree(dev, pf->peers);
+ pf->peers = NULL;
+ }
+ }
err_alloc_sw_unroll:
set_bit(__ICE_SERVICE_DIS, pf->state);
set_bit(__ICE_DOWN, pf->state);
@@ -3019,6 +3069,7 @@ static void ice_remove(struct pci_dev *pdev)
if (test_bit(ICE_FLAG_SRIOV_ENA, pf->flags))
ice_free_vfs(pf);
ice_vsi_release_all(pf);
+ ice_for_each_peer(pf, NULL, ice_unreg_peer_device);
ice_free_irq_msix_misc(pf);
ice_for_each_vsi(pf, i) {
if (!pf->vsi[i])
@@ -3223,6 +3274,7 @@ static int __init ice_module_init(void)
if (status) {
pr_err("failed to register PCI driver, err %d\n", status);
destroy_workqueue(ice_wq);
+ ida_destroy(&ice_peer_index_ida);
}
return status;
@@ -3239,6 +3291,10 @@ static void __exit ice_module_exit(void)
{
pci_unregister_driver(&ice_driver);
destroy_workqueue(ice_wq);
+ /* release all cached layer within ida tree, associated with
+ * ice_peer_index_ida object
+ */
+ ida_destroy(&ice_peer_index_ida);
pr_info("module unloaded\n");
}
module_exit(ice_module_exit);
@@ -4291,6 +4347,16 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
goto err_vsi_rebuild;
}
+ if (ice_is_peer_ena(pf)) {
+ struct ice_vsi *vsi = ice_get_main_vsi(pf);
+
+ if (!vsi) {
+ dev_err(dev, "No PF_VSI to update peer\n");
+ goto err_vsi_rebuild;
+ }
+ ice_for_each_peer(pf, vsi, ice_peer_update_vsi);
+ }
+
if (test_bit(ICE_FLAG_SRIOV_ENA, pf->flags)) {
err = ice_vsi_rebuild_by_type(pf, ICE_VSI_VF);
if (err) {
diff --git a/drivers/net/ethernet/intel/ice/ice_type.h b/drivers/net/ethernet/intel/ice/ice_type.h
index 6667d17a4206..d3e44a220d5d 100644
--- a/drivers/net/ethernet/intel/ice/ice_type.h
+++ b/drivers/net/ethernet/intel/ice/ice_type.h
@@ -174,6 +174,7 @@ struct ice_hw_common_caps {
u8 rss_table_entry_width; /* RSS Entry width in bits */
u8 dcb;
+ u8 iwarp;
};
/* Function specific capabilities */
diff --git a/include/linux/net/intel/iidc.h b/include/linux/net/intel/iidc.h
new file mode 100644
index 000000000000..406169084a95
--- /dev/null
+++ b/include/linux/net/intel/iidc.h
@@ -0,0 +1,355 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef _IIDC_H_
+#define _IIDC_H_
+
+#include <linux/dcbnl.h>
+#include <linux/device.h>
+#include <linux/if_ether.h>
+#include <linux/kernel.h>
+#include <linux/netdevice.h>
+
+/* This major and minor version represent IDC API version information.
+ * During peer driver registration, peer driver specifies major and minor
+ * version information (via. peer_driver:ver_info). It gets checked against
+ * following defines and if mismatch, then peer driver registration
+ * fails and appropriate message gets logged.
+ */
+#define IIDC_PEER_MAJOR_VER 8
+#define IIDC_PEER_MINOR_VER 0
+
+enum iidc_event_type {
+ IIDC_EVENT_LINK_CHANGE,
+ IIDC_EVENT_MTU_CHANGE,
+ IIDC_EVENT_TC_CHANGE,
+ IIDC_EVENT_API_CHANGE,
+ IIDC_EVENT_MBX_CHANGE,
+ IIDC_EVENT_NBITS /* must be last */
+};
+
+enum iidc_res_type {
+ IIDC_INVAL_RES,
+ IIDC_VSI,
+ IIDC_VEB,
+ IIDC_EVENT_Q,
+ IIDC_EGRESS_CMPL_Q,
+ IIDC_CMPL_EVENT_Q,
+ IIDC_ASYNC_EVENT_Q,
+ IIDC_DOORBELL_Q,
+ IIDC_RDMA_QSETS_TXSCHED,
+};
+
+enum iidc_peer_reset_type {
+ IIDC_PEER_PFR,
+ IIDC_PEER_CORER,
+ IIDC_PEER_CORER_SW_CORE,
+ IIDC_PEER_CORER_SW_FULL,
+ IIDC_PEER_GLOBR,
+};
+
+/* reason notified to peer driver as part of event handling */
+enum iidc_close_reason {
+ IIDC_REASON_INVAL,
+ IIDC_REASON_HW_UNRESPONSIVE,
+ IIDC_REASON_INTERFACE_DOWN, /* Administrative down */
+ IIDC_REASON_PEER_DRV_UNREG, /* peer driver getting unregistered */
+ IIDC_REASON_PEER_DEV_UNINIT,
+ IIDC_REASON_GLOBR_REQ,
+ IIDC_REASON_CORER_REQ,
+ IIDC_REASON_EMPR_REQ,
+ IIDC_REASON_PFR_REQ,
+ IIDC_REASON_HW_RESET_PENDING,
+ IIDC_REASON_RECOVERY_MODE,
+ IIDC_REASON_PARAM_CHANGE,
+};
+
+enum iidc_rdma_filter {
+ IIDC_RDMA_FILTER_INVAL,
+ IIDC_RDMA_FILTER_IWARP,
+ IIDC_RDMA_FILTER_ROCEV2,
+ IIDC_RDMA_FILTER_BOTH,
+};
+
+/* This information is needed to handle peer driver registration,
+ * instead of adding more params to peer_drv_registration function,
+ * let's get it thru' peer_drv object.
+ */
+struct iidc_ver_info {
+ u16 major;
+ u16 minor;
+ u64 support;
+};
+
+/* Struct to hold per DCB APP info */
+struct iidc_dcb_app_info {
+ u8 priority;
+ u8 selector;
+ u16 prot_id;
+};
+
+struct iidc_peer_dev;
+
+#define IIDC_MAX_USER_PRIORITY 8
+#define IIDC_MAX_APPS 8
+
+/* Struct to hold per RDMA Qset info */
+struct iidc_rdma_qset_params {
+ u32 teid; /* qset TEID */
+ u16 qs_handle; /* RDMA driver provides this */
+ u16 vsi_id; /* VSI index */
+ u8 tc; /* TC branch the QSet should belong to */
+ u8 reserved[3];
+};
+
+struct iidc_res_base {
+ /* Union for future provision e.g. other res_type */
+ union {
+ struct iidc_rdma_qset_params qsets;
+ } res;
+};
+
+struct iidc_res {
+ /* Type of resource. Filled by peer driver */
+ enum iidc_res_type res_type;
+ /* Count requested by peer driver */
+ u16 cnt_req;
+
+ /* Number of resources allocated. Filled in by callee.
+ * Based on this value, caller to fill up "resources"
+ */
+ u16 res_allocated;
+
+ /* Unique handle to resources allocated. Zero if call fails.
+ * Allocated by callee and for now used by caller for internal
+ * tracking purpose.
+ */
+ u32 res_handle;
+
+ /* Peer driver has to allocate sufficient memory, to accommodate
+ * cnt_requested before calling this function.
+ * Memory has to be zero initialized. It is input/output param.
+ * As a result of alloc_res API, this structures will be populated.
+ */
+ struct iidc_res_base res[1];
+};
+
+struct iidc_qos_info {
+ u64 tc_ctx;
+ u8 rel_bw;
+ u8 prio_type;
+ u8 egress_virt_up;
+ u8 ingress_virt_up;
+};
+
+/* Struct to hold QoS info */
+struct iidc_qos_params {
+ struct iidc_qos_info tc_info[IEEE_8021QAZ_MAX_TCS];
+ u8 up2tc[IIDC_MAX_USER_PRIORITY];
+ u8 vsi_relative_bw;
+ u8 vsi_priority_type;
+ u32 num_apps;
+ struct iidc_dcb_app_info apps[IIDC_MAX_APPS];
+ u8 num_tc;
+};
+
+union iidc_event_info {
+ /* IIDC_EVENT_LINK_CHANGE */
+ struct {
+ struct net_device *lwr_nd;
+ u16 vsi_num; /* HW index of VSI corresponding to lwr ndev */
+ u8 new_link_state;
+ u8 lport;
+ } link_info;
+ /* IIDC_EVENT_MTU_CHANGE */
+ u16 mtu;
+ /* IIDC_EVENT_TC_CHANGE */
+ struct iidc_qos_params port_qos;
+ /* IIDC_EVENT_API_CHANGE */
+ u8 api_rdy;
+ /* IIDC_EVENT_MBX_CHANGE */
+ u8 mbx_rdy;
+};
+
+/* iidc_event elements are to be passed back and forth between the device
+ * owner and the peer drivers. They are to be used to both register/unregister
+ * for event reporting and to report an event (events can be either device
+ * owner generated or peer generated).
+ *
+ * For (un)registering for events, the structure needs to be populated with:
+ * reporter - pointer to the iidc_peer_dev struct of the peer (un)registering
+ * type - bitmap with bits set for event types to (un)register for
+ *
+ * For reporting events, the structure needs to be populated with:
+ * reporter - pointer to peer that generated the event (NULL for ice)
+ * type - bitmap with single bit set for this event type
+ * info - union containing data relevant to this event type
+ */
+struct iidc_event {
+ struct iidc_peer_dev *reporter;
+ DECLARE_BITMAP(type, IIDC_EVENT_NBITS);
+ union iidc_event_info info;
+};
+
+/* Following APIs are implemented by device owner and invoked by peer
+ * drivers
+ */
+struct iidc_ops {
+ /* APIs to allocate resources such as VEB, VSI, Doorbell queues,
+ * completion queues, Tx/Rx queues, etc...
+ */
+ int (*alloc_res)(struct iidc_peer_dev *peer_dev,
+ struct iidc_res *res,
+ int partial_acceptable);
+ int (*free_res)(struct iidc_peer_dev *peer_dev,
+ struct iidc_res *res);
+
+ int (*is_vsi_ready)(struct iidc_peer_dev *peer_dev);
+ int (*peer_register)(struct iidc_peer_dev *peer_dev);
+ int (*peer_unregister)(struct iidc_peer_dev *peer_dev);
+ int (*request_reset)(struct iidc_peer_dev *dev,
+ enum iidc_peer_reset_type reset_type);
+
+ void (*notify_state_change)(struct iidc_peer_dev *dev,
+ struct iidc_event *event);
+
+ /* Notification APIs */
+ void (*reg_for_notification)(struct iidc_peer_dev *dev,
+ struct iidc_event *event);
+ void (*unreg_for_notification)(struct iidc_peer_dev *dev,
+ struct iidc_event *event);
+ int (*update_vsi_filter)(struct iidc_peer_dev *peer_dev,
+ enum iidc_rdma_filter filter, bool enable);
+ int (*vc_send)(struct iidc_peer_dev *peer_dev, u32 vf_id, u8 *msg,
+ u16 len);
+};
+
+/* Following APIs are implemented by peer drivers and invoked by device
+ * owner
+ */
+struct iidc_peer_ops {
+ void (*event_handler)(struct iidc_peer_dev *peer_dev,
+ struct iidc_event *event);
+
+ /* Why we have 'open' and when it is expected to be called:
+ * 1. symmetric set of API w.r.t close
+ * 2. To be invoked form driver initialization path
+ * - call peer_driver:open once device owner is fully
+ * initialized
+ * 3. To be invoked upon RESET complete
+ */
+ int (*open)(struct iidc_peer_dev *peer_dev);
+
+ /* Peer's close function is to be called when the peer needs to be
+ * quiesced. This can be for a variety of reasons (enumerated in the
+ * iidc_close_reason enum struct). A call to close will only be
+ * followed by a call to either remove or open. No IDC calls from the
+ * peer should be accepted until it is re-opened.
+ *
+ * The *reason* parameter is the reason for the call to close. This
+ * can be for any reason enumerated in the iidc_close_reason struct.
+ * It's primary reason is for the peer's bookkeeping and in case the
+ * peer want to perform any different tasks dictated by the reason.
+ */
+ void (*close)(struct iidc_peer_dev *peer_dev,
+ enum iidc_close_reason reason);
+
+ int (*vc_receive)(struct iidc_peer_dev *peer_dev, u32 vf_id, u8 *msg,
+ u16 len);
+ /* tell RDMA peer to prepare for TC change in a blocking call
+ * that will directly precede the change event
+ */
+ void (*prep_tc_change)(struct iidc_peer_dev *peer_dev);
+};
+
+#define IIDC_PEER_RDMA_NAME "iidc_rdma"
+#define IIDC_PEER_RDMA_ID 0x00000010
+#define IIDC_MAX_NUM_PEERS 4
+
+/* The const struct that instantiates peer_dev_id needs to be initialized
+ * in the .c with the macro ASSIGN_PEER_INFO.
+ * For example:
+ * static const struct peer_dev_id peer_dev_ids[] = ASSIGN_PEER_INFO;
+ */
+struct peer_dev_id {
+ char *name;
+ int id;
+};
+
+#define ASSIGN_PEER_INFO \
+{ \
+ { .name = IIDC_PEER_RDMA_NAME, .id = IIDC_PEER_RDMA_ID }, \
+}
+
+#define iidc_peer_priv(x) ((x)->peer_priv)
+
+/* Structure representing peer specific information, each peer using the IIDC
+ * interface will have an instance of this struct dedicated to it.
+ */
+struct iidc_peer_dev {
+ struct iidc_ver_info ver;
+ struct pci_dev *pdev; /* PCI device of corresponding to main function */
+ /* KVA / Linear address corresponding to BAR0 of underlying
+ * pci_device.
+ */
+ u8 __iomem *hw_addr;
+ int peer_dev_id;
+
+ /* Opaque pointer for peer specific data tracking. This memory will
+ * be alloc'd and freed by the peer driver and used for private data
+ * accessible only to the specific peer. It is stored here so that
+ * when this struct is passed to the peer via an IDC call, the data
+ * can be accessed by the peer at that time.
+ * The peers should only retrieve the pointer by the macro:
+ * iidc_peer_priv(struct iidc_peer_dev *)
+ */
+ void *peer_priv;
+
+ u8 ftype; /* PF(false) or VF (true) */
+
+ /* Data VSI created by driver */
+ u16 pf_vsi_num;
+
+ struct iidc_qos_params initial_qos_info;
+ struct net_device *netdev;
+
+ /* Based on peer driver type, this shall point to corresponding MSIx
+ * entries in pf->msix_entries (which were allocated as part of driver
+ * initialization) e.g. for RDMA driver, msix_entries reserved will be
+ * num_online_cpus + 1.
+ */
+ u16 msix_count; /* How many vectors are reserved for this device */
+ struct msix_entry *msix_entries;
+
+ /* Following struct contains function pointers to be initialized
+ * by device owner and called by peer driver
+ */
+ const struct iidc_ops *ops;
+
+ /* Following struct contains function pointers to be initialized
+ * by peer driver and called by device owner
+ */
+ const struct iidc_peer_ops *peer_ops;
+
+ /* Pointer to peer_drv struct to be populated by peer driver */
+ struct iidc_peer_drv *peer_drv;
+};
+
+struct iidc_peer_dev_platform_data {
+ struct iidc_peer_dev *peer_dev;
+};
+
+/* structure representing peer driver
+ * Peer driver to initialize those function ptrs and it will be invoked
+ * by device owner as part of driver_registration via bus infrastructure
+ */
+struct iidc_peer_drv {
+ u16 driver_id;
+#define IIDC_PEER_DEVICE_OWNER 0
+#define IIDC_PEER_RDMA_DRIVER 4
+
+ struct iidc_ver_info ver;
+ const char *name;
+
+};
+#endif /* _IIDC_H_*/
--
2.21.0
^ permalink raw reply related
* [RFC 04/20] RDMA/irdma: Add driver framework definitions
From: Jeff Kirsher @ 2019-09-26 16:45 UTC (permalink / raw)
To: dledford, jgg, gregkh
Cc: Mustafa Ismail, netdev, linux-rdma, Shiraz Saleem, Jeff Kirsher
In-Reply-To: <20190926164519.10471-1-jeffrey.t.kirsher@intel.com>
From: Mustafa Ismail <mustafa.ismail@intel.com>
Register irdma as a platform driver capable of supporting platform
devices from multi-generation RDMA capable Intel HW. Establish the
interface with all supported netdev peer devices and initialize HW.
Signed-off-by: Mustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
---
drivers/infiniband/hw/irdma/i40iw_if.c | 270 +++++++++++
drivers/infiniband/hw/irdma/irdma_if.c | 436 +++++++++++++++++
drivers/infiniband/hw/irdma/main.c | 531 ++++++++++++++++++++
drivers/infiniband/hw/irdma/main.h | 639 +++++++++++++++++++++++++
4 files changed, 1876 insertions(+)
create mode 100644 drivers/infiniband/hw/irdma/i40iw_if.c
create mode 100644 drivers/infiniband/hw/irdma/irdma_if.c
create mode 100644 drivers/infiniband/hw/irdma/main.c
create mode 100644 drivers/infiniband/hw/irdma/main.h
diff --git a/drivers/infiniband/hw/irdma/i40iw_if.c b/drivers/infiniband/hw/irdma/i40iw_if.c
new file mode 100644
index 000000000000..3cddb091acfb
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/i40iw_if.c
@@ -0,0 +1,270 @@
+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+#include <linux/netdevice.h>
+#include <linux/etherdevice.h>
+#include <net/addrconf.h>
+#include "main.h"
+#include "i40iw_hw.h"
+#include <linux/net/intel/i40e_client.h>
+
+/**
+ * i40iw_request_reset - Request a reset
+ * @rf: RDMA PCI function
+ *
+ */
+void i40iw_request_reset(struct irdma_pci_f *rf)
+{
+ struct i40e_info *ldev = (struct i40e_info *)rf->ldev.if_ldev;
+
+ ldev->ops->request_reset(ldev, rf->ldev.if_client, 1);
+}
+
+/**
+ * i40iw_open - client interface operation open for iwarp/uda device
+ * @ldev: LAN device information
+ * @client: iwarp client information, provided during registration
+ *
+ * Called by the LAN driver during the processing of client register
+ * Create device resources, set up queues, pble and hmc objects and
+ * register the device with the ib verbs interface
+ * Return 0 if successful, otherwise return error
+ */
+static int i40iw_open(struct i40e_info *ldev, struct i40e_client *client)
+{
+ struct irdma_l2params l2params = {};
+ struct irdma_device *iwdev = NULL;
+ struct irdma_handler *hdl = NULL;
+ struct irdma_priv_ldev *pldev;
+ u16 last_qset = IRDMA_NO_QSET;
+ struct irdma_sc_dev *dev;
+ struct irdma_pci_f *rf;
+ int err_code = -EIO;
+ u16 qset;
+ int i;
+
+ hdl = irdma_find_handler(ldev->pcidev);
+ if (hdl)
+ return 0;
+
+ hdl = kzalloc((sizeof(*hdl) + sizeof(*iwdev)), GFP_KERNEL);
+ if (!hdl)
+ return -ENOMEM;
+
+ iwdev = (struct irdma_device *)((u8 *)hdl + sizeof(*hdl));
+
+ iwdev->param_wq = alloc_ordered_workqueue("l2params", WQ_MEM_RECLAIM);
+ if (!iwdev->param_wq)
+ goto error;
+
+ rf = &hdl->rf;
+ rf->hdl = hdl;
+ dev = &rf->sc_dev;
+ dev->back_dev = rf;
+ rf->rdma_ver = IRDMA_GEN_1;
+ hdl->platform_dev = ldev->platform_dev;
+ irdma_init_rf_config_params(rf);
+ rf->init_hw = i40iw_init_hw;
+ rf->hw.hw_addr = ldev->hw_addr;
+ rf->pdev = ldev->pcidev;
+ rf->netdev = ldev->netdev;
+ dev->pci_rev = rf->pdev->revision;
+ iwdev->rf = rf;
+ iwdev->hdl = hdl;
+ iwdev->ldev = &rf->ldev;
+ iwdev->init_state = INITIAL_STATE;
+ iwdev->rcv_wnd = IRDMA_CM_DEFAULT_RCV_WND_SCALED;
+ iwdev->rcv_wscale = IRDMA_CM_DEFAULT_RCV_WND_SCALE;
+ iwdev->netdev = ldev->netdev;
+ iwdev->create_ilq = true;
+ iwdev->vsi_num = 0;
+
+ pldev = &rf->ldev;
+ hdl->ldev = pldev;
+ pldev->if_client = client;
+ pldev->if_ldev = ldev;
+ pldev->fn_num = ldev->fid;
+ pldev->ftype = ldev->ftype;
+ pldev->pf_vsi_num = 0;
+ pldev->msix_count = ldev->msix_count;
+ pldev->msix_entries = ldev->msix_entries;
+
+ if (irdma_ctrl_init_hw(rf))
+ goto error;
+
+ l2params.mtu =
+ (ldev->params.mtu) ? ldev->params.mtu : IRDMA_DEFAULT_MTU;
+ for (i = 0; i < I40E_CLIENT_MAX_USER_PRIORITY; i++) {
+ qset = ldev->params.qos.prio_qos[i].qs_handle;
+ l2params.up2tc[i] = ldev->params.qos.prio_qos[i].tc;
+ l2params.qs_handle_list[i] = qset;
+ if (last_qset == IRDMA_NO_QSET)
+ last_qset = qset;
+ else if ((qset != last_qset) && (qset != IRDMA_NO_QSET))
+ iwdev->dcb = true;
+ }
+
+ if (irdma_rt_init_hw(rf, iwdev, &l2params)) {
+ irdma_deinit_ctrl_hw(rf);
+ goto error;
+ }
+
+ irdma_add_handler(hdl);
+ return 0;
+error:
+ kfree(hdl);
+ return err_code;
+}
+
+/**
+ * i40iw_l2params_worker - worker for l2 params change
+ * @work: work pointer for l2 params
+ */
+static void i40iw_l2params_worker(struct work_struct *work)
+{
+ struct l2params_work *dwork =
+ container_of(work, struct l2params_work, work);
+ struct irdma_device *iwdev = dwork->iwdev;
+
+ irdma_change_l2params(&iwdev->vsi, &dwork->l2params);
+ atomic_dec(&iwdev->params_busy);
+ kfree(work);
+}
+
+/**
+ * i40iw_l2param_change - handle qs handles for QoS and MSS change
+ * @ldev: LAN device information
+ * @client: client for parameter change
+ * @params: new parameters from L2
+ */
+static void i40iw_l2param_change(struct i40e_info *ldev,
+ struct i40e_client *client,
+ struct i40e_params *params)
+{
+ struct irdma_l2params *l2params;
+ struct l2params_work *work;
+ struct irdma_device *iwdev;
+ struct irdma_handler *hdl;
+ int i;
+
+ hdl = irdma_find_handler(ldev->pcidev);
+ if (!hdl)
+ return;
+
+ iwdev = (struct irdma_device *)((u8 *)hdl + sizeof(*hdl));
+
+ if (atomic_read(&iwdev->params_busy))
+ return;
+ work = kzalloc(sizeof(*work), GFP_KERNEL);
+ if (!work)
+ return;
+
+ atomic_inc(&iwdev->params_busy);
+ work->iwdev = iwdev;
+ l2params = &work->l2params;
+ for (i = 0; i < I40E_CLIENT_MAX_USER_PRIORITY; i++)
+ l2params->qs_handle_list[i] = params->qos.prio_qos[i].qs_handle;
+
+ l2params->mtu = (params->mtu) ? params->mtu : iwdev->vsi.mtu;
+
+ INIT_WORK(&work->work, i40iw_l2params_worker);
+ queue_work(iwdev->param_wq, &work->work);
+}
+
+/**
+ * i40iw_close - client interface operation close for iwarp/uda device
+ * @ldev: LAN device information
+ * @client: client to close
+ * @reset: flag to indicate close on reset
+ *
+ * Called by the LAN driver during the processing of client unregister
+ * Destroy and clean up the driver resources
+ */
+static void i40iw_close(struct i40e_info *ldev, struct i40e_client *client,
+ bool reset)
+{
+ struct irdma_device *iwdev;
+ struct irdma_handler *hdl;
+ struct irdma_pci_f *rf;
+
+ hdl = irdma_find_handler(ldev->pcidev);
+ if (!hdl)
+ return;
+ rf = &hdl->rf;
+ iwdev = (struct irdma_device *)((u8 *)hdl + sizeof(*hdl));
+
+ if (iwdev->param_wq)
+ destroy_workqueue(iwdev->param_wq);
+
+ if (reset)
+ iwdev->reset = true;
+
+ irdma_deinit_rt_device(iwdev);
+ irdma_deinit_ctrl_hw(rf);
+ irdma_del_handler(irdma_find_handler(ldev->pcidev));
+ kfree(hdl);
+ pr_info("IRDMA hardware deinitialization complete\n");
+}
+
+/* client interface functions */
+static const struct i40e_client_ops i40e_ops = {
+ .open = i40iw_open,
+ .close = i40iw_close,
+ .l2_param_change = i40iw_l2param_change
+};
+
+static struct i40e_client i40iw_client = {
+ .name = "irdma",
+ .ops = &i40e_ops,
+ .version.major = I40E_CLIENT_VERSION_MAJOR,
+ .version.minor = I40E_CLIENT_VERSION_MINOR,
+ .version.build = I40E_CLIENT_VERSION_BUILD,
+ .type = I40E_CLIENT_IWARP,
+};
+
+int i40iw_probe(struct platform_device *pdev)
+{
+ struct i40e_peer_dev_platform_data *pdata =
+ dev_get_platdata(&pdev->dev);
+ struct i40e_info *ldev;
+
+ if (!pdata)
+ return -EINVAL;
+
+ ldev = pdata->ldev;
+
+ if (ldev->version.major != I40E_CLIENT_VERSION_MAJOR ||
+ ldev->version.minor != I40E_CLIENT_VERSION_MINOR) {
+ pr_err("version mismatch:\n");
+ pr_err("expected major ver %d, caller specified major ver %d\n",
+ I40E_CLIENT_VERSION_MAJOR, ldev->version.major);
+ pr_err("expected minor ver %d, caller specified minor ver %d\n",
+ I40E_CLIENT_VERSION_MINOR, ldev->version.minor);
+ return -EINVAL;
+ }
+
+ if (!ldev->ops->client_device_register)
+ return -EINVAL;
+
+ ldev->client = &i40iw_client;
+ ldev->platform_dev = pdev;
+
+ return ldev->ops->client_device_register(ldev);
+}
+
+void i40iw_remove(struct platform_device *pdev)
+{
+ struct i40e_peer_dev_platform_data *pdata =
+ dev_get_platdata(&pdev->dev);
+ struct i40e_info *ldev;
+
+ if (!pdata)
+ return;
+
+ ldev = pdata->ldev;
+
+ if (ldev->ops->client_device_unregister)
+ ldev->ops->client_device_unregister(ldev);
+}
diff --git a/drivers/infiniband/hw/irdma/irdma_if.c b/drivers/infiniband/hw/irdma/irdma_if.c
new file mode 100644
index 000000000000..a0bcb8538a5e
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/irdma_if.c
@@ -0,0 +1,436 @@
+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+#include <linux/net/intel/iidc.h>
+#include "main.h"
+#include "ws.h"
+#include "icrdma_hw.h"
+
+/**
+ * irdma_lan_register_qset - Register qset with LAN driver
+ * @vsi: vsi structure
+ * @tc_node: Traffic class node
+ */
+enum irdma_status_code irdma_lan_register_qset(struct irdma_sc_vsi *vsi,
+ struct irdma_ws_node *tc_node)
+{
+ struct irdma_device *iwdev = vsi->back_vsi;
+ struct iidc_peer_dev *ldev = (struct iidc_peer_dev *)iwdev->ldev->if_ldev;
+ struct iidc_res rdma_qset_res = {};
+ int ret;
+
+ if (ldev->ops->alloc_res) {
+ rdma_qset_res.cnt_req = 1;
+ rdma_qset_res.res_type = IIDC_RDMA_QSETS_TXSCHED;
+ rdma_qset_res.res[0].res.qsets.qs_handle = tc_node->qs_handle;
+ rdma_qset_res.res[0].res.qsets.tc = tc_node->traffic_class;
+ rdma_qset_res.res[0].res.qsets.vsi_id = vsi->vsi_idx;
+ ret = ldev->ops->alloc_res(ldev, &rdma_qset_res, 0);
+ if (ret) {
+ dev_dbg(rfdev_to_dev(vsi->dev),
+ "WS: LAN alloc_res for rdma qset failed.\n");
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ tc_node->l2_sched_node_id = rdma_qset_res.res[0].res.qsets.teid;
+ vsi->qos[tc_node->user_pri].l2_sched_node_id =
+ rdma_qset_res.res[0].res.qsets.teid;
+ }
+
+ return 0;
+}
+
+/**
+ * irdma_lan_unregister_qset - Unregister qset with LAN driver
+ * @vsi: vsi structure
+ * @tc_node: Traffic class node
+ */
+void irdma_lan_unregister_qset(struct irdma_sc_vsi *vsi,
+ struct irdma_ws_node *tc_node)
+{
+ struct irdma_device *iwdev = vsi->back_vsi;
+ struct iidc_peer_dev *ldev = (struct iidc_peer_dev *)iwdev->ldev->if_ldev;
+ struct iidc_res rdma_qset_res = {};
+
+ if (ldev->ops->free_res) {
+ rdma_qset_res.res_allocated = 1;
+ rdma_qset_res.res_type = IIDC_RDMA_QSETS_TXSCHED;
+ rdma_qset_res.res[0].res.qsets.vsi_id = vsi->vsi_idx;
+ rdma_qset_res.res[0].res.qsets.teid = tc_node->l2_sched_node_id;
+ rdma_qset_res.res[0].res.qsets.qs_handle = tc_node->qs_handle;
+
+ if (ldev->ops->free_res(ldev, &rdma_qset_res))
+ dev_dbg(rfdev_to_dev(vsi->dev),
+ "WS: LAN free_res for rdma qset failed.\n");
+ }
+}
+
+/**
+ * irdma_log_invalid_mtu: log warning on invalid mtu
+ * @mtu: maximum tranmission unit
+ */
+static void irdma_log_invalid_mtu(u16 mtu)
+{
+ if (mtu < IRDMA_MIN_MTU_IPV4)
+ pr_warn("Current MTU setting of %d is too low for RDMA traffic. Minimum MTU is 576 for IPv4 and 1280 for IPv6\n",
+ mtu);
+ else if (mtu < IRDMA_MIN_MTU_IPV6)
+ pr_warn("Current MTU setting of %d is too low for IPv6 RDMA traffic, the minimum is 1280\n",
+ mtu);
+}
+
+/**
+ * irdma_prep_tc_change - Prepare for TC changes
+ * @ldev: Peer device structure
+ */
+static void irdma_prep_tc_change(struct iidc_peer_dev *ldev)
+{
+ struct irdma_device *iwdev;
+
+ iwdev = irdma_get_device(ldev->netdev);
+ if (!iwdev)
+ return;
+
+ if (iwdev->vsi.tc_change_pending)
+ goto done;
+
+ iwdev->vsi.tc_change_pending = true;
+ irdma_suspend_qps(&iwdev->vsi);
+
+ /* Wait for all qp's to suspend */
+ wait_event_timeout(iwdev->suspend_wq,
+ !atomic_read(&iwdev->vsi.qp_suspend_reqs),
+ IRDMA_EVENT_TIMEOUT);
+ irdma_ws_reset(&iwdev->vsi);
+done:
+ irdma_put_device(iwdev);
+}
+
+/**
+ * irdma_event_handler - Called by LAN driver to notify events
+ * @ldev: Peer device structure
+ * @event: event from LAN driver
+ */
+static void irdma_event_handler(struct iidc_peer_dev *ldev,
+ struct iidc_event *event)
+{
+ struct irdma_l2params l2params = {};
+ struct irdma_device *iwdev;
+ int i;
+
+ iwdev = irdma_get_device(ldev->netdev);
+ if (!iwdev)
+ return;
+
+ if (test_bit(IIDC_EVENT_LINK_CHANGE, event->type)) {
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "CLNT: LINK_CHANGE event\n");
+ } else if (test_bit(IIDC_EVENT_MTU_CHANGE, event->type)) {
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "CLNT: new MTU = %d\n", event->info.mtu);
+ if (iwdev->vsi.mtu != event->info.mtu) {
+ l2params.mtu = event->info.mtu;
+ l2params.mtu_changed = true;
+ irdma_log_invalid_mtu(l2params.mtu);
+ irdma_change_l2params(&iwdev->vsi, &l2params);
+ }
+ } else if (test_bit(IIDC_EVENT_TC_CHANGE, event->type)) {
+ if (!iwdev->vsi.tc_change_pending)
+ goto done;
+
+ l2params.tc_changed = true;
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev), "CLNT: TC Change\n");
+ iwdev->dcb = event->info.port_qos.num_tc > 1 ? true : false;
+
+ for (i = 0; i < IIDC_MAX_USER_PRIORITY; ++i)
+ l2params.up2tc[i] = event->info.port_qos.up2tc[i];
+ irdma_change_l2params(&iwdev->vsi, &l2params);
+ } else if (test_bit(IIDC_EVENT_API_CHANGE, event->type)) {
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "CLNT: API_CHANGE\n");
+ }
+
+done:
+ irdma_put_device(iwdev);
+}
+
+/**
+ * irdma_open - client interface operation open for RDMA device
+ * @ldev: LAN device information
+ *
+ * Called by the LAN driver during the processing of client
+ * register.
+ */
+static int irdma_open(struct iidc_peer_dev *ldev)
+{
+ struct irdma_l2params l2params = {};
+ enum irdma_status_code status;
+ struct irdma_priv_ldev *pldev;
+ struct iidc_event events = {};
+ struct irdma_device *iwdev;
+ struct irdma_handler *hdl;
+ struct irdma_sc_dev *dev;
+ struct irdma_pci_f *rf;
+ int i;
+
+ hdl = irdma_find_handler(ldev->pdev);
+ if (!hdl)
+ return -ENODEV;
+
+ rf = &hdl->rf;
+ if (rf->init_state != CEQ0_CREATED)
+ return -EINVAL;
+
+ iwdev = kzalloc(sizeof(*iwdev), GFP_KERNEL);
+ if (!iwdev)
+ return -ENOMEM;
+
+ iwdev->hdl = hdl;
+ iwdev->rf = rf;
+ iwdev->ldev = &rf->ldev;
+ pldev = &rf->ldev;
+ pldev->pf_vsi_num = ldev->pf_vsi_num;
+
+ /* Set configfs default values */
+ iwdev->push_mode = 0;
+ iwdev->rcv_wnd = IRDMA_CM_DEFAULT_RCV_WND_SCALED;
+ iwdev->rcv_wscale = IRDMA_CM_DEFAULT_RCV_WND_SCALE;
+
+ dev = &hdl->rf.sc_dev;
+ iwdev->netdev = ldev->netdev;
+ iwdev->create_ilq = true;
+ if (rf->protocol_used == IRDMA_ROCE_PROTOCOL_ONLY) {
+ iwdev->roce_mode = true;
+ iwdev->create_ilq = false;
+ }
+ l2params.mtu = ldev->netdev->mtu;
+
+ l2params.num_tc = ldev->initial_qos_info.num_tc;
+ l2params.num_apps = ldev->initial_qos_info.num_apps;
+ l2params.vsi_prio_type = ldev->initial_qos_info.vsi_priority_type;
+ l2params.vsi_rel_bw = ldev->initial_qos_info.vsi_relative_bw;
+ for (i = 0; i < l2params.num_tc; i++) {
+ l2params.tc_info[i].egress_virt_up =
+ ldev->initial_qos_info.tc_info[i].egress_virt_up;
+ l2params.tc_info[i].ingress_virt_up =
+ ldev->initial_qos_info.tc_info[i].ingress_virt_up;
+ l2params.tc_info[i].prio_type =
+ ldev->initial_qos_info.tc_info[i].prio_type;
+ l2params.tc_info[i].rel_bw =
+ ldev->initial_qos_info.tc_info[i].rel_bw;
+ l2params.tc_info[i].tc_ctx =
+ ldev->initial_qos_info.tc_info[i].tc_ctx;
+ }
+ for (i = 0; i < IIDC_MAX_USER_PRIORITY; i++)
+ l2params.up2tc[i] = ldev->initial_qos_info.up2tc[i];
+
+ iwdev->vsi_num = ldev->pf_vsi_num;
+ ldev->ops->update_vsi_filter(ldev, IIDC_RDMA_FILTER_BOTH, true);
+
+ status = irdma_rt_init_hw(rf, iwdev, &l2params);
+ if (status) {
+ kfree(iwdev);
+ return -EIO;
+ }
+
+ events.reporter = ldev;
+ set_bit(IIDC_EVENT_LINK_CHANGE, events.type);
+ set_bit(IIDC_EVENT_MTU_CHANGE, events.type);
+ set_bit(IIDC_EVENT_TC_CHANGE, events.type);
+ set_bit(IIDC_EVENT_API_CHANGE, events.type);
+
+ if (ldev->ops->reg_for_notification)
+ ldev->ops->reg_for_notification(ldev, &events);
+ dev_info(rfdev_to_dev(dev), "IRDMA VSI Open Successful");
+ init_waitqueue_head(&iwdev->suspend_wq);
+
+ return 0;
+}
+
+/**
+ * irdma_close - client interface operation close for iwarp/uda device
+ * @ldev: LAN device information
+ * @reason: reason for closing
+ *
+ * Called by the LAN driver during the processing of client unregister
+ * Destroy and clean up the driver resources
+ */
+static void irdma_close(struct iidc_peer_dev *ldev, enum iidc_close_reason reason)
+{
+ struct irdma_device *iwdev;
+
+ iwdev = irdma_get_device(ldev->netdev);
+ if (!iwdev)
+ return;
+
+ irdma_put_device(iwdev);
+ if (reason == IIDC_REASON_HW_RESET_PENDING) {
+ iwdev->reset = true;
+ iwdev->rf->reset = true;
+ }
+
+ if (iwdev->init_state >= CEQ0_CREATED)
+ irdma_deinit_rt_device(iwdev);
+
+ kfree(iwdev);
+ ldev->ops->update_vsi_filter(ldev, IIDC_RDMA_FILTER_BOTH, false);
+ pr_info("IRDMA VSI close complete\n");
+}
+
+/**
+ * irdma_deinit_pf - Unrolls PF initializations done during irdma_probe()
+ * @rf: RDMA PCI function
+ */
+static void irdma_deinit_pf(struct irdma_pci_f *rf)
+{
+ if (rf->free_qp_wq)
+ destroy_workqueue(rf->free_qp_wq);
+ if (rf->free_cqbuf_wq)
+ destroy_workqueue(rf->free_cqbuf_wq);
+ irdma_deinit_ctrl_hw(rf);
+ irdma_del_handler(rf->hdl);
+ kfree(rf->hdl);
+}
+
+/**
+ * irdma_remove - GEN_2 device remove()
+ * @pdev: platform device
+ *
+ * Called on module unload.
+ */
+int irdma_remove(struct platform_device *pdev)
+{
+ struct iidc_peer_dev_platform_data *pdata =
+ dev_get_platdata(&pdev->dev);
+ struct iidc_peer_dev *ldev;
+ struct irdma_handler *hdl;
+
+ if (!pdata)
+ return -EINVAL;
+ ldev = pdata->peer_dev;
+
+ hdl = irdma_find_handler(ldev->pdev);
+ if (!hdl)
+ return 0;
+
+ if (ldev->ops->peer_unregister)
+ ldev->ops->peer_unregister(ldev);
+
+ irdma_deinit_pf(&hdl->rf);
+ pr_info("IRDMA hardware deinitialization complete\n");
+
+ return 0;
+}
+
+static const struct iidc_peer_ops irdma_peer_ops = {
+ .close = irdma_close,
+ .event_handler = irdma_event_handler,
+ .open = irdma_open,
+ .prep_tc_change = irdma_prep_tc_change,
+};
+
+static struct iidc_peer_drv irdma_peer_drv = {
+ .driver_id = IIDC_PEER_RDMA_DRIVER,
+ .name = KBUILD_MODNAME,
+ .ver.major = IIDC_PEER_MAJOR_VER,
+ .ver.minor = IIDC_PEER_MINOR_VER,
+};
+
+/**
+ * irdma_probe - GEN_2 device probe()
+ * @pdev: platform device
+ *
+ * Create device resources, set up queues, pble and hmc objects.
+ * Return 0 if successful, otherwise return error
+ */
+int irdma_probe(struct platform_device *pdev)
+{
+ struct iidc_peer_dev_platform_data *pdata =
+ dev_get_platdata(&pdev->dev);
+ struct irdma_priv_ldev *pldev;
+ struct iidc_peer_dev *ldev;
+ struct irdma_handler *hdl;
+ struct irdma_sc_dev *dev;
+ struct irdma_pci_f *rf;
+ int ret;
+
+ if (!pdata)
+ return -EINVAL;
+ ldev = pdata->peer_dev;
+
+ pr_info("probe: ldev=%p, ldev->dev.pdev.bus->number=%d, ldev->netdev=%p\n",
+ ldev, ldev->pdev->bus->number, ldev->netdev);
+
+ if (ldev->ver.major != IIDC_PEER_MAJOR_VER ||
+ ldev->ver.minor != IIDC_PEER_MINOR_VER) {
+ pr_err("version mismatch:\n");
+ pr_err("expected major ver %d, caller specified major ver %d\n",
+ IIDC_PEER_MAJOR_VER, ldev->ver.major);
+ pr_err("expected minor ver %d, caller specified minor ver %d\n",
+ IIDC_PEER_MINOR_VER, ldev->ver.minor);
+ return -EINVAL;
+ }
+
+ hdl = irdma_find_handler(ldev->pdev);
+ if (hdl)
+ return -EBUSY;
+
+ if (!ldev->ops->peer_register)
+ return -EINVAL;
+
+ hdl = kzalloc(sizeof(*hdl), GFP_KERNEL);
+ if (!hdl)
+ return -ENOMEM;
+
+ rf = &hdl->rf;
+ pldev = &rf->ldev;
+ hdl->ldev = pldev;
+ hdl->platform_dev = pdev;
+ rf->hdl = hdl;
+ dev = &rf->sc_dev;
+ dev->back_dev = rf;
+ rf->init_hw = icrdma_init_hw;
+ pldev->if_ldev = ldev;
+ rf->rdma_ver = IRDMA_GEN_2;
+ irdma_init_rf_config_params(rf);
+ dev->pci_rev = ldev->pdev->revision;
+ rf->default_vsi.vsi_idx = ldev->pf_vsi_num;
+ /* save information from ldev to priv_ldev*/
+ pldev->fn_num = PCI_FUNC(ldev->pdev->devfn);
+ rf->hw.hw_addr = ldev->hw_addr;
+ rf->pdev = ldev->pdev;
+ rf->netdev = ldev->netdev;
+ pldev->ftype = ldev->ftype;
+ pldev->msix_count = ldev->msix_count;
+ pldev->msix_entries = ldev->msix_entries;
+ irdma_add_handler(hdl);
+ if (irdma_ctrl_init_hw(rf)) {
+ irdma_del_handler(hdl);
+ kfree(hdl);
+ return -EIO;
+ }
+ ldev->peer_ops = &irdma_peer_ops;
+ ldev->peer_drv = &irdma_peer_drv;
+ ret = ldev->ops->peer_register(ldev);
+ if (ret) {
+ irdma_deinit_pf(rf);
+ return ret;
+ }
+
+ return 0;
+}
+
+/**
+ * icrdma_request_reset - Request a reset
+ * @rf: RDMA PCI function
+ *
+ */
+void icrdma_request_reset(struct irdma_pci_f *rf)
+{
+ struct iidc_peer_dev *ldev = (struct iidc_peer_dev *)rf->ldev.if_ldev;
+
+ if (ldev && ldev->ops && ldev->ops->request_reset)
+ ldev->ops->request_reset(ldev, IIDC_PEER_PFR);
+}
diff --git a/drivers/infiniband/hw/irdma/main.c b/drivers/infiniband/hw/irdma/main.c
new file mode 100644
index 000000000000..3ff29cafb92c
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/main.c
@@ -0,0 +1,531 @@
+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "main.h"
+
+/* Legacy i40iw module parameters */
+static int resource_profile;
+module_param(resource_profile, int, 0644);
+MODULE_PARM_DESC(resource_profile, "Resource Profile: 0=PF only, 1=Weighted VF, 2=Even Distribution");
+
+static int max_rdma_vfs = 32;
+module_param(max_rdma_vfs, int, 0644);
+MODULE_PARM_DESC(max_rdma_vfs, "Maximum VF count: 0-32 32=default");
+
+static int mpa_version = 2;
+module_param(mpa_version, int, 0644);
+MODULE_PARM_DESC(mpa_version, "MPA version: deprecated parameter");
+
+static int push_mode;
+module_param(push_mode, int, 0644);
+MODULE_PARM_DESC(push_mode, "Low latency mode: deprecated parameter");
+
+static int debug;
+module_param(debug, int, 0644);
+MODULE_PARM_DESC(debug, "debug flags: deprecated parameter");
+
+MODULE_ALIAS("i40iw");
+MODULE_AUTHOR("Intel Corporation, <e1000-rdma@lists.sourceforge.net>");
+MODULE_DESCRIPTION("Intel(R) Ethernet Connection RDMA Driver");
+MODULE_LICENSE("Dual BSD/GPL");
+
+LIST_HEAD(irdma_handlers);
+DEFINE_SPINLOCK(irdma_handler_lock);
+
+static struct notifier_block irdma_inetaddr_notifier = {
+ .notifier_call = irdma_inetaddr_event
+};
+
+static struct notifier_block irdma_inetaddr6_notifier = {
+ .notifier_call = irdma_inet6addr_event
+};
+
+static struct notifier_block irdma_net_notifier = {
+ .notifier_call = irdma_net_event
+};
+
+static struct notifier_block irdma_netdevice_notifier = {
+ .notifier_call = irdma_netdevice_event
+};
+
+void irdma_init_rf_config_params(struct irdma_pci_f *rf)
+{
+ struct irdma_dl_priv *dl_priv;
+
+ rf->rsrc_profile = (resource_profile < IRDMA_HMC_PROFILE_EQUAL) ?
+ (u8)resource_profile + IRDMA_HMC_PROFILE_DEFAULT :
+ IRDMA_HMC_PROFILE_DEFAULT;
+ rf->max_rdma_vfs = (rf->rsrc_profile != IRDMA_HMC_PROFILE_DEFAULT) ?
+ max_rdma_vfs : 0;
+ rf->max_ena_vfs = rf->max_rdma_vfs;
+ dl_priv = platform_get_drvdata(rf->hdl->platform_dev);
+ rf->limits_sel = dl_priv->limits_sel;
+ rf->protocol_used = dl_priv->roce_ena ? IRDMA_ROCE_PROTOCOL_ONLY :
+ IRDMA_IWARP_PROTOCOL_ONLY;
+}
+
+/**
+ * irdma_get_device - find a iwdev given a netdev
+ * @netdev: pointer to net_device
+ *
+ * This function takes a reference on ibdev and prevents ib
+ * device deregistration. The caller must call a matching
+ * irdma_put_device.
+ */
+struct irdma_device *irdma_get_device(struct net_device *netdev)
+{
+ struct ib_device *ibdev = ib_device_get_by_netdev(netdev,
+ RDMA_DRIVER_I40IW);
+
+ if (!ibdev)
+ return NULL;
+
+ return to_iwdev(ibdev);
+}
+
+/**
+ * irdma_put_device - release ibdev refcnt
+ * @iwdev: device
+ *
+ * release refcnt on ibdev taken with irdma_get_device.
+ */
+void irdma_put_device(struct irdma_device *iwdev)
+{
+ struct ib_device *ibdev = &iwdev->iwibdev->ibdev;
+
+ ib_device_put(ibdev);
+}
+
+/**
+ * irdma_find_ice_handler - find a handler given a client info
+ * @pdev: pointer to pci dev info
+ */
+struct irdma_handler *irdma_find_handler(struct pci_dev *pdev)
+{
+ struct irdma_handler *hdl;
+ unsigned long flags;
+
+ spin_lock_irqsave(&irdma_handler_lock, flags);
+ list_for_each_entry(hdl, &irdma_handlers, list) {
+ if (hdl->rf.pdev->devfn == pdev->devfn &&
+ hdl->rf.pdev->bus->number == pdev->bus->number) {
+ spin_unlock_irqrestore(&irdma_handler_lock, flags);
+ return hdl;
+ }
+ }
+ spin_unlock_irqrestore(&irdma_handler_lock, flags);
+
+ return NULL;
+}
+
+/**
+ * irdma_add_handler - add a handler to the list
+ * @hdl: handler to be added to the handler list
+ */
+void irdma_add_handler(struct irdma_handler *hdl)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&irdma_handler_lock, flags);
+ list_add(&hdl->list, &irdma_handlers);
+ spin_unlock_irqrestore(&irdma_handler_lock, flags);
+}
+
+/**
+ * irdma_del_handler - delete a handler from the list
+ * @hdl: handler to be deleted from the handler list
+ */
+void irdma_del_handler(struct irdma_handler *hdl)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&irdma_handler_lock, flags);
+ list_del(&hdl->list);
+ spin_unlock_irqrestore(&irdma_handler_lock, flags);
+}
+
+/**
+ * irdma_register_notifiers - register tcp ip notifiers
+ */
+void irdma_register_notifiers(void)
+{
+ register_inetaddr_notifier(&irdma_inetaddr_notifier);
+ register_inet6addr_notifier(&irdma_inetaddr6_notifier);
+ register_netevent_notifier(&irdma_net_notifier);
+ register_netdevice_notifier(&irdma_netdevice_notifier);
+}
+
+void irdma_unregister_notifiers(void)
+{
+ unregister_netevent_notifier(&irdma_net_notifier);
+ unregister_inetaddr_notifier(&irdma_inetaddr_notifier);
+ unregister_inet6addr_notifier(&irdma_inetaddr6_notifier);
+ unregister_netdevice_notifier(&irdma_netdevice_notifier);
+}
+
+/**
+ * irdma_add_ipv6_addr - add ipv6 address to the hw arp table
+ * @iwdev: iwarp device
+ */
+static void irdma_add_ipv6_addr(struct irdma_device *iwdev)
+{
+ struct inet6_ifaddr *ifp, *tmp;
+ struct net_device *ip_dev;
+ struct inet6_dev *idev;
+ u32 local_ipaddr6[4];
+
+ rcu_read_lock();
+ for_each_netdev_rcu(&init_net, ip_dev) {
+ if (((rdma_vlan_dev_vlan_id(ip_dev) < 0xFFFF &&
+ rdma_vlan_dev_real_dev(ip_dev) == iwdev->netdev) ||
+ ip_dev == iwdev->netdev) && ip_dev->flags & IFF_UP) {
+ idev = __in6_dev_get(ip_dev);
+ if (!idev) {
+ dev_err(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "ipv6 inet device not found\n");
+ break;
+ }
+ list_for_each_entry_safe(ifp, tmp, &idev->addr_list,
+ if_list) {
+ dev_info(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "IP=%pI6, vlan_id=%d, MAC=%pM\n",
+ &ifp->addr,
+ rdma_vlan_dev_vlan_id(ip_dev),
+ ip_dev->dev_addr);
+
+ irdma_copy_ip_ntohl(local_ipaddr6,
+ ifp->addr.in6_u.u6_addr32);
+ irdma_manage_arp_cache(iwdev->rf,
+ ip_dev->dev_addr,
+ local_ipaddr6, false,
+ IRDMA_ARP_ADD);
+ }
+ }
+ }
+ rcu_read_unlock();
+}
+
+/**
+ * irdma_add_ipv4_addr - add ipv4 address to the hw arp table
+ * @iwdev: iwarp device
+ */
+static void irdma_add_ipv4_addr(struct irdma_device *iwdev)
+{
+ struct net_device *dev;
+ struct in_device *idev;
+ bool got_lock = true;
+ u32 ip_addr;
+
+ if (!rtnl_trylock())
+ got_lock = false;
+
+ for_each_netdev(&init_net, dev) {
+ if (((rdma_vlan_dev_vlan_id(dev) < 0xFFFF &&
+ rdma_vlan_dev_real_dev(dev) == iwdev->netdev) ||
+ dev == iwdev->netdev) && dev->flags & IFF_UP) {
+ const struct in_ifaddr *ifa;
+
+ idev = in_dev_get(dev);
+ in_dev_for_each_ifa_rtnl(ifa, idev) {
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "CM: IP=%pI4, vlan_id=%d, MAC=%pM\n",
+ &ifa->ifa_address,
+ rdma_vlan_dev_vlan_id(dev),
+ dev->dev_addr);
+
+ ip_addr = ntohl(ifa->ifa_address);
+ irdma_manage_arp_cache(iwdev->rf, dev->dev_addr,
+ &ip_addr, true,
+ IRDMA_ARP_ADD);
+ }
+ in_dev_put(idev);
+ }
+ }
+ if (got_lock)
+ rtnl_unlock();
+}
+
+/**
+ * irdma_add_ip - add ip addresses
+ * @iwdev: iwarp device
+ *
+ * Add ipv4/ipv6 addresses to the arp cache
+ */
+void irdma_add_ip(struct irdma_device *iwdev)
+{
+ irdma_add_ipv4_addr(iwdev);
+ irdma_add_ipv6_addr(iwdev);
+}
+
+/**
+ * irdma_request_reset - Request a reset
+ * @rf: RDMA PCI function
+ *
+ */
+void irdma_request_reset(struct irdma_pci_f *rf)
+{
+ dev_warn(rfdev_to_dev(&rf->sc_dev),
+ "Requesting a a reset from LAN driver\n");
+ if (rf->rdma_ver == IRDMA_GEN_1)
+ i40iw_request_reset(rf);
+ else
+ icrdma_request_reset(rf);
+}
+
+static int irdma_devlink_rsrc_limits_validate(struct devlink *dl, u32 id,
+ union devlink_param_value val,
+ struct netlink_ext_ack *extack)
+{
+ u8 value = val.vu8;
+
+ if (value > 5) {
+ NL_SET_ERR_MSG_MOD(extack, "resource limits selector range is (0-5)");
+ return -ERANGE;
+ }
+
+ return 0;
+}
+
+enum irdma_dl_param_id {
+ IRDMA_DEVLINK_PARAM_ID_BASE = DEVLINK_PARAM_GENERIC_ID_MAX,
+ IRDMA_DEVLINK_PARAM_ID_LIMITS_SELECTOR,
+ IRDMA_DEVLINK_PARAM_ID_ROCE_ENABLE,
+};
+
+static const struct devlink_param irdma_devlink_params[] = {
+ /* Common */
+ DEVLINK_PARAM_DRIVER(IRDMA_DEVLINK_PARAM_ID_LIMITS_SELECTOR,
+ "resource_limits_selector", DEVLINK_PARAM_TYPE_U8,
+ BIT(DEVLINK_PARAM_CMODE_DRIVERINIT),
+ NULL, NULL, irdma_devlink_rsrc_limits_validate),
+#define IRDMA_DL_COMMON_PARAMS_ARRAY_SZ 1
+ /* GEN_2 only */
+ DEVLINK_PARAM_DRIVER(IRDMA_DEVLINK_PARAM_ID_ROCE_ENABLE,
+ "roce_enable", DEVLINK_PARAM_TYPE_BOOL,
+ BIT(DEVLINK_PARAM_CMODE_DRIVERINIT),
+ NULL, NULL, NULL),
+};
+
+static int irdma_devlink_reload(struct devlink *devlink,
+ struct netlink_ext_ack *extack)
+{
+ struct irdma_dl_priv *priv = devlink_priv(devlink);
+ const struct platform_device_id *id = platform_get_device_id(priv->pdev);
+ union devlink_param_value saved_value;
+ u8 gen_ver = id->driver_data;
+ int ret;
+
+ switch (gen_ver) {
+ case IRDMA_GEN_2:
+ irdma_remove(priv->pdev);
+ devlink_param_driverinit_value_get(devlink,
+ IRDMA_DEVLINK_PARAM_ID_ROCE_ENABLE,
+ &saved_value);
+ priv->roce_ena = saved_value.vbool;
+ devlink_param_driverinit_value_get(devlink,
+ IRDMA_DEVLINK_PARAM_ID_LIMITS_SELECTOR,
+ &saved_value);
+ priv->limits_sel = saved_value.vu8;
+ ret = irdma_probe(priv->pdev);
+ break;
+ case IRDMA_GEN_1:
+ i40iw_remove(priv->pdev);
+ devlink_param_driverinit_value_get(devlink,
+ IRDMA_DEVLINK_PARAM_ID_LIMITS_SELECTOR,
+ &saved_value);
+ priv->limits_sel = saved_value.vu8;
+ ret = i40iw_probe(priv->pdev);
+ break;
+ default:
+ ret = -ENODEV;
+ break;
+ }
+
+ return ret;
+}
+
+static const struct devlink_ops irdma_devlink_ops = {
+ .reload_up = irdma_devlink_reload,
+};
+
+static void irdma_devlink_unregister(struct platform_device *pdev)
+{
+ const struct platform_device_id *id = platform_get_device_id(pdev);
+ struct irdma_dl_priv *priv = platform_get_drvdata(pdev);
+ struct devlink *devlink = priv_to_devlink(priv);
+ u8 gen_ver = id->driver_data;
+
+ if (gen_ver == IRDMA_GEN_2)
+ devlink_params_unregister(devlink, irdma_devlink_params,
+ ARRAY_SIZE(irdma_devlink_params));
+ else if (gen_ver == IRDMA_GEN_1)
+ devlink_params_unregister(devlink, irdma_devlink_params,
+ IRDMA_DL_COMMON_PARAMS_ARRAY_SZ);
+
+ devlink_unregister(devlink);
+ devlink_free(devlink);
+}
+
+static int irdma_devlink_register(struct platform_device *pdev)
+{
+ const struct platform_device_id *id = platform_get_device_id(pdev);
+ union devlink_param_value value;
+ u8 gen_ver = id->driver_data;
+ struct irdma_dl_priv *priv;
+ struct devlink *devlink;
+ int ret;
+
+ devlink = devlink_alloc(&irdma_devlink_ops, sizeof(struct irdma_dl_priv));
+ if (!devlink)
+ return -ENOMEM;
+
+ priv = devlink_priv(devlink);
+ priv->pdev = pdev;
+ priv->roce_ena = 0;
+ platform_set_drvdata(pdev, priv);
+
+ ret = devlink_register(devlink, &pdev->dev);
+ if (ret)
+ goto err_dl_free;
+
+ switch (gen_ver) {
+ case IRDMA_GEN_2:
+ priv->limits_sel = 0;
+ ret = devlink_params_register(devlink, irdma_devlink_params,
+ ARRAY_SIZE(irdma_devlink_params));
+ if (!ret) {
+ value.vbool = priv->roce_ena;
+ devlink_param_driverinit_value_set(devlink,
+ IRDMA_DEVLINK_PARAM_ID_ROCE_ENABLE,
+ value);
+ }
+ break;
+ case IRDMA_GEN_1:
+ priv->limits_sel = 2;
+ ret = devlink_params_register(devlink, irdma_devlink_params,
+ IRDMA_DL_COMMON_PARAMS_ARRAY_SZ);
+ break;
+ default:
+ ret = -ENODEV;
+ break;
+ }
+
+ if (ret)
+ goto err_dl_unreg;
+
+ value.vu8 = priv->limits_sel;
+ devlink_param_driverinit_value_set(devlink,
+ IRDMA_DEVLINK_PARAM_ID_LIMITS_SELECTOR,
+ value);
+ devlink_params_publish(devlink);
+
+ return 0;
+
+err_dl_unreg:
+ devlink_unregister(devlink);
+err_dl_free:
+ devlink_free(devlink);
+
+ return ret;
+}
+
+static int irdma_bus_probe(struct platform_device *pdev)
+{
+ const struct platform_device_id *id = platform_get_device_id(pdev);
+ u8 gen_ver = id->driver_data;
+ int ret = -ENODEV;
+
+ ret = irdma_devlink_register(pdev);
+ if (ret)
+ return ret;
+
+ switch (gen_ver) {
+ case IRDMA_GEN_2:
+ ret = irdma_probe(pdev);
+ break;
+ case IRDMA_GEN_1:
+ ret = i40iw_probe(pdev);
+ break;
+ default:
+ break;
+ }
+
+ if (ret)
+ irdma_devlink_unregister(pdev);
+
+ return ret;
+}
+
+static int irdma_bus_remove(struct platform_device *pdev)
+{
+ const struct platform_device_id *id = platform_get_device_id(pdev);
+ u8 gen_ver = id->driver_data;
+
+ switch (gen_ver) {
+ case IRDMA_GEN_2:
+ irdma_remove(pdev);
+ break;
+ case IRDMA_GEN_1:
+ i40iw_remove(pdev);
+ break;
+ default:
+ break;
+ }
+
+ irdma_devlink_unregister(pdev);
+
+ return 0;
+}
+
+static const struct platform_device_id irdma_platform_id_table[] = {
+ {"ice_rdma", IRDMA_GEN_2},
+ {"i40e_rdma", IRDMA_GEN_1},
+ {},
+};
+
+MODULE_DEVICE_TABLE(platform, irdma_platform_id_table);
+
+static struct platform_driver irdma_pdriver = {
+ .probe = irdma_bus_probe,
+ .remove = irdma_bus_remove,
+ .id_table = irdma_platform_id_table,
+ .driver = {
+ .name = "irdma",
+ .owner = THIS_MODULE,
+ },
+};
+
+/**
+ * irdma_init_module - driver initialization function
+ *
+ * First function to call when the driver is loaded
+ * Register the driver as ice client and port mapper client
+ */
+static int __init irdma_init_module(void)
+{
+ int ret;
+
+ ret = platform_driver_register(&irdma_pdriver);
+ if (ret) {
+ pr_err("Failed irdma platform_driver_register()\n");
+ return ret;
+ }
+ irdma_register_notifiers();
+
+ return 0;
+}
+
+/**
+ * irdma_exit_module - driver exit clean up function
+ *
+ * The function is called just before the driver is unloaded
+ * Unregister the driver as ice client and port mapper client
+ */
+static void __exit irdma_exit_module(void)
+{
+ irdma_unregister_notifiers();
+ platform_driver_unregister(&irdma_pdriver);
+}
+
+module_init(irdma_init_module);
+module_exit(irdma_exit_module);
diff --git a/drivers/infiniband/hw/irdma/main.h b/drivers/infiniband/hw/irdma/main.h
new file mode 100644
index 000000000000..8b57f77cdf72
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/main.h
@@ -0,0 +1,639 @@
+/* SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef IRDMA_MAIN_H
+#define IRDMA_MAIN_H
+
+#include <linux/ip.h>
+#include <linux/tcp.h>
+#include <linux/if_vlan.h>
+#include <net/addrconf.h>
+#include <net/netevent.h>
+#include <net/devlink.h>
+#include <linux/netdevice.h>
+#include <linux/inetdevice.h>
+#include <linux/spinlock.h>
+#include <linux/kernel.h>
+#include <linux/delay.h>
+#include <linux/pci.h>
+#include <linux/dma-mapping.h>
+#include <linux/workqueue.h>
+#include <linux/slab.h>
+#include <linux/io.h>
+#include <linux/crc32c.h>
+#include <linux/kthread.h>
+#include <linux/platform_device.h>
+#include <rdma/ib_smi.h>
+#include <rdma/ib_verbs.h>
+#include <rdma/ib_pack.h>
+#include <rdma/rdma_cm.h>
+#include <rdma/iw_cm.h>
+#include <crypto/hash.h>
+#include "status.h"
+#include "osdep.h"
+#include "defs.h"
+#include "hmc.h"
+#include "type.h"
+#include "protos.h"
+#include "pble.h"
+#include "verbs.h"
+#include "cm.h"
+#include "user.h"
+#include "puda.h"
+#include <rdma/irdma-abi.h>
+
+extern struct list_head irdma_handlers;
+extern spinlock_t irdma_handler_lock;
+
+#define IRDMA_FW_VER_DEFAULT 2
+#define IRDMA_HW_VER 2
+
+#define IRDMA_ARP_ADD 1
+#define IRDMA_ARP_DELETE 2
+#define IRDMA_ARP_RESOLVE 3
+
+#define IRDMA_MACIP_ADD 1
+#define IRDMA_MACIP_DELETE 2
+
+#define IW_CCQ_SIZE (IRDMA_CQP_SW_SQSIZE_2048 + 1)
+#define IW_CEQ_SIZE 2048
+#define IW_AEQ_SIZE 2048
+
+#define RX_BUF_SIZE (1536 + 8)
+#define IW_REG0_SIZE (4 * 1024)
+#define IW_TX_TIMEOUT (6 * HZ)
+#define IW_FIRST_QPN 1
+
+#define IW_SW_CONTEXT_ALIGN 1024
+
+#define MAX_DPC_ITERATIONS 128
+
+#define IRDMA_EVENT_TIMEOUT 100000
+#define IRDMA_VCHNL_EVENT_TIMEOUT 100000
+
+#define IRDMA_NO_QSET 0xffff
+
+#define IW_CFG_FPM_QP_COUNT 32768
+#define IRDMA_MAX_PAGES_PER_FMR 512
+#define IRDMA_MIN_PAGES_PER_FMR 1
+#define IRDMA_CQP_COMPL_RQ_WQE_FLUSHED 2
+#define IRDMA_CQP_COMPL_SQ_WQE_FLUSHED 3
+
+#define IRDMA_Q_TYPE_PE_AEQ 0x80
+#define IRDMA_Q_INVALID_IDX 0xffff
+#define IRDMA_REM_ENDPOINT_TRK_QPID 3
+
+#define IRDMA_DRV_OPT_ENA_MPA_VER_0 0x00000001
+#define IRDMA_DRV_OPT_DISABLE_MPA_CRC 0x00000002
+#define IRDMA_DRV_OPT_DISABLE_FIRST_WRITE 0x00000004
+#define IRDMA_DRV_OPT_DISABLE_INTF 0x00000008
+#define IRDMA_DRV_OPT_ENA_MSI 0x00000010
+#define IRDMA_DRV_OPT_DUAL_LOGICAL_PORT 0x00000020
+#define IRDMA_DRV_OPT_NO_INLINE_DATA 0x00000080
+#define IRDMA_DRV_OPT_DISABLE_INT_MOD 0x00000100
+#define IRDMA_DRV_OPT_DISABLE_VIRT_WQ 0x00000200
+#define IRDMA_DRV_OPT_ENA_PAU 0x00000400
+#define IRDMA_DRV_OPT_MCAST_LOGPORT_MAP 0x00000800
+
+#define IW_HMC_OBJ_TYPE_NUM ARRAY_SIZE(iw_hmc_obj_types)
+
+enum init_completion_state {
+ INVALID_STATE = 0,
+ INITIAL_STATE,
+ CQP_CREATED,
+ HMC_OBJS_CREATED,
+ CCQ_CREATED,
+ AEQ_CREATED,
+ CEQ0_CREATED, /* Last state of probe */
+ CEQS_CREATED,
+ ILQ_CREATED,
+ IEQ_CREATED,
+ PBLE_CHUNK_MEM,
+ IP_ADDR_REGISTERED,
+ RDMA_DEV_REGISTERED, /* Last state of open */
+};
+
+struct irdma_rsrc_limits {
+ u32 qplimit;
+ u32 mrlimit;
+ u32 cqlimit;
+};
+
+struct irdma_cqp_compl_info {
+ u32 op_ret_val;
+ u16 maj_err_code;
+ u16 min_err_code;
+ bool error;
+ u8 op_code;
+};
+
+struct irdma_cqp_request {
+ struct cqp_cmds_info info;
+ wait_queue_head_t waitq;
+ struct list_head list;
+ atomic_t refcount;
+ void (*callback_fcn)(struct irdma_cqp_request *cqp_request);
+ void *param;
+ struct irdma_cqp_compl_info compl_info;
+ bool waiting;
+ bool request_done;
+ bool dynamic;
+};
+
+struct irdma_cqp {
+ struct irdma_sc_cqp sc_cqp;
+ spinlock_t req_lock; /* protect CQP request list */
+ spinlock_t compl_lock; /* protect CQP completion processing */
+ wait_queue_head_t waitq;
+ wait_queue_head_t remove_wq;
+ struct irdma_dma_mem sq;
+ struct irdma_dma_mem host_ctx;
+ u64 *scratch_array;
+ struct irdma_cqp_request *cqp_requests;
+ struct list_head cqp_avail_reqs;
+ struct list_head cqp_pending_reqs;
+ struct task_struct *cqp_compl_thread;
+ struct semaphore cqp_compl_sem;
+};
+
+struct irdma_ccq {
+ struct irdma_sc_cq sc_cq;
+ struct irdma_dma_mem mem_cq;
+ struct irdma_dma_mem shadow_area;
+};
+
+struct irdma_ceq {
+ struct irdma_sc_ceq sc_ceq;
+ struct irdma_dma_mem mem;
+ u32 irq;
+ u32 msix_idx;
+ struct irdma_pci_f *rf;
+ struct tasklet_struct dpc_tasklet;
+};
+
+struct irdma_aeq {
+ struct irdma_sc_aeq sc_aeq;
+ struct irdma_dma_mem mem;
+};
+
+struct irdma_arp_entry {
+ u32 ip_addr[4];
+ u8 mac_addr[ETH_ALEN];
+};
+
+struct irdma_msix_vector {
+ u32 idx;
+ u32 irq;
+ u32 cpu_affinity;
+ u32 ceq_id;
+ cpumask_t mask;
+};
+
+struct l2params_work {
+ struct work_struct work;
+ struct irdma_device *iwdev;
+ struct irdma_l2params l2params;
+};
+
+struct virtchnl_work {
+ struct work_struct work;
+ union {
+ struct irdma_cqp_request *cqp_request;
+ struct irdma_virtchnl_work_info work_info;
+ };
+};
+
+struct irdma_mc_table_info {
+ bool ipv4_valid;
+ u32 mgn;
+ u32 dest_ip[4];
+ bool lan_fwd;
+};
+
+struct mc_table_list {
+ struct list_head list;
+ struct irdma_mc_table_info mc_info;
+ struct irdma_mcast_grp_info mc_grp_ctx;
+};
+
+struct irdma_qv_info {
+ u32 v_idx; /* msix_vector */
+ u16 ceq_idx;
+ u16 aeq_idx;
+ u8 itr_idx;
+};
+
+struct irdma_qvlist_info {
+ u32 num_vectors;
+ struct irdma_qv_info qv_info[1];
+};
+
+struct irdma_priv_ldev {
+ unsigned int fn_num;
+ bool ftype;
+ u16 pf_vsi_num;
+ u16 msix_count;
+ struct msix_entry *msix_entries;
+ void *if_client;
+ void *if_ldev;
+};
+
+struct irdma_dl_priv {
+ struct platform_device *pdev;
+ bool roce_ena;
+ u8 limits_sel;
+};
+
+struct irdma_pci_f {
+ bool ooo;
+ bool reset;
+ bool rsrc_created;
+ bool stop_cqp_thread;
+ bool msix_shared;
+ u8 rsrc_profile;
+ u8 max_rdma_vfs;
+ u8 max_ena_vfs;
+ u8 *hmc_info_mem;
+ u8 *mem_rsrc;
+ u8 rdma_ver;
+ enum irdma_protocol_used protocol_used;
+ u32 sd_type;
+ u32 msix_count;
+ u32 max_mr;
+ u32 max_qp;
+ u32 max_cq;
+ u32 max_ah;
+ u32 next_ah;
+ u32 max_mcg;
+ u32 next_mcg;
+ u32 max_pd;
+ u32 next_qp;
+ u32 next_cq;
+ u32 next_pd;
+ u32 max_mr_size;
+ u32 max_cqe;
+ u32 mr_stagmask;
+ u32 used_pds;
+ u32 used_cqs;
+ u32 used_mrs;
+ u32 used_qps;
+ u32 arp_table_size;
+ u32 next_arp_index;
+ u32 ceqs_count;
+ u32 next_ws_node_id;
+ u32 max_ws_node_id;
+ u32 limits_sel;
+ unsigned long *allocated_ws_nodes;
+ unsigned long *allocated_qps;
+ unsigned long *allocated_cqs;
+ unsigned long *allocated_mrs;
+ unsigned long *allocated_pds;
+ unsigned long *allocated_mcgs;
+ unsigned long *allocated_ahs;
+ unsigned long *allocated_arps;
+ enum init_completion_state init_state;
+ struct irdma_sc_dev sc_dev;
+ struct irdma_priv_ldev ldev;
+ struct irdma_handler *hdl;
+ struct pci_dev *pdev;
+ struct net_device *netdev;
+ struct irdma_hw hw;
+ struct irdma_cqp cqp;
+ struct irdma_ccq ccq;
+ struct irdma_aeq aeq;
+ struct irdma_ceq *ceqlist;
+ struct irdma_hmc_pble_rsrc *pble_rsrc;
+ struct irdma_arp_entry *arp_table;
+ spinlock_t arp_lock; /*protect ARP table access*/
+ spinlock_t rsrc_lock; /* protect HW resource array access */
+ spinlock_t qptable_lock; /*protect QP table access*/
+ struct irdma_qp **qp_table;
+ spinlock_t qh_list_lock; /* protect mc_qht_list */
+ struct mc_table_list mc_qht_list;
+ struct irdma_msix_vector *iw_msixtbl;
+ struct irdma_qvlist_info *iw_qvlist;
+ struct tasklet_struct dpc_tasklet;
+ struct irdma_dma_mem obj_mem;
+ struct irdma_dma_mem obj_next;
+ atomic_t vchnl_msgs;
+ wait_queue_head_t vchnl_waitq;
+ struct workqueue_struct *free_qp_wq;
+ struct workqueue_struct *free_cqbuf_wq;
+ struct virtchnl_work virtchnl_w[IRDMA_MAX_PE_ENA_VF_COUNT];
+ struct irdma_sc_vsi default_vsi;
+ void *back_fcn;
+ void (*init_hw)(struct irdma_sc_dev *dev);
+};
+
+struct irdma_device {
+ struct irdma_ib_device *iwibdev;
+ struct irdma_pci_f *rf;
+ struct irdma_priv_ldev *ldev;
+ struct net_device *netdev;
+ struct irdma_handler *hdl;
+ struct irdma_sc_vsi vsi;
+ struct irdma_cm_core cm_core;
+ bool roce_mode;
+ u32 vendor_id;
+ u32 vendor_part_id;
+ u32 device_cap_flags;
+ u32 push_mode;
+ u32 rcv_wnd;
+ u16 mac_ip_table_idx;
+ u8 rcv_wscale;
+ u16 vsi_num;
+ bool create_ilq;
+ u8 iw_status;
+ struct tasklet_struct dpc_tasklet;
+ enum init_completion_state init_state;
+ bool dcb;
+ bool reset;
+ wait_queue_head_t suspend_wq;
+ struct workqueue_struct *param_wq;
+ atomic_t params_busy;
+};
+
+struct irdma_ib_device {
+ struct ib_device ibdev;
+ struct irdma_device *iwdev;
+};
+
+struct irdma_handler {
+ struct list_head list;
+ struct irdma_pci_f rf;
+ struct irdma_priv_ldev *ldev;
+ struct platform_device *platform_dev;
+ bool shared_res_created;
+};
+
+/***********************************************************/
+/**
+ * to_iwdev - get device
+ * @ibdev: ib device
+ **/
+static inline struct irdma_device *to_iwdev(struct ib_device *ibdev)
+{
+ return container_of(ibdev, struct irdma_ib_device, ibdev)->iwdev;
+}
+
+/**
+ * to_ucontext - get user context
+ * @ibucontext: ib user context
+ **/
+static inline struct irdma_ucontext *to_ucontext(struct ib_ucontext *ibucontext)
+{
+ return container_of(ibucontext, struct irdma_ucontext, ibucontext);
+}
+
+/**
+ * to_iwpd - get protection domain
+ * @ibpd: ib pd
+ **/
+static inline struct irdma_pd *to_iwpd(struct ib_pd *ibpd)
+{
+ return container_of(ibpd, struct irdma_pd, ibpd);
+}
+
+/**
+ * to_iwah - get device ah
+ * @ibdev: ib ah
+ **/
+static inline struct irdma_ah *to_iwah(struct ib_ah *ibah)
+{
+ return container_of(ibah, struct irdma_ah, ibah);
+}
+
+/**
+ * to_iwmr - get device memory region
+ * @ibdev: ib memory region
+ **/
+static inline struct irdma_mr *to_iwmr(struct ib_mr *ibmr)
+{
+ return container_of(ibmr, struct irdma_mr, ibmr);
+}
+
+/**
+ * to_iwmr_from_ibfmr - get device memory region
+ * @ibfmr: ib fmr
+ **/
+static inline struct irdma_mr *to_iwmr_from_ibfmr(struct ib_fmr *ibfmr)
+{
+ return container_of(ibfmr, struct irdma_mr, ibfmr);
+}
+
+/**
+ * to_iwmw - get device memory window
+ * @ibmw: ib memory window
+ **/
+static inline struct irdma_mr *to_iwmw(struct ib_mw *ibmw)
+{
+ return container_of(ibmw, struct irdma_mr, ibmw);
+}
+
+/**
+ * to_iwcq - get completion queue
+ * @ibcq: ib cqdevice
+ **/
+static inline struct irdma_cq *to_iwcq(struct ib_cq *ibcq)
+{
+ return container_of(ibcq, struct irdma_cq, ibcq);
+}
+
+/**
+ * to_iwqp - get device qp
+ * @ibqp: ib qp
+ **/
+static inline struct irdma_qp *to_iwqp(struct ib_qp *ibqp)
+{
+ return container_of(ibqp, struct irdma_qp, ibqp);
+}
+
+/**
+ * irdma_alloc_resource - allocate a resource
+ * @iwdev: device pointer
+ * @resource_array: resource bit array:
+ * @max_resources: maximum resource number
+ * @req_resources_num: Allocated resource number
+ * @next: next free id
+ **/
+static inline int irdma_alloc_rsrc(struct irdma_pci_f *rf,
+ unsigned long *rsrc_array, u32 max_rsrc,
+ u32 *req_rsrc_num, u32 *next)
+{
+ u32 rsrc_num;
+ unsigned long flags;
+
+ spin_lock_irqsave(&rf->rsrc_lock, flags);
+ rsrc_num = find_next_zero_bit(rsrc_array, max_rsrc, *next);
+ if (rsrc_num >= max_rsrc) {
+ rsrc_num = find_first_zero_bit(rsrc_array, max_rsrc);
+ if (rsrc_num >= max_rsrc) {
+ spin_unlock_irqrestore(&rf->rsrc_lock, flags);
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: resource [%d] allocation failed\n",
+ rsrc_num);
+ return -EOVERFLOW;
+ }
+ }
+ set_bit(rsrc_num, rsrc_array);
+ *next = rsrc_num + 1;
+ if (*next == max_rsrc)
+ *next = 0;
+ *req_rsrc_num = rsrc_num;
+ spin_unlock_irqrestore(&rf->rsrc_lock, flags);
+
+ return 0;
+}
+
+/**
+ * irdma_is_resource_allocated - detrmine if resource is
+ * allocated
+ * @iwdev: device pointer
+ * @resource_array: resource array for the resource_num
+ * @resource_num: resource number to check
+ **/
+static inline bool irdma_is_rsrc_allocated(struct irdma_pci_f *rf,
+ unsigned long *rsrc_array,
+ u32 rsrc_num)
+{
+ bool bit_is_set;
+ unsigned long flags;
+
+ spin_lock_irqsave(&rf->rsrc_lock, flags);
+
+ bit_is_set = test_bit(rsrc_num, rsrc_array);
+ spin_unlock_irqrestore(&rf->rsrc_lock, flags);
+
+ return bit_is_set;
+}
+
+/**
+ * irdma_free_resource - free a resource
+ * @iwdev: device pointer
+ * @resource_array: resource array for the resource_num
+ * @resource_num: resource number to free
+ **/
+static inline void irdma_free_rsrc(struct irdma_pci_f *rf,
+ unsigned long *rsrc_array, u32 rsrc_num)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&rf->rsrc_lock, flags);
+ clear_bit(rsrc_num, rsrc_array);
+ spin_unlock_irqrestore(&rf->rsrc_lock, flags);
+}
+
+void irdma_init_rf_config_params(struct irdma_pci_f *rf);
+enum irdma_status_code irdma_ctrl_init_hw(struct irdma_pci_f *rf);
+void irdma_deinit_ctrl_hw(struct irdma_pci_f *rf);
+enum irdma_status_code irdma_rt_init_hw(struct irdma_pci_f *rf,
+ struct irdma_device *iwdev,
+ struct irdma_l2params *l2params);
+void irdma_deinit_rt_device(struct irdma_device *iwdev);
+void irdma_add_ref(struct ib_qp *ibqp);
+void irdma_rem_ref(struct ib_qp *ibqp);
+struct ib_qp *irdma_get_qp(struct ib_device *ibdev, int qpn);
+void irdma_flush_wqes(struct irdma_pci_f *rf, struct irdma_qp *qp);
+void irdma_manage_arp_cache(struct irdma_pci_f *rf, unsigned char *mac_addr,
+ u32 *ip_addr, bool ipv4, u32 action);
+int irdma_manage_apbvt(struct irdma_device *iwdev, u16 accel_local_port,
+ bool add_port);
+struct irdma_cqp_request *irdma_get_cqp_request(struct irdma_cqp *cqp,
+ bool wait);
+void irdma_free_cqp_request(struct irdma_cqp *cqp,
+ struct irdma_cqp_request *cqp_request);
+void irdma_put_cqp_request(struct irdma_cqp *cqp,
+ struct irdma_cqp_request *cqp_request);
+struct irdma_device *irdma_get_device(struct net_device *netdev);
+void irdma_put_device(struct irdma_device *iwdev);
+struct irdma_handler *irdma_find_handler(struct pci_dev *pdev);
+struct irdma_device *irdma_find_iwdev(const char *name);
+void irdma_add_handler(struct irdma_handler *hdl);
+void irdma_del_handler(struct irdma_handler *hdl);
+void irdma_add_ip(struct irdma_device *iwdev);
+int irdma_alloc_local_mac_entry(struct irdma_pci_f *rf, u16 *mac_tbl_idx);
+int irdma_add_local_mac_entry(struct irdma_pci_f *rf, u8 *mac_addr, u16 idx);
+void irdma_del_local_mac_entry(struct irdma_pci_f *rf, u16 idx);
+
+u32 irdma_initialize_hw_rsrc(struct irdma_pci_f *rf);
+int irdma_register_rdma_device(struct irdma_device *iwdev);
+void irdma_port_ibevent(struct irdma_device *iwdev);
+void irdma_cm_disconn(struct irdma_qp *qp);
+
+enum irdma_status_code
+irdma_handle_cqp_op(struct irdma_pci_f *rf,
+ struct irdma_cqp_request *cqp_request);
+
+int irdma_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask,
+ struct ib_udata *udata);
+int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
+ int attr_mask, struct ib_udata *udata);
+void irdma_cq_wq_destroy(struct irdma_pci_f *rf, struct irdma_sc_cq *cq);
+
+void irdma_cleanup_pending_cqp_op(struct irdma_pci_f *rf);
+/* TODO: remove once VMWare implements or if not needed */
+enum irdma_status_code irdma_hw_modify_qp(struct irdma_device *iwdev,
+ struct irdma_qp *iwqp,
+ struct irdma_modify_qp_info *info,
+ bool wait);
+enum irdma_status_code irdma_qp_suspend_resume(struct irdma_sc_qp *qp,
+ bool suspend);
+enum irdma_status_code
+irdma_manage_qhash(struct irdma_device *iwdev, struct irdma_cm_info *cminfo,
+ enum irdma_quad_entry_type etype,
+ enum irdma_quad_hash_manage_type mtype, void *cmnode,
+ bool wait);
+void irdma_receive_ilq(struct irdma_sc_vsi *vsi, struct irdma_puda_buf *rbuf);
+void irdma_free_sqbuf(struct irdma_sc_vsi *vsi, void *bufp);
+void irdma_free_qp_rsrc(struct irdma_device *iwdev, struct irdma_qp *iwqp,
+ u32 qp_num);
+void irdma_request_reset(struct irdma_pci_f *rf);
+void irdma_destroy_rdma_device(struct irdma_ib_device *iwibdev);
+enum irdma_status_code irdma_setup_cm_core(struct irdma_device *iwdev, u8 ver);
+void irdma_cleanup_cm_core(struct irdma_cm_core *cm_core);
+void irdma_next_iw_state(struct irdma_qp *iwqp, u8 state, u8 del_hash, u8 term,
+ u8 term_len);
+int irdma_send_syn(struct irdma_cm_node *cm_node, u32 sendack);
+int irdma_send_reset(struct irdma_cm_node *cm_node);
+struct irdma_cm_node *irdma_find_node(struct irdma_cm_core *cm_core,
+ u16 rem_port, u32 *rem_addr, u16 loc_port,
+ u32 *loc_addr, bool add_refcnt,
+ bool accelerated_list);
+enum irdma_status_code irdma_hw_flush_wqes(struct irdma_pci_f *rf,
+ struct irdma_sc_qp *qp,
+ struct irdma_qp_flush_info *info,
+ bool wait);
+void irdma_gen_ae(struct irdma_pci_f *rf, struct irdma_sc_qp *qp,
+ struct irdma_gen_ae_info *info, bool wait);
+void irdma_copy_ip_ntohl(u32 *dst, __be32 *src);
+void irdma_copy_ip_htonl(__be32 *dst, u32 *src);
+u16 irdma_get_vlan_ipv4(u32 *addr);
+struct net_device *irdma_netdev_vlan_ipv6(u32 *addr, u16 *vlan_id, u8 *mac);
+struct ib_mr *irdma_reg_phys_mr(struct ib_pd *ib_pd, u64 addr, u64 size,
+ int acc, u64 *iova_start);
+int cqp_compl_thread(void *context);
+int irdma_inetaddr_event(struct notifier_block *notifier, unsigned long event,
+ void *ptr);
+int irdma_inet6addr_event(struct notifier_block *notifier, unsigned long event,
+ void *ptr);
+int irdma_net_event(struct notifier_block *notifier, unsigned long event,
+ void *ptr);
+int irdma_netdevice_event(struct notifier_block *notifier, unsigned long event,
+ void *ptr);
+int irdma_probe(struct platform_device *pdev);
+int irdma_remove(struct platform_device *pdev);
+int i40iw_probe(struct platform_device *pdev);
+void i40iw_remove(struct platform_device *pdev);
+void i40iw_request_reset(struct irdma_pci_f *rf);
+void icrdma_request_reset(struct irdma_pci_f *rf);
+void irdma_register_notifiers(void);
+void irdma_unregister_notifiers(void);
+void irdma_cqp_ce_handler(struct irdma_pci_f *rf, struct irdma_sc_cq *cq);
+int irdma_ah_cqp_op(struct irdma_pci_f *rf, struct irdma_sc_ah *sc_ah, u8 cmd,
+ bool wait,
+ void (*callback_fcn)(struct irdma_cqp_request *cqp_request),
+ void *cb_param);
+void irdma_gsi_ud_qp_ah_cb(struct irdma_cqp_request *cqp_request);
+int irdma_configfs_init(void);
+void irdma_configfs_exit(void);
+#endif /* IRDMA_MAIN_H */
--
2.21.0
^ permalink raw reply related
* [RFC 07/20] RDMA/irdma: Add HMC backing store setup functions
From: Jeff Kirsher @ 2019-09-26 16:45 UTC (permalink / raw)
To: dledford, jgg, gregkh; +Cc: Mustafa Ismail, netdev, linux-rdma, Shiraz Saleem
In-Reply-To: <20190926164519.10471-1-jeffrey.t.kirsher@intel.com>
From: Mustafa Ismail <mustafa.ismail@intel.com>
HW uses host memory as a backing store for a number of
protocol context objects and queue state tracking.
The Host Memory Cache (HMC) is a component responsible for
managing these objects stored in host memory.
Add the functions and data structures to manage the allocation
of backing pages used by the HMC for the various objects
Signed-off-by: Mustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
---
drivers/infiniband/hw/irdma/hmc.c | 706 ++++++++++++++++++++++++++++++
drivers/infiniband/hw/irdma/hmc.h | 219 +++++++++
2 files changed, 925 insertions(+)
create mode 100644 drivers/infiniband/hw/irdma/hmc.c
create mode 100644 drivers/infiniband/hw/irdma/hmc.h
diff --git a/drivers/infiniband/hw/irdma/hmc.c b/drivers/infiniband/hw/irdma/hmc.c
new file mode 100644
index 000000000000..ea42fd41bd55
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/hmc.c
@@ -0,0 +1,706 @@
+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "osdep.h"
+#include "status.h"
+#include "hmc.h"
+#include "defs.h"
+#include "type.h"
+#include "protos.h"
+
+/**
+ * irdma_find_sd_index_limit - finds segment descriptor index limit
+ * @hmc_info: pointer to the HMC configuration information structure
+ * @type: type of HMC resources we're searching
+ * @idx: starting index for the object
+ * @cnt: number of objects we're trying to create
+ * @sd_idx: pointer to return index of the segment descriptor in question
+ * @sd_limit: pointer to return the maximum number of segment descriptors
+ *
+ * This function calculates the segment descriptor index and index limit
+ * for the resource defined by irdma_hmc_rsrc_type.
+ */
+
+static void irdma_find_sd_index_limit(struct irdma_hmc_info *hmc_info, u32 type,
+ u32 idx, u32 cnt, u32 *sd_idx,
+ u32 *sd_limit)
+{
+ u64 fpm_addr, fpm_limit;
+
+ fpm_addr = hmc_info->hmc_obj[(type)].base +
+ hmc_info->hmc_obj[type].size * idx;
+ fpm_limit = fpm_addr + hmc_info->hmc_obj[type].size * cnt;
+ *sd_idx = (u32)(fpm_addr / IRDMA_HMC_DIRECT_BP_SIZE);
+ *sd_limit = (u32)((fpm_limit - 1) / IRDMA_HMC_DIRECT_BP_SIZE);
+ *sd_limit += 1;
+}
+
+/**
+ * irdma_find_pd_index_limit - finds page descriptor index limit
+ * @hmc_info: pointer to the HMC configuration information struct
+ * @type: HMC resource type we're examining
+ * @idx: starting index for the object
+ * @cnt: number of objects we're trying to create
+ * @pd_idx: pointer to return page descriptor index
+ * @pd_limit: pointer to return page descriptor index limit
+ *
+ * Calculates the page descriptor index and index limit for the resource
+ * defined by irdma_hmc_rsrc_type.
+ */
+
+static void irdma_find_pd_index_limit(struct irdma_hmc_info *hmc_info, u32 type,
+ u32 idx, u32 cnt, u32 *pd_idx,
+ u32 *pd_limit)
+{
+ u64 fpm_adr, fpm_limit;
+
+ fpm_adr = hmc_info->hmc_obj[type].base +
+ hmc_info->hmc_obj[type].size * idx;
+ fpm_limit = fpm_adr + (hmc_info)->hmc_obj[(type)].size * (cnt);
+ *pd_idx = (u32)(fpm_adr / IRDMA_HMC_PAGED_BP_SIZE);
+ *pd_limit = (u32)((fpm_limit - 1) / IRDMA_HMC_PAGED_BP_SIZE);
+ *pd_limit += 1;
+}
+
+/**
+ * irdma_set_sd_entry - setup entry for sd programming
+ * @pa: physical addr
+ * @idx: sd index
+ * @type: paged or direct sd
+ * @entry: sd entry ptr
+ */
+static void irdma_set_sd_entry(u64 pa, u32 idx, enum irdma_sd_entry_type type,
+ struct irdma_update_sd_entry *entry)
+{
+ entry->data = pa | (IRDMA_HMC_MAX_BP_COUNT << IRDMA_PFHMC_SDDATALOW_PMSDBPCOUNT_S) |
+ (((type == IRDMA_SD_TYPE_PAGED) ? 0 : 1) << IRDMA_PFHMC_SDDATALOW_PMSDTYPE_S) |
+ (1 << IRDMA_PFHMC_SDDATALOW_PMSDVALID_S);
+ entry->cmd = (idx | (1 << IRDMA_PFHMC_SDCMD_PMSDWR_S) | (1 << 15));
+}
+
+/**
+ * irdma_clr_sd_entry - setup entry for sd clear
+ * @idx: sd index
+ * @type: paged or direct sd
+ * @entry: sd entry ptr
+ */
+static void irdma_clr_sd_entry(u32 idx, enum irdma_sd_entry_type type,
+ struct irdma_update_sd_entry *entry)
+{
+ entry->data = (IRDMA_HMC_MAX_BP_COUNT << IRDMA_PFHMC_SDDATALOW_PMSDBPCOUNT_S) |
+ (((type == IRDMA_SD_TYPE_PAGED) ? 0 : 1) << IRDMA_PFHMC_SDDATALOW_PMSDTYPE_S);
+ entry->cmd = (idx | (1 << IRDMA_PFHMC_SDCMD_PMSDWR_S) | (1 << 15));
+}
+
+/**
+ * irdma_hmc_sd_one - setup 1 sd entry for cqp
+ * @dev: pointer to the device structure
+ * @hmc_fn_id: hmc's function id
+ * @pa: physical addr
+ * @sd_idx: sd index
+ * @type: paged or direct sd
+ * @setsd: flag to set or clear sd
+ */
+enum irdma_status_code irdma_hmc_sd_one(struct irdma_sc_dev *dev, u8 hmc_fn_id,
+ u64 pa, u32 sd_idx,
+ enum irdma_sd_entry_type type,
+ bool setsd)
+{
+ struct irdma_update_sds_info sdinfo;
+
+ sdinfo.cnt = 1;
+ sdinfo.hmc_fn_id = hmc_fn_id;
+ if (setsd)
+ irdma_set_sd_entry(pa, sd_idx, type, sdinfo.entry);
+ else
+ irdma_clr_sd_entry(sd_idx, type, sdinfo.entry);
+ return dev->cqp->process_cqp_sds(dev, &sdinfo);
+}
+
+/**
+ * irdma_hmc_sd_grp - setup group of sd entries for cqp
+ * @dev: pointer to the device structure
+ * @hmc_info: pointer to the HMC configuration information struct
+ * @sd_index: sd index
+ * @sd_cnt: number of sd entries
+ * @setsd: flag to set or clear sd
+ */
+static enum irdma_status_code irdma_hmc_sd_grp(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info,
+ u32 sd_index, u32 sd_cnt,
+ bool setsd)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+ struct irdma_update_sds_info sdinfo = {};
+ u64 pa;
+ u32 i;
+ enum irdma_status_code ret_code = 0;
+
+ sdinfo.hmc_fn_id = hmc_info->hmc_fn_id;
+ for (i = sd_index; i < sd_index + sd_cnt; i++) {
+ sd_entry = &hmc_info->sd_table.sd_entry[i];
+ if (!sd_entry || (!sd_entry->valid && setsd) ||
+ (sd_entry->valid && !setsd))
+ continue;
+ if (setsd) {
+ pa = (sd_entry->entry_type == IRDMA_SD_TYPE_PAGED) ?
+ sd_entry->u.pd_table.pd_page_addr.pa :
+ sd_entry->u.bp.addr.pa;
+ irdma_set_sd_entry(pa, i, sd_entry->entry_type,
+ &sdinfo.entry[sdinfo.cnt]);
+ } else {
+ irdma_clr_sd_entry(i, sd_entry->entry_type,
+ &sdinfo.entry[sdinfo.cnt]);
+ }
+ sdinfo.cnt++;
+ if (sdinfo.cnt == IRDMA_MAX_SD_ENTRIES) {
+ ret_code = dev->cqp->process_cqp_sds(dev, &sdinfo);
+ if (ret_code) {
+ dev_dbg(rfdev_to_dev(dev),
+ "HMC: sd_programming failed err=%d\n",
+ ret_code);
+ return ret_code;
+ }
+
+ sdinfo.cnt = 0;
+ }
+ }
+ if (sdinfo.cnt)
+ ret_code = dev->cqp->process_cqp_sds(dev, &sdinfo);
+
+ return ret_code;
+}
+
+/**
+ * irdma_hmc_finish_add_sd_reg - program sd entries for objects
+ * @dev: pointer to the device structure
+ * @info: create obj info
+ */
+static enum irdma_status_code
+irdma_hmc_finish_add_sd_reg(struct irdma_sc_dev *dev,
+ struct irdma_hmc_create_obj_info *info)
+{
+ if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt)
+ return IRDMA_ERR_INVALID_HMC_OBJ_INDEX;
+
+ if ((info->start_idx + info->count) >
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt)
+ return IRDMA_ERR_INVALID_HMC_OBJ_COUNT;
+
+ if (!info->add_sd_cnt)
+ return 0;
+ return irdma_hmc_sd_grp(dev, info->hmc_info,
+ info->hmc_info->sd_indexes[0], info->add_sd_cnt,
+ true);
+}
+
+/**
+ * irdma_sc_create_hmc_obj - allocate backing store for hmc objects
+ * @dev: pointer to the device structure
+ * @info: pointer to irdma_hmc_create_obj_info struct
+ *
+ * This will allocate memory for PDs and backing pages and populate
+ * the sd and pd entries.
+ */
+enum irdma_status_code
+irdma_sc_create_hmc_obj(struct irdma_sc_dev *dev,
+ struct irdma_hmc_create_obj_info *info)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+ u32 sd_idx, sd_lmt;
+ u32 pd_idx = 0, pd_lmt = 0;
+ u32 pd_idx1 = 0, pd_lmt1 = 0;
+ u32 i, j;
+ bool pd_error = false;
+ enum irdma_status_code ret_code = 0;
+
+ if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt)
+ return IRDMA_ERR_INVALID_HMC_OBJ_INDEX;
+
+ if ((info->start_idx + info->count) >
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt) {
+ dev_dbg(rfdev_to_dev(dev),
+ "HMC: error type %u, start = %u, req cnt %u, cnt = %u\n",
+ info->rsrc_type, info->start_idx, info->count,
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt);
+ return IRDMA_ERR_INVALID_HMC_OBJ_COUNT;
+ }
+
+ irdma_find_sd_index_limit(info->hmc_info, info->rsrc_type,
+ info->start_idx, info->count, &sd_idx,
+ &sd_lmt);
+ if (sd_idx >= info->hmc_info->sd_table.sd_cnt ||
+ sd_lmt > info->hmc_info->sd_table.sd_cnt) {
+ return IRDMA_ERR_INVALID_SD_INDEX;
+ }
+
+ irdma_find_pd_index_limit(info->hmc_info, info->rsrc_type,
+ info->start_idx, info->count, &pd_idx,
+ &pd_lmt);
+
+ for (j = sd_idx; j < sd_lmt; j++) {
+ ret_code = irdma_add_sd_table_entry(dev->hw, info->hmc_info, j,
+ info->entry_type,
+ IRDMA_HMC_DIRECT_BP_SIZE);
+ if (ret_code)
+ goto exit_sd_error;
+
+ sd_entry = &info->hmc_info->sd_table.sd_entry[j];
+ if (sd_entry->entry_type == IRDMA_SD_TYPE_PAGED &&
+ (dev->hmc_info == info->hmc_info &&
+ info->rsrc_type != IRDMA_HMC_IW_PBLE)) {
+ pd_idx1 = max(pd_idx, (j * IRDMA_HMC_MAX_BP_COUNT));
+ pd_lmt1 = min(pd_lmt, (j + 1) * IRDMA_HMC_MAX_BP_COUNT);
+ for (i = pd_idx1; i < pd_lmt1; i++) {
+ /* update the pd table entry */
+ ret_code = irdma_add_pd_table_entry(dev,
+ info->hmc_info,
+ i, NULL);
+ if (ret_code) {
+ pd_error = true;
+ break;
+ }
+ }
+ if (pd_error) {
+ while (i && (i > pd_idx1)) {
+ irdma_remove_pd_bp(dev, info->hmc_info,
+ i - 1);
+ i--;
+ }
+ }
+ }
+ if (sd_entry->valid)
+ continue;
+
+ info->hmc_info->sd_indexes[info->add_sd_cnt] = (u16)j;
+ info->add_sd_cnt++;
+ sd_entry->valid = true;
+ }
+ return irdma_hmc_finish_add_sd_reg(dev, info);
+
+exit_sd_error:
+ while (j && (j > sd_idx)) {
+ sd_entry = &info->hmc_info->sd_table.sd_entry[j - 1];
+ switch (sd_entry->entry_type) {
+ case IRDMA_SD_TYPE_PAGED:
+ pd_idx1 = max(pd_idx, (j - 1) * IRDMA_HMC_MAX_BP_COUNT);
+ pd_lmt1 = min(pd_lmt, (j * IRDMA_HMC_MAX_BP_COUNT));
+ for (i = pd_idx1; i < pd_lmt1; i++)
+ irdma_prep_remove_pd_page(info->hmc_info, i);
+ break;
+ case IRDMA_SD_TYPE_DIRECT:
+ irdma_prep_remove_pd_page(info->hmc_info, (j - 1));
+ break;
+ default:
+ ret_code = IRDMA_ERR_INVALID_SD_TYPE;
+ break;
+ }
+ j--;
+ }
+
+ return ret_code;
+}
+
+/**
+ * irdma_finish_del_sd_reg - delete sd entries for objects
+ * @dev: pointer to the device structure
+ * @info: dele obj info
+ * @reset: true if called before reset
+ */
+static enum irdma_status_code
+irdma_finish_del_sd_reg(struct irdma_sc_dev *dev,
+ struct irdma_hmc_del_obj_info *info, bool reset)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+ enum irdma_status_code ret_code = 0;
+ u32 i, sd_idx;
+ struct irdma_dma_mem *mem;
+
+ if (dev->is_pf && !reset)
+ ret_code = irdma_hmc_sd_grp(dev, info->hmc_info,
+ info->hmc_info->sd_indexes[0],
+ info->del_sd_cnt, false);
+
+ if (ret_code)
+ dev_dbg(rfdev_to_dev(dev), "HMC: error cqp sd sd_grp\n");
+ for (i = 0; i < info->del_sd_cnt; i++) {
+ sd_idx = info->hmc_info->sd_indexes[i];
+ sd_entry = &info->hmc_info->sd_table.sd_entry[sd_idx];
+ if (!sd_entry)
+ continue;
+ mem = (sd_entry->entry_type == IRDMA_SD_TYPE_PAGED) ?
+ &sd_entry->u.pd_table.pd_page_addr :
+ &sd_entry->u.bp.addr;
+
+ if (!mem || !mem->va) {
+ dev_dbg(rfdev_to_dev(dev), "HMC: error cqp sd mem\n");
+ } else {
+ dma_free_coherent(hw_to_dev(dev->hw), mem->size,
+ mem->va, mem->pa);
+ mem->va = NULL;
+ }
+ }
+
+ return ret_code;
+}
+
+/**
+ * irdma_sc_del_hmc_obj - remove pe hmc objects
+ * @dev: pointer to the device structure
+ * @info: pointer to irdma_hmc_del_obj_info struct
+ * @reset: true if called before reset
+ *
+ * This will de-populate the SDs and PDs. It frees
+ * the memory for PDS and backing storage. After this function is returned,
+ * caller should deallocate memory allocated previously for
+ * book-keeping information about PDs and backing storage.
+ */
+enum irdma_status_code irdma_sc_del_hmc_obj(struct irdma_sc_dev *dev,
+ struct irdma_hmc_del_obj_info *info,
+ bool reset)
+{
+ struct irdma_hmc_pd_table *pd_table;
+ u32 sd_idx, sd_lmt;
+ u32 pd_idx, pd_lmt, rel_pd_idx;
+ u32 i, j;
+ enum irdma_status_code ret_code = 0;
+
+ if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt) {
+ dev_dbg(rfdev_to_dev(dev),
+ "HMC: error start_idx[%04d] >= [type %04d].cnt[%04d]\n",
+ info->start_idx, info->rsrc_type,
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt);
+ return IRDMA_ERR_INVALID_HMC_OBJ_INDEX;
+ }
+
+ if ((info->start_idx + info->count) >
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt) {
+ dev_dbg(rfdev_to_dev(dev),
+ "HMC: error start_idx[%04d] + count %04d >= [type %04d].cnt[%04d]\n",
+ info->start_idx, info->count, info->rsrc_type,
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt);
+ return IRDMA_ERR_INVALID_HMC_OBJ_COUNT;
+ }
+
+ irdma_find_pd_index_limit(info->hmc_info, info->rsrc_type,
+ info->start_idx, info->count, &pd_idx,
+ &pd_lmt);
+
+ for (j = pd_idx; j < pd_lmt; j++) {
+ sd_idx = j / IRDMA_HMC_PD_CNT_IN_SD;
+
+ if (!info->hmc_info->sd_table.sd_entry[sd_idx].valid)
+ continue;
+
+ if (info->hmc_info->sd_table.sd_entry[sd_idx].entry_type !=
+ IRDMA_SD_TYPE_PAGED)
+ continue;
+
+ rel_pd_idx = j % IRDMA_HMC_PD_CNT_IN_SD;
+ pd_table = &info->hmc_info->sd_table.sd_entry[sd_idx].u.pd_table;
+ if (pd_table->pd_entry &&
+ pd_table->pd_entry[rel_pd_idx].valid) {
+ ret_code = irdma_remove_pd_bp(dev, info->hmc_info, j);
+ if (ret_code) {
+ dev_dbg(rfdev_to_dev(dev),
+ "HMC: remove_pd_bp error\n");
+ return ret_code;
+ }
+ }
+ }
+
+ irdma_find_sd_index_limit(info->hmc_info, info->rsrc_type,
+ info->start_idx, info->count, &sd_idx,
+ &sd_lmt);
+ if (sd_idx >= info->hmc_info->sd_table.sd_cnt ||
+ sd_lmt > info->hmc_info->sd_table.sd_cnt) {
+ dev_dbg(rfdev_to_dev(dev), "HMC: invalid sd_idx\n");
+ return IRDMA_ERR_INVALID_SD_INDEX;
+ }
+
+ for (i = sd_idx; i < sd_lmt; i++) {
+ pd_table = &info->hmc_info->sd_table.sd_entry[i].u.pd_table;
+ if (!info->hmc_info->sd_table.sd_entry[i].valid)
+ continue;
+ switch (info->hmc_info->sd_table.sd_entry[i].entry_type) {
+ case IRDMA_SD_TYPE_DIRECT:
+ ret_code = irdma_prep_remove_sd_bp(info->hmc_info, i);
+ if (!ret_code) {
+ info->hmc_info->sd_indexes[info->del_sd_cnt] =
+ (u16)i;
+ info->del_sd_cnt++;
+ }
+ break;
+ case IRDMA_SD_TYPE_PAGED:
+ ret_code = irdma_prep_remove_pd_page(info->hmc_info, i);
+ if (ret_code)
+ break;
+ if (dev->hmc_info != info->hmc_info &&
+ info->rsrc_type == IRDMA_HMC_IW_PBLE &&
+ pd_table->pd_entry) {
+ kfree(pd_table->pd_entry_virt_mem.va);
+ pd_table->pd_entry = NULL;
+ }
+ info->hmc_info->sd_indexes[info->del_sd_cnt] = (u16)i;
+ info->del_sd_cnt++;
+ break;
+ default:
+ break;
+ }
+ }
+ return irdma_finish_del_sd_reg(dev, info, reset);
+}
+
+/**
+ * irdma_add_sd_table_entry - Adds a segment descriptor to the table
+ * @hw: pointer to our hw struct
+ * @hmc_info: pointer to the HMC configuration information struct
+ * @sd_index: segment descriptor index to manipulate
+ * @type: what type of segment descriptor we're manipulating
+ * @direct_mode_sz: size to alloc in direct mode
+ */
+enum irdma_status_code irdma_add_sd_table_entry(struct irdma_hw *hw,
+ struct irdma_hmc_info *hmc_info,
+ u32 sd_index,
+ enum irdma_sd_entry_type type,
+ u64 direct_mode_sz)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+ struct irdma_dma_mem dma_mem;
+ u64 alloc_len;
+
+ sd_entry = &hmc_info->sd_table.sd_entry[sd_index];
+ if (!sd_entry->valid) {
+ if (type == IRDMA_SD_TYPE_PAGED)
+ alloc_len = IRDMA_HMC_PAGED_BP_SIZE;
+ else
+ alloc_len = direct_mode_sz;
+
+ /* allocate a 4K pd page or 2M backing page */
+ dma_mem.size = ALIGN(alloc_len, IRDMA_HMC_PD_BP_BUF_ALIGNMENT);
+ dma_mem.va = dma_alloc_coherent(hw_to_dev(hw),
+ dma_mem.size, &dma_mem.pa,
+ GFP_ATOMIC);
+ if (!dma_mem.va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ if (type == IRDMA_SD_TYPE_PAGED) {
+ struct irdma_virt_mem *vmem =
+ &sd_entry->u.pd_table.pd_entry_virt_mem;
+
+ vmem->size = sizeof(struct irdma_hmc_pd_entry) * 512;
+ vmem->va = kzalloc(vmem->size, GFP_ATOMIC);
+ if (!vmem->va) {
+ dma_free_coherent(hw_to_dev(hw), dma_mem.size,
+ dma_mem.va, dma_mem.pa);
+ dma_mem.va = NULL;
+ return IRDMA_ERR_NO_MEMORY;
+ }
+ sd_entry->u.pd_table.pd_entry = vmem->va;
+
+ memcpy(&sd_entry->u.pd_table.pd_page_addr, &dma_mem,
+ sizeof(sd_entry->u.pd_table.pd_page_addr));
+ } else {
+ memcpy(&sd_entry->u.bp.addr, &dma_mem,
+ sizeof(sd_entry->u.bp.addr));
+
+ sd_entry->u.bp.sd_pd_index = sd_index;
+ }
+
+ hmc_info->sd_table.sd_entry[sd_index].entry_type = type;
+ IRDMA_INC_SD_REFCNT(&hmc_info->sd_table);
+ }
+ if (sd_entry->entry_type == IRDMA_SD_TYPE_DIRECT)
+ IRDMA_INC_BP_REFCNT(&sd_entry->u.bp);
+
+ return 0;
+}
+
+/**
+ * irdma_add_pd_table_entry - Adds page descriptor to the specified table
+ * @dev: pointer to our device structure
+ * @hmc_info: pointer to the HMC configuration information structure
+ * @pd_index: which page descriptor index to manipulate
+ * @rsrc_pg: if not NULL, use preallocated page instead of allocating new one.
+ *
+ * This function:
+ * 1. Initializes the pd entry
+ * 2. Adds pd_entry in the pd_table
+ * 3. Mark the entry valid in irdma_hmc_pd_entry structure
+ * 4. Initializes the pd_entry's ref count to 1
+ * assumptions:
+ * 1. The memory for pd should be pinned down, physically contiguous and
+ * aligned on 4K boundary and zeroed memory.
+ * 2. It should be 4K in size.
+ */
+enum irdma_status_code irdma_add_pd_table_entry(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info,
+ u32 pd_index,
+ struct irdma_dma_mem *rsrc_pg)
+{
+ struct irdma_hmc_pd_table *pd_table;
+ struct irdma_hmc_pd_entry *pd_entry;
+ struct irdma_dma_mem mem;
+ struct irdma_dma_mem *page = &mem;
+ u32 sd_idx, rel_pd_idx;
+ u64 *pd_addr;
+ u64 page_desc;
+
+ if (pd_index / IRDMA_HMC_PD_CNT_IN_SD >= hmc_info->sd_table.sd_cnt)
+ return IRDMA_ERR_INVALID_PAGE_DESC_INDEX;
+
+ sd_idx = (pd_index / IRDMA_HMC_PD_CNT_IN_SD);
+ if (hmc_info->sd_table.sd_entry[sd_idx].entry_type !=
+ IRDMA_SD_TYPE_PAGED)
+ return 0;
+
+ rel_pd_idx = (pd_index % IRDMA_HMC_PD_CNT_IN_SD);
+ pd_table = &hmc_info->sd_table.sd_entry[sd_idx].u.pd_table;
+ pd_entry = &pd_table->pd_entry[rel_pd_idx];
+ if (!pd_entry->valid) {
+ if (rsrc_pg) {
+ pd_entry->rsrc_pg = true;
+ page = rsrc_pg;
+ } else {
+ page->size = ALIGN(IRDMA_HMC_PAGED_BP_SIZE,
+ IRDMA_HMC_PD_BP_BUF_ALIGNMENT);
+ page->va = dma_alloc_coherent(hw_to_dev(dev->hw),
+ page->size, &page->pa,
+ GFP_KERNEL);
+ if (!page->va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ pd_entry->rsrc_pg = false;
+ }
+
+ memcpy(&pd_entry->bp.addr, page, sizeof(pd_entry->bp.addr));
+ pd_entry->bp.sd_pd_index = pd_index;
+ pd_entry->bp.entry_type = IRDMA_SD_TYPE_PAGED;
+ page_desc = page->pa | 0x1;
+ pd_addr = pd_table->pd_page_addr.va;
+ pd_addr += rel_pd_idx;
+ memcpy(pd_addr, &page_desc, sizeof(*pd_addr));
+ pd_entry->sd_index = sd_idx;
+ pd_entry->valid = true;
+ IRDMA_INC_PD_REFCNT(pd_table);
+ if (hmc_info->hmc_fn_id < dev->hw_attrs.first_hw_vf_fpm_id)
+ IRDMA_INVALIDATE_PF_HMC_PD(dev, sd_idx, rel_pd_idx);
+ else if (dev->hw->hmc.hmc_fn_id != hmc_info->hmc_fn_id)
+ IRDMA_INVALIDATE_VF_HMC_PD(dev, sd_idx, rel_pd_idx,
+ hmc_info->hmc_fn_id);
+ }
+ IRDMA_INC_BP_REFCNT(&pd_entry->bp);
+
+ return 0;
+}
+
+/**
+ * irdma_remove_pd_bp - remove a backing page from a page descriptor
+ * @dev: pointer to our HW structure
+ * @hmc_info: pointer to the HMC configuration information structure
+ * @idx: the page index
+ *
+ * This function:
+ * 1. Marks the entry in pd table (for paged address mode) or in sd table
+ * (for direct address mode) invalid.
+ * 2. Write to register PMPDINV to invalidate the backing page in FV cache
+ * 3. Decrement the ref count for the pd _entry
+ * assumptions:
+ * 1. Caller can deallocate the memory used by backing storage after this
+ * function returns.
+ */
+enum irdma_status_code irdma_remove_pd_bp(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info,
+ u32 idx)
+{
+ struct irdma_hmc_pd_entry *pd_entry;
+ struct irdma_hmc_pd_table *pd_table;
+ struct irdma_hmc_sd_entry *sd_entry;
+ u32 sd_idx, rel_pd_idx;
+ struct irdma_dma_mem *mem;
+ u64 *pd_addr;
+
+ sd_idx = idx / IRDMA_HMC_PD_CNT_IN_SD;
+ rel_pd_idx = idx % IRDMA_HMC_PD_CNT_IN_SD;
+ if (sd_idx >= hmc_info->sd_table.sd_cnt)
+ return IRDMA_ERR_INVALID_PAGE_DESC_INDEX;
+
+ sd_entry = &hmc_info->sd_table.sd_entry[sd_idx];
+ if (sd_entry->entry_type != IRDMA_SD_TYPE_PAGED)
+ return IRDMA_ERR_INVALID_SD_TYPE;
+
+ pd_table = &hmc_info->sd_table.sd_entry[sd_idx].u.pd_table;
+ pd_entry = &pd_table->pd_entry[rel_pd_idx];
+ IRDMA_DEC_BP_REFCNT(&pd_entry->bp);
+ if (pd_entry->bp.ref_cnt)
+ return 0;
+
+ pd_entry->valid = false;
+ IRDMA_DEC_PD_REFCNT(pd_table);
+ pd_addr = pd_table->pd_page_addr.va;
+ pd_addr += rel_pd_idx;
+ memset(pd_addr, 0, sizeof(u64));
+ if (dev->is_pf) {
+ if (dev->hmc_fn_id == hmc_info->hmc_fn_id)
+ IRDMA_INVALIDATE_PF_HMC_PD(dev, sd_idx, idx);
+ else
+ IRDMA_INVALIDATE_VF_HMC_PD(dev, sd_idx, idx,
+ hmc_info->hmc_fn_id);
+ }
+
+ if (!pd_entry->rsrc_pg) {
+ mem = &pd_entry->bp.addr;
+ if (!mem || !mem->va)
+ return IRDMA_ERR_PARAM;
+
+ dma_free_coherent(hw_to_dev(dev->hw), mem->size, mem->va,
+ mem->pa);
+ mem->va = NULL;
+ }
+ if (!pd_table->ref_cnt)
+ kfree(pd_table->pd_entry_virt_mem.va);
+
+ return 0;
+}
+
+/**
+ * irdma_prep_remove_sd_bp - Prepares to remove a backing page from a sd entry
+ * @hmc_info: pointer to the HMC configuration information structure
+ * @idx: the page index
+ */
+enum irdma_status_code irdma_prep_remove_sd_bp(struct irdma_hmc_info *hmc_info,
+ u32 idx)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+
+ sd_entry = &hmc_info->sd_table.sd_entry[idx];
+ IRDMA_DEC_BP_REFCNT(&sd_entry->u.bp);
+ if (sd_entry->u.bp.ref_cnt)
+ return IRDMA_ERR_NOT_READY;
+
+ IRDMA_DEC_SD_REFCNT(&hmc_info->sd_table);
+ sd_entry->valid = false;
+
+ return 0;
+}
+
+/**
+ * irdma_prep_remove_pd_page - Prepares to remove a PD page from sd entry.
+ * @hmc_info: pointer to the HMC configuration information structure
+ * @idx: segment descriptor index to find the relevant page descriptor
+ */
+enum irdma_status_code
+irdma_prep_remove_pd_page(struct irdma_hmc_info *hmc_info, u32 idx)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+
+ sd_entry = &hmc_info->sd_table.sd_entry[idx];
+
+ if (sd_entry->u.pd_table.ref_cnt)
+ return IRDMA_ERR_NOT_READY;
+
+ sd_entry->valid = false;
+ IRDMA_DEC_SD_REFCNT(&hmc_info->sd_table);
+
+ return 0;
+}
diff --git a/drivers/infiniband/hw/irdma/hmc.h b/drivers/infiniband/hw/irdma/hmc.h
new file mode 100644
index 000000000000..9a529f89cb07
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/hmc.h
@@ -0,0 +1,219 @@
+/* SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef IRDMA_HMC_H
+#define IRDMA_HMC_H
+
+#include "defs.h"
+
+#define IRDMA_HMC_MAX_BP_COUNT 512
+#define IRDMA_MAX_SD_ENTRIES 11
+#define IRDMA_HW_DBG_HMC_INVALID_BP_MARK 0xca
+#define IRDMA_HMC_INFO_SIGNATURE 0x484d5347
+#define IRDMA_HMC_PD_CNT_IN_SD 512
+#define IRDMA_HMC_DIRECT_BP_SIZE 0x200000
+#define IRDMA_HMC_MAX_SD_COUNT 8192
+#define IRDMA_HMC_PAGED_BP_SIZE 4096
+#define IRDMA_HMC_PD_BP_BUF_ALIGNMENT 4096
+#define IRDMA_FIRST_VF_FPM_ID 8
+#define FPM_MULTIPLIER 1024
+
+#define IRDMA_INC_SD_REFCNT(sd_table) ((sd_table)->ref_cnt++)
+#define IRDMA_INC_PD_REFCNT(pd_table) ((pd_table)->ref_cnt++)
+#define IRDMA_INC_BP_REFCNT(bp) ((bp)->ref_cnt++)
+
+#define IRDMA_DEC_SD_REFCNT(sd_table) ((sd_table)->ref_cnt--)
+#define IRDMA_DEC_PD_REFCNT(pd_table) ((pd_table)->ref_cnt--)
+#define IRDMA_DEC_BP_REFCNT(bp) ((bp)->ref_cnt--)
+
+/**
+ * IRDMA_INVALIDATE_PF_HMC_PD - Invalidates the pd cache in the hardware
+ * @hw: pointer to our hw struct
+ * @sd_idx: segment descriptor index
+ * @pd_idx: page descriptor index
+ */
+#define IRDMA_INVALIDATE_PF_HMC_PD(dev, sd_idx, pd_idx) \
+ wr32((dev)->hw, (dev)->hw_regs[IRDMA_PFHMC_PDINV], \
+ (((sd_idx) << IRDMA_PFHMC_PDINV_PMSDIDX_S) | \
+ (0x1 << IRDMA_PFHMC_PDINV_PMSDPARTSEL_S) | \
+ ((pd_idx) << IRDMA_PFHMC_PDINV_PMPDIDX_S)))
+
+/**
+ * IRDMA_INVALIDATE_VF_HMC_PD - Invalidates the pd cache in the hardware
+ * @hw: pointer to our hw struct
+ * @sd_idx: segment descriptor index
+ * @pd_idx: page descriptor index
+ * @hmc_fn_id: VF's function id
+ */
+#define IRDMA_INVALIDATE_VF_HMC_PD(dev, sd_idx, pd_idx, hmc_fn_id) \
+ wr32((dev)->hw, \
+ (dev)->hw_regs[IRDMA_GLHMC_VFPDINV] + \
+ 4 * ((hmc_fn_id) - (dev)->hw_attrs.first_hw_vf_fpm_id), \
+ (((sd_idx) << IRDMA_PFHMC_PDINV_PMSDIDX_S) | \
+ ((pd_idx) << IRDMA_PFHMC_PDINV_PMPDIDX_S)))
+
+enum irdma_hmc_rsrc_type {
+ IRDMA_HMC_IW_QP = 0,
+ IRDMA_HMC_IW_CQ = 1,
+ IRDMA_HMC_IW_RESERVED = 2,
+ IRDMA_HMC_IW_HTE = 3,
+ IRDMA_HMC_IW_ARP = 4,
+ IRDMA_HMC_IW_APBVT_ENTRY = 5,
+ IRDMA_HMC_IW_MR = 6,
+ IRDMA_HMC_IW_XF = 7,
+ IRDMA_HMC_IW_XFFL = 8,
+ IRDMA_HMC_IW_Q1 = 9,
+ IRDMA_HMC_IW_Q1FL = 10,
+ IRDMA_HMC_IW_TIMER = 11,
+ IRDMA_HMC_IW_FSIMC = 12,
+ IRDMA_HMC_IW_FSIAV = 13,
+ IRDMA_HMC_IW_PBLE = 14,
+ IRDMA_HMC_IW_RRF = 15,
+ IRDMA_HMC_IW_RRFFL = 16,
+ IRDMA_HMC_IW_HDR = 17,
+ IRDMA_HMC_IW_MD = 18,
+ IRDMA_HMC_IW_OOISC = 19,
+ IRDMA_HMC_IW_OOISCFFL = 20,
+ IRDMA_HMC_IW_MAX, /* Must be last entry */
+};
+
+enum irdma_sd_entry_type {
+ IRDMA_SD_TYPE_INVALID = 0,
+ IRDMA_SD_TYPE_PAGED = 1,
+ IRDMA_SD_TYPE_DIRECT = 2,
+};
+
+struct irdma_hmc_obj_info {
+ u64 base;
+ u32 max_cnt;
+ u32 cnt;
+ u64 size;
+};
+
+struct irdma_hmc_bp {
+ enum irdma_sd_entry_type entry_type;
+ struct irdma_dma_mem addr;
+ u32 sd_pd_index;
+ u32 ref_cnt;
+};
+
+struct irdma_hmc_pd_entry {
+ struct irdma_hmc_bp bp;
+ u32 sd_index;
+ bool rsrc_pg;
+ bool valid;
+};
+
+struct irdma_hmc_pd_table {
+ struct irdma_dma_mem pd_page_addr;
+ struct irdma_hmc_pd_entry *pd_entry;
+ struct irdma_virt_mem pd_entry_virt_mem;
+ u32 ref_cnt;
+ u32 sd_index;
+};
+
+struct irdma_hmc_sd_entry {
+ enum irdma_sd_entry_type entry_type;
+ bool valid;
+ union {
+ struct irdma_hmc_pd_table pd_table;
+ struct irdma_hmc_bp bp;
+ } u;
+};
+
+struct irdma_hmc_sd_table {
+ struct irdma_virt_mem addr;
+ u32 sd_cnt;
+ u32 ref_cnt;
+ struct irdma_hmc_sd_entry *sd_entry;
+};
+
+struct irdma_hmc_info {
+ u32 signature;
+ u8 hmc_fn_id;
+ u16 first_sd_index;
+ struct irdma_hmc_obj_info *hmc_obj;
+ struct irdma_virt_mem hmc_obj_virt_mem;
+ struct irdma_hmc_sd_table sd_table;
+ u16 sd_indexes[IRDMA_HMC_MAX_SD_COUNT];
+};
+
+struct irdma_update_sd_entry {
+ u64 cmd;
+ u64 data;
+};
+
+struct irdma_update_sds_info {
+ u32 cnt;
+ u8 hmc_fn_id;
+ struct irdma_update_sd_entry entry[IRDMA_MAX_SD_ENTRIES];
+};
+
+struct irdma_ccq_cqe_info;
+struct irdma_hmc_fcn_info {
+ void (*callback_fcn)(struct irdma_sc_dev *dev, void *cqp_callback_param,
+ struct irdma_ccq_cqe_info *ccq_cqe_info);
+ void *cqp_callback_param;
+ u32 vf_id;
+ u16 iw_vf_idx;
+ bool free_fcn;
+};
+
+struct irdma_hmc_create_obj_info {
+ struct irdma_hmc_info *hmc_info;
+ struct irdma_virt_mem add_sd_virt_mem;
+ u32 rsrc_type;
+ u32 start_idx;
+ u32 count;
+ u32 add_sd_cnt;
+ enum irdma_sd_entry_type entry_type;
+ bool is_pf;
+};
+
+struct irdma_hmc_del_obj_info {
+ struct irdma_hmc_info *hmc_info;
+ struct irdma_virt_mem del_sd_virt_mem;
+ u32 rsrc_type;
+ u32 start_idx;
+ u32 count;
+ u32 del_sd_cnt;
+ bool is_pf;
+};
+
+enum irdma_status_code irdma_copy_dma_mem(struct irdma_hw *hw, void *dest_buf,
+ struct irdma_dma_mem *src_mem,
+ u64 src_offset, u64 size);
+enum irdma_status_code
+irdma_sc_create_hmc_obj(struct irdma_sc_dev *dev,
+ struct irdma_hmc_create_obj_info *info);
+enum irdma_status_code irdma_sc_del_hmc_obj(struct irdma_sc_dev *dev,
+ struct irdma_hmc_del_obj_info *info,
+ bool reset);
+enum irdma_status_code irdma_hmc_sd_one(struct irdma_sc_dev *dev, u8 hmc_fn_id,
+ u64 pa, u32 sd_idx,
+ enum irdma_sd_entry_type type,
+ bool setsd);
+enum irdma_status_code
+irdma_update_sds_noccq(struct irdma_sc_dev *dev,
+ struct irdma_update_sds_info *info);
+struct irdma_vfdev *irdma_vfdev_from_fpm(struct irdma_sc_dev *dev,
+ u8 hmc_fn_id);
+struct irdma_hmc_info *irdma_vf_hmcinfo_from_fpm(struct irdma_sc_dev *dev,
+ u8 hmc_fn_id);
+enum irdma_status_code irdma_add_sd_table_entry(struct irdma_hw *hw,
+ struct irdma_hmc_info *hmc_info,
+ u32 sd_index,
+ enum irdma_sd_entry_type type,
+ u64 direct_mode_sz);
+enum irdma_status_code irdma_add_pd_table_entry(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info,
+ u32 pd_index,
+ struct irdma_dma_mem *rsrc_pg);
+enum irdma_status_code irdma_remove_pd_bp(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info,
+ u32 idx);
+enum irdma_status_code irdma_prep_remove_sd_bp(struct irdma_hmc_info *hmc_info,
+ u32 idx);
+enum irdma_status_code
+irdma_prep_remove_pd_page(struct irdma_hmc_info *hmc_info, u32 idx);
+#endif /* IRDMA_HMC_H */
--
2.21.0
^ permalink raw reply related
* [RFC 11/20] RDMA/irdma: Add PBLE resource manager
From: Jeff Kirsher @ 2019-09-26 16:45 UTC (permalink / raw)
To: dledford, jgg, gregkh; +Cc: Mustafa Ismail, netdev, linux-rdma, Shiraz Saleem
In-Reply-To: <20190926164519.10471-1-jeffrey.t.kirsher@intel.com>
From: Mustafa Ismail <mustafa.ismail@intel.com>
Implement a Physical Buffer List Entry (PBLE) resource manager
to manage a pool of PBLE HMC resource objects.
Signed-off-by: Mustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
---
drivers/infiniband/hw/irdma/pble.c | 511 +++++++++++++++++++++++++++++
drivers/infiniband/hw/irdma/pble.h | 136 ++++++++
2 files changed, 647 insertions(+)
create mode 100644 drivers/infiniband/hw/irdma/pble.c
create mode 100644 drivers/infiniband/hw/irdma/pble.h
diff --git a/drivers/infiniband/hw/irdma/pble.c b/drivers/infiniband/hw/irdma/pble.c
new file mode 100644
index 000000000000..7237651a8dc0
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/pble.c
@@ -0,0 +1,511 @@
+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "osdep.h"
+#include "status.h"
+#include "hmc.h"
+#include "defs.h"
+#include "type.h"
+#include "protos.h"
+#include "pble.h"
+
+static enum irdma_status_code
+add_pble_prm(struct irdma_hmc_pble_rsrc *pble_rsrc);
+
+/**
+ * irdma_destroy_pble_prm - destroy prm during module unload
+ * @pble_rsrc: pble resources
+ */
+void irdma_destroy_pble_prm(struct irdma_hmc_pble_rsrc *pble_rsrc)
+{
+ struct irdma_chunk *chunk;
+ struct irdma_pble_prm *pinfo = &pble_rsrc->pinfo;
+
+ while (!list_empty(&pinfo->clist)) {
+ chunk = (struct irdma_chunk *) pinfo->clist.next;
+ list_del(&chunk->list);
+ if (chunk->type == PBLE_SD_PAGED)
+ irdma_pble_free_paged_mem(chunk);
+ if (chunk->bitmapbuf)
+ kfree(chunk->bitmapmem.va);
+ kfree(chunk->chunkmem.va);
+ }
+}
+
+/**
+ * irdma_hmc_init_pble - Initialize pble resources during module load
+ * @dev: irdma_sc_dev struct
+ * @pble_rsrc: pble resources
+ */
+enum irdma_status_code
+irdma_hmc_init_pble(struct irdma_sc_dev *dev,
+ struct irdma_hmc_pble_rsrc *pble_rsrc)
+{
+ struct irdma_hmc_info *hmc_info;
+ u32 fpm_idx = 0;
+ enum irdma_status_code status = 0;
+
+ hmc_info = dev->hmc_info;
+ pble_rsrc->dev = dev;
+ pble_rsrc->fpm_base_addr = hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].base;
+ /* Start pble' on 4k boundary */
+ if (pble_rsrc->fpm_base_addr & 0xfff)
+ fpm_idx = (4096 - (pble_rsrc->fpm_base_addr & 0xfff)) >> 3;
+ pble_rsrc->unallocated_pble =
+ hmc_info->hmc_obj[IRDMA_HMC_IW_PBLE].cnt - fpm_idx;
+ pble_rsrc->next_fpm_addr = pble_rsrc->fpm_base_addr + (fpm_idx << 3);
+ pble_rsrc->pinfo.pble_shift = PBLE_SHIFT;
+
+ spin_lock_init(&pble_rsrc->pinfo.prm_lock);
+ INIT_LIST_HEAD(&pble_rsrc->pinfo.clist);
+ if (add_pble_prm(pble_rsrc)) {
+ irdma_destroy_pble_prm(pble_rsrc);
+ status = IRDMA_ERR_NO_MEMORY;
+ }
+
+ return status;
+}
+
+/**
+ * get_sd_pd_idx - Returns sd index, pd index and rel_pd_idx from fpm address
+ * @pble_rsrc: structure containing fpm address
+ * @idx: where to return indexes
+ */
+static void get_sd_pd_idx(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct sd_pd_idx *idx)
+{
+ idx->sd_idx = (u32)pble_rsrc->next_fpm_addr / IRDMA_HMC_DIRECT_BP_SIZE;
+ idx->pd_idx = (u32)(pble_rsrc->next_fpm_addr / IRDMA_HMC_PAGED_BP_SIZE);
+ idx->rel_pd_idx = (idx->pd_idx % IRDMA_HMC_PD_CNT_IN_SD);
+}
+
+/**
+ * add_sd_direct - add sd direct for pble
+ * @pble_rsrc: pble resource ptr
+ * @info: page info for sd
+ */
+static enum irdma_status_code
+add_sd_direct(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct irdma_add_page_info *info)
+{
+ struct irdma_sc_dev *dev = pble_rsrc->dev;
+ enum irdma_status_code ret_code = 0;
+ struct sd_pd_idx *idx = &info->idx;
+ struct irdma_chunk *chunk = info->chunk;
+ struct irdma_hmc_info *hmc_info = info->hmc_info;
+ struct irdma_hmc_sd_entry *sd_entry = info->sd_entry;
+ u32 offset = 0;
+
+ if (!sd_entry->valid) {
+ ret_code = irdma_add_sd_table_entry(dev->hw, hmc_info,
+ info->idx.sd_idx,
+ IRDMA_SD_TYPE_DIRECT,
+ IRDMA_HMC_DIRECT_BP_SIZE);
+ if (ret_code)
+ return ret_code;
+
+ chunk->type = PBLE_SD_CONTIGOUS;
+ }
+
+ offset = idx->rel_pd_idx << HMC_PAGED_BP_SHIFT;
+ chunk->size = info->pages << HMC_PAGED_BP_SHIFT;
+ chunk->vaddr = (uintptr_t)(sd_entry->u.bp.addr.va + offset);
+ chunk->fpm_addr = pble_rsrc->next_fpm_addr;
+ dev_dbg(rfdev_to_dev(dev),
+ "PBLE: chunk_size[%lld] = 0x%llx vaddr=0x%llx fpm_addr = %llx\n",
+ chunk->size, chunk->size, chunk->vaddr, chunk->fpm_addr);
+
+ return 0;
+}
+
+/**
+ * fpm_to_idx - given fpm address, get pble index
+ * @pble_rsrc: pble resource management
+ * @addr: fpm address for index
+ */
+static u32 fpm_to_idx(struct irdma_hmc_pble_rsrc *pble_rsrc, u64 addr)
+{
+ u64 idx;
+
+ idx = (addr - (pble_rsrc->fpm_base_addr)) >> 3;
+
+ return (u32)idx;
+}
+
+/**
+ * add_bp_pages - add backing pages for sd
+ * @pble_rsrc: pble resource management
+ * @info: page info for sd
+ */
+static enum irdma_status_code
+add_bp_pages(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct irdma_add_page_info *info)
+{
+ struct irdma_sc_dev *dev = pble_rsrc->dev;
+ u8 *addr;
+ struct irdma_dma_mem mem;
+ struct irdma_hmc_pd_entry *pd_entry;
+ struct irdma_hmc_sd_entry *sd_entry = info->sd_entry;
+ struct irdma_hmc_info *hmc_info = info->hmc_info;
+ struct irdma_chunk *chunk = info->chunk;
+ enum irdma_status_code status = 0;
+ u32 rel_pd_idx = info->idx.rel_pd_idx;
+ u32 pd_idx = info->idx.pd_idx;
+ u32 i;
+
+ if (irdma_pble_get_paged_mem(chunk, info->pages))
+ return IRDMA_ERR_NO_MEMORY;
+
+ status = irdma_add_sd_table_entry(dev->hw, hmc_info, info->idx.sd_idx,
+ IRDMA_SD_TYPE_PAGED,
+ IRDMA_HMC_DIRECT_BP_SIZE);
+
+ if (status)
+ goto error;
+
+ addr = (u8 *)(uintptr_t)chunk->vaddr;
+ for (i = 0; i < info->pages; i++) {
+ mem.pa = (u64)chunk->dmainfo.dmaaddrs[i];
+ mem.size = 4096;
+ mem.va = addr;
+ pd_entry = &sd_entry->u.pd_table.pd_entry[rel_pd_idx++];
+ if (!pd_entry->valid) {
+ status = irdma_add_pd_table_entry(dev, hmc_info,
+ pd_idx++, &mem);
+ if (status)
+ goto error;
+
+ addr += 4096;
+ }
+ }
+
+ chunk->fpm_addr = pble_rsrc->next_fpm_addr;
+ return 0;
+
+error:
+ irdma_pble_free_paged_mem(chunk);
+
+ return status;
+}
+
+/**
+ * add_pble_prm - add a sd entry for pble resoure
+ * @pble_rsrc: pble resource management
+ */
+static enum irdma_status_code
+add_pble_prm(struct irdma_hmc_pble_rsrc *pble_rsrc)
+{
+ struct irdma_sc_dev *dev = pble_rsrc->dev;
+ struct irdma_hmc_sd_entry *sd_entry;
+ struct irdma_hmc_info *hmc_info;
+ struct irdma_chunk *chunk;
+ struct irdma_add_page_info info;
+ struct sd_pd_idx *idx = &info.idx;
+ enum irdma_status_code ret_code = 0;
+ enum irdma_sd_entry_type sd_entry_type;
+ u64 sd_reg_val = 0;
+ struct irdma_virt_mem chunkmem;
+ u32 pages;
+
+ if (pble_rsrc->unallocated_pble < PBLE_PER_PAGE)
+ return IRDMA_ERR_NO_MEMORY;
+
+ if (pble_rsrc->next_fpm_addr & 0xfff)
+ return IRDMA_ERR_INVALID_PAGE_DESC_INDEX;
+
+ chunkmem.size = sizeof(*chunk);
+ chunkmem.va = kzalloc(chunkmem.size, GFP_ATOMIC);
+ if (!chunkmem.va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ chunk = chunkmem.va;
+ chunk->chunkmem = chunkmem;
+ hmc_info = dev->hmc_info;
+ chunk->dev = dev;
+ chunk->fpm_addr = pble_rsrc->next_fpm_addr;
+ get_sd_pd_idx(pble_rsrc, idx);
+ sd_entry = &hmc_info->sd_table.sd_entry[idx->sd_idx];
+ pages = (idx->rel_pd_idx) ? (IRDMA_HMC_PD_CNT_IN_SD - idx->rel_pd_idx) :
+ IRDMA_HMC_PD_CNT_IN_SD;
+ pages = min(pages, pble_rsrc->unallocated_pble >> PBLE_512_SHIFT);
+ info.chunk = chunk;
+ info.hmc_info = hmc_info;
+ info.pages = pages;
+ info.sd_entry = sd_entry;
+ if (!sd_entry->valid)
+ sd_entry_type = (!idx->rel_pd_idx &&
+ (pages == IRDMA_HMC_PD_CNT_IN_SD) &&
+ dev->is_pf) ?
+ IRDMA_SD_TYPE_DIRECT : IRDMA_SD_TYPE_PAGED;
+ else
+ sd_entry_type = sd_entry->entry_type;
+
+ dev_dbg(rfdev_to_dev(dev),
+ "PBLE: pages = %d, unallocated_pble[%d] current_fpm_addr = %llx\n",
+ pages, pble_rsrc->unallocated_pble, pble_rsrc->next_fpm_addr);
+ dev_dbg(rfdev_to_dev(dev), "PBLE: sd_entry_type = %d\n",
+ sd_entry_type);
+ if (sd_entry_type == IRDMA_SD_TYPE_DIRECT)
+ ret_code = add_sd_direct(pble_rsrc, &info);
+
+ if (ret_code)
+ sd_entry_type = IRDMA_SD_TYPE_PAGED;
+ else
+ pble_rsrc->stats_direct_sds++;
+
+ if (sd_entry_type == IRDMA_SD_TYPE_PAGED) {
+ ret_code = add_bp_pages(pble_rsrc, &info);
+ if (ret_code)
+ goto error;
+ else
+ pble_rsrc->stats_paged_sds++;
+ }
+
+ ret_code = irdma_prm_add_pble_mem(&pble_rsrc->pinfo, chunk);
+ if (ret_code)
+ goto error;
+
+ pble_rsrc->next_fpm_addr += chunk->size;
+ dev_dbg(rfdev_to_dev(dev),
+ "PBLE: next_fpm_addr = %llx chunk_size[%llu] = 0x%llx\n",
+ pble_rsrc->next_fpm_addr, chunk->size, chunk->size);
+ pble_rsrc->unallocated_pble -= (u32)(chunk->size >> 3);
+ list_add(&chunk->list, &pble_rsrc->pinfo.clist);
+ sd_reg_val = (sd_entry_type == IRDMA_SD_TYPE_PAGED) ?
+ sd_entry->u.pd_table.pd_page_addr.pa :
+ sd_entry->u.bp.addr.pa;
+ if (sd_entry->valid)
+ return 0;
+
+ if (dev->is_pf) {
+ ret_code = irdma_hmc_sd_one(dev, hmc_info->hmc_fn_id,
+ sd_reg_val, idx->sd_idx,
+ sd_entry->entry_type, true);
+ if (ret_code)
+ goto error;
+ }
+
+ sd_entry->valid = true;
+ return 0;
+
+error:
+ if (chunk->bitmapbuf)
+ kfree(chunk->bitmapmem.va);
+
+ kfree(chunk->chunkmem.va);
+
+ return ret_code;
+}
+
+/**
+ * free_lvl2 - fee level 2 pble
+ * @pble_rsrc: pble resource management
+ * @palloc: level 2 pble allocation
+ */
+static void free_lvl2(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct irdma_pble_alloc *palloc)
+{
+ u32 i;
+ struct irdma_pble_level2 *lvl2 = &palloc->level2;
+ struct irdma_pble_info *root = &lvl2->root;
+ struct irdma_pble_info *leaf = lvl2->leaf;
+
+ for (i = 0; i < lvl2->leaf_cnt; i++, leaf++) {
+ if (leaf->addr)
+ irdma_prm_return_pbles(&pble_rsrc->pinfo,
+ &leaf->chunkinfo);
+ else
+ break;
+ }
+
+ if (root->addr)
+ irdma_prm_return_pbles(&pble_rsrc->pinfo, &root->chunkinfo);
+
+ kfree(lvl2->leafmem.va);
+ lvl2->leaf = NULL;
+}
+
+/**
+ * get_lvl2_pble - get level 2 pble resource
+ * @pble_rsrc: pble resource management
+ * @palloc: level 2 pble allocation
+ */
+static enum irdma_status_code
+get_lvl2_pble(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct irdma_pble_alloc *palloc)
+{
+ u32 lf4k, lflast, total, i;
+ u32 pblcnt = PBLE_PER_PAGE;
+ u64 *addr;
+ struct irdma_pble_level2 *lvl2 = &palloc->level2;
+ struct irdma_pble_info *root = &lvl2->root;
+ struct irdma_pble_info *leaf;
+ enum irdma_status_code ret_code;
+ u64 fpm_addr;
+
+ /* number of full 512 (4K) leafs) */
+ lf4k = palloc->total_cnt >> 9;
+ lflast = palloc->total_cnt % PBLE_PER_PAGE;
+ total = (lflast == 0) ? lf4k : lf4k + 1;
+ lvl2->leaf_cnt = total;
+
+ lvl2->leafmem.size = (sizeof(*leaf) * total);
+ lvl2->leafmem.va = kzalloc(lvl2->leafmem.size, GFP_ATOMIC);
+ if (!lvl2->leafmem.va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ lvl2->leaf = lvl2->leafmem.va;
+ leaf = lvl2->leaf;
+ ret_code = irdma_prm_get_pbles(&pble_rsrc->pinfo, &root->chunkinfo,
+ total << 3, &root->addr, &fpm_addr);
+ if (ret_code) {
+ kfree(lvl2->leafmem.va);
+ lvl2->leaf = NULL;
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ root->idx = fpm_to_idx(pble_rsrc, fpm_addr);
+ root->cnt = total;
+ addr = (u64 *)(uintptr_t)root->addr;
+ for (i = 0; i < total; i++, leaf++) {
+ pblcnt = (lflast && ((i + 1) == total)) ?
+ lflast : PBLE_PER_PAGE;
+ ret_code = irdma_prm_get_pbles(&pble_rsrc->pinfo,
+ &leaf->chunkinfo, pblcnt << 3,
+ &leaf->addr, &fpm_addr);
+ if (ret_code)
+ goto error;
+
+ leaf->idx = fpm_to_idx(pble_rsrc, fpm_addr);
+
+ leaf->cnt = pblcnt;
+ *addr = (u64)leaf->idx;
+ addr++;
+ }
+
+ palloc->level = PBLE_LEVEL_2;
+ pble_rsrc->stats_lvl2++;
+ return 0;
+
+error:
+ free_lvl2(pble_rsrc, palloc);
+
+ return IRDMA_ERR_NO_MEMORY;
+}
+
+/**
+ * get_lvl1_pble - get level 1 pble resource
+ * @pble_rsrc: pble resource management
+ * @palloc: level 1 pble allocation
+ */
+static enum irdma_status_code
+get_lvl1_pble(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct irdma_pble_alloc *palloc)
+{
+ enum irdma_status_code ret_code;
+ u64 fpm_addr, vaddr;
+ struct irdma_pble_info *lvl1 = &palloc->level1;
+
+ ret_code = irdma_prm_get_pbles(&pble_rsrc->pinfo, &lvl1->chunkinfo,
+ palloc->total_cnt << 3, &vaddr,
+ &fpm_addr);
+ if (ret_code)
+ return IRDMA_ERR_NO_MEMORY;
+
+ lvl1->addr = vaddr;
+ palloc->level = PBLE_LEVEL_1;
+ lvl1->idx = fpm_to_idx(pble_rsrc, fpm_addr);
+ lvl1->cnt = palloc->total_cnt;
+ pble_rsrc->stats_lvl1++;
+
+ return 0;
+}
+
+/**
+ * get_lvl1_lvl2_pble - calls get_lvl1 and get_lvl2 pble routine
+ * @pble_rsrc: pble resources
+ * @palloc: contains all inforamtion regarding pble (idx + pble addr)
+ * @level1_only: flag for a level 1 PBLE
+ */
+static enum irdma_status_code
+get_lvl1_lvl2_pble(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct irdma_pble_alloc *palloc, bool level1_only)
+{
+ enum irdma_status_code status = 0;
+
+ status = get_lvl1_pble(pble_rsrc, palloc);
+ if (!status || level1_only || palloc->total_cnt <= PBLE_PER_PAGE)
+ return status;
+
+ status = get_lvl2_pble(pble_rsrc, palloc);
+
+ return status;
+}
+
+/**
+ * irdma_get_pble - allocate pbles from the prm
+ * @pble_rsrc: pble resources
+ * @palloc: contains all inforamtion regarding pble (idx + pble addr)
+ * @pble_cnt: #of pbles requested
+ * @level1_only: true if only pble level 1 to acquire
+ */
+enum irdma_status_code irdma_get_pble(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct irdma_pble_alloc *palloc,
+ u32 pble_cnt, bool level1_only)
+{
+ enum irdma_status_code status = 0;
+ unsigned long flags;
+ int max_sds = 0;
+ int i;
+
+ palloc->total_cnt = pble_cnt;
+ palloc->level = PBLE_LEVEL_0;
+ spin_lock_irqsave(&pble_rsrc->pble_lock, flags);
+ /*check first to see if we can get pble's without acquiring
+ * additional sd's
+ */
+ status = get_lvl1_lvl2_pble(pble_rsrc, palloc, level1_only);
+ if (!status)
+ goto exit;
+
+ max_sds = (palloc->total_cnt >> 18) + 1;
+ for (i = 0; i < max_sds; i++) {
+ status = add_pble_prm(pble_rsrc);
+ if (status)
+ break;
+
+ status = get_lvl1_lvl2_pble(pble_rsrc, palloc, level1_only);
+ /* if level1_only, only go through it once */
+ if (!status || level1_only)
+ break;
+ }
+
+exit:
+ if (!status) {
+ pble_rsrc->allocdpbles += pble_cnt;
+ pble_rsrc->stats_alloc_ok++;
+ } else {
+ pble_rsrc->stats_alloc_fail++;
+ }
+ spin_unlock_irqrestore(&pble_rsrc->pble_lock, flags);
+
+ return status;
+}
+
+/**
+ * irdma_free_pble - put pbles back into prm
+ * @pble_rsrc: pble resources
+ * @palloc: contains all information regarding pble resource being freed
+ */
+void irdma_free_pble(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct irdma_pble_alloc *palloc)
+{
+ pble_rsrc->freedpbles += palloc->total_cnt;
+
+ if (palloc->level == PBLE_LEVEL_2)
+ free_lvl2(pble_rsrc, palloc);
+ else
+ irdma_prm_return_pbles(&pble_rsrc->pinfo,
+ &palloc->level1.chunkinfo);
+ pble_rsrc->stats_alloc_freed++;
+}
diff --git a/drivers/infiniband/hw/irdma/pble.h b/drivers/infiniband/hw/irdma/pble.h
new file mode 100644
index 000000000000..6f54ea43c00a
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/pble.h
@@ -0,0 +1,136 @@
+/* SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef IRDMA_PBLE_H
+#define IRDMA_PBLE_H
+
+#define PBLE_SHIFT 6
+#define PBLE_PER_PAGE 512
+#define HMC_PAGED_BP_SHIFT 12
+#define PBLE_512_SHIFT 9
+#define PBLE_INVALID_IDX 0xffffffff
+
+enum irdma_pble_level {
+ PBLE_LEVEL_0 = 0,
+ PBLE_LEVEL_1 = 1,
+ PBLE_LEVEL_2 = 2,
+};
+
+enum irdma_alloc_type {
+ PBLE_NO_ALLOC = 0,
+ PBLE_SD_CONTIGOUS = 1,
+ PBLE_SD_PAGED = 2,
+};
+
+struct irdma_chunk;
+
+struct irdma_pble_chunkinfo {
+ struct irdma_chunk *pchunk;
+ u64 bit_idx;
+ u64 bits_used;
+};
+
+struct irdma_pble_info {
+ u64 addr;
+ u32 idx;
+ u32 cnt;
+ struct irdma_pble_chunkinfo chunkinfo;
+};
+
+struct irdma_pble_level2 {
+ struct irdma_pble_info root;
+ struct irdma_pble_info *leaf;
+ struct irdma_virt_mem leafmem;
+ u32 leaf_cnt;
+};
+
+struct irdma_pble_alloc {
+ u32 total_cnt;
+ enum irdma_pble_level level;
+ union {
+ struct irdma_pble_info level1;
+ struct irdma_pble_level2 level2;
+ };
+};
+
+struct sd_pd_idx {
+ u32 sd_idx;
+ u32 pd_idx;
+ u32 rel_pd_idx;
+};
+
+struct irdma_add_page_info {
+ struct irdma_chunk *chunk;
+ struct irdma_hmc_sd_entry *sd_entry;
+ struct irdma_hmc_info *hmc_info;
+ struct sd_pd_idx idx;
+ u32 pages;
+};
+
+struct irdma_chunk {
+ struct list_head list;
+ struct irdma_dma_info dmainfo;
+ void *bitmapbuf;
+
+ u32 sizeofbitmap;
+ u64 size;
+ u64 vaddr;
+ u64 fpm_addr;
+ u32 pg_cnt;
+ enum irdma_alloc_type type;
+ struct irdma_sc_dev *dev;
+ struct irdma_virt_mem bitmapmem;
+ struct irdma_virt_mem chunkmem;
+};
+
+struct irdma_pble_prm {
+ struct list_head clist;
+ spinlock_t prm_lock; /* protect prm bitmap */
+ u64 total_pble_alloc;
+ u64 free_pble_cnt;
+ u8 pble_shift;
+};
+
+struct irdma_hmc_pble_rsrc {
+ u32 unallocated_pble;
+ spinlock_t pble_lock; /* to serialize PBLE resource acquisition */
+ struct irdma_sc_dev *dev;
+ u64 fpm_base_addr;
+ u64 next_fpm_addr;
+ struct irdma_pble_prm pinfo;
+ u64 allocdpbles;
+ u64 freedpbles;
+ u32 stats_direct_sds;
+ u32 stats_paged_sds;
+ u64 stats_alloc_ok;
+ u64 stats_alloc_fail;
+ u64 stats_alloc_freed;
+ u64 stats_lvl1;
+ u64 stats_lvl2;
+};
+
+void irdma_destroy_pble_prm(struct irdma_hmc_pble_rsrc *pble_rsrc);
+enum irdma_status_code
+irdma_hmc_init_pble(struct irdma_sc_dev *dev,
+ struct irdma_hmc_pble_rsrc *pble_rsrc);
+void irdma_free_pble(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct irdma_pble_alloc *palloc);
+enum irdma_status_code irdma_get_pble(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ struct irdma_pble_alloc *palloc,
+ u32 pble_cnt, bool level1_only);
+enum irdma_status_code irdma_prm_add_pble_mem(struct irdma_pble_prm *pprm,
+ struct irdma_chunk *pchunk);
+enum irdma_status_code
+irdma_prm_get_pbles(struct irdma_pble_prm *pprm,
+ struct irdma_pble_chunkinfo *chunkinfo, u32 mem_size,
+ u64 *vaddr, u64 *fpm_addr);
+void irdma_prm_return_pbles(struct irdma_pble_prm *pprm,
+ struct irdma_pble_chunkinfo *chunkinfo);
+void irdma_pble_acquire_lock(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ unsigned long *flags);
+void irdma_pble_release_lock(struct irdma_hmc_pble_rsrc *pble_rsrc,
+ unsigned long *flags);
+void irdma_pble_free_paged_mem(struct irdma_chunk *chunk);
+enum irdma_status_code irdma_pble_get_paged_mem(struct irdma_chunk *chunk,
+ int pg_cnt);
+#endif /* IRDMA_PBLE_H */
--
2.21.0
^ permalink raw reply related
* [RFC 09/20] RDMA/irdma: Add QoS definitions
From: Jeff Kirsher @ 2019-09-26 16:45 UTC (permalink / raw)
To: dledford, jgg, gregkh; +Cc: Mustafa Ismail, netdev, linux-rdma, Shiraz Saleem
In-Reply-To: <20190926164519.10471-1-jeffrey.t.kirsher@intel.com>
From: Mustafa Ismail <mustafa.ismail@intel.com>
Add definitions for managing the RDMA HW work scheduler (WS) tree.
A WS node is created via a control QP operation with the bandwidth
allocation, arbitration scheme, and traffic class of the QP specified.
The Qset handle returned associates the QoS parameters for the QP.
The Qset is registered with the LAN and a equivalent node is created
in the LAN packet scheduler tree.
Signed-off-by: Mustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
---
drivers/infiniband/hw/irdma/ws.c | 396 +++++++++++++++++++++++++++++++
drivers/infiniband/hw/irdma/ws.h | 40 ++++
2 files changed, 436 insertions(+)
create mode 100644 drivers/infiniband/hw/irdma/ws.c
create mode 100644 drivers/infiniband/hw/irdma/ws.h
diff --git a/drivers/infiniband/hw/irdma/ws.c b/drivers/infiniband/hw/irdma/ws.c
new file mode 100644
index 000000000000..8ecebc1f2ec8
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/ws.c
@@ -0,0 +1,396 @@
+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "osdep.h"
+#include "status.h"
+#include "hmc.h"
+#include "defs.h"
+#include "type.h"
+#include "protos.h"
+
+#include "ws.h"
+
+/**
+ * irdma_alloc_node - Allocate a WS node and init
+ * @vsi: vsi pointer
+ * @user_pri: user priority
+ * @node_type: Type of node, leaf or parent
+ * @parent: parent node pointer
+ */
+static struct irdma_ws_node *irdma_alloc_node(struct irdma_sc_vsi *vsi,
+ u8 user_pri,
+ enum irdma_ws_node_type node_type,
+ struct irdma_ws_node *parent)
+{
+ struct irdma_virt_mem ws_mem;
+ struct irdma_ws_node *node;
+ u16 node_index = 0;
+
+ ws_mem.size = sizeof(struct irdma_ws_node);
+ ws_mem.va = kzalloc(ws_mem.size, GFP_ATOMIC);
+ if (!ws_mem.va)
+ return NULL;
+
+ if (parent || vsi->vm_vf_type == IRDMA_VF_TYPE) {
+ node_index = irdma_alloc_ws_node_id(vsi->dev);
+ if (node_index == IRDMA_WS_NODE_INVALID) {
+ kfree(ws_mem.va);
+ return NULL;
+ }
+ }
+
+ node = ws_mem.va;
+ node->index = node_index;
+ node->vsi_index = vsi->vsi_idx;
+ INIT_LIST_HEAD(&node->child_list_head);
+ if (node_type == WS_NODE_TYPE_LEAF) {
+ node->type_leaf = true;
+ node->traffic_class = vsi->qos[user_pri].traffic_class;
+ node->user_pri = user_pri;
+ node->rel_bw = vsi->qos[user_pri].rel_bw;
+ if (!node->rel_bw)
+ node->rel_bw = 1;
+
+ node->lan_qs_handle = vsi->qos[user_pri].lan_qos_handle;
+ node->prio_type = IRDMA_PRIO_WEIGHTED_RR;
+ } else {
+ node->rel_bw = 1;
+ node->prio_type = IRDMA_PRIO_WEIGHTED_RR;
+ }
+
+ node->parent = parent;
+
+ return node;
+}
+
+/**
+ * irdma_free_node - Free a WS node
+ * @vsi: VSI stricture of device
+ * @node: Pointer to node to free
+ */
+static void irdma_free_node(struct irdma_sc_vsi *vsi,
+ struct irdma_ws_node *node)
+{
+ struct irdma_virt_mem ws_mem;
+
+ if (node->index)
+ irdma_free_ws_node_id(vsi->dev, node->index);
+
+ ws_mem.va = node;
+ ws_mem.size = sizeof(struct irdma_ws_node);
+ kfree(ws_mem.va);
+}
+
+/**
+ * irdma_ws_cqp_cmd - Post CQP work scheduler node cmd
+ * @vsi: vsi pointer
+ * @node: pointer to node
+ * @cmd: add, remove or modify
+ */
+static enum irdma_status_code
+irdma_ws_cqp_cmd(struct irdma_sc_vsi *vsi, struct irdma_ws_node *node, u8 cmd)
+{
+ struct irdma_ws_node_info node_info = {};
+
+ node_info.id = node->index;
+ node_info.vsi = node->vsi_index;
+ if (node->parent)
+ node_info.parent_id = node->parent->index;
+ else
+ node_info.parent_id = node_info.id;
+
+ node_info.weight = node->rel_bw;
+ node_info.tc = node->traffic_class;
+ node_info.prio_type = node->prio_type;
+ node_info.type_leaf = node->type_leaf;
+ node_info.enable = node->enable;
+ if (irdma_cqp_ws_node_cmd(vsi->dev, cmd, &node_info)) {
+ dev_dbg(rfdev_to_dev(vsi->dev), "WS: CQP WS CMD failed\n");
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ if (node->type_leaf && cmd == IRDMA_OP_WS_ADD_NODE) {
+ node->qs_handle = node_info.qs_handle;
+ vsi->qos[node->user_pri].qs_handle = node_info.qs_handle;
+ }
+
+ return 0;
+}
+
+/**
+ * ws_find_node - Find SC WS node based on VSI id or TC
+ * @parent: parent node of First VSI or TC node
+ * @match_val: value to match
+ * @type: match type VSI/TC
+ */
+static struct irdma_ws_node *ws_find_node(struct irdma_ws_node *parent,
+ u16 match_val,
+ enum irdma_ws_match_type type)
+{
+ struct irdma_ws_node *node;
+
+ switch (type) {
+ case WS_MATCH_TYPE_VSI:
+ list_for_each_entry(node, &parent->child_list_head, siblings) {
+ if (node->vsi_index == match_val)
+ return node;
+ }
+ break;
+ case WS_MATCH_TYPE_TC:
+ list_for_each_entry(node, &parent->child_list_head, siblings) {
+ if (node->traffic_class == match_val)
+ return node;
+ }
+ break;
+ default:
+ break;
+ }
+
+ return NULL;
+}
+
+/**
+ * irdma_tc_in_use - Checks to see if a leaf node is in use
+ * @vsi: vsi pointer
+ * @user_pri: user priority
+ */
+static bool irdma_tc_in_use(struct irdma_sc_vsi *vsi, u8 user_pri)
+{
+ unsigned long flags;
+ int i;
+
+ spin_lock_irqsave(&vsi->qos[user_pri].lock, flags);
+ if (!list_empty(&vsi->qos[user_pri].qplist)) {
+ spin_unlock_irqrestore(&vsi->qos[user_pri].lock, flags);
+ return true;
+ }
+
+ /* Check if the traffic class associated with the given user priority
+ * is in use by any other user priority. If so, nothing left to do
+ */
+ for (i = 0; i < IRDMA_MAX_USER_PRIORITY; i++) {
+ if (vsi->qos[i].traffic_class == vsi->qos[user_pri].traffic_class &&
+ !list_empty(&vsi->qos[i].qplist)) {
+ spin_unlock_irqrestore(&vsi->qos[user_pri].lock, flags);
+ return true;
+ }
+ }
+ spin_unlock_irqrestore(&vsi->qos[user_pri].lock, flags);
+
+ return false;
+}
+
+/**
+ * irdma_remove_leaf - Remove leaf node unconditionally
+ * @vsi: vsi pointer
+ * @user_pri: user priority
+ */
+static void irdma_remove_leaf(struct irdma_sc_vsi *vsi, u8 user_pri)
+{
+ struct irdma_ws_node *ws_tree_root, *vsi_node, *tc_node;
+
+ ws_tree_root = vsi->dev->ws_tree_root;
+ if (!ws_tree_root)
+ return;
+
+ vsi_node = ws_find_node(ws_tree_root, vsi->vsi_idx,
+ WS_MATCH_TYPE_VSI);
+ if (!vsi_node)
+ return;
+
+ tc_node = ws_find_node(vsi_node,
+ vsi->qos[user_pri].traffic_class,
+ WS_MATCH_TYPE_TC);
+ if (!tc_node)
+ return;
+
+ irdma_ws_cqp_cmd(vsi, tc_node, IRDMA_OP_WS_DELETE_NODE);
+ irdma_lan_unregister_qset(vsi, tc_node);
+ list_del(&tc_node->siblings);
+ irdma_free_node(vsi, tc_node);
+ /* Check if VSI node can be freed */
+ if (list_empty(&vsi_node->child_list_head)) {
+ irdma_ws_cqp_cmd(vsi, vsi_node, IRDMA_OP_WS_DELETE_NODE);
+ list_del(&vsi_node->siblings);
+ irdma_free_node(vsi, vsi_node);
+ /* Free head node there are no remaining VSI nodes */
+ if (list_empty(&ws_tree_root->child_list_head)) {
+ irdma_ws_cqp_cmd(vsi, ws_tree_root,
+ IRDMA_OP_WS_DELETE_NODE);
+ irdma_free_node(vsi, ws_tree_root);
+ vsi->dev->ws_tree_root = NULL;
+ }
+ }
+}
+
+/**
+ * irdma_ws_add - Build work scheduler tree, set RDMA qs_handle
+ * @vsi: vsi pointer
+ * @user_pri: user priority
+ */
+enum irdma_status_code irdma_ws_add(struct irdma_sc_vsi *vsi, u8 user_pri)
+{
+ struct irdma_ws_node *ws_tree_root;
+ struct irdma_ws_node *vsi_node;
+ struct irdma_ws_node *tc_node;
+ u16 traffic_class;
+ enum irdma_status_code ret = 0;
+ int i;
+
+ mutex_lock(&vsi->dev->ws_mutex);
+ if (vsi->tc_change_pending) {
+ ret = IRDMA_ERR_NOT_READY;
+ goto exit;
+ }
+
+ ws_tree_root = vsi->dev->ws_tree_root;
+ if (!ws_tree_root) {
+ dev_dbg(rfdev_to_dev(vsi->dev), "WS: Creating root node\n");
+ ws_tree_root = irdma_alloc_node(vsi, user_pri,
+ WS_NODE_TYPE_PARENT, NULL);
+ if (!ws_tree_root) {
+ ret = IRDMA_ERR_NO_MEMORY;
+ goto exit;
+ }
+
+ ret = irdma_ws_cqp_cmd(vsi, ws_tree_root, IRDMA_OP_WS_ADD_NODE);
+ if (ret) {
+ irdma_free_node(vsi, ws_tree_root);
+ goto exit;
+ }
+
+ vsi->dev->ws_tree_root = ws_tree_root;
+ }
+
+ /* Find a second tier node that matches the VSI */
+ vsi_node = ws_find_node(ws_tree_root, vsi->vsi_idx,
+ WS_MATCH_TYPE_VSI);
+
+ /* If VSI node doesn't exist, add one */
+ if (!vsi_node) {
+ dev_dbg(rfdev_to_dev(vsi->dev),
+ "WS: Node not found matching VSI %d\n", vsi->vsi_idx);
+ vsi_node = irdma_alloc_node(vsi, user_pri, WS_NODE_TYPE_PARENT,
+ ws_tree_root);
+ if (!vsi_node) {
+ ret = IRDMA_ERR_NO_MEMORY;
+ goto vsi_add_err;
+ }
+
+ ret = irdma_ws_cqp_cmd(vsi, vsi_node, IRDMA_OP_WS_ADD_NODE);
+ if (ret) {
+ irdma_free_node(vsi, vsi_node);
+ goto vsi_add_err;
+ }
+
+ list_add(&vsi_node->siblings, &ws_tree_root->child_list_head);
+ }
+
+ dev_dbg(rfdev_to_dev(vsi->dev),
+ "WS: Using node %d which represents VSI %d\n",
+ vsi_node->index, vsi->vsi_idx);
+ traffic_class = vsi->qos[user_pri].traffic_class;
+ tc_node = ws_find_node(vsi_node, traffic_class,
+ WS_MATCH_TYPE_TC);
+ if (!tc_node) {
+ /* Add leaf node */
+ dev_dbg(rfdev_to_dev(vsi->dev),
+ "WS: Node not found matching VSI %d and TC %d\n",
+ vsi->vsi_idx, traffic_class);
+ tc_node = irdma_alloc_node(vsi, user_pri, WS_NODE_TYPE_LEAF,
+ vsi_node);
+ if (!tc_node) {
+ ret = IRDMA_ERR_NO_MEMORY;
+ goto leaf_add_err;
+ }
+
+ ret = irdma_ws_cqp_cmd(vsi, tc_node, IRDMA_OP_WS_ADD_NODE);
+ if (ret) {
+ irdma_free_node(vsi, tc_node);
+ goto leaf_add_err;
+ }
+
+ list_add(&tc_node->siblings, &vsi_node->child_list_head);
+ /*
+ * callback to LAN to update the LAN tree with our node
+ */
+ ret = irdma_lan_register_qset(vsi, tc_node);
+ if (ret)
+ goto reg_err;
+
+ tc_node->enable = true;
+ ret = irdma_ws_cqp_cmd(vsi, tc_node, IRDMA_OP_WS_MODIFY_NODE);
+ if (ret)
+ goto reg_err;
+ }
+ dev_dbg(rfdev_to_dev(vsi->dev),
+ "WS: Using node %d which represents VSI %d TC %d\n",
+ tc_node->index, vsi->vsi_idx, traffic_class);
+ /*
+ * Iterate through other UPs and update the QS handle if they have
+ * a matching traffic class.
+ */
+ for (i = 0; i < IRDMA_MAX_USER_PRIORITY; i++) {
+ if (vsi->qos[i].traffic_class == traffic_class) {
+ vsi->qos[i].qs_handle = tc_node->qs_handle;
+ vsi->qos[i].lan_qos_handle = tc_node->lan_qs_handle;
+ vsi->qos[i].l2_sched_node_id = tc_node->l2_sched_node_id;
+ }
+ }
+ goto exit;
+
+leaf_add_err:
+ if (list_empty(&vsi_node->child_list_head)) {
+ if (irdma_ws_cqp_cmd(vsi, vsi_node, IRDMA_OP_WS_DELETE_NODE))
+ goto exit;
+ list_del(&vsi_node->siblings);
+ irdma_free_node(vsi, vsi_node);
+ }
+
+vsi_add_err:
+ /* Free head node there are no remaining VSI nodes */
+ if (list_empty(&ws_tree_root->child_list_head)) {
+ irdma_ws_cqp_cmd(vsi, ws_tree_root, IRDMA_OP_WS_DELETE_NODE);
+ vsi->dev->ws_tree_root = NULL;
+ irdma_free_node(vsi, ws_tree_root);
+ }
+
+exit:
+ mutex_unlock(&vsi->dev->ws_mutex);
+ return ret;
+
+reg_err:
+ mutex_unlock(&vsi->dev->ws_mutex);
+ irdma_ws_remove(vsi, user_pri);
+ return ret;
+}
+
+/**
+ * irdma_ws_remove - Free WS scheduler node, update WS tree
+ * @vsi: vsi pointer
+ * @user_pri: user priority
+ */
+void irdma_ws_remove(struct irdma_sc_vsi *vsi, u8 user_pri)
+{
+ mutex_lock(&vsi->dev->ws_mutex);
+ if (irdma_tc_in_use(vsi, user_pri))
+ goto exit;
+
+ irdma_remove_leaf(vsi, user_pri);
+exit:
+ mutex_unlock(&vsi->dev->ws_mutex);
+}
+
+/**
+ * irdma_ws_reset - Reset entire WS tree
+ * @vsi: vsi pointer
+ */
+void irdma_ws_reset(struct irdma_sc_vsi *vsi)
+{
+ u8 i;
+
+ mutex_lock(&vsi->dev->ws_mutex);
+ for (i = 0; i < IRDMA_MAX_USER_PRIORITY; ++i)
+ irdma_remove_leaf(vsi, i);
+ mutex_unlock(&vsi->dev->ws_mutex);
+}
diff --git a/drivers/infiniband/hw/irdma/ws.h b/drivers/infiniband/hw/irdma/ws.h
new file mode 100644
index 000000000000..664c9f38b16b
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/ws.h
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef IRDMA_WS_H
+#define IRDMA_WS_H
+
+#include "osdep.h"
+
+enum irdma_ws_node_type {
+ WS_NODE_TYPE_PARENT,
+ WS_NODE_TYPE_LEAF,
+};
+
+enum irdma_ws_match_type {
+ WS_MATCH_TYPE_VSI,
+ WS_MATCH_TYPE_TC,
+};
+
+struct irdma_ws_node {
+ struct list_head siblings;
+ struct list_head child_list_head;
+ struct irdma_ws_node *parent;
+ u64 lan_qs_handle; /* opaque handle used by LAN */
+ u32 l2_sched_node_id;
+ u16 index;
+ u16 qs_handle;
+ u16 vsi_index;
+ u8 traffic_class;
+ u8 user_pri;
+ u8 rel_bw;
+ u8 abstraction_layer; /* used for splitting a TC */
+ u8 prio_type;
+ bool type_leaf;
+ bool enable;
+};
+
+enum irdma_status_code irdma_ws_add(struct irdma_sc_vsi *vsi, u8 user_pri);
+void irdma_ws_remove(struct irdma_sc_vsi *vsi, u8 user_pri);
+void irdma_ws_reset(struct irdma_sc_vsi *vsi);
+#endif /* IRDMA_WS_H */
--
2.21.0
^ permalink raw reply related
* [RFC 05/20] RDMA/irdma: Implement device initialization definitions
From: Jeff Kirsher @ 2019-09-26 16:45 UTC (permalink / raw)
To: dledford, jgg, gregkh; +Cc: Mustafa Ismail, netdev, linux-rdma, Shiraz Saleem
In-Reply-To: <20190926164519.10471-1-jeffrey.t.kirsher@intel.com>
From: Mustafa Ismail <mustafa.ismail@intel.com>
Implement device initialization routines, interrupt set-up,
and allocate object bit-map tracking structures.
Also, add device specific attributes and register definitions.
Signed-off-by: Mustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
---
drivers/infiniband/hw/irdma/hw.c | 2564 +++++++++++++++++++++++
drivers/infiniband/hw/irdma/i40iw_hw.c | 210 ++
drivers/infiniband/hw/irdma/i40iw_hw.h | 163 ++
drivers/infiniband/hw/irdma/icrdma_hw.c | 75 +
drivers/infiniband/hw/irdma/icrdma_hw.h | 63 +
5 files changed, 3075 insertions(+)
create mode 100644 drivers/infiniband/hw/irdma/hw.c
create mode 100644 drivers/infiniband/hw/irdma/i40iw_hw.c
create mode 100644 drivers/infiniband/hw/irdma/i40iw_hw.h
create mode 100644 drivers/infiniband/hw/irdma/icrdma_hw.c
create mode 100644 drivers/infiniband/hw/irdma/icrdma_hw.h
diff --git a/drivers/infiniband/hw/irdma/hw.c b/drivers/infiniband/hw/irdma/hw.c
new file mode 100644
index 000000000000..5c3dda48088e
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/hw.c
@@ -0,0 +1,2564 @@
+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "main.h"
+
+static struct irdma_rsrc_limits rsrc_limits_table[] = {
+ [0] = {
+ .qplimit = 4096,
+ },
+ [1] = {
+ .qplimit = 128,
+ },
+ [2] = {
+ .qplimit = 1024,
+ },
+ [3] = {
+ .qplimit = 2048,
+ },
+ [4] = {
+ .qplimit = 16384,
+ },
+ [5] = {
+ .qplimit = 65536,
+ },
+};
+
+/* types of hmc objects */
+static enum irdma_hmc_rsrc_type iw_hmc_obj_types[] = {
+ IRDMA_HMC_IW_QP,
+ IRDMA_HMC_IW_CQ,
+ IRDMA_HMC_IW_HTE,
+ IRDMA_HMC_IW_ARP,
+ IRDMA_HMC_IW_APBVT_ENTRY,
+ IRDMA_HMC_IW_MR,
+ IRDMA_HMC_IW_XF,
+ IRDMA_HMC_IW_XFFL,
+ IRDMA_HMC_IW_Q1,
+ IRDMA_HMC_IW_Q1FL,
+ IRDMA_HMC_IW_TIMER,
+ IRDMA_HMC_IW_FSIMC,
+ IRDMA_HMC_IW_FSIAV,
+ IRDMA_HMC_IW_RRF,
+ IRDMA_HMC_IW_RRFFL,
+ IRDMA_HMC_IW_HDR,
+ IRDMA_HMC_IW_MD,
+ IRDMA_HMC_IW_OOISC,
+ IRDMA_HMC_IW_OOISCFFL,
+};
+
+/**
+ * irdma_iwarp_ce_handler - handle iwarp completions
+ * @iwcq: iwarp cq receiving event
+ */
+static void irdma_iwarp_ce_handler(struct irdma_sc_cq *iwcq)
+{
+ struct irdma_cq *cq = iwcq->back_cq;
+
+ if (cq->ibcq.comp_handler)
+ cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
+}
+
+/**
+ * irdma_puda_ce_handler - handle puda completion events
+ * @rf: RDMA PCI function
+ * @cq: puda completion q for event
+ */
+static void irdma_puda_ce_handler(struct irdma_pci_f *rf,
+ struct irdma_sc_cq *cq)
+{
+ struct irdma_sc_dev *dev = (struct irdma_sc_dev *)&rf->sc_dev;
+ enum irdma_status_code status;
+ u32 compl_error;
+
+ do {
+ status = irdma_puda_poll_cmpl(dev, cq, &compl_error);
+ if (status == IRDMA_ERR_Q_EMPTY)
+ break;
+ if (status) {
+ dev_dbg(rfdev_to_dev(dev), "ERR: puda status = %d\n",
+ status);
+ break;
+ }
+ if (compl_error) {
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: puda compl_err =0x%x\n", compl_error);
+ break;
+ }
+ } while (1);
+
+ dev->ccq_ops->ccq_arm(cq);
+}
+
+/**
+ * irdma_process_ceq - handle ceq for completions
+ * @rf: RDMA PCI function
+ * @ceq: ceq having cq for completion
+ */
+static void irdma_process_ceq(struct irdma_pci_f *rf, struct irdma_ceq *ceq)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_sc_ceq *sc_ceq;
+ struct irdma_sc_cq *cq;
+
+ sc_ceq = &ceq->sc_ceq;
+ do {
+ cq = dev->ceq_ops->process_ceq(dev, sc_ceq);
+ if (!cq)
+ break;
+
+ if (cq->cq_type == IRDMA_CQ_TYPE_CQP)
+ up(&rf->cqp.cqp_compl_sem);
+ else if (cq->cq_type == IRDMA_CQ_TYPE_IWARP)
+ irdma_iwarp_ce_handler(cq);
+ else if (cq->cq_type == IRDMA_CQ_TYPE_ILQ ||
+ cq->cq_type == IRDMA_CQ_TYPE_IEQ)
+ irdma_puda_ce_handler(rf, cq);
+ } while (1);
+}
+
+/**
+ * irdma_process_aeq - handle aeq events
+ * @rf: RDMA PCI function
+ */
+static void irdma_process_aeq(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_aeq *aeq = &rf->aeq;
+ struct irdma_sc_aeq *sc_aeq = &aeq->sc_aeq;
+ struct irdma_aeqe_info aeinfo;
+ struct irdma_aeqe_info *info = &aeinfo;
+ int ret;
+ struct irdma_qp *iwqp = NULL;
+ struct irdma_sc_cq *cq = NULL;
+ struct irdma_cq *iwcq = NULL;
+ struct irdma_sc_qp *qp = NULL;
+ struct irdma_qp_host_ctx_info *ctx_info = NULL;
+ unsigned long flags;
+
+ u32 aeqcnt = 0;
+
+ if (!sc_aeq->size)
+ return;
+
+ do {
+ memset(info, 0, sizeof(*info));
+ ret = dev->aeq_ops->get_next_aeqe(sc_aeq, info);
+ if (ret)
+ break;
+
+ aeqcnt++;
+ dev_dbg(rfdev_to_dev(dev),
+ "AEQ: ae_id = 0x%x bool qp=%d qp_id = %d\n",
+ info->ae_id, info->qp, info->qp_cq_id);
+ if (info->qp) {
+ spin_lock_irqsave(&rf->qptable_lock, flags);
+ iwqp = rf->qp_table[info->qp_cq_id];
+ if (!iwqp) {
+ spin_unlock_irqrestore(&rf->qptable_lock,
+ flags);
+ dev_dbg(rfdev_to_dev(dev),
+ "AEQ: qp_id %d is already freed\n",
+ info->qp_cq_id);
+ continue;
+ }
+ irdma_add_ref(&iwqp->ibqp);
+ spin_unlock_irqrestore(&rf->qptable_lock, flags);
+ qp = &iwqp->sc_qp;
+ spin_lock_irqsave(&iwqp->lock, flags);
+ iwqp->hw_tcp_state = info->tcp_state;
+ iwqp->hw_iwarp_state = info->iwarp_state;
+ iwqp->last_aeq = info->ae_id;
+ spin_unlock_irqrestore(&iwqp->lock, flags);
+ ctx_info = &iwqp->ctx_info;
+ if (rdma_protocol_roce(&iwqp->iwdev->iwibdev->ibdev, 1))
+ ctx_info->roce_info->err_rq_idx_valid = true;
+ else
+ ctx_info->iwarp_info->err_rq_idx_valid = true;
+ } else {
+ if (info->ae_id != IRDMA_AE_CQ_OPERATION_ERROR)
+ continue;
+ }
+
+ switch (info->ae_id) {
+ struct irdma_cm_node *cm_node;
+ case IRDMA_AE_LLP_CONNECTION_ESTABLISHED:
+ cm_node = iwqp->cm_node;
+ if (cm_node->accept_pend) {
+ atomic_dec(&cm_node->listener->pend_accepts_cnt);
+ cm_node->accept_pend = 0;
+ }
+ iwqp->rts_ae_rcvd = 1;
+ wake_up_interruptible(&iwqp->waitq);
+ break;
+ case IRDMA_AE_LLP_FIN_RECEIVED:
+ case IRDMA_AE_RDMAP_ROE_BAD_LLP_CLOSE:
+ if (qp->term_flags)
+ break;
+ if (atomic_inc_return(&iwqp->close_timer_started) == 1) {
+ iwqp->hw_tcp_state = IRDMA_TCP_STATE_CLOSE_WAIT;
+ if (iwqp->hw_tcp_state == IRDMA_TCP_STATE_CLOSE_WAIT &&
+ iwqp->ibqp_state == IB_QPS_RTS) {
+ irdma_next_iw_state(iwqp,
+ IRDMA_QP_STATE_CLOSING,
+ 0, 0, 0);
+ irdma_cm_disconn(iwqp);
+ }
+ iwqp->cm_id->add_ref(iwqp->cm_id);
+ irdma_schedule_cm_timer(iwqp->cm_node,
+ (struct irdma_puda_buf *)iwqp,
+ IRDMA_TIMER_TYPE_CLOSE,
+ 1, 0);
+ }
+ break;
+ case IRDMA_AE_LLP_CLOSE_COMPLETE:
+ if (qp->term_flags)
+ irdma_terminate_done(qp, 0);
+ else
+ irdma_cm_disconn(iwqp);
+ break;
+ case IRDMA_AE_BAD_CLOSE:
+ /* fall through */
+ case IRDMA_AE_RESET_SENT:
+ irdma_next_iw_state(iwqp, IRDMA_QP_STATE_ERROR, 1, 0,
+ 0);
+ irdma_cm_disconn(iwqp);
+ break;
+ case IRDMA_AE_LLP_CONNECTION_RESET:
+ if (atomic_read(&iwqp->close_timer_started))
+ break;
+ irdma_cm_disconn(iwqp);
+ break;
+ case IRDMA_AE_QP_SUSPEND_COMPLETE:
+ atomic_dec(&iwqp->sc_qp.vsi->qp_suspend_reqs);
+ wake_up(&iwqp->iwdev->suspend_wq);
+ break;
+ case IRDMA_AE_TERMINATE_SENT:
+ irdma_terminate_send_fin(qp);
+ break;
+ case IRDMA_AE_LLP_TERMINATE_RECEIVED:
+ irdma_terminate_received(qp, info);
+ break;
+ case IRDMA_AE_CQ_OPERATION_ERROR:
+ dev_err(rfdev_to_dev(dev),
+ "Processing an iWARP related AE for CQ misc = 0x%04X\n",
+ info->ae_id);
+ cq = (struct irdma_sc_cq *)(unsigned long)
+ info->compl_ctx;
+
+ iwcq = (struct irdma_cq *)cq->back_cq;
+
+ if (iwcq->ibcq.event_handler) {
+ struct ib_event ibevent;
+
+ ibevent.device = iwcq->ibcq.device;
+ ibevent.event = IB_EVENT_CQ_ERR;
+ ibevent.element.cq = &iwcq->ibcq;
+ iwcq->ibcq.event_handler(&ibevent,
+ iwcq->ibcq.cq_context);
+ }
+ break;
+ case IRDMA_AE_LLP_DOUBT_REACHABILITY:
+ case IRDMA_AE_RESOURCE_EXHAUSTION:
+ break;
+ case IRDMA_AE_PRIV_OPERATION_DENIED:
+ case IRDMA_AE_STAG_ZERO_INVALID:
+ case IRDMA_AE_IB_RREQ_AND_Q1_FULL:
+ case IRDMA_AE_DDP_UBE_INVALID_DDP_VERSION:
+ case IRDMA_AE_DDP_UBE_INVALID_MO:
+ case IRDMA_AE_DDP_UBE_INVALID_QN:
+ case IRDMA_AE_DDP_NO_L_BIT:
+ case IRDMA_AE_RDMAP_ROE_INVALID_RDMAP_VERSION:
+ case IRDMA_AE_RDMAP_ROE_UNEXPECTED_OPCODE:
+ case IRDMA_AE_ROE_INVALID_RDMA_READ_REQUEST:
+ case IRDMA_AE_ROE_INVALID_RDMA_WRITE_OR_READ_RESP:
+ case IRDMA_AE_INVALID_ARP_ENTRY:
+ case IRDMA_AE_INVALID_TCP_OPTION_RCVD:
+ case IRDMA_AE_STALE_ARP_ENTRY:
+ case IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR:
+ case IRDMA_AE_LLP_SEGMENT_TOO_SMALL:
+ case IRDMA_AE_LLP_SYN_RECEIVED:
+ case IRDMA_AE_LLP_TOO_MANY_RETRIES:
+ case IRDMA_AE_LCE_QP_CATASTROPHIC:
+ case IRDMA_AE_LCE_FUNCTION_CATASTROPHIC:
+ case IRDMA_AE_LCE_CQ_CATASTROPHIC:
+ case IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG:
+ if (rdma_protocol_roce(&iwqp->iwdev->iwibdev->ibdev, 1))
+ ctx_info->roce_info->err_rq_idx_valid = false;
+ else
+ ctx_info->iwarp_info->err_rq_idx_valid = false;
+ /* fall through */
+ default:
+ dev_err(rfdev_to_dev(dev),
+ "abnormal ae_id = 0x%x bool qp=%d qp_id = %d\n",
+ info->ae_id, info->qp, info->qp_cq_id);
+ if (rdma_protocol_roce(&iwqp->iwdev->iwibdev->ibdev, 1)) {
+ if (!info->sq && ctx_info->roce_info->err_rq_idx_valid) {
+ ctx_info->roce_info->err_rq_idx = info->wqe_idx;
+ ret = dev->iw_priv_qp_ops->qp_setctx_roce(&iwqp->sc_qp,
+ iwqp->host_ctx.va,
+ ctx_info);
+ }
+ irdma_cm_disconn(iwqp);
+ break;
+ }
+ if (!info->sq && ctx_info->iwarp_info->err_rq_idx_valid) {
+ ctx_info->iwarp_info->err_rq_idx = info->wqe_idx;
+ ctx_info->tcp_info_valid = false;
+ ctx_info->iwarp_info_valid = false;
+ ret = dev->iw_priv_qp_ops->qp_setctx(&iwqp->sc_qp,
+ iwqp->host_ctx.va,
+ ctx_info);
+ }
+ if (iwqp->hw_iwarp_state != IRDMA_QP_STATE_RTS &&
+ iwqp->hw_iwarp_state != IRDMA_QP_STATE_TERMINATE) {
+ irdma_next_iw_state(iwqp, IRDMA_QP_STATE_ERROR, 1, 0, 0);
+ irdma_cm_disconn(iwqp);
+ } else {
+ iwqp->sc_qp.term_flags = 1;
+ irdma_next_iw_state(iwqp, IRDMA_QP_STATE_ERROR, 1, 0,
+ 0);
+ irdma_cm_disconn(iwqp);
+ }
+ break;
+ }
+ if (info->qp)
+ irdma_rem_ref(&iwqp->ibqp);
+ } while (1);
+
+ if (aeqcnt)
+ dev->aeq_ops->repost_aeq_entries(dev, aeqcnt);
+}
+
+/**
+ * irdma_enable_intr - set up device interrupts
+ * @dev: hardware control device structure
+ * @msix_id: id of the interrupt to be enabled
+ */
+static void irdma_ena_intr(struct irdma_sc_dev *dev, u32 msix_id)
+{
+ dev->irq_ops->irdma_en_irq(dev, msix_id);
+}
+
+/**
+ * irdma_dpc - tasklet for aeq and ceq 0
+ * @data: RDMA PCI function
+ */
+static void irdma_dpc(unsigned long data)
+{
+ struct irdma_pci_f *rf = (struct irdma_pci_f *)data;
+
+ if (rf->msix_shared)
+ irdma_process_ceq(rf, rf->ceqlist);
+ irdma_process_aeq(rf);
+ irdma_ena_intr(&rf->sc_dev, rf->iw_msixtbl[0].idx);
+}
+
+/**
+ * irdma_ceq_dpc - dpc handler for CEQ
+ * @data: data points to CEQ
+ */
+static void irdma_ceq_dpc(unsigned long data)
+{
+ struct irdma_ceq *iwceq = (struct irdma_ceq *)data;
+ struct irdma_pci_f *rf = iwceq->rf;
+
+ irdma_process_ceq(rf, iwceq);
+ irdma_ena_intr(&rf->sc_dev, iwceq->msix_idx);
+}
+
+/**
+ * irdma_save_msix_info - copy msix vector information to iwarp device
+ * @rf: RDMA PCI function
+ *
+ * Allocate iwdev msix table and copy the ldev msix info to the table
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_save_msix_info(struct irdma_pci_f *rf)
+{
+ struct irdma_priv_ldev *ldev = &rf->ldev;
+ struct irdma_qvlist_info *iw_qvlist;
+ struct irdma_qv_info *iw_qvinfo;
+ struct msix_entry *pmsix;
+ u32 ceq_idx;
+ u32 i;
+ u32 size;
+
+ if (!ldev->msix_count) {
+ pr_err("No MSI-X vectors for RDMA\n");
+ return IRDMA_ERR_CFG;
+ }
+
+ rf->msix_count = ldev->msix_count;
+ size = sizeof(struct irdma_msix_vector) * rf->msix_count;
+ size += sizeof(struct irdma_qvlist_info);
+ size += sizeof(struct irdma_qv_info) * rf->msix_count - 1;
+ rf->iw_msixtbl = kzalloc(size, GFP_KERNEL);
+ if (!rf->iw_msixtbl)
+ return IRDMA_ERR_NO_MEMORY;
+
+ rf->iw_qvlist = (struct irdma_qvlist_info *)
+ (&rf->iw_msixtbl[rf->msix_count]);
+ iw_qvlist = rf->iw_qvlist;
+ iw_qvinfo = iw_qvlist->qv_info;
+ iw_qvlist->num_vectors = rf->msix_count;
+ if (rf->msix_count <= num_online_cpus())
+ rf->msix_shared = true;
+
+ for (i = 0, ceq_idx = 0, pmsix = ldev->msix_entries; i < rf->msix_count;
+ i++, iw_qvinfo++, pmsix++) {
+ rf->iw_msixtbl[i].idx = pmsix->entry;
+ rf->iw_msixtbl[i].irq = pmsix->vector;
+ rf->iw_msixtbl[i].cpu_affinity = ceq_idx;
+ if (!i) {
+ iw_qvinfo->aeq_idx = 0;
+ if (rf->msix_shared)
+ iw_qvinfo->ceq_idx = ceq_idx++;
+ else
+ iw_qvinfo->ceq_idx = IRDMA_Q_INVALID_IDX;
+ } else {
+ iw_qvinfo->aeq_idx = IRDMA_Q_INVALID_IDX;
+ iw_qvinfo->ceq_idx = ceq_idx++;
+ }
+ iw_qvinfo->itr_idx = 3;
+ iw_qvinfo->v_idx = rf->iw_msixtbl[i].idx;
+ }
+
+ return 0;
+}
+
+/**
+ * irdma_irq_handler - interrupt handler for aeq and ceq0
+ * @irq: Interrupt request number
+ * @data: RDMA PCI function
+ */
+static irqreturn_t irdma_irq_handler(int irq, void *data)
+{
+ struct irdma_pci_f *rf = data;
+
+ tasklet_schedule(&rf->dpc_tasklet);
+
+ return IRQ_HANDLED;
+}
+
+/**
+ * irdma_ceq_handler - interrupt handler for ceq
+ * @irq: interrupt request number
+ * @data: ceq pointer
+ */
+static irqreturn_t irdma_ceq_handler(int irq, void *data)
+{
+ struct irdma_ceq *iwceq = data;
+
+ if (iwceq->irq != irq)
+ dev_err(rfdev_to_dev(&iwceq->rf->sc_dev),
+ "expected irq = %d received irq = %d\n", iwceq->irq,
+ irq);
+ tasklet_schedule(&iwceq->dpc_tasklet);
+
+ return IRQ_HANDLED;
+}
+
+/**
+ * irdma_destroy_irq - destroy device interrupts
+ * @rf: RDMA PCI function
+ * @msix_vec: msix vector to disable irq
+ * @dev_id: parameter to pass to free_irq (used during irq setup)
+ *
+ * The function is called when destroying aeq/ceq
+ */
+static void irdma_destroy_irq(struct irdma_pci_f *rf,
+ struct irdma_msix_vector *msix_vec, void *dev_id)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+
+ dev->irq_ops->irdma_dis_irq(dev, msix_vec->idx);
+ irq_set_affinity_hint(msix_vec->irq, NULL);
+ free_irq(msix_vec->irq, dev_id);
+}
+
+/**
+ * irdma_destroy_cqp - destroy control qp
+ * @rf: RDMA PCI function
+ * @free_hwcqp: 1 if hw cqp should be freed
+ *
+ * Issue destroy cqp request and
+ * free the resources associated with the cqp
+ */
+static void irdma_destroy_cqp(struct irdma_pci_f *rf, bool free_hwcqp)
+{
+ enum irdma_status_code status = 0;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_cqp *cqp = &rf->cqp;
+
+ if (free_hwcqp && dev->cqp_ops->cqp_destroy)
+ status = dev->cqp_ops->cqp_destroy(dev->cqp);
+ if (status)
+ dev_dbg(rfdev_to_dev(dev), "ERR: Destroy CQP failed %d\n",
+ status);
+
+ irdma_cleanup_pending_cqp_op(rf);
+ dma_free_coherent(hw_to_dev(dev->hw), cqp->sq.size, cqp->sq.va,
+ cqp->sq.pa);
+ cqp->sq.va = NULL;
+ kfree(cqp->scratch_array);
+ cqp->scratch_array = NULL;
+ kfree(cqp->cqp_requests);
+ cqp->cqp_requests = NULL;
+}
+
+/**
+ * irdma_destroy_aeq - destroy aeq
+ * @rf: RDMA PCI function
+ *
+ * Issue a destroy aeq request and
+ * free the resources associated with the aeq
+ * The function is called during driver unload
+ */
+static void irdma_destroy_aeq(struct irdma_pci_f *rf)
+{
+ enum irdma_status_code status = IRDMA_ERR_NOT_READY;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_aeq *aeq = &rf->aeq;
+
+ if (!rf->msix_shared)
+ irdma_destroy_irq(rf, rf->iw_msixtbl, (void *)rf);
+ if (rf->reset)
+ goto exit;
+
+ if (!dev->aeq_ops->aeq_destroy(&aeq->sc_aeq, 0, 1))
+ status = dev->aeq_ops->aeq_destroy_done(&aeq->sc_aeq);
+ if (status)
+ dev_dbg(rfdev_to_dev(dev), "ERR: Destroy AEQ failed %d\n",
+ status);
+
+exit:
+ dma_free_coherent(hw_to_dev(dev->hw), aeq->mem.size, aeq->mem.va,
+ aeq->mem.pa);
+ aeq->mem.va = NULL;
+}
+
+/**
+ * irdma_destroy_ceq - destroy ceq
+ * @rf: RDMA PCI function
+ * @iwceq: ceq to be destroyed
+ *
+ * Issue a destroy ceq request and
+ * free the resources associated with the ceq
+ */
+static void irdma_destroy_ceq(struct irdma_pci_f *rf, struct irdma_ceq *iwceq)
+{
+ enum irdma_status_code status;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+
+ if (rf->reset)
+ goto exit;
+
+ status = dev->ceq_ops->ceq_destroy(&iwceq->sc_ceq, 0, 1);
+ if (status) {
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: CEQ destroy command failed %d\n", status);
+ goto exit;
+ }
+
+ status = dev->ceq_ops->cceq_destroy_done(&iwceq->sc_ceq);
+ if (status)
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: CEQ destroy completion failed %d\n", status);
+exit:
+ dma_free_coherent(hw_to_dev(dev->hw), iwceq->mem.size, iwceq->mem.va,
+ iwceq->mem.pa);
+ iwceq->mem.va = NULL;
+}
+
+/**
+ * irdma_del_ceq_0 - destroy ceq 0
+ * @rf: RDMA PCI function
+ *
+ * Disable the ceq 0 interrupt and destroy the ceq 0
+ */
+static void irdma_del_ceq_0(struct irdma_pci_f *rf)
+{
+ struct irdma_ceq *iwceq = rf->ceqlist;
+ struct irdma_msix_vector *msix_vec;
+
+ if (rf->msix_shared) {
+ msix_vec = &rf->iw_msixtbl[0];
+ irdma_destroy_irq(rf, msix_vec, (void *)rf);
+ } else {
+ msix_vec = &rf->iw_msixtbl[1];
+ irdma_destroy_irq(rf, msix_vec, (void *)iwceq);
+ }
+ irdma_destroy_ceq(rf, iwceq);
+ rf->sc_dev.ceq_valid = false;
+ rf->ceqs_count = 0;
+}
+
+/**
+ * irdma_del_ceqs - destroy all ceq's except CEQ 0 // RT mode FSL
+ * @rf: RDMA PCI function
+ *
+ * Go through all of the device ceq's, except 0, and for each
+ * ceq disable the ceq interrupt and destroy the ceq
+ */
+static void irdma_del_ceqs(struct irdma_pci_f *rf)
+{
+ struct irdma_ceq *iwceq = &rf->ceqlist[1];
+ struct irdma_msix_vector *msix_vec;
+ u32 i = 0;
+
+ if (rf->msix_shared)
+ msix_vec = &rf->iw_msixtbl[1];
+ else
+ msix_vec = &rf->iw_msixtbl[2];
+
+ for (i = 1; i < rf->ceqs_count; i++, msix_vec++, iwceq++) {
+ irdma_destroy_irq(rf, msix_vec, (void *)iwceq);
+ irdma_cqp_ceq_cmd(&rf->sc_dev, &iwceq->sc_ceq,
+ IRDMA_OP_CEQ_DESTROY);
+ dma_free_coherent(hw_to_dev(rf->sc_dev.hw), iwceq->mem.size,
+ iwceq->mem.va, iwceq->mem.pa);
+ iwceq->mem.va = NULL;
+ }
+ rf->ceqs_count = 1;
+}
+
+/**
+ * irdma_destroy_ccq - destroy control cq
+ * @rf: RDMA PCI function
+ *
+ * Issue destroy ccq request and
+ * free the resources associated with the ccq
+ */
+static void irdma_destroy_ccq(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_ccq *ccq = &rf->ccq;
+ enum irdma_status_code status = 0;
+
+ if (!rf->reset)
+ status = dev->ccq_ops->ccq_destroy(dev->ccq, 0, true);
+ if (status)
+ dev_dbg(rfdev_to_dev(dev), "ERR: CCQ destroy failed %d\n",
+ status);
+ dma_free_coherent(hw_to_dev(dev->hw), ccq->mem_cq.size,
+ ccq->mem_cq.va, ccq->mem_cq.pa);
+ ccq->mem_cq.va = NULL;
+}
+
+/**
+ * irdma_close_hmc_objects_type - delete hmc objects of a given type
+ * @dev: iwarp device
+ * @obj_type: the hmc object type to be deleted
+ * @hmc_info: host memory info struct
+ * @is_pf: true if the function is PF otherwise false
+ * @reset: true if called before reset
+ */
+static void irdma_close_hmc_objects_type(struct irdma_sc_dev *dev,
+ enum irdma_hmc_rsrc_type obj_type,
+ struct irdma_hmc_info *hmc_info,
+ bool is_pf, bool reset)
+{
+ struct irdma_hmc_del_obj_info info = {};
+
+ info.hmc_info = hmc_info;
+ info.rsrc_type = obj_type;
+ info.count = hmc_info->hmc_obj[obj_type].cnt;
+ info.is_pf = is_pf;
+ if (dev->hmc_ops->del_hmc_object(dev, &info, reset))
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: del HMC obj of type %d failed\n", obj_type);
+}
+
+/**
+ * irdma_del_hmc_objects - remove all device hmc objects
+ * @dev: iwarp device
+ * @hmc_info: hmc_info to free
+ * @is_pf: true if hmc_info belongs to PF, not vf nor allocated
+ * by PF on behalf of VF
+ * @reset: true if called before reset
+ * @vers: hardware version
+ */
+static void irdma_del_hmc_objects(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info, bool is_pf,
+ bool reset, enum irdma_vers vers)
+{
+ unsigned int i;
+
+ for (i = 0; i < IW_HMC_OBJ_TYPE_NUM; i++) {
+ if (dev->hmc_info->hmc_obj[iw_hmc_obj_types[i]].cnt)
+ irdma_close_hmc_objects_type(dev, iw_hmc_obj_types[i],
+ hmc_info, is_pf, reset);
+ if (vers == IRDMA_GEN_1 && i == IRDMA_HMC_IW_TIMER)
+ break;
+ }
+}
+
+/**
+ * irdma_create_hmc_obj_type - create hmc object of a given type
+ * @dev: hardware control device structure
+ * @info: information for the hmc object to create
+ */
+static enum irdma_status_code
+irdma_create_hmc_obj_type(struct irdma_sc_dev *dev,
+ struct irdma_hmc_create_obj_info *info)
+{
+ return dev->hmc_ops->create_hmc_object(dev, info);
+}
+
+/**
+ * irdma_create_hmc_objs - create all hmc objects for the device
+ * @rf: RDMA PCI function
+ * @is_pf: true if the function is PF otherwise false
+ * @vers: HW version
+ *
+ * Create the device hmc objects and allocate hmc pages
+ * Return 0 if successful, otherwise clean up and return error
+ */
+static enum irdma_status_code
+irdma_create_hmc_objs(struct irdma_pci_f *rf, bool is_pf, enum irdma_vers vers)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_hmc_create_obj_info info = {};
+ enum irdma_status_code status = 0;
+ int i;
+
+ info.hmc_info = dev->hmc_info;
+ info.is_pf = is_pf;
+ info.entry_type = rf->sd_type;
+
+ for (i = 0; i < IW_HMC_OBJ_TYPE_NUM; i++) {
+ if (dev->hmc_info->hmc_obj[iw_hmc_obj_types[i]].cnt) {
+ info.rsrc_type = iw_hmc_obj_types[i];
+ info.count = dev->hmc_info->hmc_obj[info.rsrc_type].cnt;
+ info.add_sd_cnt = 0;
+ status = irdma_create_hmc_obj_type(dev, &info);
+ if (status) {
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: create obj type %d status = %d\n",
+ iw_hmc_obj_types[i], status);
+ break;
+ }
+ }
+ if (vers == IRDMA_GEN_1 && i == IRDMA_HMC_IW_TIMER)
+ break;
+ }
+
+ if (!status)
+ return dev->hmc_ops->static_hmc_pages_allocated(dev->cqp, 0,
+ dev->hmc_fn_id,
+ true, true);
+
+ while (i) {
+ i--;
+ /* destroy the hmc objects of a given type */
+ irdma_close_hmc_objects_type(dev, iw_hmc_obj_types[i],
+ dev->hmc_info, is_pf, false);
+ }
+
+ return status;
+}
+
+/**
+ * irdma_obj_aligned_mem - get aligned memory from device allocated memory
+ * @rf: RDMA PCI function
+ * @memptr: points to the memory addresses
+ * @size: size of memory needed
+ * @mask: mask for the aligned memory
+ *
+ * Get aligned memory of the requested size and
+ * update the memptr to point to the new aligned memory
+ * Return 0 if successful, otherwise return no memory error
+ */
+static enum irdma_status_code
+irdma_obj_aligned_mem(struct irdma_pci_f *rf, struct irdma_dma_mem *memptr,
+ u32 size, u32 mask)
+{
+ unsigned long va, newva;
+ unsigned long extra;
+
+ va = (unsigned long)rf->obj_next.va;
+ newva = va;
+ if (mask)
+ newva = ALIGN(va, (unsigned long)mask + 1ULL);
+ extra = newva - va;
+ memptr->va = (u8 *)va + extra;
+ memptr->pa = rf->obj_next.pa + extra;
+ memptr->size = size;
+ if ((memptr->va + size) > (rf->obj_mem.va + rf->obj_mem.size))
+ return IRDMA_ERR_NO_MEMORY;
+
+ rf->obj_next.va = memptr->va + size;
+ rf->obj_next.pa = memptr->pa + size;
+
+ return 0;
+}
+
+/**
+ * irdma_create_cqp - create control qp
+ * @rf: RDMA PCI function
+ *
+ * Return 0, if the cqp and all the resources associated with it
+ * are successfully created, otherwise return error
+ */
+static enum irdma_status_code irdma_create_cqp(struct irdma_pci_f *rf)
+{
+ enum irdma_status_code status;
+ u32 sqsize = IRDMA_CQP_SW_SQSIZE_2048;
+ struct irdma_dma_mem mem;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_cqp_init_info cqp_init_info = {};
+ struct irdma_cqp *cqp = &rf->cqp;
+ u16 maj_err, min_err;
+ int i;
+
+ cqp->cqp_requests = kcalloc(sqsize, sizeof(*cqp->cqp_requests), GFP_KERNEL);
+ if (!cqp->cqp_requests)
+ return IRDMA_ERR_NO_MEMORY;
+
+ cqp->scratch_array = kcalloc(sqsize, sizeof(*cqp->scratch_array), GFP_KERNEL);
+ if (!cqp->scratch_array) {
+ kfree(cqp->cqp_requests);
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ dev->cqp = &cqp->sc_cqp;
+ dev->cqp->dev = dev;
+ cqp->sq.size = ALIGN(sizeof(struct irdma_cqp_sq_wqe) * sqsize,
+ IRDMA_CQP_ALIGNMENT);
+ cqp->sq.va = dma_alloc_coherent(hw_to_dev(dev->hw), cqp->sq.size,
+ &cqp->sq.pa, GFP_KERNEL);
+ if (!cqp->sq.va) {
+ kfree(cqp->scratch_array);
+ kfree(cqp->cqp_requests);
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ status = irdma_obj_aligned_mem(rf, &mem, sizeof(struct irdma_cqp_ctx),
+ IRDMA_HOST_CTX_ALIGNMENT_M);
+ if (status)
+ goto exit;
+
+ dev->cqp->host_ctx_pa = mem.pa;
+ dev->cqp->host_ctx = mem.va;
+ /* populate the cqp init info */
+ cqp_init_info.dev = dev;
+ cqp_init_info.sq_size = sqsize;
+ cqp_init_info.sq = cqp->sq.va;
+ cqp_init_info.sq_pa = cqp->sq.pa;
+ cqp_init_info.host_ctx_pa = mem.pa;
+ cqp_init_info.host_ctx = mem.va;
+ cqp_init_info.hmc_profile = rf->rsrc_profile;
+ cqp_init_info.ena_vf_count = rf->max_rdma_vfs;
+ cqp_init_info.scratch_array = cqp->scratch_array;
+ cqp_init_info.disable_packed = true;
+ cqp_init_info.protocol_used = rf->protocol_used;
+ status = dev->cqp_ops->cqp_init(dev->cqp, &cqp_init_info);
+ if (status) {
+ dev_dbg(rfdev_to_dev(dev), "ERR: cqp init status %d\n",
+ status);
+ goto exit;
+ }
+
+ status = dev->cqp_ops->cqp_create(dev->cqp, &maj_err, &min_err);
+ if (status) {
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: cqp create failed - status %d maj_err %d min_err %d\n",
+ status, maj_err, min_err);
+ goto exit;
+ }
+
+ spin_lock_init(&cqp->req_lock);
+ spin_lock_init(&cqp->compl_lock);
+ INIT_LIST_HEAD(&cqp->cqp_avail_reqs);
+ INIT_LIST_HEAD(&cqp->cqp_pending_reqs);
+ sema_init(&cqp->cqp_compl_sem, 0);
+
+ /* init the waitqueue of the cqp_requests and add them to the list */
+ for (i = 0; i < sqsize; i++) {
+ init_waitqueue_head(&cqp->cqp_requests[i].waitq);
+ list_add_tail(&cqp->cqp_requests[i].list, &cqp->cqp_avail_reqs);
+ }
+ init_waitqueue_head(&cqp->remove_wq);
+ return 0;
+
+exit:
+ irdma_destroy_cqp(rf, false);
+
+ return status;
+}
+
+/**
+ * irdma_create_ccq - create control cq
+ * @rf: RDMA PCI function
+ *
+ * Return 0, if the ccq and the resources associated with it
+ * are successfully created, otherwise return error
+ */
+static enum irdma_status_code irdma_create_ccq(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ enum irdma_status_code status;
+ struct irdma_ccq_init_info info = {};
+ struct irdma_ccq *ccq = &rf->ccq;
+
+ dev->ccq = &ccq->sc_cq;
+ dev->ccq->dev = dev;
+ info.dev = dev;
+ ccq->shadow_area.size = sizeof(struct irdma_cq_shadow_area);
+ ccq->mem_cq.size = ALIGN(sizeof(struct irdma_cqe) * IW_CCQ_SIZE,
+ IRDMA_CQ0_ALIGNMENT);
+ ccq->mem_cq.va = dma_alloc_coherent(hw_to_dev(dev->hw),
+ ccq->mem_cq.size, &ccq->mem_cq.pa,
+ GFP_KERNEL);
+ if (!ccq->mem_cq.va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ status = irdma_obj_aligned_mem(rf, &ccq->shadow_area,
+ ccq->shadow_area.size,
+ IRDMA_SHADOWAREA_M);
+ if (status)
+ goto exit;
+
+ ccq->sc_cq.back_cq = (void *)ccq;
+ /* populate the ccq init info */
+ info.cq_base = ccq->mem_cq.va;
+ info.cq_pa = ccq->mem_cq.pa;
+ info.num_elem = IW_CCQ_SIZE;
+ info.shadow_area = ccq->shadow_area.va;
+ info.shadow_area_pa = ccq->shadow_area.pa;
+ info.ceqe_mask = false;
+ info.ceq_id_valid = true;
+ info.shadow_read_threshold = 16;
+ info.vsi = &rf->default_vsi;
+ status = dev->ccq_ops->ccq_init(dev->ccq, &info);
+ if (!status)
+ status = dev->ccq_ops->ccq_create(dev->ccq, 0, true, true);
+exit:
+ if (status) {
+ dma_free_coherent(hw_to_dev(dev->hw), ccq->mem_cq.size,
+ ccq->mem_cq.va, ccq->mem_cq.pa);
+ ccq->mem_cq.va = NULL;
+ }
+
+ return status;
+}
+
+/**
+ * irdma_alloc_set_mac - set up a mac address table entry
+ * @iwdev: device
+ *
+ * Allocate a mac ip entry and add it to the hw table Return 0
+ * if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_alloc_set_mac(struct irdma_device *iwdev)
+{
+ enum irdma_status_code status;
+
+ status = irdma_alloc_local_mac_entry(iwdev->rf,
+ &iwdev->mac_ip_table_idx);
+ if (!status) {
+ status = irdma_add_local_mac_entry(iwdev->rf,
+ (u8 *)iwdev->netdev->dev_addr,
+ (u8)iwdev->mac_ip_table_idx);
+ if (status)
+ irdma_del_local_mac_entry(iwdev->rf,
+ (u8)iwdev->mac_ip_table_idx);
+ }
+ return status;
+}
+
+/**
+ * irdma_configure_ceq_vector - set up the msix interrupt vector for ceq
+ * @rf: RDMA PCI function
+ * @iwceq: ceq associated with the vector
+ * @ceq_id: the id number of the iwceq
+ * @msix_vec: interrupt vector information
+ *
+ * Allocate interrupt resources and enable irq handling
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code
+irdma_cfg_ceq_vector(struct irdma_pci_f *rf, struct irdma_ceq *iwceq,
+ u32 ceq_id, struct irdma_msix_vector *msix_vec)
+{
+ int status;
+
+ if (rf->msix_shared && !ceq_id) {
+ tasklet_init(&rf->dpc_tasklet, irdma_dpc, (unsigned long)rf);
+ status = request_irq(msix_vec->irq, irdma_irq_handler, 0,
+ "AEQCEQ", rf);
+ } else {
+ tasklet_init(&iwceq->dpc_tasklet, irdma_ceq_dpc,
+ (unsigned long)iwceq);
+
+ status = request_irq(msix_vec->irq, irdma_ceq_handler, 0, "CEQ",
+ iwceq);
+ }
+
+ cpumask_clear(&msix_vec->mask);
+ cpumask_set_cpu(msix_vec->cpu_affinity, &msix_vec->mask);
+ irq_set_affinity_hint(msix_vec->irq, &msix_vec->mask);
+ if (status) {
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: ceq irq config fail\n");
+ return IRDMA_ERR_CFG;
+ }
+
+ msix_vec->ceq_id = ceq_id;
+ rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, ceq_id, msix_vec->idx);
+
+ return 0;
+}
+
+/**
+ * irdma_configure_aeq_vector - set up the msix vector for aeq
+ * @rf: RDMA PCI function
+ *
+ * Allocate interrupt resources and enable irq handling
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_cfg_aeq_vector(struct irdma_pci_f *rf)
+{
+ struct irdma_msix_vector *msix_vec = rf->iw_msixtbl;
+ u32 ret = 0;
+
+ if (!rf->msix_shared) {
+ tasklet_init(&rf->dpc_tasklet, irdma_dpc, (unsigned long)rf);
+ ret = request_irq(msix_vec->irq, irdma_irq_handler, 0, "irdma",
+ rf);
+ }
+ if (ret) {
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: aeq irq config fail\n");
+ return IRDMA_ERR_CFG;
+ }
+
+ rf->sc_dev.irq_ops->irdma_cfg_aeq(&rf->sc_dev, msix_vec->idx);
+
+ return 0;
+}
+
+/**
+ * irdma_create_ceq - create completion event queue
+ * @rf: RDMA PCI function
+ * @iwceq: pointer to the ceq resources to be created
+ * @ceq_id: the id number of the iwceq
+ * @vsi: SC vsi struct
+ *
+ * Return 0, if the ceq and the resources associated with it
+ * are successfully created, otherwise return error
+ */
+static enum irdma_status_code irdma_create_ceq(struct irdma_pci_f *rf,
+ struct irdma_ceq *iwceq,
+ u32 ceq_id,
+ struct irdma_sc_vsi *vsi)
+{
+ enum irdma_status_code status;
+ struct irdma_ceq_init_info info = {};
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ u64 scratch;
+
+ info.ceq_id = ceq_id;
+ iwceq->rf = rf;
+ iwceq->mem.size = ALIGN(sizeof(struct irdma_ceqe) * rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt,
+ IRDMA_CEQ_ALIGNMENT);
+ iwceq->mem.va = dma_alloc_coherent(hw_to_dev(dev->hw),
+ iwceq->mem.size, &iwceq->mem.pa,
+ GFP_KERNEL);
+ if (!iwceq->mem.va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ info.ceq_id = ceq_id;
+ info.ceqe_base = iwceq->mem.va;
+ info.ceqe_pa = iwceq->mem.pa;
+ info.elem_cnt = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt;
+ iwceq->sc_ceq.ceq_id = ceq_id;
+ info.dev = dev;
+ info.vsi = vsi;
+ scratch = (uintptr_t)&rf->cqp.sc_cqp;
+ status = dev->ceq_ops->ceq_init(&iwceq->sc_ceq, &info);
+ if (!status) {
+ if (dev->ceq_valid)
+ status = irdma_cqp_ceq_cmd(&rf->sc_dev, &iwceq->sc_ceq,
+ IRDMA_OP_CEQ_CREATE);
+ else
+ status = dev->ceq_ops->cceq_create(&iwceq->sc_ceq,
+ scratch);
+ }
+
+ if (status) {
+ dma_free_coherent(hw_to_dev(dev->hw), iwceq->mem.size,
+ iwceq->mem.va, iwceq->mem.pa);
+ iwceq->mem.va = NULL;
+ }
+
+ return status;
+}
+
+/**
+ * irdma_setup_ceq_0 - create CEQ 0 and it's interrupt resource
+ * @rf: RDMA PCI function
+ *
+ * Allocate a list for all device completion event queues
+ * Create the ceq 0 and configure it's msix interrupt vector
+ * Return 0, if successfully set up, otherwise return error
+ */
+static enum irdma_status_code irdma_setup_ceq_0(struct irdma_pci_f *rf)
+{
+ u32 i;
+ struct irdma_ceq *iwceq;
+ struct irdma_msix_vector *msix_vec;
+ enum irdma_status_code status = 0;
+ u32 num_ceqs;
+
+ num_ceqs = min(rf->msix_count, rf->sc_dev.hmc_fpm_misc.max_ceqs);
+ rf->ceqlist = kcalloc(num_ceqs, sizeof(*rf->ceqlist), GFP_KERNEL);
+ if (!rf->ceqlist) {
+ status = IRDMA_ERR_NO_MEMORY;
+ goto exit;
+ }
+
+ i = rf->msix_shared ? 0 : 1;
+ iwceq = &rf->ceqlist[0];
+ status = irdma_create_ceq(rf, iwceq, 0, &rf->default_vsi);
+ if (status) {
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: create ceq status = %d\n", status);
+ goto exit;
+ }
+
+ msix_vec = &rf->iw_msixtbl[i];
+ iwceq->irq = msix_vec->irq;
+ iwceq->msix_idx = msix_vec->idx;
+ status = irdma_cfg_ceq_vector(rf, iwceq, 0, msix_vec);
+ if (status) {
+ irdma_destroy_ceq(rf, iwceq);
+ goto exit;
+ }
+
+ irdma_ena_intr(&rf->sc_dev, msix_vec->idx);
+ rf->ceqs_count++;
+
+exit:
+ if (status && !rf->ceqs_count) {
+ kfree(rf->ceqlist);
+ rf->ceqlist = NULL;
+ return status;
+ }
+ rf->sc_dev.ceq_valid = true;
+
+ return 0;
+}
+
+/**
+ * irdma_setup_ceqs - manage the device ceq's and their interrupt resources
+ * @rf: RDMA PCI function
+ * @vsi: VSI structure for this CEQ
+ *
+ * Allocate a list for all device completion event queues
+ * Create the ceq's and configure their msix interrupt vectors
+ * Return 0, if at least one ceq is successfully set up, otherwise return error
+ */
+static enum irdma_status_code irdma_setup_ceqs(struct irdma_pci_f *rf,
+ struct irdma_sc_vsi *vsi)
+{
+ u32 i;
+ u32 ceq_id;
+ struct irdma_ceq *iwceq;
+ struct irdma_msix_vector *msix_vec;
+ enum irdma_status_code status = 0;
+ u32 num_ceqs;
+
+ num_ceqs = min(rf->msix_count, rf->sc_dev.hmc_fpm_misc.max_ceqs);
+ i = (rf->msix_shared) ? 1 : 2;
+ for (ceq_id = 1; i < num_ceqs; i++, ceq_id++) {
+ iwceq = &rf->ceqlist[ceq_id];
+ status = irdma_create_ceq(rf, iwceq, ceq_id, vsi);
+ if (status) {
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: create ceq status = %d\n", status);
+ break;
+ }
+ msix_vec = &rf->iw_msixtbl[i];
+ iwceq->irq = msix_vec->irq;
+ iwceq->msix_idx = msix_vec->idx;
+ status = irdma_cfg_ceq_vector(rf, iwceq, ceq_id, msix_vec);
+ if (status) {
+ irdma_destroy_ceq(rf, iwceq);
+ break;
+ }
+ irdma_ena_intr(&rf->sc_dev, msix_vec->idx);
+ rf->ceqs_count++;
+ }
+
+ return status;
+}
+
+/**
+ * irdma_create_aeq - create async event queue
+ * @rf: RDMA PCI function
+ *
+ * Return 0, if the aeq and the resources associated with it
+ * are successfully created, otherwise return error
+ */
+static enum irdma_status_code irdma_create_aeq(struct irdma_pci_f *rf)
+{
+ enum irdma_status_code status;
+ struct irdma_aeq_init_info info = {};
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_aeq *aeq = &rf->aeq;
+ struct irdma_hmc_info *hmc_info = rf->sc_dev.hmc_info;
+ u64 scratch = 0;
+ u32 aeq_size;
+
+ aeq_size = 2 * hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt +
+ hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt;
+ aeq->mem.size = ALIGN(sizeof(struct irdma_sc_aeqe) * aeq_size,
+ IRDMA_AEQ_ALIGNMENT);
+ aeq->mem.va = dma_alloc_coherent(hw_to_dev(dev->hw), aeq->mem.size,
+ &aeq->mem.pa, GFP_KERNEL);
+ if (!aeq->mem.va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ info.aeqe_base = aeq->mem.va;
+ info.aeq_elem_pa = aeq->mem.pa;
+ info.elem_cnt = aeq_size;
+ info.dev = dev;
+ status = dev->aeq_ops->aeq_init(&aeq->sc_aeq, &info);
+ if (status)
+ goto exit;
+
+ status = dev->aeq_ops->aeq_create(&aeq->sc_aeq, scratch, 1);
+ if (!status)
+ status = dev->aeq_ops->aeq_create_done(&aeq->sc_aeq);
+exit:
+ if (status) {
+ dma_free_coherent(hw_to_dev(dev->hw), aeq->mem.size,
+ aeq->mem.va, aeq->mem.pa);
+ aeq->mem.va = NULL;
+ }
+
+ return status;
+}
+
+/**
+ * irdma_setup_aeq - set up the device aeq
+ * @rf: RDMA PCI function
+ *
+ * Create the aeq and configure its msix interrupt vector
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_setup_aeq(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ enum irdma_status_code status;
+
+ status = irdma_create_aeq(rf);
+ if (status)
+ return status;
+
+ status = irdma_cfg_aeq_vector(rf);
+ if (status) {
+ irdma_destroy_aeq(rf);
+ return status;
+ }
+
+ if (!rf->msix_shared)
+ irdma_ena_intr(dev, rf->iw_msixtbl[0].idx);
+
+ return 0;
+}
+
+/**
+ * irdma_initialize_ilq - create iwarp local queue for cm
+ * @iwdev: iwarp device
+ *
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_initialize_ilq(struct irdma_device *iwdev)
+{
+ struct irdma_puda_rsrc_info info = {};
+ enum irdma_status_code status;
+
+ info.type = IRDMA_PUDA_RSRC_TYPE_ILQ;
+ info.cq_id = 1;
+ info.qp_id = 1;
+ info.count = 1;
+ info.pd_id = 1;
+ info.sq_size = min(iwdev->rf->max_qp / 2, (u32)32768);
+ info.rq_size = info.sq_size;
+ info.buf_size = 1024;
+ info.tx_buf_cnt = 2 * info.sq_size;
+ info.receive = irdma_receive_ilq;
+ info.xmit_complete = irdma_free_sqbuf;
+ status = irdma_puda_create_rsrc(&iwdev->vsi, &info);
+ if (status)
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "ERR: ilq create fail\n");
+
+ return status;
+}
+
+/**
+ * irdma_initialize_ieq - create iwarp exception queue
+ * @iwdev: iwarp device
+ *
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_initialize_ieq(struct irdma_device *iwdev)
+{
+ struct irdma_puda_rsrc_info info = {};
+ enum irdma_status_code status;
+
+ info.type = IRDMA_PUDA_RSRC_TYPE_IEQ;
+ info.cq_id = 2;
+ info.qp_id = iwdev->vsi.exception_lan_q;
+ info.count = 1;
+ info.pd_id = 2;
+ info.sq_size = min(iwdev->rf->max_qp / 2, (u32)32768);
+ info.rq_size = info.sq_size;
+ info.buf_size = iwdev->vsi.mtu + IRDMA_IPV4_PAD;
+ info.tx_buf_cnt = 4096;
+ status = irdma_puda_create_rsrc(&iwdev->vsi, &info);
+ if (status)
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "ERR: ieq create fail\n");
+
+ return status;
+}
+
+/**
+ * irdma_reinitialize_ieq - destroy and re-create ieq
+ * @vsi: VSI structure
+ */
+void irdma_reinitialize_ieq(struct irdma_sc_vsi *vsi)
+{
+ struct irdma_device *iwdev = vsi->back_vsi;
+
+ irdma_puda_dele_rsrc(vsi, IRDMA_PUDA_RSRC_TYPE_IEQ, false);
+ if (irdma_initialize_ieq(iwdev)) {
+ iwdev->reset = true;
+ irdma_request_reset(iwdev->rf);
+ }
+}
+
+/**
+ * irdma_hmc_setup - create hmc objects for the device
+ * @rf: RDMA PCI function
+ *
+ * Set up the device private memory space for the number and size of
+ * the hmc objects and create the objects
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_hmc_setup(struct irdma_pci_f *rf)
+{
+ enum irdma_status_code status;
+ u32 qpcnt;
+
+ if (rf->rdma_ver == IRDMA_GEN_1)
+ qpcnt = rsrc_limits_table[rf->limits_sel].qplimit * 2;
+ else
+ qpcnt = rsrc_limits_table[rf->limits_sel].qplimit;
+
+ rf->sd_type = IRDMA_SD_TYPE_DIRECT;
+ status = irdma_cfg_fpm_val(&rf->sc_dev, qpcnt);
+ if (status)
+ return status;
+
+ status = irdma_create_hmc_objs(rf, true, rf->rdma_ver);
+
+ return status;
+}
+
+/**
+ * irdma_del_init_mem - deallocate memory resources
+ * @rf: RDMA PCI function
+ */
+static void irdma_del_init_mem(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+
+ kfree(dev->hmc_info->sd_table.sd_entry);
+ dev->hmc_info->sd_table.sd_entry = NULL;
+ kfree(rf->mem_rsrc);
+ rf->mem_rsrc = NULL;
+ dma_free_coherent(hw_to_dev(&rf->hw), rf->obj_mem.size,
+ rf->obj_mem.va, rf->obj_mem.pa);
+ rf->obj_mem.va = NULL;
+ if (rf->rdma_ver != IRDMA_GEN_1) {
+ kfree(rf->allocated_ws_nodes);
+ rf->allocated_ws_nodes = NULL;
+ }
+ kfree(rf->ceqlist);
+ rf->ceqlist = NULL;
+ kfree(rf->iw_msixtbl);
+ rf->iw_msixtbl = NULL;
+ kfree(rf->hmc_info_mem);
+ rf->hmc_info_mem = NULL;
+}
+
+/**
+ * irdma_initialize_dev - initialize device
+ * @rf: RDMA PCI function
+ * @ldev: lan device information
+ *
+ * Allocate memory for the hmc objects and initialize iwdev
+ * Return 0 if successful, otherwise clean up the resources
+ * and return error
+ */
+static enum irdma_status_code irdma_initialize_dev(struct irdma_pci_f *rf,
+ struct irdma_priv_ldev *ldev)
+{
+ enum irdma_status_code status;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_device_init_info info = {};
+ struct irdma_dma_mem mem;
+ u32 size;
+
+ size = sizeof(struct irdma_hmc_pble_rsrc) +
+ sizeof(struct irdma_hmc_info) +
+ (sizeof(struct irdma_hmc_obj_info) * IRDMA_HMC_IW_MAX);
+
+ rf->hmc_info_mem = kzalloc(size, GFP_KERNEL);
+ if (!rf->hmc_info_mem)
+ return IRDMA_ERR_NO_MEMORY;
+
+ rf->pble_rsrc = (struct irdma_hmc_pble_rsrc *)rf->hmc_info_mem;
+ dev->hmc_info = &rf->hw.hmc;
+ dev->hmc_info->hmc_obj = (struct irdma_hmc_obj_info *)
+ (rf->pble_rsrc + 1);
+
+ status = irdma_obj_aligned_mem(rf, &mem, IRDMA_QUERY_FPM_BUF_SIZE,
+ IRDMA_FPM_QUERY_BUF_ALIGNMENT_M);
+ if (status)
+ goto error;
+
+ info.fpm_query_buf_pa = mem.pa;
+ info.fpm_query_buf = mem.va;
+ info.init_hw = rf->init_hw;
+
+ status = irdma_obj_aligned_mem(rf, &mem, IRDMA_COMMIT_FPM_BUF_SIZE,
+ IRDMA_FPM_COMMIT_BUF_ALIGNMENT_M);
+ if (status)
+ goto error;
+
+ info.fpm_commit_buf_pa = mem.pa;
+ info.fpm_commit_buf = mem.va;
+
+ info.bar0 = rf->hw.hw_addr;
+ info.hmc_fn_id = (u8)ldev->fn_num;
+ info.is_pf = !ldev->ftype;
+ info.hw = &rf->hw;
+ info.vchnl_send = NULL;
+ status = irdma_sc_ctrl_init(rf->rdma_ver, &rf->sc_dev, &info);
+ if (status)
+ goto error;
+
+ return status;
+error:
+ kfree(rf->hmc_info_mem);
+ rf->hmc_info_mem = NULL;
+
+ return status;
+}
+
+/**
+ * irdma_deinit_rt_device - clean up the device resources
+ * @iwdev: iwarp device
+ *
+ * Destroy the ib device interface, remove the mac ip entry and
+ * ipv4/ipv6 addresses, destroy the device queues and free the
+ * pble and the hmc objects
+ */
+void irdma_deinit_rt_device(struct irdma_device *iwdev)
+{
+ dev_info(rfdev_to_dev(&iwdev->rf->sc_dev), "state = %d\n",
+ iwdev->init_state);
+
+ switch (iwdev->init_state) {
+ case RDMA_DEV_REGISTERED:
+ iwdev->iw_status = 0;
+ irdma_port_ibevent(iwdev);
+ irdma_destroy_rdma_device(iwdev->iwibdev);
+ /* fallthrough */
+ case IP_ADDR_REGISTERED:
+ if (iwdev->rf->sc_dev.hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_1)
+ irdma_del_local_mac_entry(iwdev->rf,
+ (u8)iwdev->mac_ip_table_idx);
+ /* fallthrough */
+ case PBLE_CHUNK_MEM:
+ /* fallthrough */
+ case CEQS_CREATED:
+ /* fallthrough */
+ case IEQ_CREATED:
+ irdma_puda_dele_rsrc(&iwdev->vsi, IRDMA_PUDA_RSRC_TYPE_IEQ,
+ iwdev->reset);
+ /* fallthrough */
+ case ILQ_CREATED:
+ if (iwdev->create_ilq)
+ irdma_puda_dele_rsrc(&iwdev->vsi,
+ IRDMA_PUDA_RSRC_TYPE_ILQ,
+ iwdev->reset);
+ break;
+ default:
+ dev_warn(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "bad init_state = %d\n", iwdev->init_state);
+ break;
+ }
+
+ irdma_cleanup_cm_core(&iwdev->cm_core);
+ if (iwdev->vsi.pestat) {
+ irdma_vsi_stats_free(&iwdev->vsi);
+ kfree(iwdev->vsi.pestat);
+ }
+}
+
+/**
+ * irdma_setup_init_state - set up the initial device struct
+ * @rf: RDMA PCI function
+ *
+ * Initialize the iwarp device and its hdl information
+ * using the ldev and client information
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_setup_init_state(struct irdma_pci_f *rf)
+{
+ struct irdma_priv_ldev *ldev = &rf->ldev;
+ enum irdma_status_code status;
+
+ status = irdma_save_msix_info(rf);
+ if (status)
+ return status;
+
+ rf->hw.pdev = rf->pdev;
+ rf->obj_mem.size = ALIGN(8192, IRDMA_HW_PAGE_SIZE);
+ rf->obj_mem.va = dma_alloc_coherent(hw_to_dev(&rf->hw),
+ rf->obj_mem.size, &rf->obj_mem.pa,
+ GFP_KERNEL);
+ if (!rf->obj_mem.va) {
+ kfree(rf->iw_msixtbl);
+ rf->iw_msixtbl = NULL;
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ rf->obj_next = rf->obj_mem;
+ rf->ooo = false;
+ init_waitqueue_head(&rf->vchnl_waitq);
+
+ status = irdma_initialize_dev(rf, ldev);
+ if (status) {
+ kfree(rf->iw_msixtbl);
+ dma_free_coherent(hw_to_dev(&rf->hw), rf->obj_mem.size,
+ rf->obj_mem.va, rf->obj_mem.pa);
+ rf->obj_mem.va = NULL;
+ rf->iw_msixtbl = NULL;
+ }
+
+ return status;
+}
+
+/**
+ * irdma_get_used_rsrc - determine resources used internally
+ * @iwdev: iwarp device
+ *
+ * Called at the end of open to get all internal allocations
+ */
+static void irdma_get_used_rsrc(struct irdma_device *iwdev)
+{
+ iwdev->rf->used_pds = find_next_zero_bit(iwdev->rf->allocated_pds,
+ iwdev->rf->max_pd, 0);
+ iwdev->rf->used_qps = find_next_zero_bit(iwdev->rf->allocated_qps,
+ iwdev->rf->max_qp, 0);
+ iwdev->rf->used_cqs = find_next_zero_bit(iwdev->rf->allocated_cqs,
+ iwdev->rf->max_cq, 0);
+ iwdev->rf->used_mrs = find_next_zero_bit(iwdev->rf->allocated_mrs,
+ iwdev->rf->max_mr, 0);
+}
+
+/**
+ * irdma_deinit_hw - De-initializes RDMA HW
+ * @rf: RDMA device information
+ *
+ */
+void irdma_deinit_ctrl_hw(struct irdma_pci_f *rf)
+{
+ enum init_completion_state state = rf->init_state;
+
+ rf->init_state = INVALID_STATE;
+ if (rf->rsrc_created) {
+ irdma_destroy_pble_prm(rf->pble_rsrc);
+ irdma_del_ceqs(rf);
+ rf->rsrc_created = false;
+ }
+ switch (state) {
+ case CEQ0_CREATED:
+ irdma_del_ceq_0(rf);
+ /* fallthrough */
+ case AEQ_CREATED:
+ irdma_destroy_aeq(rf);
+ /* fallthrough */
+ case CCQ_CREATED:
+ irdma_destroy_ccq(rf);
+ /* fallthrough */
+ case HMC_OBJS_CREATED:
+ irdma_del_hmc_objects(&rf->sc_dev, rf->sc_dev.hmc_info, true,
+ rf->reset, rf->rdma_ver);
+ /* fallthrough */
+ case CQP_CREATED:
+ if (rf->cqp.cqp_compl_thread) {
+ rf->stop_cqp_thread = true;
+ up(&rf->cqp.cqp_compl_sem);
+ kthread_stop(rf->cqp.cqp_compl_thread);
+ }
+ irdma_destroy_cqp(rf, true);
+ /* fallthrough */
+ case INITIAL_STATE:
+ irdma_del_init_mem(rf);
+ break;
+ case INVALID_STATE:
+ /* fallthrough */
+ default:
+ pr_warn("bad init_state = %d\n", rf->init_state);
+ break;
+ }
+}
+
+enum irdma_status_code irdma_rt_init_hw(struct irdma_pci_f *rf,
+ struct irdma_device *iwdev,
+ struct irdma_l2params *l2params)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ enum irdma_status_code status;
+ struct irdma_vsi_init_info vsi_info = {};
+ struct irdma_vsi_stats_info stats_info = {};
+
+ irdma_sc_rt_init(dev);
+ vsi_info.vm_vf_type = dev->is_pf ? IRDMA_PF_TYPE : IRDMA_VF_TYPE;
+ vsi_info.dev = dev;
+ vsi_info.back_vsi = (void *)iwdev;
+ vsi_info.params = l2params;
+ vsi_info.pf_data_vsi_num = iwdev->vsi_num;
+ vsi_info.exception_lan_q = 2;
+ irdma_sc_vsi_init(&iwdev->vsi, &vsi_info);
+
+ status = irdma_setup_cm_core(iwdev, rf->rdma_ver);
+ if (status)
+ return status;
+
+ stats_info.pestat = kzalloc(sizeof(*stats_info.pestat), GFP_KERNEL);
+ if (!stats_info.pestat)
+ return IRDMA_ERR_NO_MEMORY;
+
+ stats_info.fcn_id = dev->hmc_fn_id;
+ status = irdma_vsi_stats_init(&iwdev->vsi, &stats_info);
+ if (status) {
+ kfree(stats_info.pestat);
+ return status;
+ }
+
+ do {
+ if (iwdev->create_ilq) {
+ status = irdma_initialize_ilq(iwdev);
+ if (status)
+ break;
+ iwdev->init_state = ILQ_CREATED;
+ }
+ status = irdma_initialize_ieq(iwdev);
+ if (status)
+ break;
+ iwdev->init_state = IEQ_CREATED;
+ if (!rf->rsrc_created) {
+ status = irdma_setup_ceqs(rf, &iwdev->vsi);
+ if (status)
+ break;
+ iwdev->init_state = CEQS_CREATED;
+
+ status = irdma_hmc_init_pble(&rf->sc_dev,
+ rf->pble_rsrc);
+ if (status) {
+ irdma_del_ceqs(rf);
+ break;
+ }
+ spin_lock_init(&rf->pble_rsrc->pble_lock);
+ iwdev->init_state = PBLE_CHUNK_MEM;
+ rf->rsrc_created = true;
+ }
+
+ iwdev->device_cap_flags = IB_DEVICE_LOCAL_DMA_LKEY |
+ IB_DEVICE_MEM_WINDOW |
+ IB_DEVICE_MEM_MGT_EXTENSIONS;
+
+ if (iwdev->rf->sc_dev.hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_1)
+ irdma_alloc_set_mac(iwdev);
+ irdma_add_ip(iwdev);
+ iwdev->init_state = IP_ADDR_REGISTERED;
+ status = irdma_register_rdma_device(iwdev);
+ if (status)
+ break;
+ iwdev->init_state = RDMA_DEV_REGISTERED;
+ irdma_port_ibevent(iwdev);
+ iwdev->iw_status = 1;
+ irdma_get_used_rsrc(iwdev);
+ init_waitqueue_head(&iwdev->suspend_wq);
+
+ return 0;
+ } while (0);
+
+ dev_err(rfdev_to_dev(dev), "VSI open FAIL status = %d last cmpl = %d\n",
+ status, iwdev->init_state);
+ irdma_deinit_rt_device(iwdev);
+
+ return status;
+}
+
+/**
+ * irdma_ctrl_init_hw - Initializes RDMA HW
+ * @rf: RDMA PCI function
+ *
+ */
+enum irdma_status_code irdma_ctrl_init_hw(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ enum irdma_status_code status;
+
+ do {
+ status = irdma_setup_init_state(rf);
+ if (status)
+ break;
+ rf->init_state = INITIAL_STATE;
+
+ status = irdma_create_cqp(rf);
+ if (status)
+ break;
+ rf->init_state = CQP_CREATED;
+
+ status = irdma_hmc_setup(rf);
+ if (status)
+ break;
+ rf->init_state = HMC_OBJS_CREATED;
+
+ status = irdma_initialize_hw_rsrc(rf);
+ if (status)
+ break;
+
+ status = irdma_create_ccq(rf);
+ if (status)
+ break;
+ rf->init_state = CCQ_CREATED;
+
+ status = irdma_setup_aeq(rf);
+ if (status)
+ break;
+ rf->init_state = AEQ_CREATED;
+ rf->sc_dev.feature_info[IRDMA_FEATURE_FW_INFO] = IRDMA_FW_VER_DEFAULT;
+
+ if (rf->rdma_ver != IRDMA_GEN_1)
+ status = irdma_get_rdma_features(&rf->sc_dev);
+ if (!status) {
+ u32 fw_ver = dev->feature_info[IRDMA_FEATURE_FW_INFO];
+ u8 hw_rev = dev->hw_attrs.uk_attrs.hw_rev;
+
+ if ((hw_rev == IRDMA_GEN_1 && fw_ver >= IRDMA_FW_VER_0x30010) ||
+ (hw_rev != IRDMA_GEN_1 && fw_ver >= IRDMA_FW_VER_0x1000D))
+
+ dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_RTS_AE |
+ IRDMA_FEATURE_CQ_RESIZE;
+ }
+ rf->cqp.cqp_compl_thread =
+ kthread_run(cqp_compl_thread, rf, "cqp_compl_thread");
+
+ status = irdma_setup_ceq_0(rf);
+ if (status)
+ break;
+ rf->init_state = CEQ0_CREATED;
+
+ rf->free_qp_wq =
+ alloc_ordered_workqueue("free_qp_wq", WQ_MEM_RECLAIM);
+ if (!rf->free_qp_wq) {
+ status = IRDMA_ERR_NO_MEMORY;
+ break;
+ }
+
+ rf->free_cqbuf_wq =
+ alloc_ordered_workqueue("free_cqbuf_wq", WQ_MEM_RECLAIM);
+ if (!rf->free_cqbuf_wq) {
+ status = IRDMA_ERR_NO_MEMORY;
+ break;
+ }
+ dev->ccq_ops->ccq_arm(dev->ccq);
+ dev_info(rfdev_to_dev(dev), "IRDMA hardware initialization successful\n");
+ return 0;
+ } while (0);
+
+ pr_err("IRDMA hardware initialization FAILED init_state=%d status=%d\n",
+ rf->init_state, status);
+ irdma_deinit_ctrl_hw(rf);
+ return status;
+}
+
+/**
+ * irdma_initialize_hw_resources - initialize hw resource during open
+ * @rf: RDMA PCI function
+ */
+u32 irdma_initialize_hw_rsrc(struct irdma_pci_f *rf)
+{
+ unsigned long num_pds;
+ u32 rsrc_size;
+ u32 max_mr;
+ u32 max_qp;
+ u32 max_cq;
+ u32 arp_table_size;
+ u32 mrdrvbits;
+ void *rsrc_ptr;
+ u32 num_ahs;
+ u32 num_mcg;
+
+ if (rf->rdma_ver != IRDMA_GEN_1) {
+ rf->allocated_ws_nodes =
+ kcalloc(BITS_TO_LONGS(IRDMA_MAX_WS_NODES),
+ sizeof(unsigned long), GFP_KERNEL);
+ if (!rf->allocated_ws_nodes)
+ return -ENOMEM;
+
+ set_bit(0, rf->allocated_ws_nodes);
+ rf->max_ws_node_id = IRDMA_MAX_WS_NODES;
+ }
+ max_qp = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt;
+ max_cq = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt;
+ max_mr = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt;
+ arp_table_size = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].cnt;
+ rf->max_cqe = rf->sc_dev.hw_attrs.uk_attrs.max_hw_cq_size;
+ num_pds = rf->sc_dev.hw_attrs.max_hw_pds;
+ rsrc_size = sizeof(struct irdma_arp_entry) * arp_table_size;
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(max_qp);
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(max_mr);
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(max_cq);
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(num_pds);
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(arp_table_size);
+ num_ahs = max_qp * 4;
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(num_ahs);
+ num_mcg = max_qp;
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(num_mcg);
+ rsrc_size += sizeof(struct irdma_qp **) * max_qp;
+
+ rf->mem_rsrc = kzalloc(rsrc_size, GFP_KERNEL);
+ if (!rf->mem_rsrc) {
+ kfree(rf->allocated_ws_nodes);
+ rf->allocated_ws_nodes = NULL;
+ return -ENOMEM;
+ }
+
+ rf->max_qp = max_qp;
+ rf->max_mr = max_mr;
+ rf->max_cq = max_cq;
+ rf->max_pd = num_pds;
+ rf->arp_table_size = arp_table_size;
+ rf->arp_table = (struct irdma_arp_entry *)rf->mem_rsrc;
+ rsrc_ptr = rf->mem_rsrc +
+ (sizeof(struct irdma_arp_entry) * arp_table_size);
+ rf->max_ah = num_ahs;
+ rf->max_mcg = num_mcg;
+ rf->allocated_qps = rsrc_ptr;
+ rf->allocated_cqs = &rf->allocated_qps[BITS_TO_LONGS(max_qp)];
+ rf->allocated_mrs = &rf->allocated_cqs[BITS_TO_LONGS(max_cq)];
+ rf->allocated_pds = &rf->allocated_mrs[BITS_TO_LONGS(max_mr)];
+ rf->allocated_ahs = &rf->allocated_pds[BITS_TO_LONGS(num_pds)];
+ rf->allocated_mcgs = &rf->allocated_ahs[BITS_TO_LONGS(num_ahs)];
+ rf->allocated_arps = &rf->allocated_mcgs[BITS_TO_LONGS(num_mcg)];
+ rf->qp_table = (struct irdma_qp **)
+ (&rf->allocated_arps[BITS_TO_LONGS(arp_table_size)]);
+
+ set_bit(0, rf->allocated_mrs);
+ set_bit(0, rf->allocated_qps);
+ set_bit(0, rf->allocated_cqs);
+ set_bit(0, rf->allocated_pds);
+ set_bit(0, rf->allocated_arps);
+ set_bit(0, rf->allocated_ahs);
+ set_bit(0, rf->allocated_mcgs);
+ set_bit(2, rf->allocated_qps); /* qp 2 IEQ */
+ set_bit(1, rf->allocated_qps); /* qp 1 ILQ */
+ set_bit(1, rf->allocated_cqs);
+ set_bit(1, rf->allocated_pds);
+ set_bit(2, rf->allocated_cqs);
+ set_bit(2, rf->allocated_pds);
+
+ spin_lock_init(&rf->rsrc_lock);
+ spin_lock_init(&rf->arp_lock);
+ spin_lock_init(&rf->qptable_lock);
+ spin_lock_init(&rf->qh_list_lock);
+
+ INIT_LIST_HEAD(&rf->mc_qht_list.list);
+ /* stag index mask has a minimum of 14 bits */
+ mrdrvbits = 24 - max(get_count_order(rf->max_mr), 14);
+ rf->mr_stagmask = ~(((1 << mrdrvbits) - 1) << (32 - mrdrvbits));
+
+ return 0;
+}
+
+/**
+ * irdma_cqp_ce_handler - handle cqp completions
+ * @rf: RDMA PCI function
+ * @cq: cq for cqp completions
+ */
+void irdma_cqp_ce_handler(struct irdma_pci_f *rf, struct irdma_sc_cq *cq)
+{
+ struct irdma_cqp_request *cqp_request;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ u32 cqe_count = 0;
+ struct irdma_ccq_cqe_info info;
+ unsigned long flags;
+ int ret;
+
+ do {
+ memset(&info, 0, sizeof(info));
+ spin_lock_irqsave(&rf->cqp.compl_lock, flags);
+ ret = dev->ccq_ops->ccq_get_cqe_info(cq, &info);
+ spin_unlock_irqrestore(&rf->cqp.compl_lock, flags);
+ if (ret)
+ break;
+
+ cqp_request = (struct irdma_cqp_request *)
+ (unsigned long)info.scratch;
+ if (info.error)
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: opcode = 0x%x maj_err_code = 0x%x min_err_code = 0x%x\n",
+ info.op_code, info.maj_err_code,
+ info.min_err_code);
+ if (cqp_request) {
+ cqp_request->compl_info.maj_err_code = info.maj_err_code;
+ cqp_request->compl_info.min_err_code = info.min_err_code;
+ cqp_request->compl_info.op_ret_val = info.op_ret_val;
+ cqp_request->compl_info.error = info.error;
+
+ if (cqp_request->waiting) {
+ cqp_request->request_done = true;
+ wake_up(&cqp_request->waitq);
+ irdma_put_cqp_request(&rf->cqp, cqp_request);
+ } else {
+ if (cqp_request->callback_fcn)
+ cqp_request->callback_fcn(cqp_request);
+ irdma_put_cqp_request(&rf->cqp, cqp_request);
+ }
+ }
+
+ cqe_count++;
+ } while (1);
+
+ if (cqe_count) {
+ irdma_process_bh(dev);
+ dev->ccq_ops->ccq_arm(cq);
+ }
+}
+
+/**
+ * cqp_thread - Handle cqp completions
+ * @context: Pointer to RDMA PCI Function
+ */
+int cqp_compl_thread(void *context)
+{
+ struct irdma_pci_f *rf = context;
+ struct irdma_sc_cq *cq = &rf->ccq.sc_cq;
+
+ do {
+ if (down_interruptible(&rf->cqp.cqp_compl_sem) ||
+ rf->stop_cqp_thread)
+ return 0;
+
+ irdma_cqp_ce_handler(rf, cq);
+ } while (!kthread_should_stop());
+
+ return 0;
+}
+
+/**
+ * irdma_next_iw_state - modify qp state
+ * @iwqp: iwarp qp to modify
+ * @state: next state for qp
+ * @del_hash: del hash
+ * @term: term message
+ * @termlen: length of term message
+ */
+void irdma_next_iw_state(struct irdma_qp *iwqp, u8 state, u8 del_hash, u8 term,
+ u8 termlen)
+{
+ struct irdma_modify_qp_info info = {};
+
+ info.next_iwarp_state = state;
+ info.remove_hash_idx = del_hash;
+ info.cq_num_valid = true;
+ info.arp_cache_idx_valid = true;
+ info.dont_send_term = true;
+ info.dont_send_fin = true;
+ info.termlen = termlen;
+
+ if (term & IRDMAQP_TERM_SEND_TERM_ONLY)
+ info.dont_send_term = false;
+ if (term & IRDMAQP_TERM_SEND_FIN_ONLY)
+ info.dont_send_fin = false;
+ if (iwqp->sc_qp.term_flags && state == IRDMA_QP_STATE_ERROR)
+ info.reset_tcp_conn = true;
+ iwqp->hw_iwarp_state = state;
+ irdma_hw_modify_qp(iwqp->iwdev, iwqp, &info, 0);
+ iwqp->iwarp_state = info.next_iwarp_state;
+}
+
+/**
+ * irdma_del_mac_entry - remove a mac entry from the hw table
+ * @rf: RDMA PCI function
+ * @idx: the index of the mac ip address to delete
+ */
+void irdma_del_local_mac_entry(struct irdma_pci_f *rf, u16 idx)
+{
+ struct irdma_cqp *iwcqp = &rf->cqp;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ enum irdma_status_code status = 0;
+
+ cqp_request = irdma_get_cqp_request(iwcqp, true);
+ if (!cqp_request) {
+ pr_err("cqp_request memory failed\n");
+ return;
+ }
+
+ cqp_info = &cqp_request->info;
+ cqp_info->cqp_cmd = IRDMA_OP_DELETE_LOCAL_MAC_ENTRY;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.del_local_mac_entry.cqp = &iwcqp->sc_cqp;
+ cqp_info->in.u.del_local_mac_entry.scratch = (uintptr_t)cqp_request;
+ cqp_info->in.u.del_local_mac_entry.entry_idx = idx;
+ cqp_info->in.u.del_local_mac_entry.ignore_ref_count = 0;
+ status = irdma_handle_cqp_op(rf, cqp_request);
+ if (status)
+ pr_err("CQP-OP Del MAC entry fail");
+}
+
+/**
+ * irdma_add_mac_entry - add a mac ip address entry to the hw table
+ * @rf: RDMA PCI function
+ * @mac_addr: pointer to mac address
+ * @idx: the index of the mac ip address to add
+ */
+int irdma_add_local_mac_entry(struct irdma_pci_f *rf, u8 *mac_addr, u16 idx)
+{
+ struct irdma_local_mac_entry_info *info;
+ struct irdma_cqp *iwcqp = &rf->cqp;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ enum irdma_status_code status = 0;
+
+ cqp_request = irdma_get_cqp_request(iwcqp, true);
+ if (!cqp_request) {
+ pr_err("cqp_request memory failed\n");
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ cqp_info = &cqp_request->info;
+ cqp_info->post_sq = 1;
+ info = &cqp_info->in.u.add_local_mac_entry.info;
+ ether_addr_copy(info->mac_addr, mac_addr);
+ info->entry_idx = idx;
+ cqp_info->in.u.add_local_mac_entry.scratch = (uintptr_t)cqp_request;
+ cqp_info->cqp_cmd = IRDMA_OP_ADD_LOCAL_MAC_ENTRY;
+ cqp_info->in.u.add_local_mac_entry.cqp = &iwcqp->sc_cqp;
+ cqp_info->in.u.add_local_mac_entry.scratch = (uintptr_t)cqp_request;
+ status = irdma_handle_cqp_op(rf, cqp_request);
+ if (status)
+ pr_err("CQP-OP Add MAC entry fail");
+
+ return status;
+}
+
+/**
+ * irdma_alloc_local_mac_entry - allocate a mac entry
+ * @rf: RDMA PCI function
+ * @mac_tbl_idx: the index of the new mac address
+ *
+ * Allocate a mac address entry and update the mac_tbl_idx
+ * to hold the index of the newly created mac address
+ * Return 0 if successful, otherwise return error
+ */
+int irdma_alloc_local_mac_entry(struct irdma_pci_f *rf, u16 *mac_tbl_idx)
+{
+ struct irdma_cqp *iwcqp = &rf->cqp;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ enum irdma_status_code status = 0;
+
+ cqp_request = irdma_get_cqp_request(iwcqp, true);
+ if (!cqp_request) {
+ pr_err("cqp_request memory failed\n");
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ /* increment refcount, because we need the cqp request ret value */
+ atomic_inc(&cqp_request->refcount);
+ cqp_info = &cqp_request->info;
+ cqp_info->cqp_cmd = IRDMA_OP_ALLOC_LOCAL_MAC_ENTRY;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.alloc_local_mac_entry.cqp = &iwcqp->sc_cqp;
+ cqp_info->in.u.alloc_local_mac_entry.scratch = (uintptr_t)cqp_request;
+ status = irdma_handle_cqp_op(rf, cqp_request);
+ if (!status)
+ *mac_tbl_idx = (u16)cqp_request->compl_info.op_ret_val;
+ else
+ pr_err("CQP-OP Alloc MAC entry fail");
+ /* decrement refcount and free the cqp request, if no longer used */
+ irdma_put_cqp_request(iwcqp, cqp_request);
+
+ return status;
+}
+
+/**
+ * irdma_cqp_manage_apbvt_cmd - send cqp command manage apbvt
+ * @iwdev: iwarp device
+ * @accel_local_port: port for apbvt
+ * @add_port: add ordelete port
+ */
+static enum irdma_status_code
+irdma_cqp_manage_apbvt_cmd(struct irdma_device *iwdev, u16 accel_local_port,
+ bool add_port)
+{
+ struct irdma_apbvt_info *info;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ enum irdma_status_code status;
+
+ cqp_request = irdma_get_cqp_request(&iwdev->rf->cqp, add_port);
+ if (!cqp_request)
+ return IRDMA_ERR_NO_MEMORY;
+
+ cqp_info = &cqp_request->info;
+ info = &cqp_info->in.u.manage_apbvt_entry.info;
+ memset(info, 0, sizeof(*info));
+ info->add = add_port;
+ info->port = accel_local_port;
+ cqp_info->cqp_cmd = IRDMA_OP_MANAGE_APBVT_ENTRY;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.manage_apbvt_entry.cqp = &iwdev->rf->cqp.sc_cqp;
+ cqp_info->in.u.manage_apbvt_entry.scratch = (uintptr_t)cqp_request;
+ status = irdma_handle_cqp_op(iwdev->rf, cqp_request);
+ if (status)
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "ERR: CQP-OP Manage APBVT entry fail");
+
+ return status;
+}
+
+/**
+ * irdma_manage_apbvt - add or delete tcp port
+ * @iwdev: iwarp device
+ * @accel_local_port: port for apbvt
+ * @add_port: add or delete port
+ */
+enum irdma_status_code irdma_manage_apbvt(struct irdma_device *iwdev,
+ u16 accel_local_port, bool add_port)
+{
+ struct irdma_cm_core *cm_core = &iwdev->cm_core;
+ enum irdma_status_code status = 0;
+ unsigned long flags;
+ bool in_use;
+
+ /* apbvt_lock is held across CQP delete APBVT OP (non-waiting) to
+ * protect against race where add APBVT CQP can race ahead of the delete
+ * APBVT for same port.
+ */
+ if (add_port) {
+ spin_lock_irqsave(&cm_core->apbvt_lock, flags);
+ in_use = __test_and_set_bit(accel_local_port,
+ cm_core->ports_in_use);
+ spin_unlock_irqrestore(&cm_core->apbvt_lock, flags);
+ if (in_use)
+ return 0;
+ return irdma_cqp_manage_apbvt_cmd(iwdev, accel_local_port,
+ true);
+ } else {
+ spin_lock_irqsave(&cm_core->apbvt_lock, flags);
+ in_use = irdma_port_in_use(cm_core, accel_local_port);
+ if (in_use) {
+ spin_unlock_irqrestore(&cm_core->apbvt_lock, flags);
+ return 0;
+ }
+ __clear_bit(accel_local_port, cm_core->ports_in_use);
+ status = irdma_cqp_manage_apbvt_cmd(iwdev, accel_local_port,
+ false);
+ spin_unlock_irqrestore(&cm_core->apbvt_lock, flags);
+ return status;
+ }
+}
+
+/**
+ * irdma_manage_arp_cache - manage hw arp cache
+ * @rf: RDMA PCI function
+ * @mac_addr: mac address ptr
+ * @ip_addr: ip addr for arp cache
+ * @ipv4: flag inicating IPv4
+ * @action: add, delete or modify
+ */
+void irdma_manage_arp_cache(struct irdma_pci_f *rf, unsigned char *mac_addr,
+ u32 *ip_addr, bool ipv4, u32 action)
+{
+ struct irdma_add_arp_cache_entry_info *info;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ int arp_index;
+
+ arp_index = irdma_arp_table(rf, ip_addr, ipv4, mac_addr, action);
+ if (arp_index == -1)
+ return;
+
+ cqp_request = irdma_get_cqp_request(&rf->cqp, false);
+ if (!cqp_request)
+ return;
+
+ cqp_info = &cqp_request->info;
+ if (action == IRDMA_ARP_ADD) {
+ cqp_info->cqp_cmd = IRDMA_OP_ADD_ARP_CACHE_ENTRY;
+ info = &cqp_info->in.u.add_arp_cache_entry.info;
+ memset(info, 0, sizeof(*info));
+ info->arp_index = (u16)arp_index;
+ info->permanent = true;
+ ether_addr_copy(info->mac_addr, mac_addr);
+ cqp_info->in.u.add_arp_cache_entry.scratch =
+ (uintptr_t)cqp_request;
+ cqp_info->in.u.add_arp_cache_entry.cqp = &rf->cqp.sc_cqp;
+ } else {
+ cqp_info->cqp_cmd = IRDMA_OP_DELETE_ARP_CACHE_ENTRY;
+ cqp_info->in.u.del_arp_cache_entry.scratch =
+ (uintptr_t)cqp_request;
+ cqp_info->in.u.del_arp_cache_entry.cqp = &rf->cqp.sc_cqp;
+ cqp_info->in.u.del_arp_cache_entry.arp_index = arp_index;
+ }
+
+ cqp_info->in.u.add_arp_cache_entry.cqp = &rf->cqp.sc_cqp;
+ cqp_info->in.u.add_arp_cache_entry.scratch = (uintptr_t)cqp_request;
+ cqp_info->post_sq = 1;
+ if (irdma_handle_cqp_op(rf, cqp_request))
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: CQP-OP Add/Del Arp Cache entry fail");
+}
+
+/**
+ * irdma_send_syn_cqp_callback - do syn/ack after qhash
+ * @cqp_request: qhash cqp completion
+ */
+static void irdma_send_syn_cqp_callback(struct irdma_cqp_request *cqp_request)
+{
+ irdma_send_syn(cqp_request->param, 1);
+}
+
+/**
+ * irdma_manage_qhash - add or modify qhash
+ * @iwdev: iwarp device
+ * @cminfo: cm info for qhash
+ * @etype: type (syn or quad)
+ * @mtype: type of qhash
+ * @cmnode: cmnode associated with connection
+ * @wait: wait for completion
+ */
+enum irdma_status_code
+irdma_manage_qhash(struct irdma_device *iwdev, struct irdma_cm_info *cminfo,
+ enum irdma_quad_entry_type etype,
+ enum irdma_quad_hash_manage_type mtype, void *cmnode,
+ bool wait)
+{
+ struct irdma_qhash_table_info *info;
+ struct irdma_sc_dev *dev = &iwdev->rf->sc_dev;
+ enum irdma_status_code status;
+ struct irdma_cqp *iwcqp = &iwdev->rf->cqp;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+
+ cqp_request = irdma_get_cqp_request(iwcqp, wait);
+ if (!cqp_request)
+ return IRDMA_ERR_NO_MEMORY;
+
+ cqp_info = &cqp_request->info;
+ info = &cqp_info->in.u.manage_qhash_table_entry.info;
+ memset(info, 0, sizeof(*info));
+ info->vsi = &iwdev->vsi;
+ info->manage = mtype;
+ info->entry_type = etype;
+ if (cminfo->vlan_id < VLAN_N_VID) {
+ info->vlan_valid = true;
+ info->vlan_id = cminfo->vlan_id;
+ } else {
+ info->vlan_valid = false;
+ }
+ info->ipv4_valid = cminfo->ipv4;
+ info->user_pri = cminfo->user_pri;
+ ether_addr_copy(info->mac_addr, iwdev->netdev->dev_addr);
+ info->qp_num = cminfo->qh_qpid;
+ info->dest_port = cminfo->loc_port;
+ info->dest_ip[0] = cminfo->loc_addr[0];
+ info->dest_ip[1] = cminfo->loc_addr[1];
+ info->dest_ip[2] = cminfo->loc_addr[2];
+ info->dest_ip[3] = cminfo->loc_addr[3];
+ if (etype == IRDMA_QHASH_TYPE_TCP_ESTABLISHED ||
+ etype == IRDMA_QHASH_TYPE_UDP_UNICAST ||
+ etype == IRDMA_QHASH_TYPE_UDP_MCAST ||
+ etype == IRDMA_QHASH_TYPE_ROCE_MCAST ||
+ etype == IRDMA_QHASH_TYPE_ROCEV2_HW) {
+ info->src_port = cminfo->rem_port;
+ info->src_ip[0] = cminfo->rem_addr[0];
+ info->src_ip[1] = cminfo->rem_addr[1];
+ info->src_ip[2] = cminfo->rem_addr[2];
+ info->src_ip[3] = cminfo->rem_addr[3];
+ }
+ if (cmnode) {
+ cqp_request->callback_fcn = irdma_send_syn_cqp_callback;
+ cqp_request->param = cmnode;
+ }
+ if (info->ipv4_valid)
+ dev_dbg(rfdev_to_dev(dev),
+ "CM: %s IP=%pI4, port=%d, mac=%pM, vlan_id=%d\n",
+ !mtype ? "DELETE" : "ADD", info->dest_ip,
+ info->dest_port, info->mac_addr, cminfo->vlan_id);
+ else
+ dev_dbg(rfdev_to_dev(dev),
+ "CM: %s IP=%pI6, port=%d, mac=%pM, vlan_id=%d\n",
+ !mtype ? "DELETE" : "ADD", info->dest_ip,
+ info->dest_port, info->mac_addr, cminfo->vlan_id);
+ cqp_info->in.u.manage_qhash_table_entry.cqp = &iwdev->rf->cqp.sc_cqp;
+ cqp_info->in.u.manage_qhash_table_entry.scratch = (uintptr_t)cqp_request;
+ cqp_info->cqp_cmd = IRDMA_OP_MANAGE_QHASH_TABLE_ENTRY;
+ cqp_info->post_sq = 1;
+ status = irdma_handle_cqp_op(iwdev->rf, cqp_request);
+ if (status)
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: CQP-OP Manage Qhash Entry fail");
+
+ return status;
+}
+
+/**
+ * irdma_hw_flush_wqes_callback - Check return code after flush
+ * @cqp_request: qhash cqp completion
+ */
+static void irdma_hw_flush_wqes_callback(struct irdma_cqp_request *cqp_request)
+{
+ struct irdma_qp_flush_info *hw_info;
+ struct irdma_sc_qp *qp;
+ struct irdma_qp *iwqp;
+ struct cqp_cmds_info *cqp_info;
+
+ cqp_info = &cqp_request->info;
+ hw_info = &cqp_request->info.in.u.qp_flush_wqes.info;
+ qp = cqp_info->in.u.qp_flush_wqes.qp;
+ iwqp = qp->qp_uk.back_qp;
+
+ if (cqp_request->compl_info.maj_err_code)
+ return;
+ if (hw_info->rq &&
+ (cqp_request->compl_info.min_err_code == IRDMA_CQP_COMPL_SQ_WQE_FLUSHED ||
+ cqp_request->compl_info.min_err_code == 0)) {
+ /* RQ WQE flush was requested but did not happen */
+ qp->qp_uk.rq_flush_complete = true;
+ complete(&iwqp->rq_drained);
+ }
+ if (hw_info->sq &&
+ (cqp_request->compl_info.min_err_code == IRDMA_CQP_COMPL_RQ_WQE_FLUSHED ||
+ cqp_request->compl_info.min_err_code == 0)) {
+ qp->qp_uk.sq_flush_complete = true;
+ complete(&iwqp->sq_drained);
+ }
+}
+
+/**
+ * irdma_hw_flush_wqes - flush qp's wqe
+ * @rf: RDMA PCI function
+ * @qp: hardware control qp
+ * @info: info for flush
+ * @wait: flag wait for completion
+ */
+enum irdma_status_code irdma_hw_flush_wqes(struct irdma_pci_f *rf,
+ struct irdma_sc_qp *qp,
+ struct irdma_qp_flush_info *info,
+ bool wait)
+{
+ enum irdma_status_code status;
+ struct irdma_qp_flush_info *hw_info;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ struct irdma_qp *iwqp = qp->qp_uk.back_qp;
+ unsigned long flags = 0;
+
+ cqp_request = irdma_get_cqp_request(&rf->cqp, wait);
+ if (!cqp_request)
+ return IRDMA_ERR_NO_MEMORY;
+
+ cqp_info = &cqp_request->info;
+ if (!wait)
+ cqp_request->callback_fcn = irdma_hw_flush_wqes_callback;
+ hw_info = &cqp_request->info.in.u.qp_flush_wqes.info;
+ memcpy(hw_info, info, sizeof(*hw_info));
+ cqp_info->cqp_cmd = IRDMA_OP_QP_FLUSH_WQES;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.qp_flush_wqes.qp = qp;
+ cqp_info->in.u.qp_flush_wqes.scratch = (uintptr_t)cqp_request;
+ status = irdma_handle_cqp_op(rf, cqp_request);
+ if (status) {
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: CQP-OP Flush WQE's fail");
+ complete(&iwqp->sq_drained);
+ complete(&iwqp->rq_drained);
+ qp->qp_uk.sq_flush_complete = true;
+ qp->qp_uk.rq_flush_complete = true;
+ return status;
+ }
+
+ if (!wait || cqp_request->compl_info.maj_err_code)
+ return 0;
+
+ if (info->rq) {
+ if (cqp_request->compl_info.min_err_code == IRDMA_CQP_COMPL_SQ_WQE_FLUSHED ||
+ cqp_request->compl_info.min_err_code == 0) {
+ /* RQ WQE flush was requested but did not happen */
+ qp->qp_uk.rq_flush_complete = true;
+ complete(&iwqp->rq_drained);
+ }
+ }
+ if (info->sq) {
+ if (cqp_request->compl_info.min_err_code == IRDMA_CQP_COMPL_RQ_WQE_FLUSHED ||
+ cqp_request->compl_info.min_err_code == 0) {
+ spin_lock_irqsave(&iwqp->lock, flags);
+ /*
+ * Handling case where WQE is posted to empty SQ when
+ * flush has not completed
+ */
+ if (IRDMA_RING_MORE_WORK(qp->qp_uk.sq_ring)) {
+ struct irdma_cqp_request *new_req;
+
+ qp->flush_sq = false;
+ spin_unlock_irqrestore(&iwqp->lock, flags);
+ info->rq = false;
+ new_req = irdma_get_cqp_request(&rf->cqp, true);
+ if (!new_req)
+ return IRDMA_ERR_NO_MEMORY;
+ cqp_info = &new_req->info;
+ hw_info = &new_req->info.in.u.qp_flush_wqes.info;
+ memcpy(hw_info, info, sizeof(*hw_info));
+ cqp_info->cqp_cmd = IRDMA_OP_QP_FLUSH_WQES;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.qp_flush_wqes.qp = qp;
+ cqp_info->in.u.qp_flush_wqes.scratch = (uintptr_t)new_req;
+
+ status = irdma_handle_cqp_op(rf, new_req);
+ if (new_req->compl_info.maj_err_code ||
+ new_req->compl_info.min_err_code != IRDMA_CQP_COMPL_SQ_WQE_FLUSHED ||
+ status) {
+ pr_warn("SQ in error but not flushed");
+ qp->qp_uk.sq_flush_complete = true;
+ }
+ } else {
+ /* SQ WQE flush was requested but did not happen */
+ qp->qp_uk.sq_flush_complete = true;
+ spin_unlock_irqrestore(&iwqp->lock, flags);
+ complete(&iwqp->sq_drained);
+ }
+ } else {
+ spin_lock_irqsave(&iwqp->lock, flags);
+ if (!IRDMA_RING_MORE_WORK(qp->qp_uk.sq_ring)) {
+ qp->qp_uk.sq_flush_complete = true;
+ spin_unlock_irqrestore(&iwqp->lock, flags);
+ complete(&iwqp->sq_drained);
+ } else {
+ spin_unlock_irqrestore(&iwqp->lock, flags);
+ }
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * irdma_gen_ae - generate AE
+ * @rf: RDMA PCI function
+ * @qp: qp associated with AE
+ * @info: info for ae
+ * @wait: wait for completion
+ */
+void irdma_gen_ae(struct irdma_pci_f *rf, struct irdma_sc_qp *qp,
+ struct irdma_gen_ae_info *info, bool wait)
+{
+ struct irdma_gen_ae_info *ae_info;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+
+ cqp_request = irdma_get_cqp_request(&rf->cqp, wait);
+ if (!cqp_request)
+ return;
+
+ cqp_info = &cqp_request->info;
+ ae_info = &cqp_request->info.in.u.gen_ae.info;
+ memcpy(ae_info, info, sizeof(*ae_info));
+ cqp_info->cqp_cmd = IRDMA_OP_GEN_AE;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.gen_ae.qp = qp;
+ cqp_info->in.u.gen_ae.scratch = (uintptr_t)cqp_request;
+ if (irdma_handle_cqp_op(rf, cqp_request))
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: CQP OP failed attempting to generate ae_code=0x%x\n",
+ info->ae_code);
+}
+
+/**
+ * irdma_get_ib_wc - return change flush code to IB's
+ * @opcode: iwarp flush code
+ */
+static enum ib_wc_status irdma_get_ib_wc(enum irdma_flush_opcode opcode)
+{
+ switch (opcode) {
+ case FLUSH_PROT_ERR:
+ return IB_WC_LOC_PROT_ERR;
+ case FLUSH_REM_ACCESS_ERR:
+ return IB_WC_REM_ACCESS_ERR;
+ case FLUSH_LOC_QP_OP_ERR:
+ return IB_WC_LOC_QP_OP_ERR;
+ case FLUSH_REM_OP_ERR:
+ return IB_WC_REM_OP_ERR;
+ case FLUSH_LOC_LEN_ERR:
+ return IB_WC_LOC_LEN_ERR;
+ case FLUSH_GENERAL_ERR:
+ return IB_WC_GENERAL_ERR;
+ case FLUSH_FATAL_ERR:
+ default:
+ return IB_WC_FATAL_ERR;
+ }
+}
+
+/**
+ * irdma_set_flush_info - set flush info
+ * @pinfo: set flush info
+ * @min: minor err
+ * @maj: major err
+ * @opcode: flush error code
+ */
+static void irdma_set_flush_info(struct irdma_qp_flush_info *pinfo, u16 *min,
+ u16 *maj, enum irdma_flush_opcode opcode)
+{
+ *min = (u16)irdma_get_ib_wc(opcode);
+ *maj = CQE_MAJOR_DRV;
+ pinfo->userflushcode = true;
+}
+
+/**
+ * irdma_flush_wqes - flush wqe for qp
+ * @rf: RDMA PCI function
+ * @iwqp: qp to flush wqes
+ */
+void irdma_flush_wqes(struct irdma_pci_f *rf, struct irdma_qp *iwqp)
+{
+ struct irdma_qp_flush_info info = {};
+ struct irdma_sc_qp *qp = &iwqp->sc_qp;
+
+ info.sq = true;
+ info.rq = true;
+ if (qp->term_flags) {
+ irdma_set_flush_info(&info, &info.sq_minor_code,
+ &info.sq_major_code, qp->flush_code);
+ irdma_set_flush_info(&info, &info.rq_minor_code,
+ &info.rq_major_code, qp->flush_code);
+ }
+ (void)irdma_hw_flush_wqes(rf, &iwqp->sc_qp, &info, true);
+}
diff --git a/drivers/infiniband/hw/irdma/i40iw_hw.c b/drivers/infiniband/hw/irdma/i40iw_hw.c
new file mode 100644
index 000000000000..8a6333ca3675
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/i40iw_hw.c
@@ -0,0 +1,210 @@
+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "osdep.h"
+#include "type.h"
+#include "i40iw_hw.h"
+#include "status.h"
+#include "protos.h"
+
+#define I40E_CQPSQ_CQ_CQID_SHIFT 0
+#define I40E_CQPSQ_CQ_CQID_MASK \
+ (0xffffULL << I40E_CQPSQ_CQ_CQID_SHIFT)
+
+static u32 i40iw_regs[IRDMA_MAX_REGS] = {
+ I40E_PFPE_CQPTAIL,
+ I40E_PFPE_CQPDB,
+ I40E_PFPE_CCQPSTATUS,
+ I40E_PFPE_CCQPHIGH,
+ I40E_PFPE_CCQPLOW,
+ I40E_PFPE_CQARM,
+ I40E_PFPE_CQACK,
+ I40E_PFPE_AEQALLOC,
+ I40E_PFPE_CQPERRCODES,
+ I40E_PFPE_WQEALLOC,
+ I40E_PFINT_DYN_CTLN(0),
+ I40IW_DB_ADDR_OFFSET,
+
+ I40E_GLPCI_LBARCTRL,
+ I40E_GLPE_CPUSTATUS0,
+ I40E_GLPE_CPUSTATUS1,
+ I40E_GLPE_CPUSTATUS2,
+ I40E_PFINT_AEQCTL,
+ I40E_PFINT_CEQCTL(0),
+ I40E_VSIQF_CTL(0),
+ I40E_PFHMC_PDINV,
+ I40E_GLHMC_VFPDINV(0)
+};
+
+static u32 i40iw_stat_offsets_32[IRDMA_HW_STAT_INDEX_MAX_32] = {
+ I40E_GLPES_PFIP4RXDISCARD(0),
+ I40E_GLPES_PFIP4RXTRUNC(0),
+ I40E_GLPES_PFIP4TXNOROUTE(0),
+ I40E_GLPES_PFIP6RXDISCARD(0),
+ I40E_GLPES_PFIP6RXTRUNC(0),
+ I40E_GLPES_PFIP6TXNOROUTE(0),
+ I40E_GLPES_PFTCPRTXSEG(0),
+ I40E_GLPES_PFTCPRXOPTERR(0),
+ I40E_GLPES_PFTCPRXPROTOERR(0),
+ I40E_GLPES_PFRXVLANERR(0)
+};
+
+static u32 i40iw_stat_offsets_64[IRDMA_HW_STAT_INDEX_MAX_64] = {
+ I40E_GLPES_PFIP4RXOCTSLO(0),
+ I40E_GLPES_PFIP4RXPKTSLO(0),
+ I40E_GLPES_PFIP4RXFRAGSLO(0),
+ I40E_GLPES_PFIP4RXMCPKTSLO(0),
+ I40E_GLPES_PFIP4TXOCTSLO(0),
+ I40E_GLPES_PFIP4TXPKTSLO(0),
+ I40E_GLPES_PFIP4TXFRAGSLO(0),
+ I40E_GLPES_PFIP4TXMCPKTSLO(0),
+ I40E_GLPES_PFIP6RXOCTSLO(0),
+ I40E_GLPES_PFIP6RXPKTSLO(0),
+ I40E_GLPES_PFIP6RXFRAGSLO(0),
+ I40E_GLPES_PFIP6RXMCPKTSLO(0),
+ I40E_GLPES_PFIP6TXOCTSLO(0),
+ I40E_GLPES_PFIP6TXPKTSLO(0),
+ I40E_GLPES_PFIP6TXFRAGSLO(0),
+ I40E_GLPES_PFIP6TXMCPKTSLO(0),
+ I40E_GLPES_PFTCPRXSEGSLO(0),
+ I40E_GLPES_PFTCPTXSEGLO(0),
+ I40E_GLPES_PFRDMARXRDSLO(0),
+ I40E_GLPES_PFRDMARXSNDSLO(0),
+ I40E_GLPES_PFRDMARXWRSLO(0),
+ I40E_GLPES_PFRDMATXRDSLO(0),
+ I40E_GLPES_PFRDMATXSNDSLO(0),
+ I40E_GLPES_PFRDMATXWRSLO(0),
+ I40E_GLPES_PFRDMAVBNDLO(0),
+ I40E_GLPES_PFRDMAVINVLO(0),
+ I40E_GLPES_PFIP4RXMCOCTSLO(0),
+ I40E_GLPES_PFIP4TXMCOCTSLO(0),
+ I40E_GLPES_PFIP6RXMCOCTSLO(0),
+ I40E_GLPES_PFIP6TXMCOCTSLO(0),
+ I40E_GLPES_PFUDPRXPKTSLO(0),
+ I40E_GLPES_PFUDPTXPKTSLO(0)
+};
+
+static u64 i40iw_masks[IRDMA_MAX_MASKS] = {
+ I40E_PFPE_CCQPSTATUS_CCQP_DONE_MASK,
+ I40E_PFPE_CCQPSTATUS_CCQP_ERR_MASK,
+ I40E_CQPSQ_STAG_PDID_MASK,
+ I40E_CQPSQ_CQ_CEQID_MASK,
+ I40E_CQPSQ_CQ_CQID_MASK,
+};
+
+static u64 i40iw_shifts[IRDMA_MAX_SHIFTS] = {
+ I40E_PFPE_CCQPSTATUS_CCQP_DONE_SHIFT,
+ I40E_PFPE_CCQPSTATUS_CCQP_ERR_SHIFT,
+ I40E_CQPSQ_STAG_PDID_SHIFT,
+ I40E_CQPSQ_CQ_CEQID_SHIFT,
+ I40E_CQPSQ_CQ_CQID_SHIFT,
+};
+
+static struct irdma_irq_ops i40iw_irq_ops;
+
+/**
+ * i40iw_config_ceq- Configure CEQ interrupt
+ * @dev: pointer to the device structure
+ * @ceq_id: Completion Event Queue ID
+ * @idx: vector index
+ */
+static void i40iw_config_ceq(struct irdma_sc_dev *dev, u32 ceq_id, u32 idx)
+{
+ u32 reg_val;
+
+ reg_val = (ceq_id << I40E_PFINT_LNKLSTN_FIRSTQ_INDX_SHIFT);
+ reg_val |= (QUEUE_TYPE_CEQ << I40E_PFINT_LNKLSTN_FIRSTQ_TYPE_SHIFT);
+ wr32(dev->hw, I40E_PFINT_LNKLSTN(idx - 1), reg_val);
+
+ reg_val = (0x3 << I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT);
+ reg_val |= I40E_PFINT_DYN_CTLN_INTENA_MASK;
+ wr32(dev->hw, I40E_PFINT_DYN_CTLN(idx - 1), reg_val);
+
+ reg_val = (IRDMA_GLINT_CEQCTL_CAUSE_ENA_M |
+ (idx << IRDMA_GLINT_CEQCTL_MSIX_INDX_S) |
+ IRDMA_GLINT_CEQCTL_ITR_INDX_M);
+ reg_val |= (NULL_QUEUE_INDEX << I40E_PFINT_CEQCTL_NEXTQ_INDX_SHIFT);
+
+ wr32(dev->hw, dev->hw_regs[IRDMA_GLINT_CEQCTL] + 4 * ceq_id, reg_val);
+}
+
+/**
+ * i40iw_ena_irq - Enable interrupt
+ * @dev: pointer to the device structure
+ * @idx: vector index
+ */
+static void i40iw_ena_irq(struct irdma_sc_dev *dev, u32 idx)
+{
+ u32 val;
+
+ val = IRDMA_GLINT_DYN_CTL_INTENA_M | IRDMA_GLINT_DYN_CTL_CLEARPBA_M |
+ IRDMA_GLINT_DYN_CTL_ITR_INDX_M;
+ wr32(dev->hw, dev->hw_regs[IRDMA_GLINT_DYN_CTL] + 4 * (idx - 1), val);
+}
+
+/**
+ * irdma_disable_irq - Disable interrupt
+ * @dev: pointer to the device structure
+ * @idx: vector index
+ */
+static void i40iw_disable_irq(struct irdma_sc_dev *dev, u32 idx)
+{
+ wr32(dev->hw, dev->hw_regs[IRDMA_GLINT_DYN_CTL] + 4 * (idx - 1), 0);
+}
+
+void i40iw_init_hw(struct irdma_sc_dev *dev)
+{
+ int i;
+
+ for (i = 0; i < IRDMA_MAX_REGS; ++i)
+ dev->hw_regs[i] = i40iw_regs[i];
+
+ for (i = 0; i < IRDMA_HW_STAT_INDEX_MAX_32; ++i)
+ dev->hw_stats_regs_32[i] = i40iw_stat_offsets_32[i];
+
+ for (i = 0; i < IRDMA_HW_STAT_INDEX_MAX_64; ++i)
+ dev->hw_stats_regs_64[i] = i40iw_stat_offsets_64[i];
+
+ for (i = 0; i < IRDMA_MAX_SHIFTS; ++i)
+ dev->hw_shifts[i] = i40iw_shifts[i];
+
+ for (i = 0; i < IRDMA_MAX_MASKS; ++i)
+ dev->hw_masks[i] = i40iw_masks[i];
+
+ dev->wqe_alloc_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_WQEALLOC]);
+ dev->cq_arm_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQARM]);
+ dev->aeq_alloc_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_AEQALLOC]);
+ dev->cqp_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQPDB]);
+ dev->cq_ack_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQACK]);
+ dev->ceq_itr_mask_db = NULL;
+ dev->aeq_itr_mask_db = NULL;
+
+ memcpy(&i40iw_irq_ops, dev->irq_ops, sizeof(i40iw_irq_ops));
+ i40iw_irq_ops.irdma_en_irq = i40iw_ena_irq;
+ i40iw_irq_ops.irdma_dis_irq = i40iw_disable_irq;
+ i40iw_irq_ops.irdma_cfg_ceq = i40iw_config_ceq;
+ dev->irq_ops = &i40iw_irq_ops;
+
+ /* Setup the hardware limits, hmc may limit further */
+ dev->hw_attrs.uk_attrs.max_hw_wq_frags = I40IW_MAX_WQ_FRAGMENT_COUNT;
+ dev->hw_attrs.uk_attrs.max_hw_read_sges = I40IW_MAX_SGE_RD;
+ dev->hw_attrs.max_hw_device_pages = I40IW_MAX_PUSH_PAGE_COUNT;
+ dev->hw_attrs.first_hw_vf_fpm_id = I40IW_FIRST_VF_FPM_ID;
+ dev->hw_attrs.uk_attrs.max_hw_inline = I40IW_MAX_INLINE_DATA_SIZE;
+ dev->hw_attrs.max_hw_ird = I40IW_MAX_IRD_SIZE;
+ dev->hw_attrs.max_hw_ord = I40IW_MAX_ORD_SIZE;
+ dev->hw_attrs.max_hw_wqes = I40IW_MAX_WQ_ENTRIES;
+ dev->hw_attrs.uk_attrs.max_hw_rq_quanta = I40IW_QP_SW_MAX_RQ_QUANTA;
+ dev->hw_attrs.uk_attrs.max_hw_wq_quanta = I40IW_QP_SW_MAX_WQ_QUANTA;
+ dev->hw_attrs.uk_attrs.max_hw_sq_chunk = I40IW_MAX_QUANTA_PER_WR;
+ dev->hw_attrs.max_hw_pds = I40IW_MAX_PDS;
+ dev->hw_attrs.max_stat_inst = I40IW_MAX_STATS_COUNT;
+ dev->hw_attrs.max_hw_outbound_msg_size = I40IW_MAX_OUTBOUND_MSG_SIZE;
+ dev->hw_attrs.max_hw_inbound_msg_size = I40IW_MAX_INBOUND_MSG_SIZE;
+ dev->hw_attrs.max_qp_wr = I40IW_MAX_QP_WRS;
+}
diff --git a/drivers/infiniband/hw/irdma/i40iw_hw.h b/drivers/infiniband/hw/irdma/i40iw_hw.h
new file mode 100644
index 000000000000..95ff50c1fa80
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/i40iw_hw.h
@@ -0,0 +1,163 @@
+/* SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef I40IW_HW_H
+#define I40IW_HW_H
+#define I40E_VFPE_CQPTAIL1 0x0000A000 /* Reset: VFR */
+#define I40E_VFPE_CQPDB1 0x0000BC00 /* Reset: VFR */
+#define I40E_VFPE_CCQPSTATUS1 0x0000B800 /* Reset: VFR */
+#define I40E_VFPE_CCQPHIGH1 0x00009800 /* Reset: VFR */
+#define I40E_VFPE_CCQPLOW1 0x0000AC00 /* Reset: VFR */
+#define I40E_VFPE_CQARM1 0x0000B400 /* Reset: VFR */
+#define I40E_VFPE_CQACK1 0x0000B000 /* Reset: VFR */
+#define I40E_VFPE_AEQALLOC1 0x0000A400 /* Reset: VFR */
+#define I40E_VFPE_CQPERRCODES1 0x00009C00 /* Reset: VFR */
+#define I40E_VFPE_WQEALLOC1 0x0000C000 /* Reset: VFR */
+#define I40E_VFINT_DYN_CTLN(_INTVF) (0x00024800 + ((_INTVF) * 4)) /* _i=0...511 */ /* Reset: VFR */
+
+#define I40E_PFPE_CQPTAIL 0x00008080 /* Reset: PFR */
+
+#define I40E_PFPE_CQPDB 0x00008000 /* Reset: PFR */
+#define I40E_PFPE_CCQPSTATUS 0x00008100 /* Reset: PFR */
+#define I40E_PFPE_CCQPHIGH 0x00008200 /* Reset: PFR */
+#define I40E_PFPE_CCQPLOW 0x00008180 /* Reset: PFR */
+#define I40E_PFPE_CQARM 0x00131080 /* Reset: PFR */
+#define I40E_PFPE_CQACK 0x00131100 /* Reset: PFR */
+#define I40E_PFPE_AEQALLOC 0x00131180 /* Reset: PFR */
+#define I40E_PFPE_CQPERRCODES 0x00008880 /* Reset: PFR */
+#define I40E_PFPE_WQEALLOC 0x00138C00 /* Reset: PFR */
+#define I40E_GLPCI_LBARCTRL 0x000BE484 /* Reset: POR */
+#define I40E_GLPE_CPUSTATUS0 0x0000D040 /* Reset: PE_CORER */
+#define I40E_GLPE_CPUSTATUS1 0x0000D044 /* Reset: PE_CORER */
+#define I40E_GLPE_CPUSTATUS2 0x0000D048 /* Reset: PE_CORER */
+#define I40E_PFHMC_PDINV 0x000C0300 /* Reset: PFR */
+#define I40E_GLHMC_VFPDINV(_i) (0x000C8300 + ((_i) * 4)) /* _i=0...31 */ /* Reset: CORER */
+#define I40E_PFINT_DYN_CTLN(_INTPF) (0x00034800 + ((_INTPF) * 4)) /* _i=0...511 */ /* Reset: PFR */
+#define I40E_PFINT_AEQCTL 0x00038700 /* Reset: CORER */
+
+#define I40E_GLPES_PFIP4RXDISCARD(_i) (0x00010600 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXTRUNC(_i) (0x00010700 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXNOROUTE(_i) (0x00012E00 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXDISCARD(_i) (0x00011200 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXTRUNC(_i) (0x00011300 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+
+#define I40E_GLPES_PFRDMAVBNDLO(_i) (0x00014800 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXMCOCTSLO(_i) (0x00012000 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXMCOCTSLO(_i) (0x00011600 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6TXMCOCTSLO(_i) (0x00012A00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFUDPRXPKTSLO(_i) (0x00013800 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFUDPTXPKTSLO(_i) (0x00013A00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+
+#define I40E_GLPES_PFIP6TXNOROUTE(_i) (0x00012F00 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFTCPRTXSEG(_i) (0x00013600 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFTCPRXOPTERR(_i) (0x00013200 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFTCPRXPROTOERR(_i) (0x00013300 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRXVLANERR(_i) (0x00010000 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXOCTSLO(_i) (0x00010200 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXPKTSLO(_i) (0x00010400 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXFRAGSLO(_i) (0x00010800 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXMCPKTSLO(_i) (0x00010C00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXOCTSLO(_i) (0x00011A00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXPKTSLO(_i) (0x00011C00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXFRAGSLO(_i) (0x00011E00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXMCPKTSLO(_i) (0x00012200 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXOCTSLO(_i) (0x00010E00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXPKTSLO(_i) (0x00011000 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXFRAGSLO(_i) (0x00011400 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6TXOCTSLO(_i) (0x00012400 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6TXPKTSLO(_i) (0x00012600 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6TXFRAGSLO(_i) (0x00012800 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6TXMCPKTSLO(_i) (0x00012C00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFTCPTXSEGLO(_i) (0x00013400 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMARXRDSLO(_i) (0x00013E00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMARXSNDSLO(_i) (0x00014000 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMARXWRSLO(_i) (0x00013C00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMATXRDSLO(_i) (0x00014400 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMATXSNDSLO(_i) (0x00014600 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMATXWRSLO(_i) (0x00014200 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXMCOCTSLO(_i) (0x00010A00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXMCPKTSLO(_i) (0x00011800 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFTCPRXSEGSLO(_i) (0x00013000 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMAVINVLO(_i) (0x00014A00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+
+#define I40IW_DB_ADDR_OFFSET (4 * 1024 * 1024 - 64 * 1024)
+
+#define I40IW_VF_DB_ADDR_OFFSET (64 * 1024)
+
+#define I40E_PFINT_LNKLSTN(_INTPF) (0x00035000 + ((_INTPF) * 4)) /* _i=0...511 */ /* Reset: PFR */
+#define I40E_PFINT_LNKLSTN_MAX_INDEX 511
+#define I40E_PFINT_LNKLSTN_FIRSTQ_INDX_SHIFT 0
+#define I40E_PFINT_LNKLSTN_FIRSTQ_INDX_MASK I40E_MASK(0x7FF, I40E_PFINT_LNKLSTN_FIRSTQ_INDX_SHIFT)
+#define I40E_PFINT_LNKLSTN_FIRSTQ_TYPE_SHIFT 11
+#define I40E_PFINT_LNKLSTN_FIRSTQ_TYPE_MASK I40E_MASK(0x3, I40E_PFINT_LNKLSTN_FIRSTQ_TYPE_SHIFT)
+
+#define I40E_PFINT_CEQCTL(_INTPF) (0x00036800 + ((_INTPF) * 4)) /* _i=0...511 */ /* Reset: CORER */
+#define I40E_PFINT_CEQCTL_MAX_INDEX 511
+#define I40E_PFINT_CEQCTL_MSIX_INDX_SHIFT 0
+#define I40E_PFINT_CEQCTL_MSIX_INDX_MASK I40E_MASK(0xFF, I40E_PFINT_CEQCTL_MSIX_INDX_SHIFT)
+#define I40E_PFINT_CEQCTL_ITR_INDX_SHIFT 11
+#define I40E_PFINT_CEQCTL_ITR_INDX_MASK I40E_MASK(0x3, I40E_PFINT_CEQCTL_ITR_INDX_SHIFT)
+#define I40E_PFINT_CEQCTL_MSIX0_INDX_SHIFT 13
+#define I40E_PFINT_CEQCTL_MSIX0_INDX_MASK I40E_MASK(0x7, I40E_PFINT_CEQCTL_MSIX0_INDX_SHIFT)
+#define I40E_PFINT_CEQCTL_NEXTQ_INDX_SHIFT 16
+#define I40E_PFINT_CEQCTL_NEXTQ_INDX_MASK I40E_MASK(0x7FF, I40E_PFINT_CEQCTL_NEXTQ_INDX_SHIFT)
+#define I40E_PFINT_CEQCTL_NEXTQ_TYPE_SHIFT 27
+#define I40E_PFINT_CEQCTL_NEXTQ_TYPE_MASK I40E_MASK(0x3, I40E_PFINT_CEQCTL_NEXTQ_TYPE_SHIFT)
+#define I40E_PFINT_CEQCTL_CAUSE_ENA_SHIFT 30
+#define I40E_PFINT_CEQCTL_CAUSE_ENA_MASK I40E_MASK(0x1, I40E_PFINT_CEQCTL_CAUSE_ENA_SHIFT)
+#define I40E_PFINT_CEQCTL_INTEVENT_SHIFT 31
+#define I40E_PFINT_CEQCTL_INTEVENT_MASK I40E_MASK(0x1, I40E_PFINT_CEQCTL_INTEVENT_SHIFT)
+
+#define I40E_CQPSQ_STAG_PDID_SHIFT 48
+#define I40E_CQPSQ_STAG_PDID_MASK MAKEMASK(0x7FFFULL, I40E_CQPSQ_STAG_PDID_SHIFT)
+
+#define I40E_PFPE_CCQPSTATUS_CCQP_DONE_SHIFT 0
+#define I40E_PFPE_CCQPSTATUS_CCQP_DONE_MASK (0x1ULL << I40E_PFPE_CCQPSTATUS_CCQP_DONE_SHIFT)
+
+#define I40E_PFPE_CCQPSTATUS_CCQP_ERR_SHIFT 31
+#define I40E_PFPE_CCQPSTATUS_CCQP_ERR_MASK (0x1ULL << I40E_PFPE_CCQPSTATUS_CCQP_ERR_SHIFT)
+
+#define I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT 3
+#define I40E_PFINT_DYN_CTLN_ITR_INDX_MASK (0x3 << I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT)
+
+#define I40E_PFINT_DYN_CTLN_INTENA_SHIFT 0
+#define I40E_PFINT_DYN_CTLN_INTENA_MASK (0x1 << I40E_PFINT_DYN_CTLN_INTENA_SHIFT)
+
+#define I40E_CQPSQ_CQ_CEQID_SHIFT 24
+#define I40E_CQPSQ_CQ_CEQID_MASK (0x7fUL << I40E_CQPSQ_CQ_CEQID_SHIFT)
+
+#define I40E_VSIQF_CTL(_VSI) (0x0020D800 + ((_VSI) * 4))
+
+enum i40iw_device_caps_const {
+ I40IW_MAX_WQ_FRAGMENT_COUNT = 3,
+ I40IW_MAX_SGE_RD = 1,
+ I40IW_MAX_PUSH_PAGE_COUNT = 0,
+ I40IW_MAX_INLINE_DATA_SIZE = 48,
+ I40IW_MAX_IRD_SIZE = 63,
+ I40IW_MAX_ORD_SIZE = 127,
+ I40IW_MAX_WQ_ENTRIES = 2048,
+ I40IW_MAX_WQE_SIZE_RQ = 128,
+ I40IW_MAX_PDS = 32768,
+ I40IW_MAX_STATS_COUNT = 16,
+ I40IW_MAX_CQ_SIZE = 1048575,
+ I40IW_MAX_OUTBOUND_MSG_SIZE = 2147483647,
+ I40IW_MAX_INBOUND_MSG_SIZE = 2147483647,
+};
+
+#define I40IW_QP_WQE_MIN_SIZE 32
+#define I40IW_QP_WQE_MAX_SIZE 128
+#define I40IW_QP_SW_MIN_WQSIZE 4
+
+#define I40IW_MAX_RQ_WQE_SHIFT 2
+#define I40IW_MAX_QUANTA_PER_WR 2
+
+#define I40IW_QP_SW_MAX_SQ_QUANTA 2048
+#define I40IW_QP_SW_MAX_RQ_QUANTA 16384
+#define I40IW_QP_SW_MAX_WQ_QUANTA 2048
+#define I40IW_MAX_QP_WRS ((I40IW_QP_SW_MAX_SQ_QUANTA - IRDMA_SQ_RSVD) / I40IW_MAX_QUANTA_PER_WR)
+#define I40IW_FIRST_VF_FPM_ID 16
+#define QUEUE_TYPE_CEQ 2
+#define NULL_QUEUE_INDEX 0x7FF
+
+void i40iw_init_hw(struct irdma_sc_dev *dev);
+#endif /* I40IW_HW_H */
diff --git a/drivers/infiniband/hw/irdma/icrdma_hw.c b/drivers/infiniband/hw/irdma/icrdma_hw.c
new file mode 100644
index 000000000000..18f5e004ffcf
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/icrdma_hw.c
@@ -0,0 +1,75 @@
+// SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "osdep.h"
+#include "type.h"
+#include "icrdma_hw.h"
+
+static u32 icrdma_regs[IRDMA_MAX_REGS] = {
+ PFPE_CQPTAIL,
+ PFPE_CQPDB,
+ PFPE_CCQPSTATUS,
+ PFPE_CCQPHIGH,
+ PFPE_CCQPLOW,
+ PFPE_CQARM,
+ PFPE_CQACK,
+ PFPE_AEQALLOC,
+ PFPE_CQPERRCODES,
+ PFPE_WQEALLOC,
+ GLINT_DYN_CTL(0),
+ ICRDMA_DB_ADDR_OFFSET,
+
+ GLPCI_LBARCTRL,
+ GLPE_CPUSTATUS0,
+ GLPE_CPUSTATUS1,
+ GLPE_CPUSTATUS2,
+ PFINT_AEQCTL,
+ GLINT_CEQCTL(0),
+ VSIQF_PE_CTL1(0),
+ PFHMC_PDINV,
+ GLHMC_VFPDINV(0)
+};
+
+static u64 icrdma_masks[IRDMA_MAX_MASKS] = {
+ ICRDMA_CCQPSTATUS_CCQP_DONE_M,
+ ICRDMA_CCQPSTATUS_CCQP_ERR_M,
+ ICRDMA_CQPSQ_STAG_PDID_M,
+ ICRDMA_CQPSQ_CQ_CEQID_M,
+ ICRDMA_CQPSQ_CQ_CQID_M,
+};
+
+static u64 icrdma_shifts[IRDMA_MAX_SHIFTS] = {
+ ICRDMA_CCQPSTATUS_CCQP_DONE_S,
+ ICRDMA_CCQPSTATUS_CCQP_ERR_S,
+ ICRDMA_CQPSQ_STAG_PDID_S,
+ ICRDMA_CQPSQ_CQ_CEQID_S,
+ ICRDMA_CQPSQ_CQ_CQID_S,
+};
+
+void icrdma_init_hw(struct irdma_sc_dev *dev)
+{
+ int i;
+
+ for (i = 0; i < IRDMA_MAX_REGS; ++i)
+ dev->hw_regs[i] = icrdma_regs[i];
+
+ for (i = 0; i < IRDMA_MAX_SHIFTS; ++i)
+ dev->hw_shifts[i] = icrdma_shifts[i];
+
+ for (i = 0; i < IRDMA_MAX_MASKS; ++i)
+ dev->hw_masks[i] = icrdma_masks[i];
+
+ dev->wqe_alloc_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_WQEALLOC]);
+ dev->cq_arm_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQARM]);
+ dev->aeq_alloc_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_AEQALLOC]);
+ dev->cqp_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQPDB]);
+ dev->cq_ack_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQACK]);
+ dev->hw_attrs.max_stat_inst = ICRDMA_MAX_STATS_COUNT;
+
+ dev->hw_attrs.uk_attrs.max_hw_sq_chunk = IRDMA_MAX_QUANTA_PER_WR;
+}
diff --git a/drivers/infiniband/hw/irdma/icrdma_hw.h b/drivers/infiniband/hw/irdma/icrdma_hw.h
new file mode 100644
index 000000000000..485edcdd589d
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/icrdma_hw.h
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0 or Linux-OpenIB */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef ICRDMA_HW_H
+#define ICRDMA_HW_H
+
+#define VFPE_CQPTAIL1 0x0000a000
+#define VFPE_CQPDB1 0x0000bc00
+#define VFPE_CCQPSTATUS1 0x0000b800
+#define VFPE_CCQPHIGH1 0x00009800
+#define VFPE_CCQPLOW1 0x0000ac00
+#define VFPE_CQARM1 0x0000b400
+#define VFPE_CQARM1 0x0000b400
+#define VFPE_CQACK1 0x0000b000
+#define VFPE_AEQALLOC1 0x0000a400
+#define VFPE_CQPERRCODES1 0x00009c00
+#define VFPE_WQEALLOC1 0x0000c000
+#define VFINT_DYN_CTLN(_i) (0x00003800 + ((_i) * 4)) /* _i=0...63 */
+
+#define PFPE_CQPTAIL 0x00500880
+#define PFPE_CQPDB 0x00500800
+#define PFPE_CCQPSTATUS 0x0050a000
+#define PFPE_CCQPHIGH 0x0050a100
+#define PFPE_CCQPLOW 0x0050a080
+#define PFPE_CQARM 0x00502c00
+#define PFPE_CQACK 0x00502c80
+#define PFPE_AEQALLOC 0x00502d00
+#define GLINT_DYN_CTL(_INT) (0x00160000 + ((_INT) * 4)) /* _i=0...2047 */
+#define GLPCI_LBARCTRL 0x0009de74
+#define GLPE_CPUSTATUS0 0x0050ba5c
+#define GLPE_CPUSTATUS1 0x0050ba60
+#define GLPE_CPUSTATUS2 0x0050ba64
+#define PFINT_AEQCTL 0x0016cb00
+#define PFPE_CQPERRCODES 0x0050a200
+#define PFPE_WQEALLOC 0x00504400
+#define GLINT_CEQCTL(_INT) (0x0015c000 + ((_INT) * 4)) /* _i=0...2047 */
+#define VSIQF_PE_CTL1(_VSI) (0x00414000 + ((_VSI) * 4)) /* _i=0...767 */
+#define PFHMC_PDINV 0x00520300
+#define GLHMC_VFPDINV(_i) (0x00528300 + ((_i) * 4)) /* _i=0...31 */
+
+#define ICRDMA_DB_ADDR_OFFSET (8 * 1024 * 1024 - 64 * 1024)
+
+#define ICRDMA_VF_DB_ADDR_OFFSET (64 * 1024)
+
+/* CCQSTATUS */
+#define ICRDMA_CCQPSTATUS_CCQP_DONE_S 0
+#define ICRDMA_CCQPSTATUS_CCQP_DONE_M MAKEMASK(0x1ULL, ICRDMA_CCQPSTATUS_CCQP_DONE_S)
+#define ICRDMA_CCQPSTATUS_CCQP_ERR_S 31
+#define ICRDMA_CCQPSTATUS_CCQP_ERR_M MAKEMASK(0x1ULL, ICRDMA_CCQPSTATUS_CCQP_ERR_S)
+#define ICRDMA_CQPSQ_STAG_PDID_S 46
+#define ICRDMA_CQPSQ_STAG_PDID_M MAKEMASK(0x3ffffULL, ICRDMA_CQPSQ_STAG_PDID_S)
+#define ICRDMA_CQPSQ_CQ_CEQID_S 22
+#define ICRDMA_CQPSQ_CQ_CEQID_M (0x3ffULL << ICRDMA_CQPSQ_CQ_CEQID_S)
+#define ICRDMA_CQPSQ_CQ_CQID_S 0
+#define ICRDMA_CQPSQ_CQ_CQID_M \
+ (0x7ffffULL << ICRDMA_CQPSQ_CQ_CQID_S)
+
+enum icrdma_device_caps_const {
+ ICRDMA_MAX_STATS_COUNT = 128,
+};
+
+void icrdma_init_hw(struct irdma_sc_dev *dev);
+#endif /* ICRDMA_HW_H*/
--
2.21.0
^ permalink raw reply related
* [RFC 18/20] RDMA/irdma: Update MAINTAINERS file
From: Jeff Kirsher @ 2019-09-26 16:45 UTC (permalink / raw)
To: dledford, jgg, gregkh; +Cc: Shiraz Saleem, netdev, linux-rdma, Mustafa Ismail
In-Reply-To: <20190926164519.10471-1-jeffrey.t.kirsher@intel.com>
From: Shiraz Saleem <shiraz.saleem@intel.com>
Add maintainer entry for irdma driver.
Signed-off-by: Mustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
---
MAINTAINERS | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index 07c374fa1975..dec22ea4ccbc 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8394,7 +8394,15 @@ L: linux-pm@vger.kernel.org
S: Supported
F: drivers/cpufreq/intel_pstate.c
-INTEL RDMA RNIC DRIVER
+INTEL ETHERNET RDMA DRIVER
+M: Mustafa Ismail <mustafa.ismail@intel.com>
+M: Shiraz Saleem <shiraz.saleem@intel.com>
+L: linux-rdma@vger.kernel.org
+S: Supported
+F: drivers/infiniband/hw/irdma/
+F: include/uapi/rdma/irdma-abi.h
+
+INTEL X722 RDMA RNIC DRIVER
M: Faisal Latif <faisal.latif@intel.com>
M: Shiraz Saleem <shiraz.saleem@intel.com>
L: linux-rdma@vger.kernel.org
--
2.21.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox