* Re: netns_id in bpf_sk_lookup_{tcp,udp}
From: Nicolas Dichtel @ 2018-11-19 20:28 UTC (permalink / raw)
To: David Ahern, Joe Stringer; +Cc: netdev, daniel
In-Reply-To: <c9400c6a-67a7-dd89-cf64-4f1b8ece89a2@gmail.com>
Le 19/11/2018 à 20:54, David Ahern a écrit :
> On 11/19/18 12:47 PM, Joe Stringer wrote:
>> On Mon, 19 Nov 2018 at 10:39, David Ahern <dsahern@gmail.com> wrote:
>>>
>>> On 11/19/18 11:36 AM, Joe Stringer wrote:
>>>> Hi David, thanks for pointing this out.
>>>>
>>>> This is more of an oversight through iterations, the runtime lookup
>>>> will fail to find a socket if the netns value is greater than the
>>>> range of a uint32 so I think it would actually make more sense to drop
>>>> the parameter size to u32 rather than u64 so that this would be
>>>> validated at load time rather than silently returning NULL because of
>>>> a bad parameter.
>>>
>>> ok. I was wondering if it was a u64 to handle nsid of 0 which as I
>>> understand it is a legal nsid. If you drop to u32, how do you know when
>>> nsid has been set?
>>
>> I was operating under the assumption that 0 represents the root netns
>> id, and cannot be assigned to another non-root netns.
>>
>> Looking at __peernet2id_alloc(), it seems to me like it attempts to
>> find a netns and if it cannot find one, returns 0, which then leads to
>> a scroll over the idr starting from 0 to INT_MAX to find a legitimate
>> id for the netns, so I think this is a fair assumption?
The NET_ID_ZERO trick is used to manage nsid 0 in net_eq_idr() (idr_for_each()
stops when the callback returns != 0).
>>
>
> Maybe Nicolas can give a definitive answer; as I recall he added the
> NSID option. I have not had time to walk the code. But I do recall
> seeing an id of 0. e.g, on my dev box:
> $ ip netns
> vms (id: 0)
>
> And include/uapi/linux/net_namespace.h shows -1 as not assigned.
Yes, 0 is a valid value and can be assigned to any netns.
nsid are signed 32 bit values. Note that -1 (NETNSA_NSID_NOT_ASSIGNED) is used
by the kernel to express that the nsid is not assigned. It can also be used by
the user to let the kernel chooses a nsid.
$ ip netns add foo
$ ip netns add bar
$ ip netns
bar
foo
$ ip netns set foo 0
$ ip netns set bar auto
$ ip netns
bar (id: 1)
foo (id: 0)
Regards,
Nicolas
^ permalink raw reply
* Re: [PATCHv3 net-next 0/4] sctp: add subscribe per asoc and sockopt SCTP_EVENT
From: David Miller @ 2018-11-19 20:25 UTC (permalink / raw)
To: lucien.xin; +Cc: netdev, linux-sctp, marcelo.leitner, nhorman
In-Reply-To: <cover.1542528309.git.lucien.xin@gmail.com>
From: Xin Long <lucien.xin@gmail.com>
Date: Sun, 18 Nov 2018 16:08:50 +0800
> This patchset mainly adds the Event Subscription sockopt described in
> rfc6525#section-6.2:
Series applied, thanks.
^ permalink raw reply
* Re: [PATCH 00/12 net-next,v2] add flow_rule infrastructure
From: David Miller @ 2018-11-19 20:12 UTC (permalink / raw)
To: pablo
Cc: netdev, thomas.lendacky, f.fainelli, ariel.elior, michael.chan,
santosh, madalin.bucur, yisen.zhuang, salil.mehta,
jeffrey.t.kirsher, tariqt, saeedm, jiri, idosch, jakub.kicinski,
peppe.cavallaro, grygorii.strashko, andrew, vivien.didelot,
alexandre.torgue, joabreu, linux-net-drivers, ganeshgr, ogerlitz
In-Reply-To: <20181119001519.12124-1-pablo@netfilter.org>
From: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Mon, 19 Nov 2018 01:15:07 +0100
> This patchset introduces a kernel intermediate representation (IR) to
> express ACL hardware offloads, as already described in previous RFC and
> v1 patchset [1] [2]. The idea is to normalize the frontend U/APIs to use
> the flow dissectors and the flow actions so drivers can reuse the
> existing TC offload driver codebase - that has been converted to use the
> flow_rule infrastructure.
I'm go to bring up the elephant in the room.
I think the real motivation here is to offload netfilter rules to HW,
and you should be completely honest about that.
^ permalink raw reply
* Re: [PATCH net-next 01/18] net: skb_scrub_packet(): Scrub offload_fwd_mark
From: Ido Schimmel @ 2018-11-19 20:06 UTC (permalink / raw)
To: David Miller
Cc: idosch, netdev, linux-kselftest, shuah, jiri, petrm, roopa, mlxsw
In-Reply-To: <20181119.110745.12820215456503449.davem@davemloft.net>
On Mon, Nov 19, 2018 at 11:07:45AM -0800, David Miller wrote:
> From: Ido Schimmel <idosch@mellanox.com>
> Date: Mon, 19 Nov 2018 16:11:07 +0000
>
> > From: Petr Machata <petrm@mellanox.com>
> >
> > When a packet is trapped and the corresponding SKB marked as
> > already-forwarded, it retains this marking even after it is forwarded
> > across veth links into another bridge. There, since it ingresses the
> > bridge over veth, which doesn't have offload_fwd_mark, it triggers a
> > warning in nbp_switchdev_frame_mark().
> >
> > Then nbp_switchdev_allowed_egress() decides not to allow egress from
> > this bridge through another veth, because the SKB is already marked, and
> > the mark (of 0) of course matches. Thus the packet is incorrectly
> > blocked.
> >
> > Solve by resetting offload_fwd_mark() in skb_scrub_packet(). That
> > function is called from tunnels and also from veth, and thus catches the
> > cases where traffic is forwarded between bridges and transformed in a
> > way that invalidates the marking.
> >
> > Signed-off-by: Petr Machata <petrm@mellanox.com>
> > Suggested-by: Ido Schimmel <idosch@mellanox.com>
> > Signed-off-by: Ido Schimmel <idosch@mellanox.com>
>
> As a bug fix this seems relevant for 'net' instead of 'net-next'.
This seemed really obscure/specific to this selftest so I deemed it
unnecessary for 'net'.
Agree that by the book it is 'net' material, so we'll send it there.
Thanks for the feedback
^ permalink raw reply
* Re: netns_id in bpf_sk_lookup_{tcp,udp}
From: David Ahern @ 2018-11-19 19:54 UTC (permalink / raw)
To: Joe Stringer; +Cc: netdev, daniel, Nicolas Dichtel
In-Reply-To: <CAOftzPjkOp=7V1n1Vpz2VV3A=xy5qKhkpZvhyttjF21BzivNzQ@mail.gmail.com>
On 11/19/18 12:47 PM, Joe Stringer wrote:
> On Mon, 19 Nov 2018 at 10:39, David Ahern <dsahern@gmail.com> wrote:
>>
>> On 11/19/18 11:36 AM, Joe Stringer wrote:
>>> Hi David, thanks for pointing this out.
>>>
>>> This is more of an oversight through iterations, the runtime lookup
>>> will fail to find a socket if the netns value is greater than the
>>> range of a uint32 so I think it would actually make more sense to drop
>>> the parameter size to u32 rather than u64 so that this would be
>>> validated at load time rather than silently returning NULL because of
>>> a bad parameter.
>>
>> ok. I was wondering if it was a u64 to handle nsid of 0 which as I
>> understand it is a legal nsid. If you drop to u32, how do you know when
>> nsid has been set?
>
> I was operating under the assumption that 0 represents the root netns
> id, and cannot be assigned to another non-root netns.
>
> Looking at __peernet2id_alloc(), it seems to me like it attempts to
> find a netns and if it cannot find one, returns 0, which then leads to
> a scroll over the idr starting from 0 to INT_MAX to find a legitimate
> id for the netns, so I think this is a fair assumption?
>
Maybe Nicolas can give a definitive answer; as I recall he added the
NSID option. I have not had time to walk the code. But I do recall
seeing an id of 0. e.g, on my dev box:
$ ip netns
vms (id: 0)
And include/uapi/linux/net_namespace.h shows -1 as not assigned.
^ permalink raw reply
* Re: netns_id in bpf_sk_lookup_{tcp,udp}
From: Joe Stringer @ 2018-11-19 19:47 UTC (permalink / raw)
To: David Ahern; +Cc: Joe Stringer, netdev, daniel
In-Reply-To: <90ae2d6b-049a-90a2-05e5-66700e885b39@gmail.com>
On Mon, 19 Nov 2018 at 10:39, David Ahern <dsahern@gmail.com> wrote:
>
> On 11/19/18 11:36 AM, Joe Stringer wrote:
> > Hi David, thanks for pointing this out.
> >
> > This is more of an oversight through iterations, the runtime lookup
> > will fail to find a socket if the netns value is greater than the
> > range of a uint32 so I think it would actually make more sense to drop
> > the parameter size to u32 rather than u64 so that this would be
> > validated at load time rather than silently returning NULL because of
> > a bad parameter.
>
> ok. I was wondering if it was a u64 to handle nsid of 0 which as I
> understand it is a legal nsid. If you drop to u32, how do you know when
> nsid has been set?
I was operating under the assumption that 0 represents the root netns
id, and cannot be assigned to another non-root netns.
Looking at __peernet2id_alloc(), it seems to me like it attempts to
find a netns and if it cannot find one, returns 0, which then leads to
a scroll over the idr starting from 0 to INT_MAX to find a legitimate
id for the netns, so I think this is a fair assumption?
^ permalink raw reply
* Re: [PATCH] 9p/net: put a lower bound on msize
From: Dominique Martinet @ 2018-11-20 6:03 UTC (permalink / raw)
To: Dominique Martinet
Cc: Eric Van Hensbergen, Latchesar Ionkov, syzkaller-bugs,
v9fs-developer, linux-kernel, netdev
In-Reply-To: <1541407968-31350-1-git-send-email-asmadeus@codewreck.org>
Dominique Martinet wrote on Mon, Nov 05, 2018:
> From: Dominique Martinet <dominique.martinet@cea.fr>
>
> If the requested msize is too small (either from command line argument
> or from the server version reply), we won't get any work done.
> If it's *really* too small, nothing will work, and this got caught by
> syzbot recently (on a new kmem_cache_create_usercopy() call)
>
> Just set a minimum msize to 4k in both code paths, until someone
> complains they have a use-case for a smaller msize.
>
> We need to check in both mount option and server reply individually
> because the msize for the first version request would be unchecked
> with just a global check on clnt->msize.
>
> Reported-by: syzbot+0c1d61e4db7db94102ca@syzkaller.appspotmail.com
> Signed-off-by: Dominique Martinet <dominique.martinet@cea.fr>
> Cc: Eric Van Hensbergen <ericvh@gmail.com>
> Cc: Latchesar Ionkov <lucho@ionkov.net>
> ---
> net/9p/client.c | 21 +++++++++++++++++++++
> 1 file changed, 21 insertions(+)
>
> diff --git a/net/9p/client.c b/net/9p/client.c
> index 2c9a17b9b46b..b1163ebdc622 100644
> --- a/net/9p/client.c
> +++ b/net/9p/client.c
> @@ -181,6 +181,12 @@ static int parse_opts(char *opts, struct p9_client *clnt)
> ret = r;
> continue;
> }
> + if (r < 4096) {
This should read if (option < 4096), amending what I've put in for -next
last week :(
Reviews wanted!
> + p9_debug(P9_DEBUG_ERROR,
> + "msize should be at least 4k\n");
> + ret = -EINVAL;
> + continue;
> + }
> clnt->msize = option;
> break;
> case Opt_trans:
> @@ -983,10 +989,18 @@ static int p9_client_version(struct p9_client *c)
> else if (!strncmp(version, "9P2000", 6))
> c->proto_version = p9_proto_legacy;
> else {
> + p9_debug(P9_DEBUG_ERROR,
> + "server returned an unknown version: %s\n", version);
> err = -EREMOTEIO;
> goto error;
> }
>
> + if (msize < 4096) {
> + p9_debug(P9_DEBUG_ERROR,
> + "server returned a msize < 4096: %d\n", msize);
> + err = -EREMOTEIO;
> + goto error;
> + }
> if (msize < c->msize)
> c->msize = msize;
>
> @@ -1043,6 +1057,13 @@ struct p9_client *p9_client_create(const char *dev_name, char *options)
> if (clnt->msize > clnt->trans_mod->maxsize)
> clnt->msize = clnt->trans_mod->maxsize;
>
> + if (clnt->msize < 4096) {
> + p9_debug(P9_DEBUG_ERROR,
> + "Please specify a msize of at least 4k\n");
> + err = -EINVAL;
> + goto free_client;
> + }
> +
> err = p9_client_version(clnt);
> if (err)
> goto close_trans;
--
Dominique
^ permalink raw reply
* Re: [Patch net] net: invert the check of detecting hardware RX checksum fault
From: Cong Wang @ 2018-11-19 19:25 UTC (permalink / raw)
To: Herbert Xu; +Cc: Linux Kernel Network Developers, Tom Herbert, Eric Dumazet
In-Reply-To: <20181119040154.lik4dqg634gozroj@gondor.apana.org.au>
On Sun, Nov 18, 2018 at 8:02 PM Herbert Xu <herbert@gondor.apana.org.au> wrote:
>
> On Fri, Nov 16, 2018 at 01:32:50PM -0800, Cong Wang wrote:
> >
> > This is true only when there is a skb_checksum_init*() or
> > skb_checksum_validate*() prior to it, it seems not true for
> > nf_ip_checksum() where skb->csum is correctly set to pesudo header
> > checksum but there is no validation of the original skb->csum.
> > So this check should be still inverted there??
> >
> > Or am I still missing anything here?
>
> What do you mean? My copy of nf_ip_checksum seems to be doing the
> right thing as far as verifying CHECKSUM_COMPLETED goes.
Hmm, it calls csum_tcpudp_magic() directly instead of
__skb_checksum_validate_complete(), but it also sets ip_summed
to CHECKSUM_UNNECESSARY:
20 if ((protocol == 0 && !csum_fold(skb->csum)) ||
21 !csum_tcpudp_magic(iph->saddr, iph->daddr,
22 skb->len - dataoff, protocol,
23 skb->csum)) {
24 skb->ip_summed = CHECKSUM_UNNECESSARY;
25 break;
26 }
which means the rx checksum fault won't be triggered.
^ permalink raw reply
* Re: [PATCH net-next] cxgb4/cxgb4vf: Fix mac_hlist initialization and free
From: Leon Romanovsky @ 2018-11-19 19:24 UTC (permalink / raw)
To: Arjun Vynipadath
Cc: netdev, davem, nirranjan, indranil, dt, Casey Leedom,
Ganesh Goudar
In-Reply-To: <20181119100734.29989-1-arjun@chelsio.com>
[-- Attachment #1: Type: text/plain, Size: 3888 bytes --]
On Mon, Nov 19, 2018 at 03:37:34PM +0530, Arjun Vynipadath wrote:
> Null pointer dereference seen when cxgb4vf driver is unloaded
> without bringing up any interfaces, moving mac_hlist initialization
> to driver probe and free the mac_hlist in remove to fix this.
>
> Fixes 'commit 24357e06ba51 ("cxgb4vf: fix memleak in mac_hlist
> initialization")'
>
Please don't break Fixes line (it is needed to various scripts) and no
extra line between Fixes and SOB.
Thanks
> Signed-off-by: Arjun Vynipadath <arjun@chelsio.com>
> Signed-off-by: Casey Leedom <leedom@chelsio.com>
> Signed-off-by: Ganesh Goudar <ganeshgr@chelsio.com>
> ---
> drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c | 19 ++++++++++---------
> drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c | 6 +++---
> 2 files changed, 13 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
> index 956e708..cdd6f48 100644
> --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
> +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
> @@ -2280,8 +2280,6 @@ static int cxgb_up(struct adapter *adap)
> #if IS_ENABLED(CONFIG_IPV6)
> update_clip(adap);
> #endif
> - /* Initialize hash mac addr list*/
> - INIT_LIST_HEAD(&adap->mac_hlist);
> return err;
>
> irq_err:
> @@ -2295,8 +2293,6 @@ static int cxgb_up(struct adapter *adap)
>
> static void cxgb_down(struct adapter *adapter)
> {
> - struct hash_mac_addr *entry, *tmp;
> -
> cancel_work_sync(&adapter->tid_release_task);
> cancel_work_sync(&adapter->db_full_task);
> cancel_work_sync(&adapter->db_drop_task);
> @@ -2306,11 +2302,6 @@ static void cxgb_down(struct adapter *adapter)
> t4_sge_stop(adapter);
> t4_free_sge_resources(adapter);
>
> - list_for_each_entry_safe(entry, tmp, &adapter->mac_hlist, list) {
> - list_del(&entry->list);
> - kfree(entry);
> - }
> -
> adapter->flags &= ~FULL_INIT_DONE;
> }
>
> @@ -5629,6 +5620,9 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
> (is_t5(adapter->params.chip) ? STATMODE_V(0) :
> T6_STATMODE_V(0)));
>
> + /* Initialize hash mac addr list */
> + INIT_LIST_HEAD(&adapter->mac_hlist);
> +
> for_each_port(adapter, i) {
> netdev = alloc_etherdev_mq(sizeof(struct port_info),
> MAX_ETH_QSETS);
> @@ -5907,6 +5901,7 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
> static void remove_one(struct pci_dev *pdev)
> {
> struct adapter *adapter = pci_get_drvdata(pdev);
> + struct hash_mac_addr *entry, *tmp;
>
> if (!adapter) {
> pci_release_regions(pdev);
> @@ -5956,6 +5951,12 @@ static void remove_one(struct pci_dev *pdev)
> if (adapter->num_uld || adapter->num_ofld_uld)
> t4_uld_mem_free(adapter);
> free_some_resources(adapter);
> + list_for_each_entry_safe(entry, tmp, &adapter->mac_hlist,
> + list) {
> + list_del(&entry->list);
> + kfree(entry);
> + }
> +
> #if IS_ENABLED(CONFIG_IPV6)
> t4_cleanup_clip_tbl(adapter);
> #endif
> diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c
> index 8ec503c..8a2ad6b 100644
> --- a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c
> +++ b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c
> @@ -723,9 +723,6 @@ static int adapter_up(struct adapter *adapter)
> if (adapter->flags & USING_MSIX)
> name_msix_vecs(adapter);
>
> - /* Initialize hash mac addr list*/
> - INIT_LIST_HEAD(&adapter->mac_hlist);
> -
> adapter->flags |= FULL_INIT_DONE;
> }
>
> @@ -3038,6 +3035,9 @@ static int cxgb4vf_pci_probe(struct pci_dev *pdev,
> if (err)
> goto err_unmap_bar;
>
> + /* Initialize hash mac addr list */
> + INIT_LIST_HEAD(&adapter->mac_hlist);
> +
> /*
> * Allocate our "adapter ports" and stitch everything together.
> */
> --
> 2.9.5
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply
* [PATCH 4/4] net-next/hinic: fix a bug in rx data flow
From: Xue Chaojing @ 2018-11-20 5:47 UTC (permalink / raw)
To: davem
Cc: linux-kernel, xuechaojing, netdev, wulike1, chiqijun, fy.wang,
tony.qu, luoshaokai
In-Reply-To: <20181120054734.17360-1-xuechaojing@huawei.com>
In rx_alloc_pkts(), there is a loop call of tasklet, which causes
100% cpu utilization, even no packets are being received. This patch
fixes this bug.
Signed-off-by: Xue Chaojing <xuechaojing@huawei.com>
---
drivers/net/ethernet/huawei/hinic/hinic_rx.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_rx.c b/drivers/net/ethernet/huawei/hinic/hinic_rx.c
index 93e8f207f6da..f86f2e693224 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_rx.c
+++ b/drivers/net/ethernet/huawei/hinic/hinic_rx.c
@@ -229,9 +229,9 @@ static int rx_alloc_pkts(struct hinic_rxq *rxq)
wmb(); /* write all the wqes before update PI */
hinic_rq_update(rxq->rq, prod_idx);
+ tasklet_schedule(&rxq->rx_task);
}
- tasklet_schedule(&rxq->rx_task);
return i;
}
--
2.17.1
^ permalink raw reply related
* [PATCH 3/4] net-next/hinic:fix a bug in set mac address
From: Xue Chaojing @ 2018-11-20 5:47 UTC (permalink / raw)
To: davem
Cc: linux-kernel, xuechaojing, netdev, wulike1, chiqijun, fy.wang,
tony.qu, luoshaokai
In-Reply-To: <20181120054734.17360-1-xuechaojing@huawei.com>
In add_mac_addr(), if the MAC address is a muliticast address,
it will not be set, which causes the network card fail to receive
the multicast packet. This patch fixes this bug.
Signed-off-by: Xue Chaojing <xuechaojing@huawei.com>
---
drivers/net/ethernet/huawei/hinic/hinic_main.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_main.c b/drivers/net/ethernet/huawei/hinic/hinic_main.c
index c675077f9774..6d48dc62a44b 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_main.c
+++ b/drivers/net/ethernet/huawei/hinic/hinic_main.c
@@ -600,9 +600,6 @@ static int add_mac_addr(struct net_device *netdev, const u8 *addr)
u16 vid = 0;
int err;
- if (!is_valid_ether_addr(addr))
- return -EADDRNOTAVAIL;
-
netif_info(nic_dev, drv, netdev, "set mac addr = %02x %02x %02x %02x %02x %02x\n",
addr[0], addr[1], addr[2], addr[3], addr[4], addr[5]);
@@ -726,6 +723,7 @@ static void set_rx_mode(struct work_struct *work)
{
struct hinic_rx_mode_work *rx_mode_work = work_to_rx_mode_work(work);
struct hinic_dev *nic_dev = rx_mode_work_to_nic_dev(rx_mode_work);
+ struct netdev_hw_addr *ha;
netif_info(nic_dev, drv, nic_dev->netdev, "set rx mode work\n");
@@ -733,6 +731,9 @@ static void set_rx_mode(struct work_struct *work)
__dev_uc_sync(nic_dev->netdev, add_mac_addr, remove_mac_addr);
__dev_mc_sync(nic_dev->netdev, add_mac_addr, remove_mac_addr);
+
+ netdev_for_each_mc_addr(ha, nic_dev->netdev)
+ add_mac_addr(nic_dev->netdev, ha->addr);
}
static void hinic_set_rx_mode(struct net_device *netdev)
--
2.17.1
^ permalink raw reply related
* [PATCH 2/4] net-next/hinic:add rx checksum offload for HiNIC
From: Xue Chaojing @ 2018-11-20 5:47 UTC (permalink / raw)
To: davem
Cc: linux-kernel, xuechaojing, netdev, wulike1, chiqijun, fy.wang,
tony.qu, luoshaokai
In-Reply-To: <20181120054734.17360-1-xuechaojing@huawei.com>
In order to improve performance, this patch adds rx checksum offload
for the HiNIC driver. Performance test(Iperf) shows more than 80%
improvement in TCP streams.
Signed-off-by: Xue Chaojing <xuechaojing@huawei.com>
---
.../net/ethernet/huawei/hinic/hinic_hw_dev.h | 2 ++
.../net/ethernet/huawei/hinic/hinic_hw_wqe.h | 4 +++
.../net/ethernet/huawei/hinic/hinic_main.c | 7 ++++-
.../net/ethernet/huawei/hinic/hinic_port.c | 28 +++++++++++++++++++
.../net/ethernet/huawei/hinic/hinic_port.h | 10 +++++++
drivers/net/ethernet/huawei/hinic/hinic_rx.c | 24 ++++++++++++++++
drivers/net/ethernet/huawei/hinic/hinic_rx.h | 4 +++
7 files changed, 78 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_hw_dev.h b/drivers/net/ethernet/huawei/hinic/hinic_hw_dev.h
index 097b5502603f..d1a7d2522d82 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_hw_dev.h
+++ b/drivers/net/ethernet/huawei/hinic/hinic_hw_dev.h
@@ -50,6 +50,8 @@ enum hinic_port_cmd {
HINIC_PORT_CMD_GET_LINK_STATE = 24,
+ HINIC_PORT_CMD_SET_RX_CSUM = 26,
+
HINIC_PORT_CMD_SET_PORT_STATE = 41,
HINIC_PORT_CMD_FWCTXT_INIT = 69,
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_hw_wqe.h b/drivers/net/ethernet/huawei/hinic/hinic_hw_wqe.h
index 9754d6ed5f4a..138941527872 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_hw_wqe.h
+++ b/drivers/net/ethernet/huawei/hinic/hinic_hw_wqe.h
@@ -170,6 +170,10 @@
#define HINIC_RQ_CQE_STATUS_RXDONE_MASK 0x1
+#define HINIC_RQ_CQE_STATUS_CSUM_ERR_SHIFT 0
+
+#define HINIC_RQ_CQE_STATUS_CSUM_ERR_MASK 0xFFFFU
+
#define HINIC_RQ_CQE_STATUS_GET(val, member) \
(((val) >> HINIC_RQ_CQE_STATUS_##member##_SHIFT) & \
HINIC_RQ_CQE_STATUS_##member##_MASK)
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_main.c b/drivers/net/ethernet/huawei/hinic/hinic_main.c
index fdf2bdb6b0d0..c675077f9774 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_main.c
+++ b/drivers/net/ethernet/huawei/hinic/hinic_main.c
@@ -806,7 +806,8 @@ static const struct net_device_ops hinic_netdev_ops = {
static void netdev_features_init(struct net_device *netdev)
{
netdev->hw_features = NETIF_F_SG | NETIF_F_HIGHDMA | NETIF_F_IP_CSUM |
- NETIF_F_IPV6_CSUM | NETIF_F_TSO | NETIF_F_TSO6;
+ NETIF_F_IPV6_CSUM | NETIF_F_TSO | NETIF_F_TSO6 |
+ NETIF_F_RXCSUM;
netdev->vlan_features = netdev->hw_features;
@@ -869,12 +870,16 @@ static int set_features(struct hinic_dev *nic_dev,
netdev_features_t features, bool force_change)
{
netdev_features_t changed = force_change ? ~0 : pre_features ^ features;
+ u32 csum_en = HINIC_RX_CSUM_OFFLOAD_EN;
int err = 0;
if (changed & NETIF_F_TSO)
err = hinic_port_set_tso(nic_dev, (features & NETIF_F_TSO) ?
HINIC_TSO_ENABLE : HINIC_TSO_DISABLE);
+ if (changed & NETIF_F_RXCSUM)
+ err = hinic_set_rx_csum_offload(nic_dev, csum_en);
+
return err;
}
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_port.c b/drivers/net/ethernet/huawei/hinic/hinic_port.c
index 7575a7d3bd9f..e9f76e904610 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_port.c
+++ b/drivers/net/ethernet/huawei/hinic/hinic_port.c
@@ -409,3 +409,31 @@ int hinic_port_set_tso(struct hinic_dev *nic_dev, enum hinic_tso_state state)
return 0;
}
+
+int hinic_set_rx_csum_offload(struct hinic_dev *nic_dev, u32 en)
+{
+ struct hinic_checksum_offload rx_csum_cfg = {0};
+ struct hinic_hwdev *hwdev = nic_dev->hwdev;
+ struct hinic_hwif *hwif = hwdev->hwif;
+ struct pci_dev *pdev = hwif->pdev;
+ u16 out_size;
+ int err;
+
+ if (!hwdev)
+ return -EINVAL;
+
+ rx_csum_cfg.func_id = HINIC_HWIF_FUNC_IDX(hwif);
+ rx_csum_cfg.rx_csum_offload = en;
+
+ err = hinic_port_msg_cmd(hwdev, HINIC_PORT_CMD_SET_RX_CSUM,
+ &rx_csum_cfg, sizeof(rx_csum_cfg),
+ &rx_csum_cfg, &out_size);
+ if (err || !out_size || rx_csum_cfg.status) {
+ dev_err(&pdev->dev,
+ "Failed to set rx csum offload, ret = %d\n",
+ rx_csum_cfg.status);
+ return -EINVAL;
+ }
+
+ return 0;
+}
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_port.h b/drivers/net/ethernet/huawei/hinic/hinic_port.h
index f6e3220fe28f..02d896eed455 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_port.h
+++ b/drivers/net/ethernet/huawei/hinic/hinic_port.h
@@ -183,6 +183,15 @@ struct hinic_tso_config {
u8 resv2[3];
};
+struct hinic_checksum_offload {
+ u8 status;
+ u8 version;
+ u8 rsvd0[6];
+
+ u16 func_id;
+ u16 rsvd1;
+ u32 rx_csum_offload;
+};
int hinic_port_add_mac(struct hinic_dev *nic_dev, const u8 *addr,
u16 vlan_id);
@@ -213,4 +222,5 @@ int hinic_port_get_cap(struct hinic_dev *nic_dev,
int hinic_port_set_tso(struct hinic_dev *nic_dev, enum hinic_tso_state state);
+int hinic_set_rx_csum_offload(struct hinic_dev *nic_dev, u32 en);
#endif
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_rx.c b/drivers/net/ethernet/huawei/hinic/hinic_rx.c
index 4c0f7eda1166..93e8f207f6da 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_rx.c
+++ b/drivers/net/ethernet/huawei/hinic/hinic_rx.c
@@ -89,6 +89,28 @@ static void rxq_stats_init(struct hinic_rxq *rxq)
hinic_rxq_clean_stats(rxq);
}
+static void rx_csum(struct hinic_rxq *rxq, u16 cons_idx,
+ struct sk_buff *skb)
+{
+ struct net_device *netdev = rxq->netdev;
+ struct hinic_rq_cqe *cqe;
+ struct hinic_rq *rq;
+ u32 csum_err;
+ u32 status;
+
+ rq = rxq->rq;
+ cqe = rq->cqe[cons_idx];
+ status = be32_to_cpu(cqe->status);
+ csum_err = HINIC_RQ_CQE_STATUS_GET(status, CSUM_ERR);
+
+ if (!(netdev->features & NETIF_F_RXCSUM))
+ return;
+
+ if (!csum_err)
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+ else
+ skb->ip_summed = CHECKSUM_NONE;
+}
/**
* rx_alloc_skb - allocate skb and map it to dma address
* @rxq: rx queue
@@ -328,6 +350,8 @@ static int rxq_recv(struct hinic_rxq *rxq, int budget)
rx_unmap_skb(rxq, hinic_sge_to_dma(&sge));
+ rx_csum(rxq, ci, skb);
+
prefetch(skb->data);
pkt_len = sge.len;
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_rx.h b/drivers/net/ethernet/huawei/hinic/hinic_rx.h
index 27c9af4b1c12..ab3fabab91b2 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_rx.h
+++ b/drivers/net/ethernet/huawei/hinic/hinic_rx.h
@@ -23,6 +23,10 @@
#include "hinic_hw_qp.h"
+#define HINIC_RX_CSUM_OFFLOAD_EN 0xFFF
+#define HINIC_RX_CSUM_HW_CHECK_NONE BIT(7)
+#define HINIC_RX_CSUM_IPSU_OTHER_ERR BIT(8)
+
struct hinic_rxq_stats {
u64 pkts;
u64 bytes;
--
2.17.1
^ permalink raw reply related
* [PATCH 1/4] net-next/hinic:replace multiply and division operators
From: Xue Chaojing @ 2018-11-20 5:47 UTC (permalink / raw)
To: davem
Cc: linux-kernel, xuechaojing, netdev, wulike1, chiqijun, fy.wang,
tony.qu, luoshaokai
To improve performance, this patch uses bit operations to replace
multiply and division operators.
Signed-off-by: Xue Chaojing <xuechaojing@huawei.com>
---
.../net/ethernet/huawei/hinic/hinic_hw_wq.c | 55 ++++++++++++-------
.../net/ethernet/huawei/hinic/hinic_hw_wq.h | 3 +-
2 files changed, 38 insertions(+), 20 deletions(-)
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_hw_wq.c b/drivers/net/ethernet/huawei/hinic/hinic_hw_wq.c
index f92f1bf3901a..1dfa7eb05c10 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_hw_wq.c
+++ b/drivers/net/ethernet/huawei/hinic/hinic_hw_wq.c
@@ -74,12 +74,6 @@
((void *)((cmdq_pages)->shadow_page_vaddr) \
+ (wq)->block_idx * CMDQ_BLOCK_SIZE)
-#define WQE_PAGE_OFF(wq, idx) (((idx) & ((wq)->num_wqebbs_per_page - 1)) * \
- (wq)->wqebb_size)
-
-#define WQE_PAGE_NUM(wq, idx) (((idx) / ((wq)->num_wqebbs_per_page)) \
- & ((wq)->num_q_pages - 1))
-
#define WQ_PAGE_ADDR(wq, idx) \
((wq)->shadow_block_vaddr[WQE_PAGE_NUM(wq, idx)])
@@ -93,6 +87,17 @@
(((unsigned long)(wqe) - (unsigned long)(wq)->shadow_wqe) \
/ (wq)->max_wqe_size)
+static inline int WQE_PAGE_OFF(struct hinic_wq *wq, u16 idx)
+{
+ return (((idx) & ((wq)->num_wqebbs_per_page - 1))
+ << (wq)->wqebb_size_shift);
+}
+
+static inline int WQE_PAGE_NUM(struct hinic_wq *wq, u16 idx)
+{
+ return (((idx) >> ((wq)->wqebbs_per_page_shift))
+ & ((wq)->num_q_pages - 1));
+}
/**
* queue_alloc_page - allocate page for Queue
* @hwif: HW interface for allocating DMA
@@ -513,10 +518,11 @@ int hinic_wq_allocate(struct hinic_wqs *wqs, struct hinic_wq *wq,
struct hinic_hwif *hwif = wqs->hwif;
struct pci_dev *pdev = hwif->pdev;
u16 num_wqebbs_per_page;
+ u16 wqebb_size_shift;
int err;
- if (wqebb_size == 0) {
- dev_err(&pdev->dev, "wqebb_size must be > 0\n");
+ if (!is_power_of_2(wqebb_size)) {
+ dev_err(&pdev->dev, "wqebb_size must be power of 2\n");
return -EINVAL;
}
@@ -530,9 +536,11 @@ int hinic_wq_allocate(struct hinic_wqs *wqs, struct hinic_wq *wq,
return -EINVAL;
}
- num_wqebbs_per_page = ALIGN(wq_page_size, wqebb_size) / wqebb_size;
+ wqebb_size_shift = ilog2(wqebb_size);
+ num_wqebbs_per_page = ALIGN(wq_page_size, wqebb_size)
+ >> wqebb_size_shift;
- if (num_wqebbs_per_page & (num_wqebbs_per_page - 1)) {
+ if (!is_power_of_2(num_wqebbs_per_page)) {
dev_err(&pdev->dev, "num wqebbs per page must be power of 2\n");
return -EINVAL;
}
@@ -550,7 +558,8 @@ int hinic_wq_allocate(struct hinic_wqs *wqs, struct hinic_wq *wq,
wq->q_depth = q_depth;
wq->max_wqe_size = max_wqe_size;
wq->num_wqebbs_per_page = num_wqebbs_per_page;
-
+ wq->wqebbs_per_page_shift = ilog2(num_wqebbs_per_page);
+ wq->wqebb_size_shift = wqebb_size_shift;
wq->block_vaddr = WQ_BASE_VADDR(wqs, wq);
wq->shadow_block_vaddr = WQ_BASE_ADDR(wqs, wq);
wq->block_paddr = WQ_BASE_PADDR(wqs, wq);
@@ -604,11 +613,13 @@ int hinic_wqs_cmdq_alloc(struct hinic_cmdq_pages *cmdq_pages,
u16 q_depth, u16 max_wqe_size)
{
struct pci_dev *pdev = hwif->pdev;
+ u16 num_wqebbs_per_page_shift;
u16 num_wqebbs_per_page;
+ u16 wqebb_size_shift;
int i, j, err = -ENOMEM;
- if (wqebb_size == 0) {
- dev_err(&pdev->dev, "wqebb_size must be > 0\n");
+ if (!is_power_of_2(wqebb_size)) {
+ dev_err(&pdev->dev, "wqebb_size must be power of 2\n");
return -EINVAL;
}
@@ -622,9 +633,11 @@ int hinic_wqs_cmdq_alloc(struct hinic_cmdq_pages *cmdq_pages,
return -EINVAL;
}
- num_wqebbs_per_page = ALIGN(wq_page_size, wqebb_size) / wqebb_size;
+ wqebb_size_shift = ilog2(wqebb_size);
+ num_wqebbs_per_page = ALIGN(wq_page_size, wqebb_size)
+ >> wqebb_size_shift;
- if (num_wqebbs_per_page & (num_wqebbs_per_page - 1)) {
+ if (!is_power_of_2(num_wqebbs_per_page)) {
dev_err(&pdev->dev, "num wqebbs per page must be power of 2\n");
return -EINVAL;
}
@@ -636,6 +649,7 @@ int hinic_wqs_cmdq_alloc(struct hinic_cmdq_pages *cmdq_pages,
dev_err(&pdev->dev, "Failed to allocate CMDQ page\n");
return err;
}
+ num_wqebbs_per_page_shift = ilog2(num_wqebbs_per_page);
for (i = 0; i < cmdq_blocks; i++) {
wq[i].hwif = hwif;
@@ -647,7 +661,8 @@ int hinic_wqs_cmdq_alloc(struct hinic_cmdq_pages *cmdq_pages,
wq[i].q_depth = q_depth;
wq[i].max_wqe_size = max_wqe_size;
wq[i].num_wqebbs_per_page = num_wqebbs_per_page;
-
+ wq[i].wqebbs_per_page_shift = num_wqebbs_per_page_shift;
+ wq[i].wqebb_size_shift = wqebb_size_shift;
wq[i].block_vaddr = CMDQ_BASE_VADDR(cmdq_pages, &wq[i]);
wq[i].shadow_block_vaddr = CMDQ_BASE_ADDR(cmdq_pages, &wq[i]);
wq[i].block_paddr = CMDQ_BASE_PADDR(cmdq_pages, &wq[i]);
@@ -741,7 +756,7 @@ struct hinic_hw_wqe *hinic_get_wqe(struct hinic_wq *wq, unsigned int wqe_size,
*prod_idx = MASKED_WQE_IDX(wq, atomic_read(&wq->prod_idx));
- num_wqebbs = ALIGN(wqe_size, wq->wqebb_size) / wq->wqebb_size;
+ num_wqebbs = ALIGN(wqe_size, wq->wqebb_size) >> wq->wqebb_size_shift;
if (atomic_sub_return(num_wqebbs, &wq->delta) <= 0) {
atomic_add(num_wqebbs, &wq->delta);
@@ -795,7 +810,8 @@ void hinic_return_wqe(struct hinic_wq *wq, unsigned int wqe_size)
**/
void hinic_put_wqe(struct hinic_wq *wq, unsigned int wqe_size)
{
- int num_wqebbs = ALIGN(wqe_size, wq->wqebb_size) / wq->wqebb_size;
+ int num_wqebbs = ALIGN(wqe_size, wq->wqebb_size)
+ >> wq->wqebb_size_shift;
atomic_add(num_wqebbs, &wq->cons_idx);
@@ -813,7 +829,8 @@ void hinic_put_wqe(struct hinic_wq *wq, unsigned int wqe_size)
struct hinic_hw_wqe *hinic_read_wqe(struct hinic_wq *wq, unsigned int wqe_size,
u16 *cons_idx)
{
- int num_wqebbs = ALIGN(wqe_size, wq->wqebb_size) / wq->wqebb_size;
+ int num_wqebbs = ALIGN(wqe_size, wq->wqebb_size)
+ >> wq->wqebb_size_shift;
u16 curr_cons_idx, end_cons_idx;
int curr_pg, end_pg;
diff --git a/drivers/net/ethernet/huawei/hinic/hinic_hw_wq.h b/drivers/net/ethernet/huawei/hinic/hinic_hw_wq.h
index 9b66545ba563..0a936cd6709b 100644
--- a/drivers/net/ethernet/huawei/hinic/hinic_hw_wq.h
+++ b/drivers/net/ethernet/huawei/hinic/hinic_hw_wq.h
@@ -39,7 +39,8 @@ struct hinic_wq {
u16 q_depth;
u16 max_wqe_size;
u16 num_wqebbs_per_page;
-
+ u16 wqebbs_per_page_shift;
+ u16 wqebb_size_shift;
/* The addresses are 64 bit in the HW */
u64 block_paddr;
void **shadow_block_vaddr;
--
2.17.1
^ permalink raw reply related
* Re: [PATCH net-next] MAINTAINERS: Add myself as third phylib maintainer
From: David Miller @ 2018-11-19 19:14 UTC (permalink / raw)
To: f.fainelli; +Cc: hkallweit1, andrew, netdev
In-Reply-To: <aa401ce8-9dd2-07a6-6c3e-791f617e1b38@gmail.com>
From: Florian Fainelli <f.fainelli@gmail.com>
Date: Mon, 19 Nov 2018 11:01:23 -0800
> On 11/18/18 11:01 PM, Heiner Kallweit wrote:
>> Add myself as third phylib maintainer.
>>
>> Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
>
> Acked-by: Florian Fainelli <f.fainelli@gmail.com>
>
> This could probably go to "net" (not a functional code change).
Yeah that's where I will apply this right now.
Thanks.
^ permalink raw reply
* [PATCH] net: faraday: ftmac100: remove netif_running(netdev) check before disabling interrupts
From: Vincent Chen @ 2018-11-20 5:37 UTC (permalink / raw)
To: davem, gregkh, rdunlap, yuehaibing; +Cc: vincentc, netdev, linux-kernel
In the original ftmac100_interrupt(), the interrupts are only disabled when
the condition "netif_running(netdev)" is true. However, this condition
causes kerenl hang in the following case. When the user requests to
disable the network device, kernel will clear the bit __LINK_STATE_START
from the dev->state and then call the driver's ndo_stop function. Network
device interrupts are not blocked during this process. If an interrupt
occurs between clearing __LINK_STATE_START and stopping network device,
kernel cannot disable the interrupts due to the condition
"netif_running(netdev)" in the ISR. Hence, kernel will hang due to the
continuous interruption of the network device.
In order to solve the above problem, the interrupts of the network device
should always be disabled in the ISR without being restricted by the
condition "netif_running(netdev)".
Signed-off-by: Vincent Chen <vincentc@andestech.com>
---
drivers/net/ethernet/faraday/ftmac100.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/faraday/ftmac100.c b/drivers/net/ethernet/faraday/ftmac100.c
index 21d7660..22dcf4c 100644
--- a/drivers/net/ethernet/faraday/ftmac100.c
+++ b/drivers/net/ethernet/faraday/ftmac100.c
@@ -875,9 +875,9 @@ static irqreturn_t ftmac100_interrupt(int irq, void *dev_id)
struct net_device *netdev = dev_id;
struct ftmac100 *priv = netdev_priv(netdev);
+ /* Disable interrupts for polling */
+ ftmac100_disable_all_int(priv);
if (likely(netif_running(netdev))) {
- /* Disable interrupts for polling */
- ftmac100_disable_all_int(priv);
napi_schedule(&priv->napi);
}
--
1.7.1
^ permalink raw reply related
* Re: [PATCH net-next 01/18] net: skb_scrub_packet(): Scrub offload_fwd_mark
From: David Miller @ 2018-11-19 19:07 UTC (permalink / raw)
To: idosch; +Cc: netdev, linux-kselftest, shuah, jiri, petrm, roopa, mlxsw
In-Reply-To: <20181119161006.5405-2-idosch@mellanox.com>
From: Ido Schimmel <idosch@mellanox.com>
Date: Mon, 19 Nov 2018 16:11:07 +0000
> From: Petr Machata <petrm@mellanox.com>
>
> When a packet is trapped and the corresponding SKB marked as
> already-forwarded, it retains this marking even after it is forwarded
> across veth links into another bridge. There, since it ingresses the
> bridge over veth, which doesn't have offload_fwd_mark, it triggers a
> warning in nbp_switchdev_frame_mark().
>
> Then nbp_switchdev_allowed_egress() decides not to allow egress from
> this bridge through another veth, because the SKB is already marked, and
> the mark (of 0) of course matches. Thus the packet is incorrectly
> blocked.
>
> Solve by resetting offload_fwd_mark() in skb_scrub_packet(). That
> function is called from tunnels and also from veth, and thus catches the
> cases where traffic is forwarded between bridges and transformed in a
> way that invalidates the marking.
>
> Signed-off-by: Petr Machata <petrm@mellanox.com>
> Suggested-by: Ido Schimmel <idosch@mellanox.com>
> Signed-off-by: Ido Schimmel <idosch@mellanox.com>
As a bug fix this seems relevant for 'net' instead of 'net-next'.
^ permalink raw reply
* Re: [PATCH 00/10] add flow_rule infrastructure
From: Florian Fainelli @ 2018-11-19 19:04 UTC (permalink / raw)
To: Jiri Pirko, Chopra, Manish
Cc: Pablo Neira Ayuso, netdev@vger.kernel.org, davem@davemloft.net,
thomas.lendacky@amd.com, Elior, Ariel, michael.chan@broadcom.com,
santosh@chelsio.com, madalin.bucur@nxp.com,
yisen.zhuang@huawei.com, salil.mehta@huawei.com,
jeffrey.t.kirsher@intel.com, tariqt@mellanox.com,
saeedm@mellanox.com, jiri@mellanox.com, idosch@mellanox.com,
jakub.kicinski@netronome.com
In-Reply-To: <20181119095758.GA2110@nanopsycho>
On 11/19/18 1:57 AM, Jiri Pirko wrote:
> Mon, Nov 19, 2018 at 10:19:28AM CET, Manish.Chopra@cavium.com wrote:
>>> -----Original Message-----
>>> From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org> On
>>> Behalf Of Pablo Neira Ayuso
>>> Sent: Friday, November 16, 2018 7:11 AM
>>> To: netdev@vger.kernel.org
>>> Cc: davem@davemloft.net; thomas.lendacky@amd.com;
>>> f.fainelli@gmail.com; Elior, Ariel <Ariel.Elior@cavium.com>;
>>> michael.chan@broadcom.com; santosh@chelsio.com;
>>> madalin.bucur@nxp.com; yisen.zhuang@huawei.com;
>>> salil.mehta@huawei.com; jeffrey.t.kirsher@intel.com; tariqt@mellanox.com;
>>> saeedm@mellanox.com; jiri@mellanox.com; idosch@mellanox.com;
>>> jakub.kicinski@netronome.com; peppe.cavallaro@st.com;
>>> grygorii.strashko@ti.com; andrew@lunn.ch;
>>> vivien.didelot@savoirfairelinux.com; alexandre.torgue@st.com;
>>> joabreu@synopsys.com; linux-net-drivers@solarflare.com;
>>> ganeshgr@chelsio.com
>>> Subject: [PATCH 00/10] add flow_rule infrastructure
>>>
>>> External Email
>>>
>>> This patchset introduces a kernel intermediate representation (IR) to express
>>> ACL hardware offloads, this is heavily based on the existing flow dissector
>>> infrastructure and the TC actions. This IR can be used by different frontend
>>> ACL interfaces such as ethtool_rxnfc and tc to represent ACL hardware
>>> offloads. Main goal is to simplify the development of ACL hardware offloads
>>> for the existing frontend interfaces, the idea is that driver developers do not
>>> need to add one specific parser for each ACL frontend, instead each frontend
>>> can just generate this flow_rule IR and pass it to drivers to populate the
>>> hardware IR.
>>>
>>> . ethtool_rxnfc tc
>>> | (ioctl) (netlink)
>>> | | | translate native
>>> Frontend | | | interface representation
>>> | | | to flow_rule IR
>>> | | |
>>> . \/ \/
>>> . flow_rule IR
>>> | |
>>> Drivers | | parsing of flow_rule IR
>>> | | to populate hardware IR
>>> | \/
>>> . hardware IR (driver)
>>>
>>> For design and implementation details, please have a look at:
>>>
>>> https://lwn.net/Articles/766695/
>>>
>>> As an example, with this patchset, it should be possible to simplify the
>>> existing net/qede driver which already has two parsers to populate the
>>> hardware IR, one for ethtool_rxnfc interface and another for tc.
>>>
>>> This batch is composed of 10 patches:
>>>
>>> Patch #1 adds the flow_match structure, this includes the
>>> flow_rule_match_key() interface to check for existing selectors
>>> that are in used in the rule and the flow_rule_match_*()
>>> functions to fetch the selector value and the mask. This
>>> also introduces the initial flow_rule structure skeleton to
>>> avoid a follow up patch that would update the same LoCs.
>>>
>>> Patch #2 makes changes to packet edit parser of mlx5e driver, to prepare
>>> introduction of the new flow_action to mangle packets.
>>>
>>> Patch #3 Introduce flow_action infrastructure. This infrastructure is
>>> based on the TC actions. Patch #8 extends it so it also
>>> supports two new actions that are only available through the
>>> ethtool_rxnfc interface.
>>>
>>> Patch #4 Add function to translate TC action to flow_action from
>>> cls_flower.
>>>
>>> Patch #5 Add infrastructure to fetch statistics into container structure
>>> and synchronize them to TC actions from cls_flower. Another
>>> preparation patch before patch #7, so we can stop exposing the
>>> TC action native layout to the drivers.
>>>
>>> Patch #6 Use flow_action infrastructure from drivers.
>>>
>>> Patch #7 Do not expose TC actions to drivers anymore, now that drivers
>>> have been converted to use the flow_action infrastructure after
>>> patch #5.
>>>
>>> Patch #8 Support to wake-up-on-lan and queue actions for the flow_action
>>> infrastructure, two actions supported by NICs. This is used by
>>> the ethtool_rx_flow interface.
>>>
>>> Patch #9 Add a function to translate from ethtool_rx_flow_spec structure
>>> to the flow_action structure. This is a simple enough for its
>>> first client: the ethtool_rxnfc interface in the bcm_sf2 driver.
>>>
>>> Patch #10 Update bcm_sf2 to use this new translator function and
>>> update codebase to configure hardware IR using the
>>> flow_action representation. This will allow later development
>>> of cls_flower using the same codebase from the driver.
>>>
>>> This patchset has passed here functional tests of the codepath that generates
>>> the flow_rule structure and the functions to implement the parsers that
>>> populate the hardware IR.
>>>
>>> Thanks.
>>>
>>> Pablo Neira Ayuso (10):
>>> flow_dissector: add flow_rule and flow_match structures and use them
>>> net/mlx5e: support for two independent packet edit actions
>>> flow_dissector: add flow action infrastructure
>>> cls_api: add translator to flow_action representation
>>> cls_flower: add statistics retrieval infrastructure and use it
>>> drivers: net: use flow action infrastructure
>>> cls_flower: don't expose TC actions to drivers anymore
>>> flow_dissector: add wake-up-on-lan and queue to flow_action
>>> flow_dissector: add basic ethtool_rx_flow_spec to flow_rule structure
>>> translator
>>> dsa: bcm_sf2: use flow_rule infrastructure
>>>
>>> drivers/net/dsa/bcm_sf2_cfp.c | 103 ++-
>>> drivers/net/ethernet/broadcom/bnxt/bnxt_tc.c | 252 +++----
>>> .../net/ethernet/chelsio/cxgb4/cxgb4_tc_flower.c | 450 ++++++-------
>>> drivers/net/ethernet/intel/i40e/i40e_main.c | 178 ++---
>>> drivers/net/ethernet/intel/iavf/iavf_main.c | 195 +++---
>>> drivers/net/ethernet/intel/igb/igb_main.c | 64 +-
>>> drivers/net/ethernet/mellanox/mlx5/core/en_tc.c | 743 ++++++++++--------
>>> ---
>>> drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c | 2 +-
>>> .../net/ethernet/mellanox/mlxsw/spectrum_flower.c | 259 ++++---
>>> drivers/net/ethernet/netronome/nfp/flower/action.c | 196 +++---
>>> drivers/net/ethernet/netronome/nfp/flower/match.c | 416 ++++++------
>>> .../net/ethernet/netronome/nfp/flower/offload.c | 151 ++---
>>> drivers/net/ethernet/qlogic/qede/qede_filter.c | 93 ++-
>>> include/net/flow_dissector.h | 185 +++++
>>> include/net/pkt_cls.h | 23 +-
>>> net/core/flow_dissector.c | 341 ++++++++++
>>> net/sched/cls_api.c | 113 ++++
>>> net/sched/cls_flower.c | 42 +-
>>> 18 files changed, 2120 insertions(+), 1686 deletions(-)
>>>
>>> --
>>> 2.11.0
>>
>> Today, we have different kernel interfaces(aRFS/ethtool/tc) to do flow offloading.
>> One big question arises in my mind that why we need these all different interfaces to do flow offloading and
>> why can't they just be combined into a single kernel interface somehow.
>> Having these flow offloading done through various hooks makes driver implementation really complex.
>>
>> Specially when looking at ethtool/tc flows most of the use cases seems common to me -
>>
>> 1. Drop action can be done on tuples from both tc/ethtool flows.
>> 2. Flow redirection is done from both ethtool/tc
>> (May be with just slight difference ? as Ethtool does steering on PF and VF/VF-queues
>> but tc doesn't have redirection on queues which can be extended over tc)
We discussed several times before with Jiri at various conferences, and
I unfortunately don't have the time to look into doing it, but if
cls_flower gained support for ethtool_rx_flow_spec::location, we should
be good in terms of feature parity. Pablo also identified the
RX_CLS_FLOW_WAKE as something that needs to be supported, but this is a
simple thing that could be implemented as a special tc action (should?)
>> 3. Could be more such common cases which could be the reason to combine them through single interface ?
Main reason IMHO for going to an unified interface is that ultimately
the piece of HW that implements filtering/action is shared across all
programming interfaces. If they all have about roughly the same feature
set, yet some slight feature deviation or semantic differences, this
gets confusing for driver writers and users.
>
> Sure you can make one interface feature complete. But that does not mean
> you can remove the others. At least not easily. You woul break UAPI.
Indeed, though we can have a couple of strategies to unify and later
deprecate them:
1) Go after each application and make them use the chosen interface,
e.g: the tc/cls_flower UAPI, this does not really scale well because
while we can easily get mainstream reference applications converted such
as ethtool, iptables, xtables, we don't know what people have designed
around the UAPI interface, and usually this also does not mean bindings
to different languages (e.g: ethtool python) would be converted. Because
we have the application converted to the native interface, there is no
loss in translation or performance penalty from going to the kernel's IR.
2) We convert the non-chosen interfaces (e.g: ethtool, *tables) from
within the kernel through the chosen IR and programming interface, that
way we can easily identify which applications are still using the
deprecated interface and issue an appropriate warning in kernel version
XZY for instance. The drawback is that you have a translation so that
could mean loss of information and/or performance penalty if rule
insertion/deletion is a hot path (which is unfortunately likely the case).
--
Florian
^ permalink raw reply
* Re: [RFCv3 PATCH 1/6] uacce: Add documents for WarpDrive/uacce
From: Leon Romanovsky @ 2018-11-19 19:02 UTC (permalink / raw)
To: Jerome Glisse
Cc: Jason Gunthorpe, Kenneth Lee, Tim Sell, linux-doc,
Alexander Shishkin, Zaibo Xu, zhangfei.gao, linuxarm,
haojian.zhuang, Christoph Lameter, Hao Fang, Gavin Schenk,
RDMA mailing list, Zhou Wang, Doug Ledford, Uwe Kleine-König,
David Kershner, Kenneth Lee, Johan Hovold, Cyrille Pitchen
In-Reply-To: <20181119184215.GB4593@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 2514 bytes --]
On Mon, Nov 19, 2018 at 01:42:16PM -0500, Jerome Glisse wrote:
> On Mon, Nov 19, 2018 at 11:27:52AM -0700, Jason Gunthorpe wrote:
> > On Mon, Nov 19, 2018 at 11:48:54AM -0500, Jerome Glisse wrote:
> >
> > > Just to comment on this, any infiniband driver which use umem and do
> > > not have ODP (here ODP for me means listening to mmu notifier so all
> > > infiniband driver except mlx5) will be affected by same issue AFAICT.
> > >
> > > AFAICT there is no special thing happening after fork() inside any of
> > > those driver. So if parent create a umem mr before fork() and program
> > > hardware with it then after fork() the parent might start using new
> > > page for the umem range while the old memory is use by the child. The
> > > reverse is also true (parent using old memory and child new memory)
> > > bottom line you can not predict which memory the child or the parent
> > > will use for the range after fork().
> > >
> > > So no matter what you consider the child or the parent, what the hw
> > > will use for the mr is unlikely to match what the CPU use for the
> > > same virtual address. In other word:
> > >
> > > Before fork:
> > > CPU parent: virtual addr ptr1 -> physical address = 0xCAFE
> > > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE
> > >
> > > Case 1:
> > > CPU parent: virtual addr ptr1 -> physical address = 0xCAFE
> > > CPU child: virtual addr ptr1 -> physical address = 0xDEAD
> > > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE
> > >
> > > Case 2:
> > > CPU parent: virtual addr ptr1 -> physical address = 0xBEEF
> > > CPU child: virtual addr ptr1 -> physical address = 0xCAFE
> > > HARDWARE: virtual addr ptr1 -> physical address = 0xCAFE
> >
> > IIRC this is solved in IB by automatically calling
> > madvise(MADV_DONTFORK) before creating the MR.
> >
> > MADV_DONTFORK
> > .. This is useful to prevent copy-on-write semantics from changing the
> > physical location of a page if the parent writes to it after a
> > fork(2) ..
>
> This would work around the issue but this is not transparent ie
> range marked with DONTFORK no longer behave as expected from the
> application point of view.
>
> Also it relies on userspace doing the right thing (which is not
> something i usualy trust :)).
The good thing that we didn't see anyone who succeeded to run
IB stack without our user space, which does right thing under
the hood :).
>
> Cheers,
> Jérôme
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply
* Re: [PATCH net-next] MAINTAINERS: Add myself as third phylib maintainer
From: Florian Fainelli @ 2018-11-19 19:01 UTC (permalink / raw)
To: Heiner Kallweit, Andrew Lunn, David Miller; +Cc: netdev@vger.kernel.org
In-Reply-To: <acdab6fa-b552-5358-6095-8ebe6a46c99b@gmail.com>
On 11/18/18 11:01 PM, Heiner Kallweit wrote:
> Add myself as third phylib maintainer.
>
> Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
Acked-by: Florian Fainelli <f.fainelli@gmail.com>
This could probably go to "net" (not a functional code change).
--
Florian
^ permalink raw reply
* [PATCH V2 mlx5-next 11/12] {net,IB}/mlx5: Move Page fault EQ and ODP logic to RDMA
From: Saeed Mahameed @ 2018-11-19 18:52 UTC (permalink / raw)
To: Leon Romanovsky, saeedm; +Cc: netdev, linux-rdma, Jason Gunthorpe
In-Reply-To: <20181119185242.21961-1-saeedm@mellanox.com>
Use the new generic EQ API to move all ODP RDMA data structures and logic
form mlx5 core driver into mlx5_ib driver.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/infiniband/hw/mlx5/main.c | 10 +-
drivers/infiniband/hw/mlx5/mlx5_ib.h | 15 +-
drivers/infiniband/hw/mlx5/odp.c | 281 +++++++++++++++++-
drivers/net/ethernet/mellanox/mlx5/core/dev.c | 34 ---
drivers/net/ethernet/mellanox/mlx5/core/eq.c | 252 ----------------
.../net/ethernet/mellanox/mlx5/core/lib/eq.h | 8 -
.../net/ethernet/mellanox/mlx5/core/main.c | 17 +-
.../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 -
include/linux/mlx5/driver.h | 49 ---
include/linux/mlx5/eq.h | 21 ++
10 files changed, 308 insertions(+), 381 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 6fbc0cba1bac..fcf4a0328a90 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -6040,6 +6040,11 @@ static int mlx5_ib_stage_odp_init(struct mlx5_ib_dev *dev)
return mlx5_ib_odp_init_one(dev);
}
+void mlx5_ib_stage_odp_cleanup(struct mlx5_ib_dev *dev)
+{
+ mlx5_ib_odp_cleanup_one(dev);
+}
+
int mlx5_ib_stage_counters_init(struct mlx5_ib_dev *dev)
{
if (MLX5_CAP_GEN(dev->mdev, max_qp_cnt)) {
@@ -6225,7 +6230,7 @@ static const struct mlx5_ib_profile pf_profile = {
mlx5_ib_stage_dev_res_cleanup),
STAGE_CREATE(MLX5_IB_STAGE_ODP,
mlx5_ib_stage_odp_init,
- NULL),
+ mlx5_ib_stage_odp_cleanup),
STAGE_CREATE(MLX5_IB_STAGE_COUNTERS,
mlx5_ib_stage_counters_init,
mlx5_ib_stage_counters_cleanup),
@@ -6395,9 +6400,6 @@ static struct mlx5_interface mlx5_ib_interface = {
.add = mlx5_ib_add,
.remove = mlx5_ib_remove,
.event = mlx5_ib_event,
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- .pfault = mlx5_ib_pfault,
-#endif
.protocol = MLX5_INTERFACE_PROTOCOL_IB,
};
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index b651a7a6fde9..27999fd32356 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -880,6 +880,15 @@ struct mlx5_ib_lb_state {
bool enabled;
};
+struct mlx5_ib_pf_eq {
+ struct mlx5_ib_dev *dev;
+ struct mlx5_eq *core;
+ struct work_struct work;
+ spinlock_t lock; /* Pagefaults spinlock */
+ struct workqueue_struct *wq;
+ mempool_t *pool;
+};
+
struct mlx5_ib_dev {
struct ib_device ib_dev;
const struct uverbs_object_tree_def *driver_trees[7];
@@ -902,6 +911,8 @@ struct mlx5_ib_dev {
#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
struct ib_odp_caps odp_caps;
u64 odp_max_size;
+ struct mlx5_ib_pf_eq odp_pf_eq;
+
/*
* Sleepable RCU that prevents destruction of MRs while they are still
* being used by a page fault handler.
@@ -1158,9 +1169,8 @@ struct ib_mr *mlx5_ib_reg_dm_mr(struct ib_pd *pd, struct ib_dm *dm,
#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev);
-void mlx5_ib_pfault(struct mlx5_core_dev *mdev, void *context,
- struct mlx5_pagefault *pfault);
int mlx5_ib_odp_init_one(struct mlx5_ib_dev *ibdev);
+void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev);
int __init mlx5_ib_odp_init(void);
void mlx5_ib_odp_cleanup(void);
void mlx5_ib_invalidate_range(struct ib_umem_odp *umem_odp, unsigned long start,
@@ -1175,6 +1185,7 @@ static inline void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev)
}
static inline int mlx5_ib_odp_init_one(struct mlx5_ib_dev *ibdev) { return 0; }
+static inline void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev) {}
static inline int mlx5_ib_odp_init(void) { return 0; }
static inline void mlx5_ib_odp_cleanup(void) {}
static inline void mlx5_odp_init_mr_cache_entry(struct mlx5_cache_ent *ent) {}
diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
index 7d784b40e017..416d141322a0 100644
--- a/drivers/infiniband/hw/mlx5/odp.c
+++ b/drivers/infiniband/hw/mlx5/odp.c
@@ -37,6 +37,46 @@
#include "mlx5_ib.h"
#include "cmd.h"
+#include <linux/mlx5/eq.h>
+
+/* Contains the details of a pagefault. */
+struct mlx5_pagefault {
+ u32 bytes_committed;
+ u32 token;
+ u8 event_subtype;
+ u8 type;
+ union {
+ /* Initiator or send message responder pagefault details. */
+ struct {
+ /* Received packet size, only valid for responders. */
+ u32 packet_size;
+ /*
+ * Number of resource holding WQE, depends on type.
+ */
+ u32 wq_num;
+ /*
+ * WQE index. Refers to either the send queue or
+ * receive queue, according to event_subtype.
+ */
+ u16 wqe_index;
+ } wqe;
+ /* RDMA responder pagefault details */
+ struct {
+ u32 r_key;
+ /*
+ * Received packet size, minimal size page fault
+ * resolution required for forward progress.
+ */
+ u32 packet_size;
+ u32 rdma_op_len;
+ u64 rdma_va;
+ } rdma;
+ };
+
+ struct mlx5_ib_pf_eq *eq;
+ struct work_struct work;
+};
+
#define MAX_PREFETCH_LEN (4*1024*1024U)
/* Timeout in ms to wait for an active mmu notifier to complete when handling
@@ -304,14 +344,20 @@ static void mlx5_ib_page_fault_resume(struct mlx5_ib_dev *dev,
{
int wq_num = pfault->event_subtype == MLX5_PFAULT_SUBTYPE_WQE ?
pfault->wqe.wq_num : pfault->token;
- int ret = mlx5_core_page_fault_resume(dev->mdev,
- pfault->token,
- wq_num,
- pfault->type,
- error);
- if (ret)
- mlx5_ib_err(dev, "Failed to resolve the page fault on WQ 0x%x\n",
- wq_num);
+ u32 out[MLX5_ST_SZ_DW(page_fault_resume_out)] = { };
+ u32 in[MLX5_ST_SZ_DW(page_fault_resume_in)] = { };
+ int err;
+
+ MLX5_SET(page_fault_resume_in, in, opcode, MLX5_CMD_OP_PAGE_FAULT_RESUME);
+ MLX5_SET(page_fault_resume_in, in, page_fault_type, pfault->type);
+ MLX5_SET(page_fault_resume_in, in, token, pfault->token);
+ MLX5_SET(page_fault_resume_in, in, wq_number, wq_num);
+ MLX5_SET(page_fault_resume_in, in, error, !!error);
+
+ err = mlx5_cmd_exec(dev->mdev, in, sizeof(in), out, sizeof(out));
+ if (err)
+ mlx5_ib_err(dev, "Failed to resolve the page fault on WQ 0x%x err %d\n",
+ wq_num, err);
}
static struct mlx5_ib_mr *implicit_mr_alloc(struct ib_pd *pd,
@@ -1196,10 +1242,8 @@ static void mlx5_ib_mr_rdma_pfault_handler(struct mlx5_ib_dev *dev,
}
}
-void mlx5_ib_pfault(struct mlx5_core_dev *mdev, void *context,
- struct mlx5_pagefault *pfault)
+static void mlx5_ib_pfault(struct mlx5_ib_dev *dev, struct mlx5_pagefault *pfault)
{
- struct mlx5_ib_dev *dev = context;
u8 event_subtype = pfault->event_subtype;
switch (event_subtype) {
@@ -1216,6 +1260,203 @@ void mlx5_ib_pfault(struct mlx5_core_dev *mdev, void *context,
}
}
+static void mlx5_ib_eqe_pf_action(struct work_struct *work)
+{
+ struct mlx5_pagefault *pfault = container_of(work,
+ struct mlx5_pagefault,
+ work);
+ struct mlx5_ib_pf_eq *eq = pfault->eq;
+
+ mlx5_ib_pfault(eq->dev, pfault);
+ mempool_free(pfault, eq->pool);
+}
+
+static void mlx5_ib_eq_pf_process(struct mlx5_ib_pf_eq *eq)
+{
+ struct mlx5_eqe_page_fault *pf_eqe;
+ struct mlx5_pagefault *pfault;
+ struct mlx5_eqe *eqe;
+ int cc = 0;
+
+ while ((eqe = mlx5_eq_get_eqe(eq->core, cc))) {
+ pfault = mempool_alloc(eq->pool, GFP_ATOMIC);
+ if (!pfault) {
+ schedule_work(&eq->work);
+ break;
+ }
+
+ pf_eqe = &eqe->data.page_fault;
+ pfault->event_subtype = eqe->sub_type;
+ pfault->bytes_committed = be32_to_cpu(pf_eqe->bytes_committed);
+
+ mlx5_ib_dbg(eq->dev,
+ "PAGE_FAULT: subtype: 0x%02x, bytes_committed: 0x%06x\n",
+ eqe->sub_type, pfault->bytes_committed);
+
+ switch (eqe->sub_type) {
+ case MLX5_PFAULT_SUBTYPE_RDMA:
+ /* RDMA based event */
+ pfault->type =
+ be32_to_cpu(pf_eqe->rdma.pftype_token) >> 24;
+ pfault->token =
+ be32_to_cpu(pf_eqe->rdma.pftype_token) &
+ MLX5_24BIT_MASK;
+ pfault->rdma.r_key =
+ be32_to_cpu(pf_eqe->rdma.r_key);
+ pfault->rdma.packet_size =
+ be16_to_cpu(pf_eqe->rdma.packet_length);
+ pfault->rdma.rdma_op_len =
+ be32_to_cpu(pf_eqe->rdma.rdma_op_len);
+ pfault->rdma.rdma_va =
+ be64_to_cpu(pf_eqe->rdma.rdma_va);
+ mlx5_ib_dbg(eq->dev,
+ "PAGE_FAULT: type:0x%x, token: 0x%06x, r_key: 0x%08x\n",
+ pfault->type, pfault->token,
+ pfault->rdma.r_key);
+ mlx5_ib_dbg(eq->dev,
+ "PAGE_FAULT: rdma_op_len: 0x%08x, rdma_va: 0x%016llx\n",
+ pfault->rdma.rdma_op_len,
+ pfault->rdma.rdma_va);
+ break;
+
+ case MLX5_PFAULT_SUBTYPE_WQE:
+ /* WQE based event */
+ pfault->type =
+ (be32_to_cpu(pf_eqe->wqe.pftype_wq) >> 24) & 0x7;
+ pfault->token =
+ be32_to_cpu(pf_eqe->wqe.token);
+ pfault->wqe.wq_num =
+ be32_to_cpu(pf_eqe->wqe.pftype_wq) &
+ MLX5_24BIT_MASK;
+ pfault->wqe.wqe_index =
+ be16_to_cpu(pf_eqe->wqe.wqe_index);
+ pfault->wqe.packet_size =
+ be16_to_cpu(pf_eqe->wqe.packet_length);
+ mlx5_ib_dbg(eq->dev,
+ "PAGE_FAULT: type:0x%x, token: 0x%06x, wq_num: 0x%06x, wqe_index: 0x%04x\n",
+ pfault->type, pfault->token,
+ pfault->wqe.wq_num,
+ pfault->wqe.wqe_index);
+ break;
+
+ default:
+ mlx5_ib_warn(eq->dev,
+ "Unsupported page fault event sub-type: 0x%02hhx\n",
+ eqe->sub_type);
+ /* Unsupported page faults should still be
+ * resolved by the page fault handler
+ */
+ }
+
+ pfault->eq = eq;
+ INIT_WORK(&pfault->work, mlx5_ib_eqe_pf_action);
+ queue_work(eq->wq, &pfault->work);
+
+ cc = mlx5_eq_update_cc(eq->core, ++cc);
+ }
+
+ mlx5_eq_update_ci(eq->core, cc, 1);
+}
+
+static irqreturn_t mlx5_ib_eq_pf_int(int irq, void *eq_ptr)
+{
+ struct mlx5_ib_pf_eq *eq = eq_ptr;
+ unsigned long flags;
+
+ if (spin_trylock_irqsave(&eq->lock, flags)) {
+ mlx5_ib_eq_pf_process(eq);
+ spin_unlock_irqrestore(&eq->lock, flags);
+ } else {
+ schedule_work(&eq->work);
+ }
+
+ return IRQ_HANDLED;
+}
+
+/* mempool_refill() was proposed but unfortunately wasn't accepted
+ * http://lkml.iu.edu/hypermail/linux/kernel/1512.1/05073.html
+ * Cheap workaround.
+ */
+static void mempool_refill(mempool_t *pool)
+{
+ while (pool->curr_nr < pool->min_nr)
+ mempool_free(mempool_alloc(pool, GFP_KERNEL), pool);
+}
+
+static void mlx5_ib_eq_pf_action(struct work_struct *work)
+{
+ struct mlx5_ib_pf_eq *eq =
+ container_of(work, struct mlx5_ib_pf_eq, work);
+
+ mempool_refill(eq->pool);
+
+ spin_lock_irq(&eq->lock);
+ mlx5_ib_eq_pf_process(eq);
+ spin_unlock_irq(&eq->lock);
+}
+
+enum {
+ MLX5_IB_NUM_PF_EQE = 0x1000,
+ MLX5_IB_NUM_PF_DRAIN = 64,
+};
+
+static int
+mlx5_ib_create_pf_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq)
+{
+ struct mlx5_eq_param param = {};
+ int err;
+
+ INIT_WORK(&eq->work, mlx5_ib_eq_pf_action);
+ spin_lock_init(&eq->lock);
+ eq->dev = dev;
+
+ eq->pool = mempool_create_kmalloc_pool(MLX5_IB_NUM_PF_DRAIN,
+ sizeof(struct mlx5_pagefault));
+ if (!eq->pool)
+ return -ENOMEM;
+
+ eq->wq = alloc_workqueue("mlx5_ib_page_fault",
+ WQ_HIGHPRI | WQ_UNBOUND | WQ_MEM_RECLAIM,
+ MLX5_NUM_CMD_EQE);
+ if (!eq->wq) {
+ err = -ENOMEM;
+ goto err_mempool;
+ }
+
+ param = (struct mlx5_eq_param) {
+ .index = MLX5_EQ_PFAULT_IDX,
+ .mask = 1 << MLX5_EVENT_TYPE_PAGE_FAULT,
+ .nent = MLX5_IB_NUM_PF_EQE,
+ .context = eq,
+ .handler = mlx5_ib_eq_pf_int
+ };
+ eq->core = mlx5_eq_create_generic(dev->mdev, "mlx5_ib_page_fault_eq", ¶m);
+ if (IS_ERR(eq->core)) {
+ err = PTR_ERR(eq->core);
+ goto err_wq;
+ }
+
+ return 0;
+err_wq:
+ destroy_workqueue(eq->wq);
+err_mempool:
+ mempool_destroy(eq->pool);
+ return err;
+}
+
+static int
+mlx5_ib_destroy_pf_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq)
+{
+ int err;
+
+ err = mlx5_eq_destroy_generic(dev->mdev, eq->core);
+ cancel_work_sync(&eq->work);
+ destroy_workqueue(eq->wq);
+ mempool_destroy(eq->pool);
+
+ return err;
+}
+
void mlx5_odp_init_mr_cache_entry(struct mlx5_cache_ent *ent)
{
if (!(ent->dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT))
@@ -1244,7 +1485,7 @@ void mlx5_odp_init_mr_cache_entry(struct mlx5_cache_ent *ent)
int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev)
{
- int ret;
+ int ret = 0;
if (dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT) {
ret = mlx5_cmd_null_mkey(dev->mdev, &dev->null_mkey);
@@ -1254,7 +1495,20 @@ int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev)
}
}
- return 0;
+ if (!MLX5_CAP_GEN(dev->mdev, pg))
+ return ret;
+
+ ret = mlx5_ib_create_pf_eq(dev, &dev->odp_pf_eq);
+
+ return ret;
+}
+
+void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *dev)
+{
+ if (!MLX5_CAP_GEN(dev->mdev, pg))
+ return;
+
+ mlx5_ib_destroy_pf_eq(dev, &dev->odp_pf_eq);
}
int mlx5_ib_odp_init(void)
@@ -1264,4 +1518,3 @@ int mlx5_ib_odp_init(void)
return 0;
}
-
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/dev.c b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
index 37ba7c78859d..7eedbea38a78 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/dev.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
@@ -139,17 +139,6 @@ void mlx5_add_device(struct mlx5_interface *intf, struct mlx5_priv *priv)
spin_lock_irq(&priv->ctx_lock);
list_add_tail(&dev_ctx->list, &priv->ctx_list);
-
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- if (dev_ctx->intf->pfault) {
- if (priv->pfault) {
- mlx5_core_err(dev, "multiple page fault handlers not supported");
- } else {
- priv->pfault_ctx = dev_ctx->context;
- priv->pfault = dev_ctx->intf->pfault;
- }
- }
-#endif
spin_unlock_irq(&priv->ctx_lock);
}
@@ -179,15 +168,6 @@ void mlx5_remove_device(struct mlx5_interface *intf, struct mlx5_priv *priv)
if (!dev_ctx)
return;
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- spin_lock_irq(&priv->ctx_lock);
- if (priv->pfault == dev_ctx->intf->pfault)
- priv->pfault = NULL;
- spin_unlock_irq(&priv->ctx_lock);
-
- synchronize_srcu(&priv->pfault_srcu);
-#endif
-
spin_lock_irq(&priv->ctx_lock);
list_del(&dev_ctx->list);
spin_unlock_irq(&priv->ctx_lock);
@@ -447,20 +427,6 @@ void mlx5_core_event(struct mlx5_core_dev *dev, enum mlx5_dev_event event,
spin_unlock_irqrestore(&priv->ctx_lock, flags);
}
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
-void mlx5_core_page_fault(struct mlx5_core_dev *dev,
- struct mlx5_pagefault *pfault)
-{
- struct mlx5_priv *priv = &dev->priv;
- int srcu_idx;
-
- srcu_idx = srcu_read_lock(&priv->pfault_srcu);
- if (priv->pfault)
- priv->pfault(dev, priv->pfault_ctx, pfault);
- srcu_read_unlock(&priv->pfault_srcu, srcu_idx);
-}
-#endif
-
void mlx5_dev_list_lock(void)
{
mutex_lock(&mlx5_intf_mutex);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
index ec1f5018546e..895401609c63 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
@@ -56,13 +56,6 @@ enum {
MLX5_EQ_STATE_ALWAYS_ARMED = 0xb,
};
-enum {
- MLX5_NUM_SPARE_EQE = 0x80,
- MLX5_NUM_ASYNC_EQE = 0x1000,
- MLX5_NUM_CMD_EQE = 32,
- MLX5_NUM_PF_DRAIN = 64,
-};
-
enum {
MLX5_EQ_DOORBEL_OFFSET = 0x40,
};
@@ -79,9 +72,6 @@ struct mlx5_eq_table {
struct mlx5_eq async_eq;
struct mlx5_eq cmd_eq;
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- struct mlx5_eq_pagefault pfault_eq;
-#endif
struct mutex lock; /* sync async eqs creations */
int num_comp_vectors;
struct mlx5_irq_info *irq_info;
@@ -222,224 +212,6 @@ static void eq_update_ci(struct mlx5_eq *eq, int arm)
mb();
}
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
-static void eqe_pf_action(struct work_struct *work)
-{
- struct mlx5_pagefault *pfault = container_of(work,
- struct mlx5_pagefault,
- work);
- struct mlx5_eq_pagefault *eq = pfault->eq;
-
- mlx5_core_page_fault(eq->core->dev, pfault);
- mempool_free(pfault, eq->pool);
-}
-
-static void eq_pf_process(struct mlx5_eq_pagefault *eq)
-{
- struct mlx5_core_dev *dev = eq->core->dev;
- struct mlx5_eqe_page_fault *pf_eqe;
- struct mlx5_pagefault *pfault;
- struct mlx5_eqe *eqe;
- int set_ci = 0;
-
- while ((eqe = next_eqe_sw(eq->core))) {
- pfault = mempool_alloc(eq->pool, GFP_ATOMIC);
- if (!pfault) {
- schedule_work(&eq->work);
- break;
- }
-
- dma_rmb();
- pf_eqe = &eqe->data.page_fault;
- pfault->event_subtype = eqe->sub_type;
- pfault->bytes_committed = be32_to_cpu(pf_eqe->bytes_committed);
-
- mlx5_core_dbg(dev,
- "PAGE_FAULT: subtype: 0x%02x, bytes_committed: 0x%06x\n",
- eqe->sub_type, pfault->bytes_committed);
-
- switch (eqe->sub_type) {
- case MLX5_PFAULT_SUBTYPE_RDMA:
- /* RDMA based event */
- pfault->type =
- be32_to_cpu(pf_eqe->rdma.pftype_token) >> 24;
- pfault->token =
- be32_to_cpu(pf_eqe->rdma.pftype_token) &
- MLX5_24BIT_MASK;
- pfault->rdma.r_key =
- be32_to_cpu(pf_eqe->rdma.r_key);
- pfault->rdma.packet_size =
- be16_to_cpu(pf_eqe->rdma.packet_length);
- pfault->rdma.rdma_op_len =
- be32_to_cpu(pf_eqe->rdma.rdma_op_len);
- pfault->rdma.rdma_va =
- be64_to_cpu(pf_eqe->rdma.rdma_va);
- mlx5_core_dbg(dev,
- "PAGE_FAULT: type:0x%x, token: 0x%06x, r_key: 0x%08x\n",
- pfault->type, pfault->token,
- pfault->rdma.r_key);
- mlx5_core_dbg(dev,
- "PAGE_FAULT: rdma_op_len: 0x%08x, rdma_va: 0x%016llx\n",
- pfault->rdma.rdma_op_len,
- pfault->rdma.rdma_va);
- break;
-
- case MLX5_PFAULT_SUBTYPE_WQE:
- /* WQE based event */
- pfault->type =
- (be32_to_cpu(pf_eqe->wqe.pftype_wq) >> 24) & 0x7;
- pfault->token =
- be32_to_cpu(pf_eqe->wqe.token);
- pfault->wqe.wq_num =
- be32_to_cpu(pf_eqe->wqe.pftype_wq) &
- MLX5_24BIT_MASK;
- pfault->wqe.wqe_index =
- be16_to_cpu(pf_eqe->wqe.wqe_index);
- pfault->wqe.packet_size =
- be16_to_cpu(pf_eqe->wqe.packet_length);
- mlx5_core_dbg(dev,
- "PAGE_FAULT: type:0x%x, token: 0x%06x, wq_num: 0x%06x, wqe_index: 0x%04x\n",
- pfault->type, pfault->token,
- pfault->wqe.wq_num,
- pfault->wqe.wqe_index);
- break;
-
- default:
- mlx5_core_warn(dev,
- "Unsupported page fault event sub-type: 0x%02hhx\n",
- eqe->sub_type);
- /* Unsupported page faults should still be
- * resolved by the page fault handler
- */
- }
-
- pfault->eq = eq;
- INIT_WORK(&pfault->work, eqe_pf_action);
- queue_work(eq->wq, &pfault->work);
-
- ++eq->core->cons_index;
- ++set_ci;
-
- if (unlikely(set_ci >= MLX5_NUM_SPARE_EQE)) {
- eq_update_ci(eq->core, 0);
- set_ci = 0;
- }
- }
-
- eq_update_ci(eq->core, 1);
-}
-
-static irqreturn_t mlx5_eq_pf_int(int irq, void *eq_ptr)
-{
- struct mlx5_eq_pagefault *eq = eq_ptr;
- unsigned long flags;
-
- if (spin_trylock_irqsave(&eq->lock, flags)) {
- eq_pf_process(eq);
- spin_unlock_irqrestore(&eq->lock, flags);
- } else {
- schedule_work(&eq->work);
- }
-
- return IRQ_HANDLED;
-}
-
-/* mempool_refill() was proposed but unfortunately wasn't accepted
- * http://lkml.iu.edu/hypermail/linux/kernel/1512.1/05073.html
- * Chip workaround.
- */
-static void mempool_refill(mempool_t *pool)
-{
- while (pool->curr_nr < pool->min_nr)
- mempool_free(mempool_alloc(pool, GFP_KERNEL), pool);
-}
-
-static void eq_pf_action(struct work_struct *work)
-{
- struct mlx5_eq_pagefault *eq =
- container_of(work, struct mlx5_eq_pagefault, work);
-
- mempool_refill(eq->pool);
-
- spin_lock_irq(&eq->lock);
- eq_pf_process(eq);
- spin_unlock_irq(&eq->lock);
-}
-
-static int
-create_pf_eq(struct mlx5_core_dev *dev, struct mlx5_eq_pagefault *eq)
-{
- struct mlx5_eq_param param = {};
- int err;
-
- spin_lock_init(&eq->lock);
- INIT_WORK(&eq->work, eq_pf_action);
-
- eq->pool = mempool_create_kmalloc_pool(MLX5_NUM_PF_DRAIN,
- sizeof(struct mlx5_pagefault));
- if (!eq->pool)
- return -ENOMEM;
-
- eq->wq = alloc_workqueue("mlx5_page_fault",
- WQ_HIGHPRI | WQ_UNBOUND | WQ_MEM_RECLAIM,
- MLX5_NUM_CMD_EQE);
- if (!eq->wq) {
- err = -ENOMEM;
- goto err_mempool;
- }
-
- param = (struct mlx5_eq_param) {
- .index = MLX5_EQ_PFAULT_IDX,
- .mask = 1 << MLX5_EVENT_TYPE_PAGE_FAULT,
- .nent = MLX5_NUM_ASYNC_EQE,
- .context = eq,
- .handler = mlx5_eq_pf_int
- };
-
- eq->core = mlx5_eq_create_generic(dev, "mlx5_page_fault_eq", ¶m);
- if (IS_ERR(eq->core)) {
- err = PTR_ERR(eq->core);
- goto err_wq;
- }
-
- return 0;
-err_wq:
- destroy_workqueue(eq->wq);
-err_mempool:
- mempool_destroy(eq->pool);
- return err;
-}
-
-static int destroy_pf_eq(struct mlx5_core_dev *dev, struct mlx5_eq_pagefault *eq)
-{
- int err;
-
- err = mlx5_eq_destroy_generic(dev, eq->core);
- cancel_work_sync(&eq->work);
- destroy_workqueue(eq->wq);
- mempool_destroy(eq->pool);
-
- return err;
-}
-
-int mlx5_core_page_fault_resume(struct mlx5_core_dev *dev, u32 token,
- u32 wq_num, u8 type, int error)
-{
- u32 out[MLX5_ST_SZ_DW(page_fault_resume_out)] = {0};
- u32 in[MLX5_ST_SZ_DW(page_fault_resume_in)] = {0};
-
- MLX5_SET(page_fault_resume_in, in, opcode,
- MLX5_CMD_OP_PAGE_FAULT_RESUME);
- MLX5_SET(page_fault_resume_in, in, error, !!error);
- MLX5_SET(page_fault_resume_in, in, page_fault_type, type);
- MLX5_SET(page_fault_resume_in, in, wq_number, wq_num);
- MLX5_SET(page_fault_resume_in, in, token, token);
-
- return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
-}
-EXPORT_SYMBOL_GPL(mlx5_core_page_fault_resume);
-#endif
-
static void general_event_handler(struct mlx5_core_dev *dev,
struct mlx5_eqe *eqe)
{
@@ -1016,22 +788,7 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
goto err2;
}
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- if (MLX5_CAP_GEN(dev, pg)) {
- err = create_pf_eq(dev, &table->pfault_eq);
- if (err) {
- mlx5_core_warn(dev, "failed to create page fault EQ %d\n",
- err);
- goto err3;
- }
- }
-
- return err;
-err3:
- destroy_async_eq(dev, &table->pages_eq);
-#else
return err;
-#endif
err2:
destroy_async_eq(dev, &table->async_eq);
@@ -1047,15 +804,6 @@ static void destroy_async_eqs(struct mlx5_core_dev *dev)
struct mlx5_eq_table *table = dev->priv.eq_table;
int err;
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- if (MLX5_CAP_GEN(dev, pg)) {
- err = destroy_pf_eq(dev, &table->pfault_eq);
- if (err)
- mlx5_core_err(dev, "failed to destroy page fault eq, err(%d)\n",
- err);
- }
-#endif
-
err = destroy_async_eq(dev, &table->pages_eq);
if (err)
mlx5_core_err(dev, "failed to destroy pages eq, err(%d)\n",
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
index db32057ad054..4cc2d442cef6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
@@ -39,14 +39,6 @@ struct mlx5_eq_comp {
struct list_head list;
};
-struct mlx5_eq_pagefault {
- struct mlx5_eq *core;
- struct work_struct work;
- spinlock_t lock; /* Pagefaults spinlock */
- struct workqueue_struct *wq;
- mempool_t *pool;
-};
-
int mlx5_eq_table_init(struct mlx5_core_dev *dev);
void mlx5_eq_table_cleanup(struct mlx5_core_dev *dev);
int mlx5_eq_table_create(struct mlx5_core_dev *dev);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 3de83fe65f2b..91022f141855 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -1169,14 +1169,6 @@ static int init_one(struct pci_dev *pdev,
INIT_LIST_HEAD(&priv->waiting_events_list);
priv->is_accum_events = false;
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- err = init_srcu_struct(&priv->pfault_srcu);
- if (err) {
- dev_err(&pdev->dev, "init_srcu_struct failed with error code %d\n",
- err);
- goto clean_dev;
- }
-#endif
mutex_init(&priv->bfregs.reg_head.lock);
mutex_init(&priv->bfregs.wc_head.lock);
INIT_LIST_HEAD(&priv->bfregs.reg_head.list);
@@ -1185,7 +1177,7 @@ static int init_one(struct pci_dev *pdev,
err = mlx5_pci_init(dev, priv);
if (err) {
dev_err(&pdev->dev, "mlx5_pci_init failed with error code %d\n", err);
- goto clean_srcu;
+ goto clean_dev;
}
err = mlx5_health_init(dev);
@@ -1218,11 +1210,7 @@ static int init_one(struct pci_dev *pdev,
mlx5_health_cleanup(dev);
close_pci:
mlx5_pci_close(dev, priv);
-clean_srcu:
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- cleanup_srcu_struct(&priv->pfault_srcu);
clean_dev:
-#endif
devlink_free(devlink);
return err;
@@ -1246,9 +1234,6 @@ static void remove_one(struct pci_dev *pdev)
mlx5_pagealloc_cleanup(dev);
mlx5_health_cleanup(dev);
mlx5_pci_close(dev, priv);
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- cleanup_srcu_struct(&priv->pfault_srcu);
-#endif
devlink_free(devlink);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
index 4728b027cb9e..21727d9eeb84 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
@@ -100,8 +100,6 @@ int mlx5_cmd_fast_teardown_hca(struct mlx5_core_dev *dev);
void mlx5_core_event(struct mlx5_core_dev *dev, enum mlx5_dev_event event,
unsigned long param);
-void mlx5_core_page_fault(struct mlx5_core_dev *dev,
- struct mlx5_pagefault *pfault);
void mlx5_port_module_event(struct mlx5_core_dev *dev, struct mlx5_eqe *eqe);
void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force);
void mlx5_disable_device(struct mlx5_core_dev *dev);
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index fe9b552aa649..f41e6713df10 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -510,7 +510,6 @@ struct mlx5_fc_stats {
struct mlx5_mpfs;
struct mlx5_eswitch;
struct mlx5_lag;
-struct mlx5_pagefault;
struct mlx5_eq_table;
struct mlx5_rate_limit {
@@ -619,13 +618,6 @@ struct mlx5_priv {
struct mlx5_port_module_event_stats pme_stats;
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- void (*pfault)(struct mlx5_core_dev *dev,
- void *context,
- struct mlx5_pagefault *pfault);
- void *pfault_ctx;
- struct srcu_struct pfault_srcu;
-#endif
struct mlx5_bfreg_data bfregs;
struct mlx5_uars_page *uar;
};
@@ -650,44 +642,6 @@ enum mlx5_pagefault_type_flags {
MLX5_PFAULT_RDMA = 1 << 2,
};
-/* Contains the details of a pagefault. */
-struct mlx5_pagefault {
- u32 bytes_committed;
- u32 token;
- u8 event_subtype;
- u8 type;
- union {
- /* Initiator or send message responder pagefault details. */
- struct {
- /* Received packet size, only valid for responders. */
- u32 packet_size;
- /*
- * Number of resource holding WQE, depends on type.
- */
- u32 wq_num;
- /*
- * WQE index. Refers to either the send queue or
- * receive queue, according to event_subtype.
- */
- u16 wqe_index;
- } wqe;
- /* RDMA responder pagefault details */
- struct {
- u32 r_key;
- /*
- * Received packet size, minimal size page fault
- * resolution required for forward progress.
- */
- u32 packet_size;
- u32 rdma_op_len;
- u64 rdma_va;
- } rdma;
- };
-
- struct mlx5_eq_pagefault *eq;
- struct work_struct work;
-};
-
struct mlx5_td {
struct list_head tirs_list;
u32 tdn;
@@ -1118,9 +1072,6 @@ struct mlx5_interface {
void (*detach)(struct mlx5_core_dev *dev, void *context);
void (*event)(struct mlx5_core_dev *dev, void *context,
enum mlx5_dev_event event, unsigned long param);
- void (*pfault)(struct mlx5_core_dev *dev,
- void *context,
- struct mlx5_pagefault *pfault);
void * (*get_dev)(void *context);
int protocol;
struct list_head list;
diff --git a/include/linux/mlx5/eq.h b/include/linux/mlx5/eq.h
index c733673ba5f6..71d82c5a1a02 100644
--- a/include/linux/mlx5/eq.h
+++ b/include/linux/mlx5/eq.h
@@ -17,6 +17,10 @@ enum {
MLX5_EQ_VEC_COMP_BASE = MLX5_EQ_MAX_ASYNC_EQS,
};
+#define MLX5_NUM_CMD_EQE (32)
+#define MLX5_NUM_ASYNC_EQE (0x1000)
+#define MLX5_NUM_SPARE_EQE (0x80)
+
struct mlx5_eq;
struct mlx5_eq_param {
@@ -36,4 +40,21 @@ mlx5_eq_destroy_generic(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
struct mlx5_eqe *mlx5_eq_get_eqe(struct mlx5_eq *eq, u32 cc);
void mlx5_eq_update_ci(struct mlx5_eq *eq, u32 cc, bool arm);
+/* The HCA will think the queue has overflowed if we
+ * don't tell it we've been processing events. We
+ * create EQs with MLX5_NUM_SPARE_EQE extra entries,
+ * so we must update our consumer index at
+ * least that often.
+ *
+ * mlx5_eq_update_cc must be called on every EQE @EQ irq handler
+ */
+static inline u32 mlx5_eq_update_cc(struct mlx5_eq *eq, u32 cc)
+{
+ if (unlikely(cc >= MLX5_NUM_SPARE_EQE)) {
+ mlx5_eq_update_ci(eq, cc, 0);
+ cc = 0;
+ }
+ return cc;
+}
+
#endif /* MLX5_CORE_EQ_H */
--
2.19.1
^ permalink raw reply related
* [PATCH V2 mlx5-next 12/12] net/mlx5: EQ, Make EQE access methods inline
From: Saeed Mahameed @ 2018-11-19 18:52 UTC (permalink / raw)
To: Leon Romanovsky, saeedm; +Cc: netdev, linux-rdma, Jason Gunthorpe
In-Reply-To: <20181119185242.21961-1-saeedm@mellanox.com>
These are one/two liner generic EQ access methods, better have them
declared static inline in eq.h.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/eq.c | 23 -----------------
.../net/ethernet/mellanox/mlx5/core/lib/eq.h | 25 ++++++++++++++++++-
2 files changed, 24 insertions(+), 24 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
index 895401609c63..6ba8e401a0c7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
@@ -46,7 +46,6 @@
#include "diag/fw_tracer.h"
enum {
- MLX5_EQE_SIZE = sizeof(struct mlx5_eqe),
MLX5_EQE_OWNER_INIT_VAL = 0x1,
};
@@ -103,18 +102,6 @@ static int mlx5_cmd_destroy_eq(struct mlx5_core_dev *dev, u8 eqn)
return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
}
-static struct mlx5_eqe *get_eqe(struct mlx5_eq *eq, u32 entry)
-{
- return mlx5_buf_offset(&eq->buf, entry * MLX5_EQE_SIZE);
-}
-
-static struct mlx5_eqe *next_eqe_sw(struct mlx5_eq *eq)
-{
- struct mlx5_eqe *eqe = get_eqe(eq, eq->cons_index & (eq->nent - 1));
-
- return ((eqe->owner & 1) ^ !!(eq->cons_index & eq->nent)) ? NULL : eqe;
-}
-
static const char *eqe_type_str(u8 type)
{
switch (type) {
@@ -202,16 +189,6 @@ static enum mlx5_dev_event port_subtype_event(u8 subtype)
return -1;
}
-static void eq_update_ci(struct mlx5_eq *eq, int arm)
-{
- __be32 __iomem *addr = eq->doorbell + (arm ? 0 : 2);
- u32 val = (eq->cons_index & 0xffffff) | (eq->eqn << 24);
-
- __raw_writel((__force u32)cpu_to_be32(val), addr);
- /* We still want ordering, just not swabbing, so add a barrier */
- mb();
-}
-
static void general_event_handler(struct mlx5_core_dev *dev,
struct mlx5_eqe *eqe)
{
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
index 4cc2d442cef6..6d8c8a57d52b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
@@ -5,7 +5,8 @@
#define __LIB_MLX5_EQ_H__
#include <linux/mlx5/driver.h>
-#define MLX5_MAX_IRQ_NAME (32)
+#define MLX5_MAX_IRQ_NAME (32)
+#define MLX5_EQE_SIZE (sizeof(struct mlx5_eqe))
struct mlx5_eq_tasklet {
struct list_head list;
@@ -39,6 +40,28 @@ struct mlx5_eq_comp {
struct list_head list;
};
+static inline struct mlx5_eqe *get_eqe(struct mlx5_eq *eq, u32 entry)
+{
+ return mlx5_buf_offset(&eq->buf, entry * MLX5_EQE_SIZE);
+}
+
+static inline struct mlx5_eqe *next_eqe_sw(struct mlx5_eq *eq)
+{
+ struct mlx5_eqe *eqe = get_eqe(eq, eq->cons_index & (eq->nent - 1));
+
+ return ((eqe->owner & 1) ^ !!(eq->cons_index & eq->nent)) ? NULL : eqe;
+}
+
+static inline void eq_update_ci(struct mlx5_eq *eq, int arm)
+{
+ __be32 __iomem *addr = eq->doorbell + (arm ? 0 : 2);
+ u32 val = (eq->cons_index & 0xffffff) | (eq->eqn << 24);
+
+ __raw_writel((__force u32)cpu_to_be32(val), addr);
+ /* We still want ordering, just not swabbing, so add a barrier */
+ mb();
+}
+
int mlx5_eq_table_init(struct mlx5_core_dev *dev);
void mlx5_eq_table_cleanup(struct mlx5_core_dev *dev);
int mlx5_eq_table_create(struct mlx5_core_dev *dev);
--
2.19.1
^ permalink raw reply related
* [PATCH V2 mlx5-next 10/12] net/mlx5: EQ, Generic EQ
From: Saeed Mahameed @ 2018-11-19 18:52 UTC (permalink / raw)
To: Leon Romanovsky, saeedm; +Cc: netdev, linux-rdma, Jason Gunthorpe
In-Reply-To: <20181119185242.21961-1-saeedm@mellanox.com>
Add mlx5_eq_{create/destroy}_generic APIs and EQE access methods, for
mlx5 core consumers generic EQs.
This API will be used in downstream patch to move page fault (RDMA ODP)
EQ logic into mlx5_ib rdma driver, hence it will use a generic EQ.
Current mlx5 EQ allocation scheme:
On load mlx5 allocates 4 (for async) + #cores (for data completions)
MSIX vectors, mlx5 core will assign 3 MSIX vectors for internal async
EQs and will use all of the #cores MSIX vectors for completion EQs,
(One vector is going to be reserved for a generic EQ).
After this patch an external user (e.g mlx5_ib) of mlx5_core
can use this new API to create new generic EQs with the reserved msix
vector index for that eq.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/eq.c | 243 +++++++++++++-----
.../net/ethernet/mellanox/mlx5/core/lib/eq.h | 12 +-
include/linux/mlx5/eq.h | 39 +++
3 files changed, 221 insertions(+), 73 deletions(-)
create mode 100644 include/linux/mlx5/eq.h
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
index 252c9f0569b1..ec1f5018546e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
@@ -33,6 +33,7 @@
#include <linux/interrupt.h>
#include <linux/module.h>
#include <linux/mlx5/driver.h>
+#include <linux/mlx5/eq.h>
#include <linux/mlx5/cmd.h>
#ifdef CONFIG_RFS_ACCEL
#include <linux/cpu_rmap.h>
@@ -69,6 +70,7 @@ enum {
struct mlx5_irq_info {
cpumask_var_t mask;
char name[MLX5_MAX_IRQ_NAME];
+ void *context; /* dev_id provided to request_irq */
};
struct mlx5_eq_table {
@@ -81,7 +83,6 @@ struct mlx5_eq_table {
struct mlx5_eq_pagefault pfault_eq;
#endif
struct mutex lock; /* sync async eqs creations */
- u8 num_async_eqs;
int num_comp_vectors;
struct mlx5_irq_info *irq_info;
#ifdef CONFIG_RFS_ACCEL
@@ -229,19 +230,19 @@ static void eqe_pf_action(struct work_struct *work)
work);
struct mlx5_eq_pagefault *eq = pfault->eq;
- mlx5_core_page_fault(eq->core.dev, pfault);
+ mlx5_core_page_fault(eq->core->dev, pfault);
mempool_free(pfault, eq->pool);
}
static void eq_pf_process(struct mlx5_eq_pagefault *eq)
{
- struct mlx5_core_dev *dev = eq->core.dev;
+ struct mlx5_core_dev *dev = eq->core->dev;
struct mlx5_eqe_page_fault *pf_eqe;
struct mlx5_pagefault *pfault;
struct mlx5_eqe *eqe;
int set_ci = 0;
- while ((eqe = next_eqe_sw(&eq->core))) {
+ while ((eqe = next_eqe_sw(eq->core))) {
pfault = mempool_alloc(eq->pool, GFP_ATOMIC);
if (!pfault) {
schedule_work(&eq->work);
@@ -316,16 +317,16 @@ static void eq_pf_process(struct mlx5_eq_pagefault *eq)
INIT_WORK(&pfault->work, eqe_pf_action);
queue_work(eq->wq, &pfault->work);
- ++eq->core.cons_index;
+ ++eq->core->cons_index;
++set_ci;
if (unlikely(set_ci >= MLX5_NUM_SPARE_EQE)) {
- eq_update_ci(&eq->core, 0);
+ eq_update_ci(eq->core, 0);
set_ci = 0;
}
}
- eq_update_ci(&eq->core, 1);
+ eq_update_ci(eq->core, 1);
}
static irqreturn_t mlx5_eq_pf_int(int irq, void *eq_ptr)
@@ -368,6 +369,7 @@ static void eq_pf_action(struct work_struct *work)
static int
create_pf_eq(struct mlx5_core_dev *dev, struct mlx5_eq_pagefault *eq)
{
+ struct mlx5_eq_param param = {};
int err;
spin_lock_init(&eq->lock);
@@ -386,11 +388,19 @@ create_pf_eq(struct mlx5_core_dev *dev, struct mlx5_eq_pagefault *eq)
goto err_mempool;
}
- err = mlx5_create_async_eq(dev, &eq->core, MLX5_NUM_ASYNC_EQE,
- 1 << MLX5_EVENT_TYPE_PAGE_FAULT,
- "mlx5_page_fault_eq", mlx5_eq_pf_int);
- if (err)
+ param = (struct mlx5_eq_param) {
+ .index = MLX5_EQ_PFAULT_IDX,
+ .mask = 1 << MLX5_EVENT_TYPE_PAGE_FAULT,
+ .nent = MLX5_NUM_ASYNC_EQE,
+ .context = eq,
+ .handler = mlx5_eq_pf_int
+ };
+
+ eq->core = mlx5_eq_create_generic(dev, "mlx5_page_fault_eq", ¶m);
+ if (IS_ERR(eq->core)) {
+ err = PTR_ERR(eq->core);
goto err_wq;
+ }
return 0;
err_wq:
@@ -404,7 +414,7 @@ static int destroy_pf_eq(struct mlx5_core_dev *dev, struct mlx5_eq_pagefault *eq
{
int err;
- err = mlx5_destroy_async_eq(dev, &eq->core);
+ err = mlx5_eq_destroy_generic(dev, eq->core);
cancel_work_sync(&eq->work);
destroy_workqueue(eq->wq);
mempool_destroy(eq->pool);
@@ -710,25 +720,29 @@ static void init_eq_buf(struct mlx5_eq *eq)
}
static int
-mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
- int nent, u64 mask, const char *name, irq_handler_t handler)
+create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, const char *name,
+ struct mlx5_eq_param *param)
{
struct mlx5_eq_table *eq_table = dev->priv.eq_table;
struct mlx5_cq_table *cq_table = &eq->cq_table;
u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {0};
struct mlx5_priv *priv = &dev->priv;
+ u8 vecidx = param->index;
__be64 *pas;
void *eqc;
int inlen;
u32 *in;
int err;
+ if (eq_table->irq_info[vecidx].context)
+ return -EEXIST;
+
/* Init CQ table */
memset(cq_table, 0, sizeof(*cq_table));
spin_lock_init(&cq_table->lock);
INIT_RADIX_TREE(&cq_table->tree, GFP_ATOMIC);
- eq->nent = roundup_pow_of_two(nent + MLX5_NUM_SPARE_EQE);
+ eq->nent = roundup_pow_of_two(param->nent + MLX5_NUM_SPARE_EQE);
eq->cons_index = 0;
err = mlx5_buf_alloc(dev, eq->nent * MLX5_EQE_SIZE, &eq->buf);
if (err)
@@ -749,7 +763,7 @@ mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
mlx5_fill_page_array(&eq->buf, pas);
MLX5_SET(create_eq_in, in, opcode, MLX5_CMD_OP_CREATE_EQ);
- MLX5_SET64(create_eq_in, in, event_bitmask, mask);
+ MLX5_SET64(create_eq_in, in, event_bitmask, param->mask);
eqc = MLX5_ADDR_OF(create_eq_in, in, eq_context_entry);
MLX5_SET(eqc, eqc, log_eq_size, ilog2(eq->nent));
@@ -764,13 +778,15 @@ mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
snprintf(eq_table->irq_info[vecidx].name, MLX5_MAX_IRQ_NAME, "%s@pci:%s",
name, pci_name(dev->pdev));
+ eq_table->irq_info[vecidx].context = param->context;
+ eq->vecidx = vecidx;
eq->eqn = MLX5_GET(create_eq_out, out, eq_number);
eq->irqn = pci_irq_vector(dev->pdev, vecidx);
eq->dev = dev;
eq->doorbell = priv->uar->map + MLX5_EQ_DOORBEL_OFFSET;
- err = request_irq(eq->irqn, handler, 0,
- eq_table->irq_info[vecidx].name, eq);
+ err = request_irq(eq->irqn, param->handler, 0,
+ eq_table->irq_info[vecidx].name, param->context);
if (err)
goto err_eq;
@@ -799,12 +815,19 @@ mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
return err;
}
-static int mlx5_destroy_unmap_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq)
+static int destroy_unmap_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq)
{
+ struct mlx5_eq_table *eq_table = dev->priv.eq_table;
+ struct mlx5_irq_info *irq_info;
int err;
+ irq_info = &eq_table->irq_info[eq->vecidx];
+
mlx5_debug_eq_remove(dev, eq);
- free_irq(eq->irqn, eq);
+
+ free_irq(eq->irqn, irq_info->context);
+ irq_info->context = NULL;
+
err = mlx5_cmd_destroy_eq(dev, eq->eqn);
if (err)
mlx5_core_warn(dev, "failed to destroy a previously created eq: eqn %d\n",
@@ -883,48 +906,38 @@ void mlx5_eq_table_cleanup(struct mlx5_core_dev *dev)
/* Async EQs */
-int mlx5_create_async_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
- int nent, u64 mask, const char *name, irq_handler_t handler)
+static int create_async_eq(struct mlx5_core_dev *dev, const char *name,
+ struct mlx5_eq *eq, struct mlx5_eq_param *param)
{
struct mlx5_eq_table *eq_table = dev->priv.eq_table;
- u8 vecdix;
int err;
mutex_lock(&eq_table->lock);
- if (eq_table->num_async_eqs >= MLX5_EQ_MAX_ASYNC_EQS) {
+ if (param->index >= MLX5_EQ_MAX_ASYNC_EQS) {
err = -ENOSPC;
goto unlock;
}
- vecdix = eq_table->num_async_eqs + 1;
-
- err = mlx5_create_map_eq(dev, eq, vecdix, nent, mask, name, handler);
- if (!err)
- eq_table->num_async_eqs++;
-
+ err = create_map_eq(dev, eq, name, param);
unlock:
mutex_unlock(&eq_table->lock);
return err;
}
-int mlx5_destroy_async_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq)
+static int destroy_async_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq)
{
struct mlx5_eq_table *eq_table = dev->priv.eq_table;
int err;
mutex_lock(&eq_table->lock);
- err = mlx5_destroy_unmap_eq(dev, eq);
- if (!err)
- eq_table->num_async_eqs--;
+ err = destroy_unmap_eq(dev, eq);
mutex_unlock(&eq_table->lock);
return err;
}
-static int create_async_eqs(struct mlx5_core_dev *dev)
+static u64 gather_async_events_mask(struct mlx5_core_dev *dev)
{
- struct mlx5_eq_table *table = dev->priv.eq_table;
u64 async_event_mask = MLX5_ASYNC_EVENT_MASK;
- int err;
if (MLX5_VPORT_MANAGER(dev))
async_event_mask |= (1ull << MLX5_EVENT_TYPE_NIC_VPORT_CHANGE);
@@ -953,9 +966,23 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
if (MLX5_CAP_MCAM_REG(dev, tracer_registers))
async_event_mask |= (1ull << MLX5_EVENT_TYPE_DEVICE_TRACER);
- err = mlx5_create_async_eq(dev, &table->cmd_eq, MLX5_NUM_CMD_EQE,
- 1ull << MLX5_EVENT_TYPE_CMD, "mlx5_cmd_eq",
- mlx5_eq_async_int);
+ return async_event_mask;
+}
+
+static int create_async_eqs(struct mlx5_core_dev *dev)
+{
+ struct mlx5_eq_table *table = dev->priv.eq_table;
+ struct mlx5_eq_param param = {};
+ int err;
+
+ param = (struct mlx5_eq_param) {
+ .index = MLX5_EQ_CMD_IDX,
+ .mask = 1ull << MLX5_EVENT_TYPE_CMD,
+ .nent = MLX5_NUM_CMD_EQE,
+ .context = &table->cmd_eq,
+ .handler = mlx5_eq_async_int,
+ };
+ err = create_async_eq(dev, "mlx5_cmd_eq", &table->cmd_eq, ¶m);
if (err) {
mlx5_core_warn(dev, "failed to create cmd EQ %d\n", err);
return err;
@@ -963,15 +990,27 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
mlx5_cmd_use_events(dev);
- err = mlx5_create_async_eq(dev, &table->async_eq, MLX5_NUM_ASYNC_EQE,
- async_event_mask, "mlx5_async_eq", mlx5_eq_async_int);
+ param = (struct mlx5_eq_param) {
+ .index = MLX5_EQ_ASYNC_IDX,
+ .mask = gather_async_events_mask(dev),
+ .nent = MLX5_NUM_ASYNC_EQE,
+ .context = &table->async_eq,
+ .handler = mlx5_eq_async_int,
+ };
+ err = create_async_eq(dev, "mlx5_async_eq", &table->async_eq, ¶m);
if (err) {
mlx5_core_warn(dev, "failed to create async EQ %d\n", err);
goto err1;
}
- err = mlx5_create_async_eq(dev, &table->pages_eq, /* TODO: sriov max_vf + */ 1,
- 1 << MLX5_EVENT_TYPE_PAGE_REQUEST, "mlx5_pages_eq", mlx5_eq_async_int);
+ param = (struct mlx5_eq_param) {
+ .index = MLX5_EQ_PAGEREQ_IDX,
+ .mask = 1 << MLX5_EVENT_TYPE_PAGE_REQUEST,
+ .nent = /* TODO: sriov max_vf + */ 1,
+ .context = &table->pages_eq,
+ .handler = mlx5_eq_async_int,
+ };
+ err = create_async_eq(dev, "mlx5_pages_eq", &table->pages_eq, ¶m);
if (err) {
mlx5_core_warn(dev, "failed to create pages EQ %d\n", err);
goto err2;
@@ -989,17 +1028,17 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
return err;
err3:
- mlx5_destroy_async_eq(dev, &table->pages_eq);
+ destroy_async_eq(dev, &table->pages_eq);
#else
return err;
#endif
err2:
- mlx5_destroy_async_eq(dev, &table->async_eq);
+ destroy_async_eq(dev, &table->async_eq);
err1:
mlx5_cmd_use_polling(dev);
- mlx5_destroy_async_eq(dev, &table->cmd_eq);
+ destroy_async_eq(dev, &table->cmd_eq);
return err;
}
@@ -1017,18 +1056,18 @@ static void destroy_async_eqs(struct mlx5_core_dev *dev)
}
#endif
- err = mlx5_destroy_async_eq(dev, &table->pages_eq);
+ err = destroy_async_eq(dev, &table->pages_eq);
if (err)
mlx5_core_err(dev, "failed to destroy pages eq, err(%d)\n",
err);
- err = mlx5_destroy_async_eq(dev, &table->async_eq);
+ err = destroy_async_eq(dev, &table->async_eq);
if (err)
mlx5_core_err(dev, "failed to destroy async eq, err(%d)\n",
err);
mlx5_cmd_use_polling(dev);
- err = mlx5_destroy_async_eq(dev, &table->cmd_eq);
+ err = destroy_async_eq(dev, &table->cmd_eq);
if (err)
mlx5_core_err(dev, "failed to destroy command eq, err(%d)\n",
err);
@@ -1049,6 +1088,77 @@ void mlx5_eq_synchronize_cmd_irq(struct mlx5_core_dev *dev)
synchronize_irq(dev->priv.eq_table->cmd_eq.irqn);
}
+/* Generic EQ API for mlx5_core consumers
+ * Needed For RDMA ODP EQ for now
+ */
+struct mlx5_eq *
+mlx5_eq_create_generic(struct mlx5_core_dev *dev, const char *name,
+ struct mlx5_eq_param *param)
+{
+ struct mlx5_eq *eq = kvzalloc(sizeof(*eq), GFP_KERNEL);
+ int err;
+
+ if (!eq)
+ return ERR_PTR(-ENOMEM);
+
+ err = create_async_eq(dev, name, eq, param);
+ if (err) {
+ kvfree(eq);
+ eq = ERR_PTR(err);
+ }
+
+ return eq;
+}
+EXPORT_SYMBOL(mlx5_eq_create_generic);
+
+int mlx5_eq_destroy_generic(struct mlx5_core_dev *dev, struct mlx5_eq *eq)
+{
+ int err;
+
+ if (IS_ERR(eq))
+ return -EINVAL;
+
+ err = destroy_async_eq(dev, eq);
+ if (err)
+ goto out;
+
+ kvfree(eq);
+out:
+ return err;
+}
+EXPORT_SYMBOL(mlx5_eq_destroy_generic);
+
+struct mlx5_eqe *mlx5_eq_get_eqe(struct mlx5_eq *eq, u32 cc)
+{
+ u32 ci = eq->cons_index + cc;
+ struct mlx5_eqe *eqe;
+
+ eqe = get_eqe(eq, ci & (eq->nent - 1));
+ eqe = ((eqe->owner & 1) ^ !!(ci & eq->nent)) ? NULL : eqe;
+ /* Make sure we read EQ entry contents after we've
+ * checked the ownership bit.
+ */
+ if (eqe)
+ dma_rmb();
+
+ return eqe;
+}
+EXPORT_SYMBOL(mlx5_eq_get_eqe);
+
+void mlx5_eq_update_ci(struct mlx5_eq *eq, u32 cc, bool arm)
+{
+ __be32 __iomem *addr = eq->doorbell + (arm ? 0 : 2);
+ u32 val;
+
+ eq->cons_index += cc;
+ val = (eq->cons_index & 0xffffff) | (eq->eqn << 24);
+
+ __raw_writel((__force u32)cpu_to_be32(val), addr);
+ /* We still want ordering, just not swabbing, so add a barrier */
+ mb();
+}
+EXPORT_SYMBOL(mlx5_eq_update_ci);
+
/* Completion EQs */
static int set_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
@@ -1127,7 +1237,7 @@ static void destroy_comp_eqs(struct mlx5_core_dev *dev)
#endif
list_for_each_entry_safe(eq, n, &table->comp_eqs_list, list) {
list_del(&eq->list);
- if (mlx5_destroy_unmap_eq(dev, &eq->core))
+ if (destroy_unmap_eq(dev, &eq->core))
mlx5_core_warn(dev, "failed to destroy comp EQ 0x%x\n",
eq->core.eqn);
tasklet_disable(&eq->tasklet_ctx.task);
@@ -1155,6 +1265,7 @@ static int create_comp_eqs(struct mlx5_core_dev *dev)
#endif
for (i = 0; i < ncomp_vec; i++) {
int vecidx = i + MLX5_EQ_VEC_COMP_BASE;
+ struct mlx5_eq_param param = {};
eq = kzalloc(sizeof(*eq), GFP_KERNEL);
if (!eq) {
@@ -1172,8 +1283,14 @@ static int create_comp_eqs(struct mlx5_core_dev *dev)
irq_cpu_rmap_add(table->rmap, pci_irq_vector(dev->pdev, vecidx));
#endif
snprintf(name, MLX5_MAX_IRQ_NAME, "mlx5_comp%d", i);
- err = mlx5_create_map_eq(dev, &eq->core, vecidx, nent, 0,
- name, mlx5_eq_comp_int);
+ param = (struct mlx5_eq_param) {
+ .index = vecidx,
+ .mask = 0,
+ .nent = nent,
+ .context = &eq->core,
+ .handler = mlx5_eq_comp_int
+ };
+ err = create_map_eq(dev, &eq->core, name, ¶m);
if (err) {
kfree(eq);
goto clean;
@@ -1257,7 +1374,7 @@ struct mlx5_eq_comp *mlx5_eqn2comp_eq(struct mlx5_core_dev *dev, int eqn)
void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev)
{
struct mlx5_eq_table *table = dev->priv.eq_table;
- struct mlx5_eq_comp *eq;
+ int i, max_eqs;
clear_comp_irqs_affinity_hints(dev);
@@ -1267,16 +1384,16 @@ void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev)
table->rmap = NULL;
}
#endif
- list_for_each_entry(eq, &table->comp_eqs_list, list)
- free_irq(eq->core.irqn, eq);
- free_irq(table->pages_eq.irqn, &table->pages_eq);
- free_irq(table->async_eq.irqn, &table->async_eq);
- free_irq(table->cmd_eq.irqn, &table->cmd_eq);
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- if (MLX5_CAP_GEN(dev, pg))
- free_irq(table->pfault_eq.core.irqn, &table->pfault_eq.core);
-#endif
+ mutex_lock(&table->lock); /* sync with create/destroy_async_eq */
+ max_eqs = table->num_comp_vectors + MLX5_EQ_VEC_COMP_BASE;
+ for (i = max_eqs - 1; i >= 0; i--) {
+ if (!table->irq_info[i].context)
+ continue;
+ free_irq(pci_irq_vector(dev->pdev, i), table->irq_info[i].context);
+ table->irq_info[i].context = NULL;
+ }
+ mutex_unlock(&table->lock);
pci_free_irq_vectors(dev->pdev);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
index 706d58383dbd..db32057ad054 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
@@ -7,11 +7,6 @@
#define MLX5_MAX_IRQ_NAME (32)
-enum {
- MLX5_EQ_MAX_ASYNC_EQS = 4, /* mlx5_core needs at least 3 */
- MLX5_EQ_VEC_COMP_BASE = MLX5_EQ_MAX_ASYNC_EQS,
-};
-
struct mlx5_eq_tasklet {
struct list_head list;
struct list_head process_list;
@@ -31,6 +26,7 @@ struct mlx5_eq {
u32 cons_index;
struct mlx5_frag_buf buf;
int size;
+ unsigned int vecidx;
unsigned int irqn;
u8 eqn;
int nent;
@@ -44,7 +40,7 @@ struct mlx5_eq_comp {
};
struct mlx5_eq_pagefault {
- struct mlx5_eq core; /* Must be first */
+ struct mlx5_eq *core;
struct work_struct work;
spinlock_t lock; /* Pagefaults spinlock */
struct workqueue_struct *wq;
@@ -55,10 +51,6 @@ int mlx5_eq_table_init(struct mlx5_core_dev *dev);
void mlx5_eq_table_cleanup(struct mlx5_core_dev *dev);
int mlx5_eq_table_create(struct mlx5_core_dev *dev);
void mlx5_eq_table_destroy(struct mlx5_core_dev *dev);
-int mlx5_create_async_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
- int nent, u64 mask, const char *name,
- irq_handler_t handler);
-int mlx5_destroy_async_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
int mlx5_eq_add_cq(struct mlx5_eq *eq, struct mlx5_core_cq *cq);
int mlx5_eq_del_cq(struct mlx5_eq *eq, struct mlx5_core_cq *cq);
diff --git a/include/linux/mlx5/eq.h b/include/linux/mlx5/eq.h
new file mode 100644
index 000000000000..c733673ba5f6
--- /dev/null
+++ b/include/linux/mlx5/eq.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2018 Mellanox Technologies. */
+
+#ifndef MLX5_CORE_EQ_H
+#define MLX5_CORE_EQ_H
+
+#include <linux/mlx5/driver.h>
+
+enum {
+ MLX5_EQ_PAGEREQ_IDX = 0,
+ MLX5_EQ_CMD_IDX = 1,
+ MLX5_EQ_ASYNC_IDX = 2,
+ /* reserved to be used by mlx5_core ulps (mlx5e/mlx5_ib) */
+ MLX5_EQ_PFAULT_IDX = 3,
+ MLX5_EQ_MAX_ASYNC_EQS,
+ /* completion eqs vector indices start here */
+ MLX5_EQ_VEC_COMP_BASE = MLX5_EQ_MAX_ASYNC_EQS,
+};
+
+struct mlx5_eq;
+
+struct mlx5_eq_param {
+ u8 index;
+ int nent;
+ u64 mask;
+ void *context;
+ irq_handler_t handler;
+};
+
+struct mlx5_eq *
+mlx5_eq_create_generic(struct mlx5_core_dev *dev, const char *name,
+ struct mlx5_eq_param *param);
+int
+mlx5_eq_destroy_generic(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
+
+struct mlx5_eqe *mlx5_eq_get_eqe(struct mlx5_eq *eq, u32 cc);
+void mlx5_eq_update_ci(struct mlx5_eq *eq, u32 cc, bool arm);
+
+#endif /* MLX5_CORE_EQ_H */
--
2.19.1
^ permalink raw reply related
* [PATCH V2 mlx5-next 09/12] net/mlx5: EQ, Different EQ types
From: Saeed Mahameed @ 2018-11-19 18:52 UTC (permalink / raw)
To: Leon Romanovsky, saeedm; +Cc: netdev, linux-rdma, Jason Gunthorpe
In-Reply-To: <20181119185242.21961-1-saeedm@mellanox.com>
In mlx5 we have three types of usages for EQs,
1. Asynchronous EQs, used internally by mlx5 core for
a. FW command completions
b. FW page requests
c. one EQ for all other Asynchronous events
2. Completion EQs, used for CQ completion (we create one per core)
3. *Special type of EQ (page fault) used for RDMA on demand paging
(ODP).
*The 3rd type shouldn't be special at least in mlx5 core, it is yet
another async events EQ with specific use case, it will be removed in
the next two patches, and will completely move its logic to mlx5_ib,
as it is rdma specific.
In this patch we remove use case (eq type) specific fields from
struct mlx5_eq into a new eq type specific structures.
struct mlx5_eq_async;
truct mlx5_eq_comp;
struct mlx5_eq_pagefault;
Separate between their type specific flows.
In the future we will allow users to create there own generic EQs.
for now we will allow only one for ODP in next patches.
We will introduce event listeners registration API for those who
want to receive mlx5 async events.
After that mlx5 eq handling will be clean from feature/user specific
handling.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/cq.c | 10 +-
.../net/ethernet/mellanox/mlx5/core/en_main.c | 8 +-
drivers/net/ethernet/mellanox/mlx5/core/eq.c | 376 +++++++++++-------
.../net/ethernet/mellanox/mlx5/core/eswitch.c | 2 +-
.../net/ethernet/mellanox/mlx5/core/health.c | 2 +-
.../net/ethernet/mellanox/mlx5/core/lib/eq.h | 53 ++-
.../net/ethernet/mellanox/mlx5/core/main.c | 2 +-
.../ethernet/mellanox/mlx5/core/mlx5_core.h | 4 -
include/linux/mlx5/cq.h | 2 +-
include/linux/mlx5/driver.h | 10 +-
10 files changed, 270 insertions(+), 199 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cq.c b/drivers/net/ethernet/mellanox/mlx5/core/cq.c
index 6e55d2f37c6d..713a17ee3751 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/cq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/cq.c
@@ -93,10 +93,10 @@ int mlx5_core_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
u32 dout[MLX5_ST_SZ_DW(destroy_cq_out)];
u32 out[MLX5_ST_SZ_DW(create_cq_out)];
u32 din[MLX5_ST_SZ_DW(destroy_cq_in)];
- struct mlx5_eq *eq;
+ struct mlx5_eq_comp *eq;
int err;
- eq = mlx5_eqn2eq(dev, eqn);
+ eq = mlx5_eqn2comp_eq(dev, eqn);
if (IS_ERR(eq))
return PTR_ERR(eq);
@@ -120,7 +120,7 @@ int mlx5_core_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
INIT_LIST_HEAD(&cq->tasklet_ctx.list);
/* Add to comp EQ CQ tree to recv comp events */
- err = mlx5_eq_add_cq(eq, cq);
+ err = mlx5_eq_add_cq(&eq->core, cq);
if (err)
goto err_cmd;
@@ -140,7 +140,7 @@ int mlx5_core_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
return 0;
err_cq_add:
- mlx5_eq_del_cq(eq, cq);
+ mlx5_eq_del_cq(&eq->core, cq);
err_cmd:
memset(din, 0, sizeof(din));
memset(dout, 0, sizeof(dout));
@@ -162,7 +162,7 @@ int mlx5_core_destroy_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq)
if (err)
return err;
- err = mlx5_eq_del_cq(cq->eq, cq);
+ err = mlx5_eq_del_cq(&cq->eq->core, cq);
if (err)
return err;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index c23caade31bf..0d495a6b3949 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -320,7 +320,7 @@ static void mlx5e_enable_async_events(struct mlx5e_priv *priv)
static void mlx5e_disable_async_events(struct mlx5e_priv *priv)
{
clear_bit(MLX5E_STATE_ASYNC_EVENTS_ENABLED, &priv->state);
- synchronize_irq(pci_irq_vector(priv->mdev->pdev, MLX5_EQ_VEC_ASYNC));
+ mlx5_eq_synchronize_async_irq(priv->mdev);
}
static inline void mlx5e_build_umr_wqe(struct mlx5e_rq *rq,
@@ -4117,17 +4117,17 @@ static netdev_features_t mlx5e_features_check(struct sk_buff *skb,
static bool mlx5e_tx_timeout_eq_recover(struct net_device *dev,
struct mlx5e_txqsq *sq)
{
- struct mlx5_eq *eq = sq->cq.mcq.eq;
+ struct mlx5_eq_comp *eq = sq->cq.mcq.eq;
u32 eqe_count;
netdev_err(dev, "EQ 0x%x: Cons = 0x%x, irqn = 0x%x\n",
- eq->eqn, eq->cons_index, eq->irqn);
+ eq->core.eqn, eq->core.cons_index, eq->core.irqn);
eqe_count = mlx5_eq_poll_irq_disabled(eq);
if (!eqe_count)
return false;
- netdev_err(dev, "Recover %d eqes on EQ 0x%x\n", eqe_count, eq->eqn);
+ netdev_err(dev, "Recover %d eqes on EQ 0x%x\n", eqe_count, eq->core.eqn);
sq->channel->stats->eq_rearm++;
return true;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
index 32ce20221c44..252c9f0569b1 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
@@ -72,13 +72,16 @@ struct mlx5_irq_info {
};
struct mlx5_eq_table {
- struct list_head comp_eqs_list;
- struct mlx5_eq pages_eq;
- struct mlx5_eq async_eq;
- struct mlx5_eq cmd_eq;
+ struct list_head comp_eqs_list;
+ struct mlx5_eq pages_eq;
+ struct mlx5_eq async_eq;
+ struct mlx5_eq cmd_eq;
+
#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- struct mlx5_eq pfault_eq;
+ struct mlx5_eq_pagefault pfault_eq;
#endif
+ struct mutex lock; /* sync async eqs creations */
+ u8 num_async_eqs;
int num_comp_vectors;
struct mlx5_irq_info *irq_info;
#ifdef CONFIG_RFS_ACCEL
@@ -224,24 +227,24 @@ static void eqe_pf_action(struct work_struct *work)
struct mlx5_pagefault *pfault = container_of(work,
struct mlx5_pagefault,
work);
- struct mlx5_eq *eq = pfault->eq;
+ struct mlx5_eq_pagefault *eq = pfault->eq;
- mlx5_core_page_fault(eq->dev, pfault);
- mempool_free(pfault, eq->pf_ctx.pool);
+ mlx5_core_page_fault(eq->core.dev, pfault);
+ mempool_free(pfault, eq->pool);
}
-static void eq_pf_process(struct mlx5_eq *eq)
+static void eq_pf_process(struct mlx5_eq_pagefault *eq)
{
- struct mlx5_core_dev *dev = eq->dev;
+ struct mlx5_core_dev *dev = eq->core.dev;
struct mlx5_eqe_page_fault *pf_eqe;
struct mlx5_pagefault *pfault;
struct mlx5_eqe *eqe;
int set_ci = 0;
- while ((eqe = next_eqe_sw(eq))) {
- pfault = mempool_alloc(eq->pf_ctx.pool, GFP_ATOMIC);
+ while ((eqe = next_eqe_sw(&eq->core))) {
+ pfault = mempool_alloc(eq->pool, GFP_ATOMIC);
if (!pfault) {
- schedule_work(&eq->pf_ctx.work);
+ schedule_work(&eq->work);
break;
}
@@ -311,30 +314,30 @@ static void eq_pf_process(struct mlx5_eq *eq)
pfault->eq = eq;
INIT_WORK(&pfault->work, eqe_pf_action);
- queue_work(eq->pf_ctx.wq, &pfault->work);
+ queue_work(eq->wq, &pfault->work);
- ++eq->cons_index;
+ ++eq->core.cons_index;
++set_ci;
if (unlikely(set_ci >= MLX5_NUM_SPARE_EQE)) {
- eq_update_ci(eq, 0);
+ eq_update_ci(&eq->core, 0);
set_ci = 0;
}
}
- eq_update_ci(eq, 1);
+ eq_update_ci(&eq->core, 1);
}
static irqreturn_t mlx5_eq_pf_int(int irq, void *eq_ptr)
{
- struct mlx5_eq *eq = eq_ptr;
+ struct mlx5_eq_pagefault *eq = eq_ptr;
unsigned long flags;
- if (spin_trylock_irqsave(&eq->pf_ctx.lock, flags)) {
+ if (spin_trylock_irqsave(&eq->lock, flags)) {
eq_pf_process(eq);
- spin_unlock_irqrestore(&eq->pf_ctx.lock, flags);
+ spin_unlock_irqrestore(&eq->lock, flags);
} else {
- schedule_work(&eq->pf_ctx.work);
+ schedule_work(&eq->work);
}
return IRQ_HANDLED;
@@ -352,35 +355,61 @@ static void mempool_refill(mempool_t *pool)
static void eq_pf_action(struct work_struct *work)
{
- struct mlx5_eq *eq = container_of(work, struct mlx5_eq, pf_ctx.work);
+ struct mlx5_eq_pagefault *eq =
+ container_of(work, struct mlx5_eq_pagefault, work);
- mempool_refill(eq->pf_ctx.pool);
+ mempool_refill(eq->pool);
- spin_lock_irq(&eq->pf_ctx.lock);
+ spin_lock_irq(&eq->lock);
eq_pf_process(eq);
- spin_unlock_irq(&eq->pf_ctx.lock);
+ spin_unlock_irq(&eq->lock);
}
-static int init_pf_ctx(struct mlx5_eq_pagefault *pf_ctx, const char *name)
+static int
+create_pf_eq(struct mlx5_core_dev *dev, struct mlx5_eq_pagefault *eq)
{
- spin_lock_init(&pf_ctx->lock);
- INIT_WORK(&pf_ctx->work, eq_pf_action);
+ int err;
- pf_ctx->wq = alloc_workqueue(name,
- WQ_HIGHPRI | WQ_UNBOUND | WQ_MEM_RECLAIM,
- MLX5_NUM_CMD_EQE);
- if (!pf_ctx->wq)
+ spin_lock_init(&eq->lock);
+ INIT_WORK(&eq->work, eq_pf_action);
+
+ eq->pool = mempool_create_kmalloc_pool(MLX5_NUM_PF_DRAIN,
+ sizeof(struct mlx5_pagefault));
+ if (!eq->pool)
return -ENOMEM;
- pf_ctx->pool = mempool_create_kmalloc_pool
- (MLX5_NUM_PF_DRAIN, sizeof(struct mlx5_pagefault));
- if (!pf_ctx->pool)
+ eq->wq = alloc_workqueue("mlx5_page_fault",
+ WQ_HIGHPRI | WQ_UNBOUND | WQ_MEM_RECLAIM,
+ MLX5_NUM_CMD_EQE);
+ if (!eq->wq) {
+ err = -ENOMEM;
+ goto err_mempool;
+ }
+
+ err = mlx5_create_async_eq(dev, &eq->core, MLX5_NUM_ASYNC_EQE,
+ 1 << MLX5_EVENT_TYPE_PAGE_FAULT,
+ "mlx5_page_fault_eq", mlx5_eq_pf_int);
+ if (err)
goto err_wq;
return 0;
err_wq:
- destroy_workqueue(pf_ctx->wq);
- return -ENOMEM;
+ destroy_workqueue(eq->wq);
+err_mempool:
+ mempool_destroy(eq->pool);
+ return err;
+}
+
+static int destroy_pf_eq(struct mlx5_core_dev *dev, struct mlx5_eq_pagefault *eq)
+{
+ int err;
+
+ err = mlx5_destroy_async_eq(dev, &eq->core);
+ cancel_work_sync(&eq->work);
+ destroy_workqueue(eq->wq);
+ mempool_destroy(eq->pool);
+
+ return err;
}
int mlx5_core_page_fault_resume(struct mlx5_core_dev *dev, u32 token,
@@ -444,37 +473,88 @@ static struct mlx5_core_cq *mlx5_eq_cq_get(struct mlx5_eq *eq, u32 cqn)
return cq;
}
-static void mlx5_eq_cq_completion(struct mlx5_eq *eq, u32 cqn)
+static void mlx5_eq_cq_event(struct mlx5_eq *eq, u32 cqn, int event_type)
{
struct mlx5_core_cq *cq = mlx5_eq_cq_get(eq, cqn);
if (unlikely(!cq)) {
- mlx5_core_warn(eq->dev, "Completion event for bogus CQ 0x%x\n", cqn);
+ mlx5_core_warn(eq->dev, "Async event for bogus CQ 0x%x\n", cqn);
return;
}
- ++cq->arm_sn;
-
- cq->comp(cq);
+ cq->event(cq, event_type);
mlx5_cq_put(cq);
}
-static void mlx5_eq_cq_event(struct mlx5_eq *eq, u32 cqn, int event_type)
+static irqreturn_t mlx5_eq_comp_int(int irq, void *eq_ptr)
{
- struct mlx5_core_cq *cq = mlx5_eq_cq_get(eq, cqn);
+ struct mlx5_eq_comp *eq_comp = eq_ptr;
+ struct mlx5_eq *eq = eq_ptr;
+ struct mlx5_eqe *eqe;
+ int set_ci = 0;
+ u32 cqn = -1;
- if (unlikely(!cq)) {
- mlx5_core_warn(eq->dev, "Async event for bogus CQ 0x%x\n", cqn);
- return;
+ while ((eqe = next_eqe_sw(eq))) {
+ struct mlx5_core_cq *cq;
+ /* Make sure we read EQ entry contents after we've
+ * checked the ownership bit.
+ */
+ dma_rmb();
+ /* Assume (eqe->type) is always MLX5_EVENT_TYPE_COMP */
+ cqn = be32_to_cpu(eqe->data.comp.cqn) & 0xffffff;
+
+ cq = mlx5_eq_cq_get(eq, cqn);
+ if (likely(cq)) {
+ ++cq->arm_sn;
+ cq->comp(cq);
+ mlx5_cq_put(cq);
+ } else {
+ mlx5_core_warn(eq->dev, "Completion event for bogus CQ 0x%x\n", cqn);
+ }
+
+ ++eq->cons_index;
+ ++set_ci;
+
+ /* The HCA will think the queue has overflowed if we
+ * don't tell it we've been processing events. We
+ * create our EQs with MLX5_NUM_SPARE_EQE extra
+ * entries, so we must update our consumer index at
+ * least that often.
+ */
+ if (unlikely(set_ci >= MLX5_NUM_SPARE_EQE)) {
+ eq_update_ci(eq, 0);
+ set_ci = 0;
+ }
}
- cq->event(cq, event_type);
+ eq_update_ci(eq, 1);
- mlx5_cq_put(cq);
+ if (cqn != -1)
+ tasklet_schedule(&eq_comp->tasklet_ctx.task);
+
+ return IRQ_HANDLED;
}
-static irqreturn_t mlx5_eq_int(int irq, void *eq_ptr)
+/* Some architectures don't latch interrupts when they are disabled, so using
+ * mlx5_eq_poll_irq_disabled could end up losing interrupts while trying to
+ * avoid losing them. It is not recommended to use it, unless this is the last
+ * resort.
+ */
+u32 mlx5_eq_poll_irq_disabled(struct mlx5_eq_comp *eq)
+{
+ u32 count_eqe;
+
+ disable_irq(eq->core.irqn);
+ count_eqe = eq->core.cons_index;
+ mlx5_eq_comp_int(eq->core.irqn, eq);
+ count_eqe = eq->core.cons_index - count_eqe;
+ enable_irq(eq->core.irqn);
+
+ return count_eqe;
+}
+
+static irqreturn_t mlx5_eq_async_int(int irq, void *eq_ptr)
{
struct mlx5_eq *eq = eq_ptr;
struct mlx5_core_dev *dev = eq->dev;
@@ -494,10 +574,6 @@ static irqreturn_t mlx5_eq_int(int irq, void *eq_ptr)
mlx5_core_dbg(eq->dev, "eqn %d, eqe type %s\n",
eq->eqn, eqe_type_str(eqe->type));
switch (eqe->type) {
- case MLX5_EVENT_TYPE_COMP:
- cqn = be32_to_cpu(eqe->data.comp.cqn) & 0xffffff;
- mlx5_eq_cq_completion(eq, cqn);
- break;
case MLX5_EVENT_TYPE_DCT_DRAINED:
rsn = be32_to_cpu(eqe->data.dct.dctn) & 0xffffff;
rsn |= (MLX5_RES_DCT << MLX5_USER_INDEX_LEN);
@@ -619,30 +695,9 @@ static irqreturn_t mlx5_eq_int(int irq, void *eq_ptr)
eq_update_ci(eq, 1);
- if (cqn != -1)
- tasklet_schedule(&eq->tasklet_ctx.task);
-
return IRQ_HANDLED;
}
-/* Some architectures don't latch interrupts when they are disabled, so using
- * mlx5_eq_poll_irq_disabled could end up losing interrupts while trying to
- * avoid losing them. It is not recommended to use it, unless this is the last
- * resort.
- */
-u32 mlx5_eq_poll_irq_disabled(struct mlx5_eq *eq)
-{
- u32 count_eqe;
-
- disable_irq(eq->irqn);
- count_eqe = eq->cons_index;
- mlx5_eq_int(eq->irqn, eq);
- count_eqe = eq->cons_index - count_eqe;
- enable_irq(eq->irqn);
-
- return count_eqe;
-}
-
static void init_eq_buf(struct mlx5_eq *eq)
{
struct mlx5_eqe *eqe;
@@ -656,13 +711,12 @@ static void init_eq_buf(struct mlx5_eq *eq)
static int
mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
- int nent, u64 mask, const char *name, enum mlx5_eq_type type)
+ int nent, u64 mask, const char *name, irq_handler_t handler)
{
struct mlx5_eq_table *eq_table = dev->priv.eq_table;
struct mlx5_cq_table *cq_table = &eq->cq_table;
u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {0};
struct mlx5_priv *priv = &dev->priv;
- irq_handler_t handler;
__be64 *pas;
void *eqc;
int inlen;
@@ -674,20 +728,12 @@ mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
spin_lock_init(&cq_table->lock);
INIT_RADIX_TREE(&cq_table->tree, GFP_ATOMIC);
- eq->type = type;
eq->nent = roundup_pow_of_two(nent + MLX5_NUM_SPARE_EQE);
eq->cons_index = 0;
err = mlx5_buf_alloc(dev, eq->nent * MLX5_EQE_SIZE, &eq->buf);
if (err)
return err;
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- if (type == MLX5_EQ_TYPE_PF)
- handler = mlx5_eq_pf_int;
- else
-#endif
- handler = mlx5_eq_int;
-
init_eq_buf(eq);
inlen = MLX5_ST_SZ_BYTES(create_eq_in) +
@@ -732,21 +778,6 @@ mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
if (err)
goto err_irq;
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- if (type == MLX5_EQ_TYPE_PF) {
- err = init_pf_ctx(&eq->pf_ctx, name);
- if (err)
- goto err_irq;
- } else
-#endif
- {
- INIT_LIST_HEAD(&eq->tasklet_ctx.list);
- INIT_LIST_HEAD(&eq->tasklet_ctx.process_list);
- spin_lock_init(&eq->tasklet_ctx.lock);
- tasklet_init(&eq->tasklet_ctx.task, mlx5_cq_tasklet_cb,
- (unsigned long)&eq->tasklet_ctx);
- }
-
/* EQs are created in ARMED state
*/
eq_update_ci(eq, 1);
@@ -780,15 +811,6 @@ static int mlx5_destroy_unmap_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq)
eq->eqn);
synchronize_irq(eq->irqn);
- if (eq->type == MLX5_EQ_TYPE_COMP) {
- tasklet_disable(&eq->tasklet_ctx.task);
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- } else if (eq->type == MLX5_EQ_TYPE_PF) {
- cancel_work_sync(&eq->pf_ctx.work);
- destroy_workqueue(eq->pf_ctx.wq);
- mempool_destroy(eq->pf_ctx.pool);
-#endif
- }
mlx5_buf_free(dev, &eq->buf);
return err;
@@ -841,8 +863,15 @@ int mlx5_eq_table_init(struct mlx5_core_dev *dev)
err = mlx5_eq_debugfs_init(dev);
if (err)
- kvfree(eq_table);
+ goto kvfree_eq_table;
+ mutex_init(&eq_table->lock);
+
+ return 0;
+
+kvfree_eq_table:
+ kvfree(eq_table);
+ dev->priv.eq_table = NULL;
return err;
}
@@ -854,6 +883,43 @@ void mlx5_eq_table_cleanup(struct mlx5_core_dev *dev)
/* Async EQs */
+int mlx5_create_async_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
+ int nent, u64 mask, const char *name, irq_handler_t handler)
+{
+ struct mlx5_eq_table *eq_table = dev->priv.eq_table;
+ u8 vecdix;
+ int err;
+
+ mutex_lock(&eq_table->lock);
+ if (eq_table->num_async_eqs >= MLX5_EQ_MAX_ASYNC_EQS) {
+ err = -ENOSPC;
+ goto unlock;
+ }
+
+ vecdix = eq_table->num_async_eqs + 1;
+
+ err = mlx5_create_map_eq(dev, eq, vecdix, nent, mask, name, handler);
+ if (!err)
+ eq_table->num_async_eqs++;
+
+unlock:
+ mutex_unlock(&eq_table->lock);
+ return err;
+}
+
+int mlx5_destroy_async_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq)
+{
+ struct mlx5_eq_table *eq_table = dev->priv.eq_table;
+ int err;
+
+ mutex_lock(&eq_table->lock);
+ err = mlx5_destroy_unmap_eq(dev, eq);
+ if (!err)
+ eq_table->num_async_eqs--;
+ mutex_unlock(&eq_table->lock);
+ return err;
+}
+
static int create_async_eqs(struct mlx5_core_dev *dev)
{
struct mlx5_eq_table *table = dev->priv.eq_table;
@@ -887,9 +953,9 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
if (MLX5_CAP_MCAM_REG(dev, tracer_registers))
async_event_mask |= (1ull << MLX5_EVENT_TYPE_DEVICE_TRACER);
- err = mlx5_create_map_eq(dev, &table->cmd_eq, MLX5_EQ_VEC_CMD,
- MLX5_NUM_CMD_EQE, 1ull << MLX5_EVENT_TYPE_CMD,
- "mlx5_cmd_eq", MLX5_EQ_TYPE_ASYNC);
+ err = mlx5_create_async_eq(dev, &table->cmd_eq, MLX5_NUM_CMD_EQE,
+ 1ull << MLX5_EVENT_TYPE_CMD, "mlx5_cmd_eq",
+ mlx5_eq_async_int);
if (err) {
mlx5_core_warn(dev, "failed to create cmd EQ %d\n", err);
return err;
@@ -897,19 +963,15 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
mlx5_cmd_use_events(dev);
- err = mlx5_create_map_eq(dev, &table->async_eq, MLX5_EQ_VEC_ASYNC,
- MLX5_NUM_ASYNC_EQE, async_event_mask,
- "mlx5_async_eq", MLX5_EQ_TYPE_ASYNC);
+ err = mlx5_create_async_eq(dev, &table->async_eq, MLX5_NUM_ASYNC_EQE,
+ async_event_mask, "mlx5_async_eq", mlx5_eq_async_int);
if (err) {
mlx5_core_warn(dev, "failed to create async EQ %d\n", err);
goto err1;
}
- err = mlx5_create_map_eq(dev, &table->pages_eq,
- MLX5_EQ_VEC_PAGES,
- /* TODO: sriov max_vf + */ 1,
- 1 << MLX5_EVENT_TYPE_PAGE_REQUEST, "mlx5_pages_eq",
- MLX5_EQ_TYPE_ASYNC);
+ err = mlx5_create_async_eq(dev, &table->pages_eq, /* TODO: sriov max_vf + */ 1,
+ 1 << MLX5_EVENT_TYPE_PAGE_REQUEST, "mlx5_pages_eq", mlx5_eq_async_int);
if (err) {
mlx5_core_warn(dev, "failed to create pages EQ %d\n", err);
goto err2;
@@ -917,12 +979,7 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
if (MLX5_CAP_GEN(dev, pg)) {
- err = mlx5_create_map_eq(dev, &table->pfault_eq,
- MLX5_EQ_VEC_PFAULT,
- MLX5_NUM_ASYNC_EQE,
- 1 << MLX5_EVENT_TYPE_PAGE_FAULT,
- "mlx5_page_fault_eq",
- MLX5_EQ_TYPE_PF);
+ err = create_pf_eq(dev, &table->pfault_eq);
if (err) {
mlx5_core_warn(dev, "failed to create page fault EQ %d\n",
err);
@@ -932,17 +989,17 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
return err;
err3:
- mlx5_destroy_unmap_eq(dev, &table->pages_eq);
+ mlx5_destroy_async_eq(dev, &table->pages_eq);
#else
return err;
#endif
err2:
- mlx5_destroy_unmap_eq(dev, &table->async_eq);
+ mlx5_destroy_async_eq(dev, &table->async_eq);
err1:
mlx5_cmd_use_polling(dev);
- mlx5_destroy_unmap_eq(dev, &table->cmd_eq);
+ mlx5_destroy_async_eq(dev, &table->cmd_eq);
return err;
}
@@ -953,25 +1010,25 @@ static void destroy_async_eqs(struct mlx5_core_dev *dev)
#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
if (MLX5_CAP_GEN(dev, pg)) {
- err = mlx5_destroy_unmap_eq(dev, &table->pfault_eq);
+ err = destroy_pf_eq(dev, &table->pfault_eq);
if (err)
mlx5_core_err(dev, "failed to destroy page fault eq, err(%d)\n",
err);
}
#endif
- err = mlx5_destroy_unmap_eq(dev, &table->pages_eq);
+ err = mlx5_destroy_async_eq(dev, &table->pages_eq);
if (err)
mlx5_core_err(dev, "failed to destroy pages eq, err(%d)\n",
err);
- err = mlx5_destroy_unmap_eq(dev, &table->async_eq);
+ err = mlx5_destroy_async_eq(dev, &table->async_eq);
if (err)
mlx5_core_err(dev, "failed to destroy async eq, err(%d)\n",
err);
mlx5_cmd_use_polling(dev);
- err = mlx5_destroy_unmap_eq(dev, &table->cmd_eq);
+ err = mlx5_destroy_async_eq(dev, &table->cmd_eq);
if (err)
mlx5_core_err(dev, "failed to destroy command eq, err(%d)\n",
err);
@@ -982,6 +1039,16 @@ struct mlx5_eq *mlx5_get_async_eq(struct mlx5_core_dev *dev)
return &dev->priv.eq_table->async_eq;
}
+void mlx5_eq_synchronize_async_irq(struct mlx5_core_dev *dev)
+{
+ synchronize_irq(dev->priv.eq_table->async_eq.irqn);
+}
+
+void mlx5_eq_synchronize_cmd_irq(struct mlx5_core_dev *dev)
+{
+ synchronize_irq(dev->priv.eq_table->cmd_eq.irqn);
+}
+
/* Completion EQs */
static int set_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
@@ -1048,7 +1115,7 @@ static void clear_comp_irqs_affinity_hints(struct mlx5_core_dev *mdev)
static void destroy_comp_eqs(struct mlx5_core_dev *dev)
{
struct mlx5_eq_table *table = dev->priv.eq_table;
- struct mlx5_eq *eq, *n;
+ struct mlx5_eq_comp *eq, *n;
clear_comp_irqs_affinity_hints(dev);
@@ -1060,9 +1127,10 @@ static void destroy_comp_eqs(struct mlx5_core_dev *dev)
#endif
list_for_each_entry_safe(eq, n, &table->comp_eqs_list, list) {
list_del(&eq->list);
- if (mlx5_destroy_unmap_eq(dev, eq))
- mlx5_core_warn(dev, "failed to destroy EQ 0x%x\n",
- eq->eqn);
+ if (mlx5_destroy_unmap_eq(dev, &eq->core))
+ mlx5_core_warn(dev, "failed to destroy comp EQ 0x%x\n",
+ eq->core.eqn);
+ tasklet_disable(&eq->tasklet_ctx.task);
kfree(eq);
}
}
@@ -1071,7 +1139,7 @@ static int create_comp_eqs(struct mlx5_core_dev *dev)
{
struct mlx5_eq_table *table = dev->priv.eq_table;
char name[MLX5_MAX_IRQ_NAME];
- struct mlx5_eq *eq;
+ struct mlx5_eq_comp *eq;
int ncomp_vec;
int nent;
int err;
@@ -1094,17 +1162,23 @@ static int create_comp_eqs(struct mlx5_core_dev *dev)
goto clean;
}
+ INIT_LIST_HEAD(&eq->tasklet_ctx.list);
+ INIT_LIST_HEAD(&eq->tasklet_ctx.process_list);
+ spin_lock_init(&eq->tasklet_ctx.lock);
+ tasklet_init(&eq->tasklet_ctx.task, mlx5_cq_tasklet_cb,
+ (unsigned long)&eq->tasklet_ctx);
+
#ifdef CONFIG_RFS_ACCEL
irq_cpu_rmap_add(table->rmap, pci_irq_vector(dev->pdev, vecidx));
#endif
snprintf(name, MLX5_MAX_IRQ_NAME, "mlx5_comp%d", i);
- err = mlx5_create_map_eq(dev, eq, vecidx, nent, 0,
- name, MLX5_EQ_TYPE_COMP);
+ err = mlx5_create_map_eq(dev, &eq->core, vecidx, nent, 0,
+ name, mlx5_eq_comp_int);
if (err) {
kfree(eq);
goto clean;
}
- mlx5_core_dbg(dev, "allocated completion EQN %d\n", eq->eqn);
+ mlx5_core_dbg(dev, "allocated completion EQN %d\n", eq->core.eqn);
/* add tail, to keep the list ordered, for mlx5_vector2eqn to work */
list_add_tail(&eq->list, &table->comp_eqs_list);
}
@@ -1126,14 +1200,14 @@ int mlx5_vector2eqn(struct mlx5_core_dev *dev, int vector, int *eqn,
unsigned int *irqn)
{
struct mlx5_eq_table *table = dev->priv.eq_table;
- struct mlx5_eq *eq, *n;
+ struct mlx5_eq_comp *eq, *n;
int err = -ENOENT;
int i = 0;
list_for_each_entry_safe(eq, n, &table->comp_eqs_list, list) {
if (i++ == vector) {
- *eqn = eq->eqn;
- *irqn = eq->irqn;
+ *eqn = eq->core.eqn;
+ *irqn = eq->core.irqn;
err = 0;
break;
}
@@ -1166,13 +1240,13 @@ struct cpu_rmap *mlx5_eq_table_get_rmap(struct mlx5_core_dev *dev)
#endif
}
-struct mlx5_eq *mlx5_eqn2eq(struct mlx5_core_dev *dev, int eqn)
+struct mlx5_eq_comp *mlx5_eqn2comp_eq(struct mlx5_core_dev *dev, int eqn)
{
struct mlx5_eq_table *table = dev->priv.eq_table;
- struct mlx5_eq *eq;
+ struct mlx5_eq_comp *eq;
list_for_each_entry(eq, &table->comp_eqs_list, list) {
- if (eq->eqn == eqn)
+ if (eq->core.eqn == eqn)
return eq;
}
@@ -1183,7 +1257,7 @@ struct mlx5_eq *mlx5_eqn2eq(struct mlx5_core_dev *dev, int eqn)
void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev)
{
struct mlx5_eq_table *table = dev->priv.eq_table;
- struct mlx5_eq *eq;
+ struct mlx5_eq_comp *eq;
clear_comp_irqs_affinity_hints(dev);
@@ -1194,14 +1268,14 @@ void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev)
}
#endif
list_for_each_entry(eq, &table->comp_eqs_list, list)
- free_irq(eq->irqn, eq);
+ free_irq(eq->core.irqn, eq);
free_irq(table->pages_eq.irqn, &table->pages_eq);
free_irq(table->async_eq.irqn, &table->async_eq);
free_irq(table->cmd_eq.irqn, &table->cmd_eq);
#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
if (MLX5_CAP_GEN(dev, pg))
- free_irq(table->pfault_eq.irqn, &table->pfault_eq);
+ free_irq(table->pfault_eq.core.irqn, &table->pfault_eq.core);
#endif
pci_free_irq_vectors(dev->pdev);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
index 324606227b1a..2346b6ba3d54 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
@@ -1568,7 +1568,7 @@ static void esw_disable_vport(struct mlx5_eswitch *esw, int vport_num)
/* Mark this vport as disabled to discard new events */
vport->enabled = false;
- synchronize_irq(pci_irq_vector(esw->dev->pdev, MLX5_EQ_VEC_ASYNC));
+ mlx5_eq_synchronize_async_irq(esw->dev);
/* Wait for current already scheduled events to complete */
flush_workqueue(esw->work_queue);
/* Disable events from this vport */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
index b5be6f0b9ed5..066883003aea 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
@@ -85,7 +85,7 @@ static void trigger_cmd_completions(struct mlx5_core_dev *dev)
u64 vector;
/* wait for pending handlers to complete */
- synchronize_irq(pci_irq_vector(dev->pdev, MLX5_EQ_VEC_CMD));
+ mlx5_eq_synchronize_cmd_irq(dev);
spin_lock_irqsave(&dev->cmd.alloc_lock, flags);
vector = ~dev->cmd.bitmask & ((1ul << (1 << dev->cmd.log_sz)) - 1);
if (!vector)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
index 48ee37797b3f..706d58383dbd 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
@@ -8,11 +8,8 @@
#define MLX5_MAX_IRQ_NAME (32)
enum {
- MLX5_EQ_VEC_PAGES = 0,
- MLX5_EQ_VEC_CMD = 1,
- MLX5_EQ_VEC_ASYNC = 2,
- MLX5_EQ_VEC_PFAULT = 3,
- MLX5_EQ_VEC_COMP_BASE,
+ MLX5_EQ_MAX_ASYNC_EQS = 4, /* mlx5_core needs at least 3 */
+ MLX5_EQ_VEC_COMP_BASE = MLX5_EQ_MAX_ASYNC_EQS,
};
struct mlx5_eq_tasklet {
@@ -22,13 +19,6 @@ struct mlx5_eq_tasklet {
spinlock_t lock; /* lock completion tasklet list */
};
-struct mlx5_eq_pagefault {
- struct work_struct work;
- spinlock_t lock; /* Pagefaults spinlock */
- struct workqueue_struct *wq;
- mempool_t *pool;
-};
-
struct mlx5_cq_table {
spinlock_t lock; /* protect radix tree */
struct radix_tree_root tree;
@@ -44,29 +34,48 @@ struct mlx5_eq {
unsigned int irqn;
u8 eqn;
int nent;
- struct list_head list;
struct mlx5_rsc_debug *dbg;
- enum mlx5_eq_type type;
- union {
- struct mlx5_eq_tasklet tasklet_ctx;
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- struct mlx5_eq_pagefault pf_ctx;
-#endif
- };
+};
+
+struct mlx5_eq_comp {
+ struct mlx5_eq core; /* Must be first */
+ struct mlx5_eq_tasklet tasklet_ctx;
+ struct list_head list;
+};
+
+struct mlx5_eq_pagefault {
+ struct mlx5_eq core; /* Must be first */
+ struct work_struct work;
+ spinlock_t lock; /* Pagefaults spinlock */
+ struct workqueue_struct *wq;
+ mempool_t *pool;
};
int mlx5_eq_table_init(struct mlx5_core_dev *dev);
void mlx5_eq_table_cleanup(struct mlx5_core_dev *dev);
int mlx5_eq_table_create(struct mlx5_core_dev *dev);
void mlx5_eq_table_destroy(struct mlx5_core_dev *dev);
+int mlx5_create_async_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
+ int nent, u64 mask, const char *name,
+ irq_handler_t handler);
+int mlx5_destroy_async_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
+
int mlx5_eq_add_cq(struct mlx5_eq *eq, struct mlx5_core_cq *cq);
int mlx5_eq_del_cq(struct mlx5_eq *eq, struct mlx5_core_cq *cq);
-struct mlx5_eq *mlx5_eqn2eq(struct mlx5_core_dev *dev, int eqn);
+struct mlx5_eq_comp *mlx5_eqn2comp_eq(struct mlx5_core_dev *dev, int eqn);
struct mlx5_eq *mlx5_get_async_eq(struct mlx5_core_dev *dev);
-u32 mlx5_eq_poll_irq_disabled(struct mlx5_eq *eq);
void mlx5_cq_tasklet_cb(unsigned long data);
struct cpumask *mlx5_eq_comp_cpumask(struct mlx5_core_dev *dev, int ix);
+u32 mlx5_eq_poll_irq_disabled(struct mlx5_eq_comp *eq);
+void mlx5_eq_synchronize_async_irq(struct mlx5_core_dev *dev);
+void mlx5_eq_synchronize_cmd_irq(struct mlx5_core_dev *dev);
+
+int mlx5_debug_eq_add(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
+void mlx5_debug_eq_remove(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
+int mlx5_eq_debugfs_init(struct mlx5_core_dev *dev);
+void mlx5_eq_debugfs_cleanup(struct mlx5_core_dev *dev);
+
/* This function should only be called after mlx5_cmd_force_teardown_hca */
void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 5d11ef92c8b6..3de83fe65f2b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -53,8 +53,8 @@
#endif
#include <net/devlink.h>
#include "mlx5_core.h"
-#include "fs_core.h"
#include "lib/eq.h"
+#include "fs_core.h"
#include "lib/mpfs.h"
#include "eswitch.h"
#include "lib/mlx5.h"
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
index 4d39adcfb0eb..4728b027cb9e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
@@ -125,10 +125,6 @@ int mlx5_wait_for_vf_pages(struct mlx5_core_dev *dev);
u64 mlx5_read_internal_timer(struct mlx5_core_dev *dev);
void mlx5_cmd_comp_handler(struct mlx5_core_dev *dev, u64 vec, bool forced);
-int mlx5_debug_eq_add(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
-void mlx5_debug_eq_remove(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
-int mlx5_eq_debugfs_init(struct mlx5_core_dev *dev);
-void mlx5_eq_debugfs_cleanup(struct mlx5_core_dev *dev);
int mlx5_cq_debugfs_init(struct mlx5_core_dev *dev);
void mlx5_cq_debugfs_cleanup(struct mlx5_core_dev *dev);
diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
index 31a750570c38..28b757a64029 100644
--- a/include/linux/mlx5/cq.h
+++ b/include/linux/mlx5/cq.h
@@ -60,7 +60,7 @@ struct mlx5_core_cq {
} tasklet_ctx;
int reset_notify_added;
struct list_head reset_notify;
- struct mlx5_eq *eq;
+ struct mlx5_eq_comp *eq;
u16 uid;
};
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 4d6246cb6c19..fe9b552aa649 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -210,14 +210,6 @@ enum mlx5_port_status {
MLX5_PORT_DOWN = 2,
};
-enum mlx5_eq_type {
- MLX5_EQ_TYPE_COMP,
- MLX5_EQ_TYPE_ASYNC,
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- MLX5_EQ_TYPE_PF,
-#endif
-};
-
struct mlx5_bfreg_info {
u32 *sys_pages;
int num_low_latency_bfregs;
@@ -692,7 +684,7 @@ struct mlx5_pagefault {
} rdma;
};
- struct mlx5_eq *eq;
+ struct mlx5_eq_pagefault *eq;
struct work_struct work;
};
--
2.19.1
^ permalink raw reply related
* [PATCH V2 mlx5-next 08/12] net/mlx5: EQ, Privatize eq_table and friends
From: Saeed Mahameed @ 2018-11-19 18:52 UTC (permalink / raw)
To: Leon Romanovsky, saeedm; +Cc: netdev, linux-rdma, Jason Gunthorpe
In-Reply-To: <20181119185242.21961-1-saeedm@mellanox.com>
Move unnecessary EQ table structures and declaration from the
public include/linux/mlx5/driver.h into the private area of mlx5_core
and into eq.c/eq.h.
Introduce new mlx5 EQ APIs:
mlx5_comp_vectors_count(dev);
mlx5_comp_irq_get_affinity_mask(dev, vector);
And use them from mlx5_ib or mlx5e netdevice instead of direct access to
mlx5_core internal structures.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/infiniband/hw/mlx5/main.c | 5 +-
drivers/net/ethernet/mellanox/mlx5/core/cq.c | 5 +-
.../net/ethernet/mellanox/mlx5/core/debugfs.c | 1 +
drivers/net/ethernet/mellanox/mlx5/core/en.h | 3 +-
.../net/ethernet/mellanox/mlx5/core/en_main.c | 10 +-
drivers/net/ethernet/mellanox/mlx5/core/eq.c | 102 ++++++++++++++----
.../net/ethernet/mellanox/mlx5/core/eswitch.c | 1 +
.../net/ethernet/mellanox/mlx5/core/health.c | 1 +
.../net/ethernet/mellanox/mlx5/core/lib/eq.h | 77 +++++++++++++
.../net/ethernet/mellanox/mlx5/core/main.c | 7 +-
.../ethernet/mellanox/mlx5/core/mlx5_core.h | 15 ---
include/linux/mlx5/driver.h | 87 +--------------
12 files changed, 179 insertions(+), 135 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index e9c428071df3..6fbc0cba1bac 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -5337,7 +5337,7 @@ mlx5_ib_get_vector_affinity(struct ib_device *ibdev, int comp_vector)
{
struct mlx5_ib_dev *dev = to_mdev(ibdev);
- return mlx5_get_vector_affinity_hint(dev->mdev, comp_vector);
+ return mlx5_comp_irq_get_affinity_mask(dev->mdev, comp_vector);
}
/* The mlx5_ib_multiport_mutex should be held when calling this function */
@@ -5701,8 +5701,7 @@ int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
dev->ib_dev.node_type = RDMA_NODE_IB_CA;
dev->ib_dev.local_dma_lkey = 0 /* not supported for now */;
dev->ib_dev.phys_port_cnt = dev->num_ports;
- dev->ib_dev.num_comp_vectors =
- dev->mdev->priv.eq_table.num_comp_vectors;
+ dev->ib_dev.num_comp_vectors = mlx5_comp_vectors_count(mdev);
dev->ib_dev.dev.parent = &mdev->pdev->dev;
mutex_init(&dev->cap_mask_mutex);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cq.c b/drivers/net/ethernet/mellanox/mlx5/core/cq.c
index 4b85abb5c9f7..6e55d2f37c6d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/cq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/cq.c
@@ -38,6 +38,7 @@
#include <rdma/ib_verbs.h>
#include <linux/mlx5/cq.h>
#include "mlx5_core.h"
+#include "lib/eq.h"
#define TASKLET_MAX_TIME 2
#define TASKLET_MAX_TIME_JIFFIES msecs_to_jiffies(TASKLET_MAX_TIME)
@@ -124,7 +125,7 @@ int mlx5_core_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
goto err_cmd;
/* Add to async EQ CQ tree to recv async events */
- err = mlx5_eq_add_cq(&dev->priv.eq_table.async_eq, cq);
+ err = mlx5_eq_add_cq(mlx5_get_async_eq(dev), cq);
if (err)
goto err_cq_add;
@@ -157,7 +158,7 @@ int mlx5_core_destroy_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq)
u32 in[MLX5_ST_SZ_DW(destroy_cq_in)] = {0};
int err;
- err = mlx5_eq_del_cq(&dev->priv.eq_table.async_eq, cq);
+ err = mlx5_eq_del_cq(mlx5_get_async_eq(dev), cq);
if (err)
return err;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/debugfs.c b/drivers/net/ethernet/mellanox/mlx5/core/debugfs.c
index b76766fb6c67..a11e22d0b0cc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/debugfs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/debugfs.c
@@ -36,6 +36,7 @@
#include <linux/mlx5/cq.h>
#include <linux/mlx5/driver.h>
#include "mlx5_core.h"
+#include "lib/eq.h"
enum {
QP_PID,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index d7fbd5b6ac95..aea74856c702 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -178,8 +178,7 @@ static inline int mlx5e_get_max_num_channels(struct mlx5_core_dev *mdev)
{
return is_kdump_kernel() ?
MLX5E_MIN_NUM_CHANNELS :
- min_t(int, mdev->priv.eq_table.num_comp_vectors,
- MLX5E_MAX_NUM_CHANNELS);
+ min_t(int, mlx5_comp_vectors_count(mdev), MLX5E_MAX_NUM_CHANNELS);
}
/* Use this function to get max num channels after netdev was created */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 32ea47c28324..c23caade31bf 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -49,6 +49,7 @@
#include "lib/clock.h"
#include "en/port.h"
#include "en/xdp.h"
+#include "lib/eq.h"
struct mlx5e_rq_param {
u32 rqc[MLX5_ST_SZ_DW(rqc)];
@@ -1758,11 +1759,6 @@ static void mlx5e_close_cq(struct mlx5e_cq *cq)
mlx5e_free_cq(cq);
}
-static int mlx5e_get_cpu(struct mlx5e_priv *priv, int ix)
-{
- return cpumask_first(priv->mdev->priv.eq_table.irq_info[ix + MLX5_EQ_VEC_COMP_BASE].mask);
-}
-
static int mlx5e_open_tx_cqs(struct mlx5e_channel *c,
struct mlx5e_params *params,
struct mlx5e_channel_param *cparam)
@@ -1913,9 +1909,9 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
struct mlx5e_channel_param *cparam,
struct mlx5e_channel **cp)
{
+ int cpu = cpumask_first(mlx5_comp_irq_get_affinity_mask(priv->mdev, ix));
struct net_dim_cq_moder icocq_moder = {0, 0};
struct net_device *netdev = priv->netdev;
- int cpu = mlx5e_get_cpu(priv, ix);
struct mlx5e_channel *c;
unsigned int irq;
int err;
@@ -4960,7 +4956,7 @@ int mlx5e_netdev_init(struct net_device *netdev,
netif_carrier_off(netdev);
#ifdef CONFIG_MLX5_EN_ARFS
- netdev->rx_cpu_rmap = mdev->priv.eq_table.rmap;
+ netdev->rx_cpu_rmap = mlx5_eq_table_get_rmap(mdev);
#endif
return 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
index 70f62f10065e..32ce20221c44 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
@@ -38,6 +38,7 @@
#include <linux/cpu_rmap.h>
#endif
#include "mlx5_core.h"
+#include "lib/eq.h"
#include "fpga/core.h"
#include "eswitch.h"
#include "lib/clock.h"
@@ -65,6 +66,26 @@ enum {
MLX5_EQ_DOORBEL_OFFSET = 0x40,
};
+struct mlx5_irq_info {
+ cpumask_var_t mask;
+ char name[MLX5_MAX_IRQ_NAME];
+};
+
+struct mlx5_eq_table {
+ struct list_head comp_eqs_list;
+ struct mlx5_eq pages_eq;
+ struct mlx5_eq async_eq;
+ struct mlx5_eq cmd_eq;
+#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
+ struct mlx5_eq pfault_eq;
+#endif
+ int num_comp_vectors;
+ struct mlx5_irq_info *irq_info;
+#ifdef CONFIG_RFS_ACCEL
+ struct cpu_rmap *rmap;
+#endif
+};
+
#define MLX5_ASYNC_EVENT_MASK ((1ull << MLX5_EVENT_TYPE_PATH_MIG) | \
(1ull << MLX5_EVENT_TYPE_COMM_EST) | \
(1ull << MLX5_EVENT_TYPE_SQ_DRAINED) | \
@@ -633,10 +654,11 @@ static void init_eq_buf(struct mlx5_eq *eq)
}
}
-int mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
- int nent, u64 mask, const char *name,
- enum mlx5_eq_type type)
+static int
+mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
+ int nent, u64 mask, const char *name, enum mlx5_eq_type type)
{
+ struct mlx5_eq_table *eq_table = dev->priv.eq_table;
struct mlx5_cq_table *cq_table = &eq->cq_table;
u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {0};
struct mlx5_priv *priv = &dev->priv;
@@ -694,7 +716,7 @@ int mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
if (err)
goto err_in;
- snprintf(priv->eq_table.irq_info[vecidx].name, MLX5_MAX_IRQ_NAME, "%s@pci:%s",
+ snprintf(eq_table->irq_info[vecidx].name, MLX5_MAX_IRQ_NAME, "%s@pci:%s",
name, pci_name(dev->pdev));
eq->eqn = MLX5_GET(create_eq_out, out, eq_number);
@@ -702,7 +724,7 @@ int mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
eq->dev = dev;
eq->doorbell = priv->uar->map + MLX5_EQ_DOORBEL_OFFSET;
err = request_irq(eq->irqn, handler, 0,
- priv->eq_table.irq_info[vecidx].name, eq);
+ eq_table->irq_info[vecidx].name, eq);
if (err)
goto err_eq;
@@ -746,7 +768,7 @@ int mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
return err;
}
-int mlx5_destroy_unmap_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq)
+static int mlx5_destroy_unmap_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq)
{
int err;
@@ -806,25 +828,35 @@ int mlx5_eq_del_cq(struct mlx5_eq *eq, struct mlx5_core_cq *cq)
return 0;
}
-int mlx5_eq_init(struct mlx5_core_dev *dev)
+int mlx5_eq_table_init(struct mlx5_core_dev *dev)
{
+ struct mlx5_eq_table *eq_table;
int err;
+ eq_table = kvzalloc(sizeof(*eq_table), GFP_KERNEL);
+ if (!eq_table)
+ return -ENOMEM;
+
+ dev->priv.eq_table = eq_table;
+
err = mlx5_eq_debugfs_init(dev);
+ if (err)
+ kvfree(eq_table);
return err;
}
-void mlx5_eq_cleanup(struct mlx5_core_dev *dev)
+void mlx5_eq_table_cleanup(struct mlx5_core_dev *dev)
{
mlx5_eq_debugfs_cleanup(dev);
+ kvfree(dev->priv.eq_table);
}
/* Async EQs */
static int create_async_eqs(struct mlx5_core_dev *dev)
{
- struct mlx5_eq_table *table = &dev->priv.eq_table;
+ struct mlx5_eq_table *table = dev->priv.eq_table;
u64 async_event_mask = MLX5_ASYNC_EVENT_MASK;
int err;
@@ -916,7 +948,7 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
static void destroy_async_eqs(struct mlx5_core_dev *dev)
{
- struct mlx5_eq_table *table = &dev->priv.eq_table;
+ struct mlx5_eq_table *table = dev->priv.eq_table;
int err;
#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
@@ -945,6 +977,11 @@ static void destroy_async_eqs(struct mlx5_core_dev *dev)
err);
}
+struct mlx5_eq *mlx5_get_async_eq(struct mlx5_core_dev *dev)
+{
+ return &dev->priv.eq_table->async_eq;
+}
+
/* Completion EQs */
static int set_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
@@ -952,7 +989,7 @@ static int set_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
struct mlx5_priv *priv = &mdev->priv;
int vecidx = MLX5_EQ_VEC_COMP_BASE + i;
int irq = pci_irq_vector(mdev->pdev, vecidx);
- struct mlx5_irq_info *irq_info = &priv->eq_table.irq_info[vecidx];
+ struct mlx5_irq_info *irq_info = &priv->eq_table->irq_info[vecidx];
if (!zalloc_cpumask_var(&irq_info->mask, GFP_KERNEL)) {
mlx5_core_warn(mdev, "zalloc_cpumask_var failed");
@@ -974,7 +1011,7 @@ static void clear_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
int vecidx = MLX5_EQ_VEC_COMP_BASE + i;
struct mlx5_priv *priv = &mdev->priv;
int irq = pci_irq_vector(mdev->pdev, vecidx);
- struct mlx5_irq_info *irq_info = &priv->eq_table.irq_info[vecidx];
+ struct mlx5_irq_info *irq_info = &priv->eq_table->irq_info[vecidx];
irq_set_affinity_hint(irq, NULL);
free_cpumask_var(irq_info->mask);
@@ -985,7 +1022,7 @@ static int set_comp_irq_affinity_hints(struct mlx5_core_dev *mdev)
int err;
int i;
- for (i = 0; i < mdev->priv.eq_table.num_comp_vectors; i++) {
+ for (i = 0; i < mdev->priv.eq_table->num_comp_vectors; i++) {
err = set_comp_irq_affinity_hint(mdev, i);
if (err)
goto err_out;
@@ -1004,13 +1041,13 @@ static void clear_comp_irqs_affinity_hints(struct mlx5_core_dev *mdev)
{
int i;
- for (i = 0; i < mdev->priv.eq_table.num_comp_vectors; i++)
+ for (i = 0; i < mdev->priv.eq_table->num_comp_vectors; i++)
clear_comp_irq_affinity_hint(mdev, i);
}
static void destroy_comp_eqs(struct mlx5_core_dev *dev)
{
- struct mlx5_eq_table *table = &dev->priv.eq_table;
+ struct mlx5_eq_table *table = dev->priv.eq_table;
struct mlx5_eq *eq, *n;
clear_comp_irqs_affinity_hints(dev);
@@ -1032,7 +1069,7 @@ static void destroy_comp_eqs(struct mlx5_core_dev *dev)
static int create_comp_eqs(struct mlx5_core_dev *dev)
{
- struct mlx5_eq_table *table = &dev->priv.eq_table;
+ struct mlx5_eq_table *table = dev->priv.eq_table;
char name[MLX5_MAX_IRQ_NAME];
struct mlx5_eq *eq;
int ncomp_vec;
@@ -1088,7 +1125,7 @@ static int create_comp_eqs(struct mlx5_core_dev *dev)
int mlx5_vector2eqn(struct mlx5_core_dev *dev, int vector, int *eqn,
unsigned int *irqn)
{
- struct mlx5_eq_table *table = &dev->priv.eq_table;
+ struct mlx5_eq_table *table = dev->priv.eq_table;
struct mlx5_eq *eq, *n;
int err = -ENOENT;
int i = 0;
@@ -1106,9 +1143,32 @@ int mlx5_vector2eqn(struct mlx5_core_dev *dev, int vector, int *eqn,
}
EXPORT_SYMBOL(mlx5_vector2eqn);
+unsigned int mlx5_comp_vectors_count(struct mlx5_core_dev *dev)
+{
+ return dev->priv.eq_table->num_comp_vectors;
+}
+EXPORT_SYMBOL(mlx5_comp_vectors_count);
+
+struct cpumask *
+mlx5_comp_irq_get_affinity_mask(struct mlx5_core_dev *dev, int vector)
+{
+ /* TODO: consider irq_get_affinity_mask(irq) */
+ return dev->priv.eq_table->irq_info[vector + MLX5_EQ_VEC_COMP_BASE].mask;
+}
+EXPORT_SYMBOL(mlx5_comp_irq_get_affinity_mask);
+
+struct cpu_rmap *mlx5_eq_table_get_rmap(struct mlx5_core_dev *dev)
+{
+#ifdef CONFIG_RFS_ACCEL
+ return dev->priv.eq_table->rmap;
+#else
+ return NULL;
+#endif
+}
+
struct mlx5_eq *mlx5_eqn2eq(struct mlx5_core_dev *dev, int eqn)
{
- struct mlx5_eq_table *table = &dev->priv.eq_table;
+ struct mlx5_eq_table *table = dev->priv.eq_table;
struct mlx5_eq *eq;
list_for_each_entry(eq, &table->comp_eqs_list, list) {
@@ -1122,7 +1182,7 @@ struct mlx5_eq *mlx5_eqn2eq(struct mlx5_core_dev *dev, int eqn)
/* This function should only be called after mlx5_cmd_force_teardown_hca */
void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev)
{
- struct mlx5_eq_table *table = &dev->priv.eq_table;
+ struct mlx5_eq_table *table = dev->priv.eq_table;
struct mlx5_eq *eq;
clear_comp_irqs_affinity_hints(dev);
@@ -1149,7 +1209,7 @@ void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev)
static int alloc_irq_vectors(struct mlx5_core_dev *dev)
{
struct mlx5_priv *priv = &dev->priv;
- struct mlx5_eq_table *table = &priv->eq_table;
+ struct mlx5_eq_table *table = priv->eq_table;
int num_eqs = MLX5_CAP_GEN(dev, max_num_eqs) ?
MLX5_CAP_GEN(dev, max_num_eqs) :
1 << MLX5_CAP_GEN(dev, log_max_eq);
@@ -1187,7 +1247,7 @@ static void free_irq_vectors(struct mlx5_core_dev *dev)
struct mlx5_priv *priv = &dev->priv;
pci_free_irq_vectors(dev->pdev);
- kfree(priv->eq_table.irq_info);
+ kfree(priv->eq_table->irq_info);
}
int mlx5_eq_table_create(struct mlx5_core_dev *dev)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
index d004957328f9..324606227b1a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
@@ -38,6 +38,7 @@
#include "mlx5_core.h"
#include "eswitch.h"
#include "fs_core.h"
+#include "lib/eq.h"
#define UPLINK_VPORT 0xFFFF
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
index 43118de8ee99..b5be6f0b9ed5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
@@ -38,6 +38,7 @@
#include <linux/mlx5/driver.h>
#include <linux/mlx5/cmd.h>
#include "mlx5_core.h"
+#include "lib/eq.h"
enum {
MLX5_HEALTH_POLL_INTERVAL = 2 * HZ,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
new file mode 100644
index 000000000000..48ee37797b3f
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2018 Mellanox Technologies */
+
+#ifndef __LIB_MLX5_EQ_H__
+#define __LIB_MLX5_EQ_H__
+#include <linux/mlx5/driver.h>
+
+#define MLX5_MAX_IRQ_NAME (32)
+
+enum {
+ MLX5_EQ_VEC_PAGES = 0,
+ MLX5_EQ_VEC_CMD = 1,
+ MLX5_EQ_VEC_ASYNC = 2,
+ MLX5_EQ_VEC_PFAULT = 3,
+ MLX5_EQ_VEC_COMP_BASE,
+};
+
+struct mlx5_eq_tasklet {
+ struct list_head list;
+ struct list_head process_list;
+ struct tasklet_struct task;
+ spinlock_t lock; /* lock completion tasklet list */
+};
+
+struct mlx5_eq_pagefault {
+ struct work_struct work;
+ spinlock_t lock; /* Pagefaults spinlock */
+ struct workqueue_struct *wq;
+ mempool_t *pool;
+};
+
+struct mlx5_cq_table {
+ spinlock_t lock; /* protect radix tree */
+ struct radix_tree_root tree;
+};
+
+struct mlx5_eq {
+ struct mlx5_core_dev *dev;
+ struct mlx5_cq_table cq_table;
+ __be32 __iomem *doorbell;
+ u32 cons_index;
+ struct mlx5_frag_buf buf;
+ int size;
+ unsigned int irqn;
+ u8 eqn;
+ int nent;
+ struct list_head list;
+ struct mlx5_rsc_debug *dbg;
+ enum mlx5_eq_type type;
+ union {
+ struct mlx5_eq_tasklet tasklet_ctx;
+#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
+ struct mlx5_eq_pagefault pf_ctx;
+#endif
+ };
+};
+
+int mlx5_eq_table_init(struct mlx5_core_dev *dev);
+void mlx5_eq_table_cleanup(struct mlx5_core_dev *dev);
+int mlx5_eq_table_create(struct mlx5_core_dev *dev);
+void mlx5_eq_table_destroy(struct mlx5_core_dev *dev);
+int mlx5_eq_add_cq(struct mlx5_eq *eq, struct mlx5_core_cq *cq);
+int mlx5_eq_del_cq(struct mlx5_eq *eq, struct mlx5_core_cq *cq);
+struct mlx5_eq *mlx5_eqn2eq(struct mlx5_core_dev *dev, int eqn);
+struct mlx5_eq *mlx5_get_async_eq(struct mlx5_core_dev *dev);
+u32 mlx5_eq_poll_irq_disabled(struct mlx5_eq *eq);
+void mlx5_cq_tasklet_cb(unsigned long data);
+struct cpumask *mlx5_eq_comp_cpumask(struct mlx5_core_dev *dev, int ix);
+
+/* This function should only be called after mlx5_cmd_force_teardown_hca */
+void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev);
+
+#ifdef CONFIG_RFS_ACCEL
+struct cpu_rmap *mlx5_eq_table_get_rmap(struct mlx5_core_dev *dev);
+#endif
+
+#endif
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 21cc9bbc2563..5d11ef92c8b6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -54,6 +54,7 @@
#include <net/devlink.h>
#include "mlx5_core.h"
#include "fs_core.h"
+#include "lib/eq.h"
#include "lib/mpfs.h"
#include "eswitch.h"
#include "lib/mlx5.h"
@@ -728,7 +729,7 @@ static int mlx5_init_once(struct mlx5_core_dev *dev, struct mlx5_priv *priv)
goto out;
}
- err = mlx5_eq_init(dev);
+ err = mlx5_eq_table_init(dev);
if (err) {
dev_err(&pdev->dev, "failed to initialize eq\n");
goto out;
@@ -802,7 +803,7 @@ static int mlx5_init_once(struct mlx5_core_dev *dev, struct mlx5_priv *priv)
mlx5_cq_debugfs_cleanup(dev);
err_eq_cleanup:
- mlx5_eq_cleanup(dev);
+ mlx5_eq_table_cleanup(dev);
out:
return err;
@@ -823,7 +824,7 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev)
mlx5_cleanup_srq_table(dev);
mlx5_cleanup_qp_table(dev);
mlx5_cq_debugfs_cleanup(dev);
- mlx5_eq_cleanup(dev);
+ mlx5_eq_table_cleanup(dev);
}
static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
index 3fa6d26875fe..4d39adcfb0eb 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
@@ -124,21 +124,6 @@ int mlx5_destroy_scheduling_element_cmd(struct mlx5_core_dev *dev, u8 hierarchy,
int mlx5_wait_for_vf_pages(struct mlx5_core_dev *dev);
u64 mlx5_read_internal_timer(struct mlx5_core_dev *dev);
-int mlx5_eq_init(struct mlx5_core_dev *dev);
-void mlx5_eq_cleanup(struct mlx5_core_dev *dev);
-int mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
- int nent, u64 mask, const char *name,
- enum mlx5_eq_type type);
-int mlx5_destroy_unmap_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
-int mlx5_eq_add_cq(struct mlx5_eq *eq, struct mlx5_core_cq *cq);
-int mlx5_eq_del_cq(struct mlx5_eq *eq, struct mlx5_core_cq *cq);
-int mlx5_eq_table_create(struct mlx5_core_dev *dev);
-void mlx5_eq_table_destroy(struct mlx5_core_dev *dev);
-/* This function should only be called after mlx5_cmd_force_teardown_hca */
-void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev);
-struct mlx5_eq *mlx5_eqn2eq(struct mlx5_core_dev *dev, int eqn);
-u32 mlx5_eq_poll_irq_disabled(struct mlx5_eq *eq);
-void mlx5_cq_tasklet_cb(unsigned long data);
void mlx5_cmd_comp_handler(struct mlx5_core_dev *dev, u64 vec, bool forced);
int mlx5_debug_eq_add(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
void mlx5_debug_eq_remove(struct mlx5_core_dev *dev, struct mlx5_eq *eq);
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index dcc3f7aa8572..4d6246cb6c19 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -84,18 +84,6 @@ enum {
MLX5_MAX_PORTS = 2,
};
-enum {
- MLX5_EQ_VEC_PAGES = 0,
- MLX5_EQ_VEC_CMD = 1,
- MLX5_EQ_VEC_ASYNC = 2,
- MLX5_EQ_VEC_PFAULT = 3,
- MLX5_EQ_VEC_COMP_BASE,
-};
-
-enum {
- MLX5_MAX_IRQ_NAME = 32
-};
-
enum {
MLX5_ATOMIC_MODE_OFFSET = 16,
MLX5_ATOMIC_MODE_IB_COMP = 1,
@@ -366,49 +354,6 @@ struct mlx5_frag_buf_ctrl {
u8 log_frag_strides;
};
-struct mlx5_eq_tasklet {
- struct list_head list;
- struct list_head process_list;
- struct tasklet_struct task;
- /* lock on completion tasklet list */
- spinlock_t lock;
-};
-
-struct mlx5_eq_pagefault {
- struct work_struct work;
- /* Pagefaults lock */
- spinlock_t lock;
- struct workqueue_struct *wq;
- mempool_t *pool;
-};
-
-struct mlx5_cq_table {
- /* protect radix tree */
- spinlock_t lock;
- struct radix_tree_root tree;
-};
-
-struct mlx5_eq {
- struct mlx5_core_dev *dev;
- struct mlx5_cq_table cq_table;
- __be32 __iomem *doorbell;
- u32 cons_index;
- struct mlx5_frag_buf buf;
- int size;
- unsigned int irqn;
- u8 eqn;
- int nent;
- struct list_head list;
- struct mlx5_rsc_debug *dbg;
- enum mlx5_eq_type type;
- union {
- struct mlx5_eq_tasklet tasklet_ctx;
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- struct mlx5_eq_pagefault pf_ctx;
-#endif
- };
-};
-
struct mlx5_core_psv {
u32 psv_idx;
struct psv_layout {
@@ -475,21 +420,6 @@ struct mlx5_core_srq {
u16 uid;
};
-struct mlx5_eq_table {
- struct list_head comp_eqs_list;
- struct mlx5_eq pages_eq;
- struct mlx5_eq async_eq;
- struct mlx5_eq cmd_eq;
-#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING
- struct mlx5_eq pfault_eq;
-#endif
- int num_comp_vectors;
- struct mlx5_irq_info *irq_info;
-#ifdef CONFIG_RFS_ACCEL
- struct cpu_rmap *rmap;
-#endif
-};
-
struct mlx5_uars_page {
void __iomem *map;
bool wc;
@@ -572,11 +502,6 @@ struct mlx5_core_sriov {
int enabled_vfs;
};
-struct mlx5_irq_info {
- cpumask_var_t mask;
- char name[MLX5_MAX_IRQ_NAME];
-};
-
struct mlx5_fc_stats {
spinlock_t counters_idr_lock; /* protects counters_idr */
struct idr counters_idr;
@@ -594,6 +519,7 @@ struct mlx5_mpfs;
struct mlx5_eswitch;
struct mlx5_lag;
struct mlx5_pagefault;
+struct mlx5_eq_table;
struct mlx5_rate_limit {
u32 rate;
@@ -643,7 +569,7 @@ struct mlx5_port_module_event_stats {
struct mlx5_priv {
char name[MLX5_MAX_NAME_LEN];
- struct mlx5_eq_table eq_table;
+ struct mlx5_eq_table *eq_table;
/* pages stuff */
struct workqueue_struct *pg_wq;
@@ -1148,6 +1074,9 @@ int mlx5_alloc_bfreg(struct mlx5_core_dev *mdev, struct mlx5_sq_bfreg *bfreg,
bool map_wc, bool fast_path);
void mlx5_free_bfreg(struct mlx5_core_dev *mdev, struct mlx5_sq_bfreg *bfreg);
+unsigned int mlx5_comp_vectors_count(struct mlx5_core_dev *dev);
+struct cpumask *
+mlx5_comp_irq_get_affinity_mask(struct mlx5_core_dev *dev, int vector);
unsigned int mlx5_core_reserved_gids_count(struct mlx5_core_dev *dev);
int mlx5_core_roce_gid_set(struct mlx5_core_dev *dev, unsigned int index,
u8 roce_version, u8 roce_l3_type, const u8 *gid,
@@ -1299,10 +1228,4 @@ enum {
MLX5_TRIGGERED_CMD_COMP = (u64)1 << 32,
};
-static inline const struct cpumask *
-mlx5_get_vector_affinity_hint(struct mlx5_core_dev *dev, int vector)
-{
- return dev->priv.eq_table.irq_info[vector + MLX5_EQ_VEC_COMP_BASE].mask;
-}
-
#endif /* MLX5_DRIVER_H */
--
2.19.1
^ permalink raw reply related
* [PATCH V2 mlx5-next 07/12] net/mlx5: EQ, irq_info and rmap belong to eq_table
From: Saeed Mahameed @ 2018-11-19 18:52 UTC (permalink / raw)
To: Leon Romanovsky, saeedm; +Cc: netdev, linux-rdma, Jason Gunthorpe
In-Reply-To: <20181119185242.21961-1-saeedm@mellanox.com>
irq_info and rmap are EQ properties of the driver, and only needed for
EQ objects, move them to the eq_table EQs database structure.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
.../net/ethernet/mellanox/mlx5/core/en_main.c | 4 +-
drivers/net/ethernet/mellanox/mlx5/core/eq.c | 40 ++++++++++---------
include/linux/mlx5/driver.h | 10 ++---
3 files changed, 28 insertions(+), 26 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 2839c30dd3a0..32ea47c28324 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1760,7 +1760,7 @@ static void mlx5e_close_cq(struct mlx5e_cq *cq)
static int mlx5e_get_cpu(struct mlx5e_priv *priv, int ix)
{
- return cpumask_first(priv->mdev->priv.irq_info[ix + MLX5_EQ_VEC_COMP_BASE].mask);
+ return cpumask_first(priv->mdev->priv.eq_table.irq_info[ix + MLX5_EQ_VEC_COMP_BASE].mask);
}
static int mlx5e_open_tx_cqs(struct mlx5e_channel *c,
@@ -4960,7 +4960,7 @@ int mlx5e_netdev_init(struct net_device *netdev,
netif_carrier_off(netdev);
#ifdef CONFIG_MLX5_EN_ARFS
- netdev->rx_cpu_rmap = mdev->rmap;
+ netdev->rx_cpu_rmap = mdev->priv.eq_table.rmap;
#endif
return 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
index 44ccd4206104..70f62f10065e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
@@ -694,7 +694,7 @@ int mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
if (err)
goto err_in;
- snprintf(priv->irq_info[vecidx].name, MLX5_MAX_IRQ_NAME, "%s@pci:%s",
+ snprintf(priv->eq_table.irq_info[vecidx].name, MLX5_MAX_IRQ_NAME, "%s@pci:%s",
name, pci_name(dev->pdev));
eq->eqn = MLX5_GET(create_eq_out, out, eq_number);
@@ -702,7 +702,7 @@ int mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
eq->dev = dev;
eq->doorbell = priv->uar->map + MLX5_EQ_DOORBEL_OFFSET;
err = request_irq(eq->irqn, handler, 0,
- priv->irq_info[vecidx].name, eq);
+ priv->eq_table.irq_info[vecidx].name, eq);
if (err)
goto err_eq;
@@ -952,17 +952,18 @@ static int set_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
struct mlx5_priv *priv = &mdev->priv;
int vecidx = MLX5_EQ_VEC_COMP_BASE + i;
int irq = pci_irq_vector(mdev->pdev, vecidx);
+ struct mlx5_irq_info *irq_info = &priv->eq_table.irq_info[vecidx];
- if (!zalloc_cpumask_var(&priv->irq_info[vecidx].mask, GFP_KERNEL)) {
+ if (!zalloc_cpumask_var(&irq_info->mask, GFP_KERNEL)) {
mlx5_core_warn(mdev, "zalloc_cpumask_var failed");
return -ENOMEM;
}
cpumask_set_cpu(cpumask_local_spread(i, priv->numa_node),
- priv->irq_info[vecidx].mask);
+ irq_info->mask);
if (IS_ENABLED(CONFIG_SMP) &&
- irq_set_affinity_hint(irq, priv->irq_info[vecidx].mask))
+ irq_set_affinity_hint(irq, irq_info->mask))
mlx5_core_warn(mdev, "irq_set_affinity_hint failed, irq 0x%.4x", irq);
return 0;
@@ -973,9 +974,10 @@ static void clear_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
int vecidx = MLX5_EQ_VEC_COMP_BASE + i;
struct mlx5_priv *priv = &mdev->priv;
int irq = pci_irq_vector(mdev->pdev, vecidx);
+ struct mlx5_irq_info *irq_info = &priv->eq_table.irq_info[vecidx];
irq_set_affinity_hint(irq, NULL);
- free_cpumask_var(priv->irq_info[vecidx].mask);
+ free_cpumask_var(irq_info->mask);
}
static int set_comp_irq_affinity_hints(struct mlx5_core_dev *mdev)
@@ -1014,9 +1016,9 @@ static void destroy_comp_eqs(struct mlx5_core_dev *dev)
clear_comp_irqs_affinity_hints(dev);
#ifdef CONFIG_RFS_ACCEL
- if (dev->rmap) {
- free_irq_cpu_rmap(dev->rmap);
- dev->rmap = NULL;
+ if (table->rmap) {
+ free_irq_cpu_rmap(table->rmap);
+ table->rmap = NULL;
}
#endif
list_for_each_entry_safe(eq, n, &table->comp_eqs_list, list) {
@@ -1042,8 +1044,8 @@ static int create_comp_eqs(struct mlx5_core_dev *dev)
ncomp_vec = table->num_comp_vectors;
nent = MLX5_COMP_EQ_SIZE;
#ifdef CONFIG_RFS_ACCEL
- dev->rmap = alloc_irq_cpu_rmap(ncomp_vec);
- if (!dev->rmap)
+ table->rmap = alloc_irq_cpu_rmap(ncomp_vec);
+ if (!table->rmap)
return -ENOMEM;
#endif
for (i = 0; i < ncomp_vec; i++) {
@@ -1056,7 +1058,7 @@ static int create_comp_eqs(struct mlx5_core_dev *dev)
}
#ifdef CONFIG_RFS_ACCEL
- irq_cpu_rmap_add(dev->rmap, pci_irq_vector(dev->pdev, vecidx));
+ irq_cpu_rmap_add(table->rmap, pci_irq_vector(dev->pdev, vecidx));
#endif
snprintf(name, MLX5_MAX_IRQ_NAME, "mlx5_comp%d", i);
err = mlx5_create_map_eq(dev, eq, vecidx, nent, 0,
@@ -1126,9 +1128,9 @@ void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev)
clear_comp_irqs_affinity_hints(dev);
#ifdef CONFIG_RFS_ACCEL
- if (dev->rmap) {
- free_irq_cpu_rmap(dev->rmap);
- dev->rmap = NULL;
+ if (table->rmap) {
+ free_irq_cpu_rmap(table->rmap);
+ table->rmap = NULL;
}
#endif
list_for_each_entry(eq, &table->comp_eqs_list, list)
@@ -1160,8 +1162,8 @@ static int alloc_irq_vectors(struct mlx5_core_dev *dev)
if (nvec <= MLX5_EQ_VEC_COMP_BASE)
return -ENOMEM;
- priv->irq_info = kcalloc(nvec, sizeof(*priv->irq_info), GFP_KERNEL);
- if (!priv->irq_info)
+ table->irq_info = kcalloc(nvec, sizeof(*table->irq_info), GFP_KERNEL);
+ if (!table->irq_info)
return -ENOMEM;
nvec = pci_alloc_irq_vectors(dev->pdev, MLX5_EQ_VEC_COMP_BASE + 1,
@@ -1176,7 +1178,7 @@ static int alloc_irq_vectors(struct mlx5_core_dev *dev)
return 0;
err_free_irq_info:
- kfree(priv->irq_info);
+ kfree(table->irq_info);
return err;
}
@@ -1185,7 +1187,7 @@ static void free_irq_vectors(struct mlx5_core_dev *dev)
struct mlx5_priv *priv = &dev->priv;
pci_free_irq_vectors(dev->pdev);
- kfree(priv->irq_info);
+ kfree(priv->eq_table.irq_info);
}
int mlx5_eq_table_create(struct mlx5_core_dev *dev)
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 852e397c7624..dcc3f7aa8572 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -484,6 +484,10 @@ struct mlx5_eq_table {
struct mlx5_eq pfault_eq;
#endif
int num_comp_vectors;
+ struct mlx5_irq_info *irq_info;
+#ifdef CONFIG_RFS_ACCEL
+ struct cpu_rmap *rmap;
+#endif
};
struct mlx5_uars_page {
@@ -640,7 +644,6 @@ struct mlx5_port_module_event_stats {
struct mlx5_priv {
char name[MLX5_MAX_NAME_LEN];
struct mlx5_eq_table eq_table;
- struct mlx5_irq_info *irq_info;
/* pages stuff */
struct workqueue_struct *pg_wq;
@@ -851,9 +854,6 @@ struct mlx5_core_dev {
} roce;
#ifdef CONFIG_MLX5_FPGA
struct mlx5_fpga_device *fpga;
-#endif
-#ifdef CONFIG_RFS_ACCEL
- struct cpu_rmap *rmap;
#endif
struct mlx5_clock clock;
struct mlx5_ib_clock_info *clock_info;
@@ -1302,7 +1302,7 @@ enum {
static inline const struct cpumask *
mlx5_get_vector_affinity_hint(struct mlx5_core_dev *dev, int vector)
{
- return dev->priv.irq_info[vector + MLX5_EQ_VEC_COMP_BASE].mask;
+ return dev->priv.eq_table.irq_info[vector + MLX5_EQ_VEC_COMP_BASE].mask;
}
#endif /* MLX5_DRIVER_H */
--
2.19.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox