* [PATCH net-next 1/8] mlxsw: reg: Add ptps field in QoS ETS Element Configuration Register
From: Ido Schimmel @ 2019-07-04 7:07 UTC (permalink / raw)
To: netdev; +Cc: davem, richardcochran, jiri, petrm, shalomt, mlxsw, Ido Schimmel
In-Reply-To: <20190704070740.302-1-idosch@idosch.org>
From: Shalom Toledo <shalomt@mellanox.com>
The PTP Shaper field is used for enabling and disabling of port-rate based
shaper which is slightly lower than port rate.
Signed-off-by: Shalom Toledo <shalomt@mellanox.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Reviewed-by: Petr Machata <petrm@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
drivers/net/ethernet/mellanox/mlxsw/reg.h | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/reg.h b/drivers/net/ethernet/mellanox/mlxsw/reg.h
index 76ff5b217c04..d2e2a75f7983 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/reg.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/reg.h
@@ -3515,6 +3515,18 @@ MLXSW_ITEM32(reg, qeec, next_element_index, 0x08, 0, 8);
*/
MLXSW_ITEM32(reg, qeec, mise, 0x0C, 31, 1);
+/* reg_qeec_ptps
+ * PTP shaper
+ * 0: regular shaper mode
+ * 1: PTP oriented shaper
+ * Allowed only for hierarchy 0
+ * Not supported for CPU port
+ * Note that ptps mode may affect the shaper rates of all hierarchies
+ * Supported only on Spectrum-1
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, qeec, ptps, 0x0C, 29, 1);
+
enum {
MLXSW_REG_QEEC_BYTES_MODE,
MLXSW_REG_QEEC_PACKETS_MODE,
@@ -3601,6 +3613,16 @@ static inline void mlxsw_reg_qeec_pack(char *payload, u8 local_port,
mlxsw_reg_qeec_next_element_index_set(payload, next_index);
}
+static inline void mlxsw_reg_qeec_ptps_pack(char *payload, u8 local_port,
+ bool ptps)
+{
+ MLXSW_REG_ZERO(qeec, payload);
+ mlxsw_reg_qeec_local_port_set(payload, local_port);
+ mlxsw_reg_qeec_element_hierarchy_set(payload,
+ MLXSW_REG_QEEC_HIERARCY_PORT);
+ mlxsw_reg_qeec_ptps_set(payload, ptps);
+}
+
/* QRWE - QoS ReWrite Enable
* -------------------------
* This register configures the rewrite enable per receive port.
--
2.20.1
^ permalink raw reply related
* [PATCH net-next 2/8] mlxsw: spectrum: Add note about the PTP shaper
From: Ido Schimmel @ 2019-07-04 7:07 UTC (permalink / raw)
To: netdev; +Cc: davem, richardcochran, jiri, petrm, shalomt, mlxsw, Ido Schimmel
In-Reply-To: <20190704070740.302-1-idosch@idosch.org>
From: Shalom Toledo <shalomt@mellanox.com>
Add note about disabling the PTP shaper when calling to
mlxsw_sp_port_ets_maxrate_set().
Signed-off-by: Shalom Toledo <shalomt@mellanox.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Reviewed-by: Petr Machata <petrm@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index 755b14b82c8f..67133ba53015 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -3457,8 +3457,9 @@ static int mlxsw_sp_port_ets_init(struct mlxsw_sp_port *mlxsw_sp_port)
return err;
}
- /* Make sure the max shaper is disabled in all hierarchies that
- * support it.
+ /* Make sure the max shaper is disabled in all hierarchies that support
+ * it. Note that this disables ptps (PTP shaper), but that is intended
+ * for the initial configuration.
*/
err = mlxsw_sp_port_ets_maxrate_set(mlxsw_sp_port,
MLXSW_REG_QEEC_HIERARCY_PORT, 0, 0,
--
2.20.1
^ permalink raw reply related
* [PATCH net-next 0/8] mlxsw: Enable/disable PTP shapers
From: Ido Schimmel @ 2019-07-04 7:07 UTC (permalink / raw)
To: netdev; +Cc: davem, richardcochran, jiri, petrm, shalomt, mlxsw, Ido Schimmel
From: Ido Schimmel <idosch@mellanox.com>
Shalom says:
In order to get more accurate hardware time stamping in Spectrum-1, the
driver needs to apply a shaper on the port for speeds lower than 40Gbps.
This shaper is called a PTP shaper and it is applied on hierarchy 0,
which is the port hierarchy. This shaper may affect the shaper rates of
all hierarchies.
This patchset adds the ability to enable or disable the PTP shaper on
the port in two scenarios:
1. When the user wants to enable/disable the hardware time stamping
2. When the port is brought up or down (including port speed change)
Patch #1 adds the QEEC.ptps field that is used for enabling or disabling
the PTP shaper on a port.
Patch #2 adds a note about disabling the PTP shaper when calling to
mlxsw_sp_port_ets_maxrate_set().
Patch #3 adds the QPSC register that is responsible for configuring the
PTP shaper parameters per speed.
Patch #4 sets the PTP shaper parameters during the ptp_init().
Patch #5 adds new operation for getting the port's speed.
Patch #6 enables/disables the PTP shaper when turning on or off the
hardware time stamping.
Patch #7 enables/disables the PTP shaper when the port's status has
changed (including port speed change).
Patch #8 applies the PTP shaper enable/disable logic by filling the PTP
shaper parameters array.
Shalom Toledo (8):
mlxsw: reg: Add ptps field in QoS ETS Element Configuration Register
mlxsw: spectrum: Add note about the PTP shaper
mlxsw: reg: Add QoS PTP Shaper Configuration Register
mlxsw: spectrum_ptp: Set the PTP shaper parameters
mlxsw: spectrum: Add new operation for getting the port's speed
mlxsw: spectrum_ptp: Enable/disable PTP shaper on a port when getting
HWTSTAMP on/off
mlxsw: spectrum: Set up PTP shaper when port status has changed
mlxsw: spectrum_ptp: Apply the PTP shaper enable/disable logic
drivers/net/ethernet/mellanox/mlxsw/reg.h | 129 ++++++++++++++
.../net/ethernet/mellanox/mlxsw/spectrum.c | 81 +++++----
.../net/ethernet/mellanox/mlxsw/spectrum.h | 2 +
.../ethernet/mellanox/mlxsw/spectrum_ptp.c | 158 ++++++++++++++++++
.../ethernet/mellanox/mlxsw/spectrum_ptp.h | 10 ++
5 files changed, 350 insertions(+), 30 deletions(-)
--
2.20.1
^ permalink raw reply
* Re: [PATCH v2 2/4] net: dsa: vsc73xx: Split vsc73xx driver
From: Linus Walleij @ 2019-07-04 7:08 UTC (permalink / raw)
To: Pawel Dembicki
Cc: Andrew Lunn, Vivien Didelot, Florian Fainelli, David S. Miller,
Rob Herring, Mark Rutland, netdev,
open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
linux-kernel@vger.kernel.org
In-Reply-To: <20190703171924.31801-3-paweldembicki@gmail.com>
On Wed, Jul 3, 2019 at 7:21 PM Pawel Dembicki <paweldembicki@gmail.com> wrote:
> This driver (currently) only takes control of the switch chip over
> SPI and configures it to route packages around when connected to a
> CPU port. But Vitesse chip support also parallel interface.
>
> This patch split driver into two parts: core and spi. It is required
> for add support to another managing interface.
>
> Tested-by: Linus Walleij <linus.walleij@linaro.org>
> Signed-off-by: Pawel Dembicki <paweldembicki@gmail.com>
That's a nice and clean split.
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Yours,
Linus Walleij
^ permalink raw reply
* Re: [PATCH v2 1/4] net: dsa: Change DT bindings for Vitesse VSC73xx switches
From: Linus Walleij @ 2019-07-04 7:05 UTC (permalink / raw)
To: Pawel Dembicki
Cc: Andrew Lunn, Vivien Didelot, Florian Fainelli, David S. Miller,
Rob Herring, Mark Rutland, netdev,
open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
linux-kernel@vger.kernel.org
In-Reply-To: <20190703171924.31801-2-paweldembicki@gmail.com>
On Wed, Jul 3, 2019 at 7:21 PM Pawel Dembicki <paweldembicki@gmail.com> wrote:
> This commit introduce how to use vsc73xx platform driver.
>
> Signed-off-by: Pawel Dembicki <paweldembicki@gmail.com>
Nice!
> +If Platform driver is used, the device tree node is an platform device so it
> +must reside inside a platform bus device tree node.
I would write something like "when connected to a memory bus, and
used in memory-mapped I/O mode, a platform device is used to represent
the vsc73xx" so it is clear what is going on.
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Yours,
Linus Walleij
^ permalink raw reply
* Re: [RFC v2] vhost: introduce mdev based hardware vhost backend
From: Tiwei Bie @ 2019-07-04 7:02 UTC (permalink / raw)
To: Jason Wang
Cc: mst, alex.williamson, maxime.coquelin, linux-kernel, kvm,
virtualization, netdev, dan.daly, cunming.liang, zhihong.wang
In-Reply-To: <c67f628f-e0c1-9a41-6c5d-b6bbda31467d@redhat.com>
On Thu, Jul 04, 2019 at 02:35:20PM +0800, Jason Wang wrote:
> On 2019/7/4 下午2:21, Tiwei Bie wrote:
> > On Thu, Jul 04, 2019 at 12:31:48PM +0800, Jason Wang wrote:
> > > On 2019/7/3 下午9:08, Tiwei Bie wrote:
> > > > On Wed, Jul 03, 2019 at 08:16:23PM +0800, Jason Wang wrote:
> > > > > On 2019/7/3 下午7:52, Tiwei Bie wrote:
> > > > > > On Wed, Jul 03, 2019 at 06:09:51PM +0800, Jason Wang wrote:
> > > > > > > On 2019/7/3 下午5:13, Tiwei Bie wrote:
> > > > > > > > Details about this can be found here:
> > > > > > > >
> > > > > > > > https://lwn.net/Articles/750770/
> > > > > > > >
> > > > > > > > What's new in this version
> > > > > > > > ==========================
> > > > > > > >
> > > > > > > > A new VFIO device type is introduced - vfio-vhost. This addressed
> > > > > > > > some comments from here:https://patchwork.ozlabs.org/cover/984763/
> > > > > > > >
> > > > > > > > Below is the updated device interface:
> > > > > > > >
> > > > > > > > Currently, there are two regions of this device: 1) CONFIG_REGION
> > > > > > > > (VFIO_VHOST_CONFIG_REGION_INDEX), which can be used to setup the
> > > > > > > > device; 2) NOTIFY_REGION (VFIO_VHOST_NOTIFY_REGION_INDEX), which
> > > > > > > > can be used to notify the device.
> > > > > > > >
> > > > > > > > 1. CONFIG_REGION
> > > > > > > >
> > > > > > > > The region described by CONFIG_REGION is the main control interface.
> > > > > > > > Messages will be written to or read from this region.
> > > > > > > >
> > > > > > > > The message type is determined by the `request` field in message
> > > > > > > > header. The message size is encoded in the message header too.
> > > > > > > > The message format looks like this:
> > > > > > > >
> > > > > > > > struct vhost_vfio_op {
> > > > > > > > __u64 request;
> > > > > > > > __u32 flags;
> > > > > > > > /* Flag values: */
> > > > > > > > #define VHOST_VFIO_NEED_REPLY 0x1 /* Whether need reply */
> > > > > > > > __u32 size;
> > > > > > > > union {
> > > > > > > > __u64 u64;
> > > > > > > > struct vhost_vring_state state;
> > > > > > > > struct vhost_vring_addr addr;
> > > > > > > > } payload;
> > > > > > > > };
> > > > > > > >
> > > > > > > > The existing vhost-kernel ioctl cmds are reused as the message
> > > > > > > > requests in above structure.
> > > > > > > Still a comments like V1. What's the advantage of inventing a new protocol?
> > > > > > I'm trying to make it work in VFIO's way..
> > > > > >
> > > > > > > I believe either of the following should be better:
> > > > > > >
> > > > > > > - using vhost ioctl, we can start from SET_VRING_KICK/SET_VRING_CALL and
> > > > > > > extend it with e.g notify region. The advantages is that all exist userspace
> > > > > > > program could be reused without modification (or minimal modification). And
> > > > > > > vhost API hides lots of details that is not necessary to be understood by
> > > > > > > application (e.g in the case of container).
> > > > > > Do you mean reusing vhost's ioctl on VFIO device fd directly,
> > > > > > or introducing another mdev driver (i.e. vhost_mdev instead of
> > > > > > using the existing vfio_mdev) for mdev device?
> > > > > Can we simply add them into ioctl of mdev_parent_ops?
> > > > Right, either way, these ioctls have to be and just need to be
> > > > added in the ioctl of the mdev_parent_ops. But another thing we
> > > > also need to consider is that which file descriptor the userspace
> > > > will do the ioctl() on. So I'm wondering do you mean let the
> > > > userspace do the ioctl() on the VFIO device fd of the mdev
> > > > device?
> > > >
> > > Yes.
> > Got it! I'm not sure what's Alex opinion on this. If we all
> > agree with this, I can do it in this way.
> >
> > > Is there any other way btw?
> > Just a quick thought.. Maybe totally a bad idea.
>
>
> It's not for sure :)
Thanks!
>
>
> > I was thinking
> > whether it would be odd to do non-VFIO's ioctls on VFIO's device
> > fd. So I was wondering whether it's possible to allow binding
> > another mdev driver (e.g. vhost_mdev) to the supported mdev
> > devices. The new mdev driver, vhost_mdev, can provide similar
> > ways to let userspace open the mdev device and do the vhost ioctls
> > on it. To distinguish with the vfio_mdev compatible mdev devices,
> > the device API of the new vhost_mdev compatible mdev devices
> > might be e.g. "vhost-net" for net?
> >
> > So in VFIO case, the device will be for passthru directly. And
> > in VHOST case, the device can be used to accelerate the existing
> > virtualized devices.
> >
> > How do you think?
>
>
> If my understanding is correct, there will be no VFIO ioctl if we go for
> vhost_mdev?
Yeah, exactly. If we go for vhost_mdev, we may have some vhost nodes
in /dev similar to what /dev/vfio/* does to handle the $UUID and open
the device (e.g. similar to VFIO_GROUP_GET_DEVICE_FD in VFIO). And
to setup the device, we can try to reuse the ioctls of the existing
kernel vhost as much as possible.
Thanks,
Tiwei
>
> Thanks
>
>
> >
> > Thanks,
> > Tiwei
> > > Thanks
> > >
^ permalink raw reply
* Re: [RFC v2] vhost: introduce mdev based hardware vhost backend
From: Jason Wang @ 2019-07-04 6:35 UTC (permalink / raw)
To: Tiwei Bie
Cc: mst, alex.williamson, maxime.coquelin, linux-kernel, kvm,
virtualization, netdev, dan.daly, cunming.liang, zhihong.wang
In-Reply-To: <20190704062134.GA21116@___>
On 2019/7/4 下午2:21, Tiwei Bie wrote:
> On Thu, Jul 04, 2019 at 12:31:48PM +0800, Jason Wang wrote:
>> On 2019/7/3 下午9:08, Tiwei Bie wrote:
>>> On Wed, Jul 03, 2019 at 08:16:23PM +0800, Jason Wang wrote:
>>>> On 2019/7/3 下午7:52, Tiwei Bie wrote:
>>>>> On Wed, Jul 03, 2019 at 06:09:51PM +0800, Jason Wang wrote:
>>>>>> On 2019/7/3 下午5:13, Tiwei Bie wrote:
>>>>>>> Details about this can be found here:
>>>>>>>
>>>>>>> https://lwn.net/Articles/750770/
>>>>>>>
>>>>>>> What's new in this version
>>>>>>> ==========================
>>>>>>>
>>>>>>> A new VFIO device type is introduced - vfio-vhost. This addressed
>>>>>>> some comments from here:https://patchwork.ozlabs.org/cover/984763/
>>>>>>>
>>>>>>> Below is the updated device interface:
>>>>>>>
>>>>>>> Currently, there are two regions of this device: 1) CONFIG_REGION
>>>>>>> (VFIO_VHOST_CONFIG_REGION_INDEX), which can be used to setup the
>>>>>>> device; 2) NOTIFY_REGION (VFIO_VHOST_NOTIFY_REGION_INDEX), which
>>>>>>> can be used to notify the device.
>>>>>>>
>>>>>>> 1. CONFIG_REGION
>>>>>>>
>>>>>>> The region described by CONFIG_REGION is the main control interface.
>>>>>>> Messages will be written to or read from this region.
>>>>>>>
>>>>>>> The message type is determined by the `request` field in message
>>>>>>> header. The message size is encoded in the message header too.
>>>>>>> The message format looks like this:
>>>>>>>
>>>>>>> struct vhost_vfio_op {
>>>>>>> __u64 request;
>>>>>>> __u32 flags;
>>>>>>> /* Flag values: */
>>>>>>> #define VHOST_VFIO_NEED_REPLY 0x1 /* Whether need reply */
>>>>>>> __u32 size;
>>>>>>> union {
>>>>>>> __u64 u64;
>>>>>>> struct vhost_vring_state state;
>>>>>>> struct vhost_vring_addr addr;
>>>>>>> } payload;
>>>>>>> };
>>>>>>>
>>>>>>> The existing vhost-kernel ioctl cmds are reused as the message
>>>>>>> requests in above structure.
>>>>>> Still a comments like V1. What's the advantage of inventing a new protocol?
>>>>> I'm trying to make it work in VFIO's way..
>>>>>
>>>>>> I believe either of the following should be better:
>>>>>>
>>>>>> - using vhost ioctl, we can start from SET_VRING_KICK/SET_VRING_CALL and
>>>>>> extend it with e.g notify region. The advantages is that all exist userspace
>>>>>> program could be reused without modification (or minimal modification). And
>>>>>> vhost API hides lots of details that is not necessary to be understood by
>>>>>> application (e.g in the case of container).
>>>>> Do you mean reusing vhost's ioctl on VFIO device fd directly,
>>>>> or introducing another mdev driver (i.e. vhost_mdev instead of
>>>>> using the existing vfio_mdev) for mdev device?
>>>> Can we simply add them into ioctl of mdev_parent_ops?
>>> Right, either way, these ioctls have to be and just need to be
>>> added in the ioctl of the mdev_parent_ops. But another thing we
>>> also need to consider is that which file descriptor the userspace
>>> will do the ioctl() on. So I'm wondering do you mean let the
>>> userspace do the ioctl() on the VFIO device fd of the mdev
>>> device?
>>>
>> Yes.
> Got it! I'm not sure what's Alex opinion on this. If we all
> agree with this, I can do it in this way.
>
>> Is there any other way btw?
> Just a quick thought.. Maybe totally a bad idea.
It's not for sure :)
> I was thinking
> whether it would be odd to do non-VFIO's ioctls on VFIO's device
> fd. So I was wondering whether it's possible to allow binding
> another mdev driver (e.g. vhost_mdev) to the supported mdev
> devices. The new mdev driver, vhost_mdev, can provide similar
> ways to let userspace open the mdev device and do the vhost ioctls
> on it. To distinguish with the vfio_mdev compatible mdev devices,
> the device API of the new vhost_mdev compatible mdev devices
> might be e.g. "vhost-net" for net?
>
> So in VFIO case, the device will be for passthru directly. And
> in VHOST case, the device can be used to accelerate the existing
> virtualized devices.
>
> How do you think?
If my understanding is correct, there will be no VFIO ioctl if we go for
vhost_mdev?
Thanks
>
> Thanks,
> Tiwei
>> Thanks
>>
^ permalink raw reply
* Re: [RFC v2] vhost: introduce mdev based hardware vhost backend
From: Tiwei Bie @ 2019-07-04 6:21 UTC (permalink / raw)
To: Jason Wang
Cc: mst, alex.williamson, maxime.coquelin, linux-kernel, kvm,
virtualization, netdev, dan.daly, cunming.liang, zhihong.wang
In-Reply-To: <b01b8e28-8d96-31dd-56f4-ca7793498c55@redhat.com>
On Thu, Jul 04, 2019 at 12:31:48PM +0800, Jason Wang wrote:
> On 2019/7/3 下午9:08, Tiwei Bie wrote:
> > On Wed, Jul 03, 2019 at 08:16:23PM +0800, Jason Wang wrote:
> > > On 2019/7/3 下午7:52, Tiwei Bie wrote:
> > > > On Wed, Jul 03, 2019 at 06:09:51PM +0800, Jason Wang wrote:
> > > > > On 2019/7/3 下午5:13, Tiwei Bie wrote:
> > > > > > Details about this can be found here:
> > > > > >
> > > > > > https://lwn.net/Articles/750770/
> > > > > >
> > > > > > What's new in this version
> > > > > > ==========================
> > > > > >
> > > > > > A new VFIO device type is introduced - vfio-vhost. This addressed
> > > > > > some comments from here:https://patchwork.ozlabs.org/cover/984763/
> > > > > >
> > > > > > Below is the updated device interface:
> > > > > >
> > > > > > Currently, there are two regions of this device: 1) CONFIG_REGION
> > > > > > (VFIO_VHOST_CONFIG_REGION_INDEX), which can be used to setup the
> > > > > > device; 2) NOTIFY_REGION (VFIO_VHOST_NOTIFY_REGION_INDEX), which
> > > > > > can be used to notify the device.
> > > > > >
> > > > > > 1. CONFIG_REGION
> > > > > >
> > > > > > The region described by CONFIG_REGION is the main control interface.
> > > > > > Messages will be written to or read from this region.
> > > > > >
> > > > > > The message type is determined by the `request` field in message
> > > > > > header. The message size is encoded in the message header too.
> > > > > > The message format looks like this:
> > > > > >
> > > > > > struct vhost_vfio_op {
> > > > > > __u64 request;
> > > > > > __u32 flags;
> > > > > > /* Flag values: */
> > > > > > #define VHOST_VFIO_NEED_REPLY 0x1 /* Whether need reply */
> > > > > > __u32 size;
> > > > > > union {
> > > > > > __u64 u64;
> > > > > > struct vhost_vring_state state;
> > > > > > struct vhost_vring_addr addr;
> > > > > > } payload;
> > > > > > };
> > > > > >
> > > > > > The existing vhost-kernel ioctl cmds are reused as the message
> > > > > > requests in above structure.
> > > > > Still a comments like V1. What's the advantage of inventing a new protocol?
> > > > I'm trying to make it work in VFIO's way..
> > > >
> > > > > I believe either of the following should be better:
> > > > >
> > > > > - using vhost ioctl, we can start from SET_VRING_KICK/SET_VRING_CALL and
> > > > > extend it with e.g notify region. The advantages is that all exist userspace
> > > > > program could be reused without modification (or minimal modification). And
> > > > > vhost API hides lots of details that is not necessary to be understood by
> > > > > application (e.g in the case of container).
> > > > Do you mean reusing vhost's ioctl on VFIO device fd directly,
> > > > or introducing another mdev driver (i.e. vhost_mdev instead of
> > > > using the existing vfio_mdev) for mdev device?
> > > Can we simply add them into ioctl of mdev_parent_ops?
> > Right, either way, these ioctls have to be and just need to be
> > added in the ioctl of the mdev_parent_ops. But another thing we
> > also need to consider is that which file descriptor the userspace
> > will do the ioctl() on. So I'm wondering do you mean let the
> > userspace do the ioctl() on the VFIO device fd of the mdev
> > device?
> >
>
> Yes.
Got it! I'm not sure what's Alex opinion on this. If we all
agree with this, I can do it in this way.
> Is there any other way btw?
Just a quick thought.. Maybe totally a bad idea. I was thinking
whether it would be odd to do non-VFIO's ioctls on VFIO's device
fd. So I was wondering whether it's possible to allow binding
another mdev driver (e.g. vhost_mdev) to the supported mdev
devices. The new mdev driver, vhost_mdev, can provide similar
ways to let userspace open the mdev device and do the vhost ioctls
on it. To distinguish with the vfio_mdev compatible mdev devices,
the device API of the new vhost_mdev compatible mdev devices
might be e.g. "vhost-net" for net?
So in VFIO case, the device will be for passthru directly. And
in VHOST case, the device can be used to accelerate the existing
virtualized devices.
How do you think?
Thanks,
Tiwei
>
> Thanks
>
^ permalink raw reply
* RE: [PATCH v1 net-next] net: stmmac: enable clause 45 mdio support
From: Voon, Weifeng @ 2019-07-04 6:05 UTC (permalink / raw)
To: Andrew Lunn
Cc: David S. Miller, Maxime Coquelin, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org, Jose Abreu, Giuseppe Cavallaro,
Florian Fainelli, Alexandre Torgue, biao huang, Ong, Boon Leong,
Kweh, Hock Leong
In-Reply-To: <20190704033038.GA6276@lunn.ch>
> > > > @@ -155,22 +171,26 @@ static int stmmac_mdio_read(struct mii_bus
> > > > *bus,
> > > int phyaddr, int phyreg)
> > > > struct stmmac_priv *priv = netdev_priv(ndev);
> > > > unsigned int mii_address = priv->hw->mii.addr;
> > > > unsigned int mii_data = priv->hw->mii.data;
> > > > - u32 v;
> > > > - int data;
> > > > u32 value = MII_BUSY;
> > > > + int data = 0;
> > > > + u32 v;
> > > >
> > > > value |= (phyaddr << priv->hw->mii.addr_shift)
> > > > & priv->hw->mii.addr_mask;
> > > > value |= (phyreg << priv->hw->mii.reg_shift) & priv->hw-
> > > >mii.reg_mask;
> > > > value |= (priv->clk_csr << priv->hw->mii.clk_csr_shift)
> > > > & priv->hw->mii.clk_csr_mask;
> > > > - if (priv->plat->has_gmac4)
> > > > + if (priv->plat->has_gmac4) {
> > > > value |= MII_GMAC4_READ;
> > > > + if (phyreg & MII_ADDR_C45)
> > > > + stmmac_mdio_c45_setup(priv, phyreg, &value, &data);
> > > > + }
> > > >
> > > > if (readl_poll_timeout(priv->ioaddr + mii_address, v, !(v &
> > > MII_BUSY),
> > > > 100, 10000))
> > > > return -EBUSY;
> > > >
> > > > + writel(data, priv->ioaddr + mii_data);
> > >
> > > That looks odd. Could you explain why it is needed.
> > >
> > > Thanks
> > > Andrew
> >
> > Hi Andrew,
> > This mdio c45 support needed to access DWC xPCS which is a Clause-45
>
> I mean it looks odd doing a write to the data register in the middle of
> stmmac_mdio_read().
MAC is using an indirect access to access mdio devices. In order to read,
the driver needs to write into both mii_data and mii_address to select
c45, read/write command, phy address, address to read, and etc.
Weifeng
^ permalink raw reply
* RE: [EXT] Re: [PATCH net-next 4/4] qed: Add devlink support for configuration attributes.
From: Sudarsana Reddy Kalluru @ 2019-07-04 5:49 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Jiri Pirko, davem@davemloft.net, netdev@vger.kernel.org,
Michal Kalderon, Ariel Elior
In-Reply-To: <20190703104244.7d26e734@cakuba.netronome.com>
> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Wednesday, July 3, 2019 11:13 PM
> To: Sudarsana Reddy Kalluru <skalluru@marvell.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> netdev@vger.kernel.org; Michal Kalderon <mkalderon@marvell.com>; Ariel
> Elior <aelior@marvell.com>
> Subject: Re: [EXT] Re: [PATCH net-next 4/4] qed: Add devlink support for
> configuration attributes.
>
> On Wed, 3 Jul 2019 12:56:39 +0000, Sudarsana Reddy Kalluru wrote:
> > Apologies for bringing this topic again. From the driver(s) code
> > paths/'devlink man pages', I understood that devlink-port object is an
> > entity on top of the PCI bus device. Some drivers say NFP represents
> > vnics (on pci-dev) as a devlink-ports and, some represents
> > (virtual?) ports on the PF/device as devlink-ports. In the case of
> > Marvell NIC driver, we don't have [port] partitioning of the PCI
> > device. And the config attributes are specific to PCI-device (not the
> > vports/vnics of PF). Hence I didn't see a need for creating
> > devlink-port objects in the system for Marvell NICs. And planning to
> > add the config attributes to 'devlink-dev' object. Please let me know
> > if my understanding and the proposal is ok?
>
> I understand where you're coming from.
>
> We want to make that judgement call on attribute-by-attribute basis.
> We want consistency across vendors (and, frankly, multiple drivers of the
> same vendor). If attribute looks like it belongs to the port, rather than the
> entire device/ASIC operation, we should make it a port attribute.
Thanks for your mail. I'll go with creating PCI-dev/0 (i.e., port-id 0) for port based attributes as there's no port partitioning for Marvell NICs.
^ permalink raw reply
* Re: i.mx6ul with DSA in multi chip addressing mode - no MDIO access
From: Benjamin Beckmeyer @ 2019-07-04 4:33 UTC (permalink / raw)
To: Andrew Lunn; +Cc: netdev
In-Reply-To: <20190703155518.GE18473@lunn.ch>
On 03.07.19 17:55, Andrew Lunn wrote:
> On Wed, Jul 03, 2019 at 03:10:34PM +0200, Benjamin Beckmeyer wrote:
>> Hey folks,
>>
>> I'm having a problem with a custom i.mx6ul board. When DSA is loaded I can't
>> get access to the switch via MDIO, but the DSA is working properly. I set up
>> a bridge for testing and the switch is in forwarding mode and i can ping the
>> board. But the MDIO access isn't working at address 2 for the switch. When I
>> delete the DSA from the devicetree and start the board up, I can access the
>> switch via MDIO.
>>
>> With DSA up and running:
>>
>> mii -i 2 0 0x9800
>> mii -i 2 1
>> phyid:2, reg:0x01 -> 0x4000
>> mii -i 2 0 0x9803
>> mii -i 2 1
>> phyid:2, reg:0x01 -> 0x4000
>> mii -i 2 1 0x1883
>> mii -i 2 1
>> phyid:2, reg:0x01 -> 0x4000
> Hi Benjamin
>
> I'm guessing that the driver is also using register 0 and 1 at the
> same time you are, e.g. to poll the PHYs for link status etc.
>
> There are trace points for MDIO, so you can get the kernel to log all
> registers access. That should confirm if i'm right.
>
> Andrew
Hi Andrew,
I will do so. Haven't done it before, so maybe it takes some time.
I'm just wondering why I haven't these problems with another switching
chip. Here we use the 88E6390 in multi chip addressing mode, in the
other custom board we have a 88E6321 in single chip addressing mode,
but there the registers would be polled too. Maybe you know a answer?
Thanks again for your fast reply.
^ permalink raw reply
* Re: [RFC v2] vhost: introduce mdev based hardware vhost backend
From: Jason Wang @ 2019-07-04 4:31 UTC (permalink / raw)
To: Tiwei Bie
Cc: mst, alex.williamson, maxime.coquelin, linux-kernel, kvm,
virtualization, netdev, dan.daly, cunming.liang, zhihong.wang
In-Reply-To: <20190703130817.GA1978@___>
On 2019/7/3 下午9:08, Tiwei Bie wrote:
> On Wed, Jul 03, 2019 at 08:16:23PM +0800, Jason Wang wrote:
>> On 2019/7/3 下午7:52, Tiwei Bie wrote:
>>> On Wed, Jul 03, 2019 at 06:09:51PM +0800, Jason Wang wrote:
>>>> On 2019/7/3 下午5:13, Tiwei Bie wrote:
>>>>> Details about this can be found here:
>>>>>
>>>>> https://lwn.net/Articles/750770/
>>>>>
>>>>> What's new in this version
>>>>> ==========================
>>>>>
>>>>> A new VFIO device type is introduced - vfio-vhost. This addressed
>>>>> some comments from here:https://patchwork.ozlabs.org/cover/984763/
>>>>>
>>>>> Below is the updated device interface:
>>>>>
>>>>> Currently, there are two regions of this device: 1) CONFIG_REGION
>>>>> (VFIO_VHOST_CONFIG_REGION_INDEX), which can be used to setup the
>>>>> device; 2) NOTIFY_REGION (VFIO_VHOST_NOTIFY_REGION_INDEX), which
>>>>> can be used to notify the device.
>>>>>
>>>>> 1. CONFIG_REGION
>>>>>
>>>>> The region described by CONFIG_REGION is the main control interface.
>>>>> Messages will be written to or read from this region.
>>>>>
>>>>> The message type is determined by the `request` field in message
>>>>> header. The message size is encoded in the message header too.
>>>>> The message format looks like this:
>>>>>
>>>>> struct vhost_vfio_op {
>>>>> __u64 request;
>>>>> __u32 flags;
>>>>> /* Flag values: */
>>>>> #define VHOST_VFIO_NEED_REPLY 0x1 /* Whether need reply */
>>>>> __u32 size;
>>>>> union {
>>>>> __u64 u64;
>>>>> struct vhost_vring_state state;
>>>>> struct vhost_vring_addr addr;
>>>>> } payload;
>>>>> };
>>>>>
>>>>> The existing vhost-kernel ioctl cmds are reused as the message
>>>>> requests in above structure.
>>>> Still a comments like V1. What's the advantage of inventing a new protocol?
>>> I'm trying to make it work in VFIO's way..
>>>
>>>> I believe either of the following should be better:
>>>>
>>>> - using vhost ioctl, we can start from SET_VRING_KICK/SET_VRING_CALL and
>>>> extend it with e.g notify region. The advantages is that all exist userspace
>>>> program could be reused without modification (or minimal modification). And
>>>> vhost API hides lots of details that is not necessary to be understood by
>>>> application (e.g in the case of container).
>>> Do you mean reusing vhost's ioctl on VFIO device fd directly,
>>> or introducing another mdev driver (i.e. vhost_mdev instead of
>>> using the existing vfio_mdev) for mdev device?
>> Can we simply add them into ioctl of mdev_parent_ops?
> Right, either way, these ioctls have to be and just need to be
> added in the ioctl of the mdev_parent_ops. But another thing we
> also need to consider is that which file descriptor the userspace
> will do the ioctl() on. So I'm wondering do you mean let the
> userspace do the ioctl() on the VFIO device fd of the mdev
> device?
>
Yes. Is there any other way btw?
Thanks
^ permalink raw reply
* Re: [PATCH v2 2/3] vsock/virtio: stop workers during the .remove()
From: Jason Wang @ 2019-07-04 4:00 UTC (permalink / raw)
To: Stefano Garzarella, netdev
Cc: kvm, virtualization, Stefan Hajnoczi, Michael S. Tsirkin,
David S. Miller, linux-kernel
In-Reply-To: <20190628123659.139576-3-sgarzare@redhat.com>
On 2019/6/28 下午8:36, Stefano Garzarella wrote:
> Before to call vdev->config->reset(vdev) we need to be sure that
> no one is accessing the device, for this reason, we add new variables
> in the struct virtio_vsock to stop the workers during the .remove().
>
> This patch also add few comments before vdev->config->reset(vdev)
> and vdev->config->del_vqs(vdev).
>
> Suggested-by: Stefan Hajnoczi <stefanha@redhat.com>
> Suggested-by: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
> ---
> net/vmw_vsock/virtio_transport.c | 51 +++++++++++++++++++++++++++++++-
> 1 file changed, 50 insertions(+), 1 deletion(-)
This should work. But we may consider to convert the_virtio_vosck to
socket object and use socket refcnt and destructor in the future instead
of inventing something new by ourselves.
Thanks
>
> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
> index 7ad510ec12e0..1b44ec6f3f6c 100644
> --- a/net/vmw_vsock/virtio_transport.c
> +++ b/net/vmw_vsock/virtio_transport.c
> @@ -38,6 +38,7 @@ struct virtio_vsock {
> * must be accessed with tx_lock held.
> */
> struct mutex tx_lock;
> + bool tx_run;
>
> struct work_struct send_pkt_work;
> spinlock_t send_pkt_list_lock;
> @@ -53,6 +54,7 @@ struct virtio_vsock {
> * must be accessed with rx_lock held.
> */
> struct mutex rx_lock;
> + bool rx_run;
> int rx_buf_nr;
> int rx_buf_max_nr;
>
> @@ -60,6 +62,7 @@ struct virtio_vsock {
> * vqs[VSOCK_VQ_EVENT] must be accessed with event_lock held.
> */
> struct mutex event_lock;
> + bool event_run;
> struct virtio_vsock_event event_list[8];
>
> u32 guest_cid;
> @@ -94,6 +97,10 @@ static void virtio_transport_loopback_work(struct work_struct *work)
> spin_unlock_bh(&vsock->loopback_list_lock);
>
> mutex_lock(&vsock->rx_lock);
> +
> + if (!vsock->rx_run)
> + goto out;
> +
> while (!list_empty(&pkts)) {
> struct virtio_vsock_pkt *pkt;
>
> @@ -102,6 +109,7 @@ static void virtio_transport_loopback_work(struct work_struct *work)
>
> virtio_transport_recv_pkt(pkt);
> }
> +out:
> mutex_unlock(&vsock->rx_lock);
> }
>
> @@ -130,6 +138,9 @@ virtio_transport_send_pkt_work(struct work_struct *work)
>
> mutex_lock(&vsock->tx_lock);
>
> + if (!vsock->tx_run)
> + goto out;
> +
> vq = vsock->vqs[VSOCK_VQ_TX];
>
> for (;;) {
> @@ -188,6 +199,7 @@ virtio_transport_send_pkt_work(struct work_struct *work)
> if (added)
> virtqueue_kick(vq);
>
> +out:
> mutex_unlock(&vsock->tx_lock);
>
> if (restart_rx)
> @@ -323,6 +335,10 @@ static void virtio_transport_tx_work(struct work_struct *work)
>
> vq = vsock->vqs[VSOCK_VQ_TX];
> mutex_lock(&vsock->tx_lock);
> +
> + if (!vsock->tx_run)
> + goto out;
> +
> do {
> struct virtio_vsock_pkt *pkt;
> unsigned int len;
> @@ -333,6 +349,8 @@ static void virtio_transport_tx_work(struct work_struct *work)
> added = true;
> }
> } while (!virtqueue_enable_cb(vq));
> +
> +out:
> mutex_unlock(&vsock->tx_lock);
>
> if (added)
> @@ -361,6 +379,9 @@ static void virtio_transport_rx_work(struct work_struct *work)
>
> mutex_lock(&vsock->rx_lock);
>
> + if (!vsock->rx_run)
> + goto out;
> +
> do {
> virtqueue_disable_cb(vq);
> for (;;) {
> @@ -470,6 +491,9 @@ static void virtio_transport_event_work(struct work_struct *work)
>
> mutex_lock(&vsock->event_lock);
>
> + if (!vsock->event_run)
> + goto out;
> +
> do {
> struct virtio_vsock_event *event;
> unsigned int len;
> @@ -484,7 +508,7 @@ static void virtio_transport_event_work(struct work_struct *work)
> } while (!virtqueue_enable_cb(vq));
>
> virtqueue_kick(vsock->vqs[VSOCK_VQ_EVENT]);
> -
> +out:
> mutex_unlock(&vsock->event_lock);
> }
>
> @@ -619,12 +643,18 @@ static int virtio_vsock_probe(struct virtio_device *vdev)
> INIT_WORK(&vsock->send_pkt_work, virtio_transport_send_pkt_work);
> INIT_WORK(&vsock->loopback_work, virtio_transport_loopback_work);
>
> + mutex_lock(&vsock->tx_lock);
> + vsock->tx_run = true;
> + mutex_unlock(&vsock->tx_lock);
> +
> mutex_lock(&vsock->rx_lock);
> virtio_vsock_rx_fill(vsock);
> + vsock->rx_run = true;
> mutex_unlock(&vsock->rx_lock);
>
> mutex_lock(&vsock->event_lock);
> virtio_vsock_event_fill(vsock);
> + vsock->event_run = true;
> mutex_unlock(&vsock->event_lock);
>
> vdev->priv = vsock;
> @@ -659,6 +689,24 @@ static void virtio_vsock_remove(struct virtio_device *vdev)
> /* Reset all connected sockets when the device disappear */
> vsock_for_each_connected_socket(virtio_vsock_reset_sock);
>
> + /* Stop all work handlers to make sure no one is accessing the device,
> + * so we can safely call vdev->config->reset().
> + */
> + mutex_lock(&vsock->rx_lock);
> + vsock->rx_run = false;
> + mutex_unlock(&vsock->rx_lock);
> +
> + mutex_lock(&vsock->tx_lock);
> + vsock->tx_run = false;
> + mutex_unlock(&vsock->tx_lock);
> +
> + mutex_lock(&vsock->event_lock);
> + vsock->event_run = false;
> + mutex_unlock(&vsock->event_lock);
> +
> + /* Flush all device writes and interrupts, device will not use any
> + * more buffers.
> + */
> vdev->config->reset(vdev);
>
> mutex_lock(&vsock->rx_lock);
> @@ -689,6 +737,7 @@ static void virtio_vsock_remove(struct virtio_device *vdev)
> }
> spin_unlock_bh(&vsock->loopback_list_lock);
>
> + /* Delete virtqueues and flush outstanding callbacks if any */
> vdev->config->del_vqs(vdev);
>
> mutex_unlock(&the_virtio_vsock_mutex);
^ permalink raw reply
* Re: [PATCH v2 1/3] vsock/virtio: use RCU to avoid use-after-free on the_virtio_vsock
From: Jason Wang @ 2019-07-04 3:58 UTC (permalink / raw)
To: Stefano Garzarella, Stefan Hajnoczi
Cc: netdev, kvm, virtualization, Michael S. Tsirkin, David S. Miller,
linux-kernel
In-Reply-To: <20190703104135.wg34dobv64k7u4jo@steredhat>
On 2019/7/3 下午6:41, Stefano Garzarella wrote:
> On Wed, Jul 03, 2019 at 05:53:58PM +0800, Jason Wang wrote:
>> On 2019/6/28 下午8:36, Stefano Garzarella wrote:
>>> Some callbacks used by the upper layers can run while we are in the
>>> .remove(). A potential use-after-free can happen, because we free
>>> the_virtio_vsock without knowing if the callbacks are over or not.
>>>
>>> To solve this issue we move the assignment of the_virtio_vsock at the
>>> end of .probe(), when we finished all the initialization, and at the
>>> beginning of .remove(), before to release resources.
>>> For the same reason, we do the same also for the vdev->priv.
>>>
>>> We use RCU to be sure that all callbacks that use the_virtio_vsock
>>> ended before freeing it. This is not required for callbacks that
>>> use vdev->priv, because after the vdev->config->del_vqs() we are sure
>>> that they are ended and will no longer be invoked.
>>>
>>> We also take the mutex during the .remove() to avoid that .probe() can
>>> run while we are resetting the device.
>>>
>>> Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
>>> ---
>>> net/vmw_vsock/virtio_transport.c | 67 +++++++++++++++++++++-----------
>>> 1 file changed, 44 insertions(+), 23 deletions(-)
>>>
>>> diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
>>> index 9c287e3e393c..7ad510ec12e0 100644
>>> --- a/net/vmw_vsock/virtio_transport.c
>>> +++ b/net/vmw_vsock/virtio_transport.c
>>> @@ -65,19 +65,22 @@ struct virtio_vsock {
>>> u32 guest_cid;
>>> };
>>> -static struct virtio_vsock *virtio_vsock_get(void)
>>> -{
>>> - return the_virtio_vsock;
>>> -}
>>> -
>>> static u32 virtio_transport_get_local_cid(void)
>>> {
>>> - struct virtio_vsock *vsock = virtio_vsock_get();
>>> + struct virtio_vsock *vsock;
>>> + u32 ret;
>>> - if (!vsock)
>>> - return VMADDR_CID_ANY;
>>> + rcu_read_lock();
>>> + vsock = rcu_dereference(the_virtio_vsock);
>>> + if (!vsock) {
>>> + ret = VMADDR_CID_ANY;
>>> + goto out_rcu;
>>> + }
>>> - return vsock->guest_cid;
>>> + ret = vsock->guest_cid;
>>> +out_rcu:
>>> + rcu_read_unlock();
>>> + return ret;
>>> }
>>> static void virtio_transport_loopback_work(struct work_struct *work)
>>> @@ -197,14 +200,18 @@ virtio_transport_send_pkt(struct virtio_vsock_pkt *pkt)
>>> struct virtio_vsock *vsock;
>>> int len = pkt->len;
>>> - vsock = virtio_vsock_get();
>>> + rcu_read_lock();
>>> + vsock = rcu_dereference(the_virtio_vsock);
>>> if (!vsock) {
>>> virtio_transport_free_pkt(pkt);
>>> - return -ENODEV;
>>> + len = -ENODEV;
>>> + goto out_rcu;
>>> }
>>> - if (le64_to_cpu(pkt->hdr.dst_cid) == vsock->guest_cid)
>>> - return virtio_transport_send_pkt_loopback(vsock, pkt);
>>> + if (le64_to_cpu(pkt->hdr.dst_cid) == vsock->guest_cid) {
>>> + len = virtio_transport_send_pkt_loopback(vsock, pkt);
>>> + goto out_rcu;
>>> + }
>>> if (pkt->reply)
>>> atomic_inc(&vsock->queued_replies);
>>> @@ -214,6 +221,9 @@ virtio_transport_send_pkt(struct virtio_vsock_pkt *pkt)
>>> spin_unlock_bh(&vsock->send_pkt_list_lock);
>>> queue_work(virtio_vsock_workqueue, &vsock->send_pkt_work);
>>> +
>>> +out_rcu:
>>> + rcu_read_unlock();
>>> return len;
>>> }
>>> @@ -222,12 +232,14 @@ virtio_transport_cancel_pkt(struct vsock_sock *vsk)
>>> {
>>> struct virtio_vsock *vsock;
>>> struct virtio_vsock_pkt *pkt, *n;
>>> - int cnt = 0;
>>> + int cnt = 0, ret;
>>> LIST_HEAD(freeme);
>>> - vsock = virtio_vsock_get();
>>> + rcu_read_lock();
>>> + vsock = rcu_dereference(the_virtio_vsock);
>>> if (!vsock) {
>>> - return -ENODEV;
>>> + ret = -ENODEV;
>>> + goto out_rcu;
>>> }
>>> spin_lock_bh(&vsock->send_pkt_list_lock);
>>> @@ -255,7 +267,11 @@ virtio_transport_cancel_pkt(struct vsock_sock *vsk)
>>> queue_work(virtio_vsock_workqueue, &vsock->rx_work);
>>> }
>>> - return 0;
>>> + ret = 0;
>>> +
>>> +out_rcu:
>>> + rcu_read_unlock();
>>> + return ret;
>>> }
>>> static void virtio_vsock_rx_fill(struct virtio_vsock *vsock)
>>> @@ -590,8 +606,6 @@ static int virtio_vsock_probe(struct virtio_device *vdev)
>>> vsock->rx_buf_max_nr = 0;
>>> atomic_set(&vsock->queued_replies, 0);
>>> - vdev->priv = vsock;
>>> - the_virtio_vsock = vsock;
>>> mutex_init(&vsock->tx_lock);
>>> mutex_init(&vsock->rx_lock);
>>> mutex_init(&vsock->event_lock);
>>> @@ -613,6 +627,9 @@ static int virtio_vsock_probe(struct virtio_device *vdev)
>>> virtio_vsock_event_fill(vsock);
>>> mutex_unlock(&vsock->event_lock);
>>> + vdev->priv = vsock;
>>> + rcu_assign_pointer(the_virtio_vsock, vsock);
>>
>> You probably need to use rcu_dereference_protected() to access
>> the_virtio_vsock in the function in order to survive from sparse.
>>
> Ooo, thanks!
>
> Do you mean when we check if the_virtio_vsock is not null at the beginning of
> virtio_vsock_probe()?
I mean instead of:
/* Only one virtio-vsock device per guest is supported */
if (the_virtio_vsock) {
ret = -EBUSY;
goto out;
}
you should use:
if (rcu_dereference_protected(the_virtio_vosck,
lock_dep_is_held(&the_virtio_vsock_mutex))
...
>
>>> +
>>> mutex_unlock(&the_virtio_vsock_mutex);
>>> return 0;
>>> @@ -627,6 +644,12 @@ static void virtio_vsock_remove(struct virtio_device *vdev)
>>> struct virtio_vsock *vsock = vdev->priv;
>>> struct virtio_vsock_pkt *pkt;
>>> + mutex_lock(&the_virtio_vsock_mutex);
>>> +
>>> + vdev->priv = NULL;
>>> + rcu_assign_pointer(the_virtio_vsock, NULL);
>>
>> This is still suspicious, can we access the_virtio_vsock through vdev->priv?
>> If yes, we may still get use-after-free since it was not protected by RCU.
> We will free the object only after calling the del_vqs(), so we are sure
> that the vq_callbacks ended and will no longer be invoked.
> So, IIUC it shouldn't happen.
Yes, but any dereference that is not done in vq_callbacks will be very
dangerous in the future.
Thanks
>
>> Another more interesting question, I believe we will do singleton for
>> virtio_vsock structure. Then what's the point of using vdev->priv to access
>> the_virtio_vsock? It looks to me we can it brings extra troubles for doing
>> synchronization.
> I thought about it when I tried to use RCU to stop the worker and I
> think make sense. Maybe can be another series after this will be merged.
>
> @Stefan, what do you think about that?
>
> Thanks,
> Stefano
^ permalink raw reply
* [PATCH v3 net-next] net: socionext: remove set but not used variable 'pkts'
From: YueHaibing @ 2019-07-04 3:37 UTC (permalink / raw)
To: Jassi Brar, Alexei Starovoitov, David S . Miller,
Ilias Apalodimas, Daniel Borkmann, Jakub Kicinski,
Jesper Dangaard Brouer
Cc: YueHaibing, netdev, xdp-newbies, bpf, kernel-janitors
In-Reply-To: <20190704032129.169282-1-yuehaibing@huawei.com>
Fixes gcc '-Wunused-but-set-variable' warning:
drivers/net/ethernet/socionext/netsec.c: In function 'netsec_clean_tx_dring':
drivers/net/ethernet/socionext/netsec.c:637:15: warning:
variable 'pkts' set but not used [-Wunused-but-set-variable]
It is not used since commit ba2b232108d3 ("net: netsec: add XDP support")
Signed-off-by: YueHaibing <yuehaibing@huawei.com>
Acked-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
---
v3: remove misplaced cc in patch log
v2: keep reverse christmas-tree ordering of the local variables
---
drivers/net/ethernet/socionext/netsec.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/socionext/netsec.c b/drivers/net/ethernet/socionext/netsec.c
index 5544a722543f..d8d640b01119 100644
--- a/drivers/net/ethernet/socionext/netsec.c
+++ b/drivers/net/ethernet/socionext/netsec.c
@@ -634,15 +634,14 @@ static void netsec_set_rx_de(struct netsec_priv *priv,
static bool netsec_clean_tx_dring(struct netsec_priv *priv)
{
struct netsec_desc_ring *dring = &priv->desc_ring[NETSEC_RING_TX];
- unsigned int pkts, bytes;
struct netsec_de *entry;
int tail = dring->tail;
+ unsigned int bytes;
int cnt = 0;
if (dring->is_xdp)
spin_lock(&dring->lock);
- pkts = 0;
bytes = 0;
entry = dring->vaddr + DESC_SZ * tail;
^ permalink raw reply related
* Re: [PATCH v1 net-next] net: stmmac: enable clause 45 mdio support
From: Andrew Lunn @ 2019-07-04 3:30 UTC (permalink / raw)
To: Voon, Weifeng
Cc: David S. Miller, Maxime Coquelin, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org, Jose Abreu, Giuseppe Cavallaro,
Florian Fainelli, Alexandre Torgue, biao huang, Ong, Boon Leong,
Kweh, Hock Leong
In-Reply-To: <D6759987A7968C4889FDA6FA91D5CBC8147384B6@PGSMSX103.gar.corp.intel.com>
On Thu, Jul 04, 2019 at 01:33:16AM +0000, Voon, Weifeng wrote:
> > > @@ -155,22 +171,26 @@ static int stmmac_mdio_read(struct mii_bus *bus,
> > int phyaddr, int phyreg)
> > > struct stmmac_priv *priv = netdev_priv(ndev);
> > > unsigned int mii_address = priv->hw->mii.addr;
> > > unsigned int mii_data = priv->hw->mii.data;
> > > - u32 v;
> > > - int data;
> > > u32 value = MII_BUSY;
> > > + int data = 0;
> > > + u32 v;
> > >
> > > value |= (phyaddr << priv->hw->mii.addr_shift)
> > > & priv->hw->mii.addr_mask;
> > > value |= (phyreg << priv->hw->mii.reg_shift) & priv->hw-
> > >mii.reg_mask;
> > > value |= (priv->clk_csr << priv->hw->mii.clk_csr_shift)
> > > & priv->hw->mii.clk_csr_mask;
> > > - if (priv->plat->has_gmac4)
> > > + if (priv->plat->has_gmac4) {
> > > value |= MII_GMAC4_READ;
> > > + if (phyreg & MII_ADDR_C45)
> > > + stmmac_mdio_c45_setup(priv, phyreg, &value, &data);
> > > + }
> > >
> > > if (readl_poll_timeout(priv->ioaddr + mii_address, v, !(v &
> > MII_BUSY),
> > > 100, 10000))
> > > return -EBUSY;
> > >
> > > + writel(data, priv->ioaddr + mii_data);
> >
> > That looks odd. Could you explain why it is needed.
> >
> > Thanks
> > Andrew
>
> Hi Andrew,
> This mdio c45 support needed to access DWC xPCS which is a Clause-45
I mean it looks odd doing a write to the data register in the middle
of stmmac_mdio_read().
Andrew
^ permalink raw reply
* [PATCH 1/1] tools/dtrace: initial implementation of DTrace
From: Kris Van Hees @ 2019-07-04 3:14 UTC (permalink / raw)
To: netdev, bpf, dtrace-devel, linux-kernel
Cc: rostedt, mhiramat, acme, ast, daniel, Peter Zijlstra, Chris Mason
In-Reply-To: <201907040313.x643D8Pg025951@userv0121.oracle.com>
This initial implementation of a tiny subset of DTrace functionality
provides the following options:
dtrace [-lvV] [-b bufsz] -s script
-b set trace buffer size
-l list probes (only works with '-s script' for now)
-s enable or list probes for the specified BPF program
-V report DTrace API version
The patch comprises quite a bit of code due to DTrace requiring a few
crucial components, even in its most basic form.
The code is structured around the command line interface implemented in
dtrace.c. It provides option parsing and drives the three modes of
operation that are currently implemented:
1. Report DTrace API version information.
Report the version information and terminate.
2. List probes in BPF programs.
Initialize the list of probes that DTrace recognizes, load BPF
programs, parse all BPF ELF section names, resolve them into
known probes, and emit the probe names. Then terminate.
3. Load BPF programs and collect tracing data.
Initialize the list of probes that DTrace recognizes, load BPF
programs and attach them to their corresponding probes, set up
perf event output buffers, and start processing tracing data.
This implementation makes extensive use of BPF (handled by dt_bpf.c) and
the perf event output ring buffer (handled by dt_buffer.c). DTrace-style
probe handling (dt_probe.c) offers an interface to probes that hides the
implementation details of the individual probe types by provider (dt_fbt.c
and dt_syscall.c). Probe lookup by name uses a hashtable implementation
(dt_hash.c). The dt_utils.c code populates a list of online CPU ids, so
we know what CPUs we can obtain tracing data from.
Building the tool is trivial because its only dependency (libbpf) is in
the kernel tree under tools/lib/bpf. A simple 'make' in the tools/dtrace
directory suffices.
The 'dtrace' executable needs to run as root because BPF programs cannot
be loaded by non-root users.
Signed-off-by: Kris Van Hees <kris.van.hees@oracle.com>
Reviewed-by: David Mc Lean <david.mclean@oracle.com>
Reviewed-by: Eugene Loh <eugene.loh@oracle.com>
---
MAINTAINERS | 6 +
tools/dtrace/Makefile | 88 ++++++++++
tools/dtrace/bpf_sample.c | 145 ++++++++++++++++
tools/dtrace/dt_bpf.c | 188 +++++++++++++++++++++
tools/dtrace/dt_buffer.c | 331 +++++++++++++++++++++++++++++++++++++
tools/dtrace/dt_fbt.c | 201 ++++++++++++++++++++++
tools/dtrace/dt_hash.c | 211 +++++++++++++++++++++++
tools/dtrace/dt_probe.c | 230 ++++++++++++++++++++++++++
tools/dtrace/dt_syscall.c | 179 ++++++++++++++++++++
tools/dtrace/dt_utils.c | 132 +++++++++++++++
tools/dtrace/dtrace.c | 249 ++++++++++++++++++++++++++++
tools/dtrace/dtrace.h | 13 ++
tools/dtrace/dtrace_impl.h | 101 +++++++++++
13 files changed, 2074 insertions(+)
create mode 100644 tools/dtrace/Makefile
create mode 100644 tools/dtrace/bpf_sample.c
create mode 100644 tools/dtrace/dt_bpf.c
create mode 100644 tools/dtrace/dt_buffer.c
create mode 100644 tools/dtrace/dt_fbt.c
create mode 100644 tools/dtrace/dt_hash.c
create mode 100644 tools/dtrace/dt_probe.c
create mode 100644 tools/dtrace/dt_syscall.c
create mode 100644 tools/dtrace/dt_utils.c
create mode 100644 tools/dtrace/dtrace.c
create mode 100644 tools/dtrace/dtrace.h
create mode 100644 tools/dtrace/dtrace_impl.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 606d1f80bc49..668468834865 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -5474,6 +5474,12 @@ W: https://linuxtv.org
S: Odd Fixes
F: drivers/media/pci/dt3155/
+DTRACE
+M: Kris Van Hees <kris.van.hees@oracle.com>
+L: dtrace-devel@oss.oracle.com
+S: Maintained
+F: tools/dtrace/
+
DVB_USB_AF9015 MEDIA DRIVER
M: Antti Palosaari <crope@iki.fi>
L: linux-media@vger.kernel.org
diff --git a/tools/dtrace/Makefile b/tools/dtrace/Makefile
new file mode 100644
index 000000000000..99fd0f9dd1d6
--- /dev/null
+++ b/tools/dtrace/Makefile
@@ -0,0 +1,88 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# This Makefile is based on samples/bpf.
+#
+# Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+
+DT_VERSION := 2.0.0
+DT_GIT_VERSION := $(shell git rev-parse HEAD 2>/dev/null || \
+ echo Unknown)
+
+DTRACE_PATH ?= $(abspath $(srctree)/$(src))
+TOOLS_PATH := $(DTRACE_PATH)/..
+SAMPLES_PATH := $(DTRACE_PATH)/../../samples
+
+hostprogs-y := dtrace
+
+LIBBPF := $(TOOLS_PATH)/lib/bpf/libbpf.a
+OBJS := dt_bpf.o dt_buffer.o dt_utils.o dt_probe.o \
+ dt_hash.o \
+ dt_fbt.o dt_syscall.o
+
+dtrace-objs := $(OBJS) dtrace.o
+
+always := $(hostprogs-y)
+always += bpf_sample.o
+
+KBUILD_HOSTCFLAGS += -DDT_VERSION=\"$(DT_VERSION)\"
+KBUILD_HOSTCFLAGS += -DDT_GIT_VERSION=\"$(DT_GIT_VERSION)\"
+KBUILD_HOSTCFLAGS += -I$(srctree)/tools/lib
+KBUILD_HOSTCFLAGS += -I$(srctree)/tools/perf
+KBUILD_HOSTCFLAGS += -I$(srctree)/tools/include/uapi
+KBUILD_HOSTCFLAGS += -I$(srctree)/tools/include/
+KBUILD_HOSTCFLAGS += -I$(srctree)/usr/include
+
+KBUILD_HOSTLDLIBS := $(LIBBPF) -lelf
+
+LLC ?= llc
+CLANG ?= clang
+LLVM_OBJCOPY ?= llvm-objcopy
+
+ifdef CROSS_COMPILE
+HOSTCC = $(CROSS_COMPILE)gcc
+CLANG_ARCH_ARGS = -target $(ARCH)
+endif
+
+all:
+ $(MAKE) -C ../../ $(CURDIR)/ DTRACE_PATH=$(CURDIR)
+
+clean:
+ $(MAKE) -C ../../ M=$(CURDIR) clean
+ @rm -f *~
+
+$(LIBBPF): FORCE
+ $(MAKE) -C $(dir $@) RM='rm -rf' LDFLAGS= srctree=$(DTRACE_PATH)/../../ O=
+
+FORCE:
+
+.PHONY: verify_cmds verify_target_bpf $(CLANG) $(LLC)
+
+verify_cmds: $(CLANG) $(LLC)
+ @for TOOL in $^ ; do \
+ if ! (which -- "$${TOOL}" > /dev/null 2>&1); then \
+ echo "*** ERROR: Cannot find LLVM tool $${TOOL}" ;\
+ exit 1; \
+ else true; fi; \
+ done
+
+verify_target_bpf: verify_cmds
+ @if ! (${LLC} -march=bpf -mattr=help > /dev/null 2>&1); then \
+ echo "*** ERROR: LLVM (${LLC}) does not support 'bpf' target" ;\
+ echo " NOTICE: LLVM version >= 3.7.1 required" ;\
+ exit 2; \
+ else true; fi
+
+$(DTRACE_PATH)/*.c: verify_target_bpf $(LIBBPF)
+$(src)/*.c: verify_target_bpf $(LIBBPF)
+
+$(obj)/%.o: $(src)/%.c
+ @echo " CLANG-bpf " $@
+ $(Q)$(CLANG) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) $(EXTRA_CFLAGS) -I$(obj) \
+ -I$(srctree)/tools/testing/selftests/bpf/ \
+ -D__KERNEL__ -D__BPF_TRACING__ -Wno-unused-value -Wno-pointer-sign \
+ -D__TARGET_ARCH_$(ARCH) -Wno-compare-distinct-pointer-types \
+ -Wno-gnu-variable-sized-type-not-at-end \
+ -Wno-address-of-packed-member -Wno-tautological-compare \
+ -Wno-unknown-warning-option $(CLANG_ARCH_ARGS) \
+ -I$(srctree)/samples/bpf/ -include asm_goto_workaround.h \
+ -O2 -emit-llvm -c $< -o -| $(LLC) -march=bpf $(LLC_FLAGS) -filetype=obj -o $@
diff --git a/tools/dtrace/bpf_sample.c b/tools/dtrace/bpf_sample.c
new file mode 100644
index 000000000000..49f350390b5f
--- /dev/null
+++ b/tools/dtrace/bpf_sample.c
@@ -0,0 +1,145 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This sample DTrace BPF tracing program demonstrates how actions can be
+ * associated with different probe types.
+ *
+ * The kprobe/ksys_write probe is a Function Boundary Tracing (FBT) entry probe
+ * on the ksys_write(fd, buf, count) function in the kernel. Arguments to the
+ * function can be retrieved from the CPU registers (struct pt_regs).
+ *
+ * The tracepoint/syscalls/sys_enter_write probe is a System Call entry probe
+ * for the write(d, buf, count) system call. Arguments to the system call can
+ * be retrieved from the tracepoint data passed to the BPF program as context
+ * struct syscall_data) when the probe fires.
+ *
+ * The BPF program associated with each probe prepares a DTrace BPF context
+ * (struct dt_bpf_context) that stores the probe ID and up to 10 arguments.
+ * Only 3 arguments are used in this sample. Then the prorgams call a shared
+ * BPF function (bpf_action) that implements the actual action to be taken when
+ * a probe fires. It prepares a data record to be stored in the tracing buffer
+ * and submits it to the buffer. The data in the data record is obtained from
+ * the DTrace BPF context.
+ *
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#include <uapi/linux/bpf.h>
+#include <linux/ptrace.h>
+#include <linux/version.h>
+#include <uapi/linux/unistd.h>
+#include "bpf_helpers.h"
+
+#include "dtrace.h"
+
+struct syscall_data {
+ struct pt_regs *regs;
+ long syscall_nr;
+ long arg[6];
+};
+
+struct bpf_map_def SEC("maps") buffers = {
+ .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(u32),
+ .max_entries = NR_CPUS,
+};
+
+#if defined(__amd64)
+# define GET_REGS_ARG0(regs) ((regs)->di)
+# define GET_REGS_ARG1(regs) ((regs)->si)
+# define GET_REGS_ARG2(regs) ((regs)->dx)
+# define GET_REGS_ARG3(regs) ((regs)->cx)
+# define GET_REGS_ARG4(regs) ((regs)->r8)
+# define GET_REGS_ARG5(regs) ((regs)->r9)
+#else
+# warning Argument retrieval from pt_regs is not supported yet on this arch.
+# define GET_REGS_ARG0(regs) 0
+# define GET_REGS_ARG1(regs) 0
+# define GET_REGS_ARG2(regs) 0
+# define GET_REGS_ARG3(regs) 0
+# define GET_REGS_ARG4(regs) 0
+# define GET_REGS_ARG5(regs) 0
+#endif
+
+/*
+ * We must pass a valid BPF context pointer because the bpf_perf_event_output()
+ * helper requires a BPF context pointer as first argument (and the verifier is
+ * validating that we pass a value that is known to be a context pointer).
+ *
+ * This BPF function implements the following D action:
+ * {
+ * trace(curthread);
+ * trace(arg0);
+ * trace(arg1);
+ * trace(arg2);
+ * }
+ *
+ * Expected output will look like:
+ * CPU ID
+ * 15 70423 0xffff8c0968bf8ec0 0x00000000000001 0x0055e019eb3f60 0x0000000000002c
+ * 15 18876 0xffff8c0968bf8ec0 0x00000000000001 0x0055e019eb3f60 0x0000000000002c
+ * | | +-- curthread +--> arg0 (fd) +--> arg1 (buf) +-- arg2 (count)
+ * | |
+ * | +--> probe ID
+ * |
+ * +--> CPU the probe fired on
+ */
+static noinline int bpf_action(void *bpf_ctx, struct dt_bpf_context *ctx)
+{
+ int cpu = bpf_get_smp_processor_id();
+ struct data {
+ u32 probe_id; /* mandatory */
+
+ u64 task; /* first data item (current task) */
+ u64 arg0; /* 2nd data item (arg0, fd) */
+ u64 arg1; /* 3rd data item (arg1, buf) */
+ u64 arg2; /* 4th data item (arg2, count) */
+ } rec;
+
+ memset(&rec, 0, sizeof(rec));
+
+ rec.probe_id = ctx->probe_id;
+ rec.task = bpf_get_current_task();
+ rec.arg0 = ctx->argv[0];
+ rec.arg1 = ctx->argv[1];
+ rec.arg2 = ctx->argv[2];
+
+ bpf_perf_event_output(bpf_ctx, &buffers, cpu, &rec, sizeof(rec));
+
+ return 0;
+}
+
+SEC("kprobe/ksys_write")
+int bpf_kprobe(struct pt_regs *regs)
+{
+ struct dt_bpf_context ctx;
+
+ memset(&ctx, 0, sizeof(ctx));
+
+ ctx.probe_id = 18876;
+ ctx.argv[0] = GET_REGS_ARG0(regs);
+ ctx.argv[1] = GET_REGS_ARG1(regs);
+ ctx.argv[2] = GET_REGS_ARG2(regs);
+ ctx.argv[3] = GET_REGS_ARG3(regs);
+ ctx.argv[4] = GET_REGS_ARG4(regs);
+ ctx.argv[5] = GET_REGS_ARG5(regs);
+
+ return bpf_action(regs, &ctx);
+}
+
+SEC("tracepoint/syscalls/sys_enter_write")
+int bpf_tp(struct syscall_data *scd)
+{
+ struct dt_bpf_context ctx;
+
+ memset(&ctx, 0, sizeof(ctx));
+
+ ctx.probe_id = 70423;
+ ctx.argv[0] = scd->arg[0];
+ ctx.argv[1] = scd->arg[1];
+ ctx.argv[2] = scd->arg[2];
+
+ return bpf_action(scd, &ctx);
+}
+
+char _license[] SEC("license") = "GPL";
+u32 _version SEC("version") = LINUX_VERSION_CODE;
diff --git a/tools/dtrace/dt_bpf.c b/tools/dtrace/dt_bpf.c
new file mode 100644
index 000000000000..6db1833c731a
--- /dev/null
+++ b/tools/dtrace/dt_bpf.c
@@ -0,0 +1,188 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file provides the interface for handling BPF. It uses the bpf library
+ * to interact with BPF ELF object files.
+ *
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#include <errno.h>
+#include <stdarg.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <bpf/libbpf.h>
+#include <linux/kernel.h>
+#include <linux/perf_event.h>
+#include <sys/ioctl.h>
+
+#include "dtrace_impl.h"
+
+/*
+ * Validate the output buffer map that is specified in the BPF ELF object. It
+ * must match the following definition to be valid:
+ *
+ * struct bpf_map_def SEC("maps") buffers = {
+ * .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
+ * .key_size = sizeof(u32),
+ * .value_size = sizeof(u32),
+ * .max_entries = num,
+ * };
+ * where num is greater than dt_maxcpuid.
+ */
+static int is_valid_buffers(const struct bpf_map_def *mdef)
+{
+ return mdef->type == BPF_MAP_TYPE_PERF_EVENT_ARRAY &&
+ mdef->key_size == sizeof(u32) &&
+ mdef->value_size == sizeof(u32) &&
+ mdef->max_entries > dt_maxcpuid;
+}
+
+/*
+ * List the probes specified in the given BPF ELF object file.
+ */
+int dt_bpf_list_probes(const char *fn)
+{
+ struct bpf_object *obj;
+ struct bpf_program *prog;
+ int rc, fd;
+
+ libbpf_set_print(NULL);
+
+ /*
+ * Listing probes is done before the DTrace command line utility loads
+ * the supplied programs. We load them here without attaching them to
+ * probes so that we can retrieve the ELF section names for each BPF
+ * program. The section name indicates the probe that the program is
+ * associated with.
+ */
+ rc = bpf_prog_load(fn, BPF_PROG_TYPE_UNSPEC, &obj, &fd);
+ if (rc)
+ return rc;
+
+ /*
+ * Loop through the programs in the BPF ELF object, and try to resolve
+ * the section names into probes. Use the supplied callback function
+ * to emit the probe description.
+ */
+ for (prog = bpf_program__next(NULL, obj); prog != NULL;
+ prog = bpf_program__next(prog, obj)) {
+ struct dt_probe *probe;
+
+ probe = dt_probe_resolve_event(bpf_program__title(prog, false));
+
+ printf("%5d %10s %17s %33s %s\n", probe->id,
+ probe->prv_name ? probe->prv_name : "",
+ probe->mod_name ? probe->mod_name : "",
+ probe->fun_name ? probe->fun_name : "",
+ probe->prb_name ? probe->prb_name : "");
+ }
+
+
+ /* Done with the BPF ELF object. */
+ bpf_object__close(obj);
+
+ return 0;
+}
+
+/*
+ * Load the given BPF ELF object file.
+ */
+int dt_bpf_load_file(const char *fn)
+{
+ struct bpf_object *obj;
+ struct bpf_map *map;
+ struct bpf_program *prog;
+ int rc, fd;
+
+ libbpf_set_print(NULL);
+
+ /* Load the BPF ELF object file. */
+ rc = bpf_prog_load(fn, BPF_PROG_TYPE_UNSPEC, &obj, &fd);
+ if (rc)
+ return rc;
+
+ /* Validate buffers map. */
+ map = bpf_object__find_map_by_name(obj, "buffers");
+ if (map && is_valid_buffers(bpf_map__def(map)))
+ dt_bufmap_fd = bpf_map__fd(map);
+ else
+ goto fail;
+
+ /*
+ * Loop through the programs and resolve each into the matching probe.
+ * Attach the program to the probe.
+ */
+ for (prog = bpf_program__next(NULL, obj); prog != NULL;
+ prog = bpf_program__next(prog, obj)) {
+ struct dt_probe *probe;
+
+ probe = dt_probe_resolve_event(bpf_program__title(prog, false));
+ if (!probe)
+ return -ENOENT;
+ if (probe->prov && probe->prov->attach)
+ probe->prov->attach(bpf_program__title(prog, false),
+ bpf_program__fd(prog));
+ }
+
+ return 0;
+
+fail:
+ bpf_object__close(obj);
+ return -EINVAL;
+}
+
+/*
+ * Store the (key, value) pair in the map referenced by the given fd.
+ */
+int dt_bpf_map_update(int fd, const void *key, const void *val)
+{
+ union bpf_attr attr;
+
+ memset(&attr, 0, sizeof(attr));
+
+ attr.map_fd = fd;
+ attr.key = (u64)(unsigned long)key;
+ attr.value = (u64)(unsigned long)val;
+ attr.flags = 0;
+
+ return bpf(BPF_MAP_UPDATE_ELEM, &attr);
+}
+
+/*
+ * Attach a trace event and associate a BPF program with it.
+ */
+int dt_bpf_attach(int event_id, int bpf_fd)
+{
+ int event_fd;
+ int rc;
+ struct perf_event_attr attr = {};
+
+ attr.type = PERF_TYPE_TRACEPOINT;
+ attr.sample_type = PERF_SAMPLE_RAW;
+ attr.sample_period = 1;
+ attr.wakeup_events = 1;
+ attr.config = event_id;
+
+ /* Register the event (based on its id), and obtain a fd. */
+ event_fd = perf_event_open(&attr, -1, 0, -1, 0);
+ if (event_fd < 0) {
+ perror("sys_perf_event_open");
+ return -1;
+ }
+
+ /* Enable the probe. */
+ rc = ioctl(event_fd, PERF_EVENT_IOC_ENABLE, 0);
+ if (rc < 0) {
+ perror("PERF_EVENT_IOC_ENABLE");
+ return -1;
+ }
+
+ /* Associate the BPF program with the event. */
+ rc = ioctl(event_fd, PERF_EVENT_IOC_SET_BPF, bpf_fd);
+ if (rc < 0) {
+ perror("PERF_EVENT_IOC_SET_BPF");
+ return -1;
+ }
+
+ return 0;
+}
diff --git a/tools/dtrace/dt_buffer.c b/tools/dtrace/dt_buffer.c
new file mode 100644
index 000000000000..1714388a7ae6
--- /dev/null
+++ b/tools/dtrace/dt_buffer.c
@@ -0,0 +1,331 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file provides the tracing buffer handling for DTrace. It makes use of
+ * the perf event output ring buffers that can be written to from BPF programs.
+ *
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#include <errno.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <syscall.h>
+#include <unistd.h>
+#include <sys/epoll.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <linux/bpf.h>
+#include <linux/perf_event.h>
+
+#include "dtrace_impl.h"
+
+/*
+ * Probe data is recorded in per-CPU perf ring buffers.
+ */
+struct dtrace_buffer {
+ int cpu; /* ID of CPU that uses this buffer */
+ int fd; /* fd of perf output buffer */
+ size_t page_size; /* size of each page in buffer */
+ size_t data_size; /* total buffer size */
+ void *base; /* address of buffer */
+};
+
+static struct dtrace_buffer *dt_buffers;
+
+/*
+ * File descriptor for the BPF map that holds the buffers for the online CPUs.
+ * The map is a bpf_array indexed by CPU id, and it stores a file descriptor as
+ * value (the fd for the perf_event that represents the CPU buffer).
+ */
+int dt_bufmap_fd = -1;
+
+/*
+ * Create a perf_event buffer for the given DTrace buffer. This will create
+ * a perf_event ring_buffer, mmap it, and enable the perf_event that owns the
+ * buffer.
+ */
+static int perf_buffer_open(struct dtrace_buffer *buf)
+{
+ int pefd;
+ struct perf_event_attr attr = {};
+
+ /*
+ * Event configuration for BPF-generated output in perf_event ring
+ * buffers.
+ */
+ attr.config = PERF_COUNT_SW_BPF_OUTPUT;
+ attr.type = PERF_TYPE_SOFTWARE;
+ attr.sample_type = PERF_SAMPLE_RAW;
+ attr.sample_period = 1;
+ attr.wakeup_events = 1;
+ pefd = perf_event_open(&attr, -1, buf->cpu, -1, PERF_FLAG_FD_CLOEXEC);
+ if (pefd < 0) {
+ fprintf(stderr, "perf_event_open(cpu %d): %s\n", buf->cpu,
+ strerror(errno));
+ goto fail;
+ }
+
+ /*
+ * We add buf->page_size to the buf->data_size, because perf maintains
+ * a meta-data page at the beginning of the memory region. That page
+ * is used for reader/writer symchronization.
+ */
+ buf->fd = pefd;
+ buf->base = mmap(NULL, buf->page_size + buf->data_size,
+ PROT_READ | PROT_WRITE, MAP_SHARED, buf->fd, 0);
+ if (!buf->base)
+ goto fail;
+
+ if (ioctl(pefd, PERF_EVENT_IOC_ENABLE, 0) < 0) {
+ fprintf(stderr, "PERF_EVENT_IOC_ENABLE(cpu %d): %s\n",
+ buf->cpu, strerror(errno));
+ goto fail;
+ }
+
+ return 0;
+
+fail:
+ if (buf->base) {
+ munmap(buf->base, buf->page_size + buf->data_size);
+ buf->base = NULL;
+ }
+ if (buf->fd) {
+ close(buf->fd);
+ buf->fd = -1;
+ }
+
+ return -1;
+}
+
+/*
+ * Close the given DTrace buffer. This function disables the perf_event that
+ * owns the buffer, munmaps the memory space, and closes the perf buffer fd.
+ */
+static void perf_buffer_close(struct dtrace_buffer *buf)
+{
+ /*
+ * If the perf buffer failed to open, there is no need to close it.
+ */
+ if (buf->fd < 0)
+ return;
+
+ if (ioctl(buf->fd, PERF_EVENT_IOC_DISABLE, 0) < 0)
+ fprintf(stderr, "PERF_EVENT_IOC_DISABLE(cpu %d): %s\n",
+ buf->cpu, strerror(errno));
+
+ munmap(buf->base, buf->page_size + buf->data_size);
+
+ if (close(buf->fd))
+ fprintf(stderr, "perf buffer close(cpu %d): %s\n",
+ buf->cpu, strerror(errno));
+
+ buf->base = NULL;
+ buf->fd = -1;
+}
+
+/*
+ * Initialize the probe data buffers (one per online CPU). Each buffer will
+ * contain the given number of pages (i.e. total size of each buffer will be
+ * num_pages * getpagesize()). This function also sets up an event polling
+ * descriptor that monitors all CPU buffers at once.
+ */
+int dt_buffer_init(int num_pages)
+{
+ int i;
+ int epoll_fd;
+
+ if (dt_bufmap_fd < 0)
+ return -EINVAL;
+
+ /* Allocate the per-CPU buffer structs. */
+ dt_buffers = calloc(dt_numcpus, sizeof(struct dtrace_buffer));
+ if (dt_buffers == NULL)
+ return -ENOMEM;
+
+ /* Set up the event polling file descriptor. */
+ epoll_fd = epoll_create1(EPOLL_CLOEXEC);
+ if (epoll_fd < 0) {
+ free(dt_buffers);
+ return -errno;
+ }
+
+ for (i = 0; i < dt_numcpus; i++) {
+ int cpu = dt_cpuids[i];
+ struct epoll_event ev;
+ struct dtrace_buffer *buf = &dt_buffers[i];
+
+ buf->cpu = cpu;
+ buf->page_size = getpagesize();
+ buf->data_size = num_pages * buf->page_size;
+
+ /* Try to create the perf buffer for this DTrace buffer. */
+ if (perf_buffer_open(buf) == -1)
+ continue;
+
+ /* Store the perf buffer fd in the buffer map. */
+ dt_bpf_map_update(dt_bufmap_fd, &cpu, &buf->fd);
+
+ /* Add the buffer to the event polling descriptor. */
+ ev.events = EPOLLIN;
+ ev.data.ptr = buf;
+ if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, buf->fd, &ev) == -1) {
+ fprintf(stderr, "EPOLL_CTL_ADD(cpu %d): %s\n",
+ buf->cpu, strerror(errno));
+ continue;
+ }
+ }
+
+ return epoll_fd;
+}
+
+/*
+ * Clean up the buffers.
+ */
+void dt_buffer_exit(int epoll_fd)
+{
+ int i;
+
+ for (i = 0; i < dt_numcpus; i++)
+ perf_buffer_close(&dt_buffers[i]);
+
+ free(dt_buffers);
+ close(epoll_fd);
+}
+
+/*
+ * Read the data_head offset from the header page of the ring buffer. The
+ * argument is declared 'volatile' because it references a memory mapped page
+ * that the kernel may be writing to while we access it here.
+ */
+static u64 read_rb_head(volatile struct perf_event_mmap_page *rb_page)
+{
+ u64 head = rb_page->data_head;
+
+ asm volatile("" ::: "memory");
+
+ return head;
+}
+
+/*
+ * Write the data_tail offset in the header page of the ring buffer. The
+ * argument is declared 'volatile' because it references a memory mapped page
+ * that the kernel may be writing to while we access it here.
+ */
+static void write_rb_tail(volatile struct perf_event_mmap_page *rb_page,
+ u64 tail)
+{
+ asm volatile("" ::: "memory");
+
+ rb_page->data_tail = tail;
+}
+
+/*
+ * Process and output the probe data at the supplied address.
+ */
+static int output_event(int cpu, u64 *buf)
+{
+ u8 *data = (u8 *)buf;
+ struct perf_event_header *hdr;
+
+ hdr = (struct perf_event_header *)data;
+ data += sizeof(struct perf_event_header);
+
+ if (hdr->type == PERF_RECORD_SAMPLE) {
+ u8 *ptr = data;
+ u32 i, size, probe_id;
+
+ /*
+ * struct {
+ * struct perf_event_header header;
+ * u32 size;
+ * u32 probe_id;
+ * u32 gap;
+ * u64 data[n];
+ * }
+ * and data points to the 'size' member at this point.
+ */
+ if (ptr > (u8 *)buf + hdr->size) {
+ fprintf(stderr, "BAD: corrupted sample header\n");
+ goto out;
+ }
+
+ size = *(u32 *)data;
+ data += sizeof(size);
+ ptr += sizeof(size) + size;
+ if (ptr != (u8 *)buf + hdr->size) {
+ fprintf(stderr, "BAD: invalid sample size\n");
+ goto out;
+ }
+
+ probe_id = *(u32 *)data;
+ data += sizeof(probe_id);
+ size -= sizeof(probe_id);
+ data += sizeof(u32); /* skip 32-bit gap */
+ size -= sizeof(u32);
+ buf = (u64 *)data;
+
+ printf("%3d %6d ", cpu, probe_id);
+ for (i = 0, size /= sizeof(u64); i < size; i++)
+ printf("%#016lx ", buf[i]);
+ printf("\n");
+ } else if (hdr->type == PERF_RECORD_LOST) {
+ u64 lost;
+
+ /*
+ * struct {
+ * struct perf_event_header header;
+ * u64 id;
+ * u64 lost;
+ * }
+ * and data points to the 'id' member at this point.
+ */
+ lost = *(u64 *)(data + sizeof(u64));
+
+ printf("[%ld probes dropped]\n", lost);
+ } else
+ fprintf(stderr, "UNKNOWN: record type %d\n", hdr->type);
+
+out:
+ return hdr->size;
+}
+
+/*
+ * Process the available probe data in the given buffer.
+ */
+static void process_data(struct dtrace_buffer *buf)
+{
+ /* This is volatile because the kernel may be updating the content. */
+ volatile struct perf_event_mmap_page *rb_page = buf->base;
+ u8 *base = (u8 *)buf->base +
+ buf->page_size;
+ u64 head = read_rb_head(rb_page);
+
+ while (rb_page->data_tail != head) {
+ u64 tail = rb_page->data_tail;
+ u64 *ptr = (u64 *)(base + tail % buf->data_size);
+ int len;
+
+ len = output_event(buf->cpu, ptr);
+
+ write_rb_tail(rb_page, tail + len);
+ head = read_rb_head(rb_page);
+ }
+}
+
+/*
+ * Wait for data to become available in any of the buffers.
+ */
+int dt_buffer_poll(int epoll_fd, int timeout)
+{
+ struct epoll_event events[dt_numcpus];
+ int i, cnt;
+
+ cnt = epoll_wait(epoll_fd, events, dt_numcpus, timeout);
+ if (cnt < 0)
+ return -errno;
+
+ for (i = 0; i < cnt; i++)
+ process_data((struct dtrace_buffer *)events[i].data.ptr);
+
+ return cnt;
+}
diff --git a/tools/dtrace/dt_fbt.c b/tools/dtrace/dt_fbt.c
new file mode 100644
index 000000000000..fcf95243bf97
--- /dev/null
+++ b/tools/dtrace/dt_fbt.c
@@ -0,0 +1,201 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * The Function Boundary Tracing (FBT) provider for DTrace.
+ *
+ * FBT probes are exposed by the kernel as kprobes. They are listed in the
+ * TRACEFS/available_filter_functions file. Some kprobes are associated with
+ * a specific kernel module, while most are in the core kernel.
+ *
+ * Mapping from event name to DTrace probe name:
+ *
+ * <name> fbt:vmlinux:<name>:entry
+ * fbt:vmlinux:<name>:return
+ * or
+ * <name> [<modname>] fbt:<modname>:<name>:entry
+ * fbt:<modname>:<name>:return
+ *
+ * Mapping from BPF section name to DTrace probe name:
+ *
+ * kprobe/<name> fbt:vmlinux:<name>:entry
+ * kretprobe/<name> fbt:vmlinux:<name>:return
+ *
+ * (Note that the BPF section does not carry information about the module that
+ * the function is found in. This means that BPF section name cannot be used
+ * to distinguish between functions with the same name occurring in different
+ * modules.)
+ *
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <linux/bpf.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+
+#include "dtrace_impl.h"
+
+#define KPROBE_EVENTS TRACEFS "kprobe_events"
+#define PROBE_LIST TRACEFS "available_filter_functions"
+
+static const char provname[] = "fbt";
+static const char modname[] = "vmlinux";
+
+/*
+ * Scan the PROBE_LIST file and add entry and return probes for every function
+ * that is listed.
+ */
+static int fbt_populate(void)
+{
+ FILE *f;
+ char buf[256];
+ char *p;
+
+ f = fopen(PROBE_LIST, "r");
+ if (f == NULL)
+ return -1;
+
+ while (fgets(buf, sizeof(buf), f)) {
+ /*
+ * Here buf is either "funcname\n" or "funcname [modname]\n".
+ */
+ p = strchr(buf, '\n');
+ if (p) {
+ *p = '\0';
+ if (p > buf && *(--p) == ']')
+ *p = '\0';
+ } else {
+ /* If we didn't see a newline, the line was too long.
+ * Report it, and continue until the end of the line.
+ */
+ fprintf(stderr, "%s: Line too long: %s\n",
+ PROBE_LIST, buf);
+ do
+ fgets(buf, sizeof(buf), f);
+ while (strchr(buf, '\n') == NULL);
+ continue;
+ }
+
+ /*
+ * Now buf is either "funcname" or "funcname [modname". If
+ * there is no module name provided, we will use the default.
+ */
+ p = strchr(buf, ' ');
+ if (p) {
+ *p++ = '\0';
+ if (*p == '[')
+ p++;
+ }
+
+ dt_probe_new(&dt_fbt, provname, p ? p : modname, buf, "entry");
+ dt_probe_new(&dt_fbt, provname, p ? p : modname, buf, "return");
+ }
+
+ fclose(f);
+
+ return 0;
+}
+
+#define ENTRY_PREFIX "kprobe/"
+#define EXIT_PREFIX "kretprobe/"
+
+/*
+ * Perform a probe lookup based on an event name (BPF ELF section name).
+ */
+static struct dt_probe *fbt_resolve_event(const char *name)
+{
+ const char *prbname;
+ struct dt_probe tmpl;
+ struct dt_probe *probe;
+
+ if (!name)
+ return NULL;
+
+ if (strncmp(name, ENTRY_PREFIX, sizeof(ENTRY_PREFIX) - 1) == 0) {
+ name += sizeof(ENTRY_PREFIX) - 1;
+ prbname = "entry";
+ } else if (strncmp(name, EXIT_PREFIX, sizeof(EXIT_PREFIX) - 1) == 0) {
+ name += sizeof(EXIT_PREFIX) - 1;
+ prbname = "return";
+ } else
+ return NULL;
+
+ memset(&tmpl, 0, sizeof(tmpl));
+ tmpl.prv_name = provname;
+ tmpl.mod_name = modname;
+ tmpl.fun_name = name;
+ tmpl.prb_name = prbname;
+
+ probe = dt_probe_by_name(&tmpl);
+
+ return probe;
+}
+
+/*
+ * Attach the given BPF program (identified by its file descriptor) to the
+ * kprobe identified by the given section name.
+ */
+static int fbt_attach(const char *name, int bpf_fd)
+{
+ char efn[256];
+ char buf[256];
+ int event_id, fd, rc;
+
+ name += 7; /* skip "kprobe/" */
+ snprintf(buf, sizeof(buf), "p:%s %s\n", name, name);
+
+ /*
+ * Register the kprobe with the tracing subsystem. This will create
+ * a tracepoint event.
+ */
+ fd = open(KPROBE_EVENTS, O_WRONLY | O_APPEND);
+ if (fd < 0) {
+ perror(KPROBE_EVENTS);
+ return -1;
+ }
+ rc = write(fd, buf, strlen(buf));
+ if (rc < 0) {
+ perror(KPROBE_EVENTS);
+ close(fd);
+ return -1;
+ }
+ close(fd);
+
+ /*
+ * Read the tracepoint event id for the kprobe we just registered.
+ */
+ strcpy(efn, EVENTSFS);
+ strcat(efn, "kprobes/");
+ strcat(efn, name);
+ strcat(efn, "/id");
+
+ fd = open(efn, O_RDONLY);
+ if (fd < 0) {
+ perror(efn);
+ return -1;
+ }
+ rc = read(fd, buf, sizeof(buf));
+ if (rc < 0 || rc >= sizeof(buf)) {
+ perror(efn);
+ close(fd);
+ return -1;
+ }
+ close(fd);
+ buf[rc] = '\0';
+ event_id = atoi(buf);
+
+ /*
+ * Attaching a BPF program (by file descriptor) to an event (by ID) is
+ * a generic operation provided by the BPF interface code.
+ */
+ return dt_bpf_attach(event_id, bpf_fd);
+}
+
+struct dt_provider dt_fbt = {
+ .name = "fbt",
+ .populate = &fbt_populate,
+ .resolve_event = &fbt_resolve_event,
+ .attach = &fbt_attach,
+};
diff --git a/tools/dtrace/dt_hash.c b/tools/dtrace/dt_hash.c
new file mode 100644
index 000000000000..b1f563bc0773
--- /dev/null
+++ b/tools/dtrace/dt_hash.c
@@ -0,0 +1,211 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file provides a generic hashtable implementation for probes.
+ *
+ * The hashtable is created with 4 user-provided functions:
+ * hval(probe) - calculate a hash value for the given probe
+ * cmp(probe1, probe2) - compare two probes
+ * add(head, probe) - add a probe to a list of probes
+ * del(head, probe) - delete a probe from a list of probes
+ *
+ * Probes are hashed into a hashtable slot based on the return value of
+ * hval(probe). Each hashtable slot holds a list of buckets, with each
+ * bucket storing probes that are equal under the cmp(probe1, probe2)
+ * function. Probes are added to the list of probes in a bucket using the
+ * add(head, probe) function, and they are deleted using a call to the
+ * del(head, probe) function.
+ *
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#include <errno.h>
+#include <stdint.h>
+#include <stdlib.h>
+
+#include "dtrace_impl.h"
+
+/*
+ * Hashtable implementation for probes.
+ */
+struct dt_hbucket {
+ u32 hval;
+ struct dt_hbucket *next;
+ struct dt_probe *head;
+ int nprobes;
+};
+
+struct dt_htab {
+ struct dt_hbucket **tab;
+ int size;
+ int mask;
+ int nbuckets;
+ dt_hval_fn hval; /* calculate hash value */
+ dt_cmp_fn cmp; /* compare 2 probes */
+ dt_add_fn add; /* add probe to list */
+ dt_del_fn del; /* delete probe from list */
+};
+
+/*
+ * Create a new (empty) hashtable.
+ */
+struct dt_htab *dt_htab_new(dt_hval_fn hval, dt_cmp_fn cmp, dt_add_fn add,
+ dt_del_fn del)
+{
+ struct dt_htab *htab = malloc(sizeof(struct dt_htab));
+
+ if (!htab)
+ return NULL;
+
+ htab->size = 1;
+ htab->mask = htab->size - 1;
+ htab->nbuckets = 0;
+ htab->hval = hval;
+ htab->cmp = cmp;
+ htab->add = add;
+ htab->del = del;
+
+ htab->tab = calloc(htab->size, sizeof(struct dt_hbucket *));
+ if (!htab->tab) {
+ free(htab);
+ return NULL;
+ }
+
+ return htab;
+}
+
+/*
+ * Resize the hashtable by doubling the number of slots.
+ */
+static int resize(struct dt_htab *htab)
+{
+ int i;
+ int osize = htab->size;
+ int nsize = osize << 1;
+ int nmask = nsize - 1;
+ struct dt_hbucket **ntab;
+
+ ntab = calloc(nsize, sizeof(struct dt_hbucket *));
+ if (!ntab)
+ return -ENOMEM;
+
+ for (i = 0; i < osize; i++) {
+ struct dt_hbucket *bucket, *next;
+
+ for (bucket = htab->tab[i]; bucket; bucket = next) {
+ int idx = bucket->hval & nmask;
+
+ next = bucket->next;
+ bucket->next = ntab[idx];
+ ntab[idx] = bucket;
+ }
+ }
+
+ free(htab->tab);
+ htab->tab = ntab;
+ htab->size = nsize;
+ htab->mask = nmask;
+
+ return 0;
+}
+
+/*
+ * Add a probe to the hashtable. Resize if necessary, and allocate a new
+ * bucket if necessary.
+ */
+int dt_htab_add(struct dt_htab *htab, struct dt_probe *probe)
+{
+ u32 hval = htab->hval(probe);
+ int idx;
+ struct dt_hbucket *bucket;
+
+retry:
+ idx = hval & htab->mask;
+ for (bucket = htab->tab[idx]; bucket; bucket = bucket->next) {
+ if (htab->cmp(bucket->head, probe) == 0)
+ goto add;
+ }
+
+ if ((htab->nbuckets >> 1) > htab->size) {
+ int err;
+
+ err = resize(htab);
+ if (err)
+ return err;
+
+ goto retry;
+ }
+
+ bucket = malloc(sizeof(struct dt_hbucket));
+ if (!bucket)
+ return -ENOMEM;
+
+ bucket->hval = hval;
+ bucket->next = htab->tab[idx];
+ bucket->head = NULL;
+ bucket->nprobes = 0;
+ htab->tab[idx] = bucket;
+ htab->nbuckets++;
+
+add:
+ bucket->head = htab->add(bucket->head, probe);
+ bucket->nprobes++;
+
+ return 0;
+}
+
+/*
+ * Find a probe in the hashtable.
+ */
+struct dt_probe *dt_htab_lookup(const struct dt_htab *htab,
+ const struct dt_probe *probe)
+{
+ u32 hval = htab->hval(probe);
+ int idx = hval & htab->mask;
+ struct dt_hbucket *bucket;
+
+ for (bucket = htab->tab[idx]; bucket; bucket = bucket->next) {
+ if (htab->cmp(bucket->head, probe) == 0)
+ return bucket->head;
+ }
+
+ return NULL;
+}
+
+/*
+ * Remove a probe from the hashtable. If we are deleting the last probe in a
+ * bucket, get rid of the bucket.
+ */
+int dt_htab_del(struct dt_htab *htab, struct dt_probe *probe)
+{
+ u32 hval = htab->hval(probe);
+ int idx = hval & htab->mask;
+ struct dt_hbucket *bucket;
+ struct dt_probe *head;
+
+ for (bucket = htab->tab[idx]; bucket; bucket = bucket->next) {
+ if (htab->cmp(bucket->head, probe) == 0)
+ break;
+ }
+
+ if (bucket == NULL)
+ return -ENOENT;
+
+ head = htab->del(bucket->head, probe);
+ if (!head) {
+ struct dt_hbucket *b = htab->tab[idx];
+
+ if (bucket == b)
+ htab->tab[idx] = bucket->next;
+ else {
+ while (b->next != bucket)
+ b = b->next;
+
+ b->next = bucket->next;
+ }
+
+ htab->nbuckets--;
+ free(bucket);
+ } else
+ bucket->head = head;
+
+ return 0;
+}
diff --git a/tools/dtrace/dt_probe.c b/tools/dtrace/dt_probe.c
new file mode 100644
index 000000000000..0b6228eaff29
--- /dev/null
+++ b/tools/dtrace/dt_probe.c
@@ -0,0 +1,230 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This file implements the interface to probes grouped by provider.
+ *
+ * Probes are named by a set of 4 identifiers:
+ * - provider name
+ * - module name
+ * - function name
+ * - probe name
+ *
+ * The Fully Qualified Name (FQN) is "provider:module:function:name".
+ *
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#include <errno.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <linux/bpf.h>
+#include <linux/kernel.h>
+
+#include "dtrace_impl.h"
+
+static struct dt_provider *dt_providers[] = {
+ &dt_fbt,
+ &dt_syscall,
+ };
+
+static struct dt_htab *ht_byfqn;
+
+static u32 next_probe_id;
+
+/*
+ * Calculate a hash value based on a given string and an initial value. The
+ * initial value is used to calculate compound hash values, e.g.
+ *
+ * u32 hval;
+ *
+ * hval = str2hval(str1, 0);
+ * hval = str2hval(str2, hval);
+ */
+static u32 str2hval(const char *p, u32 hval)
+{
+ u32 g;
+
+ if (!p)
+ return hval;
+
+ while (*p) {
+ hval = (hval << 4) + *p++;
+ g = hval & 0xf0000000;
+ if (g != 0)
+ hval ^= g >> 24;
+
+ hval &= ~g;
+ }
+
+ return hval;
+}
+
+/*
+ * String compare function that can handle either or both strings being NULL.
+ */
+static int safe_strcmp(const char *p, const char *q)
+{
+ return (!p) ? (!q) ? 0
+ : -1
+ : (!q) ? 1
+ : strcmp(p, q);
+}
+
+/*
+ * Calculate the hash value of a probe as the cummulative hash value of the
+ * FQN.
+ */
+static u32 fqn_hval(const struct dt_probe *probe)
+{
+ u32 hval = 0;
+
+ hval = str2hval(probe->prv_name, hval);
+ hval = str2hval(":", hval);
+ hval = str2hval(probe->mod_name, hval);
+ hval = str2hval(":", hval);
+ hval = str2hval(probe->fun_name, hval);
+ hval = str2hval(":", hval);
+ hval = str2hval(probe->prb_name, hval);
+
+ return hval;
+}
+
+/*
+ * Compare two probes based on the FQN.
+ */
+static int fqn_cmp(const struct dt_probe *p, const struct dt_probe *q)
+{
+ int rc;
+
+ rc = safe_strcmp(p->prv_name, q->prv_name);
+ if (rc)
+ return rc;
+ rc = safe_strcmp(p->mod_name, q->mod_name);
+ if (rc)
+ return rc;
+ rc = safe_strcmp(p->fun_name, q->fun_name);
+ if (rc)
+ return rc;
+ rc = safe_strcmp(p->prb_name, q->prb_name);
+ if (rc)
+ return rc;
+
+ return 0;
+}
+
+/*
+ * Add the given probe 'new' to the double-linked probe list 'head'. Probe
+ * 'new' becomes the new list head.
+ */
+static struct dt_probe *fqn_add(struct dt_probe *head, struct dt_probe *new)
+{
+ if (!head)
+ return new;
+
+ new->he_fqn.next = head;
+ head->he_fqn.prev = new;
+
+ return new;
+}
+
+/*
+ * Remove the given probe 'probe' from the double-linked probe list 'head'.
+ * If we are deleting the current head, the next probe in the list is returned
+ * as the new head. If that value is NULL, the list is now empty.
+ */
+static struct dt_probe *fqn_del(struct dt_probe *head, struct dt_probe *probe)
+{
+ if (head == probe) {
+ if (!probe->he_fqn.next)
+ return NULL;
+
+ head = probe->he_fqn.next;
+ head->he_fqn.prev = NULL;
+ probe->he_fqn.next = NULL;
+
+ return head;
+ }
+
+ if (!probe->he_fqn.next) {
+ probe->he_fqn.prev->he_fqn.next = NULL;
+ probe->he_fqn.prev = NULL;
+
+ return head;
+ }
+
+ probe->he_fqn.prev->he_fqn.next = probe->he_fqn.next;
+ probe->he_fqn.next->he_fqn.prev = probe->he_fqn.prev;
+ probe->he_fqn.prev = probe->he_fqn.next = NULL;
+
+ return head;
+}
+
+/*
+ * Initialize the probe handling by populating the FQN hashtable with probes
+ * from all providers.
+ */
+int dt_probe_init(void)
+{
+ int i;
+
+ ht_byfqn = dt_htab_new(fqn_hval, fqn_cmp, fqn_add, fqn_del);
+
+ for (i = 0; i < ARRAY_SIZE(dt_providers); i++) {
+ if (dt_providers[i]->populate() < 0)
+ return -1;
+ }
+
+ return 0;
+}
+
+/*
+ * Allocate a new probe and add it to the FQN hashtable.
+ */
+int dt_probe_new(const struct dt_provider *prov, const char *pname,
+ const char *mname, const char *fname, const char *name)
+{
+ struct dt_probe *probe;
+
+ probe = malloc(sizeof(struct dt_probe));
+ if (!probe)
+ return -ENOMEM;
+
+ memset(probe, 0, sizeof(struct dt_probe));
+ probe->id = next_probe_id++;
+ probe->prov = prov;
+ probe->prv_name = pname ? strdup(pname) : NULL;
+ probe->mod_name = mname ? strdup(mname) : NULL;
+ probe->fun_name = fname ? strdup(fname) : NULL;
+ probe->prb_name = name ? strdup(name) : NULL;
+
+ dt_htab_add(ht_byfqn, probe);
+
+ return 0;
+}
+
+/*
+ * Perform a probe lookup based on FQN.
+ */
+struct dt_probe *dt_probe_by_name(const struct dt_probe *tmpl)
+{
+ return dt_htab_lookup(ht_byfqn, tmpl);
+}
+
+/*
+ * Resolve an event name (BPF ELF section name) into a probe. We query each
+ * provider, and as soon as we get a hit, we return the result.
+ */
+struct dt_probe *dt_probe_resolve_event(const char *name)
+{
+ int i;
+ struct dt_probe *probe;
+
+ for (i = 0; i < ARRAY_SIZE(dt_providers); i++) {
+ if (!dt_providers[i]->resolve_event)
+ continue;
+ probe = dt_providers[i]->resolve_event(name);
+ if (probe)
+ return probe;
+ }
+
+ return NULL;
+}
diff --git a/tools/dtrace/dt_syscall.c b/tools/dtrace/dt_syscall.c
new file mode 100644
index 000000000000..6695a4a1c701
--- /dev/null
+++ b/tools/dtrace/dt_syscall.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * The syscall provider for DTrace.
+ *
+ * System call probes are exposed by the kernel as tracepoint events in the
+ * "syscalls" group. Entry probe names start with "sys_enter_" and exit probes
+ * start with "sys_exit_".
+ *
+ * Mapping from event name to DTrace probe name:
+ *
+ * syscalls:sys_enter_<name> syscall:vmlinux:<name>:entry
+ * syscalls:sys_exit_<name> syscall:vmlinux:<name>:return
+ *
+ * Mapping from BPF section name to DTrace probe name:
+ *
+ * tracepoint/syscalls/sys_enter_<name> syscall:vmlinux:<name>:entry
+ * tracepoint/syscalls/sys_exit_<name> syscall:vmlinux:<name>:return
+ *
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#include <ctype.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <linux/bpf.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+
+#include "dtrace_impl.h"
+
+static const char provname[] = "syscall";
+static const char modname[] = "vmlinux";
+
+#define PROBE_LIST TRACEFS "available_events"
+
+#define PROV_PREFIX "syscalls:"
+#define ENTRY_PREFIX "sys_enter_"
+#define EXIT_PREFIX "sys_exit_"
+
+/*
+ * Scan the PROBE_LIST file and add probes for any syscalls events.
+ */
+static int syscall_populate(void)
+{
+ FILE *f;
+ char buf[256];
+
+ f = fopen(PROBE_LIST, "r");
+ if (f == NULL)
+ return -1;
+
+ while (fgets(buf, sizeof(buf), f)) {
+ char *p;
+
+ /* * Here buf is "group:event". */
+ p = strchr(buf, '\n');
+ if (p)
+ *p = '\0';
+ else {
+ /*
+ * If we didn't see a newline, the line was too long.
+ * Report it, and continue until the end of the line.
+ */
+ fprintf(stderr, "%s: Line too long: %s\n",
+ PROBE_LIST, buf);
+ do
+ fgets(buf, sizeof(buf), f);
+ while (strchr(buf, '\n') == NULL);
+ continue;
+ }
+
+ /* We need "group:" to match "syscalls:". */
+ p = buf;
+ if (memcmp(p, PROV_PREFIX, sizeof(PROV_PREFIX) - 1) != 0)
+ continue;
+
+ p += sizeof(PROV_PREFIX) - 1;
+ /*
+ * Now p will be just "event", and we are only interested in
+ * events that match "sys_enter_*" or "sys_exit_*".
+ */
+ if (!memcmp(p, ENTRY_PREFIX, sizeof(ENTRY_PREFIX) - 1)) {
+ p += sizeof(ENTRY_PREFIX) - 1;
+ dt_probe_new(&dt_syscall, provname, modname, p,
+ "entry");
+ } else if (!memcmp(p, EXIT_PREFIX, sizeof(EXIT_PREFIX) - 1)) {
+ p += sizeof(EXIT_PREFIX) - 1;
+ dt_probe_new(&dt_syscall, provname, modname, p,
+ "return");
+ }
+ }
+
+ fclose(f);
+
+ return 0;
+}
+
+#define EVENT_PREFIX "tracepoint/syscalls/"
+
+/*
+ * Perform a probe lookup based on an event name (BPF ELF section name).
+ */
+static struct dt_probe *systrace_resolve_event(const char *name)
+{
+ const char *prbname;
+ struct dt_probe tmpl;
+ struct dt_probe *probe;
+
+ if (!name)
+ return NULL;
+
+ /* Exclude anything that is not a syscalls tracepoint */
+ if (strncmp(name, EVENT_PREFIX, sizeof(EVENT_PREFIX) - 1) != 0)
+ return NULL;
+ name += sizeof(EVENT_PREFIX) - 1;
+
+ if (strncmp(name, ENTRY_PREFIX, sizeof(ENTRY_PREFIX) - 1) == 0) {
+ name += sizeof(ENTRY_PREFIX) - 1;
+ prbname = "entry";
+ } else if (strncmp(name, EXIT_PREFIX, sizeof(EXIT_PREFIX) - 1) == 0) {
+ name += sizeof(EXIT_PREFIX) - 1;
+ prbname = "return";
+ } else
+ return NULL;
+
+ memset(&tmpl, 0, sizeof(tmpl));
+ tmpl.prv_name = provname;
+ tmpl.mod_name = modname;
+ tmpl.fun_name = name;
+ tmpl.prb_name = prbname;
+
+ probe = dt_probe_by_name(&tmpl);
+
+ return probe;
+}
+
+#define SYSCALLSFS EVENTSFS "syscalls/"
+
+/*
+ * Attach the given BPF program (identified by its file descriptor) to the
+ * event identified by the given section name.
+ */
+static int syscall_attach(const char *name, int bpf_fd)
+{
+ char efn[256];
+ char buf[256];
+ int event_id, fd, rc;
+
+ name += sizeof(EVENT_PREFIX) - 1;
+ strcpy(efn, SYSCALLSFS);
+ strcat(efn, name);
+ strcat(efn, "/id");
+
+ fd = open(efn, O_RDONLY);
+ if (fd < 0) {
+ perror(efn);
+ return -1;
+ }
+ rc = read(fd, buf, sizeof(buf));
+ if (rc < 0 || rc >= sizeof(buf)) {
+ perror(efn);
+ close(fd);
+ return -1;
+ }
+ close(fd);
+ buf[rc] = '\0';
+ event_id = atoi(buf);
+
+ return dt_bpf_attach(event_id, bpf_fd);
+}
+
+struct dt_provider dt_syscall = {
+ .name = "syscall",
+ .populate = &syscall_populate,
+ .resolve_event = &systrace_resolve_event,
+ .attach = &syscall_attach,
+};
diff --git a/tools/dtrace/dt_utils.c b/tools/dtrace/dt_utils.c
new file mode 100644
index 000000000000..55d51bae1d97
--- /dev/null
+++ b/tools/dtrace/dt_utils.c
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "dtrace_impl.h"
+
+#define BUF_SIZE 1024 /* max size for online cpu data */
+
+int dt_numcpus; /* number of online CPUs */
+int dt_maxcpuid; /* highest CPU id */
+int *dt_cpuids; /* list of CPU ids */
+
+/*
+ * Populate the online CPU id information from sysfs data. We only do this
+ * once because we do not care about CPUs coming online after we started
+ * tracing. If a CPU goes offline during tracing, we do not care either
+ * because that simply means that it won't be writing any new probe data into
+ * its buffer.
+ */
+void cpu_list_populate(void)
+{
+ char buf[BUF_SIZE];
+ int fd, cnt, start, end, i;
+ int *cpu;
+ char *p, *q;
+
+ fd = open("/sys/devices/system/cpu/online", O_RDONLY);
+ if (fd < 0)
+ goto fail;
+ cnt = read(fd, buf, sizeof(buf));
+ close(fd);
+ if (cnt <= 0)
+ goto fail;
+
+ /*
+ * The string should always end with a newline, but let's make sure.
+ */
+ if (buf[cnt - 1] == '\n')
+ buf[--cnt] = 0;
+
+ /*
+ * Count how many CPUs we have.
+ */
+ dt_numcpus = 0;
+ p = buf;
+ do {
+ start = (int)strtol(p, &q, 10);
+ switch (*q) {
+ case '-': /* range */
+ p = q + 1;
+ end = (int)strtol(p, &q, 10);
+ dt_numcpus += end - start + 1;
+ if (*q == 0) { /* end of string */
+ p = q;
+ break;
+ }
+ if (*q != ',')
+ goto fail;
+ p = q + 1;
+ break;
+ case 0: /* end of string */
+ dt_numcpus++;
+ p = q;
+ break;
+ case ',': /* gap */
+ dt_numcpus++;
+ p = q + 1;
+ break;
+ }
+ } while (*p != 0);
+
+ dt_cpuids = calloc(dt_numcpus, sizeof(int));
+ cpu = dt_cpuids;
+
+ /*
+ * Fill in the CPU ids.
+ */
+ p = buf;
+ do {
+ start = (int)strtol(p, &q, 10);
+ switch (*q) {
+ case '-': /* range */
+ p = q + 1;
+ end = (int)strtol(p, &q, 10);
+ for (i = start; i <= end; i++)
+ *cpu++ = i;
+ if (*q == 0) { /* end of string */
+ p = q;
+ break;
+ }
+ if (*q != ',')
+ goto fail;
+ p = q + 1;
+ break;
+ case 0: /* end of string */
+ *cpu = start;
+ p = q;
+ break;
+ case ',': /* gap */
+ *cpu++ = start;
+ p = q + 1;
+ break;
+ }
+ } while (*p != 0);
+
+ /* Record the highest CPU id of the set of online CPUs. */
+ dt_maxcpuid = *(cpu - 1);
+
+ return;
+fail:
+ if (dt_cpuids)
+ free(dt_cpuids);
+
+ dt_numcpus = 0;
+ dt_maxcpuid = 0;
+ dt_cpuids = NULL;
+}
+
+void cpu_list_free(void)
+{
+ free(dt_cpuids);
+ dt_numcpus = 0;
+ dt_maxcpuid = 0;
+ dt_cpuids = NULL;
+}
diff --git a/tools/dtrace/dtrace.c b/tools/dtrace/dtrace.c
new file mode 100644
index 000000000000..36ad526c1cd4
--- /dev/null
+++ b/tools/dtrace/dtrace.c
@@ -0,0 +1,249 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#include <errno.h>
+#include <libgen.h>
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <linux/log2.h>
+
+#include "dtrace_impl.h"
+
+#define DTRACE_BUFSIZE 32 /* default buffer size (in pages) */
+
+#define DMODE_VERS 0 /* display version information (-V) */
+#define DMODE_LIST 1 /* list probes (-l) */
+#define DMODE_EXEC 2 /* compile program and start tracing */
+
+#define E_SUCCESS 0
+#define E_ERROR 1
+#define E_USAGE 2
+
+#define NUM_PAGES(sz) (((sz) + getpagesize() - 1) / getpagesize())
+
+static const char *dtrace_options = "+b:ls:V";
+
+static char *g_pname;
+static int g_mode = DMODE_EXEC;
+
+static int usage(void)
+{
+ fprintf(stderr, "Usage: %s [-lV] [-b bufsz] -s script\n", g_pname);
+ fprintf(stderr,
+ "\t-b set trace buffer size\n"
+ "\t-l list probes matching specified criteria\n"
+ "\t-s enable or list probes for the specified BPF program\n"
+ "\t-V report DTrace API version\n");
+
+ return E_USAGE;
+}
+
+static u64 parse_size(const char *arg)
+{
+ long long mul = 1;
+ long long neg, val;
+ size_t len;
+ char *end;
+
+ if (!arg)
+ return -1;
+
+ len = strlen(arg);
+ if (!len)
+ return -1;
+
+ switch (arg[len - 1]) {
+ case 't':
+ case 'T':
+ mul *= 1024;
+ /* fall-through */
+ case 'g':
+ case 'G':
+ mul *= 1024;
+ /* fall-through */
+ case 'm':
+ case 'M':
+ mul *= 1024;
+ /* fall-through */
+ case 'k':
+ case 'K':
+ mul *= 1024;
+ /* fall-through */
+ default:
+ break;
+ }
+
+ neg = strtoll(arg, NULL, 0);
+ errno = 0;
+ val = strtoull(arg, &end, 0) * mul;
+
+ if ((mul > 1 && end != &arg[len - 1]) || (mul == 1 && *end != '\0') ||
+ val < 0 || neg < 0 || errno != 0)
+ return -1;
+
+ return val;
+}
+
+int main(int argc, char *argv[])
+{
+ int i;
+ int modec = 0;
+ int bufsize = DTRACE_BUFSIZE;
+ int epoll_fd;
+ int cnt;
+ char **prgv;
+ int prgc;
+
+ g_pname = basename(argv[0]);
+
+ if (argc == 1)
+ return usage();
+
+ prgc = 0;
+ prgv = calloc(argc, sizeof(char *));
+ if (!prgv) {
+ fprintf(stderr, "failed to allocate memory for arguments: %s\n",
+ strerror(errno));
+ return E_ERROR;
+ }
+
+ argv[0] = g_pname; /* argv[0] for getopt errors */
+
+ for (optind = 1; optind < argc; optind++) {
+ int opt;
+
+ while ((opt = getopt(argc, argv, dtrace_options)) != EOF) {
+ u64 val;
+
+ switch (opt) {
+ case 'b':
+ val = parse_size(optarg);
+ if (val < 0) {
+ fprintf(stderr, "invalid: -b %s\n",
+ optarg);
+ return E_ERROR;
+ }
+
+ /*
+ * Bufsize needs to be a number of pages, and
+ * must be a power of 2. This is required by
+ * the perf event buffer code.
+ */
+ bufsize = roundup_pow_of_two(NUM_PAGES(val));
+ if ((u64)bufsize * getpagesize() > val)
+ fprintf(stderr,
+ "bufsize increased to %ld\n",
+ (u64)bufsize * getpagesize());
+
+ break;
+ case 'l':
+ g_mode = DMODE_LIST;
+ modec++;
+ break;
+ case 's':
+ prgv[prgc++] = optarg;
+ break;
+ case 'V':
+ g_mode = DMODE_VERS;
+ modec++;
+ break;
+ default:
+ if (strchr(dtrace_options, opt) == NULL)
+ return usage();
+ }
+ }
+
+ if (optind < argc) {
+ fprintf(stderr, "unknown option '%s'\n", argv[optind]);
+ return E_ERROR;
+ }
+ }
+
+ if (modec > 1) {
+ fprintf(stderr,
+ "only one of [-lV] can be specified at a time\n");
+ return E_USAGE;
+ }
+
+ /*
+ * We handle requests for version information first because we do not
+ * need probe information for it.
+ */
+ if (g_mode == DMODE_VERS) {
+ printf("%s\n"
+ "This is DTrace %s\n"
+ "dtrace(1) version-control ID: %s\n",
+ DT_VERS_STRING, DT_VERSION, DT_GIT_VERSION);
+
+ return E_SUCCESS;
+ }
+
+ /* Initialize probes. */
+ if (dt_probe_init() < 0) {
+ fprintf(stderr, "failed to initialize probes: %s\n",
+ strerror(errno));
+ return E_ERROR;
+ }
+
+ /*
+ * We handle requests to list probes next.
+ */
+ if (g_mode == DMODE_LIST) {
+ int rc = 0;
+
+ printf("%5s %10s %17s %33s %s\n",
+ "ID", "PROVIDER", "MODULE", "FUNCTION", "NAME");
+ for (i = 0; i < prgc; i++) {
+ rc = dt_bpf_list_probes(prgv[i]);
+ if (rc < 0)
+ fprintf(stderr, "failed to load %s: %s\n",
+ prgv[i], strerror(errno));
+ }
+
+ return rc ? E_ERROR : E_SUCCESS;
+ }
+
+ if (!prgc) {
+ fprintf(stderr, "missing BPF program(s)\n");
+ return E_ERROR;
+ }
+
+ /* Process the BPF program. */
+ for (i = 0; i < prgc; i++) {
+ int err;
+
+ err = dt_bpf_load_file(prgv[i]);
+ if (err) {
+ errno = -err;
+ fprintf(stderr, "failed to load %s: %s\n",
+ prgv[i], strerror(errno));
+ return E_ERROR;
+ }
+ }
+
+ /* Get the list of online CPUs. */
+ cpu_list_populate();
+
+ /* Initialize buffers. */
+ epoll_fd = dt_buffer_init(bufsize);
+ if (epoll_fd < 0) {
+ errno = -epoll_fd;
+ fprintf(stderr, "failed to allocate buffers: %s\n",
+ strerror(errno));
+ return E_ERROR;
+ }
+
+ /* Process probe data. */
+ printf("%3s %6s\n", "CPU", "ID");
+ do {
+ cnt = dt_buffer_poll(epoll_fd, 100);
+ } while (cnt >= 0);
+
+ dt_buffer_exit(epoll_fd);
+
+ return E_SUCCESS;
+}
diff --git a/tools/dtrace/dtrace.h b/tools/dtrace/dtrace.h
new file mode 100644
index 000000000000..c79398432d17
--- /dev/null
+++ b/tools/dtrace/dtrace.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#ifndef _UAPI_LINUX_DTRACE_H
+#define _UAPI_LINUX_DTRACE_H
+
+struct dt_bpf_context {
+ u32 probe_id;
+ u64 argv[10];
+};
+
+#endif /* _UAPI_LINUX_DTRACE_H */
diff --git a/tools/dtrace/dtrace_impl.h b/tools/dtrace/dtrace_impl.h
new file mode 100644
index 000000000000..9aa51b4c4aee
--- /dev/null
+++ b/tools/dtrace/dtrace_impl.h
@@ -0,0 +1,101 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ */
+#ifndef _DTRACE_H
+#define _DTRACE_H
+
+#include <unistd.h>
+#include <bpf/libbpf.h>
+#include <linux/types.h>
+#include <linux/ptrace.h>
+#include <linux/perf_event.h>
+#include <sys/syscall.h>
+
+#include "dtrace.h"
+
+#define DT_DEBUG
+
+#define DT_VERS_STRING "Oracle D 2.0.0"
+
+#define TRACEFS "/sys/kernel/debug/tracing/"
+#define EVENTSFS TRACEFS "events/"
+
+extern int dt_numcpus;
+extern int dt_maxcpuid;
+extern int *dt_cpuids;
+
+extern void cpu_list_populate(void);
+extern void cpu_list_free(void);
+
+struct dt_provider {
+ char *name;
+ int (*populate)(void);
+ struct dt_probe *(*resolve_event)(const char *name);
+ int (*attach)(const char *name, int bpf_fd);
+};
+
+extern struct dt_provider dt_fbt;
+extern struct dt_provider dt_syscall;
+
+struct dt_hentry {
+ struct dt_probe *next;
+ struct dt_probe *prev;
+};
+
+struct dt_htab;
+
+typedef u32 (*dt_hval_fn)(const struct dt_probe *);
+typedef int (*dt_cmp_fn)(const struct dt_probe *, const struct dt_probe *);
+typedef struct dt_probe *(*dt_add_fn)(struct dt_probe *, struct dt_probe *);
+typedef struct dt_probe *(*dt_del_fn)(struct dt_probe *, struct dt_probe *);
+
+extern struct dt_htab *dt_htab_new(dt_hval_fn hval, dt_cmp_fn cmp,
+ dt_add_fn add, dt_del_fn del);
+extern int dt_htab_add(struct dt_htab *htab, struct dt_probe *probe);
+extern struct dt_probe *dt_htab_lookup(const struct dt_htab *htab,
+ const struct dt_probe *probe);
+extern int dt_htab_del(struct dt_htab *htab, struct dt_probe *probe);
+
+struct dt_probe {
+ u32 id;
+ int event_fd;
+ const struct dt_provider *prov;
+ const char *prv_name; /* provider name */
+ const char *mod_name; /* module name */
+ const char *fun_name; /* function name */
+ const char *prb_name; /* probe name */
+ struct dt_hentry he_fqn;
+};
+
+typedef void (*dt_probe_fn)(const struct dt_probe *probe);
+
+extern int dt_probe_init(void);
+extern int dt_probe_new(const struct dt_provider *prov, const char *pname,
+ const char *mname, const char *fname, const char *name);
+extern struct dt_probe *dt_probe_by_name(const struct dt_probe *tmpl);
+extern struct dt_probe *dt_probe_resolve_event(const char *name);
+
+extern int dt_bpf_list_probes(const char *fn);
+extern int dt_bpf_load_file(const char *fn);
+extern int dt_bpf_map_update(int fd, const void *key, const void *val);
+extern int dt_bpf_attach(int event_id, int bpf_fd);
+
+extern int dt_bufmap_fd;
+
+extern int dt_buffer_init(int num_pages);
+extern int dt_buffer_poll(int epoll_fd, int timeout);
+extern void dt_buffer_exit(int epoll_fd);
+
+static inline int perf_event_open(struct perf_event_attr *attr, pid_t pid,
+ int cpu, int group_fd, unsigned long flags)
+{
+ return syscall(__NR_perf_event_open, attr, pid, cpu, group_fd, flags);
+}
+
+extern inline int bpf(enum bpf_cmd cmd, union bpf_attr *attr)
+{
+ return syscall(__NR_bpf, cmd, attr, sizeof(union bpf_attr));
+}
+
+#endif /* _DTRACE_H */
--
2.20.1
^ permalink raw reply related
* [PATCH v2 net-next] net: socionext: remove set but not used variable 'pkts'
From: YueHaibing @ 2019-07-04 3:21 UTC (permalink / raw)
To: Jassi Brar, Alexei Starovoitov, David S . Miller,
Ilias Apalodimas, Daniel Borkmann, Jakub Kicinski,
Jesper Dangaard Brouer
Cc: YueHaibing, netdev
In-Reply-To: <20190703024213.191191-1-yuehaibing@huawei.com>
Cc: netdev@vger.kernel.org,
xdp-newbies@vger.kernel.org,
bpf@vger.kernel.org,
kernel-janitors@vger.kernel.org
Fixes gcc '-Wunused-but-set-variable' warning:
drivers/net/ethernet/socionext/netsec.c: In function 'netsec_clean_tx_dring':
drivers/net/ethernet/socionext/netsec.c:637:15: warning:
variable 'pkts' set but not used [-Wunused-but-set-variable]
It is not used since commit ba2b232108d3 ("net: netsec: add XDP support")
Signed-off-by: YueHaibing <yuehaibing@huawei.com>
Acked-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
---
v2: keep reverse christmas-tree ordering of the local variables
---
drivers/net/ethernet/socionext/netsec.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/socionext/netsec.c b/drivers/net/ethernet/socionext/netsec.c
index 5544a722543f..d8d640b01119 100644
--- a/drivers/net/ethernet/socionext/netsec.c
+++ b/drivers/net/ethernet/socionext/netsec.c
@@ -634,15 +634,14 @@ static void netsec_set_rx_de(struct netsec_priv *priv,
static bool netsec_clean_tx_dring(struct netsec_priv *priv)
{
struct netsec_desc_ring *dring = &priv->desc_ring[NETSEC_RING_TX];
- unsigned int pkts, bytes;
struct netsec_de *entry;
int tail = dring->tail;
+ unsigned int bytes;
int cnt = 0;
if (dring->is_xdp)
spin_lock(&dring->lock);
- pkts = 0;
bytes = 0;
entry = dring->vaddr + DESC_SZ * tail;
^ permalink raw reply related
* [PATCH 0/1] tools/dtrace: initial implementation of DTrace
From: Kris Van Hees @ 2019-07-04 3:13 UTC (permalink / raw)
To: netdev, bpf, dtrace-devel, linux-kernel
Cc: rostedt, mhiramat, acme, ast, daniel, Peter Zijlstra, Chris Mason
This patch is also available, applied to bpf-next, at the following URL:
https://github.com/oracle/dtrace-linux-kernel/tree/dtrace-bpf
As suggested in feedback to my earlier patch submissions, this code takes an
approach to avoid kernel code changes as much as possible. The current patch
does not involve any kernel code changes. Further development of this code
will continue with this approach, incrementally adding features to this first
minimal implementation. The goal is a fully featured and functional DTrace
implementation involving kernel changes only when strictly necessary.
The code presented here supports two very basic functions:
1. Listing probes that are used in BPF programs
# dtrace -l -s bpf_sample.o
ID PROVIDER MODULE FUNCTION NAME
18876 fbt vmlinux ksys_write entry
70423 syscall vmlinux write entry
2. Loading BPF tracing programs and collecting data that they generate
# dtrace -s bpf_sample.o
CPU ID
15 70423 0xffff8c0968bf8ec0 0x00000000000001 0x0055e019eb3f60 0x0000000000002c
15 18876 0xffff8c0968bf8ec0 0x00000000000001 0x0055e019eb3f60 0x0000000000002c
...
Only kprobes and syscall tracepoints are supported since this is an initial
patch. It does show the use of a generic BPF function to implement the actual
probe action, called from two distinct probe types. Follow-up patches will
add more probe types, add more tracing features from the D language, add
support for D script compilation to BPF, etc.
The implementation makes use of libbpf for handling BPF ELF objects, and uses
the perf event output ring buffer (supported through BPF) to retrieve the
tracing data. The next step in development will be adding support to libbpf
for programs using shared functions from a collection of functions included in
the BPF ELF object (as suggested by Alexei).
The code is structured as follows:
tools/dtrace/dtrace.c = command line utility
tools/dtrace/dt_bpf.c = interface to libbpf
tools/dtrace/dt_buffer.c = perf event output buffer handling
tools/dtrace/dt_fbt.c = kprobes probe provider
tools/dtrace/dt_syscall.c = syscall tracepoint probe provider
tools/dtrace/dt_probe.c = generic probe and probe provider handling code
This implements a generic interface to the actual
probe providers (dt_fbt and dt_syscall).
tools/dtrace/dt_hash.c = general probe hashing implementation
tools/dtrace/dt_utils.c = support code (manage list of online CPUs)
tools/dtrace/dtrace.h = API header file (used by BPF program source code)
tools/dtrace/dtrace_impl.h = implementation header file
tools/dtrace/bpf_sample.c = sample BPF program using two probe types
I included an entry for the MAINTAINERS file. I offer to actively maintain
this code, and to keep advancing its development.
Cheers,
Kris Van Hees
^ permalink raw reply
* [PATCH] net: ethernet: allwinner: Remove unneeded memset
From: Hariprasad Kelam @ 2019-07-04 2:59 UTC (permalink / raw)
To: David S. Miller, Maxime Ripard, Chen-Yu Tsai, Petr Štetiar,
Hariprasad Kelam, Florian Fainelli, Andrew Lunn, netdev,
linux-arm-kernel, linux-kernel
Remove unneeded memset as alloc_etherdev is using kvzalloc which uses
__GFP_ZERO flag
Signed-off-by: Hariprasad Kelam <hariprasad.kelam@gmail.com>
---
drivers/net/ethernet/allwinner/sun4i-emac.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/drivers/net/ethernet/allwinner/sun4i-emac.c b/drivers/net/ethernet/allwinner/sun4i-emac.c
index 9e06dff..6253e5e 100644
--- a/drivers/net/ethernet/allwinner/sun4i-emac.c
+++ b/drivers/net/ethernet/allwinner/sun4i-emac.c
@@ -818,7 +818,6 @@ static int emac_probe(struct platform_device *pdev)
SET_NETDEV_DEV(ndev, &pdev->dev);
db = netdev_priv(ndev);
- memset(db, 0, sizeof(*db));
db->dev = &pdev->dev;
db->ndev = ndev;
--
2.7.4
^ permalink raw reply related
* [Patch v2 09/10] net/ethernet: using dev_get_drvdata directly
From: Fuqian Huang @ 2019-07-04 2:36 UTC (permalink / raw)
Cc: David S . Miller, netdev, linux-kernel, Fuqian Huang
Several drivers cast a struct device pointer to a struct
platform_device pointer only to then call platform_get_drvdata().
To improve readability, these constructs can be simplified
by using dev_get_drvdata() directly.
Signed-off-by: Fuqian Huang <huangfq.daxian@gmail.com>
---
Changes in v2:
- Make the commit message more clearly.
drivers/net/ethernet/calxeda/xgmac.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/calxeda/xgmac.c b/drivers/net/ethernet/calxeda/xgmac.c
index 11d4e91ea754..99f49d059414 100644
--- a/drivers/net/ethernet/calxeda/xgmac.c
+++ b/drivers/net/ethernet/calxeda/xgmac.c
@@ -1855,7 +1855,7 @@ static void xgmac_pmt(void __iomem *ioaddr, unsigned long mode)
static int xgmac_suspend(struct device *dev)
{
- struct net_device *ndev = platform_get_drvdata(to_platform_device(dev));
+ struct net_device *ndev = dev_get_drvdata(dev);
struct xgmac_priv *priv = netdev_priv(ndev);
u32 value;
@@ -1881,7 +1881,7 @@ static int xgmac_suspend(struct device *dev)
static int xgmac_resume(struct device *dev)
{
- struct net_device *ndev = platform_get_drvdata(to_platform_device(dev));
+ struct net_device *ndev = dev_get_drvdata(dev);
struct xgmac_priv *priv = netdev_priv(ndev);
void __iomem *ioaddr = priv->base;
--
2.11.0
^ permalink raw reply related
* [Patch v2 08/10] net/can: using dev_get_drvdata directly
From: Fuqian Huang @ 2019-07-04 2:36 UTC (permalink / raw)
Cc: Wolfgang Grandegger, Marc Kleine-Budde, David S . Miller,
linux-can, netdev, linux-kernel, Fuqian Huang
Several drivers cast a struct device pointer to a struct
platform_device pointer only to then call platform_get_drvdata().
To improve readability, these constructs can be simplified
by using dev_get_drvdata() directly.
Signed-off-by: Fuqian Huang <huangfq.daxian@gmail.com>
---
Changes in v2:
- Make the commit message more clearly.
drivers/net/can/softing/softing_main.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/can/softing/softing_main.c b/drivers/net/can/softing/softing_main.c
index 68bb58a57f3b..8242fb287cbb 100644
--- a/drivers/net/can/softing/softing_main.c
+++ b/drivers/net/can/softing/softing_main.c
@@ -683,7 +683,7 @@ static void softing_netdev_cleanup(struct net_device *netdev)
static ssize_t show_##name(struct device *dev, \
struct device_attribute *attr, char *buf) \
{ \
- struct softing *card = platform_get_drvdata(to_platform_device(dev)); \
+ struct softing *card = dev_get_drvdata(dev); \
return sprintf(buf, "%u\n", card->member); \
} \
static DEVICE_ATTR(name, 0444, show_##name, NULL)
@@ -692,7 +692,7 @@ static DEVICE_ATTR(name, 0444, show_##name, NULL)
static ssize_t show_##name(struct device *dev, \
struct device_attribute *attr, char *buf) \
{ \
- struct softing *card = platform_get_drvdata(to_platform_device(dev)); \
+ struct softing *card = dev_get_drvdata(dev); \
return sprintf(buf, "%s\n", card->member); \
} \
static DEVICE_ATTR(name, 0444, show_##name, NULL)
--
2.11.0
^ permalink raw reply related
* [rdma 06/16] RDMA/irdma: Add QoS definitions
From: Jeff Kirsher @ 2019-07-04 2:12 UTC (permalink / raw)
To: dledford, jgg, davem
Cc: Mustafa Ismail, linux-rdma, netdev, nhorman, sassmann, poswald,
david.m.ertman, Shiraz Saleem
In-Reply-To: <20190704021259.15489-1-jeffrey.t.kirsher@intel.com>
From: Mustafa Ismail <mustafa.ismail@intel.com>
Add definitions for managing the RDMA HW work scheduler (WS) tree.
A WS node is created via a control QP operation with the bandwidth
allocation, arbitration scheme, and traffic class of the QP specified.
The Qset handle returned associates the QoS parameters for the QP.
The Qset is registered with the LAN and a equivalent node is created
in the LAN packet scheduler tree.
Signed-off-by: Mustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
---
drivers/infiniband/hw/irdma/ws.c | 396 +++++++++++++++++++++++++++++++
drivers/infiniband/hw/irdma/ws.h | 40 ++++
2 files changed, 436 insertions(+)
create mode 100644 drivers/infiniband/hw/irdma/ws.c
create mode 100644 drivers/infiniband/hw/irdma/ws.h
diff --git a/drivers/infiniband/hw/irdma/ws.c b/drivers/infiniband/hw/irdma/ws.c
new file mode 100644
index 000000000000..73e1a66542ee
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/ws.c
@@ -0,0 +1,396 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "osdep.h"
+#include "status.h"
+#include "hmc.h"
+#include "defs.h"
+#include "type.h"
+#include "protos.h"
+
+#include "ws.h"
+
+/**
+ * irdma_alloc_node - Allocate a WS node and init
+ * @vsi: vsi pointer
+ * @user_pri: user priority
+ * @node_type: Type of node, leaf or parent
+ * @parent: parent node pointer
+ */
+static struct irdma_ws_node *irdma_alloc_node(struct irdma_sc_vsi *vsi,
+ u8 user_pri,
+ enum irdma_ws_node_type node_type,
+ struct irdma_ws_node *parent)
+{
+ struct irdma_virt_mem ws_mem;
+ struct irdma_ws_node *node;
+ u16 node_index = 0;
+
+ ws_mem.size = sizeof(struct irdma_ws_node);
+ ws_mem.va = kzalloc(ws_mem.size, GFP_ATOMIC);
+ if (!ws_mem.va)
+ return NULL;
+
+ if (parent || vsi->vm_vf_type == IRDMA_VF_TYPE) {
+ node_index = irdma_alloc_ws_node_id(vsi->dev);
+ if (node_index == IRDMA_WS_NODE_INVALID) {
+ kfree(ws_mem.va);
+ return NULL;
+ }
+ }
+
+ node = ws_mem.va;
+ node->index = node_index;
+ node->vsi_index = vsi->vsi_idx;
+ INIT_LIST_HEAD(&node->child_list_head);
+ if (node_type == WS_NODE_TYPE_LEAF) {
+ node->type_leaf = true;
+ node->traffic_class = vsi->qos[user_pri].traffic_class;
+ node->user_pri = user_pri;
+ node->rel_bw = vsi->qos[user_pri].rel_bw;
+ if (!node->rel_bw)
+ node->rel_bw = 1;
+
+ node->lan_qs_handle = vsi->qos[user_pri].lan_qos_handle;
+ node->prio_type = IRDMA_PRIO_WEIGHTED_RR;
+ } else {
+ node->rel_bw = 1;
+ node->prio_type = IRDMA_PRIO_WEIGHTED_RR;
+ }
+
+ node->parent = parent;
+
+ return node;
+}
+
+/**
+ * irdma_free_node - Free a WS node
+ * @vsi: VSI stricture of device
+ * @node: Pointer to node to free
+ */
+static void irdma_free_node(struct irdma_sc_vsi *vsi,
+ struct irdma_ws_node *node)
+{
+ struct irdma_virt_mem ws_mem;
+
+ if (node->index)
+ irdma_free_ws_node_id(vsi->dev, node->index);
+
+ ws_mem.va = node;
+ ws_mem.size = sizeof(struct irdma_ws_node);
+ kfree(ws_mem.va);
+}
+
+/**
+ * irdma_ws_cqp_cmd - Post CQP work scheduler node cmd
+ * @vsi: vsi pointer
+ * @node: pointer to node
+ * @cmd: add, remove or modify
+ */
+static enum irdma_status_code
+irdma_ws_cqp_cmd(struct irdma_sc_vsi *vsi, struct irdma_ws_node *node, u8 cmd)
+{
+ struct irdma_ws_node_info node_info = {};
+
+ node_info.id = node->index;
+ node_info.vsi = node->vsi_index;
+ if (node->parent)
+ node_info.parent_id = node->parent->index;
+ else
+ node_info.parent_id = node_info.id;
+
+ node_info.weight = node->rel_bw;
+ node_info.tc = node->traffic_class;
+ node_info.prio_type = node->prio_type;
+ node_info.type_leaf = node->type_leaf;
+ node_info.enable = node->enable;
+ if (irdma_cqp_ws_node_cmd(vsi->dev, cmd, &node_info)) {
+ dev_dbg(rfdev_to_dev(vsi->dev), "WS: CQP WS CMD failed\n");
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ if (node->type_leaf && cmd == IRDMA_OP_WS_ADD_NODE) {
+ node->qs_handle = node_info.qs_handle;
+ vsi->qos[node->user_pri].qs_handle = node_info.qs_handle;
+ }
+
+ return 0;
+}
+
+/**
+ * ws_find_node - Find SC WS node based on VSI id or TC
+ * @parent: parent node of First VSI or TC node
+ * @match_val: value to match
+ * @type: match type VSI/TC
+ */
+static struct irdma_ws_node *ws_find_node(struct irdma_ws_node *parent,
+ u16 match_val,
+ enum irdma_ws_match_type type)
+{
+ struct irdma_ws_node *node;
+
+ switch (type) {
+ case WS_MATCH_TYPE_VSI:
+ list_for_each_entry(node, &parent->child_list_head, siblings) {
+ if (node->vsi_index == match_val)
+ return node;
+ }
+ break;
+ case WS_MATCH_TYPE_TC:
+ list_for_each_entry(node, &parent->child_list_head, siblings) {
+ if (node->traffic_class == match_val)
+ return node;
+ }
+ break;
+ default:
+ break;
+ }
+
+ return NULL;
+}
+
+/**
+ * irdma_tc_in_use - Checks to see if a leaf node is in use
+ * @vsi: vsi pointer
+ * @user_pri: user priority
+ */
+static bool irdma_tc_in_use(struct irdma_sc_vsi *vsi, u8 user_pri)
+{
+ unsigned long flags;
+ int i;
+
+ spin_lock_irqsave(&vsi->qos[user_pri].lock, flags);
+ if (!list_empty(&vsi->qos[user_pri].qplist)) {
+ spin_unlock_irqrestore(&vsi->qos[user_pri].lock, flags);
+ return true;
+ }
+
+ /* Check if the traffic class associated with the given user priority
+ * is in use by any other user priority. If so, nothing left to do
+ */
+ for (i = 0; i < IRDMA_MAX_USER_PRIORITY; i++) {
+ if (vsi->qos[i].traffic_class == vsi->qos[user_pri].traffic_class &&
+ !list_empty(&vsi->qos[i].qplist)) {
+ spin_unlock_irqrestore(&vsi->qos[user_pri].lock, flags);
+ return true;
+ }
+ }
+ spin_unlock_irqrestore(&vsi->qos[user_pri].lock, flags);
+
+ return false;
+}
+
+/**
+ * irdma_remove_leaf - Remove leaf node unconditionally
+ * @vsi: vsi pointer
+ * @user_pri: user priority
+ */
+static void irdma_remove_leaf(struct irdma_sc_vsi *vsi, u8 user_pri)
+{
+ struct irdma_ws_node *ws_tree_root, *vsi_node, *tc_node;
+
+ ws_tree_root = vsi->dev->ws_tree_root;
+ if (!ws_tree_root)
+ return;
+
+ vsi_node = ws_find_node(ws_tree_root, vsi->vsi_idx,
+ WS_MATCH_TYPE_VSI);
+ if (!vsi_node)
+ return;
+
+ tc_node = ws_find_node(vsi_node,
+ vsi->qos[user_pri].traffic_class,
+ WS_MATCH_TYPE_TC);
+ if (!tc_node)
+ return;
+
+ irdma_ws_cqp_cmd(vsi, tc_node, IRDMA_OP_WS_DELETE_NODE);
+ irdma_lan_unregister_qset(vsi, tc_node);
+ list_del(&tc_node->siblings);
+ irdma_free_node(vsi, tc_node);
+ /* Check if VSI node can be freed */
+ if (list_empty(&vsi_node->child_list_head)) {
+ irdma_ws_cqp_cmd(vsi, vsi_node, IRDMA_OP_WS_DELETE_NODE);
+ list_del(&vsi_node->siblings);
+ irdma_free_node(vsi, vsi_node);
+ /* Free head node there are no remaining VSI nodes */
+ if (list_empty(&ws_tree_root->child_list_head)) {
+ irdma_ws_cqp_cmd(vsi, ws_tree_root,
+ IRDMA_OP_WS_DELETE_NODE);
+ irdma_free_node(vsi, ws_tree_root);
+ vsi->dev->ws_tree_root = NULL;
+ }
+ }
+}
+
+/**
+ * irdma_ws_add - Build work scheduler tree, set RDMA qs_handle
+ * @vsi: vsi pointer
+ * @user_pri: user priority
+ */
+enum irdma_status_code irdma_ws_add(struct irdma_sc_vsi *vsi, u8 user_pri)
+{
+ struct irdma_ws_node *ws_tree_root;
+ struct irdma_ws_node *vsi_node;
+ struct irdma_ws_node *tc_node;
+ u16 traffic_class;
+ enum irdma_status_code ret = 0;
+ int i;
+
+ mutex_lock(&vsi->dev->ws_mutex);
+ if (vsi->tc_change_pending) {
+ ret = IRDMA_ERR_NOT_READY;
+ goto exit;
+ }
+
+ ws_tree_root = vsi->dev->ws_tree_root;
+ if (!ws_tree_root) {
+ dev_dbg(rfdev_to_dev(vsi->dev), "WS: Creating root node\n");
+ ws_tree_root = irdma_alloc_node(vsi, user_pri,
+ WS_NODE_TYPE_PARENT, NULL);
+ if (!ws_tree_root) {
+ ret = IRDMA_ERR_NO_MEMORY;
+ goto exit;
+ }
+
+ ret = irdma_ws_cqp_cmd(vsi, ws_tree_root, IRDMA_OP_WS_ADD_NODE);
+ if (ret) {
+ irdma_free_node(vsi, ws_tree_root);
+ goto exit;
+ }
+
+ vsi->dev->ws_tree_root = ws_tree_root;
+ }
+
+ /* Find a second tier node that matches the VSI */
+ vsi_node = ws_find_node(ws_tree_root, vsi->vsi_idx,
+ WS_MATCH_TYPE_VSI);
+
+ /* If VSI node doesn't exist, add one */
+ if (!vsi_node) {
+ dev_dbg(rfdev_to_dev(vsi->dev),
+ "WS: Node not found matching VSI %d\n", vsi->vsi_idx);
+ vsi_node = irdma_alloc_node(vsi, user_pri, WS_NODE_TYPE_PARENT,
+ ws_tree_root);
+ if (!vsi_node) {
+ ret = IRDMA_ERR_NO_MEMORY;
+ goto vsi_add_err;
+ }
+
+ ret = irdma_ws_cqp_cmd(vsi, vsi_node, IRDMA_OP_WS_ADD_NODE);
+ if (ret) {
+ irdma_free_node(vsi, vsi_node);
+ goto vsi_add_err;
+ }
+
+ list_add(&vsi_node->siblings, &ws_tree_root->child_list_head);
+ }
+
+ dev_dbg(rfdev_to_dev(vsi->dev),
+ "WS: Using node %d which represents VSI %d\n",
+ vsi_node->index, vsi->vsi_idx);
+ traffic_class = vsi->qos[user_pri].traffic_class;
+ tc_node = ws_find_node(vsi_node, traffic_class,
+ WS_MATCH_TYPE_TC);
+ if (!tc_node) {
+ /* Add leaf node */
+ dev_dbg(rfdev_to_dev(vsi->dev),
+ "WS: Node not found matching VSI %d and TC %d\n",
+ vsi->vsi_idx, traffic_class);
+ tc_node = irdma_alloc_node(vsi, user_pri, WS_NODE_TYPE_LEAF,
+ vsi_node);
+ if (!tc_node) {
+ ret = IRDMA_ERR_NO_MEMORY;
+ goto leaf_add_err;
+ }
+
+ ret = irdma_ws_cqp_cmd(vsi, tc_node, IRDMA_OP_WS_ADD_NODE);
+ if (ret) {
+ irdma_free_node(vsi, tc_node);
+ goto leaf_add_err;
+ }
+
+ list_add(&tc_node->siblings, &vsi_node->child_list_head);
+ /*
+ * callback to LAN to update the LAN tree with our node
+ */
+ ret = irdma_lan_register_qset(vsi, tc_node);
+ if (ret)
+ goto reg_err;
+
+ tc_node->enable = true;
+ ret = irdma_ws_cqp_cmd(vsi, tc_node, IRDMA_OP_WS_MODIFY_NODE);
+ if (ret)
+ goto reg_err;
+ }
+ dev_dbg(rfdev_to_dev(vsi->dev),
+ "WS: Using node %d which represents VSI %d TC %d\n",
+ tc_node->index, vsi->vsi_idx, traffic_class);
+ /*
+ * Iterate through other UPs and update the QS handle if they have
+ * a matching traffic class.
+ */
+ for (i = 0; i < IRDMA_MAX_USER_PRIORITY; i++) {
+ if (vsi->qos[i].traffic_class == traffic_class) {
+ vsi->qos[i].qs_handle = tc_node->qs_handle;
+ vsi->qos[i].lan_qos_handle = tc_node->lan_qs_handle;
+ vsi->qos[i].l2_sched_node_id = tc_node->l2_sched_node_id;
+ }
+ }
+ goto exit;
+
+leaf_add_err:
+ if (list_empty(&vsi_node->child_list_head)) {
+ if (irdma_ws_cqp_cmd(vsi, vsi_node, IRDMA_OP_WS_DELETE_NODE))
+ goto exit;
+ list_del(&vsi_node->siblings);
+ irdma_free_node(vsi, vsi_node);
+ }
+
+vsi_add_err:
+ /* Free head node there are no remaining VSI nodes */
+ if (list_empty(&ws_tree_root->child_list_head)) {
+ irdma_ws_cqp_cmd(vsi, ws_tree_root, IRDMA_OP_WS_DELETE_NODE);
+ vsi->dev->ws_tree_root = NULL;
+ irdma_free_node(vsi, ws_tree_root);
+ }
+
+exit:
+ mutex_unlock(&vsi->dev->ws_mutex);
+ return ret;
+
+reg_err:
+ mutex_unlock(&vsi->dev->ws_mutex);
+ irdma_ws_remove(vsi, user_pri);
+ return ret;
+}
+
+/**
+ * irdma_ws_remove - Free WS scheduler node, update WS tree
+ * @vsi: vsi pointer
+ * @user_pri: user priority
+ */
+void irdma_ws_remove(struct irdma_sc_vsi *vsi, u8 user_pri)
+{
+ mutex_lock(&vsi->dev->ws_mutex);
+ if (irdma_tc_in_use(vsi, user_pri))
+ goto exit;
+
+ irdma_remove_leaf(vsi, user_pri);
+exit:
+ mutex_unlock(&vsi->dev->ws_mutex);
+}
+
+/**
+ * irdma_ws_reset - Reset entire WS tree
+ * @vsi: vsi pointer
+ */
+void irdma_ws_reset(struct irdma_sc_vsi *vsi)
+{
+ u8 i;
+
+ mutex_lock(&vsi->dev->ws_mutex);
+ for (i = 0; i < IRDMA_MAX_USER_PRIORITY; ++i)
+ irdma_remove_leaf(vsi, i);
+ mutex_unlock(&vsi->dev->ws_mutex);
+}
diff --git a/drivers/infiniband/hw/irdma/ws.h b/drivers/infiniband/hw/irdma/ws.h
new file mode 100644
index 000000000000..342d197ce2a0
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/ws.h
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef IRDMA_WS_H
+#define IRDMA_WS_H
+
+#include "osdep.h"
+
+enum irdma_ws_node_type {
+ WS_NODE_TYPE_PARENT,
+ WS_NODE_TYPE_LEAF,
+};
+
+enum irdma_ws_match_type {
+ WS_MATCH_TYPE_VSI,
+ WS_MATCH_TYPE_TC,
+};
+
+struct irdma_ws_node {
+ struct list_head siblings;
+ struct list_head child_list_head;
+ struct irdma_ws_node *parent;
+ u64 lan_qs_handle; /* opaque handle used by LAN */
+ u32 l2_sched_node_id;
+ u16 index;
+ u16 qs_handle;
+ u16 vsi_index;
+ u8 traffic_class;
+ u8 user_pri;
+ u8 rel_bw;
+ u8 abstraction_layer; /* used for splitting a TC */
+ u8 prio_type;
+ bool type_leaf;
+ bool enable;
+};
+
+enum irdma_status_code irdma_ws_add(struct irdma_sc_vsi *vsi, u8 user_pri);
+void irdma_ws_remove(struct irdma_sc_vsi *vsi, u8 user_pri);
+void irdma_ws_reset(struct irdma_sc_vsi *vsi);
+#endif /* IRDMA_WS_H */
--
2.21.0
^ permalink raw reply related
* [rdma 04/16] RDMA/irdma: Add HMC backing store setup functions
From: Jeff Kirsher @ 2019-07-04 2:12 UTC (permalink / raw)
To: dledford, jgg, davem
Cc: Mustafa Ismail, linux-rdma, netdev, nhorman, sassmann, poswald,
david.m.ertman, Shiraz Saleem
In-Reply-To: <20190704021259.15489-1-jeffrey.t.kirsher@intel.com>
From: Mustafa Ismail <mustafa.ismail@intel.com>
HW uses host memory as a backing store for a number of
protocol context objects and queue state tracking.
The Host Memory Cache (HMC) is a component responsible for
managing these objects stored in host memory.
Add the functions and data structures to manage the allocation
of backing pages used by the HMC for the various objects
Signed-off-by: Mustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
---
drivers/infiniband/hw/irdma/hmc.c | 706 ++++++++++++++++++++++++++++++
drivers/infiniband/hw/irdma/hmc.h | 219 +++++++++
2 files changed, 925 insertions(+)
create mode 100644 drivers/infiniband/hw/irdma/hmc.c
create mode 100644 drivers/infiniband/hw/irdma/hmc.h
diff --git a/drivers/infiniband/hw/irdma/hmc.c b/drivers/infiniband/hw/irdma/hmc.c
new file mode 100644
index 000000000000..eeaf2fb84638
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/hmc.c
@@ -0,0 +1,706 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "osdep.h"
+#include "status.h"
+#include "hmc.h"
+#include "defs.h"
+#include "type.h"
+#include "protos.h"
+
+/**
+ * irdma_find_sd_index_limit - finds segment descriptor index limit
+ * @hmc_info: pointer to the HMC configuration information structure
+ * @type: type of HMC resources we're searching
+ * @idx: starting index for the object
+ * @cnt: number of objects we're trying to create
+ * @sd_idx: pointer to return index of the segment descriptor in question
+ * @sd_limit: pointer to return the maximum number of segment descriptors
+ *
+ * This function calculates the segment descriptor index and index limit
+ * for the resource defined by irdma_hmc_rsrc_type.
+ */
+
+static void irdma_find_sd_index_limit(struct irdma_hmc_info *hmc_info, u32 type,
+ u32 idx, u32 cnt, u32 *sd_idx,
+ u32 *sd_limit)
+{
+ u64 fpm_addr, fpm_limit;
+
+ fpm_addr = hmc_info->hmc_obj[(type)].base +
+ hmc_info->hmc_obj[type].size * idx;
+ fpm_limit = fpm_addr + hmc_info->hmc_obj[type].size * cnt;
+ *sd_idx = (u32)(fpm_addr / IRDMA_HMC_DIRECT_BP_SIZE);
+ *sd_limit = (u32)((fpm_limit - 1) / IRDMA_HMC_DIRECT_BP_SIZE);
+ *sd_limit += 1;
+}
+
+/**
+ * irdma_find_pd_index_limit - finds page descriptor index limit
+ * @hmc_info: pointer to the HMC configuration information struct
+ * @type: HMC resource type we're examining
+ * @idx: starting index for the object
+ * @cnt: number of objects we're trying to create
+ * @pd_idx: pointer to return page descriptor index
+ * @pd_limit: pointer to return page descriptor index limit
+ *
+ * Calculates the page descriptor index and index limit for the resource
+ * defined by irdma_hmc_rsrc_type.
+ */
+
+static void irdma_find_pd_index_limit(struct irdma_hmc_info *hmc_info, u32 type,
+ u32 idx, u32 cnt, u32 *pd_idx,
+ u32 *pd_limit)
+{
+ u64 fpm_adr, fpm_limit;
+
+ fpm_adr = hmc_info->hmc_obj[type].base +
+ hmc_info->hmc_obj[type].size * idx;
+ fpm_limit = fpm_adr + (hmc_info)->hmc_obj[(type)].size * (cnt);
+ *pd_idx = (u32)(fpm_adr / IRDMA_HMC_PAGED_BP_SIZE);
+ *pd_limit = (u32)((fpm_limit - 1) / IRDMA_HMC_PAGED_BP_SIZE);
+ *pd_limit += 1;
+}
+
+/**
+ * irdma_set_sd_entry - setup entry for sd programming
+ * @pa: physical addr
+ * @idx: sd index
+ * @type: paged or direct sd
+ * @entry: sd entry ptr
+ */
+static void irdma_set_sd_entry(u64 pa, u32 idx, enum irdma_sd_entry_type type,
+ struct irdma_update_sd_entry *entry)
+{
+ entry->data = pa | (IRDMA_HMC_MAX_BP_COUNT << IRDMA_PFHMC_SDDATALOW_PMSDBPCOUNT_S) |
+ (((type == IRDMA_SD_TYPE_PAGED) ? 0 : 1) << IRDMA_PFHMC_SDDATALOW_PMSDTYPE_S) |
+ (1 << IRDMA_PFHMC_SDDATALOW_PMSDVALID_S);
+ entry->cmd = (idx | (1 << IRDMA_PFHMC_SDCMD_PMSDWR_S) | (1 << 15));
+}
+
+/**
+ * irdma_clr_sd_entry - setup entry for sd clear
+ * @idx: sd index
+ * @type: paged or direct sd
+ * @entry: sd entry ptr
+ */
+static void irdma_clr_sd_entry(u32 idx, enum irdma_sd_entry_type type,
+ struct irdma_update_sd_entry *entry)
+{
+ entry->data = (IRDMA_HMC_MAX_BP_COUNT << IRDMA_PFHMC_SDDATALOW_PMSDBPCOUNT_S) |
+ (((type == IRDMA_SD_TYPE_PAGED) ? 0 : 1) << IRDMA_PFHMC_SDDATALOW_PMSDTYPE_S);
+ entry->cmd = (idx | (1 << IRDMA_PFHMC_SDCMD_PMSDWR_S) | (1 << 15));
+}
+
+/**
+ * irdma_hmc_sd_one - setup 1 sd entry for cqp
+ * @dev: pointer to the device structure
+ * @hmc_fn_id: hmc's function id
+ * @pa: physical addr
+ * @sd_idx: sd index
+ * @type: paged or direct sd
+ * @setsd: flag to set or clear sd
+ */
+enum irdma_status_code irdma_hmc_sd_one(struct irdma_sc_dev *dev, u8 hmc_fn_id,
+ u64 pa, u32 sd_idx,
+ enum irdma_sd_entry_type type,
+ bool setsd)
+{
+ struct irdma_update_sds_info sdinfo;
+
+ sdinfo.cnt = 1;
+ sdinfo.hmc_fn_id = hmc_fn_id;
+ if (setsd)
+ irdma_set_sd_entry(pa, sd_idx, type, sdinfo.entry);
+ else
+ irdma_clr_sd_entry(sd_idx, type, sdinfo.entry);
+ return dev->cqp->process_cqp_sds(dev, &sdinfo);
+}
+
+/**
+ * irdma_hmc_sd_grp - setup group of sd entries for cqp
+ * @dev: pointer to the device structure
+ * @hmc_info: pointer to the HMC configuration information struct
+ * @sd_index: sd index
+ * @sd_cnt: number of sd entries
+ * @setsd: flag to set or clear sd
+ */
+static enum irdma_status_code irdma_hmc_sd_grp(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info,
+ u32 sd_index, u32 sd_cnt,
+ bool setsd)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+ struct irdma_update_sds_info sdinfo = {};
+ u64 pa;
+ u32 i;
+ enum irdma_status_code ret_code = 0;
+
+ sdinfo.hmc_fn_id = hmc_info->hmc_fn_id;
+ for (i = sd_index; i < sd_index + sd_cnt; i++) {
+ sd_entry = &hmc_info->sd_table.sd_entry[i];
+ if (!sd_entry || (!sd_entry->valid && setsd) ||
+ (sd_entry->valid && !setsd))
+ continue;
+ if (setsd) {
+ pa = (sd_entry->entry_type == IRDMA_SD_TYPE_PAGED) ?
+ sd_entry->u.pd_table.pd_page_addr.pa :
+ sd_entry->u.bp.addr.pa;
+ irdma_set_sd_entry(pa, i, sd_entry->entry_type,
+ &sdinfo.entry[sdinfo.cnt]);
+ } else {
+ irdma_clr_sd_entry(i, sd_entry->entry_type,
+ &sdinfo.entry[sdinfo.cnt]);
+ }
+ sdinfo.cnt++;
+ if (sdinfo.cnt == IRDMA_MAX_SD_ENTRIES) {
+ ret_code = dev->cqp->process_cqp_sds(dev, &sdinfo);
+ if (ret_code) {
+ dev_dbg(rfdev_to_dev(dev),
+ "HMC: sd_programming failed err=%d\n",
+ ret_code);
+ return ret_code;
+ }
+
+ sdinfo.cnt = 0;
+ }
+ }
+ if (sdinfo.cnt)
+ ret_code = dev->cqp->process_cqp_sds(dev, &sdinfo);
+
+ return ret_code;
+}
+
+/**
+ * irdma_hmc_finish_add_sd_reg - program sd entries for objects
+ * @dev: pointer to the device structure
+ * @info: create obj info
+ */
+static enum irdma_status_code
+irdma_hmc_finish_add_sd_reg(struct irdma_sc_dev *dev,
+ struct irdma_hmc_create_obj_info *info)
+{
+ if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt)
+ return IRDMA_ERR_INVALID_HMC_OBJ_INDEX;
+
+ if ((info->start_idx + info->count) >
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt)
+ return IRDMA_ERR_INVALID_HMC_OBJ_COUNT;
+
+ if (!info->add_sd_cnt)
+ return 0;
+ return irdma_hmc_sd_grp(dev, info->hmc_info,
+ info->hmc_info->sd_indexes[0], info->add_sd_cnt,
+ true);
+}
+
+/**
+ * irdma_sc_create_hmc_obj - allocate backing store for hmc objects
+ * @dev: pointer to the device structure
+ * @info: pointer to irdma_hmc_create_obj_info struct
+ *
+ * This will allocate memory for PDs and backing pages and populate
+ * the sd and pd entries.
+ */
+enum irdma_status_code
+irdma_sc_create_hmc_obj(struct irdma_sc_dev *dev,
+ struct irdma_hmc_create_obj_info *info)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+ u32 sd_idx, sd_lmt;
+ u32 pd_idx = 0, pd_lmt = 0;
+ u32 pd_idx1 = 0, pd_lmt1 = 0;
+ u32 i, j;
+ bool pd_error = false;
+ enum irdma_status_code ret_code = 0;
+
+ if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt)
+ return IRDMA_ERR_INVALID_HMC_OBJ_INDEX;
+
+ if ((info->start_idx + info->count) >
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt) {
+ dev_dbg(rfdev_to_dev(dev),
+ "HMC: error type %u, start = %u, req cnt %u, cnt = %u\n",
+ info->rsrc_type, info->start_idx, info->count,
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt);
+ return IRDMA_ERR_INVALID_HMC_OBJ_COUNT;
+ }
+
+ irdma_find_sd_index_limit(info->hmc_info, info->rsrc_type,
+ info->start_idx, info->count, &sd_idx,
+ &sd_lmt);
+ if (sd_idx >= info->hmc_info->sd_table.sd_cnt ||
+ sd_lmt > info->hmc_info->sd_table.sd_cnt) {
+ return IRDMA_ERR_INVALID_SD_INDEX;
+ }
+
+ irdma_find_pd_index_limit(info->hmc_info, info->rsrc_type,
+ info->start_idx, info->count, &pd_idx,
+ &pd_lmt);
+
+ for (j = sd_idx; j < sd_lmt; j++) {
+ ret_code = irdma_add_sd_table_entry(dev->hw, info->hmc_info, j,
+ info->entry_type,
+ IRDMA_HMC_DIRECT_BP_SIZE);
+ if (ret_code)
+ goto exit_sd_error;
+
+ sd_entry = &info->hmc_info->sd_table.sd_entry[j];
+ if (sd_entry->entry_type == IRDMA_SD_TYPE_PAGED &&
+ (dev->hmc_info == info->hmc_info &&
+ info->rsrc_type != IRDMA_HMC_IW_PBLE)) {
+ pd_idx1 = max(pd_idx, (j * IRDMA_HMC_MAX_BP_COUNT));
+ pd_lmt1 = min(pd_lmt, (j + 1) * IRDMA_HMC_MAX_BP_COUNT);
+ for (i = pd_idx1; i < pd_lmt1; i++) {
+ /* update the pd table entry */
+ ret_code = irdma_add_pd_table_entry(dev,
+ info->hmc_info,
+ i, NULL);
+ if (ret_code) {
+ pd_error = true;
+ break;
+ }
+ }
+ if (pd_error) {
+ while (i && (i > pd_idx1)) {
+ irdma_remove_pd_bp(dev, info->hmc_info,
+ i - 1);
+ i--;
+ }
+ }
+ }
+ if (sd_entry->valid)
+ continue;
+
+ info->hmc_info->sd_indexes[info->add_sd_cnt] = (u16)j;
+ info->add_sd_cnt++;
+ sd_entry->valid = true;
+ }
+ return irdma_hmc_finish_add_sd_reg(dev, info);
+
+exit_sd_error:
+ while (j && (j > sd_idx)) {
+ sd_entry = &info->hmc_info->sd_table.sd_entry[j - 1];
+ switch (sd_entry->entry_type) {
+ case IRDMA_SD_TYPE_PAGED:
+ pd_idx1 = max(pd_idx, (j - 1) * IRDMA_HMC_MAX_BP_COUNT);
+ pd_lmt1 = min(pd_lmt, (j * IRDMA_HMC_MAX_BP_COUNT));
+ for (i = pd_idx1; i < pd_lmt1; i++)
+ irdma_prep_remove_pd_page(info->hmc_info, i);
+ break;
+ case IRDMA_SD_TYPE_DIRECT:
+ irdma_prep_remove_pd_page(info->hmc_info, (j - 1));
+ break;
+ default:
+ ret_code = IRDMA_ERR_INVALID_SD_TYPE;
+ break;
+ }
+ j--;
+ }
+
+ return ret_code;
+}
+
+/**
+ * irdma_finish_del_sd_reg - delete sd entries for objects
+ * @dev: pointer to the device structure
+ * @info: dele obj info
+ * @reset: true if called before reset
+ */
+static enum irdma_status_code
+irdma_finish_del_sd_reg(struct irdma_sc_dev *dev,
+ struct irdma_hmc_del_obj_info *info, bool reset)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+ enum irdma_status_code ret_code = 0;
+ u32 i, sd_idx;
+ struct irdma_dma_mem *mem;
+
+ if (dev->is_pf && !reset)
+ ret_code = irdma_hmc_sd_grp(dev, info->hmc_info,
+ info->hmc_info->sd_indexes[0],
+ info->del_sd_cnt, false);
+
+ if (ret_code)
+ dev_dbg(rfdev_to_dev(dev), "HMC: error cqp sd sd_grp\n");
+ for (i = 0; i < info->del_sd_cnt; i++) {
+ sd_idx = info->hmc_info->sd_indexes[i];
+ sd_entry = &info->hmc_info->sd_table.sd_entry[sd_idx];
+ if (!sd_entry)
+ continue;
+ mem = (sd_entry->entry_type == IRDMA_SD_TYPE_PAGED) ?
+ &sd_entry->u.pd_table.pd_page_addr :
+ &sd_entry->u.bp.addr;
+
+ if (!mem || !mem->va) {
+ dev_dbg(rfdev_to_dev(dev), "HMC: error cqp sd mem\n");
+ } else {
+ dma_free_coherent(hw_to_dev(dev->hw), mem->size,
+ mem->va, mem->pa);
+ mem->va = NULL;
+ }
+ }
+
+ return ret_code;
+}
+
+/**
+ * irdma_sc_del_hmc_obj - remove pe hmc objects
+ * @dev: pointer to the device structure
+ * @info: pointer to irdma_hmc_del_obj_info struct
+ * @reset: true if called before reset
+ *
+ * This will de-populate the SDs and PDs. It frees
+ * the memory for PDS and backing storage. After this function is returned,
+ * caller should deallocate memory allocated previously for
+ * book-keeping information about PDs and backing storage.
+ */
+enum irdma_status_code irdma_sc_del_hmc_obj(struct irdma_sc_dev *dev,
+ struct irdma_hmc_del_obj_info *info,
+ bool reset)
+{
+ struct irdma_hmc_pd_table *pd_table;
+ u32 sd_idx, sd_lmt;
+ u32 pd_idx, pd_lmt, rel_pd_idx;
+ u32 i, j;
+ enum irdma_status_code ret_code = 0;
+
+ if (info->start_idx >= info->hmc_info->hmc_obj[info->rsrc_type].cnt) {
+ dev_dbg(rfdev_to_dev(dev),
+ "HMC: error start_idx[%04d] >= [type %04d].cnt[%04d]\n",
+ info->start_idx, info->rsrc_type,
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt);
+ return IRDMA_ERR_INVALID_HMC_OBJ_INDEX;
+ }
+
+ if ((info->start_idx + info->count) >
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt) {
+ dev_dbg(rfdev_to_dev(dev),
+ "HMC: error start_idx[%04d] + count %04d >= [type %04d].cnt[%04d]\n",
+ info->start_idx, info->count, info->rsrc_type,
+ info->hmc_info->hmc_obj[info->rsrc_type].cnt);
+ return IRDMA_ERR_INVALID_HMC_OBJ_COUNT;
+ }
+
+ irdma_find_pd_index_limit(info->hmc_info, info->rsrc_type,
+ info->start_idx, info->count, &pd_idx,
+ &pd_lmt);
+
+ for (j = pd_idx; j < pd_lmt; j++) {
+ sd_idx = j / IRDMA_HMC_PD_CNT_IN_SD;
+
+ if (!info->hmc_info->sd_table.sd_entry[sd_idx].valid)
+ continue;
+
+ if (info->hmc_info->sd_table.sd_entry[sd_idx].entry_type !=
+ IRDMA_SD_TYPE_PAGED)
+ continue;
+
+ rel_pd_idx = j % IRDMA_HMC_PD_CNT_IN_SD;
+ pd_table = &info->hmc_info->sd_table.sd_entry[sd_idx].u.pd_table;
+ if (pd_table->pd_entry &&
+ pd_table->pd_entry[rel_pd_idx].valid) {
+ ret_code = irdma_remove_pd_bp(dev, info->hmc_info, j);
+ if (ret_code) {
+ dev_dbg(rfdev_to_dev(dev),
+ "HMC: remove_pd_bp error\n");
+ return ret_code;
+ }
+ }
+ }
+
+ irdma_find_sd_index_limit(info->hmc_info, info->rsrc_type,
+ info->start_idx, info->count, &sd_idx,
+ &sd_lmt);
+ if (sd_idx >= info->hmc_info->sd_table.sd_cnt ||
+ sd_lmt > info->hmc_info->sd_table.sd_cnt) {
+ dev_dbg(rfdev_to_dev(dev), "HMC: invalid sd_idx\n");
+ return IRDMA_ERR_INVALID_SD_INDEX;
+ }
+
+ for (i = sd_idx; i < sd_lmt; i++) {
+ pd_table = &info->hmc_info->sd_table.sd_entry[i].u.pd_table;
+ if (!info->hmc_info->sd_table.sd_entry[i].valid)
+ continue;
+ switch (info->hmc_info->sd_table.sd_entry[i].entry_type) {
+ case IRDMA_SD_TYPE_DIRECT:
+ ret_code = irdma_prep_remove_sd_bp(info->hmc_info, i);
+ if (!ret_code) {
+ info->hmc_info->sd_indexes[info->del_sd_cnt] =
+ (u16)i;
+ info->del_sd_cnt++;
+ }
+ break;
+ case IRDMA_SD_TYPE_PAGED:
+ ret_code = irdma_prep_remove_pd_page(info->hmc_info, i);
+ if (ret_code)
+ break;
+ if (dev->hmc_info != info->hmc_info &&
+ info->rsrc_type == IRDMA_HMC_IW_PBLE &&
+ pd_table->pd_entry) {
+ kfree(pd_table->pd_entry_virt_mem.va);
+ pd_table->pd_entry = NULL;
+ }
+ info->hmc_info->sd_indexes[info->del_sd_cnt] = (u16)i;
+ info->del_sd_cnt++;
+ break;
+ default:
+ break;
+ }
+ }
+ return irdma_finish_del_sd_reg(dev, info, reset);
+}
+
+/**
+ * irdma_add_sd_table_entry - Adds a segment descriptor to the table
+ * @hw: pointer to our hw struct
+ * @hmc_info: pointer to the HMC configuration information struct
+ * @sd_index: segment descriptor index to manipulate
+ * @type: what type of segment descriptor we're manipulating
+ * @direct_mode_sz: size to alloc in direct mode
+ */
+enum irdma_status_code irdma_add_sd_table_entry(struct irdma_hw *hw,
+ struct irdma_hmc_info *hmc_info,
+ u32 sd_index,
+ enum irdma_sd_entry_type type,
+ u64 direct_mode_sz)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+ struct irdma_dma_mem dma_mem;
+ u64 alloc_len;
+
+ sd_entry = &hmc_info->sd_table.sd_entry[sd_index];
+ if (!sd_entry->valid) {
+ if (type == IRDMA_SD_TYPE_PAGED)
+ alloc_len = IRDMA_HMC_PAGED_BP_SIZE;
+ else
+ alloc_len = direct_mode_sz;
+
+ /* allocate a 4K pd page or 2M backing page */
+ dma_mem.size = ALIGN(alloc_len, IRDMA_HMC_PD_BP_BUF_ALIGNMENT);
+ dma_mem.va = dma_alloc_coherent(hw_to_dev(hw),
+ dma_mem.size, &dma_mem.pa,
+ GFP_ATOMIC);
+ if (!dma_mem.va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ if (type == IRDMA_SD_TYPE_PAGED) {
+ struct irdma_virt_mem *vmem =
+ &sd_entry->u.pd_table.pd_entry_virt_mem;
+
+ vmem->size = sizeof(struct irdma_hmc_pd_entry) * 512;
+ vmem->va = kzalloc(vmem->size, GFP_ATOMIC);
+ if (!vmem->va) {
+ dma_free_coherent(hw_to_dev(hw), dma_mem.size,
+ dma_mem.va, dma_mem.pa);
+ dma_mem.va = NULL;
+ return IRDMA_ERR_NO_MEMORY;
+ }
+ sd_entry->u.pd_table.pd_entry = vmem->va;
+
+ memcpy(&sd_entry->u.pd_table.pd_page_addr, &dma_mem,
+ sizeof(sd_entry->u.pd_table.pd_page_addr));
+ } else {
+ memcpy(&sd_entry->u.bp.addr, &dma_mem,
+ sizeof(sd_entry->u.bp.addr));
+
+ sd_entry->u.bp.sd_pd_index = sd_index;
+ }
+
+ hmc_info->sd_table.sd_entry[sd_index].entry_type = type;
+ IRDMA_INC_SD_REFCNT(&hmc_info->sd_table);
+ }
+ if (sd_entry->entry_type == IRDMA_SD_TYPE_DIRECT)
+ IRDMA_INC_BP_REFCNT(&sd_entry->u.bp);
+
+ return 0;
+}
+
+/**
+ * irdma_add_pd_table_entry - Adds page descriptor to the specified table
+ * @dev: pointer to our device structure
+ * @hmc_info: pointer to the HMC configuration information structure
+ * @pd_index: which page descriptor index to manipulate
+ * @rsrc_pg: if not NULL, use preallocated page instead of allocating new one.
+ *
+ * This function:
+ * 1. Initializes the pd entry
+ * 2. Adds pd_entry in the pd_table
+ * 3. Mark the entry valid in irdma_hmc_pd_entry structure
+ * 4. Initializes the pd_entry's ref count to 1
+ * assumptions:
+ * 1. The memory for pd should be pinned down, physically contiguous and
+ * aligned on 4K boundary and zeroed memory.
+ * 2. It should be 4K in size.
+ */
+enum irdma_status_code irdma_add_pd_table_entry(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info,
+ u32 pd_index,
+ struct irdma_dma_mem *rsrc_pg)
+{
+ struct irdma_hmc_pd_table *pd_table;
+ struct irdma_hmc_pd_entry *pd_entry;
+ struct irdma_dma_mem mem;
+ struct irdma_dma_mem *page = &mem;
+ u32 sd_idx, rel_pd_idx;
+ u64 *pd_addr;
+ u64 page_desc;
+
+ if (pd_index / IRDMA_HMC_PD_CNT_IN_SD >= hmc_info->sd_table.sd_cnt)
+ return IRDMA_ERR_INVALID_PAGE_DESC_INDEX;
+
+ sd_idx = (pd_index / IRDMA_HMC_PD_CNT_IN_SD);
+ if (hmc_info->sd_table.sd_entry[sd_idx].entry_type !=
+ IRDMA_SD_TYPE_PAGED)
+ return 0;
+
+ rel_pd_idx = (pd_index % IRDMA_HMC_PD_CNT_IN_SD);
+ pd_table = &hmc_info->sd_table.sd_entry[sd_idx].u.pd_table;
+ pd_entry = &pd_table->pd_entry[rel_pd_idx];
+ if (!pd_entry->valid) {
+ if (rsrc_pg) {
+ pd_entry->rsrc_pg = true;
+ page = rsrc_pg;
+ } else {
+ page->size = ALIGN(IRDMA_HMC_PAGED_BP_SIZE,
+ IRDMA_HMC_PD_BP_BUF_ALIGNMENT);
+ page->va = dma_alloc_coherent(hw_to_dev(dev->hw),
+ page->size, &page->pa,
+ GFP_KERNEL);
+ if (!page->va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ pd_entry->rsrc_pg = false;
+ }
+
+ memcpy(&pd_entry->bp.addr, page, sizeof(pd_entry->bp.addr));
+ pd_entry->bp.sd_pd_index = pd_index;
+ pd_entry->bp.entry_type = IRDMA_SD_TYPE_PAGED;
+ page_desc = page->pa | 0x1;
+ pd_addr = pd_table->pd_page_addr.va;
+ pd_addr += rel_pd_idx;
+ memcpy(pd_addr, &page_desc, sizeof(*pd_addr));
+ pd_entry->sd_index = sd_idx;
+ pd_entry->valid = true;
+ IRDMA_INC_PD_REFCNT(pd_table);
+ if (hmc_info->hmc_fn_id < dev->hw_attrs.first_hw_vf_fpm_id)
+ IRDMA_INVALIDATE_PF_HMC_PD(dev, sd_idx, rel_pd_idx);
+ else if (dev->hw->hmc.hmc_fn_id != hmc_info->hmc_fn_id)
+ IRDMA_INVALIDATE_VF_HMC_PD(dev, sd_idx, rel_pd_idx,
+ hmc_info->hmc_fn_id);
+ }
+ IRDMA_INC_BP_REFCNT(&pd_entry->bp);
+
+ return 0;
+}
+
+/**
+ * irdma_remove_pd_bp - remove a backing page from a page descriptor
+ * @dev: pointer to our HW structure
+ * @hmc_info: pointer to the HMC configuration information structure
+ * @idx: the page index
+ *
+ * This function:
+ * 1. Marks the entry in pd table (for paged address mode) or in sd table
+ * (for direct address mode) invalid.
+ * 2. Write to register PMPDINV to invalidate the backing page in FV cache
+ * 3. Decrement the ref count for the pd _entry
+ * assumptions:
+ * 1. Caller can deallocate the memory used by backing storage after this
+ * function returns.
+ */
+enum irdma_status_code irdma_remove_pd_bp(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info,
+ u32 idx)
+{
+ struct irdma_hmc_pd_entry *pd_entry;
+ struct irdma_hmc_pd_table *pd_table;
+ struct irdma_hmc_sd_entry *sd_entry;
+ u32 sd_idx, rel_pd_idx;
+ struct irdma_dma_mem *mem;
+ u64 *pd_addr;
+
+ sd_idx = idx / IRDMA_HMC_PD_CNT_IN_SD;
+ rel_pd_idx = idx % IRDMA_HMC_PD_CNT_IN_SD;
+ if (sd_idx >= hmc_info->sd_table.sd_cnt)
+ return IRDMA_ERR_INVALID_PAGE_DESC_INDEX;
+
+ sd_entry = &hmc_info->sd_table.sd_entry[sd_idx];
+ if (sd_entry->entry_type != IRDMA_SD_TYPE_PAGED)
+ return IRDMA_ERR_INVALID_SD_TYPE;
+
+ pd_table = &hmc_info->sd_table.sd_entry[sd_idx].u.pd_table;
+ pd_entry = &pd_table->pd_entry[rel_pd_idx];
+ IRDMA_DEC_BP_REFCNT(&pd_entry->bp);
+ if (pd_entry->bp.ref_cnt)
+ return 0;
+
+ pd_entry->valid = false;
+ IRDMA_DEC_PD_REFCNT(pd_table);
+ pd_addr = pd_table->pd_page_addr.va;
+ pd_addr += rel_pd_idx;
+ memset(pd_addr, 0, sizeof(u64));
+ if (dev->is_pf) {
+ if (dev->hmc_fn_id == hmc_info->hmc_fn_id)
+ IRDMA_INVALIDATE_PF_HMC_PD(dev, sd_idx, idx);
+ else
+ IRDMA_INVALIDATE_VF_HMC_PD(dev, sd_idx, idx,
+ hmc_info->hmc_fn_id);
+ }
+
+ if (!pd_entry->rsrc_pg) {
+ mem = &pd_entry->bp.addr;
+ if (!mem || !mem->va)
+ return IRDMA_ERR_PARAM;
+
+ dma_free_coherent(hw_to_dev(dev->hw), mem->size, mem->va,
+ mem->pa);
+ mem->va = NULL;
+ }
+ if (!pd_table->ref_cnt)
+ kfree(pd_table->pd_entry_virt_mem.va);
+
+ return 0;
+}
+
+/**
+ * irdma_prep_remove_sd_bp - Prepares to remove a backing page from a sd entry
+ * @hmc_info: pointer to the HMC configuration information structure
+ * @idx: the page index
+ */
+enum irdma_status_code irdma_prep_remove_sd_bp(struct irdma_hmc_info *hmc_info,
+ u32 idx)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+
+ sd_entry = &hmc_info->sd_table.sd_entry[idx];
+ IRDMA_DEC_BP_REFCNT(&sd_entry->u.bp);
+ if (sd_entry->u.bp.ref_cnt)
+ return IRDMA_ERR_NOT_READY;
+
+ IRDMA_DEC_SD_REFCNT(&hmc_info->sd_table);
+ sd_entry->valid = false;
+
+ return 0;
+}
+
+/**
+ * irdma_prep_remove_pd_page - Prepares to remove a PD page from sd entry.
+ * @hmc_info: pointer to the HMC configuration information structure
+ * @idx: segment descriptor index to find the relevant page descriptor
+ */
+enum irdma_status_code
+irdma_prep_remove_pd_page(struct irdma_hmc_info *hmc_info, u32 idx)
+{
+ struct irdma_hmc_sd_entry *sd_entry;
+
+ sd_entry = &hmc_info->sd_table.sd_entry[idx];
+
+ if (sd_entry->u.pd_table.ref_cnt)
+ return IRDMA_ERR_NOT_READY;
+
+ sd_entry->valid = false;
+ IRDMA_DEC_SD_REFCNT(&hmc_info->sd_table);
+
+ return 0;
+}
diff --git a/drivers/infiniband/hw/irdma/hmc.h b/drivers/infiniband/hw/irdma/hmc.h
new file mode 100644
index 000000000000..66df742e82e3
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/hmc.h
@@ -0,0 +1,219 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef IRDMA_HMC_H
+#define IRDMA_HMC_H
+
+#include "defs.h"
+
+#define IRDMA_HMC_MAX_BP_COUNT 512
+#define IRDMA_MAX_SD_ENTRIES 11
+#define IRDMA_HW_DBG_HMC_INVALID_BP_MARK 0xca
+#define IRDMA_HMC_INFO_SIGNATURE 0x484d5347
+#define IRDMA_HMC_PD_CNT_IN_SD 512
+#define IRDMA_HMC_DIRECT_BP_SIZE 0x200000
+#define IRDMA_HMC_MAX_SD_COUNT 8192
+#define IRDMA_HMC_PAGED_BP_SIZE 4096
+#define IRDMA_HMC_PD_BP_BUF_ALIGNMENT 4096
+#define IRDMA_FIRST_VF_FPM_ID 8
+#define FPM_MULTIPLIER 1024
+
+#define IRDMA_INC_SD_REFCNT(sd_table) ((sd_table)->ref_cnt++)
+#define IRDMA_INC_PD_REFCNT(pd_table) ((pd_table)->ref_cnt++)
+#define IRDMA_INC_BP_REFCNT(bp) ((bp)->ref_cnt++)
+
+#define IRDMA_DEC_SD_REFCNT(sd_table) ((sd_table)->ref_cnt--)
+#define IRDMA_DEC_PD_REFCNT(pd_table) ((pd_table)->ref_cnt--)
+#define IRDMA_DEC_BP_REFCNT(bp) ((bp)->ref_cnt--)
+
+/**
+ * IRDMA_INVALIDATE_PF_HMC_PD - Invalidates the pd cache in the hardware
+ * @hw: pointer to our hw struct
+ * @sd_idx: segment descriptor index
+ * @pd_idx: page descriptor index
+ */
+#define IRDMA_INVALIDATE_PF_HMC_PD(dev, sd_idx, pd_idx) \
+ wr32((dev)->hw, (dev)->hw_regs[IRDMA_PFHMC_PDINV], \
+ (((sd_idx) << IRDMA_PFHMC_PDINV_PMSDIDX_S) | \
+ (0x1 << IRDMA_PFHMC_PDINV_PMSDPARTSEL_S) | \
+ ((pd_idx) << IRDMA_PFHMC_PDINV_PMPDIDX_S)))
+
+/**
+ * IRDMA_INVALIDATE_VF_HMC_PD - Invalidates the pd cache in the hardware
+ * @hw: pointer to our hw struct
+ * @sd_idx: segment descriptor index
+ * @pd_idx: page descriptor index
+ * @hmc_fn_id: VF's function id
+ */
+#define IRDMA_INVALIDATE_VF_HMC_PD(dev, sd_idx, pd_idx, hmc_fn_id) \
+ wr32((dev)->hw, \
+ (dev)->hw_regs[IRDMA_GLHMC_VFPDINV] + \
+ 4 * ((hmc_fn_id) - (dev)->hw_attrs.first_hw_vf_fpm_id), \
+ (((sd_idx) << IRDMA_PFHMC_PDINV_PMSDIDX_S) | \
+ ((pd_idx) << IRDMA_PFHMC_PDINV_PMPDIDX_S)))
+
+enum irdma_hmc_rsrc_type {
+ IRDMA_HMC_IW_QP = 0,
+ IRDMA_HMC_IW_CQ = 1,
+ IRDMA_HMC_IW_RESERVED = 2,
+ IRDMA_HMC_IW_HTE = 3,
+ IRDMA_HMC_IW_ARP = 4,
+ IRDMA_HMC_IW_APBVT_ENTRY = 5,
+ IRDMA_HMC_IW_MR = 6,
+ IRDMA_HMC_IW_XF = 7,
+ IRDMA_HMC_IW_XFFL = 8,
+ IRDMA_HMC_IW_Q1 = 9,
+ IRDMA_HMC_IW_Q1FL = 10,
+ IRDMA_HMC_IW_TIMER = 11,
+ IRDMA_HMC_IW_FSIMC = 12,
+ IRDMA_HMC_IW_FSIAV = 13,
+ IRDMA_HMC_IW_PBLE = 14,
+ IRDMA_HMC_IW_RRF = 15,
+ IRDMA_HMC_IW_RRFFL = 16,
+ IRDMA_HMC_IW_HDR = 17,
+ IRDMA_HMC_IW_MD = 18,
+ IRDMA_HMC_IW_OOISC = 19,
+ IRDMA_HMC_IW_OOISCFFL = 20,
+ IRDMA_HMC_IW_MAX, /* Must be last entry */
+};
+
+enum irdma_sd_entry_type {
+ IRDMA_SD_TYPE_INVALID = 0,
+ IRDMA_SD_TYPE_PAGED = 1,
+ IRDMA_SD_TYPE_DIRECT = 2,
+};
+
+struct irdma_hmc_obj_info {
+ u64 base;
+ u32 max_cnt;
+ u32 cnt;
+ u64 size;
+};
+
+struct irdma_hmc_bp {
+ enum irdma_sd_entry_type entry_type;
+ struct irdma_dma_mem addr;
+ u32 sd_pd_index;
+ u32 ref_cnt;
+};
+
+struct irdma_hmc_pd_entry {
+ struct irdma_hmc_bp bp;
+ u32 sd_index;
+ bool rsrc_pg;
+ bool valid;
+};
+
+struct irdma_hmc_pd_table {
+ struct irdma_dma_mem pd_page_addr;
+ struct irdma_hmc_pd_entry *pd_entry;
+ struct irdma_virt_mem pd_entry_virt_mem;
+ u32 ref_cnt;
+ u32 sd_index;
+};
+
+struct irdma_hmc_sd_entry {
+ enum irdma_sd_entry_type entry_type;
+ bool valid;
+ union {
+ struct irdma_hmc_pd_table pd_table;
+ struct irdma_hmc_bp bp;
+ } u;
+};
+
+struct irdma_hmc_sd_table {
+ struct irdma_virt_mem addr;
+ u32 sd_cnt;
+ u32 ref_cnt;
+ struct irdma_hmc_sd_entry *sd_entry;
+};
+
+struct irdma_hmc_info {
+ u32 signature;
+ u8 hmc_fn_id;
+ u16 first_sd_index;
+ struct irdma_hmc_obj_info *hmc_obj;
+ struct irdma_virt_mem hmc_obj_virt_mem;
+ struct irdma_hmc_sd_table sd_table;
+ u16 sd_indexes[IRDMA_HMC_MAX_SD_COUNT];
+};
+
+struct irdma_update_sd_entry {
+ u64 cmd;
+ u64 data;
+};
+
+struct irdma_update_sds_info {
+ u32 cnt;
+ u8 hmc_fn_id;
+ struct irdma_update_sd_entry entry[IRDMA_MAX_SD_ENTRIES];
+};
+
+struct irdma_ccq_cqe_info;
+struct irdma_hmc_fcn_info {
+ void (*callback_fcn)(struct irdma_sc_dev *dev, void *cqp_callback_param,
+ struct irdma_ccq_cqe_info *ccq_cqe_info);
+ void *cqp_callback_param;
+ u32 vf_id;
+ u16 iw_vf_idx;
+ bool free_fcn;
+};
+
+struct irdma_hmc_create_obj_info {
+ struct irdma_hmc_info *hmc_info;
+ struct irdma_virt_mem add_sd_virt_mem;
+ u32 rsrc_type;
+ u32 start_idx;
+ u32 count;
+ u32 add_sd_cnt;
+ enum irdma_sd_entry_type entry_type;
+ bool is_pf;
+};
+
+struct irdma_hmc_del_obj_info {
+ struct irdma_hmc_info *hmc_info;
+ struct irdma_virt_mem del_sd_virt_mem;
+ u32 rsrc_type;
+ u32 start_idx;
+ u32 count;
+ u32 del_sd_cnt;
+ bool is_pf;
+};
+
+enum irdma_status_code irdma_copy_dma_mem(struct irdma_hw *hw, void *dest_buf,
+ struct irdma_dma_mem *src_mem,
+ u64 src_offset, u64 size);
+enum irdma_status_code
+irdma_sc_create_hmc_obj(struct irdma_sc_dev *dev,
+ struct irdma_hmc_create_obj_info *info);
+enum irdma_status_code irdma_sc_del_hmc_obj(struct irdma_sc_dev *dev,
+ struct irdma_hmc_del_obj_info *info,
+ bool reset);
+enum irdma_status_code irdma_hmc_sd_one(struct irdma_sc_dev *dev, u8 hmc_fn_id,
+ u64 pa, u32 sd_idx,
+ enum irdma_sd_entry_type type,
+ bool setsd);
+enum irdma_status_code
+irdma_update_sds_noccq(struct irdma_sc_dev *dev,
+ struct irdma_update_sds_info *info);
+struct irdma_vfdev *irdma_vfdev_from_fpm(struct irdma_sc_dev *dev,
+ u8 hmc_fn_id);
+struct irdma_hmc_info *irdma_vf_hmcinfo_from_fpm(struct irdma_sc_dev *dev,
+ u8 hmc_fn_id);
+enum irdma_status_code irdma_add_sd_table_entry(struct irdma_hw *hw,
+ struct irdma_hmc_info *hmc_info,
+ u32 sd_index,
+ enum irdma_sd_entry_type type,
+ u64 direct_mode_sz);
+enum irdma_status_code irdma_add_pd_table_entry(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info,
+ u32 pd_index,
+ struct irdma_dma_mem *rsrc_pg);
+enum irdma_status_code irdma_remove_pd_bp(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info,
+ u32 idx);
+enum irdma_status_code irdma_prep_remove_sd_bp(struct irdma_hmc_info *hmc_info,
+ u32 idx);
+enum irdma_status_code
+irdma_prep_remove_pd_page(struct irdma_hmc_info *hmc_info, u32 idx);
+#endif /* IRDMA_HMC_H */
--
2.21.0
^ permalink raw reply related
* [rdma 02/16] RDMA/irdma: Implement device initialization definitions
From: Jeff Kirsher @ 2019-07-04 2:12 UTC (permalink / raw)
To: dledford, jgg, davem
Cc: Mustafa Ismail, linux-rdma, netdev, nhorman, sassmann, poswald,
david.m.ertman, Shiraz Saleem
In-Reply-To: <20190704021259.15489-1-jeffrey.t.kirsher@intel.com>
From: Mustafa Ismail <mustafa.ismail@intel.com>
Implement device initialization routines, interrupt set-up,
and allocate object bit-map tracking structures.
Also, add device specific attributes and register definitions.
Signed-off-by: Mustafa Ismail <mustafa.ismail@intel.com>
Signed-off-by: Shiraz Saleem <shiraz.saleem@intel.com>
---
drivers/infiniband/hw/irdma/hw.c | 2563 +++++++++++++++++++++++
drivers/infiniband/hw/irdma/i40iw_hw.c | 210 ++
drivers/infiniband/hw/irdma/i40iw_hw.h | 163 ++
drivers/infiniband/hw/irdma/icrdma_hw.c | 75 +
drivers/infiniband/hw/irdma/icrdma_hw.h | 63 +
5 files changed, 3074 insertions(+)
create mode 100644 drivers/infiniband/hw/irdma/hw.c
create mode 100644 drivers/infiniband/hw/irdma/i40iw_hw.c
create mode 100644 drivers/infiniband/hw/irdma/i40iw_hw.h
create mode 100644 drivers/infiniband/hw/irdma/icrdma_hw.c
create mode 100644 drivers/infiniband/hw/irdma/icrdma_hw.h
diff --git a/drivers/infiniband/hw/irdma/hw.c b/drivers/infiniband/hw/irdma/hw.c
new file mode 100644
index 000000000000..12c88b5fe68c
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/hw.c
@@ -0,0 +1,2563 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "main.h"
+
+static struct irdma_rsrc_limits rsrc_limits_table[] = {
+ [0] = {
+ .qplimit = 4096,
+ },
+ [1] = {
+ .qplimit = 128,
+ },
+ [2] = {
+ .qplimit = 1024,
+ },
+ [3] = {
+ .qplimit = 2048,
+ },
+ [4] = {
+ .qplimit = 16384,
+ },
+ [5] = {
+ .qplimit = 65536,
+ },
+};
+
+/* types of hmc objects */
+static enum irdma_hmc_rsrc_type iw_hmc_obj_types[] = {
+ IRDMA_HMC_IW_QP,
+ IRDMA_HMC_IW_CQ,
+ IRDMA_HMC_IW_HTE,
+ IRDMA_HMC_IW_ARP,
+ IRDMA_HMC_IW_APBVT_ENTRY,
+ IRDMA_HMC_IW_MR,
+ IRDMA_HMC_IW_XF,
+ IRDMA_HMC_IW_XFFL,
+ IRDMA_HMC_IW_Q1,
+ IRDMA_HMC_IW_Q1FL,
+ IRDMA_HMC_IW_TIMER,
+ IRDMA_HMC_IW_FSIMC,
+ IRDMA_HMC_IW_FSIAV,
+ IRDMA_HMC_IW_RRF,
+ IRDMA_HMC_IW_RRFFL,
+ IRDMA_HMC_IW_HDR,
+ IRDMA_HMC_IW_MD,
+ IRDMA_HMC_IW_OOISC,
+ IRDMA_HMC_IW_OOISCFFL,
+};
+
+/**
+ * irdma_iwarp_ce_handler - handle iwarp completions
+ * @iwcq: iwarp cq receiving event
+ */
+static void irdma_iwarp_ce_handler(struct irdma_sc_cq *iwcq)
+{
+ struct irdma_cq *cq = iwcq->back_cq;
+
+ if (cq->ibcq.comp_handler)
+ cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context);
+}
+
+/**
+ * irdma_puda_ce_handler - handle puda completion events
+ * @rf: RDMA PCI function
+ * @cq: puda completion q for event
+ */
+static void irdma_puda_ce_handler(struct irdma_pci_f *rf,
+ struct irdma_sc_cq *cq)
+{
+ struct irdma_sc_dev *dev = (struct irdma_sc_dev *)&rf->sc_dev;
+ enum irdma_status_code status;
+ u32 compl_error;
+
+ do {
+ status = irdma_puda_poll_cmpl(dev, cq, &compl_error);
+ if (status == IRDMA_ERR_Q_EMPTY)
+ break;
+ if (status) {
+ dev_dbg(rfdev_to_dev(dev), "ERR: puda status = %d\n",
+ status);
+ break;
+ }
+ if (compl_error) {
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: puda compl_err =0x%x\n", compl_error);
+ break;
+ }
+ } while (1);
+
+ dev->ccq_ops->ccq_arm(cq);
+}
+
+/**
+ * irdma_process_ceq - handle ceq for completions
+ * @rf: RDMA PCI function
+ * @ceq: ceq having cq for completion
+ */
+static void irdma_process_ceq(struct irdma_pci_f *rf, struct irdma_ceq *ceq)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_sc_ceq *sc_ceq;
+ struct irdma_sc_cq *cq;
+
+ sc_ceq = &ceq->sc_ceq;
+ do {
+ cq = dev->ceq_ops->process_ceq(dev, sc_ceq);
+ if (!cq)
+ break;
+
+ if (cq->cq_type == IRDMA_CQ_TYPE_CQP)
+ up(&rf->cqp.cqp_compl_sem);
+ else if (cq->cq_type == IRDMA_CQ_TYPE_IWARP)
+ irdma_iwarp_ce_handler(cq);
+ else if (cq->cq_type == IRDMA_CQ_TYPE_ILQ ||
+ cq->cq_type == IRDMA_CQ_TYPE_IEQ)
+ irdma_puda_ce_handler(rf, cq);
+ } while (1);
+}
+
+/**
+ * irdma_process_aeq - handle aeq events
+ * @rf: RDMA PCI function
+ */
+static void irdma_process_aeq(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_aeq *aeq = &rf->aeq;
+ struct irdma_sc_aeq *sc_aeq = &aeq->sc_aeq;
+ struct irdma_aeqe_info aeinfo;
+ struct irdma_aeqe_info *info = &aeinfo;
+ int ret;
+ struct irdma_qp *iwqp = NULL;
+ struct irdma_sc_cq *cq = NULL;
+ struct irdma_cq *iwcq = NULL;
+ struct irdma_sc_qp *qp = NULL;
+ struct irdma_qp_host_ctx_info *ctx_info = NULL;
+ unsigned long flags;
+
+ u32 aeqcnt = 0;
+
+ if (!sc_aeq->size)
+ return;
+
+ do {
+ memset(info, 0, sizeof(*info));
+ ret = dev->aeq_ops->get_next_aeqe(sc_aeq, info);
+ if (ret)
+ break;
+
+ aeqcnt++;
+ dev_dbg(rfdev_to_dev(dev),
+ "AEQ: ae_id = 0x%x bool qp=%d qp_id = %d\n",
+ info->ae_id, info->qp, info->qp_cq_id);
+ if (info->qp) {
+ spin_lock_irqsave(&rf->qptable_lock, flags);
+ iwqp = rf->qp_table[info->qp_cq_id];
+ if (!iwqp) {
+ spin_unlock_irqrestore(&rf->qptable_lock,
+ flags);
+ dev_dbg(rfdev_to_dev(dev),
+ "AEQ: qp_id %d is already freed\n",
+ info->qp_cq_id);
+ continue;
+ }
+ irdma_add_ref(&iwqp->ibqp);
+ spin_unlock_irqrestore(&rf->qptable_lock, flags);
+ qp = &iwqp->sc_qp;
+ spin_lock_irqsave(&iwqp->lock, flags);
+ iwqp->hw_tcp_state = info->tcp_state;
+ iwqp->hw_iwarp_state = info->iwarp_state;
+ iwqp->last_aeq = info->ae_id;
+ spin_unlock_irqrestore(&iwqp->lock, flags);
+ ctx_info = &iwqp->ctx_info;
+ if (rdma_protocol_roce(&iwqp->iwdev->iwibdev->ibdev, 1))
+ ctx_info->roce_info->err_rq_idx_valid = true;
+ else
+ ctx_info->iwarp_info->err_rq_idx_valid = true;
+ } else {
+ if (info->ae_id != IRDMA_AE_CQ_OPERATION_ERROR)
+ continue;
+ }
+
+ switch (info->ae_id) {
+ struct irdma_cm_node *cm_node;
+ case IRDMA_AE_LLP_CONNECTION_ESTABLISHED:
+ cm_node = iwqp->cm_node;
+ if (cm_node->accept_pend) {
+ atomic_dec(&cm_node->listener->pend_accepts_cnt);
+ cm_node->accept_pend = 0;
+ }
+ iwqp->rts_ae_rcvd = 1;
+ wake_up_interruptible(&iwqp->waitq);
+ break;
+ case IRDMA_AE_LLP_FIN_RECEIVED:
+ case IRDMA_AE_RDMAP_ROE_BAD_LLP_CLOSE:
+ if (qp->term_flags)
+ break;
+ if (atomic_inc_return(&iwqp->close_timer_started) == 1) {
+ iwqp->hw_tcp_state = IRDMA_TCP_STATE_CLOSE_WAIT;
+ if (iwqp->hw_tcp_state == IRDMA_TCP_STATE_CLOSE_WAIT &&
+ iwqp->ibqp_state == IB_QPS_RTS) {
+ irdma_next_iw_state(iwqp,
+ IRDMA_QP_STATE_CLOSING,
+ 0, 0, 0);
+ irdma_cm_disconn(iwqp);
+ }
+ iwqp->cm_id->add_ref(iwqp->cm_id);
+ irdma_schedule_cm_timer(iwqp->cm_node,
+ (struct irdma_puda_buf *)iwqp,
+ IRDMA_TIMER_TYPE_CLOSE,
+ 1, 0);
+ }
+ break;
+ case IRDMA_AE_LLP_CLOSE_COMPLETE:
+ if (qp->term_flags)
+ irdma_terminate_done(qp, 0);
+ else
+ irdma_cm_disconn(iwqp);
+ break;
+ case IRDMA_AE_BAD_CLOSE:
+ /* fall through */
+ case IRDMA_AE_RESET_SENT:
+ irdma_next_iw_state(iwqp, IRDMA_QP_STATE_ERROR, 1, 0,
+ 0);
+ irdma_cm_disconn(iwqp);
+ break;
+ case IRDMA_AE_LLP_CONNECTION_RESET:
+ if (atomic_read(&iwqp->close_timer_started))
+ break;
+ irdma_cm_disconn(iwqp);
+ break;
+ case IRDMA_AE_QP_SUSPEND_COMPLETE:
+ atomic_dec(&iwqp->sc_qp.vsi->qp_suspend_reqs);
+ wake_up(&iwqp->iwdev->suspend_wq);
+ break;
+ case IRDMA_AE_TERMINATE_SENT:
+ irdma_terminate_send_fin(qp);
+ break;
+ case IRDMA_AE_LLP_TERMINATE_RECEIVED:
+ irdma_terminate_received(qp, info);
+ break;
+ case IRDMA_AE_CQ_OPERATION_ERROR:
+ dev_err(rfdev_to_dev(dev),
+ "Processing an iWARP related AE for CQ misc = 0x%04X\n",
+ info->ae_id);
+ cq = (struct irdma_sc_cq *)(unsigned long)
+ info->compl_ctx;
+
+ iwcq = (struct irdma_cq *)cq->back_cq;
+
+ if (iwcq->ibcq.event_handler) {
+ struct ib_event ibevent;
+
+ ibevent.device = iwcq->ibcq.device;
+ ibevent.event = IB_EVENT_CQ_ERR;
+ ibevent.element.cq = &iwcq->ibcq;
+ iwcq->ibcq.event_handler(&ibevent,
+ iwcq->ibcq.cq_context);
+ }
+ break;
+ case IRDMA_AE_LLP_DOUBT_REACHABILITY:
+ case IRDMA_AE_RESOURCE_EXHAUSTION:
+ break;
+ case IRDMA_AE_PRIV_OPERATION_DENIED:
+ case IRDMA_AE_STAG_ZERO_INVALID:
+ case IRDMA_AE_IB_RREQ_AND_Q1_FULL:
+ case IRDMA_AE_DDP_UBE_INVALID_DDP_VERSION:
+ case IRDMA_AE_DDP_UBE_INVALID_MO:
+ case IRDMA_AE_DDP_UBE_INVALID_QN:
+ case IRDMA_AE_DDP_NO_L_BIT:
+ case IRDMA_AE_RDMAP_ROE_INVALID_RDMAP_VERSION:
+ case IRDMA_AE_RDMAP_ROE_UNEXPECTED_OPCODE:
+ case IRDMA_AE_ROE_INVALID_RDMA_READ_REQUEST:
+ case IRDMA_AE_ROE_INVALID_RDMA_WRITE_OR_READ_RESP:
+ case IRDMA_AE_INVALID_ARP_ENTRY:
+ case IRDMA_AE_INVALID_TCP_OPTION_RCVD:
+ case IRDMA_AE_STALE_ARP_ENTRY:
+ case IRDMA_AE_LLP_RECEIVED_MPA_CRC_ERROR:
+ case IRDMA_AE_LLP_SEGMENT_TOO_SMALL:
+ case IRDMA_AE_LLP_SYN_RECEIVED:
+ case IRDMA_AE_LLP_TOO_MANY_RETRIES:
+ case IRDMA_AE_LCE_QP_CATASTROPHIC:
+ case IRDMA_AE_LCE_FUNCTION_CATASTROPHIC:
+ case IRDMA_AE_LCE_CQ_CATASTROPHIC:
+ case IRDMA_AE_UDA_XMIT_DGRAM_TOO_LONG:
+ if (rdma_protocol_roce(&iwqp->iwdev->iwibdev->ibdev, 1))
+ ctx_info->roce_info->err_rq_idx_valid = false;
+ else
+ ctx_info->iwarp_info->err_rq_idx_valid = false;
+ /* fall through */
+ default:
+ dev_err(rfdev_to_dev(dev),
+ "abnormal ae_id = 0x%x bool qp=%d qp_id = %d\n",
+ info->ae_id, info->qp, info->qp_cq_id);
+ if (rdma_protocol_roce(&iwqp->iwdev->iwibdev->ibdev, 1)) {
+ if (!info->sq && ctx_info->roce_info->err_rq_idx_valid) {
+ ctx_info->roce_info->err_rq_idx = info->wqe_idx;
+ ret = dev->iw_priv_qp_ops->qp_setctx_roce(&iwqp->sc_qp,
+ iwqp->host_ctx.va,
+ ctx_info);
+ }
+ irdma_cm_disconn(iwqp);
+ break;
+ }
+ if (!info->sq && ctx_info->iwarp_info->err_rq_idx_valid) {
+ ctx_info->iwarp_info->err_rq_idx = info->wqe_idx;
+ ctx_info->tcp_info_valid = false;
+ ctx_info->iwarp_info_valid = false;
+ ret = dev->iw_priv_qp_ops->qp_setctx(&iwqp->sc_qp,
+ iwqp->host_ctx.va,
+ ctx_info);
+ }
+ if (iwqp->hw_iwarp_state != IRDMA_QP_STATE_RTS &&
+ iwqp->hw_iwarp_state != IRDMA_QP_STATE_TERMINATE) {
+ irdma_next_iw_state(iwqp, IRDMA_QP_STATE_ERROR, 1, 0, 0);
+ irdma_cm_disconn(iwqp);
+ } else {
+ iwqp->sc_qp.term_flags = 1;
+ irdma_next_iw_state(iwqp, IRDMA_QP_STATE_ERROR, 1, 0,
+ 0);
+ irdma_cm_disconn(iwqp);
+ }
+ break;
+ }
+ if (info->qp)
+ irdma_rem_ref(&iwqp->ibqp);
+ } while (1);
+
+ if (aeqcnt)
+ dev->aeq_ops->repost_aeq_entries(dev, aeqcnt);
+}
+
+/**
+ * irdma_enable_intr - set up device interrupts
+ * @dev: hardware control device structure
+ * @msix_id: id of the interrupt to be enabled
+ */
+static void irdma_ena_intr(struct irdma_sc_dev *dev, u32 msix_id)
+{
+ dev->irq_ops->irdma_en_irq(dev, msix_id);
+}
+
+/**
+ * irdma_dpc - tasklet for aeq and ceq 0
+ * @data: RDMA PCI function
+ */
+static void irdma_dpc(unsigned long data)
+{
+ struct irdma_pci_f *rf = (struct irdma_pci_f *)data;
+
+ if (rf->msix_shared)
+ irdma_process_ceq(rf, rf->ceqlist);
+ irdma_process_aeq(rf);
+ irdma_ena_intr(&rf->sc_dev, rf->iw_msixtbl[0].idx);
+}
+
+/**
+ * irdma_ceq_dpc - dpc handler for CEQ
+ * @data: data points to CEQ
+ */
+static void irdma_ceq_dpc(unsigned long data)
+{
+ struct irdma_ceq *iwceq = (struct irdma_ceq *)data;
+ struct irdma_pci_f *rf = iwceq->rf;
+
+ irdma_process_ceq(rf, iwceq);
+ irdma_ena_intr(&rf->sc_dev, iwceq->msix_idx);
+}
+
+/**
+ * irdma_save_msix_info - copy msix vector information to iwarp device
+ * @rf: RDMA PCI function
+ *
+ * Allocate iwdev msix table and copy the ldev msix info to the table
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_save_msix_info(struct irdma_pci_f *rf)
+{
+ struct irdma_priv_ldev *ldev = &rf->ldev;
+ struct irdma_qvlist_info *iw_qvlist;
+ struct irdma_qv_info *iw_qvinfo;
+ struct msix_entry *pmsix;
+ u32 ceq_idx;
+ u32 i;
+ u32 size;
+
+ if (!ldev->msix_count) {
+ pr_err("No MSI-X vectors for RDMA\n");
+ return IRDMA_ERR_CFG;
+ }
+
+ rf->msix_count = ldev->msix_count;
+ size = sizeof(struct irdma_msix_vector) * rf->msix_count;
+ size += sizeof(struct irdma_qvlist_info);
+ size += sizeof(struct irdma_qv_info) * rf->msix_count - 1;
+ rf->iw_msixtbl = kzalloc(size, GFP_KERNEL);
+ if (!rf->iw_msixtbl)
+ return IRDMA_ERR_NO_MEMORY;
+
+ rf->iw_qvlist = (struct irdma_qvlist_info *)
+ (&rf->iw_msixtbl[rf->msix_count]);
+ iw_qvlist = rf->iw_qvlist;
+ iw_qvinfo = iw_qvlist->qv_info;
+ iw_qvlist->num_vectors = rf->msix_count;
+ if (rf->msix_count <= num_online_cpus())
+ rf->msix_shared = true;
+
+ for (i = 0, ceq_idx = 0, pmsix = ldev->msix_entries; i < rf->msix_count;
+ i++, iw_qvinfo++, pmsix++) {
+ rf->iw_msixtbl[i].idx = pmsix->entry;
+ rf->iw_msixtbl[i].irq = pmsix->vector;
+ rf->iw_msixtbl[i].cpu_affinity = ceq_idx;
+ if (!i) {
+ iw_qvinfo->aeq_idx = 0;
+ if (rf->msix_shared)
+ iw_qvinfo->ceq_idx = ceq_idx++;
+ else
+ iw_qvinfo->ceq_idx = IRDMA_Q_INVALID_IDX;
+ } else {
+ iw_qvinfo->aeq_idx = IRDMA_Q_INVALID_IDX;
+ iw_qvinfo->ceq_idx = ceq_idx++;
+ }
+ iw_qvinfo->itr_idx = 3;
+ iw_qvinfo->v_idx = rf->iw_msixtbl[i].idx;
+ }
+
+ return 0;
+}
+
+/**
+ * irdma_irq_handler - interrupt handler for aeq and ceq0
+ * @irq: Interrupt request number
+ * @data: RDMA PCI function
+ */
+static irqreturn_t irdma_irq_handler(int irq, void *data)
+{
+ struct irdma_pci_f *rf = data;
+
+ tasklet_schedule(&rf->dpc_tasklet);
+
+ return IRQ_HANDLED;
+}
+
+/**
+ * irdma_ceq_handler - interrupt handler for ceq
+ * @irq: interrupt request number
+ * @data: ceq pointer
+ */
+static irqreturn_t irdma_ceq_handler(int irq, void *data)
+{
+ struct irdma_ceq *iwceq = data;
+
+ if (iwceq->irq != irq)
+ dev_err(rfdev_to_dev(&iwceq->rf->sc_dev),
+ "expected irq = %d received irq = %d\n", iwceq->irq,
+ irq);
+ tasklet_schedule(&iwceq->dpc_tasklet);
+
+ return IRQ_HANDLED;
+}
+
+/**
+ * irdma_destroy_irq - destroy device interrupts
+ * @rf: RDMA PCI function
+ * @msix_vec: msix vector to disable irq
+ * @dev_id: parameter to pass to free_irq (used during irq setup)
+ *
+ * The function is called when destroying aeq/ceq
+ */
+static void irdma_destroy_irq(struct irdma_pci_f *rf,
+ struct irdma_msix_vector *msix_vec, void *dev_id)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+
+ dev->irq_ops->irdma_dis_irq(dev, msix_vec->idx);
+ irq_set_affinity_hint(msix_vec->irq, NULL);
+ free_irq(msix_vec->irq, dev_id);
+}
+
+/**
+ * irdma_destroy_cqp - destroy control qp
+ * @rf: RDMA PCI function
+ * @free_hwcqp: 1 if hw cqp should be freed
+ *
+ * Issue destroy cqp request and
+ * free the resources associated with the cqp
+ */
+static void irdma_destroy_cqp(struct irdma_pci_f *rf, bool free_hwcqp)
+{
+ enum irdma_status_code status = 0;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_cqp *cqp = &rf->cqp;
+
+ if (free_hwcqp && dev->cqp_ops->cqp_destroy)
+ status = dev->cqp_ops->cqp_destroy(dev->cqp);
+ if (status)
+ dev_dbg(rfdev_to_dev(dev), "ERR: Destroy CQP failed %d\n",
+ status);
+
+ irdma_cleanup_pending_cqp_op(rf);
+ dma_free_coherent(hw_to_dev(dev->hw), cqp->sq.size, cqp->sq.va,
+ cqp->sq.pa);
+ cqp->sq.va = NULL;
+ kfree(cqp->scratch_array);
+ cqp->scratch_array = NULL;
+ kfree(cqp->cqp_requests);
+ cqp->cqp_requests = NULL;
+}
+
+/**
+ * irdma_destroy_aeq - destroy aeq
+ * @rf: RDMA PCI function
+ *
+ * Issue a destroy aeq request and
+ * free the resources associated with the aeq
+ * The function is called during driver unload
+ */
+static void irdma_destroy_aeq(struct irdma_pci_f *rf)
+{
+ enum irdma_status_code status = IRDMA_ERR_NOT_READY;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_aeq *aeq = &rf->aeq;
+
+ if (!rf->msix_shared)
+ irdma_destroy_irq(rf, rf->iw_msixtbl, (void *)rf);
+ if (rf->reset)
+ goto exit;
+
+ if (!dev->aeq_ops->aeq_destroy(&aeq->sc_aeq, 0, 1))
+ status = dev->aeq_ops->aeq_destroy_done(&aeq->sc_aeq);
+ if (status)
+ dev_dbg(rfdev_to_dev(dev), "ERR: Destroy AEQ failed %d\n",
+ status);
+
+exit:
+ dma_free_coherent(hw_to_dev(dev->hw), aeq->mem.size, aeq->mem.va,
+ aeq->mem.pa);
+ aeq->mem.va = NULL;
+}
+
+/**
+ * irdma_destroy_ceq - destroy ceq
+ * @rf: RDMA PCI function
+ * @iwceq: ceq to be destroyed
+ *
+ * Issue a destroy ceq request and
+ * free the resources associated with the ceq
+ */
+static void irdma_destroy_ceq(struct irdma_pci_f *rf, struct irdma_ceq *iwceq)
+{
+ enum irdma_status_code status;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+
+ if (rf->reset)
+ goto exit;
+
+ status = dev->ceq_ops->ceq_destroy(&iwceq->sc_ceq, 0, 1);
+ if (status) {
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: CEQ destroy command failed %d\n", status);
+ goto exit;
+ }
+
+ status = dev->ceq_ops->cceq_destroy_done(&iwceq->sc_ceq);
+ if (status)
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: CEQ destroy completion failed %d\n", status);
+exit:
+ dma_free_coherent(hw_to_dev(dev->hw), iwceq->mem.size, iwceq->mem.va,
+ iwceq->mem.pa);
+ iwceq->mem.va = NULL;
+}
+
+/**
+ * irdma_del_ceq_0 - destroy ceq 0
+ * @rf: RDMA PCI function
+ *
+ * Disable the ceq 0 interrupt and destroy the ceq 0
+ */
+static void irdma_del_ceq_0(struct irdma_pci_f *rf)
+{
+ struct irdma_ceq *iwceq = rf->ceqlist;
+ struct irdma_msix_vector *msix_vec;
+
+ if (rf->msix_shared) {
+ msix_vec = &rf->iw_msixtbl[0];
+ irdma_destroy_irq(rf, msix_vec, (void *)rf);
+ } else {
+ msix_vec = &rf->iw_msixtbl[1];
+ irdma_destroy_irq(rf, msix_vec, (void *)iwceq);
+ }
+ irdma_destroy_ceq(rf, iwceq);
+ rf->sc_dev.ceq_valid = false;
+ rf->ceqs_count = 0;
+}
+
+/**
+ * irdma_del_ceqs - destroy all ceq's except CEQ 0 // RT mode FSL
+ * @rf: RDMA PCI function
+ *
+ * Go through all of the device ceq's, except 0, and for each
+ * ceq disable the ceq interrupt and destroy the ceq
+ */
+static void irdma_del_ceqs(struct irdma_pci_f *rf)
+{
+ struct irdma_ceq *iwceq = &rf->ceqlist[1];
+ struct irdma_msix_vector *msix_vec;
+ u32 i = 0;
+
+ if (rf->msix_shared)
+ msix_vec = &rf->iw_msixtbl[1];
+ else
+ msix_vec = &rf->iw_msixtbl[2];
+
+ for (i = 1; i < rf->ceqs_count; i++, msix_vec++, iwceq++) {
+ irdma_destroy_irq(rf, msix_vec, (void *)iwceq);
+ irdma_cqp_ceq_cmd(&rf->sc_dev, &iwceq->sc_ceq,
+ IRDMA_OP_CEQ_DESTROY);
+ dma_free_coherent(hw_to_dev(rf->sc_dev.hw), iwceq->mem.size,
+ iwceq->mem.va, iwceq->mem.pa);
+ iwceq->mem.va = NULL;
+ }
+ rf->ceqs_count = 1;
+}
+
+/**
+ * irdma_destroy_ccq - destroy control cq
+ * @rf: RDMA PCI function
+ *
+ * Issue destroy ccq request and
+ * free the resources associated with the ccq
+ */
+static void irdma_destroy_ccq(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_ccq *ccq = &rf->ccq;
+ enum irdma_status_code status = 0;
+
+ if (!rf->reset)
+ status = dev->ccq_ops->ccq_destroy(dev->ccq, 0, true);
+ if (status)
+ dev_dbg(rfdev_to_dev(dev), "ERR: CCQ destroy failed %d\n",
+ status);
+ dma_free_coherent(hw_to_dev(dev->hw), ccq->mem_cq.size,
+ ccq->mem_cq.va, ccq->mem_cq.pa);
+ ccq->mem_cq.va = NULL;
+}
+
+/**
+ * irdma_close_hmc_objects_type - delete hmc objects of a given type
+ * @dev: iwarp device
+ * @obj_type: the hmc object type to be deleted
+ * @hmc_info: host memory info struct
+ * @is_pf: true if the function is PF otherwise false
+ * @reset: true if called before reset
+ */
+static void irdma_close_hmc_objects_type(struct irdma_sc_dev *dev,
+ enum irdma_hmc_rsrc_type obj_type,
+ struct irdma_hmc_info *hmc_info,
+ bool is_pf, bool reset)
+{
+ struct irdma_hmc_del_obj_info info = {};
+
+ info.hmc_info = hmc_info;
+ info.rsrc_type = obj_type;
+ info.count = hmc_info->hmc_obj[obj_type].cnt;
+ info.is_pf = is_pf;
+ if (dev->hmc_ops->del_hmc_object(dev, &info, reset))
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: del HMC obj of type %d failed\n", obj_type);
+}
+
+/**
+ * irdma_del_hmc_objects - remove all device hmc objects
+ * @dev: iwarp device
+ * @hmc_info: hmc_info to free
+ * @is_pf: true if hmc_info belongs to PF, not vf nor allocated
+ * by PF on behalf of VF
+ * @reset: true if called before reset
+ * @vers: hardware version
+ */
+static void irdma_del_hmc_objects(struct irdma_sc_dev *dev,
+ struct irdma_hmc_info *hmc_info, bool is_pf,
+ bool reset, enum irdma_vers vers)
+{
+ unsigned int i;
+
+ for (i = 0; i < IW_HMC_OBJ_TYPE_NUM; i++) {
+ if (dev->hmc_info->hmc_obj[iw_hmc_obj_types[i]].cnt)
+ irdma_close_hmc_objects_type(dev, iw_hmc_obj_types[i],
+ hmc_info, is_pf, reset);
+ if (vers == IRDMA_GEN_1 && i == IRDMA_HMC_IW_TIMER)
+ break;
+ }
+}
+
+/**
+ * irdma_create_hmc_obj_type - create hmc object of a given type
+ * @dev: hardware control device structure
+ * @info: information for the hmc object to create
+ */
+static enum irdma_status_code
+irdma_create_hmc_obj_type(struct irdma_sc_dev *dev,
+ struct irdma_hmc_create_obj_info *info)
+{
+ return dev->hmc_ops->create_hmc_object(dev, info);
+}
+
+/**
+ * irdma_create_hmc_objs - create all hmc objects for the device
+ * @rf: RDMA PCI function
+ * @is_pf: true if the function is PF otherwise false
+ * @vers: HW version
+ *
+ * Create the device hmc objects and allocate hmc pages
+ * Return 0 if successful, otherwise clean up and return error
+ */
+static enum irdma_status_code
+irdma_create_hmc_objs(struct irdma_pci_f *rf, bool is_pf, enum irdma_vers vers)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_hmc_create_obj_info info = {};
+ enum irdma_status_code status = 0;
+ int i;
+
+ info.hmc_info = dev->hmc_info;
+ info.is_pf = is_pf;
+ info.entry_type = rf->sd_type;
+
+ for (i = 0; i < IW_HMC_OBJ_TYPE_NUM; i++) {
+ if (dev->hmc_info->hmc_obj[iw_hmc_obj_types[i]].cnt) {
+ info.rsrc_type = iw_hmc_obj_types[i];
+ info.count = dev->hmc_info->hmc_obj[info.rsrc_type].cnt;
+ info.add_sd_cnt = 0;
+ status = irdma_create_hmc_obj_type(dev, &info);
+ if (status) {
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: create obj type %d status = %d\n",
+ iw_hmc_obj_types[i], status);
+ break;
+ }
+ }
+ if (vers == IRDMA_GEN_1 && i == IRDMA_HMC_IW_TIMER)
+ break;
+ }
+
+ if (!status)
+ return dev->hmc_ops->static_hmc_pages_allocated(dev->cqp, 0,
+ dev->hmc_fn_id,
+ true, true);
+
+ while (i) {
+ i--;
+ /* destroy the hmc objects of a given type */
+ irdma_close_hmc_objects_type(dev, iw_hmc_obj_types[i],
+ dev->hmc_info, is_pf, false);
+ }
+
+ return status;
+}
+
+/**
+ * irdma_obj_aligned_mem - get aligned memory from device allocated memory
+ * @rf: RDMA PCI function
+ * @memptr: points to the memory addresses
+ * @size: size of memory needed
+ * @mask: mask for the aligned memory
+ *
+ * Get aligned memory of the requested size and
+ * update the memptr to point to the new aligned memory
+ * Return 0 if successful, otherwise return no memory error
+ */
+static enum irdma_status_code
+irdma_obj_aligned_mem(struct irdma_pci_f *rf, struct irdma_dma_mem *memptr,
+ u32 size, u32 mask)
+{
+ unsigned long va, newva;
+ unsigned long extra;
+
+ va = (unsigned long)rf->obj_next.va;
+ newva = va;
+ if (mask)
+ newva = ALIGN(va, (unsigned long)mask + 1ULL);
+ extra = newva - va;
+ memptr->va = (u8 *)va + extra;
+ memptr->pa = rf->obj_next.pa + extra;
+ memptr->size = size;
+ if ((memptr->va + size) > (rf->obj_mem.va + rf->obj_mem.size))
+ return IRDMA_ERR_NO_MEMORY;
+
+ rf->obj_next.va = memptr->va + size;
+ rf->obj_next.pa = memptr->pa + size;
+
+ return 0;
+}
+
+/**
+ * irdma_create_cqp - create control qp
+ * @rf: RDMA PCI function
+ *
+ * Return 0, if the cqp and all the resources associated with it
+ * are successfully created, otherwise return error
+ */
+static enum irdma_status_code irdma_create_cqp(struct irdma_pci_f *rf)
+{
+ enum irdma_status_code status;
+ u32 sqsize = IRDMA_CQP_SW_SQSIZE_2048;
+ struct irdma_dma_mem mem;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_cqp_init_info cqp_init_info = {};
+ struct irdma_cqp *cqp = &rf->cqp;
+ u16 maj_err, min_err;
+ int i;
+
+ cqp->cqp_requests = kcalloc(sqsize, sizeof(*cqp->cqp_requests), GFP_KERNEL);
+ if (!cqp->cqp_requests)
+ return IRDMA_ERR_NO_MEMORY;
+
+ cqp->scratch_array = kcalloc(sqsize, sizeof(*cqp->scratch_array), GFP_KERNEL);
+ if (!cqp->scratch_array) {
+ kfree(cqp->cqp_requests);
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ dev->cqp = &cqp->sc_cqp;
+ dev->cqp->dev = dev;
+ cqp->sq.size = ALIGN(sizeof(struct irdma_cqp_sq_wqe) * sqsize,
+ IRDMA_CQP_ALIGNMENT);
+ cqp->sq.va = dma_alloc_coherent(hw_to_dev(dev->hw), cqp->sq.size,
+ &cqp->sq.pa, GFP_KERNEL);
+ if (!cqp->sq.va) {
+ kfree(cqp->scratch_array);
+ kfree(cqp->cqp_requests);
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ status = irdma_obj_aligned_mem(rf, &mem, sizeof(struct irdma_cqp_ctx),
+ IRDMA_HOST_CTX_ALIGNMENT_M);
+ if (status)
+ goto exit;
+
+ dev->cqp->host_ctx_pa = mem.pa;
+ dev->cqp->host_ctx = mem.va;
+ /* populate the cqp init info */
+ cqp_init_info.dev = dev;
+ cqp_init_info.sq_size = sqsize;
+ cqp_init_info.sq = cqp->sq.va;
+ cqp_init_info.sq_pa = cqp->sq.pa;
+ cqp_init_info.host_ctx_pa = mem.pa;
+ cqp_init_info.host_ctx = mem.va;
+ cqp_init_info.hmc_profile = rf->rsrc_profile;
+ cqp_init_info.ena_vf_count = rf->max_rdma_vfs;
+ cqp_init_info.scratch_array = cqp->scratch_array;
+ cqp_init_info.disable_packed = true;
+ cqp_init_info.protocol_used = rf->protocol_used;
+ status = dev->cqp_ops->cqp_init(dev->cqp, &cqp_init_info);
+ if (status) {
+ dev_dbg(rfdev_to_dev(dev), "ERR: cqp init status %d\n",
+ status);
+ goto exit;
+ }
+
+ status = dev->cqp_ops->cqp_create(dev->cqp, &maj_err, &min_err);
+ if (status) {
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: cqp create failed - status %d maj_err %d min_err %d\n",
+ status, maj_err, min_err);
+ goto exit;
+ }
+
+ spin_lock_init(&cqp->req_lock);
+ spin_lock_init(&cqp->compl_lock);
+ INIT_LIST_HEAD(&cqp->cqp_avail_reqs);
+ INIT_LIST_HEAD(&cqp->cqp_pending_reqs);
+ sema_init(&cqp->cqp_compl_sem, 0);
+
+ /* init the waitqueue of the cqp_requests and add them to the list */
+ for (i = 0; i < sqsize; i++) {
+ init_waitqueue_head(&cqp->cqp_requests[i].waitq);
+ list_add_tail(&cqp->cqp_requests[i].list, &cqp->cqp_avail_reqs);
+ }
+ init_waitqueue_head(&cqp->remove_wq);
+ return 0;
+
+exit:
+ irdma_destroy_cqp(rf, false);
+
+ return status;
+}
+
+/**
+ * irdma_create_ccq - create control cq
+ * @rf: RDMA PCI function
+ *
+ * Return 0, if the ccq and the resources associated with it
+ * are successfully created, otherwise return error
+ */
+static enum irdma_status_code irdma_create_ccq(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ enum irdma_status_code status;
+ struct irdma_ccq_init_info info = {};
+ struct irdma_ccq *ccq = &rf->ccq;
+
+ dev->ccq = &ccq->sc_cq;
+ dev->ccq->dev = dev;
+ info.dev = dev;
+ ccq->shadow_area.size = sizeof(struct irdma_cq_shadow_area);
+ ccq->mem_cq.size = ALIGN(sizeof(struct irdma_cqe) * IW_CCQ_SIZE,
+ IRDMA_CQ0_ALIGNMENT);
+ ccq->mem_cq.va = dma_alloc_coherent(hw_to_dev(dev->hw),
+ ccq->mem_cq.size, &ccq->mem_cq.pa,
+ GFP_KERNEL);
+ if (!ccq->mem_cq.va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ status = irdma_obj_aligned_mem(rf, &ccq->shadow_area,
+ ccq->shadow_area.size,
+ IRDMA_SHADOWAREA_M);
+ if (status)
+ goto exit;
+
+ ccq->sc_cq.back_cq = (void *)ccq;
+ /* populate the ccq init info */
+ info.cq_base = ccq->mem_cq.va;
+ info.cq_pa = ccq->mem_cq.pa;
+ info.num_elem = IW_CCQ_SIZE;
+ info.shadow_area = ccq->shadow_area.va;
+ info.shadow_area_pa = ccq->shadow_area.pa;
+ info.ceqe_mask = false;
+ info.ceq_id_valid = true;
+ info.shadow_read_threshold = 16;
+ info.vsi = &rf->default_vsi;
+ status = dev->ccq_ops->ccq_init(dev->ccq, &info);
+ if (!status)
+ status = dev->ccq_ops->ccq_create(dev->ccq, 0, true, true);
+exit:
+ if (status) {
+ dma_free_coherent(hw_to_dev(dev->hw), ccq->mem_cq.size,
+ ccq->mem_cq.va, ccq->mem_cq.pa);
+ ccq->mem_cq.va = NULL;
+ }
+
+ return status;
+}
+
+/**
+ * irdma_alloc_set_mac - set up a mac address table entry
+ * @iwdev: device
+ *
+ * Allocate a mac ip entry and add it to the hw table Return 0
+ * if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_alloc_set_mac(struct irdma_device *iwdev)
+{
+ enum irdma_status_code status;
+
+ status = irdma_alloc_local_mac_entry(iwdev->rf,
+ &iwdev->mac_ip_table_idx);
+ if (!status) {
+ status = irdma_add_local_mac_entry(iwdev->rf,
+ (u8 *)iwdev->netdev->dev_addr,
+ (u8)iwdev->mac_ip_table_idx);
+ if (status)
+ irdma_del_local_mac_entry(iwdev->rf,
+ (u8)iwdev->mac_ip_table_idx);
+ }
+ return status;
+}
+
+/**
+ * irdma_configure_ceq_vector - set up the msix interrupt vector for ceq
+ * @rf: RDMA PCI function
+ * @iwceq: ceq associated with the vector
+ * @ceq_id: the id number of the iwceq
+ * @msix_vec: interrupt vector information
+ *
+ * Allocate interrupt resources and enable irq handling
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code
+irdma_cfg_ceq_vector(struct irdma_pci_f *rf, struct irdma_ceq *iwceq,
+ u32 ceq_id, struct irdma_msix_vector *msix_vec)
+{
+ int status;
+
+ if (rf->msix_shared && !ceq_id) {
+ tasklet_init(&rf->dpc_tasklet, irdma_dpc, (unsigned long)rf);
+ status = request_irq(msix_vec->irq, irdma_irq_handler, 0,
+ "AEQCEQ", rf);
+ } else {
+ tasklet_init(&iwceq->dpc_tasklet, irdma_ceq_dpc,
+ (unsigned long)iwceq);
+
+ status = request_irq(msix_vec->irq, irdma_ceq_handler, 0, "CEQ",
+ iwceq);
+ }
+
+ cpumask_clear(&msix_vec->mask);
+ cpumask_set_cpu(msix_vec->cpu_affinity, &msix_vec->mask);
+ irq_set_affinity_hint(msix_vec->irq, &msix_vec->mask);
+ if (status) {
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: ceq irq config fail\n");
+ return IRDMA_ERR_CFG;
+ }
+
+ msix_vec->ceq_id = ceq_id;
+ rf->sc_dev.irq_ops->irdma_cfg_ceq(&rf->sc_dev, ceq_id, msix_vec->idx);
+
+ return 0;
+}
+
+/**
+ * irdma_configure_aeq_vector - set up the msix vector for aeq
+ * @rf: RDMA PCI function
+ *
+ * Allocate interrupt resources and enable irq handling
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_cfg_aeq_vector(struct irdma_pci_f *rf)
+{
+ struct irdma_msix_vector *msix_vec = rf->iw_msixtbl;
+ u32 ret = 0;
+
+ if (!rf->msix_shared) {
+ tasklet_init(&rf->dpc_tasklet, irdma_dpc, (unsigned long)rf);
+ ret = request_irq(msix_vec->irq, irdma_irq_handler, 0, "irdma",
+ rf);
+ }
+ if (ret) {
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: aeq irq config fail\n");
+ return IRDMA_ERR_CFG;
+ }
+
+ rf->sc_dev.irq_ops->irdma_cfg_aeq(&rf->sc_dev, msix_vec->idx);
+
+ return 0;
+}
+
+/**
+ * irdma_create_ceq - create completion event queue
+ * @rf: RDMA PCI function
+ * @iwceq: pointer to the ceq resources to be created
+ * @ceq_id: the id number of the iwceq
+ * @vsi: SC vsi struct
+ *
+ * Return 0, if the ceq and the resources associated with it
+ * are successfully created, otherwise return error
+ */
+static enum irdma_status_code irdma_create_ceq(struct irdma_pci_f *rf,
+ struct irdma_ceq *iwceq,
+ u32 ceq_id,
+ struct irdma_sc_vsi *vsi)
+{
+ enum irdma_status_code status;
+ struct irdma_ceq_init_info info = {};
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ u64 scratch;
+
+ info.ceq_id = ceq_id;
+ iwceq->rf = rf;
+ iwceq->mem.size = ALIGN(sizeof(struct irdma_ceqe) * rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt,
+ IRDMA_CEQ_ALIGNMENT);
+ iwceq->mem.va = dma_alloc_coherent(hw_to_dev(dev->hw),
+ iwceq->mem.size, &iwceq->mem.pa,
+ GFP_KERNEL);
+ if (!iwceq->mem.va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ info.ceq_id = ceq_id;
+ info.ceqe_base = iwceq->mem.va;
+ info.ceqe_pa = iwceq->mem.pa;
+ info.elem_cnt = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt;
+ iwceq->sc_ceq.ceq_id = ceq_id;
+ info.dev = dev;
+ info.vsi = vsi;
+ scratch = (uintptr_t)&rf->cqp.sc_cqp;
+ status = dev->ceq_ops->ceq_init(&iwceq->sc_ceq, &info);
+ if (!status) {
+ if (dev->ceq_valid)
+ status = irdma_cqp_ceq_cmd(&rf->sc_dev, &iwceq->sc_ceq,
+ IRDMA_OP_CEQ_CREATE);
+ else
+ status = dev->ceq_ops->cceq_create(&iwceq->sc_ceq,
+ scratch);
+ }
+
+ if (status) {
+ dma_free_coherent(hw_to_dev(dev->hw), iwceq->mem.size,
+ iwceq->mem.va, iwceq->mem.pa);
+ iwceq->mem.va = NULL;
+ }
+
+ return status;
+}
+
+/**
+ * irdma_setup_ceq_0 - create CEQ 0 and it's interrupt resource
+ * @rf: RDMA PCI function
+ *
+ * Allocate a list for all device completion event queues
+ * Create the ceq 0 and configure it's msix interrupt vector
+ * Return 0, if successfully set up, otherwise return error
+ */
+static enum irdma_status_code irdma_setup_ceq_0(struct irdma_pci_f *rf)
+{
+ u32 i;
+ struct irdma_ceq *iwceq;
+ struct irdma_msix_vector *msix_vec;
+ enum irdma_status_code status = 0;
+ u32 num_ceqs;
+
+ num_ceqs = min(rf->msix_count, rf->sc_dev.hmc_fpm_misc.max_ceqs);
+ rf->ceqlist = kcalloc(num_ceqs, sizeof(*rf->ceqlist), GFP_KERNEL);
+ if (!rf->ceqlist) {
+ status = IRDMA_ERR_NO_MEMORY;
+ goto exit;
+ }
+
+ i = rf->msix_shared ? 0 : 1;
+ iwceq = &rf->ceqlist[0];
+ status = irdma_create_ceq(rf, iwceq, 0, &rf->default_vsi);
+ if (status) {
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: create ceq status = %d\n", status);
+ goto exit;
+ }
+
+ msix_vec = &rf->iw_msixtbl[i];
+ iwceq->irq = msix_vec->irq;
+ iwceq->msix_idx = msix_vec->idx;
+ status = irdma_cfg_ceq_vector(rf, iwceq, 0, msix_vec);
+ if (status) {
+ irdma_destroy_ceq(rf, iwceq);
+ goto exit;
+ }
+
+ irdma_ena_intr(&rf->sc_dev, msix_vec->idx);
+ rf->ceqs_count++;
+
+exit:
+ if (status && !rf->ceqs_count) {
+ kfree(rf->ceqlist);
+ rf->ceqlist = NULL;
+ return status;
+ }
+ rf->sc_dev.ceq_valid = true;
+
+ return 0;
+}
+
+/**
+ * irdma_setup_ceqs - manage the device ceq's and their interrupt resources
+ * @rf: RDMA PCI function
+ * @vsi: VSI structure for this CEQ
+ *
+ * Allocate a list for all device completion event queues
+ * Create the ceq's and configure their msix interrupt vectors
+ * Return 0, if at least one ceq is successfully set up, otherwise return error
+ */
+static enum irdma_status_code irdma_setup_ceqs(struct irdma_pci_f *rf,
+ struct irdma_sc_vsi *vsi)
+{
+ u32 i;
+ u32 ceq_id;
+ struct irdma_ceq *iwceq;
+ struct irdma_msix_vector *msix_vec;
+ enum irdma_status_code status = 0;
+ u32 num_ceqs;
+
+ num_ceqs = min(rf->msix_count, rf->sc_dev.hmc_fpm_misc.max_ceqs);
+ i = (rf->msix_shared) ? 1 : 2;
+ for (ceq_id = 1; i < num_ceqs; i++, ceq_id++) {
+ iwceq = &rf->ceqlist[ceq_id];
+ status = irdma_create_ceq(rf, iwceq, ceq_id, vsi);
+ if (status) {
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: create ceq status = %d\n", status);
+ break;
+ }
+ msix_vec = &rf->iw_msixtbl[i];
+ iwceq->irq = msix_vec->irq;
+ iwceq->msix_idx = msix_vec->idx;
+ status = irdma_cfg_ceq_vector(rf, iwceq, ceq_id, msix_vec);
+ if (status) {
+ irdma_destroy_ceq(rf, iwceq);
+ break;
+ }
+ irdma_ena_intr(&rf->sc_dev, msix_vec->idx);
+ rf->ceqs_count++;
+ }
+
+ return status;
+}
+
+/**
+ * irdma_create_aeq - create async event queue
+ * @rf: RDMA PCI function
+ *
+ * Return 0, if the aeq and the resources associated with it
+ * are successfully created, otherwise return error
+ */
+static enum irdma_status_code irdma_create_aeq(struct irdma_pci_f *rf)
+{
+ enum irdma_status_code status;
+ struct irdma_aeq_init_info info = {};
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_aeq *aeq = &rf->aeq;
+ struct irdma_hmc_info *hmc_info = rf->sc_dev.hmc_info;
+ u64 scratch = 0;
+ u32 aeq_size;
+
+ aeq_size = 2 * hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt +
+ hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt;
+ aeq->mem.size = ALIGN(sizeof(struct irdma_sc_aeqe) * aeq_size,
+ IRDMA_AEQ_ALIGNMENT);
+ aeq->mem.va = dma_alloc_coherent(hw_to_dev(dev->hw), aeq->mem.size,
+ &aeq->mem.pa, GFP_KERNEL);
+ if (!aeq->mem.va)
+ return IRDMA_ERR_NO_MEMORY;
+
+ info.aeqe_base = aeq->mem.va;
+ info.aeq_elem_pa = aeq->mem.pa;
+ info.elem_cnt = aeq_size;
+ info.dev = dev;
+ status = dev->aeq_ops->aeq_init(&aeq->sc_aeq, &info);
+ if (status)
+ goto exit;
+
+ status = dev->aeq_ops->aeq_create(&aeq->sc_aeq, scratch, 1);
+ if (!status)
+ status = dev->aeq_ops->aeq_create_done(&aeq->sc_aeq);
+exit:
+ if (status) {
+ dma_free_coherent(hw_to_dev(dev->hw), aeq->mem.size,
+ aeq->mem.va, aeq->mem.pa);
+ aeq->mem.va = NULL;
+ }
+
+ return status;
+}
+
+/**
+ * irdma_setup_aeq - set up the device aeq
+ * @rf: RDMA PCI function
+ *
+ * Create the aeq and configure its msix interrupt vector
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_setup_aeq(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ enum irdma_status_code status;
+
+ status = irdma_create_aeq(rf);
+ if (status)
+ return status;
+
+ status = irdma_cfg_aeq_vector(rf);
+ if (status) {
+ irdma_destroy_aeq(rf);
+ return status;
+ }
+
+ if (!rf->msix_shared)
+ irdma_ena_intr(dev, rf->iw_msixtbl[0].idx);
+
+ return 0;
+}
+
+/**
+ * irdma_initialize_ilq - create iwarp local queue for cm
+ * @iwdev: iwarp device
+ *
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_initialize_ilq(struct irdma_device *iwdev)
+{
+ struct irdma_puda_rsrc_info info = {};
+ enum irdma_status_code status;
+
+ info.type = IRDMA_PUDA_RSRC_TYPE_ILQ;
+ info.cq_id = 1;
+ info.qp_id = 1;
+ info.count = 1;
+ info.pd_id = 1;
+ info.sq_size = min(iwdev->rf->max_qp / 2, (u32)32768);
+ info.rq_size = info.sq_size;
+ info.buf_size = 1024;
+ info.tx_buf_cnt = 2 * info.sq_size;
+ info.receive = irdma_receive_ilq;
+ info.xmit_complete = irdma_free_sqbuf;
+ status = irdma_puda_create_rsrc(&iwdev->vsi, &info);
+ if (status)
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "ERR: ilq create fail\n");
+
+ return status;
+}
+
+/**
+ * irdma_initialize_ieq - create iwarp exception queue
+ * @iwdev: iwarp device
+ *
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_initialize_ieq(struct irdma_device *iwdev)
+{
+ struct irdma_puda_rsrc_info info = {};
+ enum irdma_status_code status;
+
+ info.type = IRDMA_PUDA_RSRC_TYPE_IEQ;
+ info.cq_id = 2;
+ info.qp_id = iwdev->vsi.exception_lan_q;
+ info.count = 1;
+ info.pd_id = 2;
+ info.sq_size = min(iwdev->rf->max_qp / 2, (u32)32768);
+ info.rq_size = info.sq_size;
+ info.buf_size = iwdev->vsi.mtu + IRDMA_IPV4_PAD;
+ info.tx_buf_cnt = 4096;
+ status = irdma_puda_create_rsrc(&iwdev->vsi, &info);
+ if (status)
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "ERR: ieq create fail\n");
+
+ return status;
+}
+
+/**
+ * irdma_reinitialize_ieq - destroy and re-create ieq
+ * @vsi: VSI structure
+ */
+void irdma_reinitialize_ieq(struct irdma_sc_vsi *vsi)
+{
+ struct irdma_device *iwdev = vsi->back_vsi;
+
+ irdma_puda_dele_rsrc(vsi, IRDMA_PUDA_RSRC_TYPE_IEQ, false);
+ if (irdma_initialize_ieq(iwdev)) {
+ iwdev->reset = true;
+ irdma_request_reset(iwdev->rf);
+ }
+}
+
+/**
+ * irdma_hmc_setup - create hmc objects for the device
+ * @rf: RDMA PCI function
+ *
+ * Set up the device private memory space for the number and size of
+ * the hmc objects and create the objects
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_hmc_setup(struct irdma_pci_f *rf)
+{
+ enum irdma_status_code status;
+ u32 qpcnt;
+
+ if (rf->rdma_ver == IRDMA_GEN_1)
+ qpcnt = rsrc_limits_table[rf->limits_sel].qplimit * 2;
+ else
+ qpcnt = rsrc_limits_table[rf->limits_sel].qplimit;
+
+ rf->sd_type = IRDMA_SD_TYPE_DIRECT;
+ status = irdma_cfg_fpm_val(&rf->sc_dev, qpcnt);
+ if (status)
+ return status;
+
+ status = irdma_create_hmc_objs(rf, true, rf->rdma_ver);
+
+ return status;
+}
+
+/**
+ * irdma_del_init_mem - deallocate memory resources
+ * @rf: RDMA PCI function
+ */
+static void irdma_del_init_mem(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+
+ kfree(dev->hmc_info->sd_table.sd_entry);
+ dev->hmc_info->sd_table.sd_entry = NULL;
+ kfree(rf->mem_rsrc);
+ rf->mem_rsrc = NULL;
+ dma_free_coherent(hw_to_dev(&rf->hw), rf->obj_mem.size,
+ rf->obj_mem.va, rf->obj_mem.pa);
+ rf->obj_mem.va = NULL;
+ if (rf->rdma_ver != IRDMA_GEN_1) {
+ kfree(rf->allocated_ws_nodes);
+ rf->allocated_ws_nodes = NULL;
+ }
+ kfree(rf->ceqlist);
+ rf->ceqlist = NULL;
+ kfree(rf->iw_msixtbl);
+ rf->iw_msixtbl = NULL;
+ kfree(rf->hmc_info_mem);
+ rf->hmc_info_mem = NULL;
+}
+
+/**
+ * irdma_initialize_dev - initialize device
+ * @rf: RDMA PCI function
+ * @ldev: lan device information
+ *
+ * Allocate memory for the hmc objects and initialize iwdev
+ * Return 0 if successful, otherwise clean up the resources
+ * and return error
+ */
+static enum irdma_status_code irdma_initialize_dev(struct irdma_pci_f *rf,
+ struct irdma_priv_ldev *ldev)
+{
+ enum irdma_status_code status;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ struct irdma_device_init_info info = {};
+ struct irdma_dma_mem mem;
+ u32 size;
+
+ size = sizeof(struct irdma_hmc_pble_rsrc) +
+ sizeof(struct irdma_hmc_info) +
+ (sizeof(struct irdma_hmc_obj_info) * IRDMA_HMC_IW_MAX);
+
+ rf->hmc_info_mem = kzalloc(size, GFP_KERNEL);
+ if (!rf->hmc_info_mem)
+ return IRDMA_ERR_NO_MEMORY;
+
+ rf->pble_rsrc = (struct irdma_hmc_pble_rsrc *)rf->hmc_info_mem;
+ dev->hmc_info = &rf->hw.hmc;
+ dev->hmc_info->hmc_obj = (struct irdma_hmc_obj_info *)
+ (rf->pble_rsrc + 1);
+
+ status = irdma_obj_aligned_mem(rf, &mem, IRDMA_QUERY_FPM_BUF_SIZE,
+ IRDMA_FPM_QUERY_BUF_ALIGNMENT_M);
+ if (status)
+ goto error;
+
+ info.fpm_query_buf_pa = mem.pa;
+ info.fpm_query_buf = mem.va;
+ info.init_hw = rf->init_hw;
+
+ status = irdma_obj_aligned_mem(rf, &mem, IRDMA_COMMIT_FPM_BUF_SIZE,
+ IRDMA_FPM_COMMIT_BUF_ALIGNMENT_M);
+ if (status)
+ goto error;
+
+ info.fpm_commit_buf_pa = mem.pa;
+ info.fpm_commit_buf = mem.va;
+
+ info.bar0 = rf->hw.hw_addr;
+ info.hmc_fn_id = (u8)ldev->fn_num;
+ info.is_pf = !ldev->ftype;
+ info.hw = &rf->hw;
+ info.vchnl_send = NULL;
+ status = irdma_sc_ctrl_init(rf->rdma_ver, &rf->sc_dev, &info);
+ if (status)
+ goto error;
+
+ return status;
+error:
+ kfree(rf->hmc_info_mem);
+ rf->hmc_info_mem = NULL;
+
+ return status;
+}
+
+/**
+ * irdma_deinit_rt_device - clean up the device resources
+ * @iwdev: iwarp device
+ *
+ * Destroy the ib device interface, remove the mac ip entry and
+ * ipv4/ipv6 addresses, destroy the device queues and free the
+ * pble and the hmc objects
+ */
+void irdma_deinit_rt_device(struct irdma_device *iwdev)
+{
+ dev_info(rfdev_to_dev(&iwdev->rf->sc_dev), "state = %d\n",
+ iwdev->init_state);
+
+ switch (iwdev->init_state) {
+ case RDMA_DEV_REGISTERED:
+ iwdev->iw_status = 0;
+ irdma_port_ibevent(iwdev);
+ irdma_destroy_rdma_device(iwdev->iwibdev);
+ /* fallthrough */
+ case IP_ADDR_REGISTERED:
+ if (iwdev->rf->sc_dev.hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_1)
+ irdma_del_local_mac_entry(iwdev->rf,
+ (u8)iwdev->mac_ip_table_idx);
+ /* fallthrough */
+ case PBLE_CHUNK_MEM:
+ /* fallthrough */
+ case CEQS_CREATED:
+ /* fallthrough */
+ case IEQ_CREATED:
+ irdma_puda_dele_rsrc(&iwdev->vsi, IRDMA_PUDA_RSRC_TYPE_IEQ,
+ iwdev->reset);
+ /* fallthrough */
+ case ILQ_CREATED:
+ if (iwdev->create_ilq)
+ irdma_puda_dele_rsrc(&iwdev->vsi,
+ IRDMA_PUDA_RSRC_TYPE_ILQ,
+ iwdev->reset);
+ break;
+ default:
+ dev_warn(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "bad init_state = %d\n", iwdev->init_state);
+ break;
+ }
+
+ irdma_cleanup_cm_core(&iwdev->cm_core);
+ if (iwdev->vsi.pestat) {
+ irdma_vsi_stats_free(&iwdev->vsi);
+ kfree(iwdev->vsi.pestat);
+ }
+}
+
+/**
+ * irdma_setup_init_state - set up the initial device struct
+ * @rf: RDMA PCI function
+ *
+ * Initialize the iwarp device and its hdl information
+ * using the ldev and client information
+ * Return 0 if successful, otherwise return error
+ */
+static enum irdma_status_code irdma_setup_init_state(struct irdma_pci_f *rf)
+{
+ struct irdma_priv_ldev *ldev = &rf->ldev;
+ enum irdma_status_code status;
+
+ status = irdma_save_msix_info(rf);
+ if (status)
+ return status;
+
+ rf->hw.pdev = rf->pdev;
+ rf->obj_mem.size = ALIGN(8192, IRDMA_HW_PAGE_SIZE);
+ rf->obj_mem.va = dma_alloc_coherent(hw_to_dev(&rf->hw),
+ rf->obj_mem.size, &rf->obj_mem.pa,
+ GFP_KERNEL);
+ if (!rf->obj_mem.va) {
+ kfree(rf->iw_msixtbl);
+ rf->iw_msixtbl = NULL;
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ rf->obj_next = rf->obj_mem;
+ rf->ooo = false;
+ init_waitqueue_head(&rf->vchnl_waitq);
+
+ status = irdma_initialize_dev(rf, ldev);
+ if (status) {
+ kfree(rf->iw_msixtbl);
+ dma_free_coherent(hw_to_dev(&rf->hw), rf->obj_mem.size,
+ rf->obj_mem.va, rf->obj_mem.pa);
+ rf->obj_mem.va = NULL;
+ rf->iw_msixtbl = NULL;
+ }
+
+ return status;
+}
+
+/**
+ * irdma_get_used_rsrc - determine resources used internally
+ * @iwdev: iwarp device
+ *
+ * Called at the end of open to get all internal allocations
+ */
+static void irdma_get_used_rsrc(struct irdma_device *iwdev)
+{
+ iwdev->rf->used_pds = find_next_zero_bit(iwdev->rf->allocated_pds,
+ iwdev->rf->max_pd, 0);
+ iwdev->rf->used_qps = find_next_zero_bit(iwdev->rf->allocated_qps,
+ iwdev->rf->max_qp, 0);
+ iwdev->rf->used_cqs = find_next_zero_bit(iwdev->rf->allocated_cqs,
+ iwdev->rf->max_cq, 0);
+ iwdev->rf->used_mrs = find_next_zero_bit(iwdev->rf->allocated_mrs,
+ iwdev->rf->max_mr, 0);
+}
+
+/**
+ * irdma_deinit_hw - De-initializes RDMA HW
+ * @rf: RDMA device information
+ *
+ */
+void irdma_deinit_ctrl_hw(struct irdma_pci_f *rf)
+{
+ enum init_completion_state state = rf->init_state;
+
+ rf->init_state = INVALID_STATE;
+ if (rf->rsrc_created) {
+ irdma_destroy_pble_prm(rf->pble_rsrc);
+ irdma_del_ceqs(rf);
+ rf->rsrc_created = false;
+ }
+ switch (state) {
+ case CEQ0_CREATED:
+ irdma_del_ceq_0(rf);
+ /* fallthrough */
+ case AEQ_CREATED:
+ irdma_destroy_aeq(rf);
+ /* fallthrough */
+ case CCQ_CREATED:
+ irdma_destroy_ccq(rf);
+ /* fallthrough */
+ case HMC_OBJS_CREATED:
+ irdma_del_hmc_objects(&rf->sc_dev, rf->sc_dev.hmc_info, true,
+ rf->reset, rf->rdma_ver);
+ /* fallthrough */
+ case CQP_CREATED:
+ if (rf->cqp.cqp_compl_thread) {
+ rf->stop_cqp_thread = true;
+ up(&rf->cqp.cqp_compl_sem);
+ kthread_stop(rf->cqp.cqp_compl_thread);
+ }
+ irdma_destroy_cqp(rf, true);
+ /* fallthrough */
+ case INITIAL_STATE:
+ irdma_del_init_mem(rf);
+ break;
+ case INVALID_STATE:
+ /* fallthrough */
+ default:
+ pr_warn("bad init_state = %d\n", rf->init_state);
+ break;
+ }
+}
+
+enum irdma_status_code irdma_rt_init_hw(struct irdma_pci_f *rf,
+ struct irdma_device *iwdev,
+ struct irdma_l2params *l2params)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ enum irdma_status_code status;
+ struct irdma_vsi_init_info vsi_info = {};
+ struct irdma_vsi_stats_info stats_info = {};
+
+ irdma_sc_rt_init(dev);
+ vsi_info.vm_vf_type = dev->is_pf ? IRDMA_PF_TYPE : IRDMA_VF_TYPE;
+ vsi_info.dev = dev;
+ vsi_info.back_vsi = (void *)iwdev;
+ vsi_info.params = l2params;
+ vsi_info.pf_data_vsi_num = iwdev->vsi_num;
+ vsi_info.exception_lan_q = 2;
+ irdma_sc_vsi_init(&iwdev->vsi, &vsi_info);
+
+ status = irdma_setup_cm_core(iwdev, rf->rdma_ver);
+ if (status)
+ return status;
+
+ stats_info.pestat = kzalloc(sizeof(*stats_info.pestat), GFP_KERNEL);
+ if (!stats_info.pestat)
+ return IRDMA_ERR_NO_MEMORY;
+
+ stats_info.fcn_id = dev->hmc_fn_id;
+ status = irdma_vsi_stats_init(&iwdev->vsi, &stats_info);
+ if (status) {
+ kfree(stats_info.pestat);
+ return status;
+ }
+
+ do {
+ if (iwdev->create_ilq) {
+ status = irdma_initialize_ilq(iwdev);
+ if (status)
+ break;
+ iwdev->init_state = ILQ_CREATED;
+ }
+ status = irdma_initialize_ieq(iwdev);
+ if (status)
+ break;
+ iwdev->init_state = IEQ_CREATED;
+ if (!rf->rsrc_created) {
+ status = irdma_setup_ceqs(rf, &iwdev->vsi);
+ if (status)
+ break;
+ iwdev->init_state = CEQS_CREATED;
+
+ status = irdma_hmc_init_pble(&rf->sc_dev,
+ rf->pble_rsrc);
+ if (status) {
+ irdma_del_ceqs(rf);
+ break;
+ }
+ spin_lock_init(&rf->pble_rsrc->pble_lock);
+ iwdev->init_state = PBLE_CHUNK_MEM;
+ rf->rsrc_created = true;
+ }
+
+ iwdev->device_cap_flags = IB_DEVICE_LOCAL_DMA_LKEY |
+ IB_DEVICE_MEM_WINDOW |
+ IB_DEVICE_MEM_MGT_EXTENSIONS;
+
+ if (iwdev->rf->sc_dev.hw_attrs.uk_attrs.hw_rev == IRDMA_GEN_1)
+ irdma_alloc_set_mac(iwdev);
+ irdma_add_ip(iwdev);
+ iwdev->init_state = IP_ADDR_REGISTERED;
+ status = irdma_register_rdma_device(iwdev);
+ if (status)
+ break;
+ iwdev->init_state = RDMA_DEV_REGISTERED;
+ irdma_port_ibevent(iwdev);
+ iwdev->iw_status = 1;
+ irdma_get_used_rsrc(iwdev);
+ init_waitqueue_head(&iwdev->suspend_wq);
+
+ return 0;
+ } while (0);
+
+ dev_err(rfdev_to_dev(dev), "VSI open FAIL status = %d last cmpl = %d\n",
+ status, iwdev->init_state);
+ irdma_deinit_rt_device(iwdev);
+
+ return status;
+}
+
+/**
+ * irdma_ctrl_init_hw - Initializes RDMA HW
+ * @rf: RDMA PCI function
+ *
+ */
+enum irdma_status_code irdma_ctrl_init_hw(struct irdma_pci_f *rf)
+{
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ enum irdma_status_code status;
+
+ do {
+ status = irdma_setup_init_state(rf);
+ if (status)
+ break;
+ rf->init_state = INITIAL_STATE;
+
+ status = irdma_create_cqp(rf);
+ if (status)
+ break;
+ rf->init_state = CQP_CREATED;
+
+ status = irdma_hmc_setup(rf);
+ if (status)
+ break;
+ rf->init_state = HMC_OBJS_CREATED;
+
+ status = irdma_initialize_hw_rsrc(rf);
+ if (status)
+ break;
+
+ status = irdma_create_ccq(rf);
+ if (status)
+ break;
+ rf->init_state = CCQ_CREATED;
+
+ status = irdma_setup_aeq(rf);
+ if (status)
+ break;
+ rf->init_state = AEQ_CREATED;
+ rf->sc_dev.feature_info[IRDMA_FEATURE_FW_INFO] = IRDMA_FW_VER_DEFAULT;
+
+ if (rf->rdma_ver != IRDMA_GEN_1)
+ status = irdma_get_rdma_features(&rf->sc_dev);
+ if (!status) {
+ u32 fw_ver = dev->feature_info[IRDMA_FEATURE_FW_INFO];
+ u8 hw_rev = dev->hw_attrs.uk_attrs.hw_rev;
+
+ if ((hw_rev == IRDMA_GEN_1 && fw_ver >= IRDMA_FW_VER_0x30010) ||
+ (hw_rev != IRDMA_GEN_1 && fw_ver >= IRDMA_FW_VER_0x1000D))
+
+ dev->hw_attrs.uk_attrs.feature_flags |= IRDMA_FEATURE_RTS_AE |
+ IRDMA_FEATURE_CQ_RESIZE;
+ }
+ rf->cqp.cqp_compl_thread =
+ kthread_run(cqp_compl_thread, rf, "cqp_compl_thread");
+
+ status = irdma_setup_ceq_0(rf);
+ if (status)
+ break;
+ rf->init_state = CEQ0_CREATED;
+
+ rf->free_qp_wq =
+ alloc_ordered_workqueue("free_qp_wq", WQ_MEM_RECLAIM);
+ if (!rf->free_qp_wq) {
+ status = IRDMA_ERR_NO_MEMORY;
+ break;
+ }
+
+ rf->free_cqbuf_wq =
+ alloc_ordered_workqueue("free_cqbuf_wq", WQ_MEM_RECLAIM);
+ if (!rf->free_cqbuf_wq) {
+ status = IRDMA_ERR_NO_MEMORY;
+ break;
+ }
+ dev->ccq_ops->ccq_arm(dev->ccq);
+ dev_info(rfdev_to_dev(dev), "IRDMA hardware initialization successful\n");
+ return 0;
+ } while (0);
+
+ pr_err("IRDMA hardware initialization FAILED init_state=%d status=%d\n",
+ rf->init_state, status);
+ irdma_deinit_ctrl_hw(rf);
+ return status;
+}
+
+/**
+ * irdma_initialize_hw_resources - initialize hw resource during open
+ * @rf: RDMA PCI function
+ */
+u32 irdma_initialize_hw_rsrc(struct irdma_pci_f *rf)
+{
+ unsigned long num_pds;
+ u32 rsrc_size;
+ u32 max_mr;
+ u32 max_qp;
+ u32 max_cq;
+ u32 arp_table_size;
+ u32 mrdrvbits;
+ void *rsrc_ptr;
+ u32 num_ahs;
+ u32 num_mcg;
+
+ if (rf->rdma_ver != IRDMA_GEN_1) {
+ rf->allocated_ws_nodes =
+ kcalloc(BITS_TO_LONGS(IRDMA_MAX_WS_NODES),
+ sizeof(unsigned long), GFP_KERNEL);
+ if (!rf->allocated_ws_nodes)
+ return -ENOMEM;
+
+ set_bit(0, rf->allocated_ws_nodes);
+ rf->max_ws_node_id = IRDMA_MAX_WS_NODES;
+ }
+ max_qp = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_QP].cnt;
+ max_cq = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_CQ].cnt;
+ max_mr = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_MR].cnt;
+ arp_table_size = rf->sc_dev.hmc_info->hmc_obj[IRDMA_HMC_IW_ARP].cnt;
+ rf->max_cqe = rf->sc_dev.hw_attrs.uk_attrs.max_hw_cq_size;
+ num_pds = rf->sc_dev.hw_attrs.max_hw_pds;
+ rsrc_size = sizeof(struct irdma_arp_entry) * arp_table_size;
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(max_qp);
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(max_mr);
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(max_cq);
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(num_pds);
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(arp_table_size);
+ num_ahs = max_qp * 4;
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(num_ahs);
+ num_mcg = max_qp;
+ rsrc_size += sizeof(unsigned long) * BITS_TO_LONGS(num_mcg);
+ rsrc_size += sizeof(struct irdma_qp **) * max_qp;
+
+ rf->mem_rsrc = kzalloc(rsrc_size, GFP_KERNEL);
+ if (!rf->mem_rsrc) {
+ kfree(rf->allocated_ws_nodes);
+ rf->allocated_ws_nodes = NULL;
+ return -ENOMEM;
+ }
+
+ rf->max_qp = max_qp;
+ rf->max_mr = max_mr;
+ rf->max_cq = max_cq;
+ rf->max_pd = num_pds;
+ rf->arp_table_size = arp_table_size;
+ rf->arp_table = (struct irdma_arp_entry *)rf->mem_rsrc;
+ rsrc_ptr = rf->mem_rsrc +
+ (sizeof(struct irdma_arp_entry) * arp_table_size);
+ rf->max_ah = num_ahs;
+ rf->max_mcg = num_mcg;
+ rf->allocated_qps = rsrc_ptr;
+ rf->allocated_cqs = &rf->allocated_qps[BITS_TO_LONGS(max_qp)];
+ rf->allocated_mrs = &rf->allocated_cqs[BITS_TO_LONGS(max_cq)];
+ rf->allocated_pds = &rf->allocated_mrs[BITS_TO_LONGS(max_mr)];
+ rf->allocated_ahs = &rf->allocated_pds[BITS_TO_LONGS(num_pds)];
+ rf->allocated_mcgs = &rf->allocated_ahs[BITS_TO_LONGS(num_ahs)];
+ rf->allocated_arps = &rf->allocated_mcgs[BITS_TO_LONGS(num_mcg)];
+ rf->qp_table = (struct irdma_qp **)
+ (&rf->allocated_arps[BITS_TO_LONGS(arp_table_size)]);
+
+ set_bit(0, rf->allocated_mrs);
+ set_bit(0, rf->allocated_qps);
+ set_bit(0, rf->allocated_cqs);
+ set_bit(0, rf->allocated_pds);
+ set_bit(0, rf->allocated_arps);
+ set_bit(0, rf->allocated_ahs);
+ set_bit(0, rf->allocated_mcgs);
+ set_bit(2, rf->allocated_qps); /* qp 2 IEQ */
+ set_bit(1, rf->allocated_qps); /* qp 1 ILQ */
+ set_bit(1, rf->allocated_cqs);
+ set_bit(1, rf->allocated_pds);
+ set_bit(2, rf->allocated_cqs);
+ set_bit(2, rf->allocated_pds);
+
+ spin_lock_init(&rf->rsrc_lock);
+ spin_lock_init(&rf->arp_lock);
+ spin_lock_init(&rf->qptable_lock);
+ spin_lock_init(&rf->qh_list_lock);
+
+ INIT_LIST_HEAD(&rf->mc_qht_list.list);
+ /* stag index mask has a minimum of 14 bits */
+ mrdrvbits = 24 - max(get_count_order(rf->max_mr), 14);
+ rf->mr_stagmask = ~(((1 << mrdrvbits) - 1) << (32 - mrdrvbits));
+
+ return 0;
+}
+
+/**
+ * irdma_cqp_ce_handler - handle cqp completions
+ * @rf: RDMA PCI function
+ * @cq: cq for cqp completions
+ */
+void irdma_cqp_ce_handler(struct irdma_pci_f *rf, struct irdma_sc_cq *cq)
+{
+ struct irdma_cqp_request *cqp_request;
+ struct irdma_sc_dev *dev = &rf->sc_dev;
+ u32 cqe_count = 0;
+ struct irdma_ccq_cqe_info info;
+ unsigned long flags;
+ int ret;
+
+ do {
+ memset(&info, 0, sizeof(info));
+ spin_lock_irqsave(&rf->cqp.compl_lock, flags);
+ ret = dev->ccq_ops->ccq_get_cqe_info(cq, &info);
+ spin_unlock_irqrestore(&rf->cqp.compl_lock, flags);
+ if (ret)
+ break;
+
+ cqp_request = (struct irdma_cqp_request *)
+ (unsigned long)info.scratch;
+ if (info.error)
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: opcode = 0x%x maj_err_code = 0x%x min_err_code = 0x%x\n",
+ info.op_code, info.maj_err_code,
+ info.min_err_code);
+ if (cqp_request) {
+ cqp_request->compl_info.maj_err_code = info.maj_err_code;
+ cqp_request->compl_info.min_err_code = info.min_err_code;
+ cqp_request->compl_info.op_ret_val = info.op_ret_val;
+ cqp_request->compl_info.error = info.error;
+
+ if (cqp_request->waiting) {
+ cqp_request->request_done = true;
+ wake_up(&cqp_request->waitq);
+ irdma_put_cqp_request(&rf->cqp, cqp_request);
+ } else {
+ if (cqp_request->callback_fcn)
+ cqp_request->callback_fcn(cqp_request);
+ irdma_put_cqp_request(&rf->cqp, cqp_request);
+ }
+ }
+
+ cqe_count++;
+ } while (1);
+
+ if (cqe_count) {
+ irdma_process_bh(dev);
+ dev->ccq_ops->ccq_arm(cq);
+ }
+}
+
+/**
+ * cqp_thread - Handle cqp completions
+ * @context: Pointer to RDMA PCI Function
+ */
+int cqp_compl_thread(void *context)
+{
+ struct irdma_pci_f *rf = context;
+ struct irdma_sc_cq *cq = &rf->ccq.sc_cq;
+
+ do {
+ if (down_interruptible(&rf->cqp.cqp_compl_sem) ||
+ rf->stop_cqp_thread)
+ return 0;
+
+ irdma_cqp_ce_handler(rf, cq);
+ } while (!kthread_should_stop());
+
+ return 0;
+}
+
+/**
+ * irdma_next_iw_state - modify qp state
+ * @iwqp: iwarp qp to modify
+ * @state: next state for qp
+ * @del_hash: del hash
+ * @term: term message
+ * @termlen: length of term message
+ */
+void irdma_next_iw_state(struct irdma_qp *iwqp, u8 state, u8 del_hash, u8 term,
+ u8 termlen)
+{
+ struct irdma_modify_qp_info info = {};
+
+ info.next_iwarp_state = state;
+ info.remove_hash_idx = del_hash;
+ info.cq_num_valid = true;
+ info.arp_cache_idx_valid = true;
+ info.dont_send_term = true;
+ info.dont_send_fin = true;
+ info.termlen = termlen;
+
+ if (term & IRDMAQP_TERM_SEND_TERM_ONLY)
+ info.dont_send_term = false;
+ if (term & IRDMAQP_TERM_SEND_FIN_ONLY)
+ info.dont_send_fin = false;
+ if (iwqp->sc_qp.term_flags && state == IRDMA_QP_STATE_ERROR)
+ info.reset_tcp_conn = true;
+ iwqp->hw_iwarp_state = state;
+ irdma_hw_modify_qp(iwqp->iwdev, iwqp, &info, 0);
+}
+
+/**
+ * irdma_del_mac_entry - remove a mac entry from the hw table
+ * @rf: RDMA PCI function
+ * @idx: the index of the mac ip address to delete
+ */
+void irdma_del_local_mac_entry(struct irdma_pci_f *rf, u16 idx)
+{
+ struct irdma_cqp *iwcqp = &rf->cqp;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ enum irdma_status_code status = 0;
+
+ cqp_request = irdma_get_cqp_request(iwcqp, true);
+ if (!cqp_request) {
+ pr_err("cqp_request memory failed\n");
+ return;
+ }
+
+ cqp_info = &cqp_request->info;
+ cqp_info->cqp_cmd = IRDMA_OP_DELETE_LOCAL_MAC_ENTRY;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.del_local_mac_entry.cqp = &iwcqp->sc_cqp;
+ cqp_info->in.u.del_local_mac_entry.scratch = (uintptr_t)cqp_request;
+ cqp_info->in.u.del_local_mac_entry.entry_idx = idx;
+ cqp_info->in.u.del_local_mac_entry.ignore_ref_count = 0;
+ status = irdma_handle_cqp_op(rf, cqp_request);
+ if (status)
+ pr_err("CQP-OP Del MAC entry fail");
+}
+
+/**
+ * irdma_add_mac_entry - add a mac ip address entry to the hw table
+ * @rf: RDMA PCI function
+ * @mac_addr: pointer to mac address
+ * @idx: the index of the mac ip address to add
+ */
+int irdma_add_local_mac_entry(struct irdma_pci_f *rf, u8 *mac_addr, u16 idx)
+{
+ struct irdma_local_mac_entry_info *info;
+ struct irdma_cqp *iwcqp = &rf->cqp;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ enum irdma_status_code status = 0;
+
+ cqp_request = irdma_get_cqp_request(iwcqp, true);
+ if (!cqp_request) {
+ pr_err("cqp_request memory failed\n");
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ cqp_info = &cqp_request->info;
+ cqp_info->post_sq = 1;
+ info = &cqp_info->in.u.add_local_mac_entry.info;
+ ether_addr_copy(info->mac_addr, mac_addr);
+ info->entry_idx = idx;
+ cqp_info->in.u.add_local_mac_entry.scratch = (uintptr_t)cqp_request;
+ cqp_info->cqp_cmd = IRDMA_OP_ADD_LOCAL_MAC_ENTRY;
+ cqp_info->in.u.add_local_mac_entry.cqp = &iwcqp->sc_cqp;
+ cqp_info->in.u.add_local_mac_entry.scratch = (uintptr_t)cqp_request;
+ status = irdma_handle_cqp_op(rf, cqp_request);
+ if (status)
+ pr_err("CQP-OP Add MAC entry fail");
+
+ return status;
+}
+
+/**
+ * irdma_alloc_local_mac_entry - allocate a mac entry
+ * @rf: RDMA PCI function
+ * @mac_tbl_idx: the index of the new mac address
+ *
+ * Allocate a mac address entry and update the mac_tbl_idx
+ * to hold the index of the newly created mac address
+ * Return 0 if successful, otherwise return error
+ */
+int irdma_alloc_local_mac_entry(struct irdma_pci_f *rf, u16 *mac_tbl_idx)
+{
+ struct irdma_cqp *iwcqp = &rf->cqp;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ enum irdma_status_code status = 0;
+
+ cqp_request = irdma_get_cqp_request(iwcqp, true);
+ if (!cqp_request) {
+ pr_err("cqp_request memory failed\n");
+ return IRDMA_ERR_NO_MEMORY;
+ }
+
+ /* increment refcount, because we need the cqp request ret value */
+ atomic_inc(&cqp_request->refcount);
+ cqp_info = &cqp_request->info;
+ cqp_info->cqp_cmd = IRDMA_OP_ALLOC_LOCAL_MAC_ENTRY;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.alloc_local_mac_entry.cqp = &iwcqp->sc_cqp;
+ cqp_info->in.u.alloc_local_mac_entry.scratch = (uintptr_t)cqp_request;
+ status = irdma_handle_cqp_op(rf, cqp_request);
+ if (!status)
+ *mac_tbl_idx = (u16)cqp_request->compl_info.op_ret_val;
+ else
+ pr_err("CQP-OP Alloc MAC entry fail");
+ /* decrement refcount and free the cqp request, if no longer used */
+ irdma_put_cqp_request(iwcqp, cqp_request);
+
+ return status;
+}
+
+/**
+ * irdma_cqp_manage_apbvt_cmd - send cqp command manage apbvt
+ * @iwdev: iwarp device
+ * @accel_local_port: port for apbvt
+ * @add_port: add ordelete port
+ */
+static enum irdma_status_code
+irdma_cqp_manage_apbvt_cmd(struct irdma_device *iwdev, u16 accel_local_port,
+ bool add_port)
+{
+ struct irdma_apbvt_info *info;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ enum irdma_status_code status;
+
+ cqp_request = irdma_get_cqp_request(&iwdev->rf->cqp, add_port);
+ if (!cqp_request)
+ return IRDMA_ERR_NO_MEMORY;
+
+ cqp_info = &cqp_request->info;
+ info = &cqp_info->in.u.manage_apbvt_entry.info;
+ memset(info, 0, sizeof(*info));
+ info->add = add_port;
+ info->port = accel_local_port;
+ cqp_info->cqp_cmd = IRDMA_OP_MANAGE_APBVT_ENTRY;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.manage_apbvt_entry.cqp = &iwdev->rf->cqp.sc_cqp;
+ cqp_info->in.u.manage_apbvt_entry.scratch = (uintptr_t)cqp_request;
+ status = irdma_handle_cqp_op(iwdev->rf, cqp_request);
+ if (status)
+ dev_dbg(rfdev_to_dev(&iwdev->rf->sc_dev),
+ "ERR: CQP-OP Manage APBVT entry fail");
+
+ return status;
+}
+
+/**
+ * irdma_manage_apbvt - add or delete tcp port
+ * @iwdev: iwarp device
+ * @accel_local_port: port for apbvt
+ * @add_port: add or delete port
+ */
+enum irdma_status_code irdma_manage_apbvt(struct irdma_device *iwdev,
+ u16 accel_local_port, bool add_port)
+{
+ struct irdma_cm_core *cm_core = &iwdev->cm_core;
+ enum irdma_status_code status = 0;
+ unsigned long flags;
+ bool in_use;
+
+ /* apbvt_lock is held across CQP delete APBVT OP (non-waiting) to
+ * protect against race where add APBVT CQP can race ahead of the delete
+ * APBVT for same port.
+ */
+ if (add_port) {
+ spin_lock_irqsave(&cm_core->apbvt_lock, flags);
+ in_use = __test_and_set_bit(accel_local_port,
+ cm_core->ports_in_use);
+ spin_unlock_irqrestore(&cm_core->apbvt_lock, flags);
+ if (in_use)
+ return 0;
+ return irdma_cqp_manage_apbvt_cmd(iwdev, accel_local_port,
+ true);
+ } else {
+ spin_lock_irqsave(&cm_core->apbvt_lock, flags);
+ in_use = irdma_port_in_use(cm_core, accel_local_port);
+ if (in_use) {
+ spin_unlock_irqrestore(&cm_core->apbvt_lock, flags);
+ return 0;
+ }
+ __clear_bit(accel_local_port, cm_core->ports_in_use);
+ status = irdma_cqp_manage_apbvt_cmd(iwdev, accel_local_port,
+ false);
+ spin_unlock_irqrestore(&cm_core->apbvt_lock, flags);
+ return status;
+ }
+}
+
+/**
+ * irdma_manage_arp_cache - manage hw arp cache
+ * @rf: RDMA PCI function
+ * @mac_addr: mac address ptr
+ * @ip_addr: ip addr for arp cache
+ * @ipv4: flag inicating IPv4
+ * @action: add, delete or modify
+ */
+void irdma_manage_arp_cache(struct irdma_pci_f *rf, unsigned char *mac_addr,
+ u32 *ip_addr, bool ipv4, u32 action)
+{
+ struct irdma_add_arp_cache_entry_info *info;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ int arp_index;
+
+ arp_index = irdma_arp_table(rf, ip_addr, ipv4, mac_addr, action);
+ if (arp_index == -1)
+ return;
+
+ cqp_request = irdma_get_cqp_request(&rf->cqp, false);
+ if (!cqp_request)
+ return;
+
+ cqp_info = &cqp_request->info;
+ if (action == IRDMA_ARP_ADD) {
+ cqp_info->cqp_cmd = IRDMA_OP_ADD_ARP_CACHE_ENTRY;
+ info = &cqp_info->in.u.add_arp_cache_entry.info;
+ memset(info, 0, sizeof(*info));
+ info->arp_index = (u16)arp_index;
+ info->permanent = true;
+ ether_addr_copy(info->mac_addr, mac_addr);
+ cqp_info->in.u.add_arp_cache_entry.scratch =
+ (uintptr_t)cqp_request;
+ cqp_info->in.u.add_arp_cache_entry.cqp = &rf->cqp.sc_cqp;
+ } else {
+ cqp_info->cqp_cmd = IRDMA_OP_DELETE_ARP_CACHE_ENTRY;
+ cqp_info->in.u.del_arp_cache_entry.scratch =
+ (uintptr_t)cqp_request;
+ cqp_info->in.u.del_arp_cache_entry.cqp = &rf->cqp.sc_cqp;
+ cqp_info->in.u.del_arp_cache_entry.arp_index = arp_index;
+ }
+
+ cqp_info->in.u.add_arp_cache_entry.cqp = &rf->cqp.sc_cqp;
+ cqp_info->in.u.add_arp_cache_entry.scratch = (uintptr_t)cqp_request;
+ cqp_info->post_sq = 1;
+ if (irdma_handle_cqp_op(rf, cqp_request))
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: CQP-OP Add/Del Arp Cache entry fail");
+}
+
+/**
+ * irdma_send_syn_cqp_callback - do syn/ack after qhash
+ * @cqp_request: qhash cqp completion
+ */
+static void irdma_send_syn_cqp_callback(struct irdma_cqp_request *cqp_request)
+{
+ irdma_send_syn(cqp_request->param, 1);
+}
+
+/**
+ * irdma_manage_qhash - add or modify qhash
+ * @iwdev: iwarp device
+ * @cminfo: cm info for qhash
+ * @etype: type (syn or quad)
+ * @mtype: type of qhash
+ * @cmnode: cmnode associated with connection
+ * @wait: wait for completion
+ */
+enum irdma_status_code
+irdma_manage_qhash(struct irdma_device *iwdev, struct irdma_cm_info *cminfo,
+ enum irdma_quad_entry_type etype,
+ enum irdma_quad_hash_manage_type mtype, void *cmnode,
+ bool wait)
+{
+ struct irdma_qhash_table_info *info;
+ struct irdma_sc_dev *dev = &iwdev->rf->sc_dev;
+ enum irdma_status_code status;
+ struct irdma_cqp *iwcqp = &iwdev->rf->cqp;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+
+ cqp_request = irdma_get_cqp_request(iwcqp, wait);
+ if (!cqp_request)
+ return IRDMA_ERR_NO_MEMORY;
+
+ cqp_info = &cqp_request->info;
+ info = &cqp_info->in.u.manage_qhash_table_entry.info;
+ memset(info, 0, sizeof(*info));
+ info->vsi = &iwdev->vsi;
+ info->manage = mtype;
+ info->entry_type = etype;
+ if (cminfo->vlan_id < VLAN_N_VID) {
+ info->vlan_valid = true;
+ info->vlan_id = cminfo->vlan_id;
+ } else {
+ info->vlan_valid = false;
+ }
+ info->ipv4_valid = cminfo->ipv4;
+ info->user_pri = cminfo->user_pri;
+ ether_addr_copy(info->mac_addr, iwdev->netdev->dev_addr);
+ info->qp_num = cminfo->qh_qpid;
+ info->dest_port = cminfo->loc_port;
+ info->dest_ip[0] = cminfo->loc_addr[0];
+ info->dest_ip[1] = cminfo->loc_addr[1];
+ info->dest_ip[2] = cminfo->loc_addr[2];
+ info->dest_ip[3] = cminfo->loc_addr[3];
+ if (etype == IRDMA_QHASH_TYPE_TCP_ESTABLISHED ||
+ etype == IRDMA_QHASH_TYPE_UDP_UNICAST ||
+ etype == IRDMA_QHASH_TYPE_UDP_MCAST ||
+ etype == IRDMA_QHASH_TYPE_ROCE_MCAST ||
+ etype == IRDMA_QHASH_TYPE_ROCEV2_HW) {
+ info->src_port = cminfo->rem_port;
+ info->src_ip[0] = cminfo->rem_addr[0];
+ info->src_ip[1] = cminfo->rem_addr[1];
+ info->src_ip[2] = cminfo->rem_addr[2];
+ info->src_ip[3] = cminfo->rem_addr[3];
+ }
+ if (cmnode) {
+ cqp_request->callback_fcn = irdma_send_syn_cqp_callback;
+ cqp_request->param = cmnode;
+ }
+ if (info->ipv4_valid)
+ dev_dbg(rfdev_to_dev(dev),
+ "CM: %s IP=%pI4, port=%d, mac=%pM, vlan_id=%d\n",
+ !mtype ? "DELETE" : "ADD", info->dest_ip,
+ info->dest_port, info->mac_addr, cminfo->vlan_id);
+ else
+ dev_dbg(rfdev_to_dev(dev),
+ "CM: %s IP=%pI6, port=%d, mac=%pM, vlan_id=%d\n",
+ !mtype ? "DELETE" : "ADD", info->dest_ip,
+ info->dest_port, info->mac_addr, cminfo->vlan_id);
+ cqp_info->in.u.manage_qhash_table_entry.cqp = &iwdev->rf->cqp.sc_cqp;
+ cqp_info->in.u.manage_qhash_table_entry.scratch = (uintptr_t)cqp_request;
+ cqp_info->cqp_cmd = IRDMA_OP_MANAGE_QHASH_TABLE_ENTRY;
+ cqp_info->post_sq = 1;
+ status = irdma_handle_cqp_op(iwdev->rf, cqp_request);
+ if (status)
+ dev_dbg(rfdev_to_dev(dev),
+ "ERR: CQP-OP Manage Qhash Entry fail");
+
+ return status;
+}
+
+/**
+ * irdma_hw_flush_wqes_callback - Check return code after flush
+ * @cqp_request: qhash cqp completion
+ */
+static void irdma_hw_flush_wqes_callback(struct irdma_cqp_request *cqp_request)
+{
+ struct irdma_qp_flush_info *hw_info;
+ struct irdma_sc_qp *qp;
+ struct irdma_qp *iwqp;
+ struct cqp_cmds_info *cqp_info;
+
+ cqp_info = &cqp_request->info;
+ hw_info = &cqp_request->info.in.u.qp_flush_wqes.info;
+ qp = cqp_info->in.u.qp_flush_wqes.qp;
+ iwqp = qp->qp_uk.back_qp;
+
+ if (cqp_request->compl_info.maj_err_code)
+ return;
+ if (hw_info->rq &&
+ (cqp_request->compl_info.min_err_code == IRDMA_CQP_COMPL_SQ_WQE_FLUSHED ||
+ cqp_request->compl_info.min_err_code == 0)) {
+ /* RQ WQE flush was requested but did not happen */
+ qp->qp_uk.rq_flush_complete = true;
+ complete(&iwqp->rq_drained);
+ }
+ if (hw_info->sq &&
+ (cqp_request->compl_info.min_err_code == IRDMA_CQP_COMPL_RQ_WQE_FLUSHED ||
+ cqp_request->compl_info.min_err_code == 0)) {
+ qp->qp_uk.sq_flush_complete = true;
+ complete(&iwqp->sq_drained);
+ }
+}
+
+/**
+ * irdma_hw_flush_wqes - flush qp's wqe
+ * @rf: RDMA PCI function
+ * @qp: hardware control qp
+ * @info: info for flush
+ * @wait: flag wait for completion
+ */
+enum irdma_status_code irdma_hw_flush_wqes(struct irdma_pci_f *rf,
+ struct irdma_sc_qp *qp,
+ struct irdma_qp_flush_info *info,
+ bool wait)
+{
+ enum irdma_status_code status;
+ struct irdma_qp_flush_info *hw_info;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+ struct irdma_qp *iwqp = qp->qp_uk.back_qp;
+ unsigned long flags = 0;
+
+ cqp_request = irdma_get_cqp_request(&rf->cqp, wait);
+ if (!cqp_request)
+ return IRDMA_ERR_NO_MEMORY;
+
+ cqp_info = &cqp_request->info;
+ if (!wait)
+ cqp_request->callback_fcn = irdma_hw_flush_wqes_callback;
+ hw_info = &cqp_request->info.in.u.qp_flush_wqes.info;
+ memcpy(hw_info, info, sizeof(*hw_info));
+ cqp_info->cqp_cmd = IRDMA_OP_QP_FLUSH_WQES;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.qp_flush_wqes.qp = qp;
+ cqp_info->in.u.qp_flush_wqes.scratch = (uintptr_t)cqp_request;
+ status = irdma_handle_cqp_op(rf, cqp_request);
+ if (status) {
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: CQP-OP Flush WQE's fail");
+ complete(&iwqp->sq_drained);
+ complete(&iwqp->rq_drained);
+ qp->qp_uk.sq_flush_complete = true;
+ qp->qp_uk.rq_flush_complete = true;
+ return status;
+ }
+
+ if (!wait || cqp_request->compl_info.maj_err_code)
+ return 0;
+
+ if (info->rq) {
+ if (cqp_request->compl_info.min_err_code == IRDMA_CQP_COMPL_SQ_WQE_FLUSHED ||
+ cqp_request->compl_info.min_err_code == 0) {
+ /* RQ WQE flush was requested but did not happen */
+ qp->qp_uk.rq_flush_complete = true;
+ complete(&iwqp->rq_drained);
+ }
+ }
+ if (info->sq) {
+ if (cqp_request->compl_info.min_err_code == IRDMA_CQP_COMPL_RQ_WQE_FLUSHED ||
+ cqp_request->compl_info.min_err_code == 0) {
+ spin_lock_irqsave(&iwqp->lock, flags);
+ /*
+ * Handling case where WQE is posted to empty SQ when
+ * flush has not completed
+ */
+ if (IRDMA_RING_MORE_WORK(qp->qp_uk.sq_ring)) {
+ struct irdma_cqp_request *new_req;
+
+ qp->flush_sq = false;
+ spin_unlock_irqrestore(&iwqp->lock, flags);
+ info->rq = false;
+ new_req = irdma_get_cqp_request(&rf->cqp, true);
+ if (!new_req)
+ return IRDMA_ERR_NO_MEMORY;
+ cqp_info = &new_req->info;
+ hw_info = &new_req->info.in.u.qp_flush_wqes.info;
+ memcpy(hw_info, info, sizeof(*hw_info));
+ cqp_info->cqp_cmd = IRDMA_OP_QP_FLUSH_WQES;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.qp_flush_wqes.qp = qp;
+ cqp_info->in.u.qp_flush_wqes.scratch = (uintptr_t)new_req;
+
+ status = irdma_handle_cqp_op(rf, new_req);
+ if (new_req->compl_info.maj_err_code ||
+ new_req->compl_info.min_err_code != IRDMA_CQP_COMPL_SQ_WQE_FLUSHED ||
+ status) {
+ pr_warn("SQ in error but not flushed");
+ qp->qp_uk.sq_flush_complete = true;
+ }
+ } else {
+ /* SQ WQE flush was requested but did not happen */
+ qp->qp_uk.sq_flush_complete = true;
+ spin_unlock_irqrestore(&iwqp->lock, flags);
+ complete(&iwqp->sq_drained);
+ }
+ } else {
+ spin_lock_irqsave(&iwqp->lock, flags);
+ if (!IRDMA_RING_MORE_WORK(qp->qp_uk.sq_ring)) {
+ qp->qp_uk.sq_flush_complete = true;
+ spin_unlock_irqrestore(&iwqp->lock, flags);
+ complete(&iwqp->sq_drained);
+ } else {
+ spin_unlock_irqrestore(&iwqp->lock, flags);
+ }
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * irdma_gen_ae - generate AE
+ * @rf: RDMA PCI function
+ * @qp: qp associated with AE
+ * @info: info for ae
+ * @wait: wait for completion
+ */
+void irdma_gen_ae(struct irdma_pci_f *rf, struct irdma_sc_qp *qp,
+ struct irdma_gen_ae_info *info, bool wait)
+{
+ struct irdma_gen_ae_info *ae_info;
+ struct irdma_cqp_request *cqp_request;
+ struct cqp_cmds_info *cqp_info;
+
+ cqp_request = irdma_get_cqp_request(&rf->cqp, wait);
+ if (!cqp_request)
+ return;
+
+ cqp_info = &cqp_request->info;
+ ae_info = &cqp_request->info.in.u.gen_ae.info;
+ memcpy(ae_info, info, sizeof(*ae_info));
+ cqp_info->cqp_cmd = IRDMA_OP_GEN_AE;
+ cqp_info->post_sq = 1;
+ cqp_info->in.u.gen_ae.qp = qp;
+ cqp_info->in.u.gen_ae.scratch = (uintptr_t)cqp_request;
+ if (irdma_handle_cqp_op(rf, cqp_request))
+ dev_dbg(rfdev_to_dev(&rf->sc_dev),
+ "ERR: CQP OP failed attempting to generate ae_code=0x%x\n",
+ info->ae_code);
+}
+
+/**
+ * irdma_get_ib_wc - return change flush code to IB's
+ * @opcode: iwarp flush code
+ */
+static enum ib_wc_status irdma_get_ib_wc(enum irdma_flush_opcode opcode)
+{
+ switch (opcode) {
+ case FLUSH_PROT_ERR:
+ return IB_WC_LOC_PROT_ERR;
+ case FLUSH_REM_ACCESS_ERR:
+ return IB_WC_REM_ACCESS_ERR;
+ case FLUSH_LOC_QP_OP_ERR:
+ return IB_WC_LOC_QP_OP_ERR;
+ case FLUSH_REM_OP_ERR:
+ return IB_WC_REM_OP_ERR;
+ case FLUSH_LOC_LEN_ERR:
+ return IB_WC_LOC_LEN_ERR;
+ case FLUSH_GENERAL_ERR:
+ return IB_WC_GENERAL_ERR;
+ case FLUSH_FATAL_ERR:
+ default:
+ return IB_WC_FATAL_ERR;
+ }
+}
+
+/**
+ * irdma_set_flush_info - set flush info
+ * @pinfo: set flush info
+ * @min: minor err
+ * @maj: major err
+ * @opcode: flush error code
+ */
+static void irdma_set_flush_info(struct irdma_qp_flush_info *pinfo, u16 *min,
+ u16 *maj, enum irdma_flush_opcode opcode)
+{
+ *min = (u16)irdma_get_ib_wc(opcode);
+ *maj = CQE_MAJOR_DRV;
+ pinfo->userflushcode = true;
+}
+
+/**
+ * irdma_flush_wqes - flush wqe for qp
+ * @rf: RDMA PCI function
+ * @iwqp: qp to flush wqes
+ */
+void irdma_flush_wqes(struct irdma_pci_f *rf, struct irdma_qp *iwqp)
+{
+ struct irdma_qp_flush_info info = {};
+ struct irdma_sc_qp *qp = &iwqp->sc_qp;
+
+ info.sq = true;
+ info.rq = true;
+ if (qp->term_flags) {
+ irdma_set_flush_info(&info, &info.sq_minor_code,
+ &info.sq_major_code, qp->flush_code);
+ irdma_set_flush_info(&info, &info.rq_minor_code,
+ &info.rq_major_code, qp->flush_code);
+ }
+ (void)irdma_hw_flush_wqes(rf, &iwqp->sc_qp, &info, true);
+}
diff --git a/drivers/infiniband/hw/irdma/i40iw_hw.c b/drivers/infiniband/hw/irdma/i40iw_hw.c
new file mode 100644
index 000000000000..bdd445e1340a
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/i40iw_hw.c
@@ -0,0 +1,210 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "osdep.h"
+#include "type.h"
+#include "i40iw_hw.h"
+#include "status.h"
+#include "protos.h"
+
+#define I40E_CQPSQ_CQ_CQID_SHIFT 0
+#define I40E_CQPSQ_CQ_CQID_MASK \
+ (0xffffULL << I40E_CQPSQ_CQ_CQID_SHIFT)
+
+static u32 i40iw_regs[IRDMA_MAX_REGS] = {
+ I40E_PFPE_CQPTAIL,
+ I40E_PFPE_CQPDB,
+ I40E_PFPE_CCQPSTATUS,
+ I40E_PFPE_CCQPHIGH,
+ I40E_PFPE_CCQPLOW,
+ I40E_PFPE_CQARM,
+ I40E_PFPE_CQACK,
+ I40E_PFPE_AEQALLOC,
+ I40E_PFPE_CQPERRCODES,
+ I40E_PFPE_WQEALLOC,
+ I40E_PFINT_DYN_CTLN(0),
+ I40IW_DB_ADDR_OFFSET,
+
+ I40E_GLPCI_LBARCTRL,
+ I40E_GLPE_CPUSTATUS0,
+ I40E_GLPE_CPUSTATUS1,
+ I40E_GLPE_CPUSTATUS2,
+ I40E_PFINT_AEQCTL,
+ I40E_PFINT_CEQCTL(0),
+ I40E_VSIQF_CTL(0),
+ I40E_PFHMC_PDINV,
+ I40E_GLHMC_VFPDINV(0)
+};
+
+static u32 i40iw_stat_offsets_32[IRDMA_HW_STAT_INDEX_MAX_32] = {
+ I40E_GLPES_PFIP4RXDISCARD(0),
+ I40E_GLPES_PFIP4RXTRUNC(0),
+ I40E_GLPES_PFIP4TXNOROUTE(0),
+ I40E_GLPES_PFIP6RXDISCARD(0),
+ I40E_GLPES_PFIP6RXTRUNC(0),
+ I40E_GLPES_PFIP6TXNOROUTE(0),
+ I40E_GLPES_PFTCPRTXSEG(0),
+ I40E_GLPES_PFTCPRXOPTERR(0),
+ I40E_GLPES_PFTCPRXPROTOERR(0),
+ I40E_GLPES_PFRXVLANERR(0)
+};
+
+static u32 i40iw_stat_offsets_64[IRDMA_HW_STAT_INDEX_MAX_64] = {
+ I40E_GLPES_PFIP4RXOCTSLO(0),
+ I40E_GLPES_PFIP4RXPKTSLO(0),
+ I40E_GLPES_PFIP4RXFRAGSLO(0),
+ I40E_GLPES_PFIP4RXMCPKTSLO(0),
+ I40E_GLPES_PFIP4TXOCTSLO(0),
+ I40E_GLPES_PFIP4TXPKTSLO(0),
+ I40E_GLPES_PFIP4TXFRAGSLO(0),
+ I40E_GLPES_PFIP4TXMCPKTSLO(0),
+ I40E_GLPES_PFIP6RXOCTSLO(0),
+ I40E_GLPES_PFIP6RXPKTSLO(0),
+ I40E_GLPES_PFIP6RXFRAGSLO(0),
+ I40E_GLPES_PFIP6RXMCPKTSLO(0),
+ I40E_GLPES_PFIP6TXOCTSLO(0),
+ I40E_GLPES_PFIP6TXPKTSLO(0),
+ I40E_GLPES_PFIP6TXFRAGSLO(0),
+ I40E_GLPES_PFIP6TXMCPKTSLO(0),
+ I40E_GLPES_PFTCPRXSEGSLO(0),
+ I40E_GLPES_PFTCPTXSEGLO(0),
+ I40E_GLPES_PFRDMARXRDSLO(0),
+ I40E_GLPES_PFRDMARXSNDSLO(0),
+ I40E_GLPES_PFRDMARXWRSLO(0),
+ I40E_GLPES_PFRDMATXRDSLO(0),
+ I40E_GLPES_PFRDMATXSNDSLO(0),
+ I40E_GLPES_PFRDMATXWRSLO(0),
+ I40E_GLPES_PFRDMAVBNDLO(0),
+ I40E_GLPES_PFRDMAVINVLO(0),
+ I40E_GLPES_PFIP4RXMCOCTSLO(0),
+ I40E_GLPES_PFIP4TXMCOCTSLO(0),
+ I40E_GLPES_PFIP6RXMCOCTSLO(0),
+ I40E_GLPES_PFIP6TXMCOCTSLO(0),
+ I40E_GLPES_PFUDPRXPKTSLO(0),
+ I40E_GLPES_PFUDPTXPKTSLO(0)
+};
+
+static u64 i40iw_masks[IRDMA_MAX_MASKS] = {
+ I40E_PFPE_CCQPSTATUS_CCQP_DONE_MASK,
+ I40E_PFPE_CCQPSTATUS_CCQP_ERR_MASK,
+ I40E_CQPSQ_STAG_PDID_MASK,
+ I40E_CQPSQ_CQ_CEQID_MASK,
+ I40E_CQPSQ_CQ_CQID_MASK,
+};
+
+static u64 i40iw_shifts[IRDMA_MAX_SHIFTS] = {
+ I40E_PFPE_CCQPSTATUS_CCQP_DONE_SHIFT,
+ I40E_PFPE_CCQPSTATUS_CCQP_ERR_SHIFT,
+ I40E_CQPSQ_STAG_PDID_SHIFT,
+ I40E_CQPSQ_CQ_CEQID_SHIFT,
+ I40E_CQPSQ_CQ_CQID_SHIFT,
+};
+
+static struct irdma_irq_ops i40iw_irq_ops;
+
+/**
+ * i40iw_config_ceq- Configure CEQ interrupt
+ * @dev: pointer to the device structure
+ * @ceq_id: Completion Event Queue ID
+ * @idx: vector index
+ */
+static void i40iw_config_ceq(struct irdma_sc_dev *dev, u32 ceq_id, u32 idx)
+{
+ u32 reg_val;
+
+ reg_val = (ceq_id << I40E_PFINT_LNKLSTN_FIRSTQ_INDX_SHIFT);
+ reg_val |= (QUEUE_TYPE_CEQ << I40E_PFINT_LNKLSTN_FIRSTQ_TYPE_SHIFT);
+ wr32(dev->hw, I40E_PFINT_LNKLSTN(idx - 1), reg_val);
+
+ reg_val = (0x3 << I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT);
+ reg_val |= I40E_PFINT_DYN_CTLN_INTENA_MASK;
+ wr32(dev->hw, I40E_PFINT_DYN_CTLN(idx - 1), reg_val);
+
+ reg_val = (IRDMA_GLINT_CEQCTL_CAUSE_ENA_M |
+ (idx << IRDMA_GLINT_CEQCTL_MSIX_INDX_S) |
+ IRDMA_GLINT_CEQCTL_ITR_INDX_M);
+ reg_val |= (NULL_QUEUE_INDEX << I40E_PFINT_CEQCTL_NEXTQ_INDX_SHIFT);
+
+ wr32(dev->hw, dev->hw_regs[IRDMA_GLINT_CEQCTL] + 4 * ceq_id, reg_val);
+}
+
+/**
+ * i40iw_ena_irq - Enable interrupt
+ * @dev: pointer to the device structure
+ * @idx: vector index
+ */
+static void i40iw_ena_irq(struct irdma_sc_dev *dev, u32 idx)
+{
+ u32 val;
+
+ val = IRDMA_GLINT_DYN_CTL_INTENA_M | IRDMA_GLINT_DYN_CTL_CLEARPBA_M |
+ IRDMA_GLINT_DYN_CTL_ITR_INDX_M;
+ wr32(dev->hw, dev->hw_regs[IRDMA_GLINT_DYN_CTL] + 4 * (idx - 1), val);
+}
+
+/**
+ * irdma_disable_irq - Disable interrupt
+ * @dev: pointer to the device structure
+ * @idx: vector index
+ */
+static void i40iw_disable_irq(struct irdma_sc_dev *dev, u32 idx)
+{
+ wr32(dev->hw, dev->hw_regs[IRDMA_GLINT_DYN_CTL] + 4 * (idx - 1), 0);
+}
+
+void i40iw_init_hw(struct irdma_sc_dev *dev)
+{
+ int i;
+
+ for (i = 0; i < IRDMA_MAX_REGS; ++i)
+ dev->hw_regs[i] = i40iw_regs[i];
+
+ for (i = 0; i < IRDMA_HW_STAT_INDEX_MAX_32; ++i)
+ dev->hw_stats_regs_32[i] = i40iw_stat_offsets_32[i];
+
+ for (i = 0; i < IRDMA_HW_STAT_INDEX_MAX_64; ++i)
+ dev->hw_stats_regs_64[i] = i40iw_stat_offsets_64[i];
+
+ for (i = 0; i < IRDMA_MAX_SHIFTS; ++i)
+ dev->hw_shifts[i] = i40iw_shifts[i];
+
+ for (i = 0; i < IRDMA_MAX_MASKS; ++i)
+ dev->hw_masks[i] = i40iw_masks[i];
+
+ dev->wqe_alloc_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_WQEALLOC]);
+ dev->cq_arm_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQARM]);
+ dev->aeq_alloc_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_AEQALLOC]);
+ dev->cqp_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQPDB]);
+ dev->cq_ack_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQACK]);
+ dev->ceq_itr_mask_db = NULL;
+ dev->aeq_itr_mask_db = NULL;
+
+ memcpy(&i40iw_irq_ops, dev->irq_ops, sizeof(i40iw_irq_ops));
+ i40iw_irq_ops.irdma_en_irq = i40iw_ena_irq;
+ i40iw_irq_ops.irdma_dis_irq = i40iw_disable_irq;
+ i40iw_irq_ops.irdma_cfg_ceq = i40iw_config_ceq;
+ dev->irq_ops = &i40iw_irq_ops;
+
+ /* Setup the hardware limits, hmc may limit further */
+ dev->hw_attrs.uk_attrs.max_hw_wq_frags = I40IW_MAX_WQ_FRAGMENT_COUNT;
+ dev->hw_attrs.uk_attrs.max_hw_read_sges = I40IW_MAX_SGE_RD;
+ dev->hw_attrs.max_hw_device_pages = I40IW_MAX_PUSH_PAGE_COUNT;
+ dev->hw_attrs.first_hw_vf_fpm_id = I40IW_FIRST_VF_FPM_ID;
+ dev->hw_attrs.uk_attrs.max_hw_inline = I40IW_MAX_INLINE_DATA_SIZE;
+ dev->hw_attrs.max_hw_ird = I40IW_MAX_IRD_SIZE;
+ dev->hw_attrs.max_hw_ord = I40IW_MAX_ORD_SIZE;
+ dev->hw_attrs.max_hw_wqes = I40IW_MAX_WQ_ENTRIES;
+ dev->hw_attrs.uk_attrs.max_hw_rq_quanta = I40IW_QP_SW_MAX_RQ_QUANTA;
+ dev->hw_attrs.uk_attrs.max_hw_wq_quanta = I40IW_QP_SW_MAX_WQ_QUANTA;
+ dev->hw_attrs.uk_attrs.max_hw_sq_chunk = I40IW_MAX_QUANTA_PER_WR;
+ dev->hw_attrs.max_hw_pds = I40IW_MAX_PDS;
+ dev->hw_attrs.max_stat_inst = I40IW_MAX_STATS_COUNT;
+ dev->hw_attrs.max_hw_outbound_msg_size = I40IW_MAX_OUTBOUND_MSG_SIZE;
+ dev->hw_attrs.max_hw_inbound_msg_size = I40IW_MAX_INBOUND_MSG_SIZE;
+ dev->hw_attrs.max_qp_wr = I40IW_MAX_QP_WRS;
+}
diff --git a/drivers/infiniband/hw/irdma/i40iw_hw.h b/drivers/infiniband/hw/irdma/i40iw_hw.h
new file mode 100644
index 000000000000..cc0a23701724
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/i40iw_hw.h
@@ -0,0 +1,163 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef I40IW_HW_H
+#define I40IW_HW_H
+#define I40E_VFPE_CQPTAIL1 0x0000A000 /* Reset: VFR */
+#define I40E_VFPE_CQPDB1 0x0000BC00 /* Reset: VFR */
+#define I40E_VFPE_CCQPSTATUS1 0x0000B800 /* Reset: VFR */
+#define I40E_VFPE_CCQPHIGH1 0x00009800 /* Reset: VFR */
+#define I40E_VFPE_CCQPLOW1 0x0000AC00 /* Reset: VFR */
+#define I40E_VFPE_CQARM1 0x0000B400 /* Reset: VFR */
+#define I40E_VFPE_CQACK1 0x0000B000 /* Reset: VFR */
+#define I40E_VFPE_AEQALLOC1 0x0000A400 /* Reset: VFR */
+#define I40E_VFPE_CQPERRCODES1 0x00009C00 /* Reset: VFR */
+#define I40E_VFPE_WQEALLOC1 0x0000C000 /* Reset: VFR */
+#define I40E_VFINT_DYN_CTLN(_INTVF) (0x00024800 + ((_INTVF) * 4)) /* _i=0...511 */ /* Reset: VFR */
+
+#define I40E_PFPE_CQPTAIL 0x00008080 /* Reset: PFR */
+
+#define I40E_PFPE_CQPDB 0x00008000 /* Reset: PFR */
+#define I40E_PFPE_CCQPSTATUS 0x00008100 /* Reset: PFR */
+#define I40E_PFPE_CCQPHIGH 0x00008200 /* Reset: PFR */
+#define I40E_PFPE_CCQPLOW 0x00008180 /* Reset: PFR */
+#define I40E_PFPE_CQARM 0x00131080 /* Reset: PFR */
+#define I40E_PFPE_CQACK 0x00131100 /* Reset: PFR */
+#define I40E_PFPE_AEQALLOC 0x00131180 /* Reset: PFR */
+#define I40E_PFPE_CQPERRCODES 0x00008880 /* Reset: PFR */
+#define I40E_PFPE_WQEALLOC 0x00138C00 /* Reset: PFR */
+#define I40E_GLPCI_LBARCTRL 0x000BE484 /* Reset: POR */
+#define I40E_GLPE_CPUSTATUS0 0x0000D040 /* Reset: PE_CORER */
+#define I40E_GLPE_CPUSTATUS1 0x0000D044 /* Reset: PE_CORER */
+#define I40E_GLPE_CPUSTATUS2 0x0000D048 /* Reset: PE_CORER */
+#define I40E_PFHMC_PDINV 0x000C0300 /* Reset: PFR */
+#define I40E_GLHMC_VFPDINV(_i) (0x000C8300 + ((_i) * 4)) /* _i=0...31 */ /* Reset: CORER */
+#define I40E_PFINT_DYN_CTLN(_INTPF) (0x00034800 + ((_INTPF) * 4)) /* _i=0...511 */ /* Reset: PFR */
+#define I40E_PFINT_AEQCTL 0x00038700 /* Reset: CORER */
+
+#define I40E_GLPES_PFIP4RXDISCARD(_i) (0x00010600 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXTRUNC(_i) (0x00010700 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXNOROUTE(_i) (0x00012E00 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXDISCARD(_i) (0x00011200 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXTRUNC(_i) (0x00011300 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+
+#define I40E_GLPES_PFRDMAVBNDLO(_i) (0x00014800 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXMCOCTSLO(_i) (0x00012000 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXMCOCTSLO(_i) (0x00011600 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6TXMCOCTSLO(_i) (0x00012A00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFUDPRXPKTSLO(_i) (0x00013800 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFUDPTXPKTSLO(_i) (0x00013A00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+
+#define I40E_GLPES_PFIP6TXNOROUTE(_i) (0x00012F00 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFTCPRTXSEG(_i) (0x00013600 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFTCPRXOPTERR(_i) (0x00013200 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFTCPRXPROTOERR(_i) (0x00013300 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRXVLANERR(_i) (0x00010000 + ((_i) * 4)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXOCTSLO(_i) (0x00010200 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXPKTSLO(_i) (0x00010400 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXFRAGSLO(_i) (0x00010800 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXMCPKTSLO(_i) (0x00010C00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXOCTSLO(_i) (0x00011A00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXPKTSLO(_i) (0x00011C00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXFRAGSLO(_i) (0x00011E00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4TXMCPKTSLO(_i) (0x00012200 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXOCTSLO(_i) (0x00010E00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXPKTSLO(_i) (0x00011000 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXFRAGSLO(_i) (0x00011400 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6TXOCTSLO(_i) (0x00012400 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6TXPKTSLO(_i) (0x00012600 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6TXFRAGSLO(_i) (0x00012800 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6TXMCPKTSLO(_i) (0x00012C00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFTCPTXSEGLO(_i) (0x00013400 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMARXRDSLO(_i) (0x00013E00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMARXSNDSLO(_i) (0x00014000 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMARXWRSLO(_i) (0x00013C00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMATXRDSLO(_i) (0x00014400 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMATXSNDSLO(_i) (0x00014600 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMATXWRSLO(_i) (0x00014200 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP4RXMCOCTSLO(_i) (0x00010A00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFIP6RXMCPKTSLO(_i) (0x00011800 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFTCPRXSEGSLO(_i) (0x00013000 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+#define I40E_GLPES_PFRDMAVINVLO(_i) (0x00014A00 + ((_i) * 8)) /* _i=0...15 */ /* Reset: PE_CORER */
+
+#define I40IW_DB_ADDR_OFFSET (4 * 1024 * 1024 - 64 * 1024)
+
+#define I40IW_VF_DB_ADDR_OFFSET (64 * 1024)
+
+#define I40E_PFINT_LNKLSTN(_INTPF) (0x00035000 + ((_INTPF) * 4)) /* _i=0...511 */ /* Reset: PFR */
+#define I40E_PFINT_LNKLSTN_MAX_INDEX 511
+#define I40E_PFINT_LNKLSTN_FIRSTQ_INDX_SHIFT 0
+#define I40E_PFINT_LNKLSTN_FIRSTQ_INDX_MASK I40E_MASK(0x7FF, I40E_PFINT_LNKLSTN_FIRSTQ_INDX_SHIFT)
+#define I40E_PFINT_LNKLSTN_FIRSTQ_TYPE_SHIFT 11
+#define I40E_PFINT_LNKLSTN_FIRSTQ_TYPE_MASK I40E_MASK(0x3, I40E_PFINT_LNKLSTN_FIRSTQ_TYPE_SHIFT)
+
+#define I40E_PFINT_CEQCTL(_INTPF) (0x00036800 + ((_INTPF) * 4)) /* _i=0...511 */ /* Reset: CORER */
+#define I40E_PFINT_CEQCTL_MAX_INDEX 511
+#define I40E_PFINT_CEQCTL_MSIX_INDX_SHIFT 0
+#define I40E_PFINT_CEQCTL_MSIX_INDX_MASK I40E_MASK(0xFF, I40E_PFINT_CEQCTL_MSIX_INDX_SHIFT)
+#define I40E_PFINT_CEQCTL_ITR_INDX_SHIFT 11
+#define I40E_PFINT_CEQCTL_ITR_INDX_MASK I40E_MASK(0x3, I40E_PFINT_CEQCTL_ITR_INDX_SHIFT)
+#define I40E_PFINT_CEQCTL_MSIX0_INDX_SHIFT 13
+#define I40E_PFINT_CEQCTL_MSIX0_INDX_MASK I40E_MASK(0x7, I40E_PFINT_CEQCTL_MSIX0_INDX_SHIFT)
+#define I40E_PFINT_CEQCTL_NEXTQ_INDX_SHIFT 16
+#define I40E_PFINT_CEQCTL_NEXTQ_INDX_MASK I40E_MASK(0x7FF, I40E_PFINT_CEQCTL_NEXTQ_INDX_SHIFT)
+#define I40E_PFINT_CEQCTL_NEXTQ_TYPE_SHIFT 27
+#define I40E_PFINT_CEQCTL_NEXTQ_TYPE_MASK I40E_MASK(0x3, I40E_PFINT_CEQCTL_NEXTQ_TYPE_SHIFT)
+#define I40E_PFINT_CEQCTL_CAUSE_ENA_SHIFT 30
+#define I40E_PFINT_CEQCTL_CAUSE_ENA_MASK I40E_MASK(0x1, I40E_PFINT_CEQCTL_CAUSE_ENA_SHIFT)
+#define I40E_PFINT_CEQCTL_INTEVENT_SHIFT 31
+#define I40E_PFINT_CEQCTL_INTEVENT_MASK I40E_MASK(0x1, I40E_PFINT_CEQCTL_INTEVENT_SHIFT)
+
+#define I40E_CQPSQ_STAG_PDID_SHIFT 48
+#define I40E_CQPSQ_STAG_PDID_MASK MAKEMASK(0x7FFFULL, I40E_CQPSQ_STAG_PDID_SHIFT)
+
+#define I40E_PFPE_CCQPSTATUS_CCQP_DONE_SHIFT 0
+#define I40E_PFPE_CCQPSTATUS_CCQP_DONE_MASK (0x1ULL << I40E_PFPE_CCQPSTATUS_CCQP_DONE_SHIFT)
+
+#define I40E_PFPE_CCQPSTATUS_CCQP_ERR_SHIFT 31
+#define I40E_PFPE_CCQPSTATUS_CCQP_ERR_MASK (0x1ULL << I40E_PFPE_CCQPSTATUS_CCQP_ERR_SHIFT)
+
+#define I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT 3
+#define I40E_PFINT_DYN_CTLN_ITR_INDX_MASK (0x3 << I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT)
+
+#define I40E_PFINT_DYN_CTLN_INTENA_SHIFT 0
+#define I40E_PFINT_DYN_CTLN_INTENA_MASK (0x1 << I40E_PFINT_DYN_CTLN_INTENA_SHIFT)
+
+#define I40E_CQPSQ_CQ_CEQID_SHIFT 24
+#define I40E_CQPSQ_CQ_CEQID_MASK (0x7fUL << I40E_CQPSQ_CQ_CEQID_SHIFT)
+
+#define I40E_VSIQF_CTL(_VSI) (0x0020D800 + ((_VSI) * 4))
+
+enum i40iw_device_caps_const {
+ I40IW_MAX_WQ_FRAGMENT_COUNT = 3,
+ I40IW_MAX_SGE_RD = 1,
+ I40IW_MAX_PUSH_PAGE_COUNT = 0,
+ I40IW_MAX_INLINE_DATA_SIZE = 48,
+ I40IW_MAX_IRD_SIZE = 63,
+ I40IW_MAX_ORD_SIZE = 127,
+ I40IW_MAX_WQ_ENTRIES = 2048,
+ I40IW_MAX_WQE_SIZE_RQ = 128,
+ I40IW_MAX_PDS = 32768,
+ I40IW_MAX_STATS_COUNT = 16,
+ I40IW_MAX_CQ_SIZE = 1048575,
+ I40IW_MAX_OUTBOUND_MSG_SIZE = 2147483647,
+ I40IW_MAX_INBOUND_MSG_SIZE = 2147483647,
+};
+
+#define I40IW_QP_WQE_MIN_SIZE 32
+#define I40IW_QP_WQE_MAX_SIZE 128
+#define I40IW_QP_SW_MIN_WQSIZE 4
+
+#define I40IW_MAX_RQ_WQE_SHIFT 2
+#define I40IW_MAX_QUANTA_PER_WR 2
+
+#define I40IW_QP_SW_MAX_SQ_QUANTA 2048
+#define I40IW_QP_SW_MAX_RQ_QUANTA 16384
+#define I40IW_QP_SW_MAX_WQ_QUANTA 2048
+#define I40IW_MAX_QP_WRS ((I40IW_QP_SW_MAX_SQ_QUANTA - IRDMA_SQ_RSVD) / I40IW_MAX_QUANTA_PER_WR)
+#define I40IW_FIRST_VF_FPM_ID 16
+#define QUEUE_TYPE_CEQ 2
+#define NULL_QUEUE_INDEX 0x7FF
+
+void i40iw_init_hw(struct irdma_sc_dev *dev);
+#endif /* I40IW_HW_H */
diff --git a/drivers/infiniband/hw/irdma/icrdma_hw.c b/drivers/infiniband/hw/irdma/icrdma_hw.c
new file mode 100644
index 000000000000..9bf5cd2ac789
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/icrdma_hw.c
@@ -0,0 +1,75 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2019, Intel Corporation. */
+
+#include "osdep.h"
+#include "type.h"
+#include "icrdma_hw.h"
+
+static u32 icrdma_regs[IRDMA_MAX_REGS] = {
+ PFPE_CQPTAIL,
+ PFPE_CQPDB,
+ PFPE_CCQPSTATUS,
+ PFPE_CCQPHIGH,
+ PFPE_CCQPLOW,
+ PFPE_CQARM,
+ PFPE_CQACK,
+ PFPE_AEQALLOC,
+ PFPE_CQPERRCODES,
+ PFPE_WQEALLOC,
+ GLINT_DYN_CTL(0),
+ ICRDMA_DB_ADDR_OFFSET,
+
+ GLPCI_LBARCTRL,
+ GLPE_CPUSTATUS0,
+ GLPE_CPUSTATUS1,
+ GLPE_CPUSTATUS2,
+ PFINT_AEQCTL,
+ GLINT_CEQCTL(0),
+ VSIQF_PE_CTL1(0),
+ PFHMC_PDINV,
+ GLHMC_VFPDINV(0)
+};
+
+static u64 icrdma_masks[IRDMA_MAX_MASKS] = {
+ ICRDMA_CCQPSTATUS_CCQP_DONE_M,
+ ICRDMA_CCQPSTATUS_CCQP_ERR_M,
+ ICRDMA_CQPSQ_STAG_PDID_M,
+ ICRDMA_CQPSQ_CQ_CEQID_M,
+ ICRDMA_CQPSQ_CQ_CQID_M,
+};
+
+static u64 icrdma_shifts[IRDMA_MAX_SHIFTS] = {
+ ICRDMA_CCQPSTATUS_CCQP_DONE_S,
+ ICRDMA_CCQPSTATUS_CCQP_ERR_S,
+ ICRDMA_CQPSQ_STAG_PDID_S,
+ ICRDMA_CQPSQ_CQ_CEQID_S,
+ ICRDMA_CQPSQ_CQ_CQID_S,
+};
+
+void icrdma_init_hw(struct irdma_sc_dev *dev)
+{
+ int i;
+
+ for (i = 0; i < IRDMA_MAX_REGS; ++i)
+ dev->hw_regs[i] = icrdma_regs[i];
+
+ for (i = 0; i < IRDMA_MAX_SHIFTS; ++i)
+ dev->hw_shifts[i] = icrdma_shifts[i];
+
+ for (i = 0; i < IRDMA_MAX_MASKS; ++i)
+ dev->hw_masks[i] = icrdma_masks[i];
+
+ dev->wqe_alloc_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_WQEALLOC]);
+ dev->cq_arm_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQARM]);
+ dev->aeq_alloc_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_AEQALLOC]);
+ dev->cqp_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQPDB]);
+ dev->cq_ack_db = (u32 __iomem *)(irdma_get_hw_addr(dev) +
+ dev->hw_regs[IRDMA_CQACK]);
+ dev->hw_attrs.max_stat_inst = ICRDMA_MAX_STATS_COUNT;
+
+ dev->hw_attrs.uk_attrs.max_hw_sq_chunk = IRDMA_MAX_QUANTA_PER_WR;
+}
diff --git a/drivers/infiniband/hw/irdma/icrdma_hw.h b/drivers/infiniband/hw/irdma/icrdma_hw.h
new file mode 100644
index 000000000000..02a6afa4e1b9
--- /dev/null
+++ b/drivers/infiniband/hw/irdma/icrdma_hw.h
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2019, Intel Corporation. */
+
+#ifndef ICRDMA_HW_H
+#define ICRDMA_HW_H
+
+#define VFPE_CQPTAIL1 0x0000a000
+#define VFPE_CQPDB1 0x0000bc00
+#define VFPE_CCQPSTATUS1 0x0000b800
+#define VFPE_CCQPHIGH1 0x00009800
+#define VFPE_CCQPLOW1 0x0000ac00
+#define VFPE_CQARM1 0x0000b400
+#define VFPE_CQARM1 0x0000b400
+#define VFPE_CQACK1 0x0000b000
+#define VFPE_AEQALLOC1 0x0000a400
+#define VFPE_CQPERRCODES1 0x00009c00
+#define VFPE_WQEALLOC1 0x0000c000
+#define VFINT_DYN_CTLN(_i) (0x00003800 + ((_i) * 4)) /* _i=0...63 */
+
+#define PFPE_CQPTAIL 0x00500880
+#define PFPE_CQPDB 0x00500800
+#define PFPE_CCQPSTATUS 0x0050a000
+#define PFPE_CCQPHIGH 0x0050a100
+#define PFPE_CCQPLOW 0x0050a080
+#define PFPE_CQARM 0x00502c00
+#define PFPE_CQACK 0x00502c80
+#define PFPE_AEQALLOC 0x00502d00
+#define GLINT_DYN_CTL(_INT) (0x00160000 + ((_INT) * 4)) /* _i=0...2047 */
+#define GLPCI_LBARCTRL 0x0009de74
+#define GLPE_CPUSTATUS0 0x0050ba5c
+#define GLPE_CPUSTATUS1 0x0050ba60
+#define GLPE_CPUSTATUS2 0x0050ba64
+#define PFINT_AEQCTL 0x0016cb00
+#define PFPE_CQPERRCODES 0x0050a200
+#define PFPE_WQEALLOC 0x00504400
+#define GLINT_CEQCTL(_INT) (0x0015c000 + ((_INT) * 4)) /* _i=0...2047 */
+#define VSIQF_PE_CTL1(_VSI) (0x00414000 + ((_VSI) * 4)) /* _i=0...767 */
+#define PFHMC_PDINV 0x00520300
+#define GLHMC_VFPDINV(_i) (0x00528300 + ((_i) * 4)) /* _i=0...31 */
+
+#define ICRDMA_DB_ADDR_OFFSET (8 * 1024 * 1024 - 64 * 1024)
+
+#define ICRDMA_VF_DB_ADDR_OFFSET (64 * 1024)
+
+/* CCQSTATUS */
+#define ICRDMA_CCQPSTATUS_CCQP_DONE_S 0
+#define ICRDMA_CCQPSTATUS_CCQP_DONE_M MAKEMASK(0x1ULL, ICRDMA_CCQPSTATUS_CCQP_DONE_S)
+#define ICRDMA_CCQPSTATUS_CCQP_ERR_S 31
+#define ICRDMA_CCQPSTATUS_CCQP_ERR_M MAKEMASK(0x1ULL, ICRDMA_CCQPSTATUS_CCQP_ERR_S)
+#define ICRDMA_CQPSQ_STAG_PDID_S 46
+#define ICRDMA_CQPSQ_STAG_PDID_M MAKEMASK(0x3ffffULL, ICRDMA_CQPSQ_STAG_PDID_S)
+#define ICRDMA_CQPSQ_CQ_CEQID_S 22
+#define ICRDMA_CQPSQ_CQ_CEQID_M (0x3ffULL << ICRDMA_CQPSQ_CQ_CEQID_S)
+#define ICRDMA_CQPSQ_CQ_CQID_S 0
+#define ICRDMA_CQPSQ_CQ_CQID_M \
+ (0x7ffffULL << ICRDMA_CQPSQ_CQ_CQID_S)
+
+enum icrdma_device_caps_const {
+ ICRDMA_MAX_STATS_COUNT = 128,
+};
+
+void icrdma_init_hw(struct irdma_sc_dev *dev);
+#endif /* ICRDMA_HW_H*/
--
2.21.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox