From mboxrd@z Thu Jan 1 00:00:00 1970 From: Leon Romanovsky Subject: Re: [PATCH rdma-next 1/2] IB/core: Add scatter end padding flags for WQ and QP Date: Wed, 25 Oct 2017 22:03:52 +0300 Message-ID: <20171025190352.GW16127@mtr-leonro.local> References: <20171017151857.11934-1-leon@kernel.org> <20171017151857.11934-2-leon@kernel.org> <1508954339.3325.45.camel@redhat.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="b9dEYEwnDXkv9lSy" Return-path: Content-Disposition: inline In-Reply-To: <1508954339.3325.45.camel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Doug Ledford Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Noa Osherovich List-Id: linux-rdma@vger.kernel.org --b9dEYEwnDXkv9lSy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wed, Oct 25, 2017 at 01:58:59PM -0400, Doug Ledford wrote: > On Tue, 2017-10-17 at 18:18 +0300, Leon Romanovsky wrote: > > From: Noa Osherovich > > > > There are root complexes that are able to optimize their > > performance when incoming data is multiple full cache lines. > > > > Scatter end padding is the device's ability to pad the ending of > > incoming packets (scatter) > > I hate this naming. I'm sure people inside of Mellanox have gotten > used to it, but this feature really has no bearing on scatter/gather at > all. This is merely final write padding. You might have a > scatter/gather list, you might have a single buffer. Either way, the > PCI root complex couldn't care less about scatter/gather or not, it's > all a byte stream to it. I would be much happier with a name that > reflected what this really does. Scatter has more broad meaning than "scatter/gather list". The name here came from statistical meaning of "scatter", I'll try to come into something more descriptive. > > > > to full cache line such that the last > > upstream write generated by an incoming packet will be a full cache > > line. > > > > Add a relevant entry to ib_device_cap_flags to report scatter end > > padding capability of an RDMA device. > > > > Add the QP and WQ create flags with an entry for scatter end padding: > > * A QP/WQ created with a scatter end padding flag will cause > > HW to pad the last upstream write generated by a packet to cache > > line. > > > > User should consider several factors before activating this feature: > > - In case of high CPU memory load (which may cause PCI back pressure > > in > > turn), if a large percent of the writes are partial cache line, > > this > > feature should be checked as an optional solution. > > - This feature might reduce performance if most packets are between > > one > > and two cache lines and PCIe throughput has reached its maximum > > capacity. E.g. 65B packet from the network port will lead to 128B > > write on PCIe, which may cause traffic on PCIe to reach high > > throughput. > > > > Signed-off-by: Noa Osherovich > > Reviewed-by: Majd Dibbiny > > Signed-off-by: Leon Romanovsky > > --- > > drivers/infiniband/core/uverbs_cmd.c | 3 ++- > > include/rdma/ib_verbs.h | 4 ++++ > > 2 files changed, 6 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/infiniband/core/uverbs_cmd.c > > b/drivers/infiniband/core/uverbs_cmd.c > > index d31e4bc58e9a..ab29a0327831 100644 > > --- a/drivers/infiniband/core/uverbs_cmd.c > > +++ b/drivers/infiniband/core/uverbs_cmd.c > > @@ -1491,7 +1491,8 @@ static int create_qp(struct ib_uverbs_file > > *file, > > IB_QP_CREATE_MANAGED_RECV | > > IB_QP_CREATE_SCATTER_FCS | > > IB_QP_CREATE_CVLAN_STRIPPING | > > - IB_QP_CREATE_SOURCE_QPN)) { > > + IB_QP_CREATE_SOURCE_QPN | > > + IB_QP_CREATE_SCATTER_END_PADDING)) { > > Maybe IB_QP_CREATE_PCI_WRITE_PAD? > > > ret = -EINVAL; > > goto err_put; > > } > > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h > > index 9810e4568635..4c0a539cd2a2 100644 > > --- a/include/rdma/ib_verbs.h > > +++ b/include/rdma/ib_verbs.h > > @@ -229,6 +229,8 @@ enum ib_device_cap_flags { > > /* Deprecated. Please use IB_RAW_PACKET_CAP_SCATTER_FCS. */ > > IB_DEVICE_RAW_SCATTER_FCS = (1ULL << 34), > > IB_DEVICE_RDMA_NETDEV_OPA_VNIC = (1ULL << > > 35), > > + /* The device supports padding incoming writes to cacheline. > > */ > /* The device supports padding the final write of a PCI write > transaction to a cacheline boundry so that the PCI root complex can > optimize its memory accesses */ > > + IB_DEVICE_SCATTER_END_PADDING = (1ULL << 36), > > }; > > > > enum ib_signature_prot_cap { > > @@ -1098,6 +1100,7 @@ enum ib_qp_create_flags { > > IB_QP_CREATE_SCATTER_FCS = 1 << 8, > > IB_QP_CREATE_CVLAN_STRIPPING = 1 << 9, > > IB_QP_CREATE_SOURCE_QPN = 1 << 10, > > + IB_QP_CREATE_SCATTER_END_PADDING = 1 << 11, > > /* reserve bits 26-31 for low level drivers' internal use */ > > IB_QP_CREATE_RESERVED_START = 1 << 26, > > IB_QP_CREATE_RESERVED_END = 1 << 31, > > @@ -1621,6 +1624,7 @@ enum ib_wq_flags { > > IB_WQ_FLAGS_CVLAN_STRIPPING = 1 << 0, > > IB_WQ_FLAGS_SCATTER_FCS = 1 << 1, > > IB_WQ_FLAGS_DELAY_DROP = 1 << 2, > > + IB_WQ_FLAGS_SCATTER_END_PADDING = 1 << 3, > > }; > > > > struct ib_wq_init_attr { > -- > Doug Ledford > GPG KeyID: B826A3330E572FDD > Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD > --b9dEYEwnDXkv9lSy Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEkhr/r4Op1/04yqaB5GN7iDZyWKcFAlnw4BgACgkQ5GN7iDZy WKeASQ//V2ozSGG50mKjKDPePN+uQKdK7Fn6Pyr9wRIlJyPpomUFA2pjuMp2I6qj TSZF/SPL9mHRVRIxWOJixCpJXc1+h3813OhzpCsHkedleue74hSn6/O1qLW8J7Yy jpKJ8ytQhvALPxlzlzNt7M//FCcIZZhQP9NhpINduWDm6DlxLeb7hCNcSgH7FZuZ q++VbcOwb4na8APR3nrTHl2GTtTxVa75TSvwf4T/VzT5zeQqxX/W3sdvoIlP/OEL UbGo6XVXBHWFC0DzwAaQpkAsvo9w4d2JDq9HGl05nucJmcbm3Wg+6/rTmrp4Erb7 1DDp33b4CX7hpdfRP6lHIqwl5TfxmAbXu8edEJponKqjrC22oY03EkOMb6Dduelo OfLShFfSGD1in1IGZCuPVgHw+2EzdWrwFIoLumWbDo22NvObz6l8vDA8z4vvwkVB QI9YV/EJ8BWozdJr5S1EKXmQ4I676QWb2M3YpYaUeyMrze6+S1iyJA4iYkptYCPn pQMK/mlwFYrT6LXfV3XPnVhm+jemXGQNMqvmq19Q+SOljGUB0HVS5eej2zu6wLRU 9fR8paa0Bjaq62emU+lbhYWTswEgQlX51Y3l2IQ31Vdx0KQtLgHR87D05fGuHWCP 7hqEPzwoEIXL1wlCr+IGU4Y6CtlZqO+j9j6EVSYOfDRfhpthQJk= =ZnAd -----END PGP SIGNATURE----- --b9dEYEwnDXkv9lSy-- -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html