[PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

This patchset introduces the following:

* add support for changing the inline header size (max_header_size) for applications
  with overlay and nested headers
* enable automatic fallback to polling mode for admin queue when interrupt is not
  available or missed
* add good checksum counter for Rx ethtool statistics
* update ena.txt
* some minor code clean-up
* some performance enhancements with doorbell calculations

Differences from V1:

* net: ena: add handling of llq max tx burst size (1/11):
 * fixed christmas tree issue

* net: ena: ethtool: add extra properties retrieval via get_priv_flags (2/11):
 * replaced snprintf with strlcpy
 * dropped confusing error message
 * added more details to  the commit message

Arthur Kiyanovski (1):
  net: ena: ethtool: add extra properties retrieval via get_priv_flags

Sameeh Jubran (10):
  net: ena: add handling of llq max tx burst size
  net: ena: replace free_tx/rx_ids union with single free_ids field in
    ena_ring
  net: ena: arrange ena_probe() function variables in reverse christmas
    tree
  net: ena: add newline at the end of pr_err prints
  net: ena: documentation: update ena.txt
  net: ena: allow automatic fallback to polling mode
  net: ena: add support for changing max_header_size in LLQ mode
  net: ena: optimise calculations for CQ doorbell
  net: ena: add good checksum counter
  net: ena: use dev_info_once instead of static variable

 .../networking/device_drivers/amazon/ena.txt  |   5 +-
 .../net/ethernet/amazon/ena/ena_admin_defs.h  |  21 +++
 drivers/net/ethernet/amazon/ena/ena_com.c     | 123 ++++++++++++++----
 drivers/net/ethernet/amazon/ena/ena_com.h     |  48 +++++++
 drivers/net/ethernet/amazon/ena/ena_eth_com.c |  28 ++--
 drivers/net/ethernet/amazon/ena/ena_eth_com.h |  73 +++++++++--
 drivers/net/ethernet/amazon/ena/ena_ethtool.c |  78 +++++++++--
 drivers/net/ethernet/amazon/ena/ena_netdev.c  |  86 +++++++-----
 drivers/net/ethernet/amazon/ena/ena_netdev.h  |  16 +--
 9 files changed, 376 insertions(+), 102 deletions(-)

-- 
2.17.1

[PATCH V2 net 01/11] net: ena: add handling of llq max tx burst size

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

There is a maximum TX burst size that the ENA device can handle.
It is exposed by the device to the driver and the driver
needs to comply with it to avoid bugs.

In this commit we:
1. Add ena_com_is_doorbell_needed(), which calculates the number of
   llq entries that will be used to hold a packet, and will return
   true if they exceed the number of allowed entries in a burst.
   If the function returns true, a doorbell needs to be invoked
   to send this packet in the next burst.

2. Follow the available entries in the current burst:
   - Every doorbell a new burst begins
   - With each write of an llq entry, the available entries in the
     current burst are decreased by 1.

Signed-off-by: Arthur Kiyanovski <akiyano@amazon.com>
Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 .../net/ethernet/amazon/ena/ena_admin_defs.h  |  5 ++
 drivers/net/ethernet/amazon/ena/ena_com.c     |  7 +++
 drivers/net/ethernet/amazon/ena/ena_com.h     |  2 +
 drivers/net/ethernet/amazon/ena/ena_eth_com.c | 28 ++++------
 drivers/net/ethernet/amazon/ena/ena_eth_com.h | 53 +++++++++++++++++++
 drivers/net/ethernet/amazon/ena/ena_netdev.c  |  7 +++
 6 files changed, 85 insertions(+), 17 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_admin_defs.h b/drivers/net/ethernet/amazon/ena/ena_admin_defs.h
index 9f80b73f9..82cff1e12 100644
--- a/drivers/net/ethernet/amazon/ena/ena_admin_defs.h
+++ b/drivers/net/ethernet/amazon/ena/ena_admin_defs.h
@@ -524,6 +524,11 @@ struct ena_admin_feature_llq_desc {
 
 	/* the stride control the driver selected to use */
 	u16 descriptors_stride_ctrl_enabled;
+
+	/* Maximum size in bytes taken by llq entries in a single tx burst.
+	 * Set to 0 when there is no such limit.
+	 */
+	u32 max_tx_burst_size;
 };
 
 struct ena_admin_queue_feature_desc {
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c
index 7f8266b19..bd0d785b2 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_com.c
@@ -396,6 +396,10 @@ static int ena_com_init_io_sq(struct ena_com_dev *ena_dev,
 		       0x0, io_sq->llq_info.desc_list_entry_size);
 		io_sq->llq_buf_ctrl.descs_left_in_line =
 			io_sq->llq_info.descs_num_before_header;
+
+		if (io_sq->llq_info.max_entries_in_tx_burst > 0)
+			io_sq->entries_in_tx_burst_left =
+				io_sq->llq_info.max_entries_in_tx_burst;
 	}
 
 	io_sq->tail = 0;
@@ -727,6 +731,9 @@ static int ena_com_config_llq_info(struct ena_com_dev *ena_dev,
 		       supported_feat, llq_info->descs_num_before_header);
 	}
 
+	llq_info->max_entries_in_tx_burst =
+		(u16)(llq_features->max_tx_burst_size /	llq_default_cfg->llq_ring_entry_size_value);
+
 	rc = ena_com_set_llq(ena_dev);
 	if (rc)
 		pr_err("Cannot set LLQ configuration: %d\n", rc);
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.h b/drivers/net/ethernet/amazon/ena/ena_com.h
index 078d6f2b4..f0cb043af 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.h
+++ b/drivers/net/ethernet/amazon/ena/ena_com.h
@@ -159,6 +159,7 @@ struct ena_com_llq_info {
 	u16 desc_list_entry_size;
 	u16 descs_num_before_header;
 	u16 descs_per_entry;
+	u16 max_entries_in_tx_burst;
 };
 
 struct ena_com_io_cq {
@@ -238,6 +239,7 @@ struct ena_com_io_sq {
 	u8 phase;
 	u8 desc_entry_size;
 	u8 dma_addr_bits;
+	u16 entries_in_tx_burst_left;
 } ____cacheline_aligned;
 
 struct ena_com_admin_cq {
diff --git a/drivers/net/ethernet/amazon/ena/ena_eth_com.c b/drivers/net/ethernet/amazon/ena/ena_eth_com.c
index f6c2d3855..cad2b5728 100644
--- a/drivers/net/ethernet/amazon/ena/ena_eth_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_eth_com.c
@@ -82,6 +82,17 @@ static inline int ena_com_write_bounce_buffer_to_dev(struct ena_com_io_sq *io_sq
 	dst_tail_mask = io_sq->tail & (io_sq->q_depth - 1);
 	dst_offset = dst_tail_mask * llq_info->desc_list_entry_size;
 
+	if (is_llq_max_tx_burst_exists(io_sq)) {
+		if (unlikely(!io_sq->entries_in_tx_burst_left)) {
+			pr_err("Error: trying to send more packets than tx burst allows\n");
+			return -ENOSPC;
+		}
+
+		io_sq->entries_in_tx_burst_left--;
+		pr_debug("decreasing entries_in_tx_burst_left of queue %d to %d\n",
+			 io_sq->qid, io_sq->entries_in_tx_burst_left);
+	}
+
 	/* Make sure everything was written into the bounce buffer before
 	 * writing the bounce buffer to the device
 	 */
@@ -274,23 +285,6 @@ static inline u16 ena_com_cdesc_rx_pkt_get(struct ena_com_io_cq *io_cq,
 	return count;
 }
 
-static inline bool ena_com_meta_desc_changed(struct ena_com_io_sq *io_sq,
-					     struct ena_com_tx_ctx *ena_tx_ctx)
-{
-	int rc;
-
-	if (ena_tx_ctx->meta_valid) {
-		rc = memcmp(&io_sq->cached_tx_meta,
-			    &ena_tx_ctx->ena_meta,
-			    sizeof(struct ena_com_tx_meta));
-
-		if (unlikely(rc != 0))
-			return true;
-	}
-
-	return false;
-}
-
 static inline int ena_com_create_and_store_tx_meta_desc(struct ena_com_io_sq *io_sq,
 							struct ena_com_tx_ctx *ena_tx_ctx)
 {
diff --git a/drivers/net/ethernet/amazon/ena/ena_eth_com.h b/drivers/net/ethernet/amazon/ena/ena_eth_com.h
index 340d02b64..0a3d9180e 100644
--- a/drivers/net/ethernet/amazon/ena/ena_eth_com.h
+++ b/drivers/net/ethernet/amazon/ena/ena_eth_com.h
@@ -125,8 +125,55 @@ static inline bool ena_com_sq_have_enough_space(struct ena_com_io_sq *io_sq,
 	return ena_com_free_desc(io_sq) > temp;
 }
 
+static inline bool ena_com_meta_desc_changed(struct ena_com_io_sq *io_sq,
+					     struct ena_com_tx_ctx *ena_tx_ctx)
+{
+	if (!ena_tx_ctx->meta_valid)
+		return false;
+
+	return !!memcmp(&io_sq->cached_tx_meta,
+			&ena_tx_ctx->ena_meta,
+			sizeof(struct ena_com_tx_meta));
+}
+
+static inline bool is_llq_max_tx_burst_exists(struct ena_com_io_sq *io_sq)
+{
+	return (io_sq->mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV) &&
+	       io_sq->llq_info.max_entries_in_tx_burst > 0;
+}
+
+static inline bool ena_com_is_doorbell_needed(struct ena_com_io_sq *io_sq,
+					      struct ena_com_tx_ctx *ena_tx_ctx)
+{
+	struct ena_com_llq_info *llq_info;
+	int descs_after_first_entry;
+	int num_entries_needed = 1;
+	u16 num_descs;
+
+	if (!is_llq_max_tx_burst_exists(io_sq))
+		return false;
+
+	llq_info = &io_sq->llq_info;
+	num_descs = ena_tx_ctx->num_bufs;
+
+	if (unlikely(ena_com_meta_desc_changed(io_sq, ena_tx_ctx)))
+		++num_descs;
+
+	if (num_descs > llq_info->descs_num_before_header) {
+		descs_after_first_entry = num_descs - llq_info->descs_num_before_header;
+		num_entries_needed += DIV_ROUND_UP(descs_after_first_entry,
+						   llq_info->descs_per_entry);
+	}
+
+	pr_debug("queue: %d num_descs: %d num_entries_needed: %d\n", io_sq->qid,
+		 num_descs, num_entries_needed);
+
+	return num_entries_needed > io_sq->entries_in_tx_burst_left;
+}
+
 static inline int ena_com_write_sq_doorbell(struct ena_com_io_sq *io_sq)
 {
+	u16 max_entries_in_tx_burst = io_sq->llq_info.max_entries_in_tx_burst;
 	u16 tail = io_sq->tail;
 
 	pr_debug("write submission queue doorbell for queue: %d tail: %d\n",
@@ -134,6 +181,12 @@ static inline int ena_com_write_sq_doorbell(struct ena_com_io_sq *io_sq)
 
 	writel(tail, io_sq->db_addr);
 
+	if (is_llq_max_tx_burst_exists(io_sq)) {
+		pr_debug("reset available entries in tx burst for queue %d to %d\n",
+			 io_sq->qid, max_entries_in_tx_burst);
+		io_sq->entries_in_tx_burst_left = max_entries_in_tx_burst;
+	}
+
 	return 0;
 }
 
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index 9c8364292..d2b82f917 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -2172,6 +2172,13 @@ static netdev_tx_t ena_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	/* set flags and meta data */
 	ena_tx_csum(&ena_tx_ctx, skb);
 
+	if (unlikely(ena_com_is_doorbell_needed(tx_ring->ena_com_io_sq, &ena_tx_ctx))) {
+		netif_dbg(adapter, tx_queued, dev,
+			  "llq tx max burst size of queue %d achieved, writing doorbell to send burst\n",
+			  qid);
+		ena_com_write_sq_doorbell(tx_ring->ena_com_io_sq);
+	}
+
 	/* prepare the packet's descriptors to dma engine */
 	rc = ena_com_prepare_tx(tx_ring->ena_com_io_sq, &ena_tx_ctx,
 				&nb_hw_desc);
-- 
2.17.1

[PATCH V2 net 02/11] net: ena: ethtool: add extra properties retrieval via get_priv_flags

[sameehj]

From: Arthur Kiyanovski <akiyano@amazon.com>

This commit adds a mechanism for exposing different device
properties via ethtool's priv_flags. The strings are provided
by the device and copied to user space through the driver.

In this commit we:

Add commands, structs and defines necessary for handling
extra properties

Add functions for:
Allocation/destruction of a buffer for extra properties strings.
Retreival of extra properties strings and flags from the network device.

Handle the allocation of a buffer for extra properties strings.

* Initialize buffer with extra properties strings from the
  network device at driver startup.

Use ethtool's get_priv_flags to expose extra properties of
the ENA device

Signed-off-by: Arthur Kiyanovski <akiyano@amazon.com>
Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 .../net/ethernet/amazon/ena/ena_admin_defs.h  | 16 ++++
 drivers/net/ethernet/amazon/ena/ena_com.c     | 56 ++++++++++++++
 drivers/net/ethernet/amazon/ena/ena_com.h     | 32 ++++++++
 drivers/net/ethernet/amazon/ena/ena_ethtool.c | 75 ++++++++++++++++---
 drivers/net/ethernet/amazon/ena/ena_netdev.c  | 14 ++++
 drivers/net/ethernet/amazon/ena/ena_netdev.h  |  2 +
 6 files changed, 184 insertions(+), 11 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_admin_defs.h b/drivers/net/ethernet/amazon/ena/ena_admin_defs.h
index 82cff1e12..414bae989 100644
--- a/drivers/net/ethernet/amazon/ena/ena_admin_defs.h
+++ b/drivers/net/ethernet/amazon/ena/ena_admin_defs.h
@@ -32,6 +32,8 @@
 #ifndef _ENA_ADMIN_H_
 #define _ENA_ADMIN_H_
 
+#define ENA_ADMIN_EXTRA_PROPERTIES_STRING_LEN 32
+#define ENA_ADMIN_EXTRA_PROPERTIES_COUNT     32
 
 enum ena_admin_aq_opcode {
 	ENA_ADMIN_CREATE_SQ                         = 1,
@@ -60,6 +62,8 @@ enum ena_admin_aq_feature_id {
 	ENA_ADMIN_MAX_QUEUES_NUM                    = 2,
 	ENA_ADMIN_HW_HINTS                          = 3,
 	ENA_ADMIN_LLQ                               = 4,
+	ENA_ADMIN_EXTRA_PROPERTIES_STRINGS          = 5,
+	ENA_ADMIN_EXTRA_PROPERTIES_FLAGS            = 6,
 	ENA_ADMIN_RSS_HASH_FUNCTION                 = 10,
 	ENA_ADMIN_STATELESS_OFFLOAD_CONFIG          = 11,
 	ENA_ADMIN_RSS_REDIRECTION_TABLE_CONFIG      = 12,
@@ -560,6 +564,14 @@ struct ena_admin_set_feature_mtu_desc {
 	u32 mtu;
 };
 
+struct ena_admin_get_extra_properties_strings_desc {
+	u32 count;
+};
+
+struct ena_admin_get_extra_properties_flags_desc {
+	u32 flags;
+};
+
 struct ena_admin_set_feature_host_attr_desc {
 	/* host OS info base address in OS memory. host info is 4KB of
 	 * physically contiguous
@@ -864,6 +876,10 @@ struct ena_admin_get_feat_resp {
 		struct ena_admin_feature_intr_moder_desc intr_moderation;
 
 		struct ena_admin_ena_hw_hints hw_hints;
+
+		struct ena_admin_get_extra_properties_strings_desc extra_properties_strings;
+
+		struct ena_admin_get_extra_properties_flags_desc extra_properties_flags;
 	} u;
 };
 
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c
index bd0d785b2..935e8fa8d 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_com.c
@@ -1877,6 +1877,62 @@ int ena_com_get_link_params(struct ena_com_dev *ena_dev,
 	return ena_com_get_feature(ena_dev, resp, ENA_ADMIN_LINK_CONFIG);
 }
 
+int ena_com_extra_properties_strings_init(struct ena_com_dev *ena_dev)
+{
+	struct ena_admin_get_feat_resp resp;
+	struct ena_extra_properties_strings *extra_properties_strings =
+			&ena_dev->extra_properties_strings;
+	u32 rc;
+
+	extra_properties_strings->size = ENA_ADMIN_EXTRA_PROPERTIES_COUNT *
+		ENA_ADMIN_EXTRA_PROPERTIES_STRING_LEN;
+
+	extra_properties_strings->virt_addr =
+		dma_alloc_coherent(ena_dev->dmadev,
+				   extra_properties_strings->size,
+				   &extra_properties_strings->dma_addr,
+				   GFP_KERNEL);
+	if (unlikely(!extra_properties_strings->virt_addr)) {
+		pr_err("Failed to allocate extra properties strings\n");
+		return 0;
+	}
+
+	rc = ena_com_get_feature_ex(ena_dev, &resp,
+				    ENA_ADMIN_EXTRA_PROPERTIES_STRINGS,
+				    extra_properties_strings->dma_addr,
+				    extra_properties_strings->size);
+	if (rc) {
+		pr_debug("Failed to get extra properties strings\n");
+		goto err;
+	}
+
+	return resp.u.extra_properties_strings.count;
+err:
+	ena_com_delete_extra_properties_strings(ena_dev);
+	return 0;
+}
+
+void ena_com_delete_extra_properties_strings(struct ena_com_dev *ena_dev)
+{
+	struct ena_extra_properties_strings *extra_properties_strings =
+				&ena_dev->extra_properties_strings;
+
+	if (extra_properties_strings->virt_addr) {
+		dma_free_coherent(ena_dev->dmadev,
+				  extra_properties_strings->size,
+				  extra_properties_strings->virt_addr,
+				  extra_properties_strings->dma_addr);
+		extra_properties_strings->virt_addr = NULL;
+	}
+}
+
+int ena_com_get_extra_properties_flags(struct ena_com_dev *ena_dev,
+				       struct ena_admin_get_feat_resp *resp)
+{
+	return ena_com_get_feature(ena_dev, resp,
+				   ENA_ADMIN_EXTRA_PROPERTIES_FLAGS);
+}
+
 int ena_com_get_dev_attr_feat(struct ena_com_dev *ena_dev,
 			      struct ena_com_dev_get_features_ctx *get_feat_ctx)
 {
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.h b/drivers/net/ethernet/amazon/ena/ena_com.h
index f0cb043af..ce36444c3 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.h
+++ b/drivers/net/ethernet/amazon/ena/ena_com.h
@@ -347,6 +347,12 @@ struct ena_host_attribute {
 	dma_addr_t host_info_dma_addr;
 };
 
+struct ena_extra_properties_strings {
+	u8 *virt_addr;
+	dma_addr_t dma_addr;
+	u32 size;
+};
+
 /* Each ena_dev is a PCI function. */
 struct ena_com_dev {
 	struct ena_com_admin_queue admin_queue;
@@ -375,6 +381,7 @@ struct ena_com_dev {
 	struct ena_intr_moder_entry *intr_moder_tbl;
 
 	struct ena_com_llq_info llq_info;
+	struct ena_extra_properties_strings extra_properties_strings;
 };
 
 struct ena_com_dev_get_features_ctx {
@@ -596,6 +603,31 @@ int ena_com_validate_version(struct ena_com_dev *ena_dev);
 int ena_com_get_link_params(struct ena_com_dev *ena_dev,
 			    struct ena_admin_get_feat_resp *resp);
 
+/* ena_com_extra_properties_strings_init - Initialize the extra properties strings buffer.
+ * @ena_dev: ENA communication layer struct
+ *
+ * Initialize the extra properties strings buffer.
+ */
+int ena_com_extra_properties_strings_init(struct ena_com_dev *ena_dev);
+
+/* ena_com_delete_extra_properties_strings - Free the extra properties strings buffer.
+ * @ena_dev: ENA communication layer struct
+ *
+ * Free the allocated extra properties strings buffer.
+ */
+void ena_com_delete_extra_properties_strings(struct ena_com_dev *ena_dev);
+
+/* ena_com_get_extra_properties_flags - Retrieve extra properties flags.
+ * @ena_dev: ENA communication layer struct
+ * @resp: Extra properties flags.
+ *
+ * Retrieve the extra properties flags.
+ *
+ * @return - 0 on Success negative value otherwise.
+ */
+int ena_com_get_extra_properties_flags(struct ena_com_dev *ena_dev,
+				       struct ena_admin_get_feat_resp *resp);
+
 /* ena_com_get_dma_width - Retrieve physical dma address width the device
  * supports.
  * @ena_dev: ENA communication layer struct
diff --git a/drivers/net/ethernet/amazon/ena/ena_ethtool.c b/drivers/net/ethernet/amazon/ena/ena_ethtool.c
index fe596bc30..28fe4c6ef 100644
--- a/drivers/net/ethernet/amazon/ena/ena_ethtool.c
+++ b/drivers/net/ethernet/amazon/ena/ena_ethtool.c
@@ -197,15 +197,24 @@ static void ena_get_ethtool_stats(struct net_device *netdev,
 	ena_dev_admin_queue_stats(adapter, &data);
 }
 
+static int get_stats_sset_count(struct ena_adapter *adapter)
+{
+	return  adapter->num_queues * (ENA_STATS_ARRAY_TX + ENA_STATS_ARRAY_RX)
+		+ ENA_STATS_ARRAY_GLOBAL + ENA_STATS_ARRAY_ENA_COM;
+}
+
 int ena_get_sset_count(struct net_device *netdev, int sset)
 {
 	struct ena_adapter *adapter = netdev_priv(netdev);
 
-	if (sset != ETH_SS_STATS)
+	switch (sset) {
+	case ETH_SS_STATS:
+		return get_stats_sset_count(adapter);
+	case ETH_SS_PRIV_FLAGS:
+		return adapter->ena_extra_properties_count;
+	default:
 		return -EOPNOTSUPP;
-
-	return  adapter->num_queues * (ENA_STATS_ARRAY_TX + ENA_STATS_ARRAY_RX)
-		+ ENA_STATS_ARRAY_GLOBAL + ENA_STATS_ARRAY_ENA_COM;
+	}
 }
 
 static void ena_queue_strings(struct ena_adapter *adapter, u8 **data)
@@ -247,26 +256,54 @@ static void ena_com_dev_strings(u8 **data)
 	}
 }
 
-static void ena_get_strings(struct net_device *netdev, u32 sset, u8 *data)
+static void get_stats_strings(struct ena_adapter *adapter, u8 *data)
 {
-	struct ena_adapter *adapter = netdev_priv(netdev);
 	const struct ena_stats *ena_stats;
 	int i;
 
-	if (sset != ETH_SS_STATS)
-		return;
-
 	for (i = 0; i < ENA_STATS_ARRAY_GLOBAL; i++) {
 		ena_stats = &ena_stats_global_strings[i];
-
 		memcpy(data, ena_stats->name, ETH_GSTRING_LEN);
 		data += ETH_GSTRING_LEN;
 	}
-
 	ena_queue_strings(adapter, &data);
 	ena_com_dev_strings(&data);
 }
 
+static void get_private_flags_strings(struct ena_adapter *adapter, u8 *data)
+{
+	struct ena_com_dev *ena_dev = adapter->ena_dev;
+	u8 *strings = ena_dev->extra_properties_strings.virt_addr;
+	int i;
+
+	if (unlikely(!strings)) {
+		adapter->ena_extra_properties_count = 0;
+		return;
+	}
+
+	for (i = 0; i < adapter->ena_extra_properties_count; i++) {
+		strlcpy(data, strings + ENA_ADMIN_EXTRA_PROPERTIES_STRING_LEN * i,
+			ETH_GSTRING_LEN);
+		data += ETH_GSTRING_LEN;
+	}
+}
+
+static void ena_get_strings(struct net_device *netdev, u32 sset, u8 *data)
+{
+	struct ena_adapter *adapter = netdev_priv(netdev);
+
+	switch (sset) {
+	case ETH_SS_STATS:
+		get_stats_strings(adapter, data);
+		break;
+	case ETH_SS_PRIV_FLAGS:
+		get_private_flags_strings(adapter, data);
+		break;
+	default:
+		break;
+	}
+}
+
 static int ena_get_link_ksettings(struct net_device *netdev,
 				  struct ethtool_link_ksettings *link_ksettings)
 {
@@ -441,6 +478,7 @@ static void ena_get_drvinfo(struct net_device *dev,
 	strlcpy(info->version, DRV_MODULE_VERSION, sizeof(info->version));
 	strlcpy(info->bus_info, pci_name(adapter->pdev),
 		sizeof(info->bus_info));
+	info->n_priv_flags = adapter->ena_extra_properties_count;
 }
 
 static void ena_get_ringparam(struct net_device *netdev,
@@ -798,6 +836,20 @@ static int ena_set_tunable(struct net_device *netdev,
 	return ret;
 }
 
+static u32 ena_get_priv_flags(struct net_device *netdev)
+{
+	struct ena_adapter *adapter = netdev_priv(netdev);
+	struct ena_com_dev *ena_dev = adapter->ena_dev;
+	struct ena_admin_get_feat_resp get_resp;
+	u32 rc;
+
+	rc = ena_com_get_extra_properties_flags(ena_dev, &get_resp);
+	if (!rc)
+		return get_resp.u.extra_properties_flags.flags;
+
+	return 0;
+}
+
 static const struct ethtool_ops ena_ethtool_ops = {
 	.get_link_ksettings	= ena_get_link_ksettings,
 	.get_drvinfo		= ena_get_drvinfo,
@@ -819,6 +871,7 @@ static const struct ethtool_ops ena_ethtool_ops = {
 	.get_channels		= ena_get_channels,
 	.get_tunable		= ena_get_tunable,
 	.set_tunable		= ena_set_tunable,
+	.get_priv_flags		= ena_get_priv_flags,
 };
 
 void ena_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index d2b82f917..33fab4f41 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -2369,6 +2369,14 @@ err:
 	ena_com_delete_debug_area(adapter->ena_dev);
 }
 
+static void ena_extra_properties_strings_destroy(struct net_device *netdev)
+{
+	struct ena_adapter *adapter = netdev_priv(netdev);
+
+	ena_com_delete_extra_properties_strings(adapter->ena_dev);
+	adapter->ena_extra_properties_count = 0;
+}
+
 static void ena_get_stats64(struct net_device *netdev,
 			    struct rtnl_link_stats64 *stats)
 {
@@ -3424,6 +3432,9 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	ena_config_debug_area(adapter);
 
+	adapter->ena_extra_properties_count =
+		ena_com_extra_properties_strings_init(ena_dev);
+
 	memcpy(adapter->netdev->perm_addr, adapter->mac_addr, netdev->addr_len);
 
 	netif_carrier_off(netdev);
@@ -3463,6 +3474,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	return 0;
 
 err_rss:
+	ena_extra_properties_strings_destroy(netdev);
 	ena_com_delete_debug_area(ena_dev);
 	ena_com_rss_destroy(ena_dev);
 err_free_msix:
@@ -3529,6 +3541,8 @@ static void ena_remove(struct pci_dev *pdev)
 
 	ena_com_delete_host_info(ena_dev);
 
+	ena_extra_properties_strings_destroy(netdev);
+
 	ena_release_bars(ena_dev, pdev);
 
 	pci_disable_device(pdev);
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h
index 63870072c..0681e18b0 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.h
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h
@@ -364,6 +364,8 @@ struct ena_adapter {
 	u32 last_monitored_tx_qid;
 
 	enum ena_regs_reset_reason_types reset_reason;
+
+	u8 ena_extra_properties_count;
 };
 
 void ena_set_ethtool_ops(struct net_device *netdev);
-- 
2.17.1

[PATCH V2 net 03/11] net: ena: replace free_tx/rx_ids union with single free_ids field in ena_ring

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

struct ena_ring holds a union of free_rx_ids and free_tx_ids.
Both of the above fields mean the exact same thing and are used
exactly the same way.
Furthermore, these fields are always used with a prefix of the
type of ring. So for tx it will be tx_ring->free_tx_ids, and for
rx it will be rx_ring->free_rx_ids, which shows how redundant the
"_tx" and "_rx" parts are.
Furthermore still, this may lead to confusing code like where
tx_ring->free_rx_ids which works correctly but looks like a mess.

This commit removes the aforementioned redundancy by replacing the
free_rx/tx_ids union with a single free_ids field.
It also changes a single goto label name from err_free_tx_ids: to
err_tx_free_ids: for consistency with the above new notation.

Signed-off-by: Arthur Kiyanovski <akiyano@amazon.com>
Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_netdev.c | 48 ++++++++++----------
 drivers/net/ethernet/amazon/ena/ena_netdev.h | 11 ++---
 2 files changed, 28 insertions(+), 31 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index 33fab4f41..b80b5eddc 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -228,11 +228,11 @@ static int ena_setup_tx_resources(struct ena_adapter *adapter, int qid)
 	}
 
 	size = sizeof(u16) * tx_ring->ring_size;
-	tx_ring->free_tx_ids = vzalloc_node(size, node);
-	if (!tx_ring->free_tx_ids) {
-		tx_ring->free_tx_ids = vzalloc(size);
-		if (!tx_ring->free_tx_ids)
-			goto err_free_tx_ids;
+	tx_ring->free_ids = vzalloc_node(size, node);
+	if (!tx_ring->free_ids) {
+		tx_ring->free_ids = vzalloc(size);
+		if (!tx_ring->free_ids)
+			goto err_tx_free_ids;
 	}
 
 	size = tx_ring->tx_max_header_size;
@@ -245,7 +245,7 @@ static int ena_setup_tx_resources(struct ena_adapter *adapter, int qid)
 
 	/* Req id ring for TX out of order completions */
 	for (i = 0; i < tx_ring->ring_size; i++)
-		tx_ring->free_tx_ids[i] = i;
+		tx_ring->free_ids[i] = i;
 
 	/* Reset tx statistics */
 	memset(&tx_ring->tx_stats, 0x0, sizeof(tx_ring->tx_stats));
@@ -256,9 +256,9 @@ static int ena_setup_tx_resources(struct ena_adapter *adapter, int qid)
 	return 0;
 
 err_push_buf_intermediate_buf:
-	vfree(tx_ring->free_tx_ids);
-	tx_ring->free_tx_ids = NULL;
-err_free_tx_ids:
+	vfree(tx_ring->free_ids);
+	tx_ring->free_ids = NULL;
+err_tx_free_ids:
 	vfree(tx_ring->tx_buffer_info);
 	tx_ring->tx_buffer_info = NULL;
 err_tx_buffer_info:
@@ -278,8 +278,8 @@ static void ena_free_tx_resources(struct ena_adapter *adapter, int qid)
 	vfree(tx_ring->tx_buffer_info);
 	tx_ring->tx_buffer_info = NULL;
 
-	vfree(tx_ring->free_tx_ids);
-	tx_ring->free_tx_ids = NULL;
+	vfree(tx_ring->free_ids);
+	tx_ring->free_ids = NULL;
 
 	vfree(tx_ring->push_buf_intermediate_buf);
 	tx_ring->push_buf_intermediate_buf = NULL;
@@ -377,10 +377,10 @@ static int ena_setup_rx_resources(struct ena_adapter *adapter,
 	}
 
 	size = sizeof(u16) * rx_ring->ring_size;
-	rx_ring->free_rx_ids = vzalloc_node(size, node);
-	if (!rx_ring->free_rx_ids) {
-		rx_ring->free_rx_ids = vzalloc(size);
-		if (!rx_ring->free_rx_ids) {
+	rx_ring->free_ids = vzalloc_node(size, node);
+	if (!rx_ring->free_ids) {
+		rx_ring->free_ids = vzalloc(size);
+		if (!rx_ring->free_ids) {
 			vfree(rx_ring->rx_buffer_info);
 			rx_ring->rx_buffer_info = NULL;
 			return -ENOMEM;
@@ -389,7 +389,7 @@ static int ena_setup_rx_resources(struct ena_adapter *adapter,
 
 	/* Req id ring for receiving RX pkts out of order */
 	for (i = 0; i < rx_ring->ring_size; i++)
-		rx_ring->free_rx_ids[i] = i;
+		rx_ring->free_ids[i] = i;
 
 	/* Reset rx statistics */
 	memset(&rx_ring->rx_stats, 0x0, sizeof(rx_ring->rx_stats));
@@ -415,8 +415,8 @@ static void ena_free_rx_resources(struct ena_adapter *adapter,
 	vfree(rx_ring->rx_buffer_info);
 	rx_ring->rx_buffer_info = NULL;
 
-	vfree(rx_ring->free_rx_ids);
-	rx_ring->free_rx_ids = NULL;
+	vfree(rx_ring->free_ids);
+	rx_ring->free_ids = NULL;
 }
 
 /* ena_setup_all_rx_resources - allocate I/O Rx queues resources for all queues
@@ -531,7 +531,7 @@ static int ena_refill_rx_bufs(struct ena_ring *rx_ring, u32 num)
 	for (i = 0; i < num; i++) {
 		struct ena_rx_buffer *rx_info;
 
-		req_id = rx_ring->free_rx_ids[next_to_use];
+		req_id = rx_ring->free_ids[next_to_use];
 		rc = validate_rx_req_id(rx_ring, req_id);
 		if (unlikely(rc < 0))
 			break;
@@ -797,7 +797,7 @@ static int ena_clean_tx_irq(struct ena_ring *tx_ring, u32 budget)
 		tx_pkts++;
 		total_done += tx_info->tx_descs;
 
-		tx_ring->free_tx_ids[next_to_clean] = req_id;
+		tx_ring->free_ids[next_to_clean] = req_id;
 		next_to_clean = ENA_TX_RING_IDX_NEXT(next_to_clean,
 						     tx_ring->ring_size);
 	}
@@ -911,7 +911,7 @@ static struct sk_buff *ena_rx_skb(struct ena_ring *rx_ring,
 
 		skb_put(skb, len);
 		skb->protocol = eth_type_trans(skb, rx_ring->netdev);
-		rx_ring->free_rx_ids[*next_to_clean] = req_id;
+		rx_ring->free_ids[*next_to_clean] = req_id;
 		*next_to_clean = ENA_RX_RING_IDX_ADD(*next_to_clean, descs,
 						     rx_ring->ring_size);
 		return skb;
@@ -935,7 +935,7 @@ static struct sk_buff *ena_rx_skb(struct ena_ring *rx_ring,
 
 		rx_info->page = NULL;
 
-		rx_ring->free_rx_ids[*next_to_clean] = req_id;
+		rx_ring->free_ids[*next_to_clean] = req_id;
 		*next_to_clean =
 			ENA_RX_RING_IDX_NEXT(*next_to_clean,
 					     rx_ring->ring_size);
@@ -1088,7 +1088,7 @@ static int ena_clean_rx_irq(struct ena_ring *rx_ring, struct napi_struct *napi,
 		/* exit if we failed to retrieve a buffer */
 		if (unlikely(!skb)) {
 			for (i = 0; i < ena_rx_ctx.descs; i++) {
-				rx_ring->free_tx_ids[next_to_clean] =
+				rx_ring->free_ids[next_to_clean] =
 					rx_ring->ena_bufs[i].req_id;
 				next_to_clean =
 					ENA_RX_RING_IDX_NEXT(next_to_clean,
@@ -2152,7 +2152,7 @@ static netdev_tx_t ena_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	skb_tx_timestamp(skb);
 
 	next_to_use = tx_ring->next_to_use;
-	req_id = tx_ring->free_tx_ids[next_to_use];
+	req_id = tx_ring->free_ids[next_to_use];
 	tx_info = &tx_ring->tx_buffer_info[req_id];
 	tx_info->num_of_bufs = 0;
 
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h
index 0681e18b0..74c316081 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.h
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h
@@ -221,13 +221,10 @@ struct ena_stats_rx {
 };
 
 struct ena_ring {
-	union {
-		/* Holds the empty requests for TX/RX
-		 * out of order completions
-		 */
-		u16 *free_tx_ids;
-		u16 *free_rx_ids;
-	};
+	/* Holds the empty requests for TX/RX
+	 * out of order completions
+	 */
+	u16 *free_ids;
 
 	union {
 		struct ena_tx_buffer *tx_buffer_info;
-- 
2.17.1

[PATCH V2 net 04/11] net: ena: arrange ena_probe() function variables in reverse christmas tree

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

Reverse christmas tree arrangement is when strings are written from longer
to shorter with each line. Most of our functions are abiding this
arrangement but this function does not.

In this commit we arrange the variables of ena_probe() in reverse christmas
tree.

Signed-off-by: Arthur Kiyanovski <akiyano@amazon.com>
Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_netdev.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index b80b5eddc..399bd5453 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -3281,17 +3281,17 @@ static int ena_calc_queue_size(struct pci_dev *pdev,
 static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 {
 	struct ena_com_dev_get_features_ctx get_feat_ctx;
-	static int version_printed;
-	struct net_device *netdev;
-	struct ena_adapter *adapter;
 	struct ena_llq_configurations llq_config;
 	struct ena_com_dev *ena_dev = NULL;
-	char *queue_type_str;
-	static int adapters_found;
+	struct ena_adapter *adapter;
+	static int version_printed;
 	int io_queue_num, bars, rc;
-	int queue_size;
+	struct net_device *netdev;
+	static int adapters_found;
+	char *queue_type_str;
 	u16 tx_sgl_size = 0;
 	u16 rx_sgl_size = 0;
+	int queue_size;
 	bool wd_state;
 
 	dev_dbg(&pdev->dev, "%s\n", __func__);
-- 
2.17.1

[PATCH V2 net 05/11] net: ena: add newline at the end of pr_err prints

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

Some pr_err prints lacked '\n' in the end. Added where missing.

Signed-off-by: Arthur Kiyanovski <akiyano@amazon.com>
Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_com.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c
index 935e8fa8d..139b31549 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_com.c
@@ -115,7 +115,7 @@ static int ena_com_admin_init_sq(struct ena_com_admin_queue *queue)
 					 GFP_KERNEL);
 
 	if (!sq->entries) {
-		pr_err("memory allocation failed");
+		pr_err("memory allocation failed\n");
 		return -ENOMEM;
 	}
 
@@ -137,7 +137,7 @@ static int ena_com_admin_init_cq(struct ena_com_admin_queue *queue)
 					 GFP_KERNEL);
 
 	if (!cq->entries) {
-		pr_err("memory allocation failed");
+		pr_err("memory allocation failed\n");
 		return -ENOMEM;
 	}
 
@@ -160,7 +160,7 @@ static int ena_com_admin_init_aenq(struct ena_com_dev *dev,
 					   GFP_KERNEL);
 
 	if (!aenq->entries) {
-		pr_err("memory allocation failed");
+		pr_err("memory allocation failed\n");
 		return -ENOMEM;
 	}
 
@@ -285,7 +285,7 @@ static inline int ena_com_init_comp_ctxt(struct ena_com_admin_queue *queue)
 
 	queue->comp_ctx = devm_kzalloc(queue->q_dmadev, size, GFP_KERNEL);
 	if (unlikely(!queue->comp_ctx)) {
-		pr_err("memory allocation failed");
+		pr_err("memory allocation failed\n");
 		return -ENOMEM;
 	}
 
@@ -356,7 +356,7 @@ static int ena_com_init_io_sq(struct ena_com_dev *ena_dev,
 		}
 
 		if (!io_sq->desc_addr.virt_addr) {
-			pr_err("memory allocation failed");
+			pr_err("memory allocation failed\n");
 			return -ENOMEM;
 		}
 	}
@@ -382,7 +382,7 @@ static int ena_com_init_io_sq(struct ena_com_dev *ena_dev,
 				devm_kzalloc(ena_dev->dmadev, size, GFP_KERNEL);
 
 		if (!io_sq->bounce_buf_ctrl.base_buffer) {
-			pr_err("bounce buffer memory allocation failed");
+			pr_err("bounce buffer memory allocation failed\n");
 			return -ENOMEM;
 		}
 
@@ -440,7 +440,7 @@ static int ena_com_init_io_cq(struct ena_com_dev *ena_dev,
 	}
 
 	if (!io_cq->cdesc_addr.virt_addr) {
-		pr_err("memory allocation failed");
+		pr_err("memory allocation failed\n");
 		return -ENOMEM;
 	}
 
@@ -829,7 +829,7 @@ static u32 ena_com_reg_bar_read32(struct ena_com_dev *ena_dev, u16 offset)
 	}
 
 	if (read_resp->reg_off != offset) {
-		pr_err("Read failure: wrong offset provided");
+		pr_err("Read failure: wrong offset provided\n");
 		ret = ENA_MMIO_READ_TIMEOUT;
 	} else {
 		ret = read_resp->reg_val;
-- 
2.17.1

[PATCH V2 net 06/11] net: ena: documentation: update ena.txt

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

Small cosmetic changes to ena.txt

Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 Documentation/networking/device_drivers/amazon/ena.txt | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/Documentation/networking/device_drivers/amazon/ena.txt b/Documentation/networking/device_drivers/amazon/ena.txt
index 2b4b6f57e..1bb55c7b6 100644
--- a/Documentation/networking/device_drivers/amazon/ena.txt
+++ b/Documentation/networking/device_drivers/amazon/ena.txt
@@ -73,7 +73,7 @@ operation.
 AQ is used for submitting management commands, and the
 results/responses are reported asynchronously through ACQ.
 
-ENA introduces a very small set of management commands with room for
+ENA introduces a small set of management commands with room for
 vendor-specific extensions. Most of the management operations are
 framed in a generic Get/Set feature command.
 
@@ -202,11 +202,14 @@ delay value to each level.
 The user can enable/disable adaptive moderation, modify the interrupt
 delay table and restore its default values through sysfs.
 
+RX copybreak:
+=============
 The rx_copybreak is initialized by default to ENA_DEFAULT_RX_COPYBREAK
 and can be configured by the ETHTOOL_STUNABLE command of the
 SIOCETHTOOL ioctl.
 
 SKB:
+====
 The driver-allocated SKB for frames received from Rx handling using
 NAPI context. The allocation method depends on the size of the packet.
 If the frame length is larger than rx_copybreak, napi_get_frags()
-- 
2.17.1

[PATCH V2 net 07/11] net: ena: allow automatic fallback to polling mode

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

Enable fallback to polling mode for Admin queue
when identified a command response arrival
without an accompanying MSI-X interrupt

Signed-off-by: Igor Chauskin <igorch@amazon.com>
Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_com.c | 34 +++++++++++++++++------
 drivers/net/ethernet/amazon/ena/ena_com.h | 14 ++++++++++
 2 files changed, 39 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c
index 139b31549..5e2abdda7 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_com.c
@@ -762,16 +762,26 @@ static int ena_com_wait_and_process_admin_cq_interrupts(struct ena_comp_ctx *com
 		admin_queue->stats.no_completion++;
 		spin_unlock_irqrestore(&admin_queue->q_lock, flags);
 
-		if (comp_ctx->status == ENA_CMD_COMPLETED)
-			pr_err("The ena device have completion but the driver didn't receive any MSI-X interrupt (cmd %d)\n",
-			       comp_ctx->cmd_opcode);
-		else
-			pr_err("The ena device doesn't send any completion for the admin cmd %d status %d\n",
+		if (comp_ctx->status == ENA_CMD_COMPLETED) {
+			pr_err("The ena device sent a completion but the driver didn't receive a MSI-X interrupt (cmd %d), autopolling mode is %s\n",
+			       comp_ctx->cmd_opcode,
+			       admin_queue->auto_polling ? "ON" : "OFF");
+			/* Check if fallback to polling is enabled */
+			if (admin_queue->auto_polling)
+				admin_queue->polling = true;
+		} else {
+			pr_err("The ena device doesn't send a completion for the admin cmd %d status %d\n",
 			       comp_ctx->cmd_opcode, comp_ctx->status);
-
-		admin_queue->running_state = false;
-		ret = -ETIME;
-		goto err;
+		}
+		/* Check if shifted to polling mode.
+		 * This will happen if there is a completion without an interrupt
+		 * and autopolling mode is enabled. Continuing normal execution in such case
+		 */
+		if (!admin_queue->polling) {
+			admin_queue->running_state = false;
+			ret = -ETIME;
+			goto err;
+		}
 	}
 
 	ret = ena_com_comp_status_to_errno(comp_ctx->comp_status);
@@ -1650,6 +1660,12 @@ void ena_com_set_admin_polling_mode(struct ena_com_dev *ena_dev, bool polling)
 	ena_dev->admin_queue.polling = polling;
 }
 
+void ena_com_set_admin_auto_polling_mode(struct ena_com_dev *ena_dev,
+					 bool polling)
+{
+	ena_dev->admin_queue.auto_polling = polling;
+}
+
 int ena_com_mmio_reg_read_request_init(struct ena_com_dev *ena_dev)
 {
 	struct ena_com_mmio_read *mmio_read = &ena_dev->mmio_read;
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.h b/drivers/net/ethernet/amazon/ena/ena_com.h
index ce36444c3..6d356cb05 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.h
+++ b/drivers/net/ethernet/amazon/ena/ena_com.h
@@ -283,6 +283,9 @@ struct ena_com_admin_queue {
 	/* Indicate if the admin queue should poll for completion */
 	bool polling;
 
+	/* Define if fallback to polling mode should occur */
+	bool auto_polling;
+
 	u16 curr_cmd_id;
 
 	/* Indicate that the ena was initialized and can
@@ -545,6 +548,17 @@ void ena_com_set_admin_polling_mode(struct ena_com_dev *ena_dev, bool polling);
  */
 bool ena_com_get_ena_admin_polling_mode(struct ena_com_dev *ena_dev);
 
+/* ena_com_set_admin_auto_polling_mode - Enable autoswitch to polling mode
+ * @ena_dev: ENA communication layer struct
+ * @polling: Enable/Disable polling mode
+ *
+ * Set the autopolling mode.
+ * If autopolling is on:
+ * In case of missing interrupt when data is available switch to polling.
+ */
+void ena_com_set_admin_auto_polling_mode(struct ena_com_dev *ena_dev,
+					 bool polling);
+
 /* ena_com_admin_q_comp_intr_handler - admin queue interrupt handler
  * @ena_dev: ENA communication layer struct
  *
-- 
2.17.1

[PATCH V2 net 08/11] net: ena: add support for changing max_header_size in LLQ mode

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

Up until now the driver always used a single setting for the sizes
of the different parts of the llq entry - 128 for entry size, 2 for
descriptors before header and 96 for maximum header size.

The current code makes sure that the parts of the llq entry are
compatible with each other and with the initial llq entry size given
by the device.

This commit changes this code to support any llq entry size

Signed-off-by: Arthur Kiyanovski <akiyano@amazon.com>
Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_com.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c
index 5e2abdda7..dbc12e383 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_com.c
@@ -2992,8 +2992,8 @@ int ena_com_config_dev_mode(struct ena_com_dev *ena_dev,
 			    struct ena_admin_feature_llq_desc *llq_features,
 			    struct ena_llq_configurations *llq_default_cfg)
 {
+	struct ena_com_llq_info *llq_info = &ena_dev->llq_info;
 	int rc;
-	int size;
 
 	if (!llq_features->max_llq_num) {
 		ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST;
@@ -3004,12 +3004,10 @@ int ena_com_config_dev_mode(struct ena_com_dev *ena_dev,
 	if (rc)
 		return rc;
 
-	/* Validate the descriptor is not too big */
-	size = ena_dev->tx_max_header_size;
-	size += ena_dev->llq_info.descs_num_before_header *
-		sizeof(struct ena_eth_io_tx_desc);
+	ena_dev->tx_max_header_size = llq_info->desc_list_entry_size -
+		(llq_info->descs_num_before_header * sizeof(struct ena_eth_io_tx_desc));
 
-	if (unlikely(ena_dev->llq_info.desc_list_entry_size < size)) {
+	if (unlikely(ena_dev->tx_max_header_size == 0)) {
 		pr_err("the size of the LLQ entry is smaller than needed\n");
 		return -EINVAL;
 	}
-- 
2.17.1

[PATCH V2 net 09/11] net: ena: optimise calculations for CQ doorbell

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

This patch initially checks if CQ doorbell
is needed before proceeding with the calculations.

Signed-off-by: Igor Chauskin <igorch@amazon.com>
Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_eth_com.h | 20 ++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_eth_com.h b/drivers/net/ethernet/amazon/ena/ena_eth_com.h
index 0a3d9180e..77986c0ea 100644
--- a/drivers/net/ethernet/amazon/ena/ena_eth_com.h
+++ b/drivers/net/ethernet/amazon/ena/ena_eth_com.h
@@ -195,15 +195,17 @@ static inline int ena_com_update_dev_comp_head(struct ena_com_io_cq *io_cq)
 	u16 unreported_comp, head;
 	bool need_update;
 
-	head = io_cq->head;
-	unreported_comp = head - io_cq->last_head_update;
-	need_update = unreported_comp > (io_cq->q_depth / ENA_COMP_HEAD_THRESH);
-
-	if (io_cq->cq_head_db_reg && need_update) {
-		pr_debug("Write completion queue doorbell for queue %d: head: %d\n",
-			 io_cq->qid, head);
-		writel(head, io_cq->cq_head_db_reg);
-		io_cq->last_head_update = head;
+	if (unlikely(io_cq->cq_head_db_reg)) {
+		head = io_cq->head;
+		unreported_comp = head - io_cq->last_head_update;
+		need_update = unreported_comp > (io_cq->q_depth / ENA_COMP_HEAD_THRESH);
+
+		if (unlikely(need_update)) {
+			pr_debug("Write completion queue doorbell for queue %d: head: %d\n",
+				 io_cq->qid, head);
+			writel(head, io_cq->cq_head_db_reg);
+			io_cq->last_head_update = head;
+		}
 	}
 
 	return 0;
-- 
2.17.1

[PATCH V2 net 10/11] net: ena: add good checksum counter

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

Add a new statistics to ETHTOOL to specify if the device calculated
and validated the Rx csum.

Signed-off-by: Evgeny Shmeilin <evgeny@annapurnaLabs.com>
Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_ethtool.c | 3 ++-
 drivers/net/ethernet/amazon/ena/ena_netdev.c  | 3 +++
 drivers/net/ethernet/amazon/ena/ena_netdev.h  | 3 ++-
 3 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_ethtool.c b/drivers/net/ethernet/amazon/ena/ena_ethtool.c
index 28fe4c6ef..5687a2860 100644
--- a/drivers/net/ethernet/amazon/ena/ena_ethtool.c
+++ b/drivers/net/ethernet/amazon/ena/ena_ethtool.c
@@ -88,13 +88,14 @@ static const struct ena_stats ena_stats_tx_strings[] = {
 static const struct ena_stats ena_stats_rx_strings[] = {
 	ENA_STAT_RX_ENTRY(cnt),
 	ENA_STAT_RX_ENTRY(bytes),
+	ENA_STAT_RX_ENTRY(rx_copybreak_pkt),
+	ENA_STAT_RX_ENTRY(csum_good),
 	ENA_STAT_RX_ENTRY(refil_partial),
 	ENA_STAT_RX_ENTRY(bad_csum),
 	ENA_STAT_RX_ENTRY(page_alloc_fail),
 	ENA_STAT_RX_ENTRY(skb_alloc_fail),
 	ENA_STAT_RX_ENTRY(dma_mapping_err),
 	ENA_STAT_RX_ENTRY(bad_desc_num),
-	ENA_STAT_RX_ENTRY(rx_copybreak_pkt),
 	ENA_STAT_RX_ENTRY(bad_req_id),
 	ENA_STAT_RX_ENTRY(empty_rx_ring),
 	ENA_STAT_RX_ENTRY(csum_unchecked),
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index 399bd5453..cff297e19 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -1001,6 +1001,9 @@ static inline void ena_rx_checksum(struct ena_ring *rx_ring,
 
 		if (likely(ena_rx_ctx->l4_csum_checked)) {
 			skb->ip_summed = CHECKSUM_UNNECESSARY;
+			u64_stats_update_begin(&rx_ring->syncp);
+			rx_ring->rx_stats.csum_good++;
+			u64_stats_update_end(&rx_ring->syncp);
 		} else {
 			u64_stats_update_begin(&rx_ring->syncp);
 			rx_ring->rx_stats.csum_unchecked++;
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h
index 74c316081..ec111cfc5 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.h
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h
@@ -208,13 +208,14 @@ struct ena_stats_tx {
 struct ena_stats_rx {
 	u64 cnt;
 	u64 bytes;
+	u64 rx_copybreak_pkt;
+	u64 csum_good;
 	u64 refil_partial;
 	u64 bad_csum;
 	u64 page_alloc_fail;
 	u64 skb_alloc_fail;
 	u64 dma_mapping_err;
 	u64 bad_desc_num;
-	u64 rx_copybreak_pkt;
 	u64 bad_req_id;
 	u64 empty_rx_ring;
 	u64 csum_unchecked;
-- 
2.17.1

[PATCH V2 net 11/11] net: ena: use dev_info_once instead of static variable

[sameehj]

From: Sameeh Jubran <sameehj@amazon.com>

Signed-off-by: Sameeh Jubran <sameehj@amazon.com>
---
 drivers/net/ethernet/amazon/ena/ena_netdev.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index cff297e19..68bed2417 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -3287,7 +3287,6 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	struct ena_llq_configurations llq_config;
 	struct ena_com_dev *ena_dev = NULL;
 	struct ena_adapter *adapter;
-	static int version_printed;
 	int io_queue_num, bars, rc;
 	struct net_device *netdev;
 	static int adapters_found;
@@ -3299,8 +3298,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	dev_dbg(&pdev->dev, "%s\n", __func__);
 
-	if (version_printed++ == 0)
-		dev_info(&pdev->dev, "%s", version);
+	dev_info_once(&pdev->dev, "%s", version);
 
 	rc = pci_enable_device_mem(pdev);
 	if (rc) {
-- 
2.17.1

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[David Miller]

From: <sameehj@amazon.com>
Date: Mon, 3 Jun 2019 17:43:18 +0300

> From: Sameeh Jubran <sameehj@amazon.com>
> 
> This patchset introduces the following:
> 
> * add support for changing the inline header size (max_header_size) for applications
>   with overlay and nested headers
> * enable automatic fallback to polling mode for admin queue when interrupt is not
>   available or missed
> * add good checksum counter for Rx ethtool statistics
> * update ena.txt
> * some minor code clean-up
> * some performance enhancements with doorbell calculations
> 
> Differences from V1:
> 
> * net: ena: add handling of llq max tx burst size (1/11):
>  * fixed christmas tree issue
> 
> * net: ena: ethtool: add extra properties retrieval via get_priv_flags (2/11):
>  * replaced snprintf with strlcpy
>  * dropped confusing error message
>  * added more details to  the commit message

Series applied.

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jakub Kicinski]

On Mon, 3 Jun 2019 17:43:18 +0300, sameehj@amazon.com wrote:
> * net: ena: ethtool: add extra properties retrieval via get_priv_flags (2/11):
>  * replaced snprintf with strlcpy
>  * dropped confusing error message
>  * added more details to  the commit message

I asked you to clearly state that you are using the blindly passing
this info from the FW to user space.  Stating that you "retrieve" it
is misleading.

IMHO it's a dangerous precedent, you're extending kernel's uAPI down to
the proprietary FW of the device.  Now we have no idea what the flags
are, so we can't refactor and create common APIs among drivers, or even
use the same names.  We have no idea what you're exposing.

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jakub Kicinski]

On Mon, 3 Jun 2019 22:27:28 +0000, Woodhouse, David wrote:
> On Mon, 2019-06-03 at 14:32 -0700, Jakub Kicinski wrote:
> > On Mon, 3 Jun 2019 17:43:18 +0300, sameehj@amazon.com wrote:  
> > > * net: ena: ethtool: add extra properties retrieval via get_priv_flags (2/11):
> > >   * replaced snprintf with strlcpy
> > >   * dropped confusing error message
> > >   * added more details to  the commit message  
> > 
> > I asked you to clearly state that you are using the blindly passing
> > this info from the FW to user space.  Stating that you "retrieve" it
> > is misleading.  
> 
> Yes, we should be very clear about that.
> 
> > IMHO it's a dangerous precedent, you're extending kernel's uAPI down to
> > the proprietary FW of the device.  Now we have no idea what the flags
> > are, so we can't refactor and create common APIs among drivers, or even
> > use the same names.  We have no idea what you're exposing.  
> 
> Yes, that should be documented very clearly too, and we should
> absolutely make sure that anything that makes sense for other devices
> is considered for making a common API.
> 
> However, the deployment environment for ENA is kind of weird — we get
> to upgrade the *firmware*, while guest kernels might get updated only
> once a decade. So the passthrough you're objecting to, actually gives
> us fairly much the only viable way to offer many users the tuning
> options they need — or at least to experiment and establish whether
> they *do* need them or not, and thus if we should turn them into a
> "real" generic property.
> 
> You do have a valid concern, but I think we can handle it, and I
> suspect that the approach taken here does make sense. Let's just make
> it explicitly clear, and also document the properties that we expect to
> be exposing, for visibility.

Thank you.

It's generally easier to push FW updates, also to enterprises, rather
than rolling out full new kernels.  People are less concerned with FW
updates, because the only thing those can break is the NIC.  Some
customers also run their own modified kernels.  I'd dispute the fact
that Amazon is so very special.

I don't dispute that this is convenient for Amazon and your FW team.
However, what's best for the vendors differs here from what's good for
Open Source (the FW is proprietary) and IMHO health of Linux ecosystem
in general.

Any "SmartNIC" vendor has temptation of uAPI-level hand off to the
firmware (including my employer), we all run pretty beefy processors
inside "the NIC" after all.  The device centric ethtool configuration
can be implemented by just forwarding the uAPI structures as they are
to the FW.  I'm sure Andrew and others who would like to see Linux
takes more control over PHYs etc. would not like this scenario, either.

To address your specific points, let's be realistic, after initial
submission it's unlikely the features will be updated in a timely
manner anywhere else than perhaps some Amazon's own docs.  Can you
be more specific on what those tuning options are today?  There are
users who don't care to update their kernels for 10 years, yet they
care about some minor tuning option of the NIC?

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Andrew Lunn]

> Any "SmartNIC" vendor has temptation of uAPI-level hand off to the
> firmware (including my employer), we all run pretty beefy processors
> inside "the NIC" after all.  The device centric ethtool configuration
> can be implemented by just forwarding the uAPI structures as they are
> to the FW.  I'm sure Andrew and others who would like to see Linux
> takes more control over PHYs etc. would not like this scenario, either.

No, i would not. There are a few good examples of both firmware and
open drivers being used to control the same PHY, on different
boards. The PHY driver was developed by the community, and has more
features than the firmware driver. And it keeps gaining features. The
firmware i stuck, no updates. The community driver can be debugged,
the firmware is a black box, no chance of the community fixing any
bugs in it.

And PHYs are commodity devices. I doubt there is any value add in the
firmware for a PHY, any real IPR which makes the product better, magic
sauce related to the PHY. So just save the cost of writing and
maintaining firmware, export the MDIO bus, and let Linux control it.
Concentrate the engineers on the interesting parts of the NIC, the
Smart parts, where there can be real IPR.

And i would say this is true for any NIC. Let Linux control the PHY.

      Andrew

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Bshara, Nafea]

Andrew,

Sent from my iPhone

On Jun 3, 2019, at 6:52 PM, Andrew Lunn <andrew@lunn.ch> wrote:

>> Any "SmartNIC" vendor has temptation of uAPI-level hand off to the
>> firmware (including my employer), we all run pretty beefy processors
>> inside "the NIC" after all.  The device centric ethtool configuration
>> can be implemented by just forwarding the uAPI structures as they are
>> to the FW.  I'm sure Andrew and others who would like to see Linux
>> takes more control over PHYs etc. would not like this scenario, either.
> 
> No, i would not. There are a few good examples of both firmware and
> open drivers being used to control the same PHY, on different
> boards. The PHY driver was developed by the community, and has more
> features than the firmware driver. And it keeps gaining features. The
> firmware i stuck, no updates. The community driver can be debugged,
> the firmware is a black box, no chance of the community fixing any
> bugs in it.
> 
> And PHYs are commodity devices. I doubt there is any value add in the
> firmware for a PHY, any real IPR which makes the product better, magic
> sauce related to the PHY. So just save the cost of writing and
> maintaining firmware, export the MDIO bus, and let Linux control it.
> Concentrate the engineers on the interesting parts of the NIC, the
> Smart parts, where there can be real IPR.
> 
> And i would say this is true for any NIC. Let Linux control the PHY.
> 
>      Andrew
> 

It may be true for old GbE PHYs where it’s a discrete chip from the likes of Marvell or broadcom

But at 25/50/100G, the PHy is actually part of the nic. It’s a very complex SERDES.  Cloud providers like us spend enormous amount of time testing the PHY across process and voltage variations, all cable types, length and manufacturing variations, and against all switches we use.  Community drivers won’t be able to validate and tune all this.

Plus we would need exact same setting for Linux, including all distributions even 10year old like RHEL6, for all Windows, ESX, DPDK, FreeBSD,  and support millions of different customers with different sets of Machine images. 

In this case, there is no practical choice by have the firmware to manage the PHY

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 2542 bytes --]

On Tue, 2019-06-04 at 02:15 +0000, Bshara, Nafea wrote:
> Andrew,
> 
> Sent from my iPhone
> 
> On Jun 3, 2019, at 6:52 PM, Andrew Lunn <andrew@lunn.ch> wrote:
> 
> > > Any "SmartNIC" vendor has temptation of uAPI-level hand off to the
> > > firmware (including my employer), we all run pretty beefy processors
> > > inside "the NIC" after all.  The device centric ethtool configuration
> > > can be implemented by just forwarding the uAPI structures as they are
> > > to the FW.  I'm sure Andrew and others who would like to see Linux
> > > takes more control over PHYs etc. would not like this scenario, either.
> > 
> > No, i would not. There are a few good examples of both firmware and
> > open drivers being used to control the same PHY, on different
> > boards. The PHY driver was developed by the community, and has more
> > features than the firmware driver. And it keeps gaining features. The
> > firmware i stuck, no updates. The community driver can be debugged,
> > the firmware is a black box, no chance of the community fixing any
> > bugs in it.
> > 
> > And PHYs are commodity devices. I doubt there is any value add in the
> > firmware for a PHY, any real IPR which makes the product better, magic
> > sauce related to the PHY. So just save the cost of writing and
> > maintaining firmware, export the MDIO bus, and let Linux control it.
> > Concentrate the engineers on the interesting parts of the NIC, the
> > Smart parts, where there can be real IPR.
> > 
> > And i would say this is true for any NIC. Let Linux control the PHY.
> > 
> >      Andrew
> > 
> 
> It may be true for old GbE PHYs where it’s a discrete chip from the
> likes of Marvell or broadcom
> 
> But at 25/50/100G, the PHy is actually part of the nic. It’s a very
> complex SERDES.  Cloud providers like us spend enormous amount of
> time testing the PHY across process and voltage variations, all cable
> types, length and manufacturing variations, and against all switches
> we use.  Community drivers won’t be able to validate and tune all
> this.
> 
> Plus we would need exact same setting for Linux, including all
> distributions even 10year old like RHEL6, for all Windows, ESX, DPDK,
> FreeBSD,  and support millions of different customers with different
> sets of Machine images. 
> 
> In this case, there is no practical choice by have the firmware to
> manage the PHY

I don't quite know why we're talking about PHYs in this context.
ENA is basically a virtio NIC. It has no PHY.


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jakub Kicinski]

On Tue, 04 Jun 2019 07:57:48 +0100, David Woodhouse wrote:
> On Tue, 2019-06-04 at 02:15 +0000, Bshara, Nafea wrote:
> > On Jun 3, 2019, at 6:52 PM, Andrew Lunn <andrew@lunn.ch> wrote:
> > > > Any "SmartNIC" vendor has temptation of uAPI-level hand off to the
> > > > firmware (including my employer), we all run pretty beefy processors
> > > > inside "the NIC" after all.  The device centric ethtool configuration
> > > > can be implemented by just forwarding the uAPI structures as they are
> > > > to the FW.  I'm sure Andrew and others who would like to see Linux
> > > > takes more control over PHYs etc. would not like this scenario, either.  
> > > 
> > > No, i would not. There are a few good examples of both firmware and
> > > open drivers being used to control the same PHY, on different
> > > boards. The PHY driver was developed by the community, and has more
> > > features than the firmware driver. And it keeps gaining features. The
> > > firmware i stuck, no updates. The community driver can be debugged,
> > > the firmware is a black box, no chance of the community fixing any
> > > bugs in it.
> > > 
> > > And PHYs are commodity devices. I doubt there is any value add in the
> > > firmware for a PHY, any real IPR which makes the product better, magic
> > > sauce related to the PHY. So just save the cost of writing and
> > > maintaining firmware, export the MDIO bus, and let Linux control it.
> > > Concentrate the engineers on the interesting parts of the NIC, the
> > > Smart parts, where there can be real IPR.
> > > 
> > > And i would say this is true for any NIC. Let Linux control the PHY.
> > 
> > It may be true for old GbE PHYs where it’s a discrete chip from the
> > likes of Marvell or broadcom
> > 
> > But at 25/50/100G, the PHy is actually part of the nic. It’s a very
> > complex SERDES.  Cloud providers like us spend enormous amount of
> > time testing the PHY across process and voltage variations, all cable
> > types, length and manufacturing variations, and against all switches
> > we use.  Community drivers won’t be able to validate and tune all
> > this.
> > 
> > Plus we would need exact same setting for Linux, including all
> > distributions even 10year old like RHEL6, for all Windows, ESX, DPDK,
> > FreeBSD,  and support millions of different customers with different
> > sets of Machine images. 
> > 
> > In this case, there is no practical choice by have the firmware to
> > manage the PHY  
> 
> I don't quite know why we're talking about PHYs in this context.
> ENA is basically a virtio NIC. It has no PHY.

I brought it up as an example, to illustrate that we'd rather see less
trust and control being blindly handed over to the firmware.

Would you mind answering what are the examples of very important flags
you need to expose to users with 10yo kernels?  Or any examples for that
matter..

RE: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jubran, Samih]

As of today there are no flags exposed by ENA NIC device, however, we are planning to use them in the near future.
We want to provide customers with extra methods to identify (and differentiate) multiple network interfaces that can be attached to a single VM. 
Currently, customers can identify a specific network interface (ENI) only by MAC address, and later look up this MAC among other multiple ENIs that they have.
In some cases it might not be convenient. Using these flags will let us inform the customer about ENI`s specific properties. It's not finalized,
but tentatively it can look like this:
primary-eni: on /* Differentiate between primary and secondary ENIs */
has-associated-efa: on /* Will specify ENA device has another associated EFA device */

> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Tuesday, June 4, 2019 8:24 PM
> To: David Woodhouse <dwmw2@infradead.org>
> Cc: Bshara, Nafea <nafea@amazon.com>; Andrew Lunn <andrew@lunn.ch>;
> Jubran, Samih <sameehj@amazon.com>; Kiyanovski, Arthur
> <akiyano@amazon.com>; Bshara, Saeed <saeedb@amazon.com>; Tzalik,
> Guy <gtzalik@amazon.com>; Matushevsky, Alexander
> <matua@amazon.com>; Liguori, Anthony <aliguori@amazon.com>; Saidi, Ali
> <alisaidi@amazon.com>; Machulsky, Zorik <zorik@amazon.com>;
> netdev@vger.kernel.org; Wilson, Matt <msw@amazon.com>;
> davem@davemloft.net; Belgazal, Netanel <netanel@amazon.com>;
> Herrenschmidt, Benjamin <benh@amazon.com>
> Subject: Re: [PATCH V2 net 00/11] Extending the ena driver to support new
> features and enhance performance
> 
> On Tue, 04 Jun 2019 07:57:48 +0100, David Woodhouse wrote:
> > On Tue, 2019-06-04 at 02:15 +0000, Bshara, Nafea wrote:
> > > On Jun 3, 2019, at 6:52 PM, Andrew Lunn <andrew@lunn.ch> wrote:
> > > > > Any "SmartNIC" vendor has temptation of uAPI-level hand off to
> > > > > the firmware (including my employer), we all run pretty beefy
> > > > > processors inside "the NIC" after all.  The device centric
> > > > > ethtool configuration can be implemented by just forwarding the
> > > > > uAPI structures as they are to the FW.  I'm sure Andrew and
> > > > > others who would like to see Linux takes more control over PHYs etc.
> would not like this scenario, either.
> > > >
> > > > No, i would not. There are a few good examples of both firmware
> > > > and open drivers being used to control the same PHY, on different
> > > > boards. The PHY driver was developed by the community, and has
> > > > more features than the firmware driver. And it keeps gaining
> > > > features. The firmware i stuck, no updates. The community driver
> > > > can be debugged, the firmware is a black box, no chance of the
> > > > community fixing any bugs in it.
> > > >
> > > > And PHYs are commodity devices. I doubt there is any value add in
> > > > the firmware for a PHY, any real IPR which makes the product
> > > > better, magic sauce related to the PHY. So just save the cost of
> > > > writing and maintaining firmware, export the MDIO bus, and let Linux
> control it.
> > > > Concentrate the engineers on the interesting parts of the NIC, the
> > > > Smart parts, where there can be real IPR.
> > > >
> > > > And i would say this is true for any NIC. Let Linux control the PHY.
> > >
> > > It may be true for old GbE PHYs where it’s a discrete chip from the
> > > likes of Marvell or broadcom
> > >
> > > But at 25/50/100G, the PHy is actually part of the nic. It’s a very
> > > complex SERDES.  Cloud providers like us spend enormous amount of
> > > time testing the PHY across process and voltage variations, all
> > > cable types, length and manufacturing variations, and against all
> > > switches we use.  Community drivers won’t be able to validate and
> > > tune all this.
> > >
> > > Plus we would need exact same setting for Linux, including all
> > > distributions even 10year old like RHEL6, for all Windows, ESX,
> > > DPDK, FreeBSD,  and support millions of different customers with
> > > different sets of Machine images.
> > >
> > > In this case, there is no practical choice by have the firmware to
> > > manage the PHY
> >
> > I don't quite know why we're talking about PHYs in this context.
> > ENA is basically a virtio NIC. It has no PHY.
> 
> I brought it up as an example, to illustrate that we'd rather see less trust and
> control being blindly handed over to the firmware.
> 
> Would you mind answering what are the examples of very important flags
> you need to expose to users with 10yo kernels?  Or any examples for that
> matter..

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jakub Kicinski]

Hi Samih!

Please don't top post on Linux kernel mailing lists.

On Thu, 6 Jun 2019 10:23:40 +0000, Jubran, Samih wrote:
> As of today there are no flags exposed by ENA NIC device, however, we
> are planning to use them in the near future. We want to provide
> customers with extra methods to identify (and differentiate) multiple
> network interfaces that can be attached to a single VM. Currently,
> customers can identify a specific network interface (ENI) only by MAC
> address, and later look up this MAC among other multiple ENIs that
> they have. In some cases it might not be convenient. Using these
> flags will let us inform the customer about ENI`s specific
> properties.

Oh no :(  You're using private _feature_ flags as labels or tags.

> It's not finalized, but tentatively it can look like this: 
> primary-eni: on /* Differentiate between primary and secondary ENIs */

Did you consider using phys_port_name for this use case?

If the intent is to have those interfaces bonded, even better would
be to extend the net_failover module or work on user space ABI for VM
failover.  That might be a bigger effort, though.

> has-associated-efa: on /* Will specify ENA device has another associated EFA device */

IDK how your ENA/EFA thing works, but sounds like something that should
be solved in the device model.

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Bshara, Nafea]

On 6/6/19, 10:16 AM, "Jakub Kicinski" <jakub.kicinski@netronome.com> wrote:

    Hi Samih!
    
    Please don't top post on Linux kernel mailing lists.
    
    On Thu, 6 Jun 2019 10:23:40 +0000, Jubran, Samih wrote:
    > As of today there are no flags exposed by ENA NIC device, however, we
    > are planning to use them in the near future. We want to provide
    > customers with extra methods to identify (and differentiate) multiple
    > network interfaces that can be attached to a single VM. Currently,
    > customers can identify a specific network interface (ENI) only by MAC
    > address, and later look up this MAC among other multiple ENIs that
    > they have. In some cases it might not be convenient. Using these
    > flags will let us inform the customer about ENI`s specific
    > properties.
    
    Oh no :(  You're using private _feature_ flags as labels or tags.
    
    > It's not finalized, but tentatively it can look like this: 
    > primary-eni: on /* Differentiate between primary and secondary ENIs */
    
    Did you consider using phys_port_name for this use case?
    
    If the intent is to have those interfaces bonded, even better would
    be to extend the net_failover module or work on user space ABI for VM
    failover.  That might be a bigger effort, though.
    
    > has-associated-efa: on /* Will specify ENA device has another associated EFA device */
    
    IDK how your ENA/EFA thing works, but sounds like something that should
    be solved in the device model.

[NB] ENA is a netDev,  EFA is an ib_dev.   Both share the same physical network
All previous attempt to make them share at device level leads to a lot of complexity at the OS level, with inter-dependencies that are pronged to error
Not to mention that OS distribution that backport from mainline would backport different subset of each driver,  let alone they belong to two subtrees with two different maintainers and we don’t want to be in a place where we have to coordinate releases between two subgroups

As such, we selected a much safer path, that operational, development and deployment of two different drivers (netdev vs ib_dev) much easier but exposing the nic as two different Physical functions/devices

Only cost we have is that we need this extra propriety to help user identify which two devices are related hence Samih's comments
    

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jakub Kicinski]

On Thu, 6 Jun 2019 21:40:19 +0000, Bshara, Nafea wrote:
> On 6/6/19, 10:16 AM, "Jakub Kicinski" <jakub.kicinski@netronome.com> wrote:
> 
>     Hi Samih!
>     
>     Please don't top post on Linux kernel mailing lists.
>     
>     On Thu, 6 Jun 2019 10:23:40 +0000, Jubran, Samih wrote:
>     > As of today there are no flags exposed by ENA NIC device, however, we
>     > are planning to use them in the near future. We want to provide
>     > customers with extra methods to identify (and differentiate) multiple
>     > network interfaces that can be attached to a single VM. Currently,
>     > customers can identify a specific network interface (ENI) only by MAC
>     > address, and later look up this MAC among other multiple ENIs that
>     > they have. In some cases it might not be convenient. Using these
>     > flags will let us inform the customer about ENI`s specific
>     > properties.  
>     
>     Oh no :(  You're using private _feature_ flags as labels or tags.
>     
>     > It's not finalized, but tentatively it can look like this: 
>     > primary-eni: on /* Differentiate between primary and secondary ENIs */  
>     
>     Did you consider using phys_port_name for this use case?
>     
>     If the intent is to have those interfaces bonded, even better would
>     be to extend the net_failover module or work on user space ABI for VM
>     failover.  That might be a bigger effort, though.

Someone else will address this point?

>     > has-associated-efa: on /* Will specify ENA device has another associated EFA device */  
>     
>     IDK how your ENA/EFA thing works, but sounds like something that should
>     be solved in the device model.
> 
> [NB] ENA is a netDev,  EFA is an ib_dev.   Both share the same physical network
> All previous attempt to make them share at device level leads to a
> lot of complexity at the OS level, with inter-dependencies that are
> pronged to error Not to mention that OS distribution that backport
> from mainline would backport different subset of each driver,  let
> alone they belong to two subtrees with two different maintainers and
> we don’t want to be in a place where we have to coordinate releases
> between two subgroups
>
> As such, we selected a much safer path, that operational, development and deployment of two different drivers (netdev vs ib_dev) much easier but exposing the nic as two different Physical functions/devices
> 
> Only cost we have is that we need this extra propriety to help user identify which two devices are related hence Samih's comments

I think you're arguing with a different point than the one I made :)
Do you think I said they should use the same PCI device?  I haven't.

IIUC you are exposing a "tag" through the feature API on the netdev to
indicate that there is another PCI device present on the system?

What I said is if there is a relation between PCI devices the best
level to expose this relation at is at the device model level.  IOW
have some form on "link" in sysfs, not in a random place on a netdev.

Having said that, it's entirely unclear to me what the user scenario is
here.  You say "which two devices related", yet you only have one bit,
so it can indicate that there is another device, not _which_ device is
related.  Information you can full well get from running lspci 🤷
Do the devices have the same PCI ID/vendor:model?

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Bshara, Nafea]

Sent from my iPhone

> On Jun 6, 2019, at 3:05 PM, Jakub Kicinski <jakub.kicinski@netronome.com> wrote:
> 
>> On Thu, 6 Jun 2019 21:40:19 +0000, Bshara, Nafea wrote:
>> On 6/6/19, 10:16 AM, "Jakub Kicinski" <jakub.kicinski@netronome.com> wrote:
>> 
>>    Hi Samih!
>> 
>>    Please don't top post on Linux kernel mailing lists.
>> 
>>>    On Thu, 6 Jun 2019 10:23:40 +0000, Jubran, Samih wrote:
>>> As of today there are no flags exposed by ENA NIC device, however, we
>>> are planning to use them in the near future. We want to provide
>>> customers with extra methods to identify (and differentiate) multiple
>>> network interfaces that can be attached to a single VM. Currently,
>>> customers can identify a specific network interface (ENI) only by MAC
>>> address, and later look up this MAC among other multiple ENIs that
>>> they have. In some cases it might not be convenient. Using these
>>> flags will let us inform the customer about ENI`s specific
>>> properties.  
>> 
>>    Oh no :(  You're using private _feature_ flags as labels or tags.
>> 
>>> It's not finalized, but tentatively it can look like this: 
>>> primary-eni: on /* Differentiate between primary and secondary ENIs */  
>> 
>>    Did you consider using phys_port_name for this use case?
>> 
>>    If the intent is to have those interfaces bonded, even better would
>>    be to extend the net_failover module or work on user space ABI for VM
>>    failover.  That might be a bigger effort, though.
> 
> Someone else will address this point?
> 
>>> has-associated-efa: on /* Will specify ENA device has another associated EFA device */  
>> 
>>    IDK how your ENA/EFA thing works, but sounds like something that should
>>    be solved in the device model.
>> 
>> [NB] ENA is a netDev,  EFA is an ib_dev.   Both share the same physical network
>> All previous attempt to make them share at device level leads to a
>> lot of complexity at the OS level, with inter-dependencies that are
>> pronged to error Not to mention that OS distribution that backport
>> from mainline would backport different subset of each driver,  let
>> alone they belong to two subtrees with two different maintainers and
>> we don’t want to be in a place where we have to coordinate releases
>> between two subgroups
>> 
>> As such, we selected a much safer path, that operational, development and deployment of two different drivers (netdev vs ib_dev) much easier but exposing the nic as two different Physical functions/devices
>> 
>> Only cost we have is that we need this extra propriety to help user identify which two devices are related hence Samih's comments
> 
> I think you're arguing with a different point than the one I made :)
> Do you think I said they should use the same PCI device?  I haven't.
> 
> IIUC you are exposing a "tag" through the feature API on the netdev to
> indicate that there is another PCI device present on the system?
> 
> What I said is if there is a relation between PCI devices the best
> level to expose this relation at is at the device model level.  IOW
> have some form on "link" in sysfs, not in a random place on a netdev.
> 
> Having said that, it's entirely unclear to me what the user scenario is
> here.  You say "which two devices related", yet you only have one bit,
> so it can indicate that there is another device, not _which_ device is
> related.  Information you can full well get from running lspci 🤷
> Do the devices have the same PCI ID/vendor:model?

Different model id
Will look into sysfs 

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jakub Kicinski]

On Thu, 6 Jun 2019 22:57:21 +0000, Bshara, Nafea wrote:
> > Having said that, it's entirely unclear to me what the user scenario is
> > here.  You say "which two devices related", yet you only have one bit,
> > so it can indicate that there is another device, not _which_ device is
> > related.  Information you can full well get from running lspci 🤷
> > Do the devices have the same PCI ID/vendor:model?  
> 
> Different model id

Okay, then you know which one is which.  Are there multiple ENAs but
one EFA?

> Will look into sysfs 

I still don't understand what is the problem you're trying to solve,
perhaps phys_port_id is the way to go...


The larger point here is that we can't guide you to the right API
unless we know what you're trying to achieve.  And we don't have 
the slightest clue of what're trying to achieve if uAPI is forwarded 
to the device.  

Honestly this is worse, and way more basic than I thought, I think
315c28d2b714 ("net: ena: ethtool: add extra properties retrieval via get_priv_flags")
needs to be reverted.

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Bshara, Nafea]

Sent from my iPhone

> On Jun 6, 2019, at 4:08 PM, Jakub Kicinski <jakub.kicinski@netronome.com> wrote:
> 
> On Thu, 6 Jun 2019 22:57:21 +0000, Bshara, Nafea wrote:
>>> Having said that, it's entirely unclear to me what the user scenario is
>>> here.  You say "which two devices related", yet you only have one bit,
>>> so it can indicate that there is another device, not _which_ device is
>>> related.  Information you can full well get from running lspci 🤷
>>> Do the devices have the same PCI ID/vendor:model?  
>> 
>> Different model id
> 
> Okay, then you know which one is which.  Are there multiple ENAs but
> one EFA?
> 

Yes,  very possible. Very common

Typical use case that instances have one ena for control plane, one for internet facing , and one 100G ena that also have efa capabilities

>> Will look into sysfs 
> 
> I still don't understand what is the problem you're trying to solve,
> perhaps phys_port_id is the way to go...
> 
> 
> The larger point here is that we can't guide you to the right API
> unless we know what you're trying to achieve.  And we don't have 
> the slightest clue of what're trying to achieve if uAPI is forwarded 
> to the device.  
> 
> Honestly this is worse, and way more basic than I thought, I think
> 315c28d2b714 ("net: ena: ethtool: add extra properties retrieval via get_priv_flags")
> needs to be reverted.

Let’s not do that until we finish this discussion and explain the various use cases

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jakub Kicinski]

On Thu, 6 Jun 2019 23:21:25 +0000, Bshara, Nafea wrote:
> > On Jun 6, 2019, at 4:08 PM, Jakub Kicinski <jakub.kicinski@netronome.com> wrote:
> > On Thu, 6 Jun 2019 22:57:21 +0000, Bshara, Nafea wrote:  
> >>> Having said that, it's entirely unclear to me what the user scenario is
> >>> here.  You say "which two devices related", yet you only have one bit,
> >>> so it can indicate that there is another device, not _which_ device is
> >>> related.  Information you can full well get from running lspci 🤷
> >>> Do the devices have the same PCI ID/vendor:model?    
> >> 
> >> Different model id  
> > 
> > Okay, then you know which one is which.  Are there multiple ENAs but
> > one EFA?
> 
> Yes,  very possible. Very common
> 
> Typical use case that instances have one ena for control plane, one
> for internet facing , and one 100G ena that also have efa capabilities

I see, and those are PCI devices..  Some form of platform data would
seem like the best fit to me.  There is something called:

/sys/bus/pci/${dbdf}/label

It seems to come from some ACPI table - DSM maybe?  I think you can put
whatever string you want there 🤔

> >> Will look into sysfs   
> > 
> > I still don't understand what is the problem you're trying to solve,
> > perhaps phys_port_id is the way to go...
> > 
> > 
> > The larger point here is that we can't guide you to the right API
> > unless we know what you're trying to achieve.  And we don't have 
> > the slightest clue of what're trying to achieve if uAPI is forwarded 
> > to the device.  
> > 
> > Honestly this is worse, and way more basic than I thought, I think
> > 315c28d2b714 ("net: ena: ethtool: add extra properties retrieval via get_priv_flags")
> > needs to be reverted.  
> 
> Let’s not do that until we finish this discussion and explain the various use cases

Whatever we decide is the right API for tagging interfaces in a virtual
environment, it's definitely not going to be private feature flags.

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Bshara, Nafea]

Sent from my iPhone

On Jun 6, 2019, at 4:43 PM, Jakub Kicinski <jakub.kicinski@netronome.com> wrote:

>>> Okay, then you know which one is which.  Are there multiple ENAs but
>>> one EFA?
>> 
>> Yes,  very possible. Very common
>> 
>> Typical use case that instances have one ena for control plane, one
>> for internet facing , and one 100G ena that also have efa capabilities
> 
> I see, and those are PCI devices..  Some form of platform data would
> seem like the best fit to me.  There is something called:
> 
> /sys/bus/pci/${dbdf}/label
> 
> It seems to come from some ACPI table - DSM maybe?  I think you can put
> whatever string you want there 🤔

Acpi path won’t work, much of thee interface are hot attached, using native pcie hot plug and acpi won’t be involved. 

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jakub Kicinski]

On Fri, 7 Jun 2019 01:04:14 +0000, Bshara, Nafea wrote:
> On Jun 6, 2019, at 4:43 PM, Jakub Kicinski wrote:
> >>> Okay, then you know which one is which.  Are there multiple ENAs but
> >>> one EFA?
> >> 
> >> Yes,  very possible. Very common
> >> 
> >> Typical use case that instances have one ena for control plane, one
> >> for internet facing , and one 100G ena that also have efa capabilities
> > 
> > I see, and those are PCI devices..  Some form of platform data would
> > seem like the best fit to me.  There is something called:
> > 
> > /sys/bus/pci/${dbdf}/label
> > 
> > It seems to come from some ACPI table - DSM maybe?  I think you can put
> > whatever string you want there 🤔  
> 
> Acpi path won’t work, much of thee interface are hot attached, using
> native pcie hot plug and acpi won’t be involved. 

Perhaps hotplug break DSM, I won't pretend to be an ACPI expert.  So you
can find a way to stuff the label into that file from another source.
There's also VPD, or custom PCI caps, but platform data generally seems
like a better idea.

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jakub Kicinski]

On Thu, 6 Jun 2019 18:14:16 -0700, Jakub Kicinski wrote:
> On Fri, 7 Jun 2019 01:04:14 +0000, Bshara, Nafea wrote:
> > On Jun 6, 2019, at 4:43 PM, Jakub Kicinski wrote:  
> > >>> Okay, then you know which one is which.  Are there multiple ENAs but
> > >>> one EFA?  
> > >> 
> > >> Yes,  very possible. Very common
> > >> 
> > >> Typical use case that instances have one ena for control plane, one
> > >> for internet facing , and one 100G ena that also have efa capabilities  
> > > 
> > > I see, and those are PCI devices..  Some form of platform data would
> > > seem like the best fit to me.  There is something called:
> > > 
> > > /sys/bus/pci/${dbdf}/label
> > > 
> > > It seems to come from some ACPI table - DSM maybe?  I think you can put
> > > whatever string you want there 🤔    
> > 
> > Acpi path won’t work, much of thee interface are hot attached, using
> > native pcie hot plug and acpi won’t be involved.   
> 
> Perhaps hotplug break DSM, I won't pretend to be an ACPI expert.  So you
> can find a way to stuff the label into that file from another source.
> There's also VPD, or custom PCI caps, but platform data generally seems
> like a better idea.

Jiri, do you have any thoughts about using phys_port_name for exposing
topology in virtual environments.  Perhaps that's the best fit on the
netdev side.  It's not like we want any other port name in such case,
and having it automatically appended to the netdev name may be useful.

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Bshara, Nafea]

On 6/7/19, 2:27 PM, "Jakub Kicinski" <jakub.kicinski@netronome.com> wrote:

    On Thu, 6 Jun 2019 18:14:16 -0700, Jakub Kicinski wrote:
    > On Fri, 7 Jun 2019 01:04:14 +0000, Bshara, Nafea wrote:
    > > On Jun 6, 2019, at 4:43 PM, Jakub Kicinski wrote:  
    > > >>> Okay, then you know which one is which.  Are there multiple ENAs but
    > > >>> one EFA?  
    > > >> 
    > > >> Yes,  very possible. Very common
    > > >> 
    > > >> Typical use case that instances have one ena for control plane, one
    > > >> for internet facing , and one 100G ena that also have efa capabilities  
    > > > 
    > > > I see, and those are PCI devices..  Some form of platform data would
    > > > seem like the best fit to me.  There is something called:
    > > > 
    > > > /sys/bus/pci/${dbdf}/label
    > > > 
    > > > It seems to come from some ACPI table - DSM maybe?  I think you can put
    > > > whatever string you want there 🤔    
    > > 
    > > Acpi path won’t work, much of thee interface are hot attached, using
    > > native pcie hot plug and acpi won’t be involved.   
    > 
    > Perhaps hotplug break DSM, I won't pretend to be an ACPI expert.  So you
    > can find a way to stuff the label into that file from another source.
    > There's also VPD, or custom PCI caps, but platform data generally seems
    > like a better idea.
    
    Jiri, do you have any thoughts about using phys_port_name for exposing
    topology in virtual environments.  Perhaps that's the best fit on the
    netdev side.  It's not like we want any other port name in such case,
    and having it automatically appended to the netdev name may be useful.
    
any preference for that vs /sysfs ?

Re: [PATCH V2 net 00/11] Extending the ena driver to support new features and enhance performance

[Jakub Kicinski]

On Fri, 7 Jun 2019 21:34:00 +0000, Bshara, Nafea wrote:
> On 6/7/19, 2:27 PM, "Jakub Kicinski" <jakub.kicinski@netronome.com> wrote:
> 
>     On Thu, 6 Jun 2019 18:14:16 -0700, Jakub Kicinski wrote:
>     > On Fri, 7 Jun 2019 01:04:14 +0000, Bshara, Nafea wrote:  
>     > > On Jun 6, 2019, at 4:43 PM, Jakub Kicinski wrote:    
>     > > >>> Okay, then you know which one is which.  Are there multiple ENAs but
>     > > >>> one EFA?    
>     > > >> 
>     > > >> Yes,  very possible. Very common
>     > > >> 
>     > > >> Typical use case that instances have one ena for control plane, one
>     > > >> for internet facing , and one 100G ena that also have efa capabilities    
>     > > > 
>     > > > I see, and those are PCI devices..  Some form of platform data would
>     > > > seem like the best fit to me.  There is something called:
>     > > > 
>     > > > /sys/bus/pci/${dbdf}/label
>     > > > 
>     > > > It seems to come from some ACPI table - DSM maybe?  I think you can put
>     > > > whatever string you want there 🤔      
>     > > 
>     > > Acpi path won’t work, much of thee interface are hot attached, using
>     > > native pcie hot plug and acpi won’t be involved.     
>     > 
>     > Perhaps hotplug break DSM, I won't pretend to be an ACPI expert.  So you
>     > can find a way to stuff the label into that file from another source.
>     > There's also VPD, or custom PCI caps, but platform data generally seems
>     > like a better idea.  
>     
>     Jiri, do you have any thoughts about using phys_port_name for exposing
>     topology in virtual environments.  Perhaps that's the best fit on the
>     netdev side.  It's not like we want any other port name in such case,
>     and having it automatically appended to the netdev name may be useful.
>     
> any preference for that vs /sysfs ?

I think so.  For the topology (control, internal, external) I feel like
its a reasonable fit, and well supported by udev/systemd.  We use it for
appending port names of the switch today (p0, p1, p2, etc).  Jiri was
working on making the core kernel generate those automatically so lets
give him a chance to speak up.

For the ENA<>EFA link I think you may still need to resort to sysfs,
unless it can be somehow implied by the topology label?