netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH net-next 00/14] ionic: putting ionic on a diet
@ 2024-03-06 23:29 Shannon Nelson
  2024-03-06 23:29 ` [PATCH net-next 01/14] ionic: remove desc, sg_desc and cmb_desc from desc_info Shannon Nelson
                   ` (14 more replies)
  0 siblings, 15 replies; 16+ messages in thread
From: Shannon Nelson @ 2024-03-06 23:29 UTC (permalink / raw)
  To: netdev, davem, kuba, edumazet, pabeni
  Cc: brett.creeley, drivers, Shannon Nelson

Building on the performance work done in the previous patchset
    [Link] https://lore.kernel.org/netdev/20240229193935.14197-1-shannon.nelson@amd.com/
this patchset puts the ionic driver on a diet, decreasing the memory
requirements per queue, and simplifies a few more bits of logic.

We trimmed the queue management structs and gained some ground, but
the most savings came from trimming the individual buffer descriptors.
The original design used a single generic buffer descriptor for Tx, Rx and
Adminq needs, but the Rx and Adminq descriptors really don't need all the
info that the Tx descriptors track.  By splitting up the descriptor types
we can significantly reduce the descriptor sizes for Rx and Adminq use.

There is a small reduction in the queue management structs, saving about
3 cachelines per queuepair:

    ionic_qcq:
	Before:	/* size: 2176, cachelines: 34, members: 23 */
	After:	/* size: 2048, cachelines: 32, members: 23 */

We also remove an array of completion descriptor pointers, or about
8 Kbytes per queue.

But the biggest savings came from splitting the desc_info struct into
queue specific structs and trimming out what was unnecessary.

    Before:
	ionic_desc_info:
		/* size: 496, cachelines: 8, members: 10 */
    After:
	ionic_tx_desc_info:
		/* size: 496, cachelines: 8, members: 6 */
	ionic_rx_desc_info:
		/* size: 224, cachelines: 4, members: 2 */
	ionic_admin_desc_info:
		/* size: 8, cachelines: 1, members: 1 */

In a 64 core host the ionic driver will default to 64 queuepairs of
1024 descriptors for Rx, 1024 for Tx, and 80 for Adminq and Notifyq. 

The total memory usage for 64 queues:
    Before:
	  65 * sizeof(ionic_qcq)			   141,440
	+ 64 * 1024 * sizeof(ionic_desc_info)		32,505,856
	+ 64 * 1024 * sizeof(ionic_desc_info)		32,505,856
	+ 64 * 1024 * 2 * sizeof(ionic_qc_info)		    16,384
	+  1 *   80 * sizeof(ionic_desc_info)		    39,690
							----------
							65,201,038

    After:
	  65 * sizeof(ionic_qcq)			   133,120
	+ 64 * 1024 * sizeof(ionic_tx_desc_info)	32,505,856
	+ 64 * 1024 * sizeof(ionic_rx_desc_info)	14,680,064
	+                           (removed)		         0
	+  1 *   80 * sizeof(ionic_admin desc_info)	       640
							----------
							47,319,680

This saves us approximately 18 Mbytes per port in a 64 core machine,
a 28% savings in our memory needs.

In addition, this improves our simple single thread / single queue
iperf case on a 9100 MTU connection from 86.7 to 95 Gbits/sec.


Shannon Nelson (14):
  ionic: remove desc, sg_desc and cmb_desc from desc_info
  ionic: drop q mapping
  ionic: move adminq-notifyq handling to main file
  ionic: remove callback pointer from desc_info
  ionic: remove the cq_info to save more memory
  ionic: use specialized desc info structs
  ionic: fold adminq clean into service routine
  ionic: refactor skb building
  ionic: carry idev in ionic_cq struct
  ionic: rearrange ionic_qcq
  ionic: rearrange ionic_queue for better layout
  ionic: remove unnecessary NULL test
  ionic: better dma-map error handling
  ionic: keep stats struct local to error handling

 drivers/net/ethernet/pensando/ionic/ionic.h   |   2 +
 .../net/ethernet/pensando/ionic/ionic_dev.c   | 105 +----
 .../net/ethernet/pensando/ionic/ionic_dev.h   |  79 ++--
 .../net/ethernet/pensando/ionic/ionic_lif.c   | 165 +++-----
 .../net/ethernet/pensando/ionic/ionic_lif.h   |   8 +-
 .../net/ethernet/pensando/ionic/ionic_main.c  | 115 ++++-
 .../net/ethernet/pensando/ionic/ionic_txrx.c  | 399 ++++++++----------
 .../net/ethernet/pensando/ionic/ionic_txrx.h  |   2 +-
 8 files changed, 371 insertions(+), 504 deletions(-)

-- 
2.17.1


^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2024-03-08 12:00 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-03-06 23:29 [PATCH net-next 00/14] ionic: putting ionic on a diet Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 01/14] ionic: remove desc, sg_desc and cmb_desc from desc_info Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 02/14] ionic: drop q mapping Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 03/14] ionic: move adminq-notifyq handling to main file Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 04/14] ionic: remove callback pointer from desc_info Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 05/14] ionic: remove the cq_info to save more memory Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 06/14] ionic: use specialized desc info structs Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 07/14] ionic: fold adminq clean into service routine Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 08/14] ionic: refactor skb building Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 09/14] ionic: carry idev in ionic_cq struct Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 10/14] ionic: rearrange ionic_qcq Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 11/14] ionic: rearrange ionic_queue for better layout Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 12/14] ionic: remove unnecessary NULL test Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 13/14] ionic: better dma-map error handling Shannon Nelson
2024-03-06 23:29 ` [PATCH net-next 14/14] ionic: keep stats struct local to " Shannon Nelson
2024-03-08 12:00 ` [PATCH net-next 00/14] ionic: putting ionic on a diet patchwork-bot+netdevbpf

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).