[PATCH] ixgbe: Reduce memory consumption with larger page sizes

[PATCH] ixgbe: Reduce memory consumption with larger page sizes

[Anton Blanchard]

The ixgbe driver allocates pages for its receive rings. It currently
uses 512 pages, regardless of page size. During receive handling it
adds the unused part of the page back into the rx ring, avoiding the
need for a new allocation.

On a ppc64 box with 64 threads and 64kB pages, we end up with
512 entries * 64 rx queues * 64kB = 2GB memory used. Even more of a
concern is that we use up 2GB of IOMMU space in order to map all this
memory.

The driver makes a number of decisions based on if PAGE_SIZE is less
than 8kB, so use this as the breakpoint and only allocate 128 entries
on 8kB or larger page sizes.

Signed-off-by: Anton Blanchard <anton@samba.org>
---

Jeff: The breakpoint and the ring size I chose was pretty arbitrary,
feel free to adjust as you see fit. Our main concern is we get that 2GB
consumption down to something more reasonable :)

Index: b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
===================================================================
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
@@ -67,7 +67,11 @@
 #define IXGBE_MAX_TXD			   4096
 #define IXGBE_MIN_TXD			     64
 
+#if (PAGE_SIZE < 8192)
 #define IXGBE_DEFAULT_RXD		    512
+#else
+#define IXGBE_DEFAULT_RXD		    128
+#endif
 #define IXGBE_MAX_RXD			   4096
 #define IXGBE_MIN_RXD			     64
 

Re: [PATCH] ixgbe: Reduce memory consumption with larger page sizes

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

On Tue, 2013-10-22 at 10:37 +1100, Anton Blanchard wrote:
> The ixgbe driver allocates pages for its receive rings. It currently
> uses 512 pages, regardless of page size. During receive handling it
> adds the unused part of the page back into the rx ring, avoiding the
> need for a new allocation.
> 
> On a ppc64 box with 64 threads and 64kB pages, we end up with
> 512 entries * 64 rx queues * 64kB = 2GB memory used. Even more of a
> concern is that we use up 2GB of IOMMU space in order to map all this
> memory.
> 
> The driver makes a number of decisions based on if PAGE_SIZE is less
> than 8kB, so use this as the breakpoint and only allocate 128 entries
> on 8kB or larger page sizes.
> 
> Signed-off-by: Anton Blanchard <anton@samba.org>
> ---
> 
> Jeff: The breakpoint and the ring size I chose was pretty arbitrary,
> feel free to adjust as you see fit. Our main concern is we get that
> 2GB
> consumption down to something more reasonable :)

Thanks Anton, I will add your patch to my queue.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]