Re: [PATCH net] net/mlx5e: Do not recycle pages from emergency reserve

[Jesper Dangaard Brouer <brouer@redhat.com>]

On Sat, 21 Jan 2017 22:31:26 +0200 Saeed Mahameed <saeedm@dev.mellanox.co.il> wrote:

> > My previous measurements show approx 20℅ speedup on a UDP test with delivery
> > to remote CPU.
> >
> > Removing the cache would of cause be a good usecase for speeding up the page
> > allocator (PCP). Which Mel Gorman and me are working on. AFAIK current page
> > order0 cost 240 cycles. Mel have reduced til to 180, and without NUMA 150
> > cycles. And with bulking this can be amortized to 80 cycles.
> >  
> 
> Are you trying to say that we won't need the cache if you manage to
> deliver those optimizations ?

These page alloc optimization are good, but not fast-enough to replace
your cache.  Maybe I can improve it further, but it will be very hard
to compete with a recycle cache (it can skip many checks the page alloc
need, as it knows the page state).

Said in another way, the gap will be significantly smaller, and you
will not see a big boost from using this cache.


BUT there are other advantages of using a guaranteed recycle pool
facility (like the page_pool).  Namely, (1) DMA-overhead: keeping page
DMA mapped to counter DMA+IOMMU overhead, (2) RX-zero-copy: opens up
for a software memory model solution for pre-VMA-mapping pages to
userspace (See: [1] for argument how this avoids leaking kernel mem,
but only expose/leak packet-data mem)


> can you compare those optimizations with the page_frag_cache from
> dev_alloc_skb ?

IHMO the page_frag_cache is indirectly a bulk page allocator facility,
which is limiting the number of times the page allocator is called, and
thus amortize the cost of talking to the page allocator.  Performance
wise, it is likely comparable to your page cache, and likely faster
than the 80 cycles order0-bulking.
BUT we generally worry about using these 32K pages, as it opens a
window for attackers pinning down memory.


[1] https://prototype-kernel.readthedocs.io/en/latest/vm/page_pool/design/memory_model_nic.html
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer