Re: AF_XDP umem and jumbo frames?

[Jesper Dangaard Brouer <brouer@redhat.com>]

On Fri, 5 Oct 2018 11:47:25 -0700 Zvi Effron <zeffron@riotgames.com> wrote:

> If the requirement is just for contiguous memory, could this be
> resolved with allowing the driver to request multiple contiguous 4KB
> pages instead of one higher order page? Does that reduce the cost? (Or
> does it actually increase it?)

Sorry, but your question does not make sense. A higher order page _is_
multiple contiguous 4KB pages. Thus, the answer is that it is the same.

--Jesper

> On Thu, Oct 4, 2018 at 12:45 PM Jesper Dangaard Brouer
> <brouer@redhat.com> wrote:
> >
> > On Thu, 4 Oct 2018 08:47:45 -0700
> > Rob Sherwood <rob.sherwood@gmail.com> wrote:
> >  
> > > [not speaking for my current employer, but just from past experience ]
> > >
> > > Certainly a lot of the 'hard' requirements (hard meaning - "without
> > > this it won't work")  I've seen could be served with a ~3k non-full
> > > jumbo frame.  
> >
> > Glad to hear that _most_ use-cases can be solved with a ~3k non-full
> > jumbo-frame.
> >  
> > > But at least what I've seen in the past was that because
> > > many of the host-side operations are per-packet limited (e.g., because
> > > of CPU or RAM, but ultimately turns into a max pps per host), a
> > > trivial way to increase application performance/reduce CPU for
> > > networking was to run at as large a frame size as possible.  For
> > > example, if your application/host is really pps limited, then getting
> > > the frame size to increase from 3k to 9k means either 3x more
> > > bandwidth for the same cpu usage (assuming the application is
> > > bandwidth limited) or 1/3x the CPU usage for the same bandwidth (if
> > > the application is not bandwidth limited).  Either way, IMHO it's a
> > > pretty big win.  
> >
> > With XDP we have basically solved the issue of being PPS (packets per
> > sec) limited.  And we can avoid these workarounds of using jumbo frames.
> > That is why it is a bit provoking to ask for jumbo-frames ;-)
> >
> >
> > People on this list might not realize that there is a significant
> > overhead in supporting larger that 4K frames for XDP, that is larger
> > than one memory-page. So let me explain...
> >
> > It is actually trivially easy for XDP to support jumbo frames, if the
> > NIC hardware supports storing RX frames into higher order pages (aka
> > compound pages, more 4K pages physically after each-other) which most
> > HW does. (Page order0 = 4KB, order1=8KB, order2=16KB, order3=32KB).
> > As then XDP will work out-of-the-box, as the requirement is really that
> > packet-payload is layout as phys continuous memory.
> >
> > Kernel page allocator can give us high-order pages, sure, but is cost
> > more, see slide 12 of [1].  The large jump to order-1, is because
> > order-0 have a Per-Cpu-Pages (PCP) cache.  From order-1 and above, the
> > page allocator goes through a central (per NUMA) lock, which makes
> > thing even worse, as this does not scale to multiple CPUs.  And there
> > is also the point of wasting memory when processing 64Byte packets.
> > So, it is not 100% of the picture, that we could support jumbo-frames
> > for XDP.  Mostly because we can workaround this cost/issue, by having
> > recycle caches for these pages, which we even do for order-0 pages.
> > Hint, I actually left this door open, as you can specify page-order
> > when setting up the page_pool API in the driver...
> >
> > [1] http://people.netfilter.org/hawk/presentations/MM-summit2017/MM-summit2017-JesperBrouer.pdf
> >
> > --
> > Best regards,
> >   Jesper Dangaard Brouer
> >   MSc.CS, Principal Kernel Engineer at Red Hat
> >   LinkedIn: http://www.linkedin.com/in/brouer
> >
> >
> >
> >  
> > > On Thu, Oct 4, 2018 at 12:52 AM Jesper Dangaard Brouer
> > > <brouer@redhat.com> wrote:  
> > > >
> > > > On Thu, 4 Oct 2018 08:44:27 +0200
> > > > Björn Töpel <bjorn.topel@gmail.com> wrote:
> > > >  
> > > > > Den tors 27 sep. 2018 kl 02:56 skrev Rob Sherwood <rob.sherwood@gmail.com>:  
> > > > > >
> > > > > > Thanks for the reference and the page-per-packet point makes sense.
> > > > > > At the same time, not supporting jumbo frames seems like a non-trivial
> > > > > > limitation.  Are there a subset of drivers that do support jumbo
> > > > > > frames (or LRO or the other features that require multiple pages per
> > > > > > packet)?
> > > > > >  
> > > > >
> > > > > No, not at the moment. XDP has a strict "one frame cannot exceed a
> > > > > page" constraint. Everything that applies to XDP in terms of
> > > > > constraints, applies to AF_XDP as well.
> > > > >
> > > > > Just to clarify, XDP supports jumbo frames -- i.e. larger than 1500B
> > > > > payload, just not the maximum 9000B size. My personal observation is
> > > > > that many deployments that "require jumbo frames", are usually OK with
> > > > > an of MTU ~3000B. Jumbo frames, yes. Full jumbo frames, no. :-)  
> > > >
> > > > Thank you for clarifying that Bjørn.
> > > >
> > > > Can Alex or Rob explain:
> > > >
> > > > (1) What is your use-case for wanting jumbo-frames?
> > > >
> > > > And (2) will an MTU of ~3000Bytes be sufficient? (which XDP does support)
> > > >
> > > >  
> > > > > > On Tue, Sep 25, 2018 at 9:44 AM Alex Forster <aforster@cloudflare.com> wrote:  
> > > > > > >  
> > > > > > > > On my test box running 4.18 if XDP is in use the MTU can not be
> > > > > > > > set higher than 3050.  
> > > > > > >
> > > > > > > Ah, that answers a few questions for me. Thanks!
> > > > > > >
> > > > > > > Alex Forster  
> > > >
> > > > --
> > > > Best regards,
> > > >   Jesper Dangaard Brouer
> > > >   MSc.CS, Principal Kernel Engineer at Red Hat
> > > >   LinkedIn: http://www.linkedin.com/in/brouer  
> >
> >  



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer