Re: [PATCHv15 bpf-next 1/6] bpf: run devmap xdp_prog on flush instead of bulk enqueue

Hangbin Liu
2021-01-21T04:39:10Z

Hi Maciej,
On Wed, Jan 20, 2021 at 11:42:38PM +0100, Maciej Fijalkowski wrote:
> > +static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog,
> > + struct xdp_frame **frames, int n,
> > + struct net_device *dev)
> > +{
> > + struct xdp_txq_info txq = { .dev = dev };
> > + struct xdp_buff xdp;
> > + int i, nframes = 0;
> > +
> > + for (i = 0; i < n; i++) {
> > + struct xdp_frame *xdpf = frames[i];
> > + u32 act;
> > + int err;
> > +
> > + xdp_convert_frame_to_buff(xdpf, &xdp);
> > + xdp.txq = &txq;
> > +
> > + act = bpf_prog_run_xdp(xdp_prog, &xdp);
> > + switch (act) {
> > + case XDP_PASS:
> > + err = xdp_update_frame_from_buff(&xdp, xdpf);
>
> Bump on John's question.

Hi Jesper, would you please help answer John's question?
> >
> > - sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags);
> > + /* Init sent to cnt in case there is no xdp_prog */
> > + sent = cnt;
> > + if (bq->xdp_prog) {
> > + sent = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev);
> > + if (!sent)
> > + goto out;
>
> Sorry, but 'sent' is a bit confusing to me, actual sending happens below
> via ndo_xdp_xmit, right? This hook will not actually send frames.
> Can we do a subtle change to have it in separate variable 'to_send' ?

Makes sense to me.
>
> Although I'm a huge goto advocate, I feel like this particular usage could
> be simplified. Not sure why we had that in first place.
>
> I gave a shot at rewriting/refactoring whole bq_xmit_all and I feel like
> it's more readable. I introduced 'to_send' variable and got rid of 'error'
> label.
>
> Thoughts?
>
> I might have missed something, though.
>
> static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
> {
> struct net_device *dev = bq->dev;
> unsigned int cnt = bq->count;
> int drops = 0, err = 0;
> int to_send = 0;

The to_send also need to init to cnt.

> int sent = cnt;
> int i;
>
> if (unlikely(!cnt))
> return;
>
> for (i = 0; i < cnt; i++) {
> struct xdp_frame *xdpf = bq->q[i];
>
> prefetch(xdpf);
> }
>
> if (bq->xdp_prog) {
> to_send = dev_map_bpf_prog_run(bq->xdp_prog, bq->q, cnt, dev);
> if (!to_send) {
> sent = 0;
> goto out;
> }
> }
>
> drops = cnt - to_send;

This line could move in to the xdp_prog brackets to save time when no xdp_prog.

if (bq->xdp_prog) {
to_send = ...
if (!to_send) {
...
}
drops = cnt - to_send;
}

> sent = dev->netdev_ops->ndo_xdp_xmit(dev, to_send, bq->q, flags);

If we don't have xdp_prog, the to_send should be cnt.

> if (sent < 0) {
> err = sent;
> sent = 0;
>
> /* If ndo_xdp_xmit fails with an errno, no frames have been
> * xmit'ed and it's our responsibility to them free all.
> */
> for (i = 0; i < cnt - drops; i++) {
> struct xdp_frame *xdpf = bq->q[i];
>
> xdp_return_frame_rx_napi(xdpf);
> }
> }
> out:
> drops = cnt - sent;
> bq->count = 0;
>
> trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err);
> bq->dev_rx = NULL;
> bq->xdp_prog = NULL;
> __list_del_clearprev(&bq->flush_node);
>
> return;
> }

Thanks for your code, looks much clear now.

Hangbin