Re: error loading xdp program on virtio nic

Re: error loading xdp program on virtio nic

[Jesper Dangaard Brouer]

On Fri, 22 Nov 2019 08:43:50 -0700
David Ahern <dsahern@gmail.com> wrote:

> On 11/21/19 11:09 PM, Jason Wang wrote:
> >> Doubling the number of queues for each tap device adds overhead to the
> >> hypervisor if you only want to allow XDP_DROP or XDP_DIRECT. Am I
> >> understanding that correctly?  
> > 
> > 
> > Yes, but there's almost impossible to know whether or not XDP_TX will be
> > used by the program. If we don't use per CPU TX queue, it must be
> > serialized through locks, not sure it's worth try that (not by default,
> > of course).
> >   
> 
> This restriction is going to prevent use of XDP in VMs in general cloud
> hosting environments. 2x vhost threads for vcpus is a non-starter.
> 
> If one XDP feature has high resource needs, then we need to subdivide
> the capabilities to let some work and others fail. For example, a flag
> can be added to xdp_buff / xdp_md that indicates supported XDP features.
> If there are insufficient resources for XDP_TX, do not show support for
> it. If a program returns XDP_TX anyways, packets will be dropped.
> 

This sounds like concrete use-case and solid argument why we need XDP
feature detection and checks. (Last part of LPC talk[1] were about
XDP features).

An interesting perspective you bring up, is that XDP features are not
static per device driver.  It actually needs to be dynamic, as your
XDP_TX feature request depend on the queue resources available.

Implementation wise, I would not add flags to xdp_buff / xdp_md.
Instead I propose in[1] slide 46, that the verifier should detect the
XDP features used by a BPF-prog.  If you XDP prog doesn't use e.g.
XDP_TX, then you should be allowed to run it on a virtio_net device
with less queue configured, right?


[1] http://people.netfilter.org/hawk/presentations/LinuxPlumbers2019/xdp-distro-view.pdf
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: error loading xdp program on virtio nic

[David Ahern]

On 11/22/19 9:57 AM, Jesper Dangaard Brouer wrote:
> Implementation wise, I would not add flags to xdp_buff / xdp_md.
> Instead I propose in[1] slide 46, that the verifier should detect the
> XDP features used by a BPF-prog.  If you XDP prog doesn't use e.g.
> XDP_TX, then you should be allowed to run it on a virtio_net device
> with less queue configured, right?

Thanks for the reference and yes, that is the goal: allow XDP in the
most use cases possible. e.g., Why limit XDP_DROP which requires no
resources because XDP_TX does not work?

I agree a flag in the api is an ugly way to allow it. For the verifier
approach, you mean add an internal flag (e.g., bitmask of return codes)
that the program uses and the NIC driver can check at attach time?

Re: error loading xdp program on virtio nic

[Toke Høiland-Jørgensen]

David Ahern <dsahern@gmail.com> writes:

> On 11/22/19 9:57 AM, Jesper Dangaard Brouer wrote:
>> Implementation wise, I would not add flags to xdp_buff / xdp_md.
>> Instead I propose in[1] slide 46, that the verifier should detect the
>> XDP features used by a BPF-prog.  If you XDP prog doesn't use e.g.
>> XDP_TX, then you should be allowed to run it on a virtio_net device
>> with less queue configured, right?
>
> Thanks for the reference and yes, that is the goal: allow XDP in the
> most use cases possible. e.g., Why limit XDP_DROP which requires no
> resources because XDP_TX does not work?
>
> I agree a flag in the api is an ugly way to allow it. For the verifier
> approach, you mean add an internal flag (e.g., bitmask of return codes)
> that the program uses and the NIC driver can check at attach time?

Yes, that's more or less what we've discussed. With the actual set of
flags, and the API for the driver (new ndo?) TBD. Suggestions welcome; I
anticipate this is something Jesper and I need to circle back to soonish
in any case (unless someone beats us to it!).

-Toke

Re: error loading xdp program on virtio nic

[Jason Wang]

On 2019/11/23 上午12:57, Jesper Dangaard Brouer wrote:
> On Fri, 22 Nov 2019 08:43:50 -0700
> David Ahern <dsahern@gmail.com> wrote:
>
>> On 11/21/19 11:09 PM, Jason Wang wrote:
>>>> Doubling the number of queues for each tap device adds overhead to the
>>>> hypervisor if you only want to allow XDP_DROP or XDP_DIRECT. Am I
>>>> understanding that correctly?
>>>
>>> Yes, but there's almost impossible to know whether or not XDP_TX will be
>>> used by the program. If we don't use per CPU TX queue, it must be
>>> serialized through locks, not sure it's worth try that (not by default,
>>> of course).
>>>    
>> This restriction is going to prevent use of XDP in VMs in general cloud
>> hosting environments. 2x vhost threads for vcpus is a non-starter.
>>
>> If one XDP feature has high resource needs, then we need to subdivide
>> the capabilities to let some work and others fail. For example, a flag
>> can be added to xdp_buff / xdp_md that indicates supported XDP features.
>> If there are insufficient resources for XDP_TX, do not show support for
>> it. If a program returns XDP_TX anyways, packets will be dropped.
>>
> This sounds like concrete use-case and solid argument why we need XDP
> feature detection and checks. (Last part of LPC talk[1] were about
> XDP features).
>
> An interesting perspective you bring up, is that XDP features are not
> static per device driver.  It actually needs to be dynamic, as your
> XDP_TX feature request depend on the queue resources available.
>
> Implementation wise, I would not add flags to xdp_buff / xdp_md.
> Instead I propose in[1] slide 46, that the verifier should detect the
> XDP features used by a BPF-prog.  If you XDP prog doesn't use e.g.
> XDP_TX, then you should be allowed to run it on a virtio_net device
> with less queue configured, right?


Yes, I think so. But I remember we used to have something like 
header_adjust in the past but finally removed ...

Thanks


>
>
> [1] http://people.netfilter.org/hawk/presentations/LinuxPlumbers2019/xdp-distro-view.pdf