xdpsock problem testing multiple queues

xdpsock problem testing multiple queues

[Július Milan]

Hi all

We are trying to test multiple RX queues with sample program xdpsock from kernel on vmware virtual machine with 2 queues.
The driver on the NIC is:
# ethtool -i ens192
driver: vmxnet3
version: 1.4.16.0-k-NAPI

NIC has 2 queues, I can check it by ethtool -S.

But when I try to use queue 1, I am getting following:
# ./xdpsock -i ens192 -q 1
/home/jmilan/ws/pt-xdp/linux/samples/bpf/xdpsock_user.c:xsk_configure_socket:315: errno: 1/"Operation not permitted"

Any ideas what the problem could be? Maybe vmxnet3 driver does not support some necessary operations related to queues?

Best Regards
Július

Re: xdpsock problem testing multiple queues

[Pavel Popa]

I don't think vmxnet3 has XDP driver support atm, thus just generic
(i.e. skb) mode. Anyone confirming?


Il giorno mer 2 ott 2019 alle ore 14:11 Július Milan
<Julius.Milan@pantheon.tech> ha scritto:
>
> Hi all
>
> We are trying to test multiple RX queues with sample program xdpsock from kernel on vmware virtual machine with 2 queues.
> The driver on the NIC is:
> # ethtool -i ens192
> driver: vmxnet3
> version: 1.4.16.0-k-NAPI
>
> NIC has 2 queues, I can check it by ethtool -S.
>
> But when I try to use queue 1, I am getting following:
> # ./xdpsock -i ens192 -q 1
> /home/jmilan/ws/pt-xdp/linux/samples/bpf/xdpsock_user.c:xsk_configure_socket:315: errno: 1/"Operation not permitted"
>
> Any ideas what the problem could be? Maybe vmxnet3 driver does not support some necessary operations related to queues?
>
> Best Regards
> Július

Re: xdpsock problem testing multiple queues

[Björn Töpel]

On Wed, 2 Oct 2019 at 14:11, Július Milan <Julius.Milan@pantheon.tech> wrote:
>
> Hi all
>
> We are trying to test multiple RX queues with sample program xdpsock from kernel on vmware virtual machine with 2 queues.
> The driver on the NIC is:
> # ethtool -i ens192
> driver: vmxnet3
> version: 1.4.16.0-k-NAPI
>
> NIC has 2 queues, I can check it by ethtool -S.
>
> But when I try to use queue 1, I am getting following:
> # ./xdpsock -i ens192 -q 1
> /home/jmilan/ws/pt-xdp/linux/samples/bpf/xdpsock_user.c:xsk_configure_socket:315: errno: 1/"Operation not permitted"
>
> Any ideas what the problem could be? Maybe vmxnet3 driver does not support some necessary operations related to queues?
>

XDP support is missing for that driver, but the XDP_SKB/generic mode
is available, and should work.

Can you run the xdp1 application in the samples directory, to rule out
that you can run XDP.


Björn

> Best Regards
> Július

Re: xdpsock problem testing multiple queues

[Maciej Fijalkowski]

On Wed, 2 Oct 2019 15:52:04 +0200
Björn Töpel <bjorn.topel@gmail.com> wrote:

> On Wed, 2 Oct 2019 at 14:11, Július Milan <Julius.Milan@pantheon.tech> wrote:
> >
> > Hi all
> >
> > We are trying to test multiple RX queues with sample program xdpsock from kernel on vmware virtual machine with 2 queues.
> > The driver on the NIC is:
> > # ethtool -i ens192
> > driver: vmxnet3
> > version: 1.4.16.0-k-NAPI
> >
> > NIC has 2 queues, I can check it by ethtool -S.
> >
> > But when I try to use queue 1, I am getting following:
> > # ./xdpsock -i ens192 -q 1
> > /home/jmilan/ws/pt-xdp/linux/samples/bpf/xdpsock_user.c:xsk_configure_socket:315: errno: 1/"Operation not permitted"
> >
> > Any ideas what the problem could be? Maybe vmxnet3 driver does not support some necessary operations related to queues?
> >  
> 
> XDP support is missing for that driver, but the XDP_SKB/generic mode
> is available, and should work.
> 
> Can you run the xdp1 application in the samples directory, to rule out
> that you can run XDP.

xdp1 should work fine but I think the reason for a reported failure is because
vmxnet3 driver doesn't expose the {set,get}_channels ethtool API (at least I
don't see it?) which is used by libbpf's xsk part for querying the underlying
driver how many queues does it has and use that info for creating that many
entries in XSKMAP.

So in that case you're limited to use queue 0 because the XSKMAP map has
only single entry.

Maciej

> 
> 
> Björn
> 
> > Best Regards
> > Július  

RE: xdpsock problem testing multiple queues

[Július Milan]

Many Thanks folks

>On Wed, 2 Oct 2019 15:52:04 +0200
>Björn Töpel <bjorn.topel@gmail.com> wrote:
>
>>On Wed, 2 Oct 2019 at 14:11, Július Milan <Julius.Milan@pantheon.tech> wrote:
>>>
>>> Hi all
>>>
>>> We are trying to test multiple RX queues with sample program xdpsock from kernel on vmware virtual machine with 2 queues.
>>> The driver on the NIC is:
>>> # ethtool -i ens192
>>> driver: vmxnet3
>>> version: 1.4.16.0-k-NAPI
>>>
>>> NIC has 2 queues, I can check it by ethtool -S.
>>>
>>> But when I try to use queue 1, I am getting following:
>>> # ./xdpsock -i ens192 -q 1
>>> /home/jmilan/ws/pt-xdp/linux/samples/bpf/xdpsock_user.c:xsk_configure_socket:315: errno: 1/"Operation not permitted"
>>>
>>> Any ideas what the problem could be? Maybe vmxnet3 driver does not support some necessary operations related to queues?
>>>
>>
>> XDP support is missing for that driver, but the XDP_SKB/generic mode 
>> is available, and should work.
>>
>> Can you run the xdp1 application in the samples directory, to rule out 
>> that you can run XDP.
>
> Xdp1 should work fine but I think the reason for a reported failure is because
> Vmxnet3 driver doesn't expose the {set,get}_channels ethtool API (at least I don't see it?) which is used by libbpf's xsk part for querying the underlying driver how many queues does it has and use that info for creating that many entries in XSKMAP.
>
> So in that case you're limited to use queue 0 because the XSKMAP map has only single entry.
>
> Maciej
>
>>
>>
>> Björn
>>
>>> Best Regards
>>> Július

Re: xdpsock problem testing multiple queues

[Maciej Fijalkowski]

On Wed, 2 Oct 2019 18:28:57 +0200
Maciej Fijalkowski <maciejromanfijalkowski@gmail.com> wrote:

> On Wed, 2 Oct 2019 15:52:04 +0200
> Björn Töpel <bjorn.topel@gmail.com> wrote:
> 
> > On Wed, 2 Oct 2019 at 14:11, Július Milan <Julius.Milan@pantheon.tech> wrote:
> > >
> > > Hi all
> > >
> > > We are trying to test multiple RX queues with sample program xdpsock from kernel on vmware virtual machine with 2 queues.
> > > The driver on the NIC is:
> > > # ethtool -i ens192
> > > driver: vmxnet3
> > > version: 1.4.16.0-k-NAPI
> > >
> > > NIC has 2 queues, I can check it by ethtool -S.
> > >
> > > But when I try to use queue 1, I am getting following:
> > > # ./xdpsock -i ens192 -q 1
> > > /home/jmilan/ws/pt-xdp/linux/samples/bpf/xdpsock_user.c:xsk_configure_socket:315: errno: 1/"Operation not permitted"
> > >
> > > Any ideas what the problem could be? Maybe vmxnet3 driver does not support some necessary operations related to queues?
> > >  
> > 
> > XDP support is missing for that driver, but the XDP_SKB/generic mode
> > is available, and should work.
> > 
> > Can you run the xdp1 application in the samples directory, to rule out
> > that you can run XDP.
> 
> xdp1 should work fine but I think the reason for a reported failure is because
> vmxnet3 driver doesn't expose the {set,get}_channels ethtool API (at least I
> don't see it?) which is used by libbpf's xsk part for querying the underlying
> driver how many queues does it has and use that info for creating that many
> entries in XSKMAP.
> 
> So in that case you're limited to use queue 0 because the XSKMAP map has
> only single entry.

After putting some thoughts into it, I am wondering whether we really need to
query the driver for its max queue count via ethtool during the XSKMAP creation.

eBPF resources are being created *after* the bind() syscall, so at that time if
it succeeded we are sure that the queue id provided by user is valid, no?
Network drivers are usually allocating queues per each CPU on the system, so
IMO using the libbpf_num_possible_cpus() for XSKMAP entries would be just fine.
Obviously, mlx5 is a special case here, but we can double the value we got in
case the xsk->queue_id is higher than the cpu count.

Thoughts?

> 
> Maciej
> 
> > 
> > 
> > Björn
> > 
> > > Best Regards
> > > Július  
>