Re: [DPDK/ethdev Bug 1381] TAP device can not support 17 queues

[Stephen Hemminger <stephen@networkplumber.org>]

On Tue, 20 Feb 2024 03:29:04 +0000
bugzilla@dpdk.org wrote:

> https://bugs.dpdk.org/show_bug.cgi?id=1381
> 
>             Bug ID: 1381
>            Summary: TAP device can not support 17 queues
>            Product: DPDK
>            Version: 23.11
>           Hardware: All
>                 OS: All
>             Status: UNCONFIRMED
>           Severity: normal
>           Priority: Normal
>          Component: ethdev
>           Assignee: dev@dpdk.org
>           Reporter: stephen@networkplumber.org
>   Target Milestone: ---
> 
> If you try:
> # dpdk-testpmd --log-level=pmd.net.tap:debug -l 1-2 --vdev=net_tap0 -- -i
> --rxq=8 --txq=8
> 
> It will fail because:
> EAL: Detected CPU lcores: 8
> EAL: Detected NUMA nodes: 1
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> rte_pmd_tap_probe(): Initializing pmd_tap for net_tap0
> eth_dev_tap_create(): TAP device on numa 0
> tun_alloc(): /dev/net/tun Features 00007173
> tun_alloc():   Multi-queue support for 16 queues
> tun_alloc(): Device name is 'dtap0'
> tun_alloc(): Using rt-signal 35
> eth_dev_tap_create(): allocated dtap0
> Interactive-mode selected
> testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
> 
> Warning! port-topology=paired and odd forward ports number, the last port will
> pair with itself.
> 
> Configuring Port 0 (socket 0)
> tap_dev_configure(): net_tap0: dtap0: TX configured queues number: 8
> tap_dev_configure(): net_tap0: dtap0: RX configured queues number: 8
> tun_alloc(): /dev/net/tun Features 00007173
> tun_alloc():   Multi-queue support for 16 queues
> tun_alloc(): Device name is 'dtap0'
> tap_setup_queue(): dtap0: add tx queue for qid 0 fd 26
> tap_tx_queue_setup():   TX TUNTAP device name dtap0, qid 0 on fd 26 csum off
> tun_alloc(): /dev/net/tun Features 00007173
> tun_alloc():   Multi-queue support for 16 queues
> tun_alloc(): Device name is 'dtap0'
> tap_setup_queue(): dtap0: add tx queue for qid 1 fd 212
> tap_tx_queue_setup():   TX TUNTAP device name dtap0, qid 1 on fd 212 csum off
> tun_alloc(): /dev/net/tun Features 00007173
> tun_alloc():   Multi-queue support for 16 queues
> tun_alloc(): Device name is 'dtap0'
> tap_setup_queue(): dtap0: add tx queue for qid 2 fd 213
> tap_tx_queue_setup():   TX TUNTAP device name dtap0, qid 2 on fd 213 csum off
> tun_alloc(): /dev/net/tun Features 00007173
> tun_alloc():   Multi-queue support for 16 queues
> tun_alloc(): Device name is 'dtap0'
> tap_setup_queue(): dtap0: add tx queue for qid 3 fd 214
> tap_tx_queue_setup():   TX TUNTAP device name dtap0, qid 3 on fd 214 csum off
> tun_alloc(): /dev/net/tun Features 00007173
> tun_alloc():   Multi-queue support for 16 queues
> tun_alloc(): Device name is 'dtap0'
> tap_setup_queue(): dtap0: add tx queue for qid 4 fd 215
> tap_tx_queue_setup():   TX TUNTAP device name dtap0, qid 4 on fd 215 csum off
> tun_alloc(): /dev/net/tun Features 00007173
> tun_alloc():   Multi-queue support for 16 queues
> tun_alloc(): Device name is 'dtap0'
> tap_setup_queue(): dtap0: add tx queue for qid 5 fd 216
> tap_tx_queue_setup():   TX TUNTAP device name dtap0, qid 5 on fd 216 csum off
> tun_alloc(): /dev/net/tun Features 00007173
> tun_alloc():   Multi-queue support for 16 queues
> tun_alloc(): Device name is 'dtap0'
> tap_setup_queue(): dtap0: add tx queue for qid 6 fd 217
> tap_tx_queue_setup():   TX TUNTAP device name dtap0, qid 6 on fd 217 csum off
> tun_alloc(): /dev/net/tun Features 00007173
> tun_alloc():   Multi-queue support for 16 queues
> tun_alloc(): Device name is 'dtap0'
> tap_setup_queue(): dtap0: add tx queue for qid 7 fd 218
> tap_tx_queue_setup():   TX TUNTAP device name dtap0, qid 7 on fd 218 csum off
> tap_setup_queue(): dtap0: dup fd 26 for rx queue qid 0 (219)
> tap_rx_queue_setup():   RX TUNTAP device name dtap0, qid 0 on fd 219
> tap_setup_queue(): dtap0: dup fd 212 for rx queue qid 1 (220)
> tap_rx_queue_setup():   RX TUNTAP device name dtap0, qid 1 on fd 220
> tap_setup_queue(): dtap0: dup fd 213 for rx queue qid 2 (221)
> tap_rx_queue_setup():   RX TUNTAP device name dtap0, qid 2 on fd 221
> tap_setup_queue(): dtap0: dup fd 214 for rx queue qid 3 (222)
> tap_rx_queue_setup():   RX TUNTAP device name dtap0, qid 3 on fd 222
> tap_setup_queue(): dtap0: dup fd 215 for rx queue qid 4 (223)
> tap_rx_queue_setup():   RX TUNTAP device name dtap0, qid 4 on fd 223
> tap_setup_queue(): dtap0: dup fd 216 for rx queue qid 5 (224)
> tap_rx_queue_setup():   RX TUNTAP device name dtap0, qid 5 on fd 224
> tap_setup_queue(): dtap0: dup fd 217 for rx queue qid 6 (225)
> tap_rx_queue_setup():   RX TUNTAP device name dtap0, qid 6 on fd 225
> tap_setup_queue(): dtap0: dup fd 218 for rx queue qid 7 (226)
> tap_rx_queue_setup():   RX TUNTAP device name dtap0, qid 7 on fd 226
> EAL: Cannot send more than 8 FDs
> tap_mp_req_on_rxtx(): Failed to send start req to secondary 7
> 
> This is a regression caused by:
> commit c36ce7099c2187926cd62cff7ebd479823554929
> Author: Kumara Parameshwaran <kumaraparamesh92@gmail.com>
> Date:   Mon Jan 31 20:02:34 2022 +0530
> 
>     net/tap: fix to populate FDs in secondary process
> 
>     When a tap device is hotplugged to primary process which in turn
>     adds the device to all secondary process, the secondary process
>     does a tap_mp_attach_queues, but the fds are not populated in
>     the primary during the probe they are populated during the queue_setup,
>     added a fix to sync the queues during rte_eth_dev_start
> 
>     Fixes: 4852aa8f6e21 ("drivers/net: enable hotplug on secondary process")
>     Cc: stable@dpdk.org
> 
>     Signed-off-by: Kumara Parameshwaran <kparameshwar@vmware.com>
>     Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
> 

The number of file descriptors allowed in rte_mp_msg should be much larger.
The Linux kernel has an upper limit SCM_MAX_FD which is 253 (see net/scm.h)

But fixing this will break ABI because rte_mp_msg structure was exposed
in rte_eal.h.  It should have been internal!

Alternatively, since fds[] is the last field in rte_mp_msg, and num_fds is
also there, fds[] could have been a flexible array, rather than hard coded.