Re: [PATCH] net/tap: allow more that 4 queues

[Ferruh Yigit <ferruh.yigit@amd.com>]

On 3/6/2024 8:21 PM, Stephen Hemminger wrote:
> On Wed, 6 Mar 2024 16:14:51 +0000
> Ferruh Yigit <ferruh.yigit@amd.com> wrote:
> 
>> On 2/29/2024 5:56 PM, Stephen Hemminger wrote:
>>> The tap device needs to exchange file descriptors for tx and rx.
>>> But the EAL MP layer has limit of 8 file descriptors per message.
>>> The ideal resolution would be to increase the number of file
>>> descriptors allowed for rte_mp_sendmsg(), but this would break
>>> the ABI. Workaround the constraint by breaking into multiple messages.
>>>
>>> Do not hide errors about MP message failures.
>>>
>>> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
>>> ---
>>>  drivers/net/tap/rte_eth_tap.c | 40 +++++++++++++++++++++++++++++------
>>>  1 file changed, 33 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
>>> index 69d9da695bed..df18c328f498 100644
>>> --- a/drivers/net/tap/rte_eth_tap.c
>>> +++ b/drivers/net/tap/rte_eth_tap.c
>>> @@ -863,21 +863,44 @@ tap_mp_req_on_rxtx(struct rte_eth_dev *dev)
>>>  		msg.fds[fd_iterator++] = process_private->txq_fds[i];
>>>  		msg.num_fds++;
>>>  		request_param->txq_count++;
>>> +
>>> +		/* Need to break request into chunks */
>>> +		if (fd_iterator >= RTE_MP_MAX_FD_NUM) {
>>> +			err = rte_mp_sendmsg(&msg);
>>> +			if (err < 0)
>>> +				goto fail;
>>> +
>>> +			fd_iterator = 0;
>>> +			msg.num_fds = 0;
>>> +			request_param->txq_count = 0;
>>> +		}
>>>  	}
>>>  	for (i = 0; i < dev->data->nb_rx_queues; i++) {
>>>  		msg.fds[fd_iterator++] = process_private->rxq_fds[i];
>>>  		msg.num_fds++;
>>>  		request_param->rxq_count++;
>>> +
>>> +		if (fd_iterator >= RTE_MP_MAX_FD_NUM) {
>>> +			err = rte_mp_sendmsg(&msg);
>>> +			if (err < 0)
>>> +				goto fail;
>>> +
>>> +			fd_iterator = 0;
>>> +			msg.num_fds = 0;
>>> +			request_param->rxq_count = 0;
>>> +		}
>>>  	}  
>>
>> Hi Stephen,
>>
>> Did you able to verify with more than 4 queues?
>>
>> As far as I can see, in the secondary counterpart of the
>> 'rte_mp_sendmsg()', each time secondary index starts from 0, and
>> subsequent calls overwrites the fds in secondary.
>> So practically still only 4 queues works.
> 
> I got 4 queues setup, but looks like they are trash in secondary.
> Probably best to revert this and fix it by bumping RTE_MP_MAX_FD_NUM.
> This is better, but does take some ABI issue handling.
>

We can increase RTE_MP_MAX_FD_NUM but still there will be a limit.

Can't it be possible to update 'rte_mp_sendmsg()' to support multiple
'rte_mp_sendmsg()' calls in this patch?

Also need to check if fds size is less than 'RTE_PMD_TAP_MAX_QUEUES'
with multiple 'rte_mp_sendmsg()' call support.