Re: [PATCH RFC net-next 2/3] crypto: caam/qi2 - skip isolated CPUs when setting up queue pairs

[Josua Mayer <josua@solid-run.com>]

Hi Ioana,

Thank you for taking a look at my patch!

On 10/6/26 11:27, Ioana Ciornei wrote:
> On Wed, Sep 30, 2026 at 06:54:59PM +0200, Josua Mayer wrote:
>> The driver looks up the affine DPIO and sets up one Rx/Tx queue pair for
>> each online CPU, up to the number of queues. Response notifications and
>> NAPI polling are handled on the DPIO object's assigned CPU. All online
>> may enqueue requests.
>>
>> When isolcpus=managed_irq is used to reserve CPUs for latency-sensitive
>> applications, those CPUs are expected to be free of routine device
>> interrupts.
>>
>> The DPIO driver assigns affine CPUs only from the HK_TYPE_MANAGED_IRQ
>> housekeeping mask, leaving isolated CPUs without an affine DPIO. This
>> defers caam qi2 probe indefinitely once DPIO setup reaches an isolated
>> CPU.
>>
>> Restrict queue pairs, notifications and NAPI polling to the online CPUs
>> in the HK_TYPE_MANAGED_IRQ housekeeping mask, so that isolated CPUs are
>> skipped and don't defer probe. Isolated CPUs may still enqueue requests,
>> with responses handled on housekeeping CPUs.
>
> Don't you need to also change the for_each_online_cpu() from
> dpaa2_dpseci_setup() to take into account only the housekeeping
> managed_irq cores?
I initially thought no, because of the "Allow all cores to enqueue"
comment in code.
>
>
> Something like:
>
> --- a/drivers/crypto/caam/caamalg_qi2.c
> +++ b/drivers/crypto/caam/caamalg_qi2.c
> @@ -5101,7 +5101,7 @@ static int __cold dpaa2_dpseci_setup(struct fsl_mc_device *ls_dev)
>          }
>
>          i = 0;
> -       for_each_online_cpu(cpu) {
> +       for_each_cpu_and(cpu, cpu_online_mask, hk_mask) {
>                  u8 j;
>
>                  j = i % priv->num_pairs;

Correct.

The loop is doing two things in one: assign tx queue to each cpu,
and assign rx queue to each cpu.

Only rx side is an issue for isolcpus. Do you have any idea why each cpu
got a tx queue assigned, and if this may be changed?


Here is a copy of the complete loop, with inline comments on the
control flow:

> 	i = 0;
> 	for_each_online_cpu(cpu) {
> 		u8 j;
>
> 		j = i % priv->num_pairs;
>
> 		ppriv = per_cpu_ptr(priv->ppriv, cpu);
> 		ppriv->req_fqid = priv->tx_queue_attr[j].fqid;
Assigns a tx queue to the cpu, one tx queue may be assigned to
multiple cores due to modulo above (if i >= num_pairs).
> 		/*
> 		 * Allow all cores to enqueue, while only some of them
> 		 * will take part in dequeuing.
> 		 */
> 		if (++i > priv->num_pairs)
> 			continue;
After this i <= num_pairs.
> 		ppriv->rsp_fqid = priv->rx_queue_attr[j].fqid;
> 		ppriv->prio = j;
Assigns a rx queue to the cpu.
This is the case which causes issues with isolcpus, and it should
be avoided for non-housekeeping cpus.
> 		dev_dbg(dev, "pair %d: rx queue %d, tx queue %d\n", j,
> 			priv->rx_queue_attr[j].fqid,
> 			priv->tx_queue_attr[j].fqid);
>
> 		ppriv->net_dev = alloc_netdev_dummy(0);
> 		if (!ppriv->net_dev) {
> 			err = -ENOMEM;
> 			goto err_alloc_netdev;
> 		}
> 		cpumask_set_cpu(cpu, priv->clean_mask);
> 		ppriv->net_dev->dev = *dev;
>
> 		netif_napi_add_tx_weight(ppriv->net_dev, &ppriv->napi,
> 					 dpaa2_dpseci_poll,
> 					 DPAA2_CAAM_NAPI_WEIGHT);
> 	}
dpaa2_caam_enqueue unconditionally accesses ppriv->dpio per-cpu
and ppriv->req_fqid, not gated by any checks, percpu (any cpu).

This causes two issues:
1. each cpu is still expected to have a dpio object
2. each cpu is still expected to have an assigned tx queue

I don't quite understand how we get into dpaa2_caam_enqueue,
I suppose it can legitimately happen on any cpu core whatsoever?

sincerely
Josua Mayer