Re: [RFC PATCH v1 1/8] misc/arm-cla: Add driver skeleton and documentation

[Ryan Roberts <ryan.roberts@arm.com>]

On 17/07/2026 17:10, Arnd Bergmann wrote:
> On Fri, Jul 17, 2026, at 17:44, Ryan Roberts wrote:
>> On 17/07/2026 14:49, Arnd Bergmann wrote:
>>> On Fri, Jul 17, 2026, at 12:47, Ryan Roberts wrote:
>>> This sounds a lot like st64bv or st64bv0, passing an 8-word payload and returning
>>> a single word per accelerator operation with shared addressing.
>>
>> We're actually passing 9 words here; 8 DATA words plus the LAUNCH word. CLA
>> supports only 64 bit aligned and sized accesses (other accesses are RAZ/WI) so
>> they have to be written as 9x64bit stores. Then poll using 64bit loads.
>>
>>>
>>> Why are there now two interfaces to do the same thing?
>>
>> Good question. This is how the HW operates.
>>
>>>
>>> Can a user process use st64bv to do the four steps in a
>>> single instruction?
>>
>> No, unfortunately not.
>>
> 
> Ok
> 
>>> Can a CPU start multiple asynchronous operations concurrently?
>>
>> Yes; STATUS indicates READY while it can accept more asynchronous operations
>> ("comamnds").
> 
> How does userspace know which operations have already completed
> then? (not worried about this bit, just trying to understand)

There is nothing architected in the CLA to determine this; You can determine
READY (i.e. can accept new commands) and IDLE (i.e. completed all previously
submitted commands). If you want to know when a specific command completed
without having to wait for all commands to complete, then you need extra
knowledge of the specific accelerator.

> 
>>> Do these continue to run if the starting process is scheduled out
>>> and another process also tries to use CLA?
>>
>> Yes; the driver manages assignment of a CLA to a process context completely
>> separately from the thread scheduler's decisions about which threads run on
>> which CPUs and when. If another process is scheduled onto the CPU and it
>> attempts to access it's VA for the CLA, it will fault into the driver's handler
>> and be put to sleep until the driver decides to reassign the CLA.
> 
> This part does sound dangerous, not in the sense that I think it's
> fundamentally broken, but in the complexity it adds.

Can you describe more about the complexity you see here? The intention is to
_reduce_ complexity by not coupling with the CPU scheduler.

> 
> I wonder if it's feasible to simplify this by always canceling
> any ongoing CLA operations during switch_mm():

The advice I've had so far is to avoid at all costs, coupling the CLA context
switch with the CPU context switch; there are concerns about the latency
impliciaitons for the CLA context switch and also a desire not to couple the CLA
device driver with arch code.

> 
> Is there an upper bound on how long a single operation can take,
> or a guarantee that an operation at least provides a partial result
> in hardware?

For the compute accelerator we have, operations are gauranteed to complete in
finite time, but that time could technically be very large. In practice
operations for known use cases range from a few microseconds to ~10 milliseconds.

As I described in a previous response, the accelerator does not support full
save/restore so if it is interrupted while running an operation, the operation
appears aborted when the state is next restored. Due to forward progress
concerns, we are looking at the possibility of a grace period (of the order of a
few milliseconds) during CLA reassignment. I'm pretty sure we wouldn't want to
insert that into the (non-preemptible) CPU context switch path...

> 
> From your earlier descriptions, it sounds like the CPU is usually
> assumed to wait for completion with WFE anyway, so from the
> scheduler's perspective, the thread is active while waiting for
> the accelerator to complete a job (even if from hardware side
> the CPU is powered down during WFE).

In practice the CPU will have some parallel work to prepare the next
operation(s) for submission, but yes, eventually it will enter a polling loop
waiting for accelarator completion with WFE.

> 
> If this is how it generally operates, and the accelerator jobs
> are usually fast, forcing the CLA TTBR0 to be the same as the
> CPU TTBR0 would avoid the entire problem of unmapping the registers
> on context switch, but instead let this hook into the same place
> as the corresponding iommu_mm_data switch on x86, which seems> to handle this
more nicely.
>

I agree this feels like a much simpler model on the surface but because the jobs
are not always fast and because the accelerator doesn't support full
save/restore, I don't think this approach flies in practice.

Thanks,
Ryan


>> I'm not sure what you mean by "Which instruction specifically causes the fault".
>> A fault occurs within the accelerator if it tries to access a virtual address
>> that is not mapped by the page table or if the permissions of the mapping are
>> not sufficient, etc... The fact that the accelerator has faulted is reported to
>> the SW that is polling the accelerator's STATUS register within user space. That
>> SW is expected to trigger fault handling by the usual kernel mechanisms by
>> accessing the VA. Then it issues a RESOLVE operation to tell the accelerator it
>> can continue.
> 
> Got it now. I had assumed that the page fault is delivered asynchronously
> to the kernel without user space getting involved. In this case, I think
> the interface is actually cleaner, though it does add a little bit
> of overhead for userspace having to decipher the status. 
>  
>>>> +User space software is expected to create a thread to drive each CLA it is
>>>> +using, and for each thread to be pinned to the CLA's local CPU.
>>>
>>> What happens if multiple processes have the same chardev open and
>>> each mmap() that, e.g. after a fork()? Does each process see its
>>> own virtual instance of the accelerator and interact with it through
>>> the same physical MMIO register range but its own process address space,
>>> or do you have to rely on the registers being mapped only into a
>>> single mm_struct to prevent a process from messing with another process
>>> data?
>>
>> The driver maintains a cla_ctx for each {file description, mm_struct} pair. So
>> in this case, even though the file description is shared between the parent and
>> child processes, they still have distinct mm_structs so still have separate
>> contexts allowing the driver to virtualize access correctly.
> 
> Ok, got it.
> 
>      Arnd