Re: [PATCH v1 03/14] xen/riscv: introduce ioremap()

[Oleksii Kurochko <oleksii.kurochko@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6969 bytes --]


On 4/17/25 4:49 PM, Jan Beulich wrote:
> On 17.04.2025 16:37, Oleksii Kurochko wrote:
>> On 4/17/25 4:24 PM, Jan Beulich wrote:
>>> On 17.04.2025 16:20, Oleksii Kurochko wrote:
>>>> On 4/15/25 1:02 PM, Jan Beulich wrote:
>>>>> On 15.04.2025 12:29, Oleksii Kurochko wrote:
>>>>>> On 4/10/25 5:13 PM, Jan Beulich wrote:
>>>>>>> On 08.04.2025 17:57, Oleksii Kurochko wrote:
>>>>>>>> Based on RISC-V unpriviliged spec ( Version 20240411 ):
>>>>>>>> ```
>>>>>>>> For implementations that conform to the RISC-V Unix Platform Specification,
>>>>>>>> I/O devices and DMA operations are required to access memory coherently and
>>>>>>>> via strongly ordered I/O channels. Therefore, accesses to regular main memory
>>>>>>>> regions that are concurrently accessed by external devices can also use the
>>>>>>>> standard synchronization mechanisms. Implementations that do not conform
>>>>>>>> to the Unix Platform Specification and/or in which devices do not access
>>>>>>>> memory coherently will need to use mechanisms
>>>>>>>> (which are currently platform-specific or device-specific) to enforce
>>>>>>>> coherency.
>>>>>>>>
>>>>>>>> I/O regions in the address space should be considered non-cacheable
>>>>>>>> regions in the PMAs for those regions. Such regions can be considered coherent
>>>>>>>> by the PMA if they are not cached by any agent.
>>>>>>>> ```
>>>>>>>> and [1]:
>>>>>>>> ```
>>>>>>>> The current riscv linux implementation requires SOC system to support
>>>>>>>> memory coherence between all I/O devices and CPUs. But some SOC systems
>>>>>>>> cannot maintain the coherence and they need support cache clean/invalid
>>>>>>>> operations to synchronize data.
>>>>>>>>
>>>>>>>> Current implementation is no problem with SiFive FU540, because FU540
>>>>>>>> keeps all IO devices and DMA master devices coherence with CPU. But to a
>>>>>>>> traditional SOC vendor, it may already have a stable non-coherency SOC
>>>>>>>> system, the need is simply to replace the CPU with RV CPU and rebuild
>>>>>>>> the whole system with IO-coherency is very expensive.
>>>>>>>> ```
>>>>>>>>
>>>>>>>> and the fact that all known ( to me ) CPUs that support the H-extension
>>>>>>>> and that ones is going to be supported by Xen have memory coherency
>>>>>>>> between all I/O devices and CPUs, so it is currently safe to use the
>>>>>>>> PAGE_HYPERVISOR attribute.
>>>>>>>> However, in cases where a platform does not support memory coherency, it
>>>>>>>> should support CMO extensions and Svpbmt. In this scenario, updates to
>>>>>>>> ioremap will be necessary.
>>>>>>>> For now, a compilation error will be generated to ensure that the need to
>>>>>>>> update ioremap() is not overlooked.
>>>>>>>>
>>>>>>>> [1]https://patchwork.kernel.org/project/linux-riscv/patch/1555947870-23014-1-git-send-email-guoren@kernel.org/
>>>>>>> But MMIO access correctness isn't just a matter of coherency. There may not
>>>>>>> be any caching involved in most cases, or else you may observe significantly
>>>>>>> delayed or even dropped (folded with later ones) writes, and reads may be
>>>>>>> serviced from the cache instead of going to actual MMIO. Therefore ...
>>>>>>>
>>>>>>>> --- a/xen/arch/riscv/Kconfig
>>>>>>>> +++ b/xen/arch/riscv/Kconfig
>>>>>>>> @@ -15,6 +15,18 @@ config ARCH_DEFCONFIG
>>>>>>>>      	string
>>>>>>>>      	default "arch/riscv/configs/tiny64_defconfig"
>>>>>>>>      
>>>>>>>> +config HAS_SVPBMT
>>>>>>>> +	bool
>>>>>>>> +	help
>>>>>>>> +	  This config enables usage of Svpbmt ISA-extension ( Supervisor-mode:
>>>>>>>> +	  page-based memory types).
>>>>>>>> +
>>>>>>>> +	  The memory type for a page contains a combination of attributes
>>>>>>>> +	  that indicate the cacheability, idempotency, and ordering
>>>>>>>> +	  properties for access to that page.
>>>>>>>> +
>>>>>>>> +	  The Svpbmt extension is only available on 64-bit cpus.
>>>>>>> ... I kind of expect this extension (or anything else that there might be) will need
>>>>>>> making use of.
>>>>>> In cases where the Svpbmt extension isn't available, PMA (Physical Memory Attributes)
>>>>>> is used to control which memory regions are cacheable, non-cacheable, readable, writable,
>>>>>> etc. PMA is configured in M-mode by the firmware (e.g., OpenSBI), as is done in Andes
>>>>>> cores, or it can be fixed at design time, as in SiFive cores.
>>>>> How would things work if there was a need to map a RAM page uncacheable (via
>>>>> ioremap() or otherwise)?
>>>> My understanding is that Svpbmt is only needed when someone wants to change the memory
>>>> attribute of a page set by PMA.
>>>>
>>>> The question is if non-cacheable RAM page is really needed if we have a coherency?
>>> Aiui coherency here is among CPUs.
>> ```
>> For implementations that conform to the RISC-V Unix Platform Specification,
>> I/O devices and DMA operations are required to access memory coherently and
>> via strongly ordered I/O channels. Therefore, accesses to regular main memory
>> regions that are concurrently accessed by external devices can also use the
>> standard synchronization mechanisms. Implementations that do not conform
>> to the Unix Platform Specification and/or in which devices do not access
>> memory coherently will need to use mechanisms
>> (which are currently platform-specific or device-specific) to enforce
>> coherency.
>> ```
>> Based on this from the spec, coherency here is not only among CPUs.
>>
>>
>>> Properties of devices in the system are
>>> largely unknown?
>> Yes, but still not sure what kind of property requires ioremap() which won't work
>> without Svpmbt. Could you please tell me an example?
> Well, above you said they all need to access memory coherently. That's the
> "property" I was referring to.

Do you mean that device could have a property which tell that it would like to have non-cachable
region used for that? I haven't seen such property in device tree files.

Do we have in Xen cases when Xen wants to have map part of RAM as non-cachebale and it is only the
one option?

I am also thinking why it can't be used cachable region + barrier (if we don't have memory coherency
for everything).

Anyway, if it isn't an option to have mapped cacheble region + barrier then there is no any choice
and the support of Svpmbt is required.

>
>>> (Beyond this there may also be special situations in which
>>> one really cares about data going directly to RAM.)
>> If there are such special cases, I assume that the firmware or hardware (in the case
>> of fixed PMA) will provide a non-cacheable region.
> How could they? Firmware may be unaware of specific properties of specific
> devices a user adds to a system.

(this is not real case, just thoughts) Firmware could by default provide part of RAM as
non-cacheable region and then hypervisor/kernel use this region for allocation. But I agree
that it isn't the best thing to manage that.


~ Oleksii

>> In that case, the user should be
>> aware of this region and use it for those specific scenarios.
>>
>> ~ Oleksii
>>

[-- Attachment #2: Type: text/html, Size: 8997 bytes --]