[EVL] Problems writing to physical memory with mmap on oob context

[EVL] Problems writing to physical memory with mmap on oob context

[Jesus Villena]

Hello,

We are using an ARM64 based platform which includes a small Cortex M4 processor to run some algorithms, and both subsystems use an internal shared SRAM memory to implement the inter-processor communication.

In mainline Linux, we open a "/dev/mem" file in the ARM64 core to get access to physical memory, and mmap the shared area to read and write directly to the SRAM from userland.

With EVL, it doesn't work: after opening the file and doing mmap, we call evl_attach_self to switch to the oob context, and then, reading on the shared area works, but writing aborts the program with the message:

EVL: evl_test switching in-band [pid=363, excpt=0, user_pc=0x404b5c]
oop...
CPU time limit exceeded

I don't understand why EVL is switching to in-band context.

Do you know where is the problem? Why does reading work but writing doesn't?

Regards.

Jesus Villena

Re: [EVL] Problems writing to physical memory with mmap on oob context

[Philippe Gerum]

Jesus Villena <j.villena@ecler.com> writes:

> Hello,
>
> We are using an ARM64 based platform which includes a small Cortex M4 processor to run some algorithms, and both subsystems use an internal shared SRAM memory to implement the inter-processor communication.
>
> In mainline Linux, we open a "/dev/mem" file in the ARM64 core to get access to physical memory, and mmap the shared area to read and write directly to the SRAM from userland.
>
> With EVL, it doesn't work: after opening the file and doing mmap, we call evl_attach_self to switch to the oob context, and then, reading on the shared area works, but writing aborts the program with the message:
>
> EVL: evl_test switching in-band [pid=363, excpt=0, user_pc=0x404b5c]
> oop...
> CPU time limit exceeded
>
> I don't understand why EVL is switching to in-band context.
>
> Do you know where is the problem? Why does reading work but writing doesn't?
>

Looking at the output you get, it most likely behaves the same and does
work in both cases, w/ and w/o EVL involved. In both cases you receive a
so-called major fault when writing to this area: the only difference is
that EVL reports such kind of event which requires heavyweight fixups
when running oob by sending a SIGXCPU signal. You definitely want to
read this doc [1] about these notifications.

Those major faults cannot be handled directly from the oob stage, so EVL
downgrades the thread issuing the write access request to in-band
context first, then notifies the issuer about it via SIGXCPU. You could
disable this signal by clearing the EVL_T_HMSIG flag for that thread
using evl_set_thread_mode(), or set a dummy handler for this signal. But
that would obviously only paper over the problem, which is that the MMU
wants some help from the kernel for dealing with this write access in
particular.

If not already set, you may want to turn CONFIG_STRICT_DEVMEM on,
checking whether mmap() still succeeds or fails with EPERM. That would
give you a first hint about how the kernel views this SRAM range,
i.e. dev exclusive or other. In any case, figuring out why the CPU traps
for that access may require some debug work with the kernel code (I
would start looking at arch/arm64/mm/fault.c, do_page_fault()
typically). You may also want to include the KERN_ALERT log level in the
console output, in order to get a number of low-level memory management
traces on arm64.

PS: Since /dev/mem is a liability when it comes to security, you may
want to write your own (small) driver only for the purpose of creating
memory mappings specifically from the SRAM to your application,
disabling /dev/mem entirely next. EVL would still have no part in this.

[1] https://v4.xenomai.org/core/user-api/thread/#hm-notification-methods

-- 
Philippe.

RE: [EVL] Problems writing to physical memory with mmap on oob context

[Jesus Villena]

Thanks Philippe,

I have already tried the custom driver solution you mention, mapping the shared memory in kernel with ioremap, and adding read and write handlers instead of mapping the memory to userland, and it works, but I would prefer to do a proper mmap to access to the memory in a transparent way.

Following your comments, I will try to debug the issue in order to see where is the problem.

Regards.

Jesus Villena


> -----Mensaje original-----
> De: Philippe Gerum <rpm@xenomai.org>
> Enviado el: martes, 26 de septiembre de 2023 16:16
> Para: Jesus Villena <j.villena@ecler.com>
> CC: xenomai@lists.linux.dev
> Asunto: Re: [EVL] Problems writing to physical memory with mmap on oob
> context
> 
> 
> Jesus Villena <j.villena@ecler.com> writes:
> 
> > Hello,
> >
> > We are using an ARM64 based platform which includes a small Cortex M4
> processor to run some algorithms, and both subsystems use an internal shared
> SRAM memory to implement the inter-processor communication.
> >
> > In mainline Linux, we open a "/dev/mem" file in the ARM64 core to get access
> to physical memory, and mmap the shared area to read and write directly to the
> SRAM from userland.
> >
> > With EVL, it doesn't work: after opening the file and doing mmap, we call
> evl_attach_self to switch to the oob context, and then, reading on the shared
> area works, but writing aborts the program with the message:
> >
> > EVL: evl_test switching in-band [pid=363, excpt=0, user_pc=0x404b5c]
> > oop...
> > CPU time limit exceeded
> >
> > I don't understand why EVL is switching to in-band context.
> >
> > Do you know where is the problem? Why does reading work but writing
> doesn't?
> >
> 
> Looking at the output you get, it most likely behaves the same and does work in
> both cases, w/ and w/o EVL involved. In both cases you receive a so-called
> major fault when writing to this area: the only difference is that EVL reports
> such kind of event which requires heavyweight fixups when running oob by
> sending a SIGXCPU signal. You definitely want to read this doc [1] about these
> notifications.
> 
> Those major faults cannot be handled directly from the oob stage, so EVL
> downgrades the thread issuing the write access request to in-band context first,
> then notifies the issuer about it via SIGXCPU. You could disable this signal by
> clearing the EVL_T_HMSIG flag for that thread using evl_set_thread_mode(), or
> set a dummy handler for this signal. But that would obviously only paper over
> the problem, which is that the MMU wants some help from the kernel for
> dealing with this write access in particular.
> 
> If not already set, you may want to turn CONFIG_STRICT_DEVMEM on, checking
> whether mmap() still succeeds or fails with EPERM. That would give you a first
> hint about how the kernel views this SRAM range, i.e. dev exclusive or other. In
> any case, figuring out why the CPU traps for that access may require some
> debug work with the kernel code (I would start looking at
> arch/arm64/mm/fault.c, do_page_fault() typically). You may also want to
> include the KERN_ALERT log level in the console output, in order to get a
> number of low-level memory management traces on arm64.
> 
> PS: Since /dev/mem is a liability when it comes to security, you may want to
> write your own (small) driver only for the purpose of creating memory
> mappings specifically from the SRAM to your application, disabling /dev/mem
> entirely next. EVL would still have no part in this.
> 
> [1] https://v4.xenomai.org/core/user-api/thread/#hm-notification-methods
> 
> --
> Philippe.

Re: [EVL] Problems writing to physical memory with mmap on oob context

[Philippe Gerum]

Jesus Villena <j.villena@ecler.com> writes:

> Thanks Philippe,
>
> I have already tried the custom driver solution you mention, mapping the shared memory in kernel with ioremap, and adding read and write handlers instead of mapping the memory to userland, and it works, but I would prefer to do a proper mmap to access to the memory in a transparent way.
>

I meant writing a driver exporting a mmap interface restricted to your
SRAM area, instead of using the generic /dev/mem device for this.

-- 
Philippe.

RE: [EVL] Problems writing to physical memory with mmap on oob context

[Jesus Villena]

Yes, that's what I understood!

Anyway, I have made some progress: if I write something to the shared area (which is less than 4KB) before going to oob context, it works as expected!

So, it seems to do with a page_fault the first time a write is issued in the shared area. Sorry, but I don't understand properly all complexity about accessing Linux virtual and physical memory, security, and protection.

Regards.

Jesus Villena


> -----Mensaje original-----
> De: Philippe Gerum <rpm@xenomai.org>
> Enviado el: martes, 26 de septiembre de 2023 17:46
> Para: Jesus Villena <j.villena@ecler.com>
> CC: xenomai@lists.linux.dev
> Asunto: Re: [EVL] Problems writing to physical memory with mmap on oob
> context
> 
> 
> Jesus Villena <j.villena@ecler.com> writes:
> 
> > Thanks Philippe,
> >
> > I have already tried the custom driver solution you mention, mapping the
> shared memory in kernel with ioremap, and adding read and write handlers
> instead of mapping the memory to userland, and it works, but I would prefer to
> do a proper mmap to access to the memory in a transparent way.
> >
> 
> I meant writing a driver exporting a mmap interface restricted to your SRAM
> area, instead of using the generic /dev/mem device for this.
> 
> --
> Philippe.

Re: [EVL] Problems writing to physical memory with mmap on oob context

[Philippe Gerum]

Jesus Villena <j.villena@ecler.com> writes:

> Yes, that's what I understood!
>
> Anyway, I have made some progress: if I write something to the shared area (which is less than 4KB) before going to oob context, it works as expected!
>
> So, it seems to do with a page_fault the first time a write is issued in the shared area. Sorry, but I don't understand properly all complexity about accessing Linux virtual and physical memory, security, and protection.
>

This memory range may have copy-on-write semantics, which would explain
the major fault when first touched for update. It would make sense to
pre-fault this area in order to break COW early for it.

-- 
Philippe.