[Xenomai] accessing mmap'ed memory causes mode switches

[Xenomai] accessing mmap'ed memory causes mode switches

[Giulio Moro]

Hello there,

I am new to the list, I am working on this project
 https://github.com/BelaPlatform/Bela/ for real-time audio 
on the BeagleBone Black using Xenomai.

I am trying to use Xenomai's POSIX skin to create and access
 shared memory objects, but I am getting mode switches to 
secondary mode every time I try to access (read or write) the 
mapped memory, using Xenomai 2.6.3 on Debian ARM Linux 3.8.13r81 
(https://github.com/RobertCNelson/bb-kernel/tree/3.8.13-xenomai-r81 ).
I would expect no mode switches to take place once the memory 
is mapped and accessed at least once.
Full listing below,
Notes:

- I build with 
gcc -I. shm.c `xeno-config --skin posix --cflags` `xeno-config --skin posix --ldflags`  `xeno-config --skin native --ldflags` -o shm
which evaluates to 
gcc -I. shm.c -I/usr/xenomai/include -D_GNU_SOURCE -D_REENTRANT -D__XENO__ -I/usr/xenomai/include/posix -Wl,@/usr/xenomai/lib/posix.wrappers -L/usr/xenomai/lib -lpthread_rt -lxenomai -lpthread -lrt -lnative -L/usr/xenomai/lib -lxenomai -lpthread -lrt -o shm


- I tried to explicitly use __wrap_shm_open and __wrap_mmap , no change.
- I tried with and without the CONFIG_XENO_OPT_POSIX_SHM config line in the kernel, no change.
- the documentation does not state that O_DIRECT can only be used when CONFIG_XENO_OPT_POSIX_SHM is activated
- the description of the return values for mmap is wrong in the docs http://www.xenomai.org/documentation/trunk/html/api/group__posix__shm.html : it says it returns 0 on success, while it returns the pointer to the memory.


minimal example (does not compile, gives you an idea)

fd = shm_open(SHM_NAME, O_RDWR | O_CREAT | O_DIRECT, 0);
size_t len = 40;
ftruncate(fd, len);void* addr = mmap(ignored, len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
float* arr = (float*)addr;
//causes mode switches:
printf("%f\n", arr[n]);



Full listing: 
https://gist.github.com/giuliomoro/457c8cde1eda6190a28566b3288d38c0
Kernel config file: 
https://gist.github.com/giuliomoro/283c92049704652c06a26ed08669966a


Best,
Giulio

Re: [Xenomai] accessing mmap'ed memory causes mode switches

[Leopold Palomo-Avellaneda]

Hi Giulio,

El Dijous, 27 d'octubre de 2016, a les 01:43:21, Giulio Moro va escriure:
> Hello there,
> 
> I am new to the list, I am working on this project
>  https://github.com/BelaPlatform/Bela/ for real-time audio
> on the BeagleBone Black using Xenomai.

welcome!!!

Nice project.

> I am trying to use Xenomai's POSIX skin to create and access
>  shared memory objects, but I am getting mode switches to
> secondary mode every time I try to access (read or write) the
> mapped memory, using Xenomai 2.6.3 on Debian ARM Linux 3.8.13r81
> (https://github.com/RobertCNelson/bb-kernel/tree/3.8.13-xenomai-r81 ).
> I would expect no mode switches to take place once the memory
> is mapped and accessed at least once.

Just some notes:

- xenomai 2.6.3 is old. 2.6.5 is the last stable version of 2.6.x branch. 
Also, 2.6.x branch AFAIK is just in maintenance mode, so I would recommend you 
if you are beginning a new project use Xenomai 3.x

Best regards,

Leopold

-- 
--
Linux User 152692     GPG: 05F4A7A949A2D9AA
Catalonia
-------------------------------------
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://xenomai.org/pipermail/xenomai/attachments/20161027/9c2c698d/attachment.sig>

Re: [Xenomai] accessing mmap'ed memory causes mode switches

[Giulio Moro]

>- xenomai 2.6.3 is old. 2.6.5 is the last stable version of 2.6.x branch. 
>Also, 2.6.x branch AFAIK is just in maintenance mode, so I would recommend you 
>if you are beginning a new project use Xenomai 3.x


Thanks for the advice, the project started three years ago so it was 2.x from the beginning and then 
It is also difficult to find BeagleBone kernels patched with Xenomai and they are all 2.6, so I guess we'll stick with 2.6.x for a while, though I will look into update to 2.6.5 soon. Do you reckon this could be related to the mode switches I am seeing?

Best,
Giulio

Re: [Xenomai] accessing mmap'ed memory causes mode switches

[Leopold Palomo-Avellaneda]

El Dijous, 27 d'octubre de 2016, a les 11:50:50, Giulio Moro va escriure:
> >- xenomai 2.6.3 is old. 2.6.5 is the last stable version of 2.6.x branch.
> >Also, 2.6.x branch AFAIK is just in maintenance mode, so I would recommend
> >you if you are beginning a new project use Xenomai 3.x
> 
> Thanks for the advice, the project started three years ago so it was 2.x
> from the beginning and then It is also difficult to find BeagleBone kernels
> patched with Xenomai and they are all 2.6, so I guess we'll stick with
> 2.6.x for a while, though I will look into update to 2.6.5 soon.


Are you telling me that it's a problem of compiled kernels? Then it would be 
worth putting some effort to have any of them and expect them to find the net. 
Even I can try to provide one.

> Do you
> reckon this could be related to the mode switches I am seeing?

I don't know, but who knows?

Leopold

-- 
--
Linux User 152692     GPG: 05F4A7A949A2D9AA
Catalonia
-------------------------------------
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://xenomai.org/pipermail/xenomai/attachments/20161027/5614caf7/attachment.sig>

Re: [Xenomai] accessing mmap'ed memory causes mode switches

[Philippe Gerum]

On 10/27/2016 03:43 AM, Giulio Moro wrote:
> Hello there,
> 
> I am new to the list, I am working on this project
>  https://github.com/BelaPlatform/Bela/ for real-time audio 
> on the BeagleBone Black using Xenomai.
> 
> I am trying to use Xenomai's POSIX skin to create and access
>  shared memory objects, but I am getting mode switches to 
> secondary mode every time I try to access (read or write) the 
> mapped memory, using Xenomai 2.6.3 on Debian ARM Linux 3.8.13r81 
> (https://github.com/RobertCNelson/bb-kernel/tree/3.8.13-xenomai-r81 ).
> I would expect no mode switches to take place once the memory 
> is mapped and accessed at least once.
> Full listing below,
> Notes:
> 
> - I build with 
> gcc -I. shm.c `xeno-config --skin posix --cflags` `xeno-config --skin posix --ldflags`  `xeno-config --skin native --ldflags` -o shm
> which evaluates to 
> gcc -I. shm.c -I/usr/xenomai/include -D_GNU_SOURCE -D_REENTRANT -D__XENO__ -I/usr/xenomai/include/posix -Wl,@/usr/xenomai/lib/posix.wrappers -L/usr/xenomai/lib -lpthread_rt -lxenomai -lpthread -lrt -lnative -L/usr/xenomai/lib -lxenomai -lpthread -lrt -o shm
> 
> 
> - I tried to explicitly use __wrap_shm_open and __wrap_mmap , no change.
> - I tried with and without the CONFIG_XENO_OPT_POSIX_SHM config line in the kernel, no change.
> - the documentation does not state that O_DIRECT can only be used when CONFIG_XENO_OPT_POSIX_SHM is activated

Xenomai's legacy POSIX shm support available with 2.6 did not bring any
actual upside, and was more or less a wrapper around the regular shm
support. For this reason, it has been dropped in 3.x. You may want to
use the latter directly, even with 2.6.

> - the description of the return values for mmap is wrong in the docs http://www.xenomai.org/documentation/trunk/html/api/group__posix__shm.html : it says it returns 0 on success, while it returns the pointer to the memory.
> 

The documentation is inlined in the source code, issuing patches to
update it is the best way to get it fixed.

> 
> minimal example (does not compile, gives you an idea)
> 
> fd = shm_open(SHM_NAME, O_RDWR | O_CREAT | O_DIRECT, 0);
> size_t len = 40;
> ftruncate(fd, len);void* addr = mmap(ignored, len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
> float* arr = (float*)addr;

No memory reference up to this point anyway.

> //causes mode switches:
> printf("%f\n", arr[n]);
> 

This is due to minor faults, when the memory is there but not referenced
to by a valid PTE in the page table for the current process yet.

Before the 3.10 series, the pipeline used to switch to secondary mode
eagerly upon any kind of page fault including minor ones (see
do_translation_fault() in arm/mm/fault.c), this is what your code is
experiencing with 3.8.

Starting from 3.10 and on, those faults should be transparently handled
by the pipeline, and should not require any transition to secondary mode
for armv6+ CPUs.

In short, you need to upgrade the pipeline code.

-- 
Philippe.

Re: [Xenomai] accessing mmap'ed memory causes mode switches

[Giulio Moro]

Thanks Philippe,

>> //causes mode switches:
>> rt_printf("%f\n", arr[n]);
>> 
>
>This is due to minor faults, when the memory is there but not referenced
>to by a valid PTE in the page table for the current process yet.
>
>Before the 3.10 series, the pipeline used to switch to secondary mode
>eagerly upon any kind of page fault including minor ones (see
>do_translation_fault() in arm/mm/fault.c), this is what your code is
>experiencing with 3.8.
>
>Starting from 3.10 and on, those faults should be transparently handled
>by the pipeline, and should not require any transition to secondary mode
>for armv6+ CPUs.
>
>In short, you need to upgrade the pipeline code.


I noticed that the mode switch happens over and over again even when accessing the same memory location multiple times.
e.g., this code will cause 10 mode switches:

for(j = 0; j < 10; ++j){
  arr[0] = 1.f; // switches to secondary mode
  rt_task_sleep(200000000);  // sleep, switches to primary mode
}

Is this phenomenon still explained by the issue in pre-3.10 you outlined ?

Thanks,
Giulio

Re: [Xenomai] accessing mmap'ed memory causes mode switches

[Philippe Gerum]

On 10/31/2016 07:12 PM, Giulio Moro wrote:
> Thanks Philippe,
> 
>>> //causes mode switches:
>>> rt_printf("%f\n", arr[n]);
>>>
>>
>> This is due to minor faults, when the memory is there but not referenced
>> to by a valid PTE in the page table for the current process yet.
>>
>> Before the 3.10 series, the pipeline used to switch to secondary mode
>> eagerly upon any kind of page fault including minor ones (see
>> do_translation_fault() in arm/mm/fault.c), this is what your code is
>> experiencing with 3.8.
>>
>> Starting from 3.10 and on, those faults should be transparently handled
>> by the pipeline, and should not require any transition to secondary mode
>> for armv6+ CPUs.
>>
>> In short, you need to upgrade the pipeline code.
> 
> 
> I noticed that the mode switch happens over and over again even when accessing the same memory location multiple times.
> e.g., this code will cause 10 mode switches:
> 
> for(j = 0; j < 10; ++j){
>   arr[0] = 1.f; // switches to secondary mode
>   rt_task_sleep(200000000);  // sleep, switches to primary mode
> }
> 
> Is this phenomenon still explained by the issue in pre-3.10 you outlined ?
> 

This fix of an obvious issue we had in pre-3.10 pipelines for ARM should
give a hint:

diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index d3259cb..cb46d76 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -440,18 +440,11 @@ do_translation_fault(unsigned long addr, unsigned
int fsr,
 	pgd_t *pgd, *pgd_k;
 	pud_t *pud, *pud_k;
 	pmd_t *pmd, *pmd_k;
+	unsigned long flags;

 	if (addr < TASK_SIZE)
 		return do_page_fault(addr, fsr, regs);

-	if (__ipipe_report_trap(IPIPE_TRAP_ACCESS,regs))
-		return 0;
-
-#ifdef CONFIG_IPIPE
-	ipipe_stall_root();
-	hard_local_irq_enable();
-#endif
-
 	if (user_mode(regs))
 		goto bad_area;

@@ -499,7 +492,20 @@ do_translation_fault(unsigned long addr, unsigned
int fsr,
 	return 0;

 bad_area:
+	if (__ipipe_report_trap(IPIPE_TRAP_ACCESS,regs))
+		return 0;
+
+#ifdef CONFIG_IPIPE
+	flags = ipipe_test_and_stall_root();
+	hard_local_irq_enable();
+#endif
+
 	do_bad_area(addr, fsr, regs);
+
+#ifdef CONFIG_IPIPE
+	hard_local_irq_disable();
+	ipipe_restore_root_nosync(flags);
+#endif
 	return 0;
 }
 #else					/* CONFIG_MMU */

-- 
Philippe.