[Xenomai-help] ARM cached,non-cached memory

[Xenomai-help] ARM cached,non-cached memory

[Sinisa Denic]

Hi, last year working with xenomai 2.4.2-3 we met problem (BUG) with memory 
caching on ARM AT91sam9260 arch. 
Then we were advised using non-cached memory and we've used it till now,but in 
our next application it has big performace slow down effect and I'm 
interested is this BUG solved in latest 2.4.10 - xenomai 
Regards,Sinisa

Re: [Xenomai-help] ARM cached,non-cached memory

[Gilles Chanteperdrix]

Sinisa Denic wrote:
> Hi, last year working with xenomai 2.4.2-3 we met problem (BUG) with memory 
> caching on ARM AT91sam9260 arch. 
> Then we were advised using non-cached memory and we've used it till now,but in 
> our next application it has big performace slow down effect and I'm 
> interested is this BUG solved in latest 2.4.10 - xenomai 
> Regards,Sinisa

No, this is not a BUG in Xenomai. It is an architectural BUG^H^Hehaviour
of ARMv5, they have a VIVT cache, which is the cause of these issues.
Alternatively to using non-cached memory, you can use cached memory, and
flush cache when needed. Beware however defining "when needed", may be
complicated.

A note: Xenomai 2.4.10 is not the latest, the latest is Xenomai 2.5.1
and it has a lot of performance improvements for ARM.

-- 
					    Gilles.

Re: [Xenomai-help] ARM cached,non-cached memory

[Sinisa Denic]

On Tuesday 02 March 2010 15:32:06 you wrote:
> Sinisa Denic wrote:
> > Hi, last year working with xenomai 2.4.2-3 we met problem (BUG) with
> > memory caching on ARM AT91sam9260 arch.
> > Then we were advised using non-cached memory and we've used it till
> > now,but in our next application it has big performace slow down effect
> > and I'm interested is this BUG solved in latest 2.4.10 - xenomai
> > Regards,Sinisa
>
> No, this is not a BUG in Xenomai. It is an architectural BUG^H^Hehaviour
> of ARMv5, they have a VIVT cache, which is the cause of these issues.
> Alternatively to using non-cached memory, you can use cached memory, and
> flush cache when needed. Beware however defining "when needed", may be
> complicated.
>
> A note: Xenomai 2.4.10 is not the latest, the latest is Xenomai 2.5.1
> and it has a lot of performance improvements for ARM.

Ok,sorry for my mistake I thought it is due to the xenomai.
Of course we know new xeno 2.5.1 and all about ARM improvements, 
but we are afraid of switching to new kernel(current 2.6.21) and thus to new 
xenomai, because we have short time to deliver product and thus to well test 
all new components. 
Apart from memory caching 2.4 works well for us.
We work on critical relay protection device and new potential bug isn't  
desirable at all.
I hope that xenomai 2.5.1 is heavily tested and stable for ARM arch,despite of 
some problems on the mailing list eg. native skin etc.
Al in all thank you very much we hope to see you in Belgrade soon.
Sinisa

Re: [Xenomai-help] ARM cached,non-cached memory

[Gilles Chanteperdrix]

Sinisa Denic wrote:
> On Tuesday 02 March 2010 15:32:06 you wrote:
>> Sinisa Denic wrote:
>>> Hi, last year working with xenomai 2.4.2-3 we met problem (BUG) with
>>> memory caching on ARM AT91sam9260 arch.
>>> Then we were advised using non-cached memory and we've used it till
>>> now,but in our next application it has big performace slow down effect
>>> and I'm interested is this BUG solved in latest 2.4.10 - xenomai
>>> Regards,Sinisa
>> No, this is not a BUG in Xenomai. It is an architectural BUG^H^Hehaviour
>> of ARMv5, they have a VIVT cache, which is the cause of these issues.
>> Alternatively to using non-cached memory, you can use cached memory, and
>> flush cache when needed. Beware however defining "when needed", may be
>> complicated.
>>
>> A note: Xenomai 2.4.10 is not the latest, the latest is Xenomai 2.5.1
>> and it has a lot of performance improvements for ARM.
> 
> Ok,sorry for my mistake I thought it is due to the xenomai.
> Of course we know new xeno 2.5.1 and all about ARM improvements, 
> but we are afraid of switching to new kernel(current 2.6.21) and thus to new 
> xenomai, because we have short time to deliver product and thus to well test 
> all new components. 
> Apart from memory caching 2.4 works well for us.

Ok. Understood.


> We work on critical relay protection device and new potential bug isn't  
> desirable at all.
> I hope that xenomai 2.5.1 is heavily tested and stable for ARM arch,despite of 
> some problems on the mailing list eg. native skin etc.

There is no such thing as bug-free software, as you probably know, but
we do our best to have the best stability possible. We have recently
fixed Xenomai for running on AT91SAM9263, and got good results over a
long testing period, the hardware support looks stable. The issue we
have such as with the native API are plain deterministic stupid bugs,
nothing which has any influence on stability.

> Al in all thank you very much we hope to see you in Belgrade soon.
> Sinisa

That could be arranged :-)

-- 
					    Gilles.

Re: [Xenomai-help] ARM cached,non-cached memory

[Gilles Chanteperdrix]

Sinisa Denic wrote:
> Hi, last year working with xenomai 2.4.2-3 we met problem (BUG) with memory 
> caching on ARM AT91sam9260 arch. 
> Then we were advised using non-cached memory and we've used it till now,but in 
> our next application it has big performace slow down effect and I'm 
> interested is this BUG solved in latest 2.4.10 - xenomai 
> Regards,Sinisa

Actually, there may be something we can do. In heap.c, could you try 
replacing calls to pgprot_noncached with calls to pgprot_writecombine ?

Here is a patch doing this for 2.4 head. I do not know whether it will
apply to the version you are using. But I guess you will get the idea.

diff --git a/ksrc/nucleus/heap.c b/ksrc/nucleus/heap.c
index 1c02183..d40844b 100644
--- a/ksrc/nucleus/heap.c
+++ b/ksrc/nucleus/heap.c
@@ -962,7 +962,7 @@ static inline void *__alloc_and_reserve_heap(size_t size, in
t kmflags)
                else
                        ptr = __vmalloc(size,
                                        GFP_KERNEL | __GFP_HIGHMEM,
-                                       pgprot_noncached(PAGE_KERNEL));
+                                       pgprot_writecombine(PAGE_KERNEL));
                if (ptr == NULL)
                        return NULL;

@@ -1126,7 +1126,7 @@ static int xnheap_mmap(struct file *file, struct vm_area_struct *vma)
                unsigned long maddr = vma->vm_start;

                if (heap->archdep.kmflags == XNHEAP_GFP_NONCACHED)
-                       vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+                       vma->vm_page_prot = pgprot_writecombine(vma->vm_page_pro
t);

                while (size > 0) {
                        if (xnarch_remap_vm_page(vma, maddr, vaddr))
@@ -1147,7 +1147,7 @@ static int xnheap_mmap(struct file *file, struct vm_area_s
truct *vma)
        (void)vaddr;
        if ((heap->archdep.kmflags & ~XNHEAP_GFP_NONCACHED) != 0 ||
            heap->archdep.kmflags == XNHEAP_GFP_NONCACHED)
-               vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+               vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
 #endif /* !CONFIG_MMU */

        atomic_inc(&heap->archdep.numaps);


-- 
					    Gilles.

Re: [Xenomai-help] ARM cached,non-cached memory

[Sinisa Denic]

On Saturday 06 March 2010 11:42:07 you wrote:
> Sinisa Denic wrote:
> > Hi, last year working with xenomai 2.4.2-3 we met problem (BUG) with
> > memory caching on ARM AT91sam9260 arch.
> > Then we were advised using non-cached memory and we've used it till
> > now,but in our next application it has big performace slow down effect
> > and I'm interested is this BUG solved in latest 2.4.10 - xenomai
> > Regards,Sinisa
>
> Actually, there may be something we can do. In heap.c, could you try
> replacing calls to pgprot_noncached with calls to pgprot_writecombine ?
>
> Here is a patch doing this for 2.4 head. I do not know whether it will
> apply to the version you are using. But I guess you will get the idea.
>
> diff --git a/ksrc/nucleus/heap.c b/ksrc/nucleus/heap.c
> index 1c02183..d40844b 100644
> --- a/ksrc/nucleus/heap.c
> +++ b/ksrc/nucleus/heap.c
> @@ -962,7 +962,7 @@ static inline void *__alloc_and_reserve_heap(size_t
> size, in t kmflags)
>                 else
>                         ptr = __vmalloc(size,
>                                         GFP_KERNEL | __GFP_HIGHMEM,
> -                                       pgprot_noncached(PAGE_KERNEL));
> +                                       pgprot_writecombine(PAGE_KERNEL));
>                 if (ptr == NULL)
>                         return NULL;
>
> @@ -1126,7 +1126,7 @@ static int xnheap_mmap(struct file *file, struct
> vm_area_struct *vma) unsigned long maddr = vma->vm_start;
>
>                 if (heap->archdep.kmflags == XNHEAP_GFP_NONCACHED)
> -                       vma->vm_page_prot =
> pgprot_noncached(vma->vm_page_prot); +                      
> vma->vm_page_prot = pgprot_writecombine(vma->vm_page_pro t);
>
>                 while (size > 0) {
>                         if (xnarch_remap_vm_page(vma, maddr, vaddr))
> @@ -1147,7 +1147,7 @@ static int xnheap_mmap(struct file *file, struct
> vm_area_s truct *vma)
>         (void)vaddr;
>         if ((heap->archdep.kmflags & ~XNHEAP_GFP_NONCACHED) != 0 ||
>             heap->archdep.kmflags == XNHEAP_GFP_NONCACHED)
> -               vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
> +               vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
>  #endif /* !CONFIG_MMU */
>
>         atomic_inc(&heap->archdep.numaps);

Ah, thank you Gilles we've already reconciled with destiny not to having 
cached memory. 
This is good news,but as I've read "write-combining does not guarantee that 
the combination of writes and reads is done in the correct order" and we 
could meet inconsistency in data - maybe.. 
Does Xeno treat this problem properly?
Sinisa

Re: [Xenomai-help] ARM cached,non-cached memory

[Gilles Chanteperdrix]

Sinisa Denic wrote:
> On Saturday 06 March 2010 11:42:07 you wrote:
>> Sinisa Denic wrote:
>>> Hi, last year working with xenomai 2.4.2-3 we met problem (BUG) with
>>> memory caching on ARM AT91sam9260 arch.
>>> Then we were advised using non-cached memory and we've used it till
>>> now,but in our next application it has big performace slow down effect
>>> and I'm interested is this BUG solved in latest 2.4.10 - xenomai
>>> Regards,Sinisa
>> Actually, there may be something we can do. In heap.c, could you try
>> replacing calls to pgprot_noncached with calls to pgprot_writecombine ?
>>
>> Here is a patch doing this for 2.4 head. I do not know whether it will
>> apply to the version you are using. But I guess you will get the idea.
>>
>> diff --git a/ksrc/nucleus/heap.c b/ksrc/nucleus/heap.c
>> index 1c02183..d40844b 100644
>> --- a/ksrc/nucleus/heap.c
>> +++ b/ksrc/nucleus/heap.c
>> @@ -962,7 +962,7 @@ static inline void *__alloc_and_reserve_heap(size_t
>> size, in t kmflags)
>>                 else
>>                         ptr = __vmalloc(size,
>>                                         GFP_KERNEL | __GFP_HIGHMEM,
>> -                                       pgprot_noncached(PAGE_KERNEL));
>> +                                       pgprot_writecombine(PAGE_KERNEL));
>>                 if (ptr == NULL)
>>                         return NULL;
>>
>> @@ -1126,7 +1126,7 @@ static int xnheap_mmap(struct file *file, struct
>> vm_area_struct *vma) unsigned long maddr = vma->vm_start;
>>
>>                 if (heap->archdep.kmflags == XNHEAP_GFP_NONCACHED)
>> -                       vma->vm_page_prot =
>> pgprot_noncached(vma->vm_page_prot); +                      
>> vma->vm_page_prot = pgprot_writecombine(vma->vm_page_pro t);
>>
>>                 while (size > 0) {
>>                         if (xnarch_remap_vm_page(vma, maddr, vaddr))
>> @@ -1147,7 +1147,7 @@ static int xnheap_mmap(struct file *file, struct
>> vm_area_s truct *vma)
>>         (void)vaddr;
>>         if ((heap->archdep.kmflags & ~XNHEAP_GFP_NONCACHED) != 0 ||
>>             heap->archdep.kmflags == XNHEAP_GFP_NONCACHED)
>> -               vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
>> +               vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
>>  #endif /* !CONFIG_MMU */
>>
>>         atomic_inc(&heap->archdep.numaps);
> 
> Ah, thank you Gilles we've already reconciled with destiny not to having 
> cached memory. 
> This is good news,but as I've read "write-combining does not guarantee that 
> the combination of writes and reads is done in the correct order" and we 
> could meet inconsistency in data - maybe.. 
> Does Xeno treat this problem properly?

The ARM kernel routinely uses this mode for shared mapping mapped at
several places in the same address space. So, I think it works. Are you
sure the sentence you quote applies to ARMv5?

-- 
					    Gilles.

Re: [Xenomai-help] ARM cached,non-cached memory

[Sinisa Denic]

On Monday 08 March 2010 14:52:22 you wrote:
> "write-combining does not guarantee that
>
> > the combination of writes and reads is done in the correct order"


No, I've briefly read general description on wikipedia. :), so I don't know 
wether it has impact on ARM arch.

Re: [Xenomai-help] ARM cached,non-cached memory

[Gilles Chanteperdrix]

Sinisa Denic wrote:
> On Monday 08 March 2010 14:52:22 you wrote:
>> "write-combining does not guarantee that
>>
>>> the combination of writes and reads is done in the correct order"
> 
> 
> No, I've briefly read general description on wikipedia. :), so I don't know 
> wether it has impact on ARM arch.

As this is used by the kernel for its mapping (when you create aliases,
i.e. map the same area several times in the same address space), it must
work reliably. Anyway, what I am interested in is a quick test with your
application, measuring performances, so that we know whether
implementing this feature completely is worth the trouble, and in that
case, we will investigate further these issues.

-- 
					    Gilles.

Re: [Xenomai-help] ARM cached,non-cached memory

[Sinisa Denic]

On Monday 08 March 2010 16:13:42 you wrote:
> Sinisa Denic wrote:
> > On Monday 08 March 2010 14:52:22 you wrote:
> >> "write-combining does not guarantee that
> >>
> >>> the combination of writes and reads is done in the correct order"
> >
> > No, I've briefly read general description on wikipedia. :), so I don't
> > know wether it has impact on ARM arch.
>
> As this is used by the kernel for its mapping (when you create aliases,
> i.e. map the same area several times in the same address space), it must
> work reliably. Anyway, what I am interested in is a quick test with your
> application, measuring performances, so that we know whether
> implementing this feature completely is worth the trouble, and in that
> case, we will investigate further these issues.

Ok, now we have some deadlines but I will apply and try this as soon as 
possible.
Regards...

Re: [Xenomai-help] ARM cached,non-cached memory

[Bosko Radivojevic]

On Mon, Mar 8, 2010 at 5:13 PM, Gilles Chanteperdrix
> Anyway, what I am interested in is a quick test with your
> application, measuring performances, so that we know whether
> implementing this feature completely is worth the trouble, and in that
> case, we will investigate further these issues.

Just for the sake of archiving our efforts, in the meantime I
performed some kind of testing and here are some, I hope, representing
results of this patch. System is AT91SAM9260 based (ARM926EJ-S), Linux
2.6.33.4 + Xenomai 2.5.3.

Writes are about 30% faster, while read is on the same performance
rate. Raw results are:

1: Results with Gilles patch, rt_heap created without H_NONCACHED:
 Writing:  11172ms
 Reading:  10232ms

2: Results with Gilles patch, rt_heap created with H_NONCACHED:
 Writing:  13207ms
 Reading: 19369ms

3: Results without Gilles patch, rt_heap created with H_NONCACHED:
 Writing:  18385ms
 Reading:  19425ms

Workload is simple, writing and reading 4 bytes at a time, int by int.
Something like 100mil iterations.

Gilles, thanks again :)