[Xenomai] xenomai , irq and pci

[Xenomai] xenomai , irq and pci

[Johann Obermayr]

Hello,

celeron dual core with PCIe - PCI bridge
we use kernel 3.10.xx smp with  (isolcpus=0)
smi is disabled.
xenomai 2.6.3

we have a pci card with a fpga (with irq) and sram.

we have 2 task running.

task0 is one high prior in primary domain on core0.
this task wait for event, that is fired by fpga irq.
than this task will make some pci accesses to the fpga.

the other task (task1) is a shadow task, but running in secondary domain 
on core 1.
This task make many big memcpy to sram.

in the ipipe_handler_irq we have a function callback.
a small driver hooks to this callback add make some tracepoint into
a shared memory.

with the trace we see some time a very high jitter in the ipipe_handler_irq
function (around 250 us).
has anyone an explanation for this?

and a 4 byte access from task0 to fpga need more than 300us,
  while task1 is writing to sram.

Thanks
   Johann

Re: [Xenomai] xenomai , irq and pci

[Gilles Chanteperdrix]

On 03/05/2014 05:22 PM, Johann Obermayr wrote:
> Hello,
> 
> celeron dual core with PCIe - PCI bridge
> we use kernel 3.10.xx smp with  (isolcpus=0)
> smi is disabled.
> xenomai 2.6.3
> 
> we have a pci card with a fpga (with irq) and sram.
> 
> we have 2 task running.
> 
> task0 is one high prior in primary domain on core0.
> this task wait for event, that is fired by fpga irq.
> than this task will make some pci accesses to the fpga.
> 
> the other task (task1) is a shadow task, but running in secondary domain 
> on core 1.
> This task make many big memcpy to sram.
> 
> in the ipipe_handler_irq we have a function callback.
> a small driver hooks to this callback add make some tracepoint into
> a shared memory.
> 
> with the trace we see some time a very high jitter in the ipipe_handler_irq
> function (around 250 us).
> has anyone an explanation for this?
> 
> and a 4 byte access from task0 to fpga need more than 300us,
>   while task1 is writing to sram.

Maybe the problem is the size of the PCI burst when writing to the SRAM?
How large is it? If the sram is mapped with "write-combine", it will be
buffered and written in large bursts. Maybe you could try mapping the
sram area uncached?

-- 
                                                                Gilles.

Re: [Xenomai] xenomai , irq and pci

[Gilles Chanteperdrix]

On 03/05/2014 05:41 PM, Gilles Chanteperdrix wrote:
> On 03/05/2014 05:22 PM, Johann Obermayr wrote:
>> Hello,
>>
>> celeron dual core with PCIe - PCI bridge
>> we use kernel 3.10.xx smp with  (isolcpus=0)
>> smi is disabled.
>> xenomai 2.6.3
>>
>> we have a pci card with a fpga (with irq) and sram.
>>
>> we have 2 task running.
>>
>> task0 is one high prior in primary domain on core0.
>> this task wait for event, that is fired by fpga irq.
>> than this task will make some pci accesses to the fpga.
>>
>> the other task (task1) is a shadow task, but running in secondary domain 
>> on core 1.
>> This task make many big memcpy to sram.
>>
>> in the ipipe_handler_irq we have a function callback.
>> a small driver hooks to this callback add make some tracepoint into
>> a shared memory.
>>
>> with the trace we see some time a very high jitter in the ipipe_handler_irq
>> function (around 250 us).
>> has anyone an explanation for this?
>>
>> and a 4 byte access from task0 to fpga need more than 300us,
>>   while task1 is writing to sram.
> 
> Maybe the problem is the size of the PCI burst when writing to the SRAM?
> How large is it? If the sram is mapped with "write-combine", it will be
> buffered and written in large bursts. Maybe you could try mapping the
> sram area uncached?
> 

Also, you do not tell us if your processor has two real cores, or if
they are hyperthreaded?

-- 
                                                                Gilles.

Re: [Xenomai] xenomai , irq and pci

[Jeroen Van den Keybus]

>
> >> task0 is one high prior in primary domain on core0.
> >> this task wait for event, that is fired by fpga irq.
> >> than this task will make some pci accesses to the fpga.
>

A while ago (11/02) you wrote to this list ('[Xenomai] Xenomai and pci
access') because you had issues with PCI transfers. Did you fix those ? How
large are your transfers ?

>> the other task (task1) is a shadow task, but running in secondary domain
> >> on core 1.
> >> This task make many big memcpy to sram.
>

I remember having asked for the complete PCI bus layout and to try out
other memory access methods besides memcpy. Did this do anything ?

Since you mention you have an FPGA, I would now suggest instrumenting it
(add some logging) as to find out when and how (regular intervals or with
interruptions) your FPGA is accessed by the PCI bus.


J.

 >>
> >> in the ipipe_handler_irq we have a function callback.
> >> a small driver hooks to this callback add make some tracepoint into
> >> a shared memory.
> >>
> >> with the trace we see some time a very high jitter in the
> ipipe_handler_irq
> >> function (around 250 us).
> >> has anyone an explanation for this?
> >>
> >> and a 4 byte access from task0 to fpga need more than 300us,
> >>   while task1 is writing to sram.
> >
> > Maybe the problem is the size of the PCI burst when writing to the SRAM?
> > How large is it? If the sram is mapped with "write-combine", it will be
> > buffered and written in large bursts. Maybe you could try mapping the
> > sram area uncached?
> >
>
> Also, you do not tell us if your processor has two real cores, or if
> they are hyperthreaded?
>
> --
>                                                                 Gilles.
>
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> http://www.xenomai.org/mailman/listinfo/xenomai
>

Re: [Xenomai] xenomai , irq and pci

[Johann Obermayr]

Am 05.03.2014 20:59, schrieb Jeroen Van den Keybus:
>
>     >> task0 is one high prior in primary domain on core0.
>     >> this task wait for event, that is fired by fpga irq.
>     >> than this task will make some pci accesses to the fpga.
>
>
> A while ago (11/02) you wrote to this list ('[Xenomai] Xenomai and pci 
> access') because you had issues with PCI transfers. Did you fix those 
> ? How large are your transfers ?
The old part we have fixed. (but not very beautyful). But it work.

Our write transfer has 10kByte, and this need 1,8 ms. And our system 
work with
1ms ticks. That  mean , our rt-task will start every 1ms.

>
>     >> the other task (task1) is a shadow task, but running in
>     secondary domain
>     >> on core 1.
>     >> This task make many big memcpy to sram.
>
>
> I remember having asked for the complete PCI bus layout and to try out 
> other memory access methods besides memcpy. Did this do anything ?
>
We can't change the memcpy function.

> Since you mention you have an FPGA, I would now suggest instrumenting 
> it (add some logging) as to find out when and how (regular intervals 
> or with interruptions) your FPGA is accessed by the PCI bus.
>
Now we will measure all available hardware points.
FPGA output, PCI bus, PCIe ??,  and than the interrupt line to the core.

Re: [Xenomai] xenomai , irq and pci

[Johann Obermayr]

Am 05.03.2014 17:41, schrieb Gilles Chanteperdrix:
> On 03/05/2014 05:22 PM, Johann Obermayr wrote:
>> Hello,
>>
>> celeron dual core with PCIe - PCI bridge
>> we use kernel 3.10.xx smp with  (isolcpus=0)
>> smi is disabled.
>> xenomai 2.6.3
>>
>> we have a pci card with a fpga (with irq) and sram.
>>
>> we have 2 task running.
>>
>> task0 is one high prior in primary domain on core0.
>> this task wait for event, that is fired by fpga irq.
>> than this task will make some pci accesses to the fpga.
>>
>> the other task (task1) is a shadow task, but running in secondary domain
>> on core 1.
>> This task make many big memcpy to sram.
>>
>> in the ipipe_handler_irq we have a function callback.
>> a small driver hooks to this callback add make some tracepoint into
>> a shared memory.
>>
>> with the trace we see some time a very high jitter in the ipipe_handler_irq
>> function (around 250 us).
>> has anyone an explanation for this?
>>
>> and a 4 byte access from task0 to fpga need more than 300us,
>>    while task1 is writing to sram.
> Maybe the problem is the size of the PCI burst when writing to the SRAM?
> How large is it? If the sram is mapped with "write-combine", it will be
> buffered and written in large bursts. Maybe you could try mapping the
> sram area uncached?
>
Hi,

we use 2 real cores, no HT.
task1 do a memcpy with 10000 bytes.

For mapping we use, rtdm_iomap_to_user
rval = rtdm_iomap_to_user(user_info, lv.phys_addr, lv.len, PROT_READ | 
PROT_WRITE, &lv.virt_addr, NULL, NULL);

Thanks
   Johann

Re: [Xenomai] xenomai , irq and pci

[Gilles Chanteperdrix]

On 03/06/2014 09:25 AM, Johann Obermayr wrote:
> Am 05.03.2014 17:41, schrieb Gilles Chanteperdrix:
>> On 03/05/2014 05:22 PM, Johann Obermayr wrote:
>>> Hello,
>>>
>>> celeron dual core with PCIe - PCI bridge
>>> we use kernel 3.10.xx smp with  (isolcpus=0)
>>> smi is disabled.
>>> xenomai 2.6.3
>>>
>>> we have a pci card with a fpga (with irq) and sram.
>>>
>>> we have 2 task running.
>>>
>>> task0 is one high prior in primary domain on core0.
>>> this task wait for event, that is fired by fpga irq.
>>> than this task will make some pci accesses to the fpga.
>>>
>>> the other task (task1) is a shadow task, but running in secondary domain
>>> on core 1.
>>> This task make many big memcpy to sram.
>>>
>>> in the ipipe_handler_irq we have a function callback.
>>> a small driver hooks to this callback add make some tracepoint into
>>> a shared memory.
>>>
>>> with the trace we see some time a very high jitter in the ipipe_handler_irq
>>> function (around 250 us).
>>> has anyone an explanation for this?
>>>
>>> and a 4 byte access from task0 to fpga need more than 300us,
>>>    while task1 is writing to sram.
>> Maybe the problem is the size of the PCI burst when writing to the SRAM?
>> How large is it? If the sram is mapped with "write-combine", it will be
>> buffered and written in large bursts. Maybe you could try mapping the
>> sram area uncached?
>>
> Hi,
> 
> we use 2 real cores, no HT.
> task1 do a memcpy with 10000 bytes.

have you tried splitting the copy in smaller chunks, and use a mutex for 
exclusion between the two threads?

> 
> For mapping we use, rtdm_iomap_to_user
> rval = rtdm_iomap_to_user(user_info, lv.phys_addr, lv.len, PROT_READ | 
> PROT_WRITE, &lv.virt_addr, NULL, NULL);

You may want to try the following patch:

http://git.xenomai.org/xenomai-2.6.git/commit/?id=1b8fb14e486ac0604e44067a0f1f89dc14153e95



-- 
                                                                Gilles.

Re: [Xenomai] xenomai , irq and pci

[Johann Obermayr]

Am 06.03.2014 20:20, schrieb Gilles Chanteperdrix:
> On 03/06/2014 09:25 AM, Johann Obermayr wrote:
>> Am 05.03.2014 17:41, schrieb Gilles Chanteperdrix:
>>> On 03/05/2014 05:22 PM, Johann Obermayr wrote:
>>>> Hello,
>>>>
>>>> celeron dual core with PCIe - PCI bridge
>>>> we use kernel 3.10.xx smp with  (isolcpus=0)
>>>> smi is disabled.
>>>> xenomai 2.6.3
>>>>
>>>> we have a pci card with a fpga (with irq) and sram.
>>>>
>>>> we have 2 task running.
>>>>
>>>> task0 is one high prior in primary domain on core0.
>>>> this task wait for event, that is fired by fpga irq.
>>>> than this task will make some pci accesses to the fpga.
>>>>
>>>> the other task (task1) is a shadow task, but running in secondary domain
>>>> on core 1.
>>>> This task make many big memcpy to sram.
>>>>
>>>> in the ipipe_handler_irq we have a function callback.
>>>> a small driver hooks to this callback add make some tracepoint into
>>>> a shared memory.
>>>>
>>>> with the trace we see some time a very high jitter in the ipipe_handler_irq
>>>> function (around 250 us).
>>>> has anyone an explanation for this?
>>>>
>>>> and a 4 byte access from task0 to fpga need more than 300us,
>>>>     while task1 is writing to sram.
>>> Maybe the problem is the size of the PCI burst when writing to the SRAM?
>>> How large is it? If the sram is mapped with "write-combine", it will be
>>> buffered and written in large bursts. Maybe you could try mapping the
>>> sram area uncached?
>>>
>> Hi,
>>
>> we use 2 real cores, no HT.
>> task1 do a memcpy with 10000 bytes.
> have you tried splitting the copy in smaller chunks, and use a mutex for
> exclusion between the two threads?
That can we do.
Only in our testsystem, but in customer application, we have no way to 
change this.
>> For mapping we use, rtdm_iomap_to_user
>> rval = rtdm_iomap_to_user(user_info, lv.phys_addr, lv.len, PROT_READ |
>> PROT_WRITE, &lv.virt_addr, NULL, NULL);
> You may want to try the following patch:
>
> http://git.xenomai.org/xenomai-2.6.git/commit/?id=1b8fb14e486ac0604e44067a0f1f89dc14153e95
>
>
we will test this.

Thanks,
   Johann

Re: [Xenomai] xenomai , irq and pci

[Johann Obermayr]

Am 06.03.2014 20:46, schrieb Gilles Chanteperdrix:
> On 03/06/2014 08:44 PM, Johann Obermayr wrote:
>> Am 06.03.2014 20:20, schrieb Gilles Chanteperdrix:
>>> On 03/06/2014 09:25 AM, Johann Obermayr wrote:
>>>> Am 05.03.2014 17:41, schrieb Gilles Chanteperdrix:
>>>>> On 03/05/2014 05:22 PM, Johann Obermayr wrote:
>>>>>> Hello,
>>>>>>
>>>>>> celeron dual core with PCIe - PCI bridge
>>>>>> we use kernel 3.10.xx smp with  (isolcpus=0)
>>>>>> smi is disabled.
>>>>>> xenomai 2.6.3
>>>>>>
>>>>>> we have a pci card with a fpga (with irq) and sram.
>>>>>>
>>>>>> we have 2 task running.
>>>>>>
>>>>>> task0 is one high prior in primary domain on core0.
>>>>>> this task wait for event, that is fired by fpga irq.
>>>>>> than this task will make some pci accesses to the fpga.
>>>>>>
>>>>>> the other task (task1) is a shadow task, but running in secondary domain
>>>>>> on core 1.
>>>>>> This task make many big memcpy to sram.
>>>>>>
>>>>>> in the ipipe_handler_irq we have a function callback.
>>>>>> a small driver hooks to this callback add make some tracepoint into
>>>>>> a shared memory.
>>>>>>
>>>>>> with the trace we see some time a very high jitter in the ipipe_handler_irq
>>>>>> function (around 250 us).
>>>>>> has anyone an explanation for this?
>>>>>>
>>>>>> and a 4 byte access from task0 to fpga need more than 300us,
>>>>>>      while task1 is writing to sram.
>>>>> Maybe the problem is the size of the PCI burst when writing to the SRAM?
>>>>> How large is it? If the sram is mapped with "write-combine", it will be
>>>>> buffered and written in large bursts. Maybe you could try mapping the
>>>>> sram area uncached?
>>>>>
>>>> Hi,
>>>>
>>>> we use 2 real cores, no HT.
>>>> task1 do a memcpy with 10000 bytes.
>>> have you tried splitting the copy in smaller chunks, and use a mutex for
>>> exclusion between the two threads?
>> That can we do.
>> Only in our testsystem, but in customer application, we have no way to
>> change this.
> Ok, I suppose you can change the FPGA then? Have you tried to make the
> BAR with the SRAM appear as register intead of memory?
>
i'll talk to the fpga department.