[Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Andrey Nechypurenko]

Hi Folks,

Following my tests with PWM generation using GPIO in user space [1],
I've made the RTDM module [2] to further reduce the jitter. As a
result, jitter was improved, but still under heavy system load the
servo motor I am trying to control starts shaking. Now, I fill stuck
and hope to get some help here.

At one hand, I can not imaging that 800MHz ARM (BeagleBoard xM) could
not manage to generate 20mS PWMs from RTDM driver precise enough to
avoid sporadic servo movements. So probably I am doing something
wrong. On the other hand, I do not see where the possible mistake can
happen and hope that someone experienced in with Xenomai could help.

There is an article about observed behavior [3] with more details, but
the core problem, I guess, boils down to the following code fragment:

void pwm_task_proc(void *arg) {
  const int which = (int)arg;

  // Toggling GPIO pin
  for(;;) {
      //set_data_out has offset 0x94 . Set gpio pin to 1 (up)
      iowrite32(0x40000000, gpio + 0x6094);

      // wait requested pulse width time (duty)
      if(0 != rtdm_task_sleep(up_period[which]))
        rtdm_printk("PWM: rtdm_task_sleep() returns error\n");

      //clear_data_out has offset 0x90 . Set gpio pin to 0 (down)
      iowrite32(0x40000000, gpio + 0x6090);

      // wait until the next pulse should start (20mS interval)
      if(0 != rtdm_task_wait_period())
        rtdm_printk("PWM: rtdm_task_wait_period() returns error\n");
  }

This is the function running as a periodic task started with the following call:

retval = rtdm_task_init(&pwm_task[i], // there is currently only one
element in this array
			"pwm-task",
			pwm_task_proc,
			0,
			RTDM_TASK_HIGHEST_PRIORITY,
			20000000); // 20ms period

It works fine until I put the system under about 95% load (just by
running tar jcf f.tbz2 /usr/lib). After that, the jitter of the
generated PWMs increasing to the level where it is not only observable
on the scope (as presented in [3]) but also servo motor starts
shaking, so it is clearly visible even without any measurements.

Honesly, I am failing to see any ways to improve this code. It is
already running in the kernel space and there are only memory writes
and RTDM wati functions involved. So I am wondering whether I hit the
limit of what is possible with Xenomai+Linux? I hope I am not and
would highly appreciate any hints or suggestions.

Thank you,
Andrey.

[1] http://veter-project.blogspot.com/2011/09/real-time-enough-about-pwms-and-shaky.html
[2] https://github.com/andreynech/rtdm-pwm
[3] http://veter-project.blogspot.com/2012/04/precise-pwms-with-gpio-using-xenomai.html

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Gilles Chanteperdrix]

On 04/23/2012 01:56 PM, Andrey Nechypurenko wrote:
> Hi Folks,
> 
> Following my tests with PWM generation using GPIO in user space [1],
> I've made the RTDM module [2] to further reduce the jitter. As a
> result, jitter was improved, but still under heavy system load the
> servo motor I am trying to control starts shaking. Now, I fill stuck
> and hope to get some help here.
> 
> At one hand, I can not imaging that 800MHz ARM (BeagleBoard xM) could
> not manage to generate 20mS PWMs from RTDM driver precise enough to
> avoid sporadic servo movements. So probably I am doing something
> wrong. On the other hand, I do not see where the possible mistake can
> happen and hope that someone experienced in with Xenomai could help.
> 
> There is an article about observed behavior [3] with more details, but
> the core problem, I guess, boils down to the following code fragment:
> 
> void pwm_task_proc(void *arg) {
>   const int which = (int)arg;
> 
>   // Toggling GPIO pin
>   for(;;) {
>       //set_data_out has offset 0x94 . Set gpio pin to 1 (up)
>       iowrite32(0x40000000, gpio + 0x6094);
> 
>       // wait requested pulse width time (duty)
>       if(0 != rtdm_task_sleep(up_period[which]))
>         rtdm_printk("PWM: rtdm_task_sleep() returns error\n");
> 
>       //clear_data_out has offset 0x90 . Set gpio pin to 0 (down)
>       iowrite32(0x40000000, gpio + 0x6090);
> 
>       // wait until the next pulse should start (20mS interval)
>       if(0 != rtdm_task_wait_period())
>         rtdm_printk("PWM: rtdm_task_wait_period() returns error\n");
>   }
> 
> This is the function running as a periodic task started with the following call:
> 
> retval = rtdm_task_init(&pwm_task[i], // there is currently only one
> element in this array
> 			"pwm-task",
> 			pwm_task_proc,
> 			0,
> 			RTDM_TASK_HIGHEST_PRIORITY,
> 			20000000); // 20ms period

Do not use a thread, use a timer.

-- 
					    Gilles.

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Andrey Nechypurenko]

>> retval = rtdm_task_init(&pwm_task[i], // there is currently only one
>> element in this array
>>                       "pwm-task",
>>                       pwm_task_proc,
>>                       0,
>>                       RTDM_TASK_HIGHEST_PRIORITY,
>>                       20000000); // 20ms period
>
> Do not use a thread, use a timer.

So you mean instead of starting periodic task with rtdm_task_init() it
is better use timer functions to trigger pin toggling piece of code?
Could you please elaborate on it a little? I thought that
rtdm_task_sleep() and rtdm_task_wait_period() are using timers
internally to wake up the thread at the right moment. Is not they?

Is not it a kind of work-around you suggesting? If there are some
problems which led to the imprecise timing of the sleep/wait functions
mentioned above, then, if technically possible, it would be better to
fix them instead of working around this behavior.

Anyway, I am really appreciate your help and will try to rewrite this
functionality using timer functions.

Thank you very much,
Andrey.

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Gilles Chanteperdrix]

On 04/23/2012 02:35 PM, Andrey Nechypurenko wrote:
>>> retval = rtdm_task_init(&pwm_task[i], // there is currently only one
>>> element in this array
>>>                       "pwm-task",
>>>                       pwm_task_proc,
>>>                       0,
>>>                       RTDM_TASK_HIGHEST_PRIORITY,
>>>                       20000000); // 20ms period
>>
>> Do not use a thread, use a timer.
> 
> So you mean instead of starting periodic task with rtdm_task_init() it
> is better use timer functions to trigger pin toggling piece of code?
> Could you please elaborate on it a little? I thought that
> rtdm_task_sleep() and rtdm_task_wait_period() are using timers
> internally to wake up the thread at the right moment. Is not they?

Yes, but once the timer is woken up, a context switch is needed to wake
up the thread, this adds time.

> 
> Is not it a kind of work-around you suggesting? If there are some
> problems which led to the imprecise timing of the sleep/wait functions
> mentioned above, then, if technically possible, it would be better to
> fix them instead of working around this behavior.

No, the time it takes to switch context between threads is unavoidable.
So, if you want to avoid it, you have to use a timer (rtdm_timer_init),
note that it is really common in RTOS interfaces to offer a timer API,
this is not a workaround at all.

The other alternative I describe in my last mail, that is, using a
dedicated hardware timer with its own irq handler, is a bit more of a
workaround, but still not uncommon in the RTOS world.

-- 
					    Gilles.

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Andrey Nechypurenko]

>>>> retval = rtdm_task_init(&pwm_task[i], // there is currently only one
>>>> element in this array
>>>>                       "pwm-task",
>>>>                       pwm_task_proc,
>>>>                       0,
>>>>                       RTDM_TASK_HIGHEST_PRIORITY,
>>>>                       20000000); // 20ms period
>>>
>>> Do not use a thread, use a timer.
>>
>> So you mean instead of starting periodic task with rtdm_task_init() it
>> is better use timer functions to trigger pin toggling piece of code?
>> Could you please elaborate on it a little? I thought that
>> rtdm_task_sleep() and rtdm_task_wait_period() are using timers
>> internally to wake up the thread at the right moment. Is not they?
>
> Yes, but once the timer is woken up, a context switch is needed to wake
> up the thread, this adds time.

I see... Nevertheless, I am surprised that context switch is so expensive.

> The other alternative I describe in my last mail, that is, using a
> dedicated hardware timer with its own irq handler, is a bit more of a
> workaround, but still not uncommon in the RTOS world.

Well, if nothing else would help, I'll try this way :-) .

Thank you!
Andrey.

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Philippe Gerum]

On 04/23/2012 02:57 PM, Andrey Nechypurenko wrote:
>>>>> retval = rtdm_task_init(&pwm_task[i], // there is currently only one
>>>>> element in this array
>>>>>                        "pwm-task",
>>>>>                        pwm_task_proc,
>>>>>                        0,
>>>>>                        RTDM_TASK_HIGHEST_PRIORITY,
>>>>>                        20000000); // 20ms period
>>>>
>>>> Do not use a thread, use a timer.
>>>
>>> So you mean instead of starting periodic task with rtdm_task_init() it
>>> is better use timer functions to trigger pin toggling piece of code?
>>> Could you please elaborate on it a little? I thought that
>>> rtdm_task_sleep() and rtdm_task_wait_period() are using timers
>>> internally to wake up the thread at the right moment. Is not they?
>>
>> Yes, but once the timer is woken up, a context switch is needed to wake
>> up the thread, this adds time.
>
> I see... Nevertheless, I am surprised that context switch is so expensive.

Keep in mind that in a dual kernel system, the RTOS part is competing 
with the linux kernel on core resources such as CPU caches. So each time 
your code sleeps for the next period, it leaves a 20 ms time window to 
linux to run and happily trash all I/D caches, which is a lot.

Now, if you have to schedule a thread, you have to traverse the Xenomai 
scheduling core to do the switch, in addition to the timer interrupt 
handling, which translates into significantly more code and data being 
accessed.

You can check this effect using the latency test: the faster the 
sampling period, the shorter the latencies, and conversely.

>
>> The other alternative I describe in my last mail, that is, using a
>> dedicated hardware timer with its own irq handler, is a bit more of a
>> workaround, but still not uncommon in the RTOS world.
>
> Well, if nothing else would help, I'll try this way :-) .
>
> Thank you!
> Andrey.
>
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
>


-- 
Philippe.

Re: [Xenomai] Jitter while generating PWMs with GPIO from RTDM driver

[Christoph]

Hi Andrey,

 
> 
> I see... Nevertheless, I am surprised that context switch is so expensive.

Especially on ARM9 Platform a context switch takes very long time. The
reason is the cache implementation, that requires to flush instruction cache
and data cache at each context switch.

There is a patch for the linux kernel called Fast Context Switch Extension
(FSCE), that speeds up the context switch a lot. If you have installed
i-pipe you should find it within the kernel config. Have a look at it's
restriction that are mentioned in the config's help.

Regards
Christoph

Re: [Xenomai] Jitter while generating PWMs with GPIO from RTDM driver

[Andrey Nechypurenko]

Hi Christoph,

> There is a patch for the linux kernel called Fast Context Switch Extension

Thanks for the reference! I will definitely take a look on it.

Thank you,
Andrey.

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Gilles Chanteperdrix]

On 04/23/2012 02:51 PM, Gilles Chanteperdrix wrote:
> On 04/23/2012 02:35 PM, Andrey Nechypurenko wrote:
>>>> retval = rtdm_task_init(&pwm_task[i], // there is currently only one
>>>> element in this array
>>>>                       "pwm-task",
>>>>                       pwm_task_proc,
>>>>                       0,
>>>>                       RTDM_TASK_HIGHEST_PRIORITY,
>>>>                       20000000); // 20ms period
>>>
>>> Do not use a thread, use a timer.
>>
>> So you mean instead of starting periodic task with rtdm_task_init() it
>> is better use timer functions to trigger pin toggling piece of code?
>> Could you please elaborate on it a little? I thought that
>> rtdm_task_sleep() and rtdm_task_wait_period() are using timers
>> internally to wake up the thread at the right moment. Is not they?
> 
> Yes, but once the timer is woken up, a context switch is needed to wake
> up the thread, this adds time.
> 
>>
>> Is not it a kind of work-around you suggesting? If there are some
>> problems which led to the imprecise timing of the sleep/wait functions
>> mentioned above, then, if technically possible, it would be better to
>> fix them instead of working around this behavior.
> 
> No, the time it takes to switch context between threads is unavoidable.
> So, if you want to avoid it, you have to use a timer (rtdm_timer_init),
> note that it is really common in RTOS interfaces to offer a timer API,
> this is not a workaround at all.
> 
> The other alternative I describe in my last mail, that is, using a
> dedicated hardware timer with its own irq handler, is a bit more of a
> workaround, but still not uncommon in the RTOS world.
> 

Another trick, which this time really is a workaround is to program the
timers to wake up a little bit early, and spin to wait for the exact
time wanted.

-- 
					    Gilles.

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Andrey Nechypurenko]

> Another trick, which this time really is a workaround is to program the
> timers to wake up a little bit early, and spin to wait for the exact
> time wanted.

Sounds like the black magic exercise! :-) I would try to stay on the
bright side as long as I can :-)

Thanks Gilles! It is really great to have such quick response from you!

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Gilles Chanteperdrix]

On 04/23/2012 02:27 PM, Gilles Chanteperdrix wrote:
> On 04/23/2012 01:56 PM, Andrey Nechypurenko wrote:
>> Hi Folks,
>>
>> Following my tests with PWM generation using GPIO in user space [1],
>> I've made the RTDM module [2] to further reduce the jitter. As a
>> result, jitter was improved, but still under heavy system load the
>> servo motor I am trying to control starts shaking. Now, I fill stuck
>> and hope to get some help here.
>>
>> At one hand, I can not imaging that 800MHz ARM (BeagleBoard xM) could
>> not manage to generate 20mS PWMs from RTDM driver precise enough to
>> avoid sporadic servo movements. So probably I am doing something
>> wrong. On the other hand, I do not see where the possible mistake can
>> happen and hope that someone experienced in with Xenomai could help.
>>
>> There is an article about observed behavior [3] with more details, but
>> the core problem, I guess, boils down to the following code fragment:
>>
>> void pwm_task_proc(void *arg) {
>>   const int which = (int)arg;
>>
>>   // Toggling GPIO pin
>>   for(;;) {
>>       //set_data_out has offset 0x94 . Set gpio pin to 1 (up)
>>       iowrite32(0x40000000, gpio + 0x6094);
>>
>>       // wait requested pulse width time (duty)
>>       if(0 != rtdm_task_sleep(up_period[which]))
>>         rtdm_printk("PWM: rtdm_task_sleep() returns error\n");
>>
>>       //clear_data_out has offset 0x90 . Set gpio pin to 0 (down)
>>       iowrite32(0x40000000, gpio + 0x6090);
>>
>>       // wait until the next pulse should start (20mS interval)
>>       if(0 != rtdm_task_wait_period())
>>         rtdm_printk("PWM: rtdm_task_wait_period() returns error\n");
>>   }
>>
>> This is the function running as a periodic task started with the following call:
>>
>> retval = rtdm_task_init(&pwm_task[i], // there is currently only one
>> element in this array
>> 			"pwm-task",
>> 			pwm_task_proc,
>> 			0,
>> 			RTDM_TASK_HIGHEST_PRIORITY,
>> 			20000000); // 20ms period
> 
> Do not use a thread, use a timer.
> 

So, rtdm_timer_init

If this still has too much overhead, you can use the OMAP3 API do
dedicate a GPTIMER to this task, and simply register an irq for the
GPTIMER which does the job (as opposed to a timer created with
rtdm_timer_init).

-- 
					    Gilles.

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Wolfgang Grandegger]

On 04/23/2012 01:56 PM, Andrey Nechypurenko wrote:
> Hi Folks,
> 
> Following my tests with PWM generation using GPIO in user space [1],
> I've made the RTDM module [2] to further reduce the jitter. As a
> result, jitter was improved, but still under heavy system load the
> servo motor I am trying to control starts shaking. Now, I fill stuck
> and hope to get some help here.

What jitters in micro-seconds are you speaking about? What are you
requirements? I have not found that information yet in this thread.

Wolfgang.

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Andrey Nechypurenko]

>> Following my tests with PWM generation using GPIO in user space [1],
>> I've made the RTDM module [2] to further reduce the jitter. As a
>> result, jitter was improved, but still under heavy system load the
>> servo motor I am trying to control starts shaking. Now, I fill stuck
>> and hope to get some help here.
>
> What jitters in micro-seconds are you speaking about? What are you
> requirements? I have not found that information yet in this thread.

The short answer is - I do not know exactly :-) .

The long answer is - my goal is to keep servo motor stable under the
high system load. I was unable to find any concrete numbers regarding
the tolerance level of the no-name servo motor. So I am just
experimenting. My recent observations, presented in the blog I've
mentioned in the original post, suggest that 20-30micro-seconds jitter
will force my concrete servo to start the movements.

I was unable to further reduce jitter, so maybe this number is
actually lower. I will try to implement what Gilles suggested (using
timer api) and see if I can a) further reduce jitter; and b) if this
jitter will be small enough to prevent servo from shaking.

Andrey.

Re: [Xenomai-help] Jitter while generating PWMs with GPIO from RTDM driver

[Wolfgang Grandegger]

On 04/23/2012 03:25 PM, Andrey Nechypurenko wrote:
>>> Following my tests with PWM generation using GPIO in user space [1],
>>> I've made the RTDM module [2] to further reduce the jitter. As a
>>> result, jitter was improved, but still under heavy system load the
>>> servo motor I am trying to control starts shaking. Now, I fill stuck
>>> and hope to get some help here.
>>
>> What jitters in micro-seconds are you speaking about? What are you
>> requirements? I have not found that information yet in this thread.
> 
> The short answer is - I do not know exactly :-) .
> 
> The long answer is - my goal is to keep servo motor stable under the
> high system load. I was unable to find any concrete numbers regarding
> the tolerance level of the no-name servo motor. So I am just
> experimenting. My recent observations, presented in the blog I've
> mentioned in the original post, suggest that 20-30micro-seconds jitter
> will force my concrete servo to start the movements.

You can run Xenomai's latency test under load to get an idea what you
can achieve on that system... and it will be more than 20-30us. Also try
the -t1 and t2 options.

> I was unable to further reduce jitter, so maybe this number is
> actually lower. I will try to implement what Gilles suggested (using
> timer api) and see if I can a) further reduce jitter; and b) if this
> jitter will be small enough to prevent servo from shaking.

The timer anticipation trick Gilles suggested would work fine, I think,
whithout costing too much (as your period is just 20ms).

Wolfgang.