[Xenomai] New domain migration process with ipipe-core and forge

[Xenomai] New domain migration process with ipipe-core and forge

[Jan Kiszka]

Hi Philippe,

I'm trying to understand how the new Linux-to-RT migration process works
under forge and specifically I-pipe core. It currently looks to me like
we are delaying this needlessly:

- __ipipe_migrate_head() suspends the calling Linux thread and triggers
  a context switch
- after the switch, __ipipe_switch_tail and then
  complete_domain_migration is invoked
- complete_domain_migration stalls head and calls ipipe_migration_hook.
  We are still running over the root domain.
- ipipe_migration_hook in forge wakes up the RT thread and calls
  xnpod_schedule
- as we are in root, xnarch_escalate triggers and calls ipipe_raise_irq
- as the head domain is stalled, dispatch_irq_head just marks the
  escalation VIRQ pending, not handler is called, no actual RT
  rescheduling can be performed on its return
- we eventually return to complete_domain_migration and simply clear
  the stall flag. No pending IRQs are process, again no VIRQ is
  delivered
- only when we receive the next IRQ or have some other reason for
  checking the head domain's IRQ log, the pending VIRQ will finally be
  processed

Did I miss something or should we actually perform some
__ipipe_sync_stage in complete_domain_migration to accelerate the switch?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] New domain migration process with ipipe-core and forge

[Jan Kiszka]

On 2013-03-26 20:17, Jan Kiszka wrote:
> Hi Philippe,
> 
> I'm trying to understand how the new Linux-to-RT migration process works
> under forge and specifically I-pipe core. It currently looks to me like
> we are delaying this needlessly:
> 
> - __ipipe_migrate_head() suspends the calling Linux thread and triggers
>   a context switch
> - after the switch, __ipipe_switch_tail and then
>   complete_domain_migration is invoked
> - complete_domain_migration stalls head and calls ipipe_migration_hook.
>   We are still running over the root domain.
> - ipipe_migration_hook in forge wakes up the RT thread and calls
>   xnpod_schedule
> - as we are in root, xnarch_escalate triggers and calls ipipe_raise_irq
> - as the head domain is stalled, dispatch_irq_head just marks the
>   escalation VIRQ pending, not handler is called, no actual RT
>   rescheduling can be performed on its return
> - we eventually return to complete_domain_migration and simply clear
>   the stall flag. No pending IRQs are process, again no VIRQ is
>   delivered
> - only when we receive the next IRQ or have some other reason for
>   checking the head domain's IRQ log, the pending VIRQ will finally be
>   processed
> 
> Did I miss something or should we actually perform some
> __ipipe_sync_stage in complete_domain_migration to accelerate the switch?

Or rather this:

diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
index 53e82ef..d164b4c 100644
--- a/kernel/ipipe/core.c
+++ b/kernel/ipipe/core.c
@@ -1102,6 +1102,7 @@ static void complete_domain_migration(void) /* hw IRQs off */
 	__set_bit(IPIPE_STALL_FLAG, &p->status);
 	ipipe_migration_hook(t);
 	__clear_bit(IPIPE_STALL_FLAG, &p->status);
+	__ipipe_do_sync_pipeline(p->domain);
 }
 
 #endif /* !CONFIG_IPIPE_LEGACY */


Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] New domain migration process with ipipe-core and forge

[Philippe Gerum]

On 03/26/2013 08:29 PM, Jan Kiszka wrote:
> On 2013-03-26 20:17, Jan Kiszka wrote:
>> Hi Philippe,
>>
>> I'm trying to understand how the new Linux-to-RT migration process works
>> under forge and specifically I-pipe core. It currently looks to me like
>> we are delaying this needlessly:
>>
>> - __ipipe_migrate_head() suspends the calling Linux thread and triggers
>>    a context switch
>> - after the switch, __ipipe_switch_tail and then
>>    complete_domain_migration is invoked
>> - complete_domain_migration stalls head and calls ipipe_migration_hook.
>>    We are still running over the root domain.
>> - ipipe_migration_hook in forge wakes up the RT thread and calls
>>    xnpod_schedule
>> - as we are in root, xnarch_escalate triggers and calls ipipe_raise_irq
>> - as the head domain is stalled, dispatch_irq_head just marks the
>>    escalation VIRQ pending, not handler is called, no actual RT
>>    rescheduling can be performed on its return
>> - we eventually return to complete_domain_migration and simply clear
>>    the stall flag. No pending IRQs are process, again no VIRQ is
>>    delivered
>> - only when we receive the next IRQ or have some other reason for
>>    checking the head domain's IRQ log, the pending VIRQ will finally be
>>    processed
>>
>> Did I miss something or should we actually perform some
>> __ipipe_sync_stage in complete_domain_migration to accelerate the switch?
>
> Or rather this:
>
> diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
> index 53e82ef..d164b4c 100644
> --- a/kernel/ipipe/core.c
> +++ b/kernel/ipipe/core.c
> @@ -1102,6 +1102,7 @@ static void complete_domain_migration(void) /* hw IRQs off */
>   	__set_bit(IPIPE_STALL_FLAG, &p->status);
>   	ipipe_migration_hook(t);
>   	__clear_bit(IPIPE_STALL_FLAG, &p->status);
> +	__ipipe_do_sync_pipeline(p->domain);
>   }
>
>   #endif /* !CONFIG_IPIPE_LEGACY */
>

Nitpicking: in theory, the code should expect the migration hook to have 
actually done the work, and not delayed it like Xenomai currently does 
in practice. A safer check would be as follows, allowing further 
arch-dependent handling in case the migration did succeed (typically for 
Blackfin):

diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
index 96b3a98..c4598ad 100644
--- a/kernel/ipipe/core.c
+++ b/kernel/ipipe/core.c
@@ -1102,6 +1102,8 @@ static void complete_domain_migration(void) /* hw 
IRQs off */
  	__set_bit(IPIPE_STALL_FLAG, &p->status);
  	ipipe_migration_hook(t);
  	__clear_bit(IPIPE_STALL_FLAG, &p->status);
+	if (__ipipe_ipending_p(p))
+		__ipipe_sync_pipeline(p->domain);
  }

  #endif /* !CONFIG_IPIPE_LEGACY */

-- 
Philippe.

Re: [Xenomai] New domain migration process with ipipe-core and forge

[Jan Kiszka]

On 2013-03-27 11:25, Philippe Gerum wrote:
> On 03/26/2013 08:29 PM, Jan Kiszka wrote:
>> On 2013-03-26 20:17, Jan Kiszka wrote:
>>> Hi Philippe,
>>>
>>> I'm trying to understand how the new Linux-to-RT migration process works
>>> under forge and specifically I-pipe core. It currently looks to me like
>>> we are delaying this needlessly:
>>>
>>> - __ipipe_migrate_head() suspends the calling Linux thread and triggers
>>>    a context switch
>>> - after the switch, __ipipe_switch_tail and then
>>>    complete_domain_migration is invoked
>>> - complete_domain_migration stalls head and calls ipipe_migration_hook.
>>>    We are still running over the root domain.
>>> - ipipe_migration_hook in forge wakes up the RT thread and calls
>>>    xnpod_schedule
>>> - as we are in root, xnarch_escalate triggers and calls ipipe_raise_irq
>>> - as the head domain is stalled, dispatch_irq_head just marks the
>>>    escalation VIRQ pending, not handler is called, no actual RT
>>>    rescheduling can be performed on its return
>>> - we eventually return to complete_domain_migration and simply clear
>>>    the stall flag. No pending IRQs are process, again no VIRQ is
>>>    delivered
>>> - only when we receive the next IRQ or have some other reason for
>>>    checking the head domain's IRQ log, the pending VIRQ will finally be
>>>    processed
>>>
>>> Did I miss something or should we actually perform some
>>> __ipipe_sync_stage in complete_domain_migration to accelerate the switch?
>>
>> Or rather this:
>>
>> diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
>> index 53e82ef..d164b4c 100644
>> --- a/kernel/ipipe/core.c
>> +++ b/kernel/ipipe/core.c
>> @@ -1102,6 +1102,7 @@ static void complete_domain_migration(void) /* hw IRQs off */
>>   	__set_bit(IPIPE_STALL_FLAG, &p->status);
>>   	ipipe_migration_hook(t);
>>   	__clear_bit(IPIPE_STALL_FLAG, &p->status);
>> +	__ipipe_do_sync_pipeline(p->domain);
>>   }
>>
>>   #endif /* !CONFIG_IPIPE_LEGACY */
>>
> 
> Nitpicking: in theory, the code should expect the migration hook to have 
> actually done the work, and not delayed it like Xenomai currently does 
> in practice.

Where does Xenomai do that? I was looking for it but didn't find a log
flush.

> A safer check would be as follows, allowing further 
> arch-dependent handling in case the migration did succeed (typically for 
> Blackfin):
> 
> diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
> index 96b3a98..c4598ad 100644
> --- a/kernel/ipipe/core.c
> +++ b/kernel/ipipe/core.c
> @@ -1102,6 +1102,8 @@ static void complete_domain_migration(void) /* hw 
> IRQs off */
>   	__set_bit(IPIPE_STALL_FLAG, &p->status);
>   	ipipe_migration_hook(t);
>   	__clear_bit(IPIPE_STALL_FLAG, &p->status);
> +	if (__ipipe_ipending_p(p))
> +		__ipipe_sync_pipeline(p->domain);
>   }
> 
>   #endif /* !CONFIG_IPIPE_LEGACY */
> 

OK, will queue this up here for the next pull request.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] New domain migration process with ipipe-core and forge

[Philippe Gerum]

On 03/27/2013 11:27 AM, Jan Kiszka wrote:

>> Nitpicking: in theory, the code should expect the migration hook to have
>> actually done the work, and not delayed it like Xenomai currently does
>> in practice.
>
> Where does Xenomai do that? I was looking for it but didn't find a log
> flush.
>

You mean delaying? Check per-arch xnarch_escalate(). Flushing occurs as 
soon as the xnpod_schedule() caller unstalls the head domain, which has 
to happen quickly after the delay was enforced.

>> diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
>> index 96b3a98..c4598ad 100644
>> --- a/kernel/ipipe/core.c
>> +++ b/kernel/ipipe/core.c
>> @@ -1102,6 +1102,8 @@ static void complete_domain_migration(void) /* hw
>> IRQs off */
>>
> OK, will queue this up here for the next pull request.
>

Ok.


-- 
Philippe.

Re: [Xenomai] New domain migration process with ipipe-core and forge

[Jan Kiszka]

On 2013-03-27 11:32, Philippe Gerum wrote:
> On 03/27/2013 11:27 AM, Jan Kiszka wrote:
> 
>>> Nitpicking: in theory, the code should expect the migration hook to have
>>> actually done the work, and not delayed it like Xenomai currently does
>>> in practice.
>>
>> Where does Xenomai do that? I was looking for it but didn't find a log
>> flush.
>>
> 
> You mean delaying? Check per-arch xnarch_escalate(). Flushing occurs as 
> soon as the xnpod_schedule() caller unstalls the head domain, which has 
> to happen quickly after the delay was enforced.

Hmm, I still do not see where we should flush in Xenomai forge before
the return to complete_domain_migration and its clearance of the stall flag.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] New domain migration process with ipipe-core and forge

[Philippe Gerum]

On 03/27/2013 11:35 AM, Jan Kiszka wrote:
> On 2013-03-27 11:32, Philippe Gerum wrote:
>> On 03/27/2013 11:27 AM, Jan Kiszka wrote:
>>
>>>> Nitpicking: in theory, the code should expect the migration hook to have
>>>> actually done the work, and not delayed it like Xenomai currently does
>>>> in practice.
>>>
>>> Where does Xenomai do that? I was looking for it but didn't find a log
>>> flush.
>>>
>>
>> You mean delaying? Check per-arch xnarch_escalate(). Flushing occurs as
>> soon as the xnpod_schedule() caller unstalls the head domain, which has
>> to happen quickly after the delay was enforced.
>
> Hmm, I still do not see where we should flush in Xenomai forge before
> the return to complete_domain_migration and its clearance of the stall flag.
>

With the new migration interface, we don't have to do that from Xenomai, 
since the migration hook assumes it is called with the head domain 
stalled, so the caller has to unstall shortly after anyway. But for that 
to work, we need your latest fix.

2.x was immune because everything happened on behalf of the gatekeeper, 
which triggered a rescheduling with head unstalled. This is how the bug 
slipped into the -forge logic.

-- 
Philippe.

Re: [Xenomai] New domain migration process with ipipe-core and forge

[Philippe Gerum]

On 03/26/2013 08:17 PM, Jan Kiszka wrote:
> Hi Philippe,
>
> I'm trying to understand how the new Linux-to-RT migration process works
> under forge and specifically I-pipe core. It currently looks to me like
> we are delaying this needlessly:
>
> - __ipipe_migrate_head() suspends the calling Linux thread and triggers
>    a context switch
> - after the switch, __ipipe_switch_tail and then
>    complete_domain_migration is invoked
> - complete_domain_migration stalls head and calls ipipe_migration_hook.
>    We are still running over the root domain.
> - ipipe_migration_hook in forge wakes up the RT thread and calls
>    xnpod_schedule
> - as we are in root, xnarch_escalate triggers and calls ipipe_raise_irq
> - as the head domain is stalled, dispatch_irq_head just marks the
>    escalation VIRQ pending, not handler is called, no actual RT
>    rescheduling can be performed on its return
> - we eventually return to complete_domain_migration and simply clear
>    the stall flag. No pending IRQs are process, again no VIRQ is
>    delivered
> - only when we receive the next IRQ or have some other reason for
>    checking the head domain's IRQ log, the pending VIRQ will finally be
>    processed
>
> Did I miss something or should we actually perform some
> __ipipe_sync_stage in complete_domain_migration to accelerate the switch?
>

Yep. The current code wrongly assumes that we are covered by a root 
unstall from finish_task_switch -> finish_lock_switch, but we are not 
since the pipeline won't consider the head domain log when deciding 
whether irqs should be synced.

-- 
Philippe.