[PATCH] sched/rt: overrun could happen in start_hrtick_dl

[PATCH] sched/rt: overrun could happen in start_hrtick_dl

[xiaofeng.yan]

It could be wrong for the precision of runtime and deadline
when the precision is within microsecond level. For example:
Task runtime deadline period
 P1   200us   500us   500us

This case need enbale HRTICK feature by the next command
PC#echo "HRTICK" > /sys/kernel/debug/sched_features
PC#./schedtool -E -t 200000:500000 -e ./test&
PC#trace-cmd record -e sched_switch

Some of runtime and deadline run with millisecond level by
reading kernershark.
The problem is caused by a conditional judgment "delta > 10000".
Because no hrtimer start up to control the runtime when runtime is less than 10us.
So the process will continue to run until tick-period coming.
For fixing this problem, Let delta is equal to 10us when it is less than 10us.
So the hrtimer will start up to control the end of process.

Signed-off-by: xiaofeng.yan <xiaofeng.yan@huawei.com>
---
 kernel/sched/deadline.c |    6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index fc4f98b1..dfefa82 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -997,10 +997,8 @@ static void check_preempt_curr_dl(struct rq *rq, struct task_struct *p,
 #ifdef CONFIG_SCHED_HRTICK
 static void start_hrtick_dl(struct rq *rq, struct task_struct *p)
 {
-	s64 delta = p->dl.dl_runtime - p->dl.runtime;
-
-	if (delta > 10000)
-		hrtick_start(rq, p->dl.runtime);
+	s64 delta = p->dl.runtime > 10000 ? p->dl.runtime : 10000;
+	hrtick_start(rq, delta);
 }
 #endif
 
-- 
1.7.9.5

Re: [PATCH] sched/rt: overrun could happen in start_hrtick_dl

[Peter Zijlstra]

[-- Attachment #1: Type: text/plain, Size: 2139 bytes --]

On Fri, Jul 04, 2014 at 12:02:21PM +0000, xiaofeng.yan wrote:
> It could be wrong for the precision of runtime and deadline
> when the precision is within microsecond level. For example:
> Task runtime deadline period
>  P1   200us   500us   500us
> 
> This case need enbale HRTICK feature by the next command
> PC#echo "HRTICK" > /sys/kernel/debug/sched_features
> PC#./schedtool -E -t 200000:500000 -e ./test&
> PC#trace-cmd record -e sched_switch

Are you actually using HRTICK ?

> Some of runtime and deadline run with millisecond level by
> reading kernershark.
> The problem is caused by a conditional judgment "delta > 10000".
> Because no hrtimer start up to control the runtime when runtime is less than 10us.
> So the process will continue to run until tick-period coming.
> For fixing this problem, Let delta is equal to 10us when it is less than 10us.
> So the hrtimer will start up to control the end of process.
> 
> Signed-off-by: xiaofeng.yan <xiaofeng.yan@huawei.com>

Always when sending patches for deadline, also CC Juri.

> ---
>  kernel/sched/deadline.c |    6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
> index fc4f98b1..dfefa82 100644
> --- a/kernel/sched/deadline.c
> +++ b/kernel/sched/deadline.c
> @@ -997,10 +997,8 @@ static void check_preempt_curr_dl(struct rq *rq, struct task_struct *p,
>  #ifdef CONFIG_SCHED_HRTICK
>  static void start_hrtick_dl(struct rq *rq, struct task_struct *p)
>  {
> -	s64 delta = p->dl.dl_runtime - p->dl.runtime;
> -
> -	if (delta > 10000)
> -		hrtick_start(rq, p->dl.runtime);
> +	s64 delta = p->dl.runtime > 10000 ? p->dl.runtime : 10000;
> +	hrtick_start(rq, delta);

Yeah, that looks funny. And seeing how the only other user does
something similar:

hrtick_start_fair()
	delta = max(10000ULL, delta)
	hrtick_start(rq, delta)


Does it make sense to move this max() into hrtick_start()?

Also; and I don't think you mentioned that but did fix, the argument to
hrtick_start() is wrong, it should be the delta, not the absolute
timeout.

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH] sched/rt: overrun could happen in start_hrtick_dl

[xiaofeng.yan]

On 2014/7/7 16:41, Peter Zijlstra wrote:
> On Fri, Jul 04, 2014 at 12:02:21PM +0000, xiaofeng.yan wrote:
>> It could be wrong for the precision of runtime and deadline
>> when the precision is within microsecond level. For example:
>> Task runtime deadline period
>>   P1   200us   500us   500us
>>
>> This case need enbale HRTICK feature by the next command
>> PC#echo "HRTICK" > /sys/kernel/debug/sched_features
>> PC#./schedtool -E -t 200000:500000 -e ./test&
>> PC#trace-cmd record -e sched_switch
> Are you actually using HRTICK ?
yes, If HRTICK is close , then all of runtime and deadline will be wrong.
>> Some of runtime and deadline run with millisecond level by
>> reading kernershark.
>> The problem is caused by a conditional judgment "delta > 10000".
>> Because no hrtimer start up to control the runtime when runtime is less than 10us.
>> So the process will continue to run until tick-period coming.
>> For fixing this problem, Let delta is equal to 10us when it is less than 10us.
>> So the hrtimer will start up to control the end of process.
>>
>> Signed-off-by: xiaofeng.yan <xiaofeng.yan@huawei.com>
> Always when sending patches for deadline, also CC Juri.
>
>> ---
>>   kernel/sched/deadline.c |    6 ++----
>>   1 file changed, 2 insertions(+), 4 deletions(-)
>>
>> diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
>> index fc4f98b1..dfefa82 100644
>> --- a/kernel/sched/deadline.c
>> +++ b/kernel/sched/deadline.c
>> @@ -997,10 +997,8 @@ static void check_preempt_curr_dl(struct rq *rq, struct task_struct *p,
>>   #ifdef CONFIG_SCHED_HRTICK
>>   static void start_hrtick_dl(struct rq *rq, struct task_struct *p)
>>   {
>> -	s64 delta = p->dl.dl_runtime - p->dl.runtime;
>> -
>> -	if (delta > 10000)
>> -		hrtick_start(rq, p->dl.runtime);
>> +	s64 delta = p->dl.runtime > 10000 ? p->dl.runtime : 10000;
>> +	hrtick_start(rq, delta);
> Yeah, that looks funny. And seeing how the only other user does
> something similar:
>
> hrtick_start_fair()
> 	delta = max(10000ULL, delta)
> 	hrtick_start(rq, delta)
I will modify my code according to your suggest.
>
> Does it make sense to move this max() into hrtick_start()?
>
> Also; and I don't think you mentioned that but did fix, the argument to
> hrtick_start() is wrong, it should be the delta, not the absolute
> timeout.
Perhaps , if the runtime  is less than 10us,  the context switch 
overhead for system could be closed to 10us.
So it could loss more then you gain.

Thanks for your reply.

Thanks
Yan

Re: [PATCH] sched/rt: overrun could happen in start_hrtick_dl

[Li Zefan]

On 2014/7/8 9:10, xiaofeng.yan wrote:
> On 2014/7/7 16:41, Peter Zijlstra wrote:
>> On Fri, Jul 04, 2014 at 12:02:21PM +0000, xiaofeng.yan wrote:
>>> It could be wrong for the precision of runtime and deadline
>>> when the precision is within microsecond level. For example:
>>> Task runtime deadline period
>>>   P1   200us   500us   500us
>>>
>>> This case need enbale HRTICK feature by the next command
>>> PC#echo "HRTICK" > /sys/kernel/debug/sched_features
>>> PC#./schedtool -E -t 200000:500000 -e ./test&
>>> PC#trace-cmd record -e sched_switch
>> Are you actually using HRTICK ?
> yes, If HRTICK is close , then all of runtime and deadline will be wrong.

I think what peter meant is, do you use HRTICK in products or
just use it for testing/experiment?

Re: [PATCH] sched/rt: overrun could happen in start_hrtick_dl

[xiaofeng.yan]

On 2014/7/8 10:40, Li Zefan wrote:
> On 2014/7/8 9:10, xiaofeng.yan wrote:
>> On 2014/7/7 16:41, Peter Zijlstra wrote:
>>> On Fri, Jul 04, 2014 at 12:02:21PM +0000, xiaofeng.yan wrote:
>>>> It could be wrong for the precision of runtime and deadline
>>>> when the precision is within microsecond level. For example:
>>>> Task runtime deadline period
>>>>    P1   200us   500us   500us
>>>>
>>>> This case need enbale HRTICK feature by the next command
>>>> PC#echo "HRTICK" > /sys/kernel/debug/sched_features
>>>> PC#./schedtool -E -t 200000:500000 -e ./test&
>>>> PC#trace-cmd record -e sched_switch
>>> Are you actually using HRTICK ?
>> yes, If HRTICK is close , then all of runtime and deadline will be wrong.
> I think what peter meant is, do you use HRTICK in products or
> just use it for testing/experiment?
>
Thanks for your timely comments. In fact, We use HRTICK feature in 
product. We need microsecond level
precision.

Thanks
Yan
> .
>

Re: [PATCH] sched/rt: overrun could happen in start_hrtick_dl

[Peter Zijlstra]

[-- Attachment #1: Type: text/plain, Size: 1349 bytes --]

On Tue, Jul 08, 2014 at 10:51:02AM +0800, xiaofeng.yan wrote:
> On 2014/7/8 10:40, Li Zefan wrote:
> >On 2014/7/8 9:10, xiaofeng.yan wrote:
> >>On 2014/7/7 16:41, Peter Zijlstra wrote:
> >>>On Fri, Jul 04, 2014 at 12:02:21PM +0000, xiaofeng.yan wrote:
> >>>>It could be wrong for the precision of runtime and deadline
> >>>>when the precision is within microsecond level. For example:
> >>>>Task runtime deadline period
> >>>>   P1   200us   500us   500us
> >>>>
> >>>>This case need enbale HRTICK feature by the next command
> >>>>PC#echo "HRTICK" > /sys/kernel/debug/sched_features
> >>>>PC#./schedtool -E -t 200000:500000 -e ./test&
> >>>>PC#trace-cmd record -e sched_switch
> >>>Are you actually using HRTICK ?
> >>yes, If HRTICK is close , then all of runtime and deadline will be wrong.
> >I think what peter meant is, do you use HRTICK in products or
> >just use it for testing/experiment?
> >
> Thanks for your timely comments. In fact, We use HRTICK feature in product.
> We need microsecond level
> precision.

Ah, thanks. Be advised that currently HRTICK is rather expensive. The
cost is twofold: 1) doing all the kernel side hrtimer things and 2)
programming clock hardware.

Of course, if that's what you need, you're willing to pay the price.

I'll see if I can put making it less expensive slightly higher on the
(endless) todo list.

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH] sched/rt: overrun could happen in start_hrtick_dl

[xiaofeng.yan]

On 2014/7/8 15:49, Peter Zijlstra wrote:
> On Tue, Jul 08, 2014 at 10:51:02AM +0800, xiaofeng.yan wrote:
>> On 2014/7/8 10:40, Li Zefan wrote:
>>> On 2014/7/8 9:10, xiaofeng.yan wrote:
>>>> On 2014/7/7 16:41, Peter Zijlstra wrote:
>>>>> On Fri, Jul 04, 2014 at 12:02:21PM +0000, xiaofeng.yan wrote:
>>>>>> It could be wrong for the precision of runtime and deadline
>>>>>> when the precision is within microsecond level. For example:
>>>>>> Task runtime deadline period
>>>>>>    P1   200us   500us   500us
>>>>>>
>>>>>> This case need enbale HRTICK feature by the next command
>>>>>> PC#echo "HRTICK" > /sys/kernel/debug/sched_features
>>>>>> PC#./schedtool -E -t 200000:500000 -e ./test&
>>>>>> PC#trace-cmd record -e sched_switch
>>>>> Are you actually using HRTICK ?
>>>> yes, If HRTICK is close , then all of runtime and deadline will be wrong.
>>> I think what peter meant is, do you use HRTICK in products or
>>> just use it for testing/experiment?
>>>
>> Thanks for your timely comments. In fact, We use HRTICK feature in product.
>> We need microsecond level
>> precision.
> Ah, thanks. Be advised that currently HRTICK is rather expensive. The
> cost is twofold: 1) doing all the kernel side hrtimer things and 2)
> programming clock hardware.
>
> Of course, if that's what you need, you're willing to pay the price.
>
> I'll see if I can put making it less expensive slightly higher on the
> (endless) todo list.

another fold: 3) Frequent migration  :)
In fact, frequent migration lead to higher overload.  In our product  we 
design new migration solution.
The simple description is as follow:
1 Set affinity in user space program at the beginning
2 Migrate happen per 100ms
3 Free task (runtime < dl_runtime)is migrated to free cpu (rt task 
bandwidth < 65%) .
4 Busy task will run more time in a CPU by dynamic quota.
So the condition of migration depends on whether task is busy or idle or 
not instead of deadline at some point.
This could not meet EDF requirement but meet our product :)