Process scheduling

Process scheduling

[Nitin Varyani]

Hello,
         I want to understand the flow of code of process scheduler of
linux kernel. What I have understood is that
The task marks itself as sleeping,
puts itself on a wait queue,
removes itself from the red-black tree of runnable, and
calls schedule() to select a new process to execute.

for Waking back up
The task is set as runnable,
removed from the wait queue,
and added back to the red-black tree.

Can I get the details of which function does what? in sched/core.c and in
sched/fair.c
I am concerned only with fair scheduler. There are so many functions in
these two files that I am totally confused.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160213/6f9cccab/attachment.html 

Process scheduling

[Henrik Austad]

On Sat, Feb 13, 2016 at 11:42:57AM +0530, Nitin Varyani wrote:
> Hello,

Hi Nitin,

>          I want to understand the flow of code of process scheduler of
> linux kernel. What I have understood is that
> The task marks itself as sleeping,
> puts itself on a wait queue,
> removes itself from the red-black tree of runnable, and
> calls schedule() to select a new process to execute.
> 
> for Waking back up
> The task is set as runnable,
> removed from the wait queue,
> and added back to the red-black tree.
> 
> Can I get the details of which function does what? in sched/core.c and in
> sched/fair.c
> I am concerned only with fair scheduler. There are so many functions in
> these two files that I am totally confused.

Then core.c and fair.c is the best bet.

You could also pick up a copy of Linux kernel development (By Love), it 
gives a nice introduction to the overall flow of .. well mostly everything. 
:)

In kernel/sched/sched.h you have a struct called 'struct sched_class" which 
is a set of function-points. This is used by the core machinery to call 
into scheduling-class specific code. At the bottom of fair.c, you see said 
struct being populated.

Also, if you want to see what really happens, try enabling 
function-tracing, but limit it to sched-functions only (and sched-events, 
those are also useful to see what triggers things)

mount -t debugfs nodev /sys/kernel/debug
cd /sys/kernel/debug/tracing
echo 0 > tracing_on	
echo function > current_tracer
echo "sched*" > set_ftrace_filter
echo 1 > events/sched/enable
echo 1 > tracing_on
... wait for a few secs
echo 0 > tracing_on

cat trace > /tmp/trace.txt

Now, look at trace.txt and correlate it to the scheduler code :)

Good luck!

-- 
Henrik Austad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160213/5c031fa3/attachment.bin 

Process scheduling

[Nitin Varyani]

thanks

On Sat, Feb 13, 2016 at 2:19 PM, Henrik Austad <henrik@austad.us> wrote:

> On Sat, Feb 13, 2016 at 11:42:57AM +0530, Nitin Varyani wrote:
> > Hello,
>
> Hi Nitin,
>
> >          I want to understand the flow of code of process scheduler of
> > linux kernel. What I have understood is that
> > The task marks itself as sleeping,
> > puts itself on a wait queue,
> > removes itself from the red-black tree of runnable, and
> > calls schedule() to select a new process to execute.
> >
> > for Waking back up
> > The task is set as runnable,
> > removed from the wait queue,
> > and added back to the red-black tree.
> >
> > Can I get the details of which function does what? in sched/core.c and in
> > sched/fair.c
> > I am concerned only with fair scheduler. There are so many functions in
> > these two files that I am totally confused.
>
> Then core.c and fair.c is the best bet.
>
> You could also pick up a copy of Linux kernel development (By Love), it
> gives a nice introduction to the overall flow of .. well mostly everything.
> :)
>
> In kernel/sched/sched.h you have a struct called 'struct sched_class" which
> is a set of function-points. This is used by the core machinery to call
> into scheduling-class specific code. At the bottom of fair.c, you see said
> struct being populated.
>
> Also, if you want to see what really happens, try enabling
> function-tracing, but limit it to sched-functions only (and sched-events,
> those are also useful to see what triggers things)
>
> mount -t debugfs nodev /sys/kernel/debug
> cd /sys/kernel/debug/tracing
> echo 0 > tracing_on
> echo function > current_tracer
> echo "sched*" > set_ftrace_filter
> echo 1 > events/sched/enable
> echo 1 > tracing_on
> ... wait for a few secs
> echo 0 > tracing_on
>
> cat trace > /tmp/trace.txt
>
> Now, look at trace.txt and correlate it to the scheduler code :)
>
> Good luck!
>
> --
> Henrik Austad
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160213/93eb460d/attachment.html 

Process scheduling

[Nitin Varyani]

On Mon, Feb 15, 2016 at 6:06 PM, Nitin Varyani <varyani.nitin1@gmail.com>
wrote:

> Hi
>         I have studied LInux kernel CFS scheduling algorithm - the
> vruntime, weights, nice value, etc. I am able to understand the code.
>  Actually the task given to me is really very huge. I am told to design a
> distributed process scheduling algorithm. A very simple implementation of
> it will be sufficient for me. Current distributed OS are patch work over
> the linux kernels, that is, they are responsible for load balancing through
> process migration but the scheduling is taken care by the single machine
> linux kernels. My task is to make the scheduling algorithm itself as
> distributed. That is a scheduler makes a decision whether to migrate a task
> or to keep the task in the current system.  I need some design aspects of
> how to achieve it. Another thing which I want to know is that whether this
> job is possible for a kernel newbie like me.
>
> On Sat, Feb 13, 2016 at 3:12 PM, Nitin Varyani <varyani.nitin1@gmail.com>
> wrote:
>
>> thanks
>>
>> On Sat, Feb 13, 2016 at 2:19 PM, Henrik Austad <henrik@austad.us> wrote:
>>
>>> On Sat, Feb 13, 2016 at 11:42:57AM +0530, Nitin Varyani wrote:
>>> > Hello,
>>>
>>> Hi Nitin,
>>>
>>> >          I want to understand the flow of code of process scheduler of
>>> > linux kernel. What I have understood is that
>>> > The task marks itself as sleeping,
>>> > puts itself on a wait queue,
>>> > removes itself from the red-black tree of runnable, and
>>> > calls schedule() to select a new process to execute.
>>> >
>>> > for Waking back up
>>> > The task is set as runnable,
>>> > removed from the wait queue,
>>> > and added back to the red-black tree.
>>> >
>>> > Can I get the details of which function does what? in sched/core.c and
>>> in
>>> > sched/fair.c
>>> > I am concerned only with fair scheduler. There are so many functions in
>>> > these two files that I am totally confused.
>>>
>>> Then core.c and fair.c is the best bet.
>>>
>>> You could also pick up a copy of Linux kernel development (By Love), it
>>> gives a nice introduction to the overall flow of .. well mostly
>>> everything.
>>> :)
>>>
>>> In kernel/sched/sched.h you have a struct called 'struct sched_class"
>>> which
>>> is a set of function-points. This is used by the core machinery to call
>>> into scheduling-class specific code. At the bottom of fair.c, you see
>>> said
>>> struct being populated.
>>>
>>> Also, if you want to see what really happens, try enabling
>>> function-tracing, but limit it to sched-functions only (and sched-events,
>>> those are also useful to see what triggers things)
>>>
>>> mount -t debugfs nodev /sys/kernel/debug
>>> cd /sys/kernel/debug/tracing
>>> echo 0 > tracing_on
>>> echo function > current_tracer
>>> echo "sched*" > set_ftrace_filter
>>> echo 1 > events/sched/enable
>>> echo 1 > tracing_on
>>> ... wait for a few secs
>>> echo 0 > tracing_on
>>>
>>> cat trace > /tmp/trace.txt
>>>
>>> Now, look at trace.txt and correlate it to the scheduler code :)
>>>
>>> Good luck!
>>>
>>> --
>>> Henrik Austad
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160215/d8031e45/attachment-0001.html 

Process scheduling

[Mulyadi Santosa]

On Mon, Feb 15, 2016 at 7:37 PM, Nitin Varyani <varyani.nitin1@gmail.com>
wrote:

>
>
> On Mon, Feb 15, 2016 at 6:06 PM, Nitin Varyani <varyani.nitin1@gmail.com>
> wrote:
>
>> Hi
>>         I have studied LInux kernel CFS scheduling algorithm - the
>> vruntime, weights, nice value, etc. I am able to understand the code.
>>  Actually the task given to me is really very huge. I am told to design a
>> distributed process scheduling algorithm. A very simple implementation of
>> it will be sufficient for me. Current distributed OS are patch work over
>> the linux kernels, that is, they are responsible for load balancing through
>> process migration but the scheduling is taken care by the single machine
>> linux kernels. My task is to make the scheduling algorithm itself as
>> distributed. That is a scheduler makes a decision whether to migrate a task
>> or to keep the task in the current system.  I need some design aspects of
>> how to achieve it. Another thing which I want to know is that whether this
>> job is possible for a kernel newbie like me.
>>
>> On Sat, Feb 13, 2016 at 3:12 PM, Nitin Varyani <varyani.nitin1@gmail.com>
>> wrote:
>>
>>> thanks
>>>
>>> On Sat, Feb 13, 2016 at 2:19 PM, Henrik Austad <henrik@austad.us> wrote:
>>>
>>>> On Sat, Feb 13, 2016 at 11:42:57AM +0530, Nitin Varyani wrote:
>>>> > Hello,
>>>>
>>>> Hi Nitin,
>>>>
>>>> >          I want to understand the flow of code of process scheduler of
>>>> > linux kernel. What I have understood is that
>>>> > The task marks itself as sleeping,
>>>> > puts itself on a wait queue,
>>>> > removes itself from the red-black tree of runnable, and
>>>> > calls schedule() to select a new process to execute.
>>>> >
>>>> > for Waking back up
>>>> > The task is set as runnable,
>>>> > removed from the wait queue,
>>>> > and added back to the red-black tree.
>>>> >
>>>> > Can I get the details of which function does what? in sched/core.c
>>>> and in
>>>> > sched/fair.c
>>>> > I am concerned only with fair scheduler. There are so many functions
>>>> in
>>>> > these two files that I am totally confused.
>>>>
>>>> Then core.c and fair.c is the best bet.
>>>>
>>>> You could also pick up a copy of Linux kernel development (By Love), it
>>>> gives a nice introduction to the overall flow of .. well mostly
>>>> everything.
>>>> :)
>>>>
>>>> In kernel/sched/sched.h you have a struct called 'struct sched_class"
>>>> which
>>>> is a set of function-points. This is used by the core machinery to call
>>>> into scheduling-class specific code. At the bottom of fair.c, you see
>>>> said
>>>> struct being populated.
>>>>
>>>> Also, if you want to see what really happens, try enabling
>>>> function-tracing, but limit it to sched-functions only (and
>>>> sched-events,
>>>> those are also useful to see what triggers things)
>>>>
>>>> mount -t debugfs nodev /sys/kernel/debug
>>>> cd /sys/kernel/debug/tracing
>>>> echo 0 > tracing_on
>>>> echo function > current_tracer
>>>> echo "sched*" > set_ftrace_filter
>>>> echo 1 > events/sched/enable
>>>> echo 1 > tracing_on
>>>> ... wait for a few secs
>>>> echo 0 > tracing_on
>>>>
>>>> cat trace > /tmp/trace.txt
>>>>
>>>> Now, look at trace.txt and correlate it to the scheduler code :)
>>>>
>>>> Good luck!
>>>>
>>>> --
>>>> Henrik Austad
>>>>
>>>
>>>
>>
>
>

Please don't top post :) Use bottom post .

Sounds like what you're going to do is highly similar to openMosix. check
their source code.

Please note that openmosix is patch against 2.4.x linux kernel. When
they're about to made it compatible to 2.6.x, the project stalls. See Linux
IPMI project and see if you can help them out

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

This email has been sent from a virus-free computer protected by Avast.
www.avast.com <https://www.avast.com/sig-email>
<#DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160217/b3e165b8/attachment.html 

Process scheduling

[Nitin Varyani]

It is similar to openMosix but still quite different. Open Mosix is built
on the top of existing linux kernels. The scheduling is taken care by the
existing linux kernels. Open Mosix is responsible for workload
distribution. This project is first of its kind.

On Wed, Feb 17, 2016 at 11:40 AM, Mulyadi Santosa <mulyadi.santosa@gmail.com
> wrote:

>
>
> On Mon, Feb 15, 2016 at 7:37 PM, Nitin Varyani <varyani.nitin1@gmail.com>
> wrote:
>
>>
>>
>> On Mon, Feb 15, 2016 at 6:06 PM, Nitin Varyani <varyani.nitin1@gmail.com>
>> wrote:
>>
>>> Hi
>>>         I have studied LInux kernel CFS scheduling algorithm - the
>>> vruntime, weights, nice value, etc. I am able to understand the code.
>>>  Actually the task given to me is really very huge. I am told to design
>>> a distributed process scheduling algorithm. A very simple implementation of
>>> it will be sufficient for me. Current distributed OS are patch work over
>>> the linux kernels, that is, they are responsible for load balancing through
>>> process migration but the scheduling is taken care by the single machine
>>> linux kernels. My task is to make the scheduling algorithm itself as
>>> distributed. That is a scheduler makes a decision whether to migrate a task
>>> or to keep the task in the current system.  I need some design aspects of
>>> how to achieve it. Another thing which I want to know is that whether this
>>> job is possible for a kernel newbie like me.
>>>
>>> On Sat, Feb 13, 2016 at 3:12 PM, Nitin Varyani <varyani.nitin1@gmail.com
>>> > wrote:
>>>
>>>> thanks
>>>>
>>>> On Sat, Feb 13, 2016 at 2:19 PM, Henrik Austad <henrik@austad.us>
>>>> wrote:
>>>>
>>>>> On Sat, Feb 13, 2016 at 11:42:57AM +0530, Nitin Varyani wrote:
>>>>> > Hello,
>>>>>
>>>>> Hi Nitin,
>>>>>
>>>>> >          I want to understand the flow of code of process scheduler
>>>>> of
>>>>> > linux kernel. What I have understood is that
>>>>> > The task marks itself as sleeping,
>>>>> > puts itself on a wait queue,
>>>>> > removes itself from the red-black tree of runnable, and
>>>>> > calls schedule() to select a new process to execute.
>>>>> >
>>>>> > for Waking back up
>>>>> > The task is set as runnable,
>>>>> > removed from the wait queue,
>>>>> > and added back to the red-black tree.
>>>>> >
>>>>> > Can I get the details of which function does what? in sched/core.c
>>>>> and in
>>>>> > sched/fair.c
>>>>> > I am concerned only with fair scheduler. There are so many functions
>>>>> in
>>>>> > these two files that I am totally confused.
>>>>>
>>>>> Then core.c and fair.c is the best bet.
>>>>>
>>>>> You could also pick up a copy of Linux kernel development (By Love), it
>>>>> gives a nice introduction to the overall flow of .. well mostly
>>>>> everything.
>>>>> :)
>>>>>
>>>>> In kernel/sched/sched.h you have a struct called 'struct sched_class"
>>>>> which
>>>>> is a set of function-points. This is used by the core machinery to call
>>>>> into scheduling-class specific code. At the bottom of fair.c, you see
>>>>> said
>>>>> struct being populated.
>>>>>
>>>>> Also, if you want to see what really happens, try enabling
>>>>> function-tracing, but limit it to sched-functions only (and
>>>>> sched-events,
>>>>> those are also useful to see what triggers things)
>>>>>
>>>>> mount -t debugfs nodev /sys/kernel/debug
>>>>> cd /sys/kernel/debug/tracing
>>>>> echo 0 > tracing_on
>>>>> echo function > current_tracer
>>>>> echo "sched*" > set_ftrace_filter
>>>>> echo 1 > events/sched/enable
>>>>> echo 1 > tracing_on
>>>>> ... wait for a few secs
>>>>> echo 0 > tracing_on
>>>>>
>>>>> cat trace > /tmp/trace.txt
>>>>>
>>>>> Now, look at trace.txt and correlate it to the scheduler code :)
>>>>>
>>>>> Good luck!
>>>>>
>>>>> --
>>>>> Henrik Austad
>>>>>
>>>>
>>>>
>>>
>>
>>
>
> Please don't top post :) Use bottom post .
>
> Sounds like what you're going to do is highly similar to openMosix. check
> their source code.
>
> Please note that openmosix is patch against 2.4.x linux kernel. When
> they're about to made it compatible to 2.6.x, the project stalls. See Linux
> IPMI project and see if you can help them out
>
> --
> regards,
>
> Mulyadi Santosa
> Freelance Linux trainer and consultant
>
> blog: the-hydra.blogspot.com
> training: mulyaditraining.blogspot.com
>
> This email has been sent from a virus-free computer protected by Avast.
> www.avast.com <https://www.avast.com/sig-email>
> <#1657133857_DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160217/3c7bb1c7/attachment.html 

Process scheduling

[Greg KH]

On Wed, Feb 17, 2016 at 12:48:33PM +0530, Nitin Varyani wrote:
> It is similar to openMosix but still quite different. Open Mosix is built on
> the top of existing linux kernels. The scheduling is taken care by the existing
> linux kernels. Open Mosix is responsible for workload distribution. This
> project is first of its kind.

No, it's not, it's been tried before, I suggest doing a bit more
research before making statements like that.

good luck,

greg k-h

Process scheduling

[Boyan Vladinov]

Check here:

https://www.kernel.org/doc/Documentation/scheduler/sched-design-CFS.txt
and
http://www.ibm.com/developerworks/library/l-completely-fair-scheduler/

Regards,
Boyan Vladinov

On 17.02.2016 18:23, Greg KH wrote:
> On Wed, Feb 17, 2016 at 12:48:33PM +0530, Nitin Varyani wrote:
>> It is similar to openMosix but still quite different. Open Mosix is built on
>> the top of existing linux kernels. The scheduling is taken care by the existing
>> linux kernels. Open Mosix is responsible for workload distribution. This
>> project is first of its kind.
> No, it's not, it's been tried before, I suggest doing a bit more
> research before making statements like that.
>
> good luck,
>
> greg k-h
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Process scheduling

[Boyan Vladinov]

Check here:

https://www.kernel.org/doc/Documentation/scheduler/sched-design-CFS.txt
and
http://www.ibm.com/developerworks/library/l-completely-fair-scheduler/

Regards,
Boyan Vladinov

On 17.02.2016 18:23, Greg KH wrote:
> On Wed, Feb 17, 2016 at 12:48:33PM +0530, Nitin Varyani wrote:
>> It is similar to openMosix but still quite different. Open Mosix is built on
>> the top of existing linux kernels. The scheduling is taken care by the existing
>> linux kernels. Open Mosix is responsible for workload distribution. This
>> project is first of its kind.
> No, it's not, it's been tried before, I suggest doing a bit more
> research before making statements like that.
>
> good luck,
>
> greg k-h
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Process Scheduling

[Nitin Varyani]

Hi,
      I am new to kernel source. I want to plugin a new process scheduling
algorithm. Can someone elaborate the steps to do it?
Nitin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20160208/7d7fdd93/attachment-0001.html