Re: A problem regarding SMP

Re: A problem regarding SMP

[JiajunDu]

Thanks for your reply!

By using a barrier to synchronize all evl kthreads, I did achieve the
effect of multiple processes running in parallel. My solution is
in the next patch, and the branch I tried on is the latest
`v5.13-evl-rebase`.

But I still ran into two other problems while trying this solution.

The first problem is that `evl_flag` cann't solve the problem. The wait
condition in `evl_wait_flag_timeout` is `evl_read_flag`, in which the
flag is set to false as soon as the flag is found to be raised in
`evl_raise_flag`, so `evl_raise_flag` can only wake up one waiting process
at a time. And it's not possible to wake up more processes by calling
`evl_raise_flag` multiple times, because it's not possible to determine
the time when processes are woken up in the SMP environment. If we did, it's
possible to have a situation where two processes are woken up at the same
time after the flag was raised twice, and then only one of them can be
woken up at the same time. So in light of this, I defined two new functions,
`evl_wait_flag_same` and `evl_wait_flag_timeout_same`, to read flags using
`evl_peek_flag`.

The second problem is that after using barrier, load balancing is no longer
possible when creating processes using `kthread_run`, which is called in
`evl_run_kthread`. The phenomenon is that all processes are created on the
same CPU. Therefore, I can only specify CPUs manually for load balancing
purpose by calling `evl_run_kthread_on_cpu`.

The code I tried is available in the next patch. Is there anything we can do
about these two problems?

Best regards,
Jiajun

Re: A problem regarding SMP

[Philippe Gerum]

JiajunDu <wonderboy512@163.com> writes:

> Thanks for your reply!
>
> By using a barrier to synchronize all evl kthreads, I did achieve the
> effect of multiple processes running in parallel. My solution is
> in the next patch, and the branch I tried on is the latest
> `v5.13-evl-rebase`.
>
> But I still ran into two other problems while trying this solution.
>
> The first problem is that `evl_flag` cann't solve the problem. The wait
> condition in `evl_wait_flag_timeout` is `evl_read_flag`, in which the
> flag is set to false as soon as the flag is found to be raised in
> `evl_raise_flag`, so `evl_raise_flag` can only wake up one waiting process
> at a time. And it's not possible to wake up more processes by calling
> `evl_raise_flag` multiple times, because it's not possible to determine
> the time when processes are woken up in the SMP environment. If we did, it's
> possible to have a situation where two processes are woken up at the same
> time after the flag was raised twice, and then only one of them can be
> woken up at the same time. So in light of this, I defined two new functions,
> `evl_wait_flag_same` and `evl_wait_flag_timeout_same`, to read flags using
> `evl_peek_flag`.

The semantics you are looking for are available from evl_wait_queue,
which basically implements a condition variable.

>
> The second problem is that after using barrier, load balancing is no longer
> possible when creating processes using `kthread_run`, which is called in
> `evl_run_kthread`. The phenomenon is that all processes are created on the
> same CPU. Therefore, I can only specify CPUs manually for load balancing
> purpose by calling `evl_run_kthread_on_cpu`.

Which is not an issue, is it? Since dynamic load balancing is not
supported by design by evl (or any earlier xenomai cores for that
matter), static pinning is the way. So evl_run_kthread_on_cpu should
indeed be used. Another option would be that each spawned kthread pins
itself to the right CPU when emerging (set_cpus_allowed).

>
> The code I tried is available in the next patch. Is there anything we can do
> about these two problems?
>
> Best regards,
> Jiajun


-- 
Philippe.

Re: A problem regarding SMP

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> JiajunDu <wonderboy512@163.com> writes:
>
>> Thanks for your reply!
>>
>> By using a barrier to synchronize all evl kthreads, I did achieve the
>> effect of multiple processes running in parallel. My solution is
>> in the next patch, and the branch I tried on is the latest
>> `v5.13-evl-rebase`.
>>
>> But I still ran into two other problems while trying this solution.
>>
>> The first problem is that `evl_flag` cann't solve the problem. The wait
>> condition in `evl_wait_flag_timeout` is `evl_read_flag`, in which the
>> flag is set to false as soon as the flag is found to be raised in
>> `evl_raise_flag`, so `evl_raise_flag` can only wake up one waiting process
>> at a time. And it's not possible to wake up more processes by calling
>> `evl_raise_flag` multiple times, because it's not possible to determine
>> the time when processes are woken up in the SMP environment. If we did, it's
>> possible to have a situation where two processes are woken up at the same
>> time after the flag was raised twice, and then only one of them can be
>> woken up at the same time. So in light of this, I defined two new functions,
>> `evl_wait_flag_same` and `evl_wait_flag_timeout_same`, to read flags using
>> `evl_peek_flag`.
>
> The semantics you are looking for are available from evl_wait_queue,
> which basically implements a condition variable.

This said, I agree that the current implementation of evl_wait_flag()
prevents broadcast, which is wrong.

-- 
Philippe.

Re: A problem regarding SMP

[JiajunDu]

Ok, thanks for your reply. Now I understand.

Best regards,
Jiajun

Re: A problem regarding SMP

[Philippe Gerum]

JiajunDu <wonderboy512@163.com> writes:

> Ok, thanks for your reply. Now I understand.
>
> Best regards,
> Jiajun

You do have another option beyond the flag which is restricted to
unicast for the time being, which is using an evl (kernel) semaphore,
depleted at init:

evl_init_ksem(&ksem, 0);
...
evl_down(&ksem); /* kthread */
...
evl_broadcast(&ksem); /* master (spawner) */


-- 
Philippe.

A problem regarding SMP

[JiajunDu]

Hello!

Currently I encountered a problem with SMP while using linux-evl core.

I created multiple kernel processes sequentially in the kernel state using
`evl_run_kthread` and would like to have them run in parallel on multiple
CPUs, but the parallelism I was expecting is not happening in EVL core.

After reading the relevant code in the EVL kernel, I discovered the reason
as follows. When calling `evl_run_kthread` in the master process to create
multiple processes sequentially, the master process creates the first process
first and then waits for the first process to finish initializing:

wait_for_completion(&kthread->done);

The newly created process enters the `kthread_trampoline` function and switches
the OOB state first, then calls `irq_work_queue` to put the inband irq_work
that wakes up the master process into the current CPU. However, this results
in that the inband irq_work that wakes up the master process having no way to
be handled until the first created process finishes executing, because it is
OOB. This would also cause the creation and running of the second process being
delayed, since these events occur in the master process. So in summary, there
is no way to run all the processes in parallel.

By the way, the /kernel/evl/thread.c file also exists a comment explaining this
problem:

/*
* We are now running OOB, therefore __evl_run_kthread() can't
* start us before we enter the dormant state because
* irq_work_queue() schedules the in-band wakeup request on
* the current CPU. If we fail switching to OOB context,
* kthread->status tells __evl_run_kthread() not to start but
* cancel us instead.
*/

Now I'm thinking that theoretically this phenomenon is correct for RTOS, since
the main process that creates the new processes is not OOB, so it does should
be woken up using inband ways. But this does cause a lot of wasted CPU
resources in SMP environments, because it would have been possible to have the
inband processes and the OOB processes running at the same time on different
CPUs.

So, here I would like to ask, should this problem to be solved? And is there
anyone in the community working on this?

Best regards,
Jiajun

Re: A problem regarding SMP

[Philippe Gerum]

JiajunDu <wonderboy512@163.com> writes:

> Hello!
>
> Currently I encountered a problem with SMP while using linux-evl core.
>
> I created multiple kernel processes sequentially in the kernel state using
> `evl_run_kthread` and would like to have them run in parallel on multiple
> CPUs, but the parallelism I was expecting is not happening in EVL core.
>
> After reading the relevant code in the EVL kernel, I discovered the reason
> as follows. When calling `evl_run_kthread` in the master process to create
> multiple processes sequentially, the master process creates the first process
> first and then waits for the first process to finish initializing:
>
> wait_for_completion(&kthread->done);
>
> The newly created process enters the `kthread_trampoline` function and switches
> the OOB state first, then calls `irq_work_queue` to put the inband irq_work
> that wakes up the master process into the current CPU. However, this results
> in that the inband irq_work that wakes up the master process having no way to
> be handled until the first created process finishes executing, because it is
> OOB. This would also cause the creation and running of the second process being
> delayed, since these events occur in the master process. So in summary, there
> is no way to run all the processes in parallel.
>

Well, yes there is: you need all evl kthreads to synchronize on a
barrier (e.g. evl_flag) before entering their work loop. The master
process would post that barrier when evl_run_kthread() has returned for
the last kthread, unleashing them all on each CPU you placed them. Per
the property you mentioned, we know for sure that an evl kthread is
ready to run when the master process resumes execution after a call to
evl_run_kthread().

> By the way, the /kernel/evl/thread.c file also exists a comment explaining this
> problem:
>
> /*
> * We are now running OOB, therefore __evl_run_kthread() can't
> * start us before we enter the dormant state because
> * irq_work_queue() schedules the in-band wakeup request on
> * the current CPU. If we fail switching to OOB context,
> * kthread->status tells __evl_run_kthread() not to start but
> * cancel us instead.
> */
>
> Now I'm thinking that theoretically this phenomenon is correct for RTOS, since
> the main process that creates the new processes is not OOB, so it does should
> be woken up using inband ways. But this does cause a lot of wasted CPU
> resources in SMP environments, because it would have been possible to have the
> inband processes and the OOB processes running at the same time on different
> CPUs.
>
> So, here I would like to ask, should this problem to be solved? And is there
> anyone in the community working on this?
>

I don't think there is any problem in this case. You only need to
account for the inband/oob duality of any evl (k)thread. Inband is where
the basic OS resources are allocated, oob is where scheduling happens
with real-time guarantee.

-- 
Philippe.