Re: A problem regarding SMP

[Philippe Gerum <rpm@xenomai.org>]

JiajunDu <wonderboy512@163.com> writes:

> Hello!
>
> Currently I encountered a problem with SMP while using linux-evl core.
>
> I created multiple kernel processes sequentially in the kernel state using
> `evl_run_kthread` and would like to have them run in parallel on multiple
> CPUs, but the parallelism I was expecting is not happening in EVL core.
>
> After reading the relevant code in the EVL kernel, I discovered the reason
> as follows. When calling `evl_run_kthread` in the master process to create
> multiple processes sequentially, the master process creates the first process
> first and then waits for the first process to finish initializing:
>
> wait_for_completion(&kthread->done);
>
> The newly created process enters the `kthread_trampoline` function and switches
> the OOB state first, then calls `irq_work_queue` to put the inband irq_work
> that wakes up the master process into the current CPU. However, this results
> in that the inband irq_work that wakes up the master process having no way to
> be handled until the first created process finishes executing, because it is
> OOB. This would also cause the creation and running of the second process being
> delayed, since these events occur in the master process. So in summary, there
> is no way to run all the processes in parallel.
>

Well, yes there is: you need all evl kthreads to synchronize on a
barrier (e.g. evl_flag) before entering their work loop. The master
process would post that barrier when evl_run_kthread() has returned for
the last kthread, unleashing them all on each CPU you placed them. Per
the property you mentioned, we know for sure that an evl kthread is
ready to run when the master process resumes execution after a call to
evl_run_kthread().

> By the way, the /kernel/evl/thread.c file also exists a comment explaining this
> problem:
>
> /*
> * We are now running OOB, therefore __evl_run_kthread() can't
> * start us before we enter the dormant state because
> * irq_work_queue() schedules the in-band wakeup request on
> * the current CPU. If we fail switching to OOB context,
> * kthread->status tells __evl_run_kthread() not to start but
> * cancel us instead.
> */
>
> Now I'm thinking that theoretically this phenomenon is correct for RTOS, since
> the main process that creates the new processes is not OOB, so it does should
> be woken up using inband ways. But this does cause a lot of wasted CPU
> resources in SMP environments, because it would have been possible to have the
> inband processes and the OOB processes running at the same time on different
> CPUs.
>
> So, here I would like to ask, should this problem to be solved? And is there
> anyone in the community working on this?
>

I don't think there is any problem in this case. You only need to
account for the inband/oob duality of any evl (k)thread. Inband is where
the basic OS resources are allocated, oob is where scheduling happens
with real-time guarantee.

-- 
Philippe.