Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3

[Evgeniy Polyakov <johnpol@2ka.mipt.ru>]

On Mon, Feb 26, 2007 at 09:35:43PM +0100, Ingo Molnar (mingo@elte.hu) wrote:
> > If kernelspace rescheduling is that fast, then please explain me why 
> > userspace one always beats kernel/userspace?
> 
> because 'user space scheduling' makes no sense? I explained my thinking 
> about that in a past mail:
> 
> -------------------------->
> One often repeated (because pretty much only) performance advantage of 
> 'light threads' is context-switch performance between user-space 
> threads. But reality is, nobody /cares/ about being able to 
> context-switch between "light user-space threads"! Why? Because there 
> are only two reasons why such a high-performance context-switch would 
> occur:
> 
>  1) there's contention between those two tasks. Wonderful: now two
>     artificial threads are running on the /same/ CPU and they are even
>     contending each other. Why not run a single context on a single CPU
>     instead and only get contended if /another/ CPU runs a conflicting
>     context?? While this makes for nice "pthread locking benchmarks",
>     it is not really useful for anything real.

Obviously there must be several real threads, each of them can reschedule 
userspace threads, because it is fast and scalable.
So, one CPU - one real thread.

>  2) there has been an IO event. The thing is, for IO events we enter the
>     kernel no matter what - and we'll do so for the next 10 years at
>     minimum. We want to abstract away the hardware, we want to do
>     reliable resource accounting, we want to share hardware resources,
>     we want to rate-limit, etc., etc. While in /theory/ you could handle
>     IO purely from user-space, in practice you dont want to do that. And
>     if we accept the premise that we'll enter the kernel anyway, there's
>     zero performance difference between scheduling right there in the
>     kernel, or returning back to user-space to schedule there. (in fact
>     i submit that the former is faster). Or if we accept the theoretical
>     possibility of 'perfect IO hardware' that implements /all/ the
>     features that the kernel wants (in a secure and generic way, and
>     mind you, such IO hardware does not exist yet), then /at most/ the
>     performance advantage of user-space doing the scheduling is the
>     overhead of a null syscall entry. Which is a whopping 100 nsecs on
>     modern CPUs! That's roughly the latency of a /single/ DRAM access!
> ....

And here were start our discussion again from the begining :)
We entered kernel, of course, but then kernel decides to move thread
away and put another one on its place under hardware sun - so kernel
starts to move registers, change some states and so on.

While in case of userspace threads we arelady returned back to userspace
(on behalf of written above overhead) and start doing new task - move
registers, change some states and so one.

And somehow it happens that doing it in userspace (for example with
setjmp/longjmp) is faster than in kernel. I do not know why - I never
investigated that case, but that is it.

> <-----------
> 
> (see http://lwn.net/Articles/219958/)
> 
> btw., the words that follow that section are quite interesting in 
> retrospect:
> 
> | Furthermore, 'light thread' concepts can no way approach the 
> | performance of #2 state-machines: if you /know/ what the structure of 
> | your context is, and you can program it in a specialized state-machine 
> | way, there's just so many shortcuts possible that it's not even funny.
> 
> [ oops! ;-) ]
> 
> i severely under-estimated the kind of performance one can reach even 
> with pure procedural concepts. Btw., when i wrote this mail was when i 
> started thinking about "is it really true that the only way to get good 
> performance are 100% event-based servers and nonblocking designs?", and 
> started coding syslets and then threadlets.

:-)

Threads happen to be really fast, and easy to programm, but the maximum
performance still can not be achieved with them just because event
handling is faster - read one cacheline from the ring or queue, or
reschedule threads?

Anyway, what we are talking about, Ingo?
I understand your point, I also understand that you are not going to
change it (yes, that's it, but I need to confirm that I'm guilty too - I
doubtly will think that thousands of threads doing IO is a win), so we
can close the discussion at this point.

My main point in participating in it was kevent introduction - although
I created kevent AIO as a state machine, I never wanted to promote
kevent as AIO - kevent is event processing mechanism, one of its usage
models is AIO, another ones are signals, file events, timers, whatever...
You drawn a line - kevent is not needed - that is your point.
I wanted to hear definitive wordsa half a year ago, but community kept
silence. Eventually things are done.

Thanks.

> 	Ingo

-- 
	Evgeniy Polyakov