From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jamie Lokier Subject: Re: [RFC] sched: implement the exclusive wait queue as a LIFO queue Date: Wed, 28 Apr 2010 16:25:02 +0100 Message-ID: <20100428152502.GA25569@shareable.org> References: <1272430986-20436-1-git-send-email-xiaosuo@gmail.com> <20100428081545.GA19027@windriver.com> <8482.1272446987@redhat.com> <20100428132135.GA22268@shareable.org> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: David Howells , Yong Zhang , Xiaotian Feng , Ingo Molnar , Alexander Viro , Andrew Morton , "Eric W. Biederman" , Davide Libenzi , Roland Dreier , Stefan Richter , Peter Zijlstra , "David S. Miller" , Eric Dumazet , Christoph Lameter , Andreas Herrmann , Thomas Gleixner , Takashi Iwai , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org To: Changli Gao Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org Changli Gao wrote: > On Wed, Apr 28, 2010 at 9:21 PM, Jamie Lokier w= rote: > > Changli Gao wrote: > >> > >> fs/eventpoll.c: 1443. > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 wait.flags |=3D WQ_FLAG_EXCLUSIVE; > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __add_wait_queue(&ep->wq, &wait); > > > > The same thing about assumptions applies here. =A0The userspace pro= cess > > may be waiting for an epoll condition to get access to a resource, > > rather than being a worker thread interchangeable with others. >=20 > Oh, the lines above are the current ones. So the assumptions applies > and works here. No, because WQ_FLAG_EXCLUSIVE doesn't have your LIFO semantic at the mo= ment. Your patch changes the behaviour of epoll, though I don't know if it matters. Perhaps all programs which have multiple tasks waiting on the same epoll fd are "interchangeable worker thread" types anyway :-) > > For example, userspace might be using a pipe as a signal-safe lock,= or > > signal-safe multi-token semaphore, and epoll to wait for that pipe. > > > > WQ_FLAG_EXCLUSIVE means there is no point waking all tasks, to avoi= d a > > pointless thundering herd. =A0It doesn't mean unfairness is ok. >=20 > The users should not make any assumption about the waking up sequence= , > neither LIFO nor FIFO. Correct, but they should be able to assume non-starvation (eventual progress) for all waiters. It's one of those subtle things, possibly a unixy thing: Non-RT tasks should always make progress when the competition is just other non-RT tasks, even if the progress is slow. Starvation can spread out beyond the starved process, to cause priority inversions in other tasks that are waiting on a resource locked by the starved process. Among other things, that can cause higher priority tasks, and RT priority tasks, to block permanently. Very unpleasant. > > The LIFO idea _might_ make sense for interchangeable worker-thread > > situations - including userspace. =A0It would make sense for pipe > > waiters, socket waiters (especially accept), etc. >=20 > Yea, and my following patches are for socket waiters. Occasionally unix socketpairs are occasionally used in the above ways t= oo. I'm not against your patch, but I worry that starvation is a new semantic, and it may have a significant effect on something - either in the kernel, or in userspace which is harder to check. > > Do you have any measurements which showing the LIFO mode performing > > better than FIFO, and by how much? >=20 > I didn't do any test yet. But some work done by LSE project years ago > showed that it is better. >=20 > http://lse.sourceforge.net/io/aionotes.txt >=20 > " Also in view of > better cache utilization the wake queue mechanism is LIFO by default. > (A new exclusive LIFO wakeup option has been introduced for this purp= ose)" I suspect it's possible to combine LIFO-ish and FIFO-ish queuing to prevent starvation while getting some of the locality benefit. Something like add-LIFO and increment a small counter in the next wait entry, but never add in front of an entry whose counter has reached MAX_LIFO_WAITERS? :-) -- Jamie