[RFC] bufmap-related wait logics (Re: Orangefs ABI documentation)

[Al Viro <viro@ZenIV.linux.org.uk>]

On Sun, Feb 07, 2016 at 03:53:31AM +0000, Al Viro wrote:

> BTW, could you try to reproduce that WARN_ON with these two patches added
> and with bufmap debugging turned on?  Both double-free and lack of rewinding
> are real; I can see scenarios where they would trigger, and I'm pretty sure
> that the latter is triggering in your reproducer.  Moreover, I'm absolutely
> sure that spurious dropping of bufmap references is happening there; what I'm
> not sure is whether it was on this double-free or on something else...

AFAICS, with bufmap we have 6 kinds of events -
	1) daemon installs a bufmap
	2) daemon shuts down
	3) wait_for_direct_io() requests a read/write slot
	4) orangefs_readdir() requests a readdir slot
	5) wait_for_direct_io() releases a slot
	6) orangefs_readdir() releases a slot
and the whole thing can be described via two counters and two waitqueues.
Rules:

Initially C1 = C2 = -1
(1)	if C1 >= 0
		sod off, we'd already installed that thing
	else
		C1 = number of read/write slots
		wake up that many of those who wait on Q1
		C2 = number of readdir slots
		wake up that many of those who wait on Q2

(2)	C1 -= number of read/write slots + 1
	C2 -= number of readdir slots + 1
	wait on Q1 for C1 == -1
	wait on Q2 for C2 == -1

(3)	if C1 <= 0
		end = now + 15 minutes
		while true
			if C1 < 0
				interruptibly wait on Q1 for (C1 > 0)
						up to min(end - now, 30s)
				if C1 < 0
					return -ETIMEDOUT
			else
				interruptibly wait on Q1 for (C1 > 0)
						up to end - now, exclusive
			if C1 > 0
				break
			if signal arrived
				return -EINTR
			if now after end
				return -ETIMEDOUT
	C1--, and grab a slot in read/write slots bitmap

(5)	release a slot in bitmap; C1++; wake up Q1

(4,6)	same as (3,5) with s/C1/C2/, s/Q1/Q2/

I'd probably use Q1.lock for serializing C1 and Q2.lock for C2; the only
obstacle is the lack of timeout versions of
	wait_event_interruptible{,_exclusive}locked()
(and obscene identifier length of such beasts, of course).

The really annoying thing is that it's very similar to a couple of counting
semaphores; home-grown wait primitive is almost always a Bad Idea(tm) and
if somebody sees a sane way to cobble that out of higher-level ones, I'd
very much prefer that.  Suggestions?