Re: [PATCH RESEND v2 1/1] percpu_rw_semaphore: reimplement to not block the readers unnecessarily

[Oleg Nesterov <oleg@redhat.com>]

On 11/09, Paul E. McKenney wrote:
>
> On Fri, Nov 09, 2012 at 04:46:56PM +0100, Oleg Nesterov wrote:
> > Contrary, I am going to try to add some complications later, so that
> > it can have more users. In particular, I think it can replace
> > get_online_cpus/cpu_hotplug_begin, just we need
> > percpu_down_write_but_dont_deadlock_with_recursive_readers().
>
> I must confess that I am a bit concerned about possible scalability
> bottlenecks in the current get_online_cpus(), so +1 from me on this one.

OK, thanks...

And btw percpu_down_write_but_dont_deadlock_with_recursive_readers() is
trivial, just it needs down_write(rw_sem) "inside" wait_event(), not
before. But I'm afraid I will never manage to write the comments ;)

	static bool xxx(brw)
	{
		down_write(&brw->rw_sem);
		if (!atomic_read(&brw->slow_read_ctr))
			return true;

		up_write(&brw->rw_sem);
		return false;
	}

	static void __percpu_down_write(struct percpu_rw_semaphore *brw, bool recursive_readers)
	{
		mutex_lock(&brw->writer_mutex);

		synchronize_sched();

		atomic_add(clear_fast_ctr(brw), &brw->slow_read_ctr);

		if (recursive_readers)  {
			wait_event(brw->write_waitq, xxx(brw));
		} else {
			down_write(&brw->rw_sem);

			wait_event(brw->write_waitq, !atomic_read(&brw->slow_read_ctr));
		}
	}

Of course, cpu.c still needs .active_writer to allow get_online_cpus()
under cpu_hotplug_begin(), but this is simple.

But first we should do other changes, I think. IMHO we should not do
synchronize_sched() under mutex_lock() and this will add (a bit) more
complications. We will see.

Oleg.