Re: [patch] cpufreq: mark cpufreq_tsc() as core_initcall_sync

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Mon, Nov 20, 2006 at 08:15:14AM +0100, Jens Axboe wrote:
> On Sun, Nov 19 2006, Paul E. McKenney wrote:
> > On Sat, Nov 18, 2006 at 09:46:24PM +0300, Oleg Nesterov wrote:
> > > On 11/17, Paul E. McKenney wrote:
> > > >
> > > > Oleg, any thoughts about Jens's optimization?  He would code something
> > > > like:
> > > >
> > > > 	if (srcu_readers_active(&my_srcu))
> > > > 		synchronize_srcu();
> > > > 	else
> > > > 		smp_mb();
> > >
> > > Well, this is clearly racy, no? I am not sure, but may be we can do
> > >
> > > 	smp_mb();
> > > 	if (srcu_readers_active(&my_srcu))
> > > 		synchronize_srcu();
> > >
> > > in this case we also need to add 'smp_mb()' into srcu_read_lock() after
> > > 'atomic_inc(&sp->hardluckref)'.
> > >
> > > > However, he is doing ordered I/O requests rather than protecting data
> > > > structures.
> > >
> > > Probably this makes a difference, but I don't understand this.
> >
> > OK, one hypothesis here...
> >
> > 	The I/Os must be somehow explicitly ordered to qualify
> > 	for I/O-barrier separation.  If two independent processes
> > 	issue I/Os concurrently with a third process doing an
> > 	I/O barrier, the I/O barrier is free to separate the
> > 	two concurrent I/Os or not, on its whim.
> >
> > Jens, is the above correct?  If so, what would the two processes
> 
> That's completely correct, hence my somewhat relaxed approach with SRCU.

OK, less scary in that case.  ;-)

> > need to do in order to ensure that their I/O was considered to be
> > ordered with respect to the I/O barrier?  Here are some possibilities:
> 
> If we consider the barrier a barrier in a certain stream of requests,
> it is the responsibility of the issuer of that barrier to ensure that
> the queueing is ordered. So if two "unrelated" streams of requests with
> barriers hit __make_request() at the same time, we don't go to great
> lengths to ensure who gets there firt.

So the "preceding" requests have to have completed their I/O system
calls?  If this is the case, does this include normal (non-direct/raw)
writes and asynchronous reads?  My guess is that it would include
asynchronous I/O, but not buffered writes.

> > 1.	I/O barriers apply only to preceding and following I/Os from
> > 	the process issuing the I/O barrier.
> >
> > 2.	As for #1 above, but restricted to task rather than process.
> >
> > 3.	I/O system calls that have completed are ordered by the
> > 	barrier to precede I/O system calls that have not yet
> > 	started, but I/O system calls still in flight could legally
> > 	land on either side of the concurrently executing I/O
> > 	barrier.
> >
> > 4.	Something else entirely?
> >
> > Given some restriction like one of the above, it is entirely possible
> > that we don't even need the memory barrier...
> 
> 3 is the closest. The request queue doesn't really know the scope of the
> barrier, it has to rely on the issuer getting it right. If you have two
> competing processes issuing io and process A relies on process B issuing
> a barrier, they have to synchronize that between them. Normally that is
> not a problem, since that's how the file systems always did io before
> barriers on items that need to be on disk (it was a serialization point
> anyway, it's just a stronger one now).

So something like a user-level mutex or atomic instructions must be used
by the tasks doing the pre-barrier I/Os to announce that these I/Os have
been started in the kernel.

> That said, I think the
> 
>         smp_mb();
>         if (srcu_readers_active(sp))
>                 synchronize_srcu();
> 
> makes the most sense.

If the user-level tasks/threads/processes must explicitly synchronize,
and if the pre-barrier I/O-initation syscalls have to have completed,
then I am not sure that the smp_mb() is needed.  Seems like the queuing
mechanisms in the syscall and the user-level synchronization would have
supplied the needed memory barriers.  Or are you using some extremely
lightweight user-level synchronization?

							Thanx, Paul