pnfs code suggestions

[Fred Isaman <iisaman@netapp.com>]

We are starting to think about wave 2 of the code submission, which
will be LAYOUTRETURN and callback handling.  Current code does not
correctly wait for outstanding LAYOUTGETS on CB_RECALL.  Remembering
that this is for whole file layouts with the forgetful model, my
intent is to work on patches implementing the following.  Let me know
if anyone sees problems with this approach.

On CB_RECALL, set a "barrier" equal to one less than the seqid
embedded in the cb_recall stateid.  Then, instead of waiting on any
outstanding LAYOUTGETs whose seqid is less than the barrier, as
required by the spec, we just drop any LAYOUTGET reply that has a
seqid less than the barrier.  Because we are using the forgetful
model, this is functionally equivalent to waiting, and simpler.
We similarly increase the barrier based on replies from LAYOUTRETURNs.
The major issue I see with this approach is that we have to be very
careful on reset of the stateid to either drain or detect outstanding
LAYOUTGETS.

We will still need to wait on outstanding IO within any existing
layout.  I suggest we use Benny's rpc waitq code, shifting it up into
the submission branch.

Thanks,
Fred