pynfs clean_init issue

pynfs clean_init issue

[Chuck Lever III]

Hi Bruce-

I'm running DELEG21 to unit-test delegations, and this message comes
out at the end:

Making sure b'DELEG21-1' is writable: operation OP_SETATTR should return NFS4_OK, instead got NFS4ERR_DELAY

I guess there's no callback service running during the test's clean-up phase.

Then if I run the test again immediately:

[cel@morisot pynfs]$ sudo nfs4.0/testserver.py manet:/export/tmp --maketree --rundeps -v cel
Initialization failed, no tests run.
Perhaps you need to use the --secure option or configure server to allow connections from high ports
Traceback (most recent call last):
  File "/home/cel/src/pynfs/nfs4.0/testserver.py", line 394, in <module>
    main()
  File "/home/cel/src/pynfs/nfs4.0/testserver.py", line 346, in main
    env.init()
  File "/home/cel/src/pynfs/nfs4.0/servertests/environment.py", line 150, in init
    c.clean_dir(self.opts.path)
  File "/home/cel/src/pynfs/nfs4.0/nfs4lib.py", line 579, in clean_dir
    check_result(res, "Making sure %s is writable" % repr(e.name))
  File "/home/cel/src/pynfs/nfs4.0/nfs4lib.py", line 906, in check_result
    raise BadCompoundRes(resop, res.status, msg)
nfs4lib.BadCompoundRes: Making sure b'DELEG21-1' is writable: operation OP_SETATTR should return NFS4_OK, instead got NFS4ERR_DELAY

And I think this condition persists until the old lease expires and
the server permits the client to delete that file.


--
Chuck Lever

Re: pynfs clean_init issue

[J. Bruce Fields]

On Thu, Aug 04, 2022 at 03:23:44PM +0000, Chuck Lever III wrote:
> Hi Bruce-
> 
> I'm running DELEG21 to unit-test delegations, and this message comes
> out at the end:
> 
> Making sure b'DELEG21-1' is writable: operation OP_SETATTR should return NFS4_OK, instead got NFS4ERR_DELAY
> 
> I guess there's no callback service running during the test's clean-up phase.
> 
> Then if I run the test again immediately:
> 
> [cel@morisot pynfs]$ sudo nfs4.0/testserver.py manet:/export/tmp --maketree --rundeps -v cel
> Initialization failed, no tests run.
> Perhaps you need to use the --secure option or configure server to allow connections from high ports
> Traceback (most recent call last):
>   File "/home/cel/src/pynfs/nfs4.0/testserver.py", line 394, in <module>
>     main()
>   File "/home/cel/src/pynfs/nfs4.0/testserver.py", line 346, in main
>     env.init()
>   File "/home/cel/src/pynfs/nfs4.0/servertests/environment.py", line 150, in init
>     c.clean_dir(self.opts.path)
>   File "/home/cel/src/pynfs/nfs4.0/nfs4lib.py", line 579, in clean_dir
>     check_result(res, "Making sure %s is writable" % repr(e.name))
>   File "/home/cel/src/pynfs/nfs4.0/nfs4lib.py", line 906, in check_result
>     raise BadCompoundRes(resop, res.status, msg)
> nfs4lib.BadCompoundRes: Making sure b'DELEG21-1' is writable: operation OP_SETATTR should return NFS4_OK, instead got NFS4ERR_DELAY
> 
> And I think this condition persists until the old lease expires and
> the server permits the client to delete that file.

DELEG21 should pass on any recent kernel.

But possibly cleanup should also be better.  I'm not sure what the right
fix is.

--b.

Re: pynfs clean_init issue

[Chuck Lever III]

> On Aug 4, 2022, at 11:38 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> 
> On Thu, Aug 04, 2022 at 03:23:44PM +0000, Chuck Lever III wrote:
>> Hi Bruce-
>> 
>> I'm running DELEG21 to unit-test delegations, and this message comes
>> out at the end:
>> 
>> Making sure b'DELEG21-1' is writable: operation OP_SETATTR should return NFS4_OK, instead got NFS4ERR_DELAY
>> 
>> I guess there's no callback service running during the test's clean-up phase.
>> 
>> Then if I run the test again immediately:
>> 
>> [cel@morisot pynfs]$ sudo nfs4.0/testserver.py manet:/export/tmp --maketree --rundeps -v cel
>> Initialization failed, no tests run.
>> Perhaps you need to use the --secure option or configure server to allow connections from high ports
>> Traceback (most recent call last):
>>  File "/home/cel/src/pynfs/nfs4.0/testserver.py", line 394, in <module>
>>    main()
>>  File "/home/cel/src/pynfs/nfs4.0/testserver.py", line 346, in main
>>    env.init()
>>  File "/home/cel/src/pynfs/nfs4.0/servertests/environment.py", line 150, in init
>>    c.clean_dir(self.opts.path)
>>  File "/home/cel/src/pynfs/nfs4.0/nfs4lib.py", line 579, in clean_dir
>>    check_result(res, "Making sure %s is writable" % repr(e.name))
>>  File "/home/cel/src/pynfs/nfs4.0/nfs4lib.py", line 906, in check_result
>>    raise BadCompoundRes(resop, res.status, msg)
>> nfs4lib.BadCompoundRes: Making sure b'DELEG21-1' is writable: operation OP_SETATTR should return NFS4_OK, instead got NFS4ERR_DELAY
>> 
>> And I think this condition persists until the old lease expires and
>> the server permits the client to delete that file.
> 
> DELEG21 should pass on any recent kernel.

It passes, but leaves the test file so that clean_dir does not work
again until the old lease expires.


> But possibly cleanup should also be better.

This bug might prevent running these tests in an automation harness.
I'd say cleanup does need to be better about this.


> I'm not sure what the right fix is.

Brute force: keep trying to delete that file if clean_dir receives
NFS4ERR_DELAY?

init_connection somewhere needs to set up a callback service and
leave it running.


--
Chuck Lever

Re: pynfs clean_init issue

[J. Bruce Fields]

On Thu, Aug 04, 2022 at 03:43:56PM +0000, Chuck Lever III wrote:
> It passes, but leaves the test file so that clean_dir does not work
> again until the old lease expires.

Oh, right.

> > But possibly cleanup should also be better.
> 
> This bug might prevent running these tests in an automation harness.
> I'd say cleanup does need to be better about this.
> 
> > I'm not sure what the right fix is.
> 
> Brute force: keep trying to delete that file if clean_dir receives
> NFS4ERR_DELAY?

Delegations block unlinks too so that probably doesn't help.

> init_connection somewhere needs to set up a callback service and
> leave it running.

The callback isn't too important, I think, if we want to return the
delegation at the end of the test we can do that without waiting for the
server to remind us.

Or maybe destroy the client at the end.  We have no DESTROY_CLIENTID
(this is 4.0), but we could do a client-rebooting SETCLIENTID/CONFIRM to
wipe out its state.

There may be limits to the kind of cleanup pynfs can do.  A test harness
should probably reboot the server between test runs.

--b.

Re: pynfs clean_init issue

[Chuck Lever III]

> On Aug 4, 2022, at 2:14 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> 
> On Thu, Aug 04, 2022 at 03:43:56PM +0000, Chuck Lever III wrote:
>> It passes, but leaves the test file so that clean_dir does not work
>> again until the old lease expires.
> 
> Oh, right.
> 
>>> But possibly cleanup should also be better.
>> 
>> This bug might prevent running these tests in an automation harness.
>> I'd say cleanup does need to be better about this.
>> 
>>> I'm not sure what the right fix is.
>> 
>> Brute force: keep trying to delete that file if clean_dir receives
>> NFS4ERR_DELAY?
> 
> Delegations block unlinks too so that probably doesn't help.

Just keep trying until the lease expires. But that's a naive
approach.


>> init_connection somewhere needs to set up a callback service and
>> leave it running.
> 
> The callback isn't too important, I think, if we want to return the
> delegation at the end of the test we can do that without waiting for the
> server to remind us.

That would help a lot, I think, and would not require an active
callback service on the client.

Btw, DELEG5 has the same problem... and btw, it's final OPEN fails
with EACCES... I'm looking into that too.


> Or maybe destroy the client at the end.  We have no DESTROY_CLIENTID
> (this is 4.0), but we could do a client-rebooting SETCLIENTID/CONFIRM to
> wipe out its state.

--
Chuck Lever