OPEN_XOR_DELEGATION performance problems

OPEN_XOR_DELEGATION performance problems

[Jeff Layton]

We attempted to implement the "delstid" draft for v6.13, but have had
to drop the patches for it. After merge, we got a couple of reports of
a performance issue due to the OPEN_XOR_DELEGATION patch:

    https://lore.kernel.org/linux-nfs/202409161645.d44bced5-oliver.sang@intel.com/

Once we enable OPEN_XOR_DELEGATION support, the fsmark "App Overhead"
statistic spikes significantly. The kernel patch for this is very
simple, and doesn't seem likely to cause a performance issue on its
own. My theory is that this test is one that causes the client to
return the delegation, and since it doesn't have an open stateid, it
has to reestablish one during the test run, and that causes the app
overhead stat to spike.

Trond, Tom, Mike -- I know that the HS Anvil has support for
OPEN_XOR_DELEGATION. If you run the fsmark test against it with that
support both enabled and disabled (either on the client or server
side), do you see a similar spike in "App Overhead"?

If so, then I suspect we need to consider limiting the use of that flag
in some cases. I have no idea what heuristic we'd use to decide this
though.
-- 
Jeff Layton <jlayton@kernel.org>

Re: OPEN_XOR_DELEGATION performance problems

[Trond Myklebust]

On Tue, 2024-11-19 at 06:45 -0500, Jeff Layton wrote:
> We attempted to implement the "delstid" draft for v6.13, but have had
> to drop the patches for it. After merge, we got a couple of reports
> of
> a performance issue due to the OPEN_XOR_DELEGATION patch:
>
>
> https://lore.kernel.org/linux-nfs/202409161645.d44bced5-oliver.sang@intel.com/
>
> Once we enable OPEN_XOR_DELEGATION support, the fsmark "App Overhead"
> statistic spikes significantly. The kernel patch for this is very
> simple, and doesn't seem likely to cause a performance issue on its
> own. My theory is that this test is one that causes the client to
> return the delegation, and since it doesn't have an open stateid, it
> has to reestablish one during the test run, and that causes the app
> overhead stat to spike.
>
> Trond, Tom, Mike -- I know that the HS Anvil has support for
> OPEN_XOR_DELEGATION. If you run the fsmark test against it with that
> support both enabled and disabled (either on the client or server
> side), do you see a similar spike in "App Overhead"?
>
> If so, then I suspect we need to consider limiting the use of that
> flag
> in some cases. I have no idea what heuristic we'd use to decide this
> though.

As already stated when we discussed this at Bakeathon: the server is
still in charge of heuristics w.r.t. whether or not there may be
contention for the file. The OPEN_XOR_DELEGATION flag changes nothing
in that respect.

Yes, I'm sure you can find tests which cause recalls of delegations,
and those will be marginally slower when the client has to re-establish
an open stateid. However the issue with those tests is that they are
deliberately setting up a situation where the server ideally shouldn't
be handing out a delegation at all.

Furthermore, this is no different than a situation where the client
used a delegation to cache the open (i.e. avoid sending an OPEN call)
after the application closed the file and then later re-opened it.
So the point is that this is not a situation that is unique to
OPEN_XOR_DELEGATION. It is just a consequence of the client's ability
to cache open state.

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: OPEN_XOR_DELEGATION performance problems

[Chuck Lever III]

> On Nov 19, 2024, at 10:09 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
> 
> On Tue, 2024-11-19 at 06:45 -0500, Jeff Layton wrote:
>> We attempted to implement the "delstid" draft for v6.13, but have had
>> to drop the patches for it. After merge, we got a couple of reports
>> of
>> a performance issue due to the OPEN_XOR_DELEGATION patch:
>> 
>> 
>> https://lore.kernel.org/linux-nfs/202409161645.d44bced5-oliver.sang@intel.com/
>> 
>> Once we enable OPEN_XOR_DELEGATION support, the fsmark "App Overhead"
>> statistic spikes significantly. The kernel patch for this is very
>> simple, and doesn't seem likely to cause a performance issue on its
>> own. My theory is that this test is one that causes the client to
>> return the delegation, and since it doesn't have an open stateid, it
>> has to reestablish one during the test run, and that causes the app
>> overhead stat to spike.
>> 
>> Trond, Tom, Mike -- I know that the HS Anvil has support for
>> OPEN_XOR_DELEGATION. If you run the fsmark test against it with that
>> support both enabled and disabled (either on the client or server
>> side), do you see a similar spike in "App Overhead"?
>> 
>> If so, then I suspect we need to consider limiting the use of that
>> flag
>> in some cases. I have no idea what heuristic we'd use to decide this
>> though.
> 
> As already stated when we discussed this at Bakeathon: the server is
> still in charge of heuristics w.r.t. whether or not there may be
> contention for the file. The OPEN_XOR_DELEGATION flag changes nothing
> in that respect.

fsmark is a single-client test. There should be no contention
for any files during this test.


> Yes, I'm sure you can find tests which cause recalls of delegations,
> and those will be marginally slower when the client has to re-establish
> an open stateid.

The fsmark result regressed 92%.


> However the issue with those tests is that they are
> deliberately setting up a situation where the server ideally shouldn't
> be handing out a delegation at all.
> 
> Furthermore, this is no different than a situation where the client
> used a delegation to cache the open (i.e. avoid sending an OPEN call)
> after the application closed the file and then later re-opened it.
> So the point is that this is not a situation that is unique to
> OPEN_XOR_DELEGATION. It is just a consequence of the client's ability
> to cache open state.

The regression was bisected to Jeff's XOR patch on two
separate occasions. This does indeed appear to be a
situation that is unique to OPEN_XOR_DELEGATION.

It's possible that our theory of the failure is wrong.
As developers of the only other server implementation of
OPEN_XOR_DELEGATION, can Hammerspace help us troubleshoot
this issue?


--
Chuck Lever

Re: OPEN_XOR_DELEGATION performance problems

[Jeff Layton]

On Tue, 2024-11-19 at 16:23 +0000, Chuck Lever III wrote:
> 
> > On Nov 19, 2024, at 10:09 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
> > 
> > On Tue, 2024-11-19 at 06:45 -0500, Jeff Layton wrote:
> > > We attempted to implement the "delstid" draft for v6.13, but have had
> > > to drop the patches for it. After merge, we got a couple of reports
> > > of
> > > a performance issue due to the OPEN_XOR_DELEGATION patch:
> > > 
> > > 
> > > https://lore.kernel.org/linux-nfs/202409161645.d44bced5-oliver.sang@intel.com/
> > > 
> > > Once we enable OPEN_XOR_DELEGATION support, the fsmark "App Overhead"
> > > statistic spikes significantly. The kernel patch for this is very
> > > simple, and doesn't seem likely to cause a performance issue on its
> > > own. My theory is that this test is one that causes the client to
> > > return the delegation, and since it doesn't have an open stateid, it
> > > has to reestablish one during the test run, and that causes the app
> > > overhead stat to spike.
> > > 
> > > Trond, Tom, Mike -- I know that the HS Anvil has support for
> > > OPEN_XOR_DELEGATION. If you run the fsmark test against it with that
> > > support both enabled and disabled (either on the client or server
> > > side), do you see a similar spike in "App Overhead"?
> > > 
> > > If so, then I suspect we need to consider limiting the use of that
> > > flag
> > > in some cases. I have no idea what heuristic we'd use to decide this
> > > though.
> > 
> > As already stated when we discussed this at Bakeathon: the server is
> > still in charge of heuristics w.r.t. whether or not there may be
> > contention for the file. The OPEN_XOR_DELEGATION flag changes nothing
> > in that respect.
> 
> fsmark is a single-client test. There should be no contention
> for any files during this test.
> 
> 
> > Yes, I'm sure you can find tests which cause recalls of delegations,
> > and those will be marginally slower when the client has to re-establish
> > an open stateid.
> 
> The fsmark result regressed 92%.
> 

To be clear, the fsmark "App Overhead" regressed 92%. Which has a
curious definition: "App overhead is time in microseconds spent in the
test not doing file writing related system calls."

It encompasses a bunch of different test setup stuff, and it's been
difficult to nail down the part that is slower. See Oliver's email
here:

https://lore.kernel.org/linux-nfs/ZwTm4e5JxOOJc7JC@xsang-OptiPlex-9020/

-- 
Jeff Layton <jlayton@kernel.org>

Re: OPEN_XOR_DELEGATION performance problems

[Cedric Blancher]

On Tue, 19 Nov 2024 at 17:31, Chuck Lever III <chuck.lever@oracle.com> wrote:
>
>
>
> > On Nov 19, 2024, at 10:09 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
> >
> > On Tue, 2024-11-19 at 06:45 -0500, Jeff Layton wrote:
> >> We attempted to implement the "delstid" draft for v6.13, but have had
> >> to drop the patches for it. After merge, we got a couple of reports
> >> of
> >> a performance issue due to the OPEN_XOR_DELEGATION patch:
> >>
> >>
> >> https://lore.kernel.org/linux-nfs/202409161645.d44bced5-oliver.sang@intel.com/
> >>
> >> Once we enable OPEN_XOR_DELEGATION support, the fsmark "App Overhead"
> >> statistic spikes significantly. The kernel patch for this is very
> >> simple, and doesn't seem likely to cause a performance issue on its
> >> own. My theory is that this test is one that causes the client to
> >> return the delegation, and since it doesn't have an open stateid, it
> >> has to reestablish one during the test run, and that causes the app
> >> overhead stat to spike.
> >>
> >> Trond, Tom, Mike -- I know that the HS Anvil has support for
> >> OPEN_XOR_DELEGATION. If you run the fsmark test against it with that
> >> support both enabled and disabled (either on the client or server
> >> side), do you see a similar spike in "App Overhead"?
> >>
> >> If so, then I suspect we need to consider limiting the use of that
> >> flag
> >> in some cases. I have no idea what heuristic we'd use to decide this
> >> though.
> >
> > As already stated when we discussed this at Bakeathon: the server is
> > still in charge of heuristics w.r.t. whether or not there may be
> > contention for the file. The OPEN_XOR_DELEGATION flag changes nothing
> > in that respect.
>
> fsmark is a single-client test. There should be no contention
> for any files during this test.
>
>
> > Yes, I'm sure you can find tests which cause recalls of delegations,
> > and those will be marginally slower when the client has to re-establish
> > an open stateid.
>
> The fsmark result regressed 92%.
>
>
> > However the issue with those tests is that they are
> > deliberately setting up a situation where the server ideally shouldn't
> > be handing out a delegation at all.
> >
> > Furthermore, this is no different than a situation where the client
> > used a delegation to cache the open (i.e. avoid sending an OPEN call)
> > after the application closed the file and then later re-opened it.
> > So the point is that this is not a situation that is unique to
> > OPEN_XOR_DELEGATION. It is just a consequence of the client's ability
> > to cache open state.
>
> The regression was bisected to Jeff's XOR patch on two
> separate occasions. This does indeed appear to be a
> situation that is unique to OPEN_XOR_DELEGATION.
>
> It's possible that our theory of the failure is wrong.
> As developers of the only other server implementation of
> OPEN_XOR_DELEGATION, can Hammerspace help us troubleshoot
> this issue?
>

Doesn't Tigran's dcache.org nfs4j server also support OPEN_XOR_DELEGATION?

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: OPEN_XOR_DELEGATION performance problems

[Mkrtchyan, Tigran]

[-- Attachment #1: Type: text/plain, Size: 3665 bytes --]

Unfortunately, for now, nfs4j doesn't support any type of delegation.

Tigran.

----- Original Message -----
> From: "Cedric Blancher" <cedric.blancher@gmail.com>
> To: "Linux NFS Mailing List" <linux-nfs@vger.kernel.org>, "Tiramisu Mokka" <kofemann@gmail.com>
> Sent: Wednesday, 20 November, 2024 08:39:00
> Subject: Re: OPEN_XOR_DELEGATION performance problems

> On Tue, 19 Nov 2024 at 17:31, Chuck Lever III <chuck.lever@oracle.com> wrote:
>>
>>
>>
>> > On Nov 19, 2024, at 10:09 AM, Trond Myklebust <trondmy@hammerspace.com> wrote:
>> >
>> > On Tue, 2024-11-19 at 06:45 -0500, Jeff Layton wrote:
>> >> We attempted to implement the "delstid" draft for v6.13, but have had
>> >> to drop the patches for it. After merge, we got a couple of reports
>> >> of
>> >> a performance issue due to the OPEN_XOR_DELEGATION patch:
>> >>
>> >>
>> >> https://lore.kernel.org/linux-nfs/202409161645.d44bced5-oliver.sang@intel.com/
>> >>
>> >> Once we enable OPEN_XOR_DELEGATION support, the fsmark "App Overhead"
>> >> statistic spikes significantly. The kernel patch for this is very
>> >> simple, and doesn't seem likely to cause a performance issue on its
>> >> own. My theory is that this test is one that causes the client to
>> >> return the delegation, and since it doesn't have an open stateid, it
>> >> has to reestablish one during the test run, and that causes the app
>> >> overhead stat to spike.
>> >>
>> >> Trond, Tom, Mike -- I know that the HS Anvil has support for
>> >> OPEN_XOR_DELEGATION. If you run the fsmark test against it with that
>> >> support both enabled and disabled (either on the client or server
>> >> side), do you see a similar spike in "App Overhead"?
>> >>
>> >> If so, then I suspect we need to consider limiting the use of that
>> >> flag
>> >> in some cases. I have no idea what heuristic we'd use to decide this
>> >> though.
>> >
>> > As already stated when we discussed this at Bakeathon: the server is
>> > still in charge of heuristics w.r.t. whether or not there may be
>> > contention for the file. The OPEN_XOR_DELEGATION flag changes nothing
>> > in that respect.
>>
>> fsmark is a single-client test. There should be no contention
>> for any files during this test.
>>
>>
>> > Yes, I'm sure you can find tests which cause recalls of delegations,
>> > and those will be marginally slower when the client has to re-establish
>> > an open stateid.
>>
>> The fsmark result regressed 92%.
>>
>>
>> > However the issue with those tests is that they are
>> > deliberately setting up a situation where the server ideally shouldn't
>> > be handing out a delegation at all.
>> >
>> > Furthermore, this is no different than a situation where the client
>> > used a delegation to cache the open (i.e. avoid sending an OPEN call)
>> > after the application closed the file and then later re-opened it.
>> > So the point is that this is not a situation that is unique to
>> > OPEN_XOR_DELEGATION. It is just a consequence of the client's ability
>> > to cache open state.
>>
>> The regression was bisected to Jeff's XOR patch on two
>> separate occasions. This does indeed appear to be a
>> situation that is unique to OPEN_XOR_DELEGATION.
>>
>> It's possible that our theory of the failure is wrong.
>> As developers of the only other server implementation of
>> OPEN_XOR_DELEGATION, can Hammerspace help us troubleshoot
>> this issue?
>>
> 
> Doesn't Tigran's dcache.org nfs4j server also support OPEN_XOR_DELEGATION?
> 
> Ced
> --
> Cedric Blancher <cedric.blancher@gmail.com>
> [https://plus.google.com/u/0/+CedricBlancher/]
> Institute Pasteur

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2826 bytes --]