* Re: Questions Regarding Delegation Claim Behavior
2025-05-26 11:10 ` Petro Pavlov
@ 2025-05-26 13:36 ` Rick Macklem
2025-05-26 14:55 ` Jeff Layton
` (2 subsequent siblings)
3 siblings, 0 replies; 9+ messages in thread
From: Rick Macklem @ 2025-05-26 13:36 UTC (permalink / raw)
To: Petro Pavlov; +Cc: Chuck Lever, Roi Azarzar, linux-nfs
On Mon, May 26, 2025 at 4:10 AM Petro Pavlov <petro.pavlov@vastdata.com> wrote:
>
> Hello Chuck,
>
> Thank you for your response, and apologies for the confusion regarding the kernel version — the correct version is 6.15.0-rc3+ (I believe it's from the branch you gave us). Regarding the client, I'm using hand-written tests based on pynfs.
>
> I believe the following section of the RFC may be relevant to the use of a delegation stateid in relation to the file being accessed:
>
> If the stateid type is not valid for the context in which the stateid appears, return NFS4ERR_BAD_STATEID. Note that a stateid may be valid in general, as would be reported by the TEST_STATEID operation, but be invalid for a particular operation, as, for example, when a stateid that doesn't represent byte-range locks is passed to the non-from_open case of LOCK or to LOCKU, or when a stateid that does not represent an open is passed to CLOSE or OPEN_DOWNGRADE. In such cases, the server MUST return NFS4ERR_BAD_STATEID.
>
>
> I did some further investigation and identified another scenario that seems problematic:
>
> Client1 creates file1 without a delegation, with read-write access. It writes some data, changes the file mode to 444, and then closes the file.
>
> Client2 opens file1 with read access, receives a read delegation (deleg1), and closes the file without returning the delegation.
>
> Client2 then creates file2 with read-write access, receives a write delegation (deleg2), and leaves the file open (delegation is not returned).
>
> Client2 tries to open file1 with write access and receives an ACCESS_DENIED error (expected).
>
> Next, Client2 attempts to open file1 with write access using CLAIM_DELEGATE_CUR, providing the stateid from deleg2 (which was issued for file2) — unexpectedly, the operation succeeds.
>
> Client2 proceeds to write to file1, and it also succeeds — despite the file being set to 444, where no write access should be allowed.
>
> This behavior seems incorrect, as I would expect the write operation to fail due to file permissions.
The behaviour is incorrect because the client is being malicious (or
buggy, if you prefer) (not playing by the rules).
Then the question becomes "How well should a server handle a client
that is not playing by the rules?".
Imho (I am not a Linux developer), the most that a server
implementation should strive for is to:
- Not crash.
- Avoid DOS attacks, if possible.
Other than that, malicious clients should be blocked via /etc/exports
or a firewall or ???
Just my $0.02 worth, rick
>
> Please see the attached PCAP file for reference.
>
> Best regards,
> Petro Pavlov
>
>
> On Fri, May 23, 2025 at 5:41 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>>
>> On 5/22/25 11:51 AM, Petro Pavlov wrote:
>> > Hello,
>> >
>> > My name is Petro Pavlov, I'm a developer at VAST.
>> >
>> > I have a few questions about the delegation claim behavior observed in
>> > the Linux kernel version 3.10.0-1160.118.1.el7.x86_64.
>> >
>> > I’ve written the following test case:
>> >
>> > 1. Client1 opens *file1* with a write delegation; the server grants
>> > both the open and delegation (*delegation1*).
>>
>> Since you mention a write delegation, I'll assume you are using Linux
>> as an NFS client, and the server is not Linux, since that kernel version
>> does not implement server-side write delegation.
>>
>>
>> > 2. Client1 closes the open but does not return the delegation.
>> > 3. Client2 opens *file2* and also receives a write delegation
>> > (*delegation2*).
>> > 4. Client1 then issues an open request with CLAIM_DELEGATE_CUR,
>> > providing the filename from step 3 *(file2*), but using the
>> > delegation stateid from step 1 (*delegation1*).
>>
>> Would that be a client bug?
>>
>>
>> > 5. The server begins a recall of *delegation2*, treating the request in
>> > step 4 as a normal open rather than returning a BAD_STATEID error.
>>
>> That seems OK to me. The server has correctly noticed that the
>> client is opening file2, and delegation2 is associated with a
>> previous open of file2.
>>
>> A better place to seek an authoritative answer might be RFC 8881.
>>
>> The server returns BAD_STATEID if the stateid doesn't pass various
>> checks as outlined in Section 8.2. I don't see any text requiring the
>> server to report BAD_STATEID if delegate_stateid does not match the
>> component4 on a DELEGATE_CUR OPEN -- in fact, Table 19 says that
>> DELEGATE_CUR considers only the current file handle (the parent
>> directory) and the component4 argument.
>>
>>
>> > My understanding is that the server should have verified whether the
>> > delegation stateid provided actually belongs to the file being opened.
>> > Since it does not, I expected the server to return a BAD_STATEID error
>> > instead of proceeding with a standard open.
>> >
>> > From additional tests, it seems the server only checks whether the
>> > delegation stateid is valid (i.e., whether it was ever granted), but
>> > does not verify that it is associated with the correct file or client.
>> > Please see the attached PCAP for reference.
>> >
>> > Questions:
>> >
>> > Is this behavior considered a bug?
>> >
>> > Are there any known plans to address or fix this issue in future kernel
>> > versions?
>>
>> AFAICT at the moment, NOTABUG
>>
>>
>> --
>> Chuck Lever
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Questions Regarding Delegation Claim Behavior
2025-05-26 11:10 ` Petro Pavlov
2025-05-26 13:36 ` Rick Macklem
@ 2025-05-26 14:55 ` Jeff Layton
2025-05-27 12:58 ` Chuck Lever
2025-06-09 20:34 ` Dai Ngo
3 siblings, 0 replies; 9+ messages in thread
From: Jeff Layton @ 2025-05-26 14:55 UTC (permalink / raw)
To: Petro Pavlov, Chuck Lever; +Cc: Roi Azarzar, linux-nfs
On Mon, 2025-05-26 at 14:10 +0300, Petro Pavlov wrote:
> Hello Chuck,
> Thank you for your response, and apologies for the confusion regarding the kernel version — the correct version is 6.15.0-rc3+ (I believe it's from the branch you gave us). Regarding the client, I'm using hand-written tests based on pynfs.
> I believe the following section of the RFC may be relevant to the use of a delegation stateid in relation to the file being accessed:
> > If the stateid type is not valid for the context in which the stateid appears, return NFS4ERR_BAD_STATEID. Note that a stateid may be valid in general, as would be reported by the TEST_STATEID operation, but be invalid for a particular operation, as, for example, when a stateid that doesn't represent byte-range locks is passed to the non-from_open case of LOCK or to LOCKU, or when a stateid that does not represent an open is passed to CLOSE or OPEN_DOWNGRADE. In such cases, the server MUST return NFS4ERR_BAD_STATEID.
>
> I did some further investigation and identified another scenario that seems problematic:
> 1.
> Client1 creates file1 without a delegation, with read-write access. It writes some data, changes the file mode to 444, and then closes the file.
> 2.
> Client2 opens file1 with read access, receives a read delegation (deleg1), and closes the file without returning the delegation.
> 3.
> Client2 then creates file2 with read-write access, receives a write delegation (deleg2), and leaves the file open (delegation is not returned).
> 4.
> Client2 tries to open file1 with write access and receives an ACCESS_DENIED error (expected).
> 5. Next, Client2 attempts to open file1 with write access using CLAIM_DELEGATE_CUR, providing the stateid from deleg2 (which was issued for file2) — unexpectedly, the operation succeeds.
> 6.
> Client2 proceeds to write to file1, and it also succeeds — despite the file being set to 444, where no write access should be allowed.
> This behavior seems incorrect, as I would expect the write operation to fail due to file permissions.
> Please see the attached PCAP file for reference.
> Best regards,
> Petro Pavlov
>
Yeah, that does seem like a bug. Mostly, the server treats
CLAIM_DELEGATE_CUR and CLAIM_NULL identically. We have this in
do_open_lookup() though:
if (open->op_created ||
open->op_claim_type == NFS4_OPEN_CLAIM_DELEGATE_CUR)
accmode |= NFSD_MAY_OWNER_OVERRIDE;
status = do_open_permission(rqstp, *resfh, open, accmode);
set_change_info(&open->op_cinfo, current_fh);
...and that effectively bypasses the permission check in this case.
Maybe we should add a fh check there?
I'm not sure why CLAIM_DELEGATE_CUR wasn't deprecated when
CLAIM_DELEGATE_CUR_FH was added to the spec. Is there some use case
where CUR is necessary instead of CUR_FH?
> On Fri, May 23, 2025 at 5:41 PM Chuck Lever <chuck.lever@oracle.com> wrote:
> > On 5/22/25 11:51 AM, Petro Pavlov wrote:
> > > Hello,
> > >
> > > My name is Petro Pavlov, I'm a developer at VAST.
> > >
> > > I have a few questions about the delegation claim behavior observed in
> > > the Linux kernel version 3.10.0-1160.118.1.el7.x86_64.
> > >
> > > I’ve written the following test case:
> > >
> > > 1. Client1 opens *file1* with a write delegation; the server grants
> > > both the open and delegation (*delegation1*).
> >
> > Since you mention a write delegation, I'll assume you are using Linux
> > as an NFS client, and the server is not Linux, since that kernel version
> > does not implement server-side write delegation.
> >
> >
> > > 2. Client1 closes the open but does not return the delegation.
> > > 3. Client2 opens *file2* and also receives a write delegation
> > > (*delegation2*).
> > > 4. Client1 then issues an open request with CLAIM_DELEGATE_CUR,
> > > providing the filename from step 3 *(file2*), but using the
> > > delegation stateid from step 1 (*delegation1*).
> >
> > Would that be a client bug?
> >
> >
> > > 5. The server begins a recall of *delegation2*, treating the request in
> > > step 4 as a normal open rather than returning a BAD_STATEID error.
> >
> > That seems OK to me. The server has correctly noticed that the
> > client is opening file2, and delegation2 is associated with a
> > previous open of file2.
> >
> > A better place to seek an authoritative answer might be RFC 8881.
> >
> > The server returns BAD_STATEID if the stateid doesn't pass various
> > checks as outlined in Section 8.2. I don't see any text requiring the
> > server to report BAD_STATEID if delegate_stateid does not match the
> > component4 on a DELEGATE_CUR OPEN -- in fact, Table 19 says that
> > DELEGATE_CUR considers only the current file handle (the parent
> > directory) and the component4 argument.
> >
> >
> > > My understanding is that the server should have verified whether the
> > > delegation stateid provided actually belongs to the file being opened.
> > > Since it does not, I expected the server to return a BAD_STATEID error
> > > instead of proceeding with a standard open.
> > >
> > > From additional tests, it seems the server only checks whether the
> > > delegation stateid is valid (i.e., whether it was ever granted), but
> > > does not verify that it is associated with the correct file or client.
> > > Please see the attached PCAP for reference.
> > >
> > > Questions:
> > >
> > > Is this behavior considered a bug?
> > >
> > > Are there any known plans to address or fix this issue in future kernel
> > > versions?
> >
> > AFAICT at the moment, NOTABUG
> >
> >
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Questions Regarding Delegation Claim Behavior
2025-05-26 11:10 ` Petro Pavlov
2025-05-26 13:36 ` Rick Macklem
2025-05-26 14:55 ` Jeff Layton
@ 2025-05-27 12:58 ` Chuck Lever
2025-05-27 13:22 ` Chuck Lever
2025-06-09 20:34 ` Dai Ngo
3 siblings, 1 reply; 9+ messages in thread
From: Chuck Lever @ 2025-05-27 12:58 UTC (permalink / raw)
To: Petro Pavlov; +Cc: Roi Azarzar, linux-nfs
On 5/26/25 7:10 AM, Petro Pavlov wrote:
> Hello Chuck,
>
> Thank you for your response, and apologies for the confusion regarding
> the kernel version — the correct version is 6.15.0-rc3+ (I believe it's
> from the branch you gave us). Regarding the client, I'm using hand-
> written tests based on pynfs.
>
> I believe the following section of the RFC may be relevant to the use of
> a delegation |stateid| in relation to the file being accessed:
>
> If the stateid type is not valid for the context in which the
> stateid appears, return NFS4ERR_BAD_STATEID. Note that a stateid may
> be valid in general, as would be reported by the TEST_STATEID
> operation, but be invalid for a particular operation, as, for
> example, when a stateid that doesn't represent byte-range locks is
> passed to the non-from_open case of LOCK or to LOCKU, or when a
> stateid that does not represent an open is passed to CLOSE or
> OPEN_DOWNGRADE. In such cases, the server MUST return
> NFS4ERR_BAD_STATEID.
>
>
> I did some further investigation and identified another scenario that
> seems problematic:
>
> 1.
>
> *Client1* creates |file1| without a delegation, with read-write
> access. It writes some data, changes the file mode to |444|, and
> then closes the file.
>
> 2.
>
> *Client2* opens |file1| with read access, receives a read delegation
> (|deleg1|), and closes the file without returning the delegation.
>
> 3.
>
> *Client2* then creates |file2| with read-write access, receives a
> write delegation (|deleg2|), and leaves the file open (delegation is
> not returned).
>
> 4.
>
> *Client2* tries to open |file1| with write access and receives an |
> ACCESS_DENIED| error (expected).
>
> 5.
>
> Next, *Client2* attempts to open |file1| with *write *access using
> CLAIM_DELEGATE_CUR, providing the stateid from deleg2 (which was
> issued for |file2|) — unexpectedly, the operation succeeds.
>
> 6.
>
> *Client2* proceeds to write to |file1|, and it also succeeds —
> despite the file being set to |444|, where no write access should be
> allowed.
>
> This behavior seems incorrect, as I would expect the write operation to
> fail due to file permissions.
This is a server bug, since the permissions on file1 prohibit
writes to the file.
> Please see the attached PCAP file for reference.
>
> Best regards,
> Petro Pavlov
>
>
> On Fri, May 23, 2025 at 5:41 PM Chuck Lever <chuck.lever@oracle.com
> <mailto:chuck.lever@oracle.com>> wrote:
>
> On 5/22/25 11:51 AM, Petro Pavlov wrote:
> > Hello,
> >
> > My name is Petro Pavlov, I'm a developer at VAST.
> >
> > I have a few questions about the delegation claim behavior observed in
> > the Linux kernel version 3.10.0-1160.118.1.el7.x86_64.
> >
> > I’ve written the following test case:
> >
> > 1. Client1 opens *file1* with a write delegation; the server grants
> > both the open and delegation (*delegation1*).
>
> Since you mention a write delegation, I'll assume you are using Linux
> as an NFS client, and the server is not Linux, since that kernel version
> does not implement server-side write delegation.
>
>
> > 2. Client1 closes the open but does not return the delegation.
> > 3. Client2 opens *file2* and also receives a write delegation
> > (*delegation2*).
> > 4. Client1 then issues an open request with CLAIM_DELEGATE_CUR,
> > providing the filename from step 3 *(file2*), but using the
> > delegation stateid from step 1 (*delegation1*).
>
> Would that be a client bug?
>
>
> > 5. The server begins a recall of *delegation2*, treating the
> request in
> > step 4 as a normal open rather than returning a BAD_STATEID error.
>
> That seems OK to me. The server has correctly noticed that the
> client is opening file2, and delegation2 is associated with a
> previous open of file2.
>
> A better place to seek an authoritative answer might be RFC 8881.
>
> The server returns BAD_STATEID if the stateid doesn't pass various
> checks as outlined in Section 8.2. I don't see any text requiring the
> server to report BAD_STATEID if delegate_stateid does not match the
> component4 on a DELEGATE_CUR OPEN -- in fact, Table 19 says that
> DELEGATE_CUR considers only the current file handle (the parent
> directory) and the component4 argument.
>
>
> > My understanding is that the server should have verified whether the
> > delegation stateid provided actually belongs to the file being opened.
> > Since it does not, I expected the server to return a BAD_STATEID error
> > instead of proceeding with a standard open.
> >
> > From additional tests, it seems the server only checks whether the
> > delegation stateid is valid (i.e., whether it was ever granted), but
> > does not verify that it is associated with the correct file or client.
> > Please see the attached PCAP for reference.
> >
> > Questions:
> >
> > Is this behavior considered a bug?
> >
> > Are there any known plans to address or fix this issue in future
> kernel
> > versions?
>
> AFAICT at the moment, NOTABUG
>
>
> --
> Chuck Lever
>
--
Chuck Lever
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Questions Regarding Delegation Claim Behavior
2025-05-27 12:58 ` Chuck Lever
@ 2025-05-27 13:22 ` Chuck Lever
[not found] ` <CAN5pLa66Fg0kWcwreggDP1btu=guC7ZgZrKX74sKSuej3mXwfQ@mail.gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: Chuck Lever @ 2025-05-27 13:22 UTC (permalink / raw)
To: Petro Pavlov; +Cc: Roi Azarzar, linux-nfs
On 5/27/25 8:58 AM, Chuck Lever wrote:
> On 5/26/25 7:10 AM, Petro Pavlov wrote:
>>
>> I believe the following section of the RFC may be relevant to the use of
>> a delegation |stateid| in relation to the file being accessed:
>>
>> If the stateid type is not valid for the context in which the
>> stateid appears, return NFS4ERR_BAD_STATEID. Note that a stateid may
>> be valid in general, as would be reported by the TEST_STATEID
>> operation, but be invalid for a particular operation, as, for
>> example, when a stateid that doesn't represent byte-range locks is
>> passed to the non-from_open case of LOCK or to LOCKU, or when a
>> stateid that does not represent an open is passed to CLOSE or
>> OPEN_DOWNGRADE. In such cases, the server MUST return
>> NFS4ERR_BAD_STATEID.
For the record, this is the sixth bullet point in the final paragraph of
Section 8.2.4 of RFC 8881.
Perhaps this text is applicable, but I have trouble with a compliance
statement that mandates specific on-the-wire behavior but does not have
an exhaustive list of cases where it applies. Such statements are not
supposed to be so hand-wavy.
And in any event, I prefer NFS4ERR_INVAL in such cases, as the stateid
is clearly valid. BAD_STATEID will most likely trigger the client to
perform state recovery, which is almost certain to cause the client to
simply resend the same request after determining that no state recovery
is needed. Lather, rinse, repeat.
But, a pynfs test case or two here is a good thing. The client is
misbehaving, and our CI isn't looking for this corner case. I hope you
are permitted to contribute your test cases to upstream pynfs.
--
Chuck Lever
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Questions Regarding Delegation Claim Behavior
2025-05-26 11:10 ` Petro Pavlov
` (2 preceding siblings ...)
2025-05-27 12:58 ` Chuck Lever
@ 2025-06-09 20:34 ` Dai Ngo
3 siblings, 0 replies; 9+ messages in thread
From: Dai Ngo @ 2025-06-09 20:34 UTC (permalink / raw)
To: Petro Pavlov, Chuck Lever; +Cc: Roi Azarzar, linux-nfs
On 5/26/25 4:10 AM, Petro Pavlov wrote:
>
> Hello Chuck,
>
> Thank you for your response, and apologies for the confusion regarding
> the kernel version — the correct version is 6.15.0-rc3+ (I believe
> it's from the branch you gave us). Regarding the client, I'm using
> hand-written tests based on pynfs.
>
> I believe the following section of the RFC may be relevant to the use
> of a delegation |stateid| in relation to the file being accessed:
>
> If the stateid type is not valid for the context in which the
> stateid appears, return NFS4ERR_BAD_STATEID. Note that a stateid
> may be valid in general, as would be reported by the TEST_STATEID
> operation, but be invalid for a particular operation, as, for
> example, when a stateid that doesn't represent byte-range locks is
> passed to the non-from_open case of LOCK or to LOCKU, or when a
> stateid that does not represent an open is passed to CLOSE or
> OPEN_DOWNGRADE. In such cases, the server MUST return
> NFS4ERR_BAD_STATEID.
>
>
> I did some further investigation and identified another scenario that
> seems problematic:
>
> 1.
>
> *Client1* creates |file1| without a delegation, with read-write
> access. It writes some data, changes the file mode to |444|, and
> then closes the file.
>
> 2.
>
> *Client2* opens |file1| with read access, receives a read
> delegation (|deleg1|), and closes the file without returning the
> delegation.
>
> 3.
>
> *Client2* then creates |file2| with read-write access, receives a
> write delegation (|deleg2|), and leaves the file open (delegation
> is not returned).
>
> 4.
>
> *Client2* tries to open |file1| with write access and receives an
> |ACCESS_DENIED| error (expected).
>
> 5.
>
> Next, *Client2* attempts to open |file1| with *write *access
> using CLAIM_DELEGATE_CUR, providing the stateid from deleg2
> (which was issued for |file2|) — unexpectedly, the operation succeeds.
>
I think the server should detect that the delegation stateid and the file
component do not belong to the same file and returns NFS4ERR_BAD_STATEID.
This will also prevent the subsequent access check problem.
-Dai
> 1.
>
>
> 2.
>
> *Client2* proceeds to write to |file1|, and it also succeeds —
> despite the file being set to |444|, where no write access should
> be allowed.
>
> This behavior seems incorrect, as I would expect the write operation
> to fail due to file permissions.
>
> Please see the attached PCAP file for reference.
>
> Best regards,
> Petro Pavlov
>
>
> On Fri, May 23, 2025 at 5:41 PM Chuck Lever <chuck.lever@oracle.com>
> wrote:
>
> On 5/22/25 11:51 AM, Petro Pavlov wrote:
> > Hello,
> >
> > My name is Petro Pavlov, I'm a developer at VAST.
> >
> > I have a few questions about the delegation claim behavior
> observed in
> > the Linux kernel version 3.10.0-1160.118.1.el7.x86_64.
> >
> > I’ve written the following test case:
> >
> > 1. Client1 opens *file1* with a write delegation; the server grants
> > both the open and delegation (*delegation1*).
>
> Since you mention a write delegation, I'll assume you are using Linux
> as an NFS client, and the server is not Linux, since that kernel
> version
> does not implement server-side write delegation.
>
>
> > 2. Client1 closes the open but does not return the delegation.
> > 3. Client2 opens *file2* and also receives a write delegation
> > (*delegation2*).
> > 4. Client1 then issues an open request with CLAIM_DELEGATE_CUR,
> > providing the filename from step 3 *(file2*), but using the
> > delegation stateid from step 1 (*delegation1*).
>
> Would that be a client bug?
>
>
> > 5. The server begins a recall of *delegation2*, treating the
> request in
> > step 4 as a normal open rather than returning a BAD_STATEID
> error.
>
> That seems OK to me. The server has correctly noticed that the
> client is opening file2, and delegation2 is associated with a
> previous open of file2.
>
> A better place to seek an authoritative answer might be RFC 8881.
>
> The server returns BAD_STATEID if the stateid doesn't pass various
> checks as outlined in Section 8.2. I don't see any text requiring the
> server to report BAD_STATEID if delegate_stateid does not match the
> component4 on a DELEGATE_CUR OPEN -- in fact, Table 19 says that
> DELEGATE_CUR considers only the current file handle (the parent
> directory) and the component4 argument.
>
>
> > My understanding is that the server should have verified whether the
> > delegation stateid provided actually belongs to the file being
> opened.
> > Since it does not, I expected the server to return a BAD_STATEID
> error
> > instead of proceeding with a standard open.
> >
> > From additional tests, it seems the server only checks whether the
> > delegation stateid is valid (i.e., whether it was ever granted), but
> > does not verify that it is associated with the correct file or
> client.
> > Please see the attached PCAP for reference.
> >
> > Questions:
> >
> > Is this behavior considered a bug?
> >
> > Are there any known plans to address or fix this issue in future
> kernel
> > versions?
>
> AFAICT at the moment, NOTABUG
>
>
> --
> Chuck Lever
>
^ permalink raw reply [flat|nested] 9+ messages in thread