Re: Question about nlmclnt_lock

[Tom Talpey <tom@talpey.com>]

On 8/6/2022 3:49 PM, Trond Myklebust wrote:
> On Sat, 2022-08-06 at 11:03 -0400, Jan Kasiak wrote:
>> Hi Trond,
>>
>> The v4 RFCs do mention protocol design flaws, but don't go into more
>> detail.
>>
>> I was trying to understand those flaws in order to understand how and
>> why v3 was problematic.
>>
>>
> 
> The main issues derive from the fact that NLM is a side band protocol,
> meaning that it has no ability to influence the NFS protocol
> operations. In particular, there is no way to ensure safe ordering of
> locks and I/O. e.g. if your readahead code kicks in while you are
> unlocking the file, then there is nothing that guarantees the page
> reads happened while the lock was in place on the server.
> The same weakness also causes problems for reboots: if your client
> doesn't notice that the server rebooted (and lost your locks) because
> the statd callback mechanism failed, then you're SOL. Your I/O may
> succeed, but can end up causing problems for another client that has
> since grabbed the lock and assumes it now has exclusive access to the
> file.
> 
> NLM also suffers from intrinsic problems of its own such as lack of
> only-once semantics. If you send a blocking LOCK request, and
> subsequently send a CANCEL operation, then who knows whether or not the
> lock or the cancel get processed first by the server? Many servers will
> reply LCK_GRANTED to the CANCEL even if they did not find the lock
> request. Sending an UNLOCK can also cause issues if the lock was
> granted via a blocking lock callback (NLM_GRANTED) since there is no
> ordering between the reply to the NLM_GRANTED and the UNLOCK.
> 
> Finally, as already mentioned, there are multiple issues associated
> with client or server reboot. The NLM mechanism is pretty dependent on
> yet another side band mechanism (STATD) to tell you when this occurs,
> but that mechanism does not work to release the locks held by a client
> if it fails to come back after reboot. Even if the client does come
> back, it might forget to invoke the statd process, or it might use a
> different identifier than it did during the last boot instance (e.g.
> because DHCP allocated a different IP address, or the IP address it not
> unique due to use of NAT, or a hostname was used that is non-unique,
> ...).
> If the server reboots, then it may fail to notify the client of that
> reboot through the callback mechanism. Reasons may include the
> existence of a NAT, failure of the rpcbind/portmapper process on the
> client, firewalls,...

That brought back memories.

http://www.nfsv4bat.org/Documents/ConnectAThon/2006/talpey-cthon06-nsm.pdf

Here's an even older issues list for nlm on Solaris circa 1996.
The portrait-mode slides are in reverse order. :)

http://www.nfsv4bat.org/Documents/ConnectAThon/1996/lockmgr.pdf

The NLM protocol is an antique and hasn't been looked at in well
over a decade (or two!). NLMv4 (circa 1995) widened offsets to
64-bit, which was the last innovation it got. None of the RPC
sideband protocols were ever standardized, btw.

Jan, what are you planning to use it for? Personally I'd advise
against pretty much anything.

Tom.

> 
>> -Jan
>>
>>
>> On Fri, Aug 5, 2022 at 10:27 PM Trond Myklebust
>> <trondmy@hammerspace.com> wrote:
>>>
>>> On Fri, 2022-08-05 at 19:17 -0400, Jan Kasiak wrote:
>>>> Hi,
>>>>
>>>> I was looking at the code for nlmclnt_lock and wanted to ask a
>>>> question about how the Linux kernel client and the NLM 4 protocol
>>>> handle some errors around certain edge cases.
>>>>
>>>> Specifically, I think there is a race condition around two
>>>> threads of
>>>> the same program acquiring a lock, one of the threads being
>>>> interrupted, and the NFS client sending an unlock when none of
>>>> the
>>>> program threads called unlock.
>>>>
>>>> On NFS server machine S:
>>>> there exists an unlocked file F
>>>>
>>>> On NFS client machine C:
>>>> in program P:
>>>> thread 1 tries to lock(F) with fd A
>>>> thread 2 tries to lock(F) with fd B
>>>>
>>>> The Linux client will issue two NLM_LOCK calls with the same svid
>>>> and
>>>> same range, because it uses the program id to map to an svid.
>>>>
>>>> For whatever reason, assume the connection is broken (cable gets
>>>> pulled etc...)
>>>> and `status = nlmclnt_call(cred, req, NLMPROC_LOCK);` fails.
>>>>
>>>> The Linux client will retry the request, but at some point thread
>>>> 1
>>>> receives a signal and nlmclnt_lock breaks out of its loop.
>>>> Because
>>>> the
>>>> Linux client request failed, it will fall through and go to the
>>>> out_unlock label, where it will want to send an unlock request.
>>>>
>>>> Assume that at some point the connection is reestablished.
>>>>
>>>> The Linux kernel client now has two outstanding lock requests to
>>>> send
>>>> to the remote server: one for a lock that thread 2 is still
>>>> trying to
>>>> acquire, and one for an unlock of thread 1 that failed and was
>>>> interrupted.
>>>>
>>>> I'm worried that the Linux client may first send the lock
>>>> request,
>>>> and
>>>> tell thread 2 that it acquired the lock, and then send an unlock
>>>> request from the cancelled thread 1 request.
>>>>
>>>> The server will successfully process both requests, because the
>>>> svid
>>>> is the same for both, and the true server side state will be that
>>>> the
>>>> file is unlocked.
>>>>
>>>> One can talk about the wisdom of using multiple threads to
>>>> acquire
>>>> the
>>>> same file lock, but this behavior is weird, because none of the
>>>> threads called unlock.
>>>>
>>>> I have experimented with reproducing this, but have not been
>>>> successful in triggering this ordering of events.
>>>>
>>>> I've also looked at the code of in clntproc.c and I don't see a
>>>> spot
>>>> where outstanding failed lock/unlock requests are checked while
>>>> processing lock requests?
>>>>
>>>> Thanks,
>>>> -Jan
>>>
>>> Nobody here is likely to want to waste much time trying to 'fix'
>>> the
>>> NLM locking protocol. The protocol itself is known to be extremely
>>> fragile, and the endemic problems constitute some of the main
>>> motivations for the development of the NFSv4 protocol
>>> (See https://datatracker.ietf.org/doc/html/rfc2624#section-8
>>> and https://datatracker.ietf.org/doc/html/rfc7530#section-9).
>>>
>>> If you need more reliable support for POSIX locks beyond what
>>> exists
>>> today for NLM, then please consider NFSv4.
>>>
>>> --
>>> Trond Myklebust
>>> Linux NFS client maintainer, Hammerspace
>>> trond.myklebust@hammerspace.com
>>>
>>>
>