Linux NFS development
 help / color / mirror / Atom feed
From: Jeff Layton <jlayton@primarydata.com>
To: bfields@fieldses.org
Cc: linux-nfs@vger.kernel.org
Subject: [PATCH v1 104/104] nfsd: add file documenting new state object model
Date: Thu, 19 Jun 2014 10:50:50 -0400	[thread overview]
Message-ID: <1403189450-18729-105-git-send-email-jlayton@primarydata.com> (raw)
In-Reply-To: <1403189450-18729-1-git-send-email-jlayton@primarydata.com>

Signed-off-by: Jeff Layton <jlayton@primarydata.com>
---
 .../filesystems/nfs/nfsd4-state-objects.txt        | 110 +++++++++++++++++++++
 1 file changed, 110 insertions(+)
 create mode 100644 Documentation/filesystems/nfs/nfsd4-state-objects.txt

diff --git a/Documentation/filesystems/nfs/nfsd4-state-objects.txt b/Documentation/filesystems/nfs/nfsd4-state-objects.txt
new file mode 100644
index 000000000000..fa3f70608422
--- /dev/null
+++ b/Documentation/filesystems/nfs/nfsd4-state-objects.txt
@@ -0,0 +1,110 @@
+KNFSD4 State Object Model:
+==========================
+Written 2014 by Jeff Layton <jlayton@primarydata.com>
+
+Introduction:
+-------------
+Until recently, knfsd relied heavily on a global mutex to ensure that objects
+didn't disappear while they were being operated on. That has proven to be a
+scalability bottleneck however, so the code has been overhauled to make heavy
+use of reference counting and spinlocks for tracking the different objects.
+
+The state model in NFSv4 is quite complex. Thus, the relationship between the
+objects to track this state in knfsd is also complex. This document attempts to
+lay out how all of the different objects relate to one another, and which ones
+hold references to others.
+
+In addition to the "persistent" references documented here, references to these
+objects can also be taken during the processing of compounds in order to ensure
+that the objects don't disappear suddenly.
+
+struct nfsd_net:
+----------------
+Represents a nfsd "container". With respect to nfsv4 state tracking, the fields
+of interest are the *_id_hashtbls and the *_name_tree. These track the
+nfs4_client objects by either short or long form clientid.
+
+Each nfsd_net runs a nfs4_laundromat workqueue job every lease period to clean
+up expired clients and delegations within the container.
+
+struct nfs4_file:
+-----------------
+These objects are global. nfsd only keeps one instance of a nfs4_file per inode
+(though it may keep multiple file descriptors open per inode). These are
+tracked in the file_hashtbl which is protected by the state_lock spinlock.
+
+struct nfs4_client:
+-------------------
+The initial object created by an NFS client using SETCLIENTID (for NFSv4.0) or
+EXCHANGE_ID (for NFSv4.1+). These objects are refcounted and timestamped. Each
+nfsd_net_ns object contains a set of these and they are tracked via short and
+long form clientid. They are hashed and searched for under the per-nfsd-net
+client_lock spinlock.
+
+The lifecycle of these is a little strange. References to it are only held
+during the processing of compounds, and in certain other operations. In their
+"resting state" they have a refcount of 0. If they are not renewed within a
+lease period, they become eligible for destruction by the laundromat.
+
+These objects can also be destroyed prematurely by the fault injection code,
+or if the client sends certain forms of SETCLIENTID or EXCHANGE_ID updates.
+Care is taken *not* to do this however when the objects have an elevated
+refcount.
+
+struct nfsd4_session:
+---------------------
+Represents a v4.1+ session. These are refcounted in a similar fashion to
+the nfs4_client. References are only taken when the server is actively working
+on the object (primarily during the processing of compounds).
+
+struct nfs4_stateowner:
+-----------------------
+A core object that represents either an open or lock owner. The object and lock
+owner objects have one of these embedded within them. Refcounts and other
+fields common to both owner types are contained within these structures.
+
+struct nfs4_openowner:
+----------------------
+When a file is opened, the client provides an open state owner opaque string
+that indicates the "owner" of that open. These objects are refcounted.
+References to it are held by each open state associated with it. This object is
+a superset of the nfs4_stateowner struct.
+
+struct nfs4_lockowner:
+----------------------
+Represents a generic "lockowner". Similar to an openowner. References to it are
+held by the lock stateids that are created on its behalf. This object is a
+superset of the nfs4_stateowner struct.
+
+strict nfs4_stid:
+-----------------
+A core object that represents a "generic" stateid. These are generally embedded
+within the different (more specific) stateid objects and contain fields that
+are of general use to any stateid.
+
+struct nfs4_ol_stateid:
+-----------------------
+A generic struct representing either a open or lock stateid. The nfs4_client
+holds a reference to each of these objects, and they in turn hold a reference
+to their respective stateowners. The client's reference is released in response
+to a close or unlock (depending on whether it's an open or lock stateid) or
+when the client is being destroyed.
+
+In the case of v4.0, these objects are preserved for a little while after close
+in order to handle CLOSE replays. They are eventually reclaimed via a LRU
+scheme by the laundromat.
+
+This object is a superset of the nfs4_stid.
+
+struct nfs4_delegation:
+-----------------------
+Represents a delegation stateid. The nfs4_client holds references to these and
+they are put when it is being destroyed or when the delegation is returned by
+the client.
+
+If the server attempts to recall a delegation and the client doesn't do so
+before a timeout, the server may also revoke the delegation. In that case, the
+object will either be destroyed (v4.0) or moved to a per-client list of revoked
+delegations (v4.1+).
+
+This object is a superset of the nfs4_stid.
-- 
1.9.3


  parent reply	other threads:[~2014-06-19 14:53 UTC|newest]

Thread overview: 144+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-06-19 14:49 [PATCH v1 000/104] nfsd: eliminate the client_mutex Jeff Layton
2014-06-19 14:49 ` [PATCH v1 001/104] nfsd: Protect addition to the file_hashtbl Jeff Layton
2014-06-19 14:49 ` [PATCH v1 002/104] NFSd: Avoid taking state_lock while holding inode lock in nfsd_break_one_deleg Jeff Layton
2014-06-23 13:54   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 003/104] NFSd: nfs4_preprocess_seqid_op should only set *stpp on success Jeff Layton
2014-06-19 14:49 ` [PATCH v1 004/104] nfsd4: use cl_lock to synchronize all stateid idr calls Jeff Layton
2014-06-23 16:02   ` Christoph Hellwig
2014-06-23 16:19     ` Jeff Layton
2014-06-19 14:49 ` [PATCH v1 005/104] NFSd: Add fine grained protection for the nfs4_file->fi_stateids list Jeff Layton
2014-06-23 16:04   ` Christoph Hellwig
2014-06-23 16:20     ` Jeff Layton
2014-06-23 16:23       ` Christoph Hellwig
2014-06-23 16:24         ` Jeff Layton
2014-06-23 16:32           ` J. Bruce Fields
2014-06-19 14:49 ` [PATCH v1 006/104] NFSd: Add a mutex to protect the NFSv4.0 open owner replay cache Jeff Layton
2014-06-23 16:05   ` Christoph Hellwig
2014-06-23 16:24     ` Jeff Layton
2014-06-19 14:49 ` [PATCH v1 007/104] NFSd: Add locking to the nfs4_file->fi_fds[] array Jeff Layton
2014-06-23 16:06   ` Christoph Hellwig
2014-06-23 17:14     ` Jeff Layton
2014-06-23 17:18       ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 008/104] NFSd: Ensure that nfs4_file_get_access enforces share access modes Jeff Layton
2014-06-23 16:12   ` Christoph Hellwig
2014-06-23 16:40     ` Jeff Layton
2014-06-19 14:49 ` [PATCH v1 009/104] locks: add file_has_lease Jeff Layton
2014-06-23 16:12   ` Christoph Hellwig
2014-06-23 16:48     ` Jeff Layton
2014-06-19 14:49 ` [PATCH v1 010/104] NFSd: Protect the nfs4_file delegation fields using the fi_lock Jeff Layton
2014-06-19 14:49 ` [PATCH v1 011/104] NFSd: Lock owners are not per open stateid Jeff Layton
2014-06-24 11:50   ` Christoph Hellwig
2014-06-24 12:04     ` Jeff Layton
2014-06-24 12:06       ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 012/104] NFSd: Clean up helper __release_lock_stateid Jeff Layton
2014-06-24 11:53   ` Christoph Hellwig
2014-06-24 12:00     ` Jeff Layton
2014-06-19 14:49 ` [PATCH v1 013/104] NFSd: Allow lockowners to hold several stateids Jeff Layton
2014-06-24 11:56   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 014/104] NFSd: NFSv4 lock-owners are not associated to a specific file Jeff Layton
2014-06-24 11:59   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 015/104] NFSd: Cleanup nfs4svc_encode_compoundres Jeff Layton
2014-06-24 12:00   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 016/104] NFSd: Don't get a session reference without a client reference Jeff Layton
2014-06-24 12:02   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 017/104] NFSd: Allow struct nfsd4_compound_state to cache the nfs4_client Jeff Layton
2014-06-24 12:02   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 018/104] NFSd: Move the delegation reference counter into the struct nfs4_stid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 019/104] NFSd: Simplify stateid management Jeff Layton
2014-06-19 14:49 ` [PATCH v1 020/104] NFSd: Fix delegation revocation Jeff Layton
2014-06-19 14:49 ` [PATCH v1 021/104] NFSd: Add reference counting to the lock and open stateids Jeff Layton
2014-06-19 14:49 ` [PATCH v1 022/104] NFSd: clean up nfsd4_close_open_stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 023/104] NFSd: Add a struct nfs4_file field to struct nfs4_stid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 024/104] NFSd: Replace nfs4_ol_stateid->st_file with the st_stid.sc_file Jeff Layton
2014-06-19 14:49 ` [PATCH v1 025/104] NFSd: Ensure stateids remain unique until they are freed Jeff Layton
2014-06-19 14:49 ` [PATCH v1 026/104] NFSd: Ensure atomicity of stateid destruction and idr tree removal Jeff Layton
2014-06-19 14:49 ` [PATCH v1 027/104] NFSd: Cleanup the freeing of stateids Jeff Layton
2014-06-19 14:49 ` [PATCH v1 028/104] nfsd: do filp_close in sc_free callback for lock stateids Jeff Layton
2014-06-19 14:49 ` [PATCH v1 029/104] NFSd: Add locking to protect the state owner lists Jeff Layton
2014-06-19 14:49 ` [PATCH v1 030/104] nfsd: clean up races in lock stateid searching and creation Jeff Layton
2014-06-19 14:49 ` [PATCH v1 031/104] NFSd: Convert delegation counter to an atomic_long_t type Jeff Layton
2014-06-19 14:49 ` [PATCH v1 032/104] NFSd: Slight cleanup of find_stateid() Jeff Layton
2014-06-19 14:49 ` [PATCH v1 033/104] NFSd: Add reference counting to find_stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 034/104] NFSd: Add reference counting to lock stateids Jeff Layton
2014-06-19 14:49 ` [PATCH v1 035/104] NFSd: nfsd4_locku() must reference the lock stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 036/104] NFSd: Ensure that nfs4_open_delegation() references the delegation stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 037/104] NFSd: nfsd4_process_open2() must reference " Jeff Layton
2014-06-19 14:49 ` [PATCH v1 038/104] NFSd: nfsd4_process_open2() must reference the open stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 039/104] NFSd: Prepare nfsd4_close() for open stateid referencing Jeff Layton
2014-06-19 14:49 ` [PATCH v1 040/104] NFSd: nfsd4_open_confirm() must reference the open stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 041/104] NFSd: Add reference counting to nfs4_preprocess_confirmed_seqid_op Jeff Layton
2014-06-19 14:49 ` [PATCH v1 042/104] NFSd: Migrate the stateid reference into nfs4_preprocess_seqid_op Jeff Layton
2014-06-19 14:49 ` [PATCH v1 043/104] NFSd: Migrate the stateid reference into nfs4_lookup_stateid() Jeff Layton
2014-06-19 14:49 ` [PATCH v1 044/104] NFSd: Migrate the stateid reference into nfs4_find_stateid_by_type() Jeff Layton
2014-06-19 14:49 ` [PATCH v1 045/104] NFSd: Add reference counting to state owners Jeff Layton
2014-06-19 14:49 ` [PATCH v1 046/104] NFSd: Keep a reference to the open stateid for the NFSv4.0 replay cache Jeff Layton
2014-06-19 14:49 ` [PATCH v1 047/104] nfsd: clean up lockowner refcounting when finding them Jeff Layton
2014-06-19 14:49 ` [PATCH v1 048/104] nfsd: add an operation for unhashing a stateowner Jeff Layton
2014-06-19 14:49 ` [PATCH v1 049/104] NFSd: Make lock stateid take a reference to the lockowner Jeff Layton
2014-06-19 14:49 ` [PATCH v1 050/104] nfsd: clean up nfs4_release_lockowner Jeff Layton
2014-06-19 14:49 ` [PATCH v1 051/104] nfsd: clean up refcounting for lockowners Jeff Layton
2014-06-19 14:49 ` [PATCH v1 052/104] nfsd: declare v4.1+ openowners confirmed on creation Jeff Layton
2014-06-19 14:49 ` [PATCH v1 053/104] nfsd: make openstateids hold references to their openowners Jeff Layton
2014-06-19 14:50 ` [PATCH v1 054/104] nfsd: don't allow CLOSE to proceed until refcount on stateid drops Jeff Layton
2014-06-19 14:50 ` [PATCH v1 055/104] NFSd: Protect adding/removing open state owners using client_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 056/104] NFSd: Protect adding/removing lock " Jeff Layton
2014-06-19 14:50 ` [PATCH v1 057/104] NFSd: Cleanup - Let nfsd4_lookup_stateid() take a cstate argument Jeff Layton
2014-06-19 14:50 ` [PATCH v1 058/104] NFSd: Cache the client that was looked up in lookup_clientid() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 059/104] NFSd: Convert nfsd4_process_open1() to work with lookup_clientid() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 060/104] NFSd: Always use lookup_clientid() in nfsd4_process_open1 Jeff Layton
2014-06-19 14:50 ` [PATCH v1 061/104] NFSd: Move the open owner hash table into struct nfs4_client Jeff Layton
2014-06-19 14:50 ` [PATCH v1 062/104] lockdep: add lockdep_assert_not_held Jeff Layton
2014-06-19 14:50 ` [PATCH v1 063/104] nfsd: add locking to stateowner release Jeff Layton
2014-06-19 14:50 ` [PATCH v1 064/104] nfsd: optimize destroy_lockowner cl_lock thrashing Jeff Layton
2014-06-19 14:50 ` [PATCH v1 065/104] nfsd: reduce cl_lock trashing in release_openowner Jeff Layton
2014-06-19 14:50 ` [PATCH v1 066/104] NFSd: Convert nfs4_check_open_reclaim() to work with lookup_clientid() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 067/104] NFSd: Ensure struct nfs4_client is unhashed before we try to destroy it Jeff Layton
2014-06-19 14:50 ` [PATCH v1 068/104] NFSd: Ensure that the laundromat unhashes the client before releasing locks Jeff Layton
2014-06-19 14:50 ` [PATCH v1 069/104] NFSd: Don't require client_lock in free_client Jeff Layton
2014-06-19 14:50 ` [PATCH v1 070/104] NFSd: Move create_client() call outside the lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 071/104] NFSd: Protect unconfirmed client creation using client_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 072/104] NFSd: Protect session creation and client confirm " Jeff Layton
2014-06-19 14:50 ` [PATCH v1 073/104] NFSd: Protect nfsd4_destroy_clientid " Jeff Layton
2014-06-19 14:50 ` [PATCH v1 074/104] NFSd: Ensure lookup_clientid() takes client_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 075/104] NFSd: Add assertions to document the nfs4_client/session locking Jeff Layton
2014-06-19 14:50 ` [PATCH v1 076/104] nfsd: protect the close_lru list and oo_last_closed_stid with client_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 077/104] nfsd: ensure that clp->cl_revoked list is protected by clp->cl_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 078/104] nfsd: move unhash_client_locked call into mark_client_expired_locked Jeff Layton
2014-06-19 14:50 ` [PATCH v1 079/104] nfsd: fix misleading comment Jeff Layton
2014-06-19 14:50 ` [PATCH v1 080/104] nfsd: don't destroy client if mark_client_expired_locked fails Jeff Layton
2014-06-19 14:50 ` [PATCH v1 081/104] nfsd: don't destroy clients that are busy Jeff Layton
2014-06-19 14:50 ` [PATCH v1 082/104] nfsd: abstract out the get and set routines into the fault injection ops Jeff Layton
2014-06-19 14:50 ` [PATCH v1 083/104] nfsd: add a forget_clients "get" routine with proper locking Jeff Layton
2014-06-19 14:50 ` [PATCH v1 084/104] nfsd: add a forget_client set_clnt routine Jeff Layton
2014-06-19 14:50 ` [PATCH v1 085/104] nfsd: add nfsd_inject_forget_clients Jeff Layton
2014-06-19 14:50 ` [PATCH v1 086/104] nfsd: add a list_head arg to nfsd_foreach_client_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 087/104] nfsd: add more granular locking to forget_locks fault injector Jeff Layton
2014-06-19 14:50 ` [PATCH v1 088/104] nfsd: add more granular locking to forget_openowners " Jeff Layton
2014-06-19 14:50 ` [PATCH v1 089/104] nfsd: add more granular locking to *_delegations fault injectors Jeff Layton
2014-06-19 14:50 ` [PATCH v1 090/104] nfsd: remove old fault injection infrastructure Jeff Layton
2014-06-19 14:50 ` [PATCH v1 091/104] NFSd: Remove nfs4_lock_state(): nfs4_preprocess_stateid_op() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 092/104] NFSd: Remove nfs4_lock_state(): nfsd4_test_stateid/nfsd4_free_stateid Jeff Layton
2014-06-19 14:50 ` [PATCH v1 093/104] NFSd: Remove nfs4_lock_state(): nfsd4_release_lockowner Jeff Layton
2014-06-19 14:50 ` [PATCH v1 094/104] NFSd: Remove nfs4_lock_state(): nfsd4_lock/locku/lockt() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 095/104] NFSd: Remove nfs4_lock_state(): nfsd4_open_downgrade + nfsd4_close Jeff Layton
2014-06-19 14:50 ` [PATCH v1 096/104] NFSd: Remove nfs4_lock_state(): nfsd4_delegreturn() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 097/104] NFSd: Remove nfs4_lock_state(): nfsd4_open and nfsd4_open_confirm Jeff Layton
2014-06-19 14:50 ` [PATCH v1 098/104] NFSd: Remove nfs4_lock_state(): exchange_id, create/destroy_session() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 099/104] NFSd: Remove nfs4_lock_state(): setclientid, setclientid_confirm, renew Jeff Layton
2014-06-19 14:50 ` [PATCH v1 100/104] NFSd: Remove nfs4_lock_state(): reclaim_complete() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 101/104] nfsd: remove nfs4_lock_state: nfs4_laundromat Jeff Layton
2014-06-19 14:50 ` [PATCH v1 102/104] nfsd: remove nfs4_lock_state: nfs4_state_shutdown_net Jeff Layton
2014-06-19 14:50 ` [PATCH v1 103/104] nfsd: remove the client_mutex and the nfs4_lock/unlock_state wrappers Jeff Layton
2014-06-19 14:50 ` Jeff Layton [this message]
2014-06-24 12:11   ` [PATCH v1 104/104] nfsd: add file documenting new state object model Christoph Hellwig
2014-06-24 12:23     ` Jeff Layton
2014-06-19 15:33 ` [PATCH v1 000/104] nfsd: eliminate the client_mutex Jeff Layton
2014-06-26  1:26   ` J. Bruce Fields
2014-06-26  1:34     ` Jeff Layton
2014-06-23 13:39 ` Christoph Hellwig
2014-06-23 13:56   ` Jeff Layton
2014-06-23 14:11     ` J. Bruce Fields
2014-06-23 16:00     ` Christoph Hellwig
2014-06-23 16:10       ` J. Bruce Fields
2014-06-23 16:13         ` Christoph Hellwig
2014-06-23 18:26         ` Jeff Layton

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1403189450-18729-105-git-send-email-jlayton@primarydata.com \
    --to=jlayton@primarydata.com \
    --cc=bfields@fieldses.org \
    --cc=linux-nfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox