All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jeff Layton <jlayton@primarydata.com>
To: bfields@fieldses.org
Cc: linux-nfs@vger.kernel.org
Subject: [PATCH v1 104/104] nfsd: add file documenting new state object model
Date: Thu, 19 Jun 2014 10:50:50 -0400	[thread overview]
Message-ID: <1403189450-18729-105-git-send-email-jlayton@primarydata.com> (raw)
In-Reply-To: <1403189450-18729-1-git-send-email-jlayton@primarydata.com>

Signed-off-by: Jeff Layton <jlayton@primarydata.com>
---
 .../filesystems/nfs/nfsd4-state-objects.txt        | 110 +++++++++++++++++++++
 1 file changed, 110 insertions(+)
 create mode 100644 Documentation/filesystems/nfs/nfsd4-state-objects.txt

diff --git a/Documentation/filesystems/nfs/nfsd4-state-objects.txt b/Documentation/filesystems/nfs/nfsd4-state-objects.txt
new file mode 100644
index 000000000000..fa3f70608422
--- /dev/null
+++ b/Documentation/filesystems/nfs/nfsd4-state-objects.txt
@@ -0,0 +1,110 @@
+KNFSD4 State Object Model:
+==========================
+Written 2014 by Jeff Layton <jlayton@primarydata.com>
+
+Introduction:
+-------------
+Until recently, knfsd relied heavily on a global mutex to ensure that objects
+didn't disappear while they were being operated on. That has proven to be a
+scalability bottleneck however, so the code has been overhauled to make heavy
+use of reference counting and spinlocks for tracking the different objects.
+
+The state model in NFSv4 is quite complex. Thus, the relationship between the
+objects to track this state in knfsd is also complex. This document attempts to
+lay out how all of the different objects relate to one another, and which ones
+hold references to others.
+
+In addition to the "persistent" references documented here, references to these
+objects can also be taken during the processing of compounds in order to ensure
+that the objects don't disappear suddenly.
+
+struct nfsd_net:
+----------------
+Represents a nfsd "container". With respect to nfsv4 state tracking, the fields
+of interest are the *_id_hashtbls and the *_name_tree. These track the
+nfs4_client objects by either short or long form clientid.
+
+Each nfsd_net runs a nfs4_laundromat workqueue job every lease period to clean
+up expired clients and delegations within the container.
+
+struct nfs4_file:
+-----------------
+These objects are global. nfsd only keeps one instance of a nfs4_file per inode
+(though it may keep multiple file descriptors open per inode). These are
+tracked in the file_hashtbl which is protected by the state_lock spinlock.
+
+struct nfs4_client:
+-------------------
+The initial object created by an NFS client using SETCLIENTID (for NFSv4.0) or
+EXCHANGE_ID (for NFSv4.1+). These objects are refcounted and timestamped. Each
+nfsd_net_ns object contains a set of these and they are tracked via short and
+long form clientid. They are hashed and searched for under the per-nfsd-net
+client_lock spinlock.
+
+The lifecycle of these is a little strange. References to it are only held
+during the processing of compounds, and in certain other operations. In their
+"resting state" they have a refcount of 0. If they are not renewed within a
+lease period, they become eligible for destruction by the laundromat.
+
+These objects can also be destroyed prematurely by the fault injection code,
+or if the client sends certain forms of SETCLIENTID or EXCHANGE_ID updates.
+Care is taken *not* to do this however when the objects have an elevated
+refcount.
+
+struct nfsd4_session:
+---------------------
+Represents a v4.1+ session. These are refcounted in a similar fashion to
+the nfs4_client. References are only taken when the server is actively working
+on the object (primarily during the processing of compounds).
+
+struct nfs4_stateowner:
+-----------------------
+A core object that represents either an open or lock owner. The object and lock
+owner objects have one of these embedded within them. Refcounts and other
+fields common to both owner types are contained within these structures.
+
+struct nfs4_openowner:
+----------------------
+When a file is opened, the client provides an open state owner opaque string
+that indicates the "owner" of that open. These objects are refcounted.
+References to it are held by each open state associated with it. This object is
+a superset of the nfs4_stateowner struct.
+
+struct nfs4_lockowner:
+----------------------
+Represents a generic "lockowner". Similar to an openowner. References to it are
+held by the lock stateids that are created on its behalf. This object is a
+superset of the nfs4_stateowner struct.
+
+strict nfs4_stid:
+-----------------
+A core object that represents a "generic" stateid. These are generally embedded
+within the different (more specific) stateid objects and contain fields that
+are of general use to any stateid.
+
+struct nfs4_ol_stateid:
+-----------------------
+A generic struct representing either a open or lock stateid. The nfs4_client
+holds a reference to each of these objects, and they in turn hold a reference
+to their respective stateowners. The client's reference is released in response
+to a close or unlock (depending on whether it's an open or lock stateid) or
+when the client is being destroyed.
+
+In the case of v4.0, these objects are preserved for a little while after close
+in order to handle CLOSE replays. They are eventually reclaimed via a LRU
+scheme by the laundromat.
+
+This object is a superset of the nfs4_stid.
+
+struct nfs4_delegation:
+-----------------------
+Represents a delegation stateid. The nfs4_client holds references to these and
+they are put when it is being destroyed or when the delegation is returned by
+the client.
+
+If the server attempts to recall a delegation and the client doesn't do so
+before a timeout, the server may also revoke the delegation. In that case, the
+object will either be destroyed (v4.0) or moved to a per-client list of revoked
+delegations (v4.1+).
+
+This object is a superset of the nfs4_stid.
-- 
1.9.3


  parent reply	other threads:[~2014-06-19 14:53 UTC|newest]

Thread overview: 144+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-06-19 14:49 [PATCH v1 000/104] nfsd: eliminate the client_mutex Jeff Layton
2014-06-19 14:49 ` [PATCH v1 001/104] nfsd: Protect addition to the file_hashtbl Jeff Layton
2014-06-19 14:49 ` [PATCH v1 002/104] NFSd: Avoid taking state_lock while holding inode lock in nfsd_break_one_deleg Jeff Layton
2014-06-23 13:54   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 003/104] NFSd: nfs4_preprocess_seqid_op should only set *stpp on success Jeff Layton
2014-06-19 14:49 ` [PATCH v1 004/104] nfsd4: use cl_lock to synchronize all stateid idr calls Jeff Layton
2014-06-23 16:02   ` Christoph Hellwig
2014-06-23 16:19     ` Jeff Layton
2014-06-19 14:49 ` [PATCH v1 005/104] NFSd: Add fine grained protection for the nfs4_file->fi_stateids list Jeff Layton
2014-06-23 16:04   ` Christoph Hellwig
2014-06-23 16:20     ` Jeff Layton
2014-06-23 16:23       ` Christoph Hellwig
2014-06-23 16:24         ` Jeff Layton
2014-06-23 16:32           ` J. Bruce Fields
2014-06-19 14:49 ` [PATCH v1 006/104] NFSd: Add a mutex to protect the NFSv4.0 open owner replay cache Jeff Layton
2014-06-23 16:05   ` Christoph Hellwig
2014-06-23 16:24     ` Jeff Layton
2014-06-19 14:49 ` [PATCH v1 007/104] NFSd: Add locking to the nfs4_file->fi_fds[] array Jeff Layton
2014-06-23 16:06   ` Christoph Hellwig
2014-06-23 17:14     ` Jeff Layton
2014-06-23 17:18       ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 008/104] NFSd: Ensure that nfs4_file_get_access enforces share access modes Jeff Layton
2014-06-23 16:12   ` Christoph Hellwig
2014-06-23 16:40     ` Jeff Layton
2014-06-19 14:49 ` [PATCH v1 009/104] locks: add file_has_lease Jeff Layton
2014-06-23 16:12   ` Christoph Hellwig
2014-06-23 16:48     ` Jeff Layton
2014-06-19 14:49 ` [PATCH v1 010/104] NFSd: Protect the nfs4_file delegation fields using the fi_lock Jeff Layton
2014-06-19 14:49 ` [PATCH v1 011/104] NFSd: Lock owners are not per open stateid Jeff Layton
2014-06-24 11:50   ` Christoph Hellwig
2014-06-24 12:04     ` Jeff Layton
2014-06-24 12:06       ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 012/104] NFSd: Clean up helper __release_lock_stateid Jeff Layton
2014-06-24 11:53   ` Christoph Hellwig
2014-06-24 12:00     ` Jeff Layton
2014-06-19 14:49 ` [PATCH v1 013/104] NFSd: Allow lockowners to hold several stateids Jeff Layton
2014-06-24 11:56   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 014/104] NFSd: NFSv4 lock-owners are not associated to a specific file Jeff Layton
2014-06-24 11:59   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 015/104] NFSd: Cleanup nfs4svc_encode_compoundres Jeff Layton
2014-06-24 12:00   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 016/104] NFSd: Don't get a session reference without a client reference Jeff Layton
2014-06-24 12:02   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 017/104] NFSd: Allow struct nfsd4_compound_state to cache the nfs4_client Jeff Layton
2014-06-24 12:02   ` Christoph Hellwig
2014-06-19 14:49 ` [PATCH v1 018/104] NFSd: Move the delegation reference counter into the struct nfs4_stid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 019/104] NFSd: Simplify stateid management Jeff Layton
2014-06-19 14:49 ` [PATCH v1 020/104] NFSd: Fix delegation revocation Jeff Layton
2014-06-19 14:49 ` [PATCH v1 021/104] NFSd: Add reference counting to the lock and open stateids Jeff Layton
2014-06-19 14:49 ` [PATCH v1 022/104] NFSd: clean up nfsd4_close_open_stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 023/104] NFSd: Add a struct nfs4_file field to struct nfs4_stid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 024/104] NFSd: Replace nfs4_ol_stateid->st_file with the st_stid.sc_file Jeff Layton
2014-06-19 14:49 ` [PATCH v1 025/104] NFSd: Ensure stateids remain unique until they are freed Jeff Layton
2014-06-19 14:49 ` [PATCH v1 026/104] NFSd: Ensure atomicity of stateid destruction and idr tree removal Jeff Layton
2014-06-19 14:49 ` [PATCH v1 027/104] NFSd: Cleanup the freeing of stateids Jeff Layton
2014-06-19 14:49 ` [PATCH v1 028/104] nfsd: do filp_close in sc_free callback for lock stateids Jeff Layton
2014-06-19 14:49 ` [PATCH v1 029/104] NFSd: Add locking to protect the state owner lists Jeff Layton
2014-06-19 14:49 ` [PATCH v1 030/104] nfsd: clean up races in lock stateid searching and creation Jeff Layton
2014-06-19 14:49 ` [PATCH v1 031/104] NFSd: Convert delegation counter to an atomic_long_t type Jeff Layton
2014-06-19 14:49 ` [PATCH v1 032/104] NFSd: Slight cleanup of find_stateid() Jeff Layton
2014-06-19 14:49 ` [PATCH v1 033/104] NFSd: Add reference counting to find_stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 034/104] NFSd: Add reference counting to lock stateids Jeff Layton
2014-06-19 14:49 ` [PATCH v1 035/104] NFSd: nfsd4_locku() must reference the lock stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 036/104] NFSd: Ensure that nfs4_open_delegation() references the delegation stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 037/104] NFSd: nfsd4_process_open2() must reference " Jeff Layton
2014-06-19 14:49 ` [PATCH v1 038/104] NFSd: nfsd4_process_open2() must reference the open stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 039/104] NFSd: Prepare nfsd4_close() for open stateid referencing Jeff Layton
2014-06-19 14:49 ` [PATCH v1 040/104] NFSd: nfsd4_open_confirm() must reference the open stateid Jeff Layton
2014-06-19 14:49 ` [PATCH v1 041/104] NFSd: Add reference counting to nfs4_preprocess_confirmed_seqid_op Jeff Layton
2014-06-19 14:49 ` [PATCH v1 042/104] NFSd: Migrate the stateid reference into nfs4_preprocess_seqid_op Jeff Layton
2014-06-19 14:49 ` [PATCH v1 043/104] NFSd: Migrate the stateid reference into nfs4_lookup_stateid() Jeff Layton
2014-06-19 14:49 ` [PATCH v1 044/104] NFSd: Migrate the stateid reference into nfs4_find_stateid_by_type() Jeff Layton
2014-06-19 14:49 ` [PATCH v1 045/104] NFSd: Add reference counting to state owners Jeff Layton
2014-06-19 14:49 ` [PATCH v1 046/104] NFSd: Keep a reference to the open stateid for the NFSv4.0 replay cache Jeff Layton
2014-06-19 14:49 ` [PATCH v1 047/104] nfsd: clean up lockowner refcounting when finding them Jeff Layton
2014-06-19 14:49 ` [PATCH v1 048/104] nfsd: add an operation for unhashing a stateowner Jeff Layton
2014-06-19 14:49 ` [PATCH v1 049/104] NFSd: Make lock stateid take a reference to the lockowner Jeff Layton
2014-06-19 14:49 ` [PATCH v1 050/104] nfsd: clean up nfs4_release_lockowner Jeff Layton
2014-06-19 14:49 ` [PATCH v1 051/104] nfsd: clean up refcounting for lockowners Jeff Layton
2014-06-19 14:49 ` [PATCH v1 052/104] nfsd: declare v4.1+ openowners confirmed on creation Jeff Layton
2014-06-19 14:49 ` [PATCH v1 053/104] nfsd: make openstateids hold references to their openowners Jeff Layton
2014-06-19 14:50 ` [PATCH v1 054/104] nfsd: don't allow CLOSE to proceed until refcount on stateid drops Jeff Layton
2014-06-19 14:50 ` [PATCH v1 055/104] NFSd: Protect adding/removing open state owners using client_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 056/104] NFSd: Protect adding/removing lock " Jeff Layton
2014-06-19 14:50 ` [PATCH v1 057/104] NFSd: Cleanup - Let nfsd4_lookup_stateid() take a cstate argument Jeff Layton
2014-06-19 14:50 ` [PATCH v1 058/104] NFSd: Cache the client that was looked up in lookup_clientid() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 059/104] NFSd: Convert nfsd4_process_open1() to work with lookup_clientid() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 060/104] NFSd: Always use lookup_clientid() in nfsd4_process_open1 Jeff Layton
2014-06-19 14:50 ` [PATCH v1 061/104] NFSd: Move the open owner hash table into struct nfs4_client Jeff Layton
2014-06-19 14:50 ` [PATCH v1 062/104] lockdep: add lockdep_assert_not_held Jeff Layton
2014-06-19 14:50 ` [PATCH v1 063/104] nfsd: add locking to stateowner release Jeff Layton
2014-06-19 14:50 ` [PATCH v1 064/104] nfsd: optimize destroy_lockowner cl_lock thrashing Jeff Layton
2014-06-19 14:50 ` [PATCH v1 065/104] nfsd: reduce cl_lock trashing in release_openowner Jeff Layton
2014-06-19 14:50 ` [PATCH v1 066/104] NFSd: Convert nfs4_check_open_reclaim() to work with lookup_clientid() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 067/104] NFSd: Ensure struct nfs4_client is unhashed before we try to destroy it Jeff Layton
2014-06-19 14:50 ` [PATCH v1 068/104] NFSd: Ensure that the laundromat unhashes the client before releasing locks Jeff Layton
2014-06-19 14:50 ` [PATCH v1 069/104] NFSd: Don't require client_lock in free_client Jeff Layton
2014-06-19 14:50 ` [PATCH v1 070/104] NFSd: Move create_client() call outside the lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 071/104] NFSd: Protect unconfirmed client creation using client_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 072/104] NFSd: Protect session creation and client confirm " Jeff Layton
2014-06-19 14:50 ` [PATCH v1 073/104] NFSd: Protect nfsd4_destroy_clientid " Jeff Layton
2014-06-19 14:50 ` [PATCH v1 074/104] NFSd: Ensure lookup_clientid() takes client_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 075/104] NFSd: Add assertions to document the nfs4_client/session locking Jeff Layton
2014-06-19 14:50 ` [PATCH v1 076/104] nfsd: protect the close_lru list and oo_last_closed_stid with client_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 077/104] nfsd: ensure that clp->cl_revoked list is protected by clp->cl_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 078/104] nfsd: move unhash_client_locked call into mark_client_expired_locked Jeff Layton
2014-06-19 14:50 ` [PATCH v1 079/104] nfsd: fix misleading comment Jeff Layton
2014-06-19 14:50 ` [PATCH v1 080/104] nfsd: don't destroy client if mark_client_expired_locked fails Jeff Layton
2014-06-19 14:50 ` [PATCH v1 081/104] nfsd: don't destroy clients that are busy Jeff Layton
2014-06-19 14:50 ` [PATCH v1 082/104] nfsd: abstract out the get and set routines into the fault injection ops Jeff Layton
2014-06-19 14:50 ` [PATCH v1 083/104] nfsd: add a forget_clients "get" routine with proper locking Jeff Layton
2014-06-19 14:50 ` [PATCH v1 084/104] nfsd: add a forget_client set_clnt routine Jeff Layton
2014-06-19 14:50 ` [PATCH v1 085/104] nfsd: add nfsd_inject_forget_clients Jeff Layton
2014-06-19 14:50 ` [PATCH v1 086/104] nfsd: add a list_head arg to nfsd_foreach_client_lock Jeff Layton
2014-06-19 14:50 ` [PATCH v1 087/104] nfsd: add more granular locking to forget_locks fault injector Jeff Layton
2014-06-19 14:50 ` [PATCH v1 088/104] nfsd: add more granular locking to forget_openowners " Jeff Layton
2014-06-19 14:50 ` [PATCH v1 089/104] nfsd: add more granular locking to *_delegations fault injectors Jeff Layton
2014-06-19 14:50 ` [PATCH v1 090/104] nfsd: remove old fault injection infrastructure Jeff Layton
2014-06-19 14:50 ` [PATCH v1 091/104] NFSd: Remove nfs4_lock_state(): nfs4_preprocess_stateid_op() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 092/104] NFSd: Remove nfs4_lock_state(): nfsd4_test_stateid/nfsd4_free_stateid Jeff Layton
2014-06-19 14:50 ` [PATCH v1 093/104] NFSd: Remove nfs4_lock_state(): nfsd4_release_lockowner Jeff Layton
2014-06-19 14:50 ` [PATCH v1 094/104] NFSd: Remove nfs4_lock_state(): nfsd4_lock/locku/lockt() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 095/104] NFSd: Remove nfs4_lock_state(): nfsd4_open_downgrade + nfsd4_close Jeff Layton
2014-06-19 14:50 ` [PATCH v1 096/104] NFSd: Remove nfs4_lock_state(): nfsd4_delegreturn() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 097/104] NFSd: Remove nfs4_lock_state(): nfsd4_open and nfsd4_open_confirm Jeff Layton
2014-06-19 14:50 ` [PATCH v1 098/104] NFSd: Remove nfs4_lock_state(): exchange_id, create/destroy_session() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 099/104] NFSd: Remove nfs4_lock_state(): setclientid, setclientid_confirm, renew Jeff Layton
2014-06-19 14:50 ` [PATCH v1 100/104] NFSd: Remove nfs4_lock_state(): reclaim_complete() Jeff Layton
2014-06-19 14:50 ` [PATCH v1 101/104] nfsd: remove nfs4_lock_state: nfs4_laundromat Jeff Layton
2014-06-19 14:50 ` [PATCH v1 102/104] nfsd: remove nfs4_lock_state: nfs4_state_shutdown_net Jeff Layton
2014-06-19 14:50 ` [PATCH v1 103/104] nfsd: remove the client_mutex and the nfs4_lock/unlock_state wrappers Jeff Layton
2014-06-19 14:50 ` Jeff Layton [this message]
2014-06-24 12:11   ` [PATCH v1 104/104] nfsd: add file documenting new state object model Christoph Hellwig
2014-06-24 12:23     ` Jeff Layton
2014-06-19 15:33 ` [PATCH v1 000/104] nfsd: eliminate the client_mutex Jeff Layton
2014-06-26  1:26   ` J. Bruce Fields
2014-06-26  1:34     ` Jeff Layton
2014-06-23 13:39 ` Christoph Hellwig
2014-06-23 13:56   ` Jeff Layton
2014-06-23 14:11     ` J. Bruce Fields
2014-06-23 16:00     ` Christoph Hellwig
2014-06-23 16:10       ` J. Bruce Fields
2014-06-23 16:13         ` Christoph Hellwig
2014-06-23 18:26         ` Jeff Layton

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1403189450-18729-105-git-send-email-jlayton@primarydata.com \
    --to=jlayton@primarydata.com \
    --cc=bfields@fieldses.org \
    --cc=linux-nfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.