* [PATCH 0/4] NFSD: Add support for session dynamic slot management
@ 2012-11-29 3:17 Trond Myklebust
2012-11-29 3:17 ` [PATCH 1/4] NFSD: Convert the slot table to use a linked list Trond Myklebust
0 siblings, 1 reply; 8+ messages in thread
From: Trond Myklebust @ 2012-11-29 3:17 UTC (permalink / raw)
To: bfields; +Cc: linux-nfs
This patch series adds support for session dynamic slot management
in accordance with Section 2.10.6.1 of RFC5661.
It implements a very simple policy that is controlled by the function
nfsd4_sequence_adjust_slot_table(): If the client is matching the server
target_highest_slotid, then we increase the window size by 1/4. Otherwise
we try to shrink the window.
The window size is bounded from below by the negotiated session table size
for backward compatibility with clients that do not support dynamic slot
management.
Trond Myklebust (4):
NFSD: Convert the slot table to use a linked list
NFSD: Make DRC memory limits work with dynamic slot allocation
NFSD: Add slot table support for target_highest_slotid and
highest_slotid
NFSD: Add support for dynamic slot changes
fs/nfsd/nfs4state.c | 279 ++++++++++++++++++++++++++++++++++++++++------------
fs/nfsd/nfs4xdr.c | 7 +-
fs/nfsd/nfsd.h | 4 +-
fs/nfsd/nfssvc.c | 4 +-
fs/nfsd/state.h | 15 ++-
fs/nfsd/xdr4.h | 6 +-
6 files changed, 240 insertions(+), 75 deletions(-)
--
1.7.11.7
^ permalink raw reply [flat|nested] 8+ messages in thread* [PATCH 1/4] NFSD: Convert the slot table to use a linked list 2012-11-29 3:17 [PATCH 0/4] NFSD: Add support for session dynamic slot management Trond Myklebust @ 2012-11-29 3:17 ` Trond Myklebust 2012-11-29 3:17 ` [PATCH 2/4] NFSD: Make DRC memory limits work with dynamic slot allocation Trond Myklebust 0 siblings, 1 reply; 8+ messages in thread From: Trond Myklebust @ 2012-11-29 3:17 UTC (permalink / raw) To: bfields; +Cc: linux-nfs In preparation for the dynamic slot table allocation, convert the nfsd slot table to a linked list model. Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com> --- fs/nfsd/nfs4state.c | 151 +++++++++++++++++++++++++++++++++++++++++----------- fs/nfsd/state.h | 10 +++- 2 files changed, 129 insertions(+), 32 deletions(-) diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index d0237f8..5717ea1 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -704,15 +704,6 @@ gen_sessionid(struct nfsd4_session *ses) */ #define NFSD_MIN_HDR_SEQ_SZ (24 + 12 + 44) -static void -free_session_slots(struct nfsd4_session *ses) -{ - int i; - - for (i = 0; i < ses->se_fchannel.maxreqs; i++) - kfree(ses->se_slots[i]); -} - /* * We don't actually need to cache the rpc and session headers, so we * can allocate a little less for each slot: @@ -758,29 +749,127 @@ static void nfsd4_put_drc_mem(int slotsize, int num) spin_unlock(&nfsd_drc_lock); } -static struct nfsd4_session *__alloc_session(int slotsize, int numslots) +static void nfsd4_free_slot(struct nfsd4_slot *slot) { - struct nfsd4_session *new; - int mem, i; + kfree(slot); +} + +static struct nfsd4_slot *nfsd4_new_slot(size_t slotsize, gfp_t gfp_mask) +{ + return kzalloc(sizeof(struct nfsd4_slot) + slotsize, gfp_mask); +} + +static struct nfsd4_slot *nfsd4_find_slot_locked(const struct nfsd4_slot_table *tbl, + u32 slotid) +{ + struct nfsd4_slot *slot; + + list_for_each_entry(slot, &tbl->slt_head, sl_list) { + if (slot->sl_slotid == slotid) + return slot; + } + return NULL; +} + +static struct nfsd4_slot *nfsd4_find_slot(struct nfsd4_slot_table *tbl, + u32 slotid) +{ + struct nfsd4_slot *slot; + + spin_lock(&tbl->slt_lock); + slot = nfsd4_find_slot_locked(tbl, slotid); + spin_unlock(&tbl->slt_lock); + return slot; +} + +static struct nfsd4_slot *nfsd4_find_last_slot_locked(const struct nfsd4_slot_table *tbl) +{ + if (list_empty(&tbl->slt_head)) + return NULL; + return list_entry(tbl->slt_head.prev, struct nfsd4_slot, sl_list); +} + +static bool nfsd4_grow_slot_table_locked(struct nfsd4_slot_table *tbl, + u32 slotid, gfp_t gfp_mask) +{ + struct nfsd4_slot *slot = NULL; + struct nfsd4_slot *prev; + u32 next_slotid; + + for (;;) { + prev = nfsd4_find_last_slot_locked(tbl); + if (prev != NULL) { + if (prev->sl_slotid >= slotid) + break; + next_slotid = prev->sl_slotid + 1; + } else + next_slotid = 0; + if (slot != NULL){ + slot->sl_slotid = next_slotid; + list_add_tail(&slot->sl_list, &tbl->slt_head); + } + spin_unlock(&tbl->slt_lock); + slot = nfsd4_new_slot(tbl->slt_slotsize, gfp_mask); + spin_lock(&tbl->slt_lock); + if (slot == NULL) + return false; + } + nfsd4_free_slot(slot); + return true; +} + +static bool nfsd4_grow_slot_table(struct nfsd4_slot_table *tbl, + u32 slotid, gfp_t gfp_mask) +{ + bool ret; + + spin_lock(&tbl->slt_lock); + ret = nfsd4_grow_slot_table_locked(tbl, slotid, gfp_mask); + spin_unlock(&tbl->slt_lock); + return ret; +} + +static void nfsd4_truncate_slot_table_locked(struct nfsd4_slot_table *tbl, + u32 slotid) +{ + struct nfsd4_slot *slot; + + for (;;) { + slot = nfsd4_find_last_slot_locked(tbl); + if (slot == NULL || slot->sl_slotid < slotid) + break; + list_del(&slot->sl_list); + nfsd4_free_slot(slot); + } +} + +static void nfsd4_free_slot_table(struct nfsd4_slot_table *tbl) +{ + spin_lock(&tbl->slt_lock); + nfsd4_truncate_slot_table_locked(tbl, 0); + spin_unlock(&tbl->slt_lock); +} + +static void nfsd4_init_slot_table(struct nfsd4_slot_table *tbl, size_t slotsize) +{ + INIT_LIST_HEAD(&tbl->slt_head); + spin_lock_init(&tbl->slt_lock); + tbl->slt_slotsize = slotsize; +} - BUILD_BUG_ON(NFSD_MAX_SLOTS_PER_SESSION * sizeof(struct nfsd4_slot *) - + sizeof(struct nfsd4_session) > PAGE_SIZE); - mem = numslots * sizeof(struct nfsd4_slot *); +static struct nfsd4_session *__alloc_session(size_t slotsize, u32 highest_slotid) +{ + struct nfsd4_session *new; - new = kzalloc(sizeof(*new) + mem, GFP_KERNEL); + new = kzalloc(sizeof(*new), GFP_KERNEL); if (!new) return NULL; + nfsd4_init_slot_table(&new->se_slots, slotsize); /* allocate each struct nfsd4_slot and data cache in one piece */ - for (i = 0; i < numslots; i++) { - mem = sizeof(struct nfsd4_slot) + slotsize; - new->se_slots[i] = kzalloc(mem, GFP_KERNEL); - if (!new->se_slots[i]) - goto out_free; - } - return new; -out_free: - while (i--) - kfree(new->se_slots[i]); + if (nfsd4_grow_slot_table(&new->se_slots, highest_slotid, GFP_KERNEL)) + return new; + + nfsd4_free_slot_table(&new->se_slots); kfree(new); return NULL; } @@ -899,7 +988,7 @@ static void nfsd4_del_conns(struct nfsd4_session *s) static void __free_session(struct nfsd4_session *ses) { nfsd4_put_drc_mem(slot_bytes(&ses->se_fchannel), ses->se_fchannel.maxreqs); - free_session_slots(ses); + nfsd4_free_slot_table(&ses->se_slots); kfree(ses); } @@ -936,7 +1025,7 @@ static struct nfsd4_session *alloc_session(struct nfsd4_channel_attrs *fchan) if (numslots < 1) return NULL; - new = __alloc_session(slotsize, numslots); + new = __alloc_session(slotsize, numslots - 1); if (!new) { nfsd4_put_drc_mem(slotsize, fchan->maxreqs); return NULL; @@ -2032,11 +2121,11 @@ nfsd4_sequence(struct svc_rqst *rqstp, if (nfsd4_request_too_big(rqstp, session)) goto out; + /* Note: nfsd4_find_slot checks the validity of seq->slotid */ status = nfserr_badslot; - if (seq->slotid >= session->se_fchannel.maxreqs) + slot = nfsd4_find_slot(&session->se_slots, seq->slotid); + if (slot == NULL) goto out; - - slot = session->se_slots[seq->slotid]; dprintk("%s: slotid %d\n", __func__, seq->slotid); /* We do not negotiate the number of slots yet, so set the diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h index e036894..7d319e9a 100644 --- a/fs/nfsd/state.h +++ b/fs/nfsd/state.h @@ -128,6 +128,8 @@ static inline struct nfs4_delegation *delegstateid(struct nfs4_stid *s) (NFSD_CACHE_SIZE_SLOTS_PER_SESSION * NFSD_SLOT_CACHE_SIZE) struct nfsd4_slot { + struct list_head sl_list; + u32 sl_slotid; u32 sl_seqid; __be32 sl_status; u32 sl_datalen; @@ -139,6 +141,12 @@ struct nfsd4_slot { char sl_data[]; }; +struct nfsd4_slot_table { + struct list_head slt_head; + spinlock_t slt_lock; + size_t slt_slotsize; +}; + struct nfsd4_channel_attrs { u32 headerpadsz; u32 maxreq_sz; @@ -195,7 +203,7 @@ struct nfsd4_session { struct list_head se_conns; u32 se_cb_prog; u32 se_cb_seq_nr; - struct nfsd4_slot *se_slots[]; /* forward channel slots */ + struct nfsd4_slot_table se_slots; }; extern void nfsd4_put_session(struct nfsd4_session *ses); -- 1.7.11.7 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 2/4] NFSD: Make DRC memory limits work with dynamic slot allocation 2012-11-29 3:17 ` [PATCH 1/4] NFSD: Convert the slot table to use a linked list Trond Myklebust @ 2012-11-29 3:17 ` Trond Myklebust 2012-11-29 3:17 ` [PATCH 3/4] NFSD: Add slot table support for target_highest_slotid and highest_slotid Trond Myklebust 0 siblings, 1 reply; 8+ messages in thread From: Trond Myklebust @ 2012-11-29 3:17 UTC (permalink / raw) To: bfields; +Cc: linux-nfs When growing the slot table, we always want to ensure that we're not exceeding DRC size limits. Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com> --- fs/nfsd/nfs4state.c | 56 ++++++++++++++++++++++++++--------------------------- fs/nfsd/nfsd.h | 4 ++-- fs/nfsd/nfssvc.c | 4 ++-- fs/nfsd/state.h | 1 + 4 files changed, 33 insertions(+), 32 deletions(-) diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 5717ea1..9cb2450 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -704,15 +704,6 @@ gen_sessionid(struct nfsd4_session *ses) */ #define NFSD_MIN_HDR_SEQ_SZ (24 + 12 + 44) -/* - * We don't actually need to cache the rpc and session headers, so we - * can allocate a little less for each slot: - */ -static inline int slot_bytes(struct nfsd4_channel_attrs *ca) -{ - return ca->maxresp_cached - NFSD_MIN_HDR_SEQ_SZ; -} - static int nfsd4_sanitize_slot_size(u32 size) { size -= NFSD_MIN_HDR_SEQ_SZ; /* We don't cache the rpc header */ @@ -726,27 +717,33 @@ static int nfsd4_sanitize_slot_size(u32 size) * re-negotiate active sessions and reduce their slot usage to make * room for new connections. For now we just fail the create session. */ -static int nfsd4_get_drc_mem(int slotsize, u32 num) +static bool nfsd4_get_drc_mem(struct nfsd4_slot_table *tbl) { - int avail; - - num = min_t(u32, num, NFSD_MAX_SLOTS_PER_SESSION); + unsigned long new_memused; + unsigned long avail; + bool ret = false; + new_memused = tbl->slt_memused + tbl->slt_slotsize; + if (new_memused > NFSD_MAX_MEM_PER_SESSION) + goto out; spin_lock(&nfsd_drc_lock); - avail = min_t(int, NFSD_MAX_MEM_PER_SESSION, - nfsd_drc_max_mem - nfsd_drc_mem_used); - num = min_t(int, num, avail / slotsize); - nfsd_drc_mem_used += num * slotsize; + avail = nfsd_drc_max_mem - nfsd_drc_mem_used; + if (avail >= tbl->slt_slotsize) { + nfsd_drc_mem_used += tbl->slt_slotsize; + tbl->slt_memused = new_memused; + ret = true; + } spin_unlock(&nfsd_drc_lock); - - return num; +out: + return ret; } -static void nfsd4_put_drc_mem(int slotsize, int num) +static void nfsd4_put_drc_mem(struct nfsd4_slot_table *tbl) { spin_lock(&nfsd_drc_lock); - nfsd_drc_mem_used -= slotsize * num; + nfsd_drc_mem_used -= tbl->slt_slotsize; spin_unlock(&nfsd_drc_lock); + tbl->slt_memused -= tbl->slt_slotsize; } static void nfsd4_free_slot(struct nfsd4_slot *slot) @@ -795,16 +792,21 @@ static bool nfsd4_grow_slot_table_locked(struct nfsd4_slot_table *tbl, struct nfsd4_slot *slot = NULL; struct nfsd4_slot *prev; u32 next_slotid; + bool ret; for (;;) { prev = nfsd4_find_last_slot_locked(tbl); if (prev != NULL) { + ret = true; if (prev->sl_slotid >= slotid) break; next_slotid = prev->sl_slotid + 1; } else next_slotid = 0; + ret = false; if (slot != NULL){ + if (!nfsd4_get_drc_mem(tbl)) + break; slot->sl_slotid = next_slotid; list_add_tail(&slot->sl_list, &tbl->slt_head); } @@ -812,10 +814,10 @@ static bool nfsd4_grow_slot_table_locked(struct nfsd4_slot_table *tbl, slot = nfsd4_new_slot(tbl->slt_slotsize, gfp_mask); spin_lock(&tbl->slt_lock); if (slot == NULL) - return false; + break; } nfsd4_free_slot(slot); - return true; + return ret; } static bool nfsd4_grow_slot_table(struct nfsd4_slot_table *tbl, @@ -840,6 +842,7 @@ static void nfsd4_truncate_slot_table_locked(struct nfsd4_slot_table *tbl, break; list_del(&slot->sl_list); nfsd4_free_slot(slot); + nfsd4_put_drc_mem(tbl); } } @@ -987,7 +990,6 @@ static void nfsd4_del_conns(struct nfsd4_session *s) static void __free_session(struct nfsd4_session *ses) { - nfsd4_put_drc_mem(slot_bytes(&ses->se_fchannel), ses->se_fchannel.maxreqs); nfsd4_free_slot_table(&ses->se_slots); kfree(ses); } @@ -1021,15 +1023,13 @@ static struct nfsd4_session *alloc_session(struct nfsd4_channel_attrs *fchan) * decrease number of slots instead of their size. */ slotsize = nfsd4_sanitize_slot_size(fchan->maxresp_cached); - numslots = nfsd4_get_drc_mem(slotsize, fchan->maxreqs); + numslots = min_t(u32, fchan->maxreqs, NFSD_MAX_SLOTS_PER_SESSION); if (numslots < 1) return NULL; new = __alloc_session(slotsize, numslots - 1); - if (!new) { - nfsd4_put_drc_mem(slotsize, fchan->maxreqs); + if (!new) return NULL; - } init_forechannel_attrs(&new->se_fchannel, fchan, numslots, slotsize); return new; } diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h index 80d5ce4..322eb93 100644 --- a/fs/nfsd/nfsd.h +++ b/fs/nfsd/nfsd.h @@ -57,8 +57,8 @@ extern u32 nfsd_supported_minorversion; extern struct mutex nfsd_mutex; extern struct svc_serv *nfsd_serv; extern spinlock_t nfsd_drc_lock; -extern unsigned int nfsd_drc_max_mem; -extern unsigned int nfsd_drc_mem_used; +extern unsigned long nfsd_drc_max_mem; +extern unsigned long nfsd_drc_mem_used; extern const struct seq_operations nfs_exports_op; diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index 2013aa00..180d205 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -61,8 +61,8 @@ struct svc_serv *nfsd_serv; * nfsd_drc_pages_used tracks the current version 4.1 DRC memory usage. */ spinlock_t nfsd_drc_lock; -unsigned int nfsd_drc_max_mem; -unsigned int nfsd_drc_mem_used; +unsigned long nfsd_drc_max_mem; +unsigned long nfsd_drc_mem_used; #if defined(CONFIG_NFSD_V2_ACL) || defined(CONFIG_NFSD_V3_ACL) static struct svc_stat nfsd_acl_svcstats; diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h index 7d319e9a..6bcc8a1 100644 --- a/fs/nfsd/state.h +++ b/fs/nfsd/state.h @@ -145,6 +145,7 @@ struct nfsd4_slot_table { struct list_head slt_head; spinlock_t slt_lock; size_t slt_slotsize; + unsigned long slt_memused; }; struct nfsd4_channel_attrs { -- 1.7.11.7 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 3/4] NFSD: Add slot table support for target_highest_slotid and highest_slotid 2012-11-29 3:17 ` [PATCH 2/4] NFSD: Make DRC memory limits work with dynamic slot allocation Trond Myklebust @ 2012-11-29 3:17 ` Trond Myklebust 2012-11-29 3:17 ` [PATCH 4/4] NFSD: Add support for dynamic slot changes Trond Myklebust 0 siblings, 1 reply; 8+ messages in thread From: Trond Myklebust @ 2012-11-29 3:17 UTC (permalink / raw) To: bfields; +Cc: linux-nfs This commit just enables the slot table to set and maintain values for the target_highest_slotid and highest_slotid, as used in the replies to a SEQUENCE request from the client. Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com> --- fs/nfsd/nfs4state.c | 18 +++++++++++------- fs/nfsd/nfs4xdr.c | 7 +++---- fs/nfsd/state.h | 2 ++ fs/nfsd/xdr4.h | 6 ++---- 4 files changed, 18 insertions(+), 15 deletions(-) diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 9cb2450..a1208cc 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -853,11 +853,14 @@ static void nfsd4_free_slot_table(struct nfsd4_slot_table *tbl) spin_unlock(&tbl->slt_lock); } -static void nfsd4_init_slot_table(struct nfsd4_slot_table *tbl, size_t slotsize) +static void nfsd4_init_slot_table(struct nfsd4_slot_table *tbl, + size_t slotsize, u32 highest_slotid) { INIT_LIST_HEAD(&tbl->slt_head); spin_lock_init(&tbl->slt_lock); tbl->slt_slotsize = slotsize; + tbl->slt_highest_slotid = highest_slotid; + tbl->slt_target_highest_slotid = highest_slotid; } static struct nfsd4_session *__alloc_session(size_t slotsize, u32 highest_slotid) @@ -867,7 +870,7 @@ static struct nfsd4_session *__alloc_session(size_t slotsize, u32 highest_slotid new = kzalloc(sizeof(*new), GFP_KERNEL); if (!new) return NULL; - nfsd4_init_slot_table(&new->se_slots, slotsize); + nfsd4_init_slot_table(&new->se_slots, slotsize, highest_slotid); /* allocate each struct nfsd4_slot and data cache in one piece */ if (nfsd4_grow_slot_table(&new->se_slots, highest_slotid, GFP_KERNEL)) return new; @@ -2128,11 +2131,6 @@ nfsd4_sequence(struct svc_rqst *rqstp, goto out; dprintk("%s: slotid %d\n", __func__, seq->slotid); - /* We do not negotiate the number of slots yet, so set the - * maxslots to the session maxreqs which is used to encode - * sr_highest_slotid and the sr_target_slot id to maxslots */ - seq->maxslots = session->se_fchannel.maxreqs; - status = check_slot_seqid(seq->seqid, slot->sl_seqid, slot->sl_flags & NFSD4_SLOT_INUSE); if (status == nfserr_replay_cache) { @@ -2161,6 +2159,12 @@ nfsd4_sequence(struct svc_rqst *rqstp, else slot->sl_flags &= ~NFSD4_SLOT_CACHETHIS; + /* Retrieve new target/highest slotid values */ + spin_lock(&session->se_slots.slt_lock); + seq->target_highest_slotid = session->se_slots.slt_target_highest_slotid; + seq->highest_slotid = session->se_slots.slt_highest_slotid; + spin_unlock(&session->se_slots.slt_lock); + cstate->slot = slot; cstate->session = session; diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c index fd548d1..52e2923 100644 --- a/fs/nfsd/nfs4xdr.c +++ b/fs/nfsd/nfs4xdr.c @@ -1371,7 +1371,7 @@ nfsd4_decode_sequence(struct nfsd4_compoundargs *argp, COPYMEM(seq->sessionid.data, NFS4_MAX_SESSIONID_LEN); READ32(seq->seqid); READ32(seq->slotid); - READ32(seq->maxslots); + READ32(seq->highest_slotid); READ32(seq->cachethis); DECODE_TAIL; @@ -3415,9 +3415,8 @@ nfsd4_encode_sequence(struct nfsd4_compoundres *resp, __be32 nfserr, WRITEMEM(seq->sessionid.data, NFS4_MAX_SESSIONID_LEN); WRITE32(seq->seqid); WRITE32(seq->slotid); - /* Note slotid's are numbered from zero: */ - WRITE32(seq->maxslots - 1); /* sr_highest_slotid */ - WRITE32(seq->maxslots - 1); /* sr_target_highest_slotid */ + WRITE32(seq->highest_slotid); + WRITE32(seq->target_highest_slotid); WRITE32(seq->status_flags); ADJUST_ARGS(); diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h index 6bcc8a1..7a0d8e4 100644 --- a/fs/nfsd/state.h +++ b/fs/nfsd/state.h @@ -146,6 +146,8 @@ struct nfsd4_slot_table { spinlock_t slt_lock; size_t slt_slotsize; unsigned long slt_memused; + u32 slt_target_highest_slotid; + u32 slt_highest_slotid; }; struct nfsd4_channel_attrs { diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h index acd127d..0c5c704 100644 --- a/fs/nfsd/xdr4.h +++ b/fs/nfsd/xdr4.h @@ -405,11 +405,9 @@ struct nfsd4_sequence { struct nfs4_sessionid sessionid; /* request/response */ u32 seqid; /* request/response */ u32 slotid; /* request/response */ - u32 maxslots; /* request/response */ + u32 highest_slotid; /* request/response */ u32 cachethis; /* request */ -#if 0 - u32 target_maxslots; /* response */ -#endif /* not yet */ + u32 target_highest_slotid; /* response */ u32 status_flags; /* response */ }; -- 1.7.11.7 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 4/4] NFSD: Add support for dynamic slot changes 2012-11-29 3:17 ` [PATCH 3/4] NFSD: Add slot table support for target_highest_slotid and highest_slotid Trond Myklebust @ 2012-11-29 3:17 ` Trond Myklebust 2012-12-06 21:29 ` J. Bruce Fields 0 siblings, 1 reply; 8+ messages in thread From: Trond Myklebust @ 2012-11-29 3:17 UTC (permalink / raw) To: bfields; +Cc: linux-nfs The algorithm chosen for growing/shrinking the dynamic slot table size is as follows: If the client is sending a sa_highest_slotid value that is greater than or equal to the our server target_highest_slotid, then multiply the value of the latter by 5/4. If, on the other hand, the sa_highest_slotid value is smaller than our target_highest_slotid, then try adjusting the latter to half its current value, or to the value of sa_highest_slotid+1, whichever is larger. Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com> --- fs/nfsd/nfs4state.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++---- fs/nfsd/state.h | 2 ++ 2 files changed, 69 insertions(+), 5 deletions(-) diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index a1208cc..88d0de9 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -807,6 +807,7 @@ static bool nfsd4_grow_slot_table_locked(struct nfsd4_slot_table *tbl, if (slot != NULL){ if (!nfsd4_get_drc_mem(tbl)) break; + slot->sl_generation = tbl->slt_generation - 1; slot->sl_slotid = next_slotid; list_add_tail(&slot->sl_list, &tbl->slt_head); } @@ -853,6 +854,30 @@ static void nfsd4_free_slot_table(struct nfsd4_slot_table *tbl) spin_unlock(&tbl->slt_lock); } +static void nfsd4_fc_set_highest_slotid_locked(struct nfsd4_session *session, + struct nfsd4_slot_table *tbl, + u32 target_highest_slotid, + u32 highest_slotid) +{ + /* Lower limit for target_highest_slotid is negotiated limit */ + target_highest_slotid = max(target_highest_slotid, + session->se_fchannel.maxreqs - 1); + /* Lower limit for highest_slotid is target_highest_slotid */ + highest_slotid = max(highest_slotid, target_highest_slotid); + + if (target_highest_slotid == tbl->slt_target_highest_slotid && + highest_slotid == tbl->slt_highest_slotid) + return; + + if (!nfsd4_grow_slot_table_locked(tbl, highest_slotid, GFP_KERNEL)) + return; + + tbl->slt_target_highest_slotid = target_highest_slotid; + tbl->slt_highest_slotid = highest_slotid; + tbl->slt_generation++; + nfsd4_truncate_slot_table_locked(tbl, tbl->slt_highest_slotid); +} + static void nfsd4_init_slot_table(struct nfsd4_slot_table *tbl, size_t slotsize, u32 highest_slotid) { @@ -2039,6 +2064,45 @@ out: return status; } +static void nfsd4_sequence_adjust_slot_table(struct nfsd4_session *session, + struct nfsd4_slot *slot, u32 sa_highest_slotid, + struct nfsd4_sequence *res) +{ + struct nfsd4_slot_table *tbl = &session->se_slots; + u32 next_target; + u32 next_highest; + + spin_lock(&tbl->slt_lock); + /* + * If this slot hasn't seen our previous values, then don't trust + * that the client has seen them. Don't adjust the slot table yet. + */ + if (slot->sl_generation != tbl->slt_generation) + goto out; + + next_target = tbl->slt_target_highest_slotid; + + /* Is the client bumping against our current window limits? */ + if (sa_highest_slotid >= tbl->slt_target_highest_slotid) { + /* Yes! Grow the window size by 1/4 */ + next_target += 1 + (tbl->slt_target_highest_slotid >> 2); + next_highest = next_target; + } else { + /* No! Try to shrink the window size by 1/2 */ + next_target >>= 1; + if (sa_highest_slotid + 1 > next_target) + next_target = sa_highest_slotid + 1; + next_highest = tbl->slt_target_highest_slotid; + } + nfsd4_fc_set_highest_slotid_locked(session, tbl, + next_target, next_highest); +out: + slot->sl_generation = tbl->slt_generation; + res->highest_slotid = tbl->slt_highest_slotid; + res->target_highest_slotid = tbl->slt_target_highest_slotid; + spin_unlock(&tbl->slt_lock); +} + static struct nfsd4_conn *__nfsd4_find_conn(struct svc_xprt *xpt, struct nfsd4_session *s) { struct nfsd4_conn *c; @@ -2159,11 +2223,9 @@ nfsd4_sequence(struct svc_rqst *rqstp, else slot->sl_flags &= ~NFSD4_SLOT_CACHETHIS; - /* Retrieve new target/highest slotid values */ - spin_lock(&session->se_slots.slt_lock); - seq->target_highest_slotid = session->se_slots.slt_target_highest_slotid; - seq->highest_slotid = session->se_slots.slt_highest_slotid; - spin_unlock(&session->se_slots.slt_lock); + /* Adjust slot table, and retrieve new target/highest slotid values */ + nfsd4_sequence_adjust_slot_table(session, slot, + seq->highest_slotid, seq); cstate->slot = slot; cstate->session = session; diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h index 7a0d8e4..c2f897f 100644 --- a/fs/nfsd/state.h +++ b/fs/nfsd/state.h @@ -129,6 +129,7 @@ static inline struct nfs4_delegation *delegstateid(struct nfs4_stid *s) struct nfsd4_slot { struct list_head sl_list; + unsigned long sl_generation; u32 sl_slotid; u32 sl_seqid; __be32 sl_status; @@ -146,6 +147,7 @@ struct nfsd4_slot_table { spinlock_t slt_lock; size_t slt_slotsize; unsigned long slt_memused; + unsigned long slt_generation; u32 slt_target_highest_slotid; u32 slt_highest_slotid; }; -- 1.7.11.7 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 4/4] NFSD: Add support for dynamic slot changes 2012-11-29 3:17 ` [PATCH 4/4] NFSD: Add support for dynamic slot changes Trond Myklebust @ 2012-12-06 21:29 ` J. Bruce Fields 2012-12-06 23:20 ` Myklebust, Trond 0 siblings, 1 reply; 8+ messages in thread From: J. Bruce Fields @ 2012-12-06 21:29 UTC (permalink / raw) To: Trond Myklebust; +Cc: linux-nfs On Wed, Nov 28, 2012 at 10:17:43PM -0500, Trond Myklebust wrote: > The algorithm chosen for growing/shrinking the dynamic slot table > size is as follows: > > If the client is sending a sa_highest_slotid value that is greater than > or equal to the our server target_highest_slotid, then multiply the > value of the latter by 5/4. > > If, on the other hand, the sa_highest_slotid value is smaller than > our target_highest_slotid, then try adjusting the latter to half its > current value, or to the value of sa_highest_slotid+1, whichever is larger. > > Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com> > --- > fs/nfsd/nfs4state.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++---- > fs/nfsd/state.h | 2 ++ > 2 files changed, 69 insertions(+), 5 deletions(-) > > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c > index a1208cc..88d0de9 100644 > --- a/fs/nfsd/nfs4state.c > +++ b/fs/nfsd/nfs4state.c > @@ -807,6 +807,7 @@ static bool nfsd4_grow_slot_table_locked(struct nfsd4_slot_table *tbl, > if (slot != NULL){ > if (!nfsd4_get_drc_mem(tbl)) > break; > + slot->sl_generation = tbl->slt_generation - 1; > slot->sl_slotid = next_slotid; > list_add_tail(&slot->sl_list, &tbl->slt_head); > } > @@ -853,6 +854,30 @@ static void nfsd4_free_slot_table(struct nfsd4_slot_table *tbl) > spin_unlock(&tbl->slt_lock); > } > > +static void nfsd4_fc_set_highest_slotid_locked(struct nfsd4_session *session, > + struct nfsd4_slot_table *tbl, > + u32 target_highest_slotid, > + u32 highest_slotid) > +{ > + /* Lower limit for target_highest_slotid is negotiated limit */ > + target_highest_slotid = max(target_highest_slotid, > + session->se_fchannel.maxreqs - 1); > + /* Lower limit for highest_slotid is target_highest_slotid */ > + highest_slotid = max(highest_slotid, target_highest_slotid); > + > + if (target_highest_slotid == tbl->slt_target_highest_slotid && > + highest_slotid == tbl->slt_highest_slotid) > + return; > + > + if (!nfsd4_grow_slot_table_locked(tbl, highest_slotid, GFP_KERNEL)) > + return; > + > + tbl->slt_target_highest_slotid = target_highest_slotid; > + tbl->slt_highest_slotid = highest_slotid; > + tbl->slt_generation++; > + nfsd4_truncate_slot_table_locked(tbl, tbl->slt_highest_slotid); > +} > + > static void nfsd4_init_slot_table(struct nfsd4_slot_table *tbl, > size_t slotsize, u32 highest_slotid) > { > @@ -2039,6 +2064,45 @@ out: > return status; > } > > +static void nfsd4_sequence_adjust_slot_table(struct nfsd4_session *session, > + struct nfsd4_slot *slot, u32 sa_highest_slotid, > + struct nfsd4_sequence *res) > +{ > + struct nfsd4_slot_table *tbl = &session->se_slots; > + u32 next_target; > + u32 next_highest; > + > + spin_lock(&tbl->slt_lock); > + /* > + * If this slot hasn't seen our previous values, then don't trust > + * that the client has seen them. Don't adjust the slot table yet. > + */ > + if (slot->sl_generation != tbl->slt_generation) > + goto out; > + > + next_target = tbl->slt_target_highest_slotid; > + > + /* Is the client bumping against our current window limits? */ > + if (sa_highest_slotid >= tbl->slt_target_highest_slotid) { > + /* Yes! Grow the window size by 1/4 */ > + next_target += 1 + (tbl->slt_target_highest_slotid >> 2); > + next_highest = next_target; > + } else { > + /* No! Try to shrink the window size by 1/2 */ > + next_target >>= 1; > + if (sa_highest_slotid + 1 > next_target) > + next_target = sa_highest_slotid + 1; > + next_highest = tbl->slt_target_highest_slotid; > + } > + nfsd4_fc_set_highest_slotid_locked(session, tbl, > + next_target, next_highest); > +out: > + slot->sl_generation = tbl->slt_generation; > + res->highest_slotid = tbl->slt_highest_slotid; > + res->target_highest_slotid = tbl->slt_target_highest_slotid; > + spin_unlock(&tbl->slt_lock); > +} > + > static struct nfsd4_conn *__nfsd4_find_conn(struct svc_xprt *xpt, struct nfsd4_session *s) > { > struct nfsd4_conn *c; > @@ -2159,11 +2223,9 @@ nfsd4_sequence(struct svc_rqst *rqstp, > else > slot->sl_flags &= ~NFSD4_SLOT_CACHETHIS; > > - /* Retrieve new target/highest slotid values */ > - spin_lock(&session->se_slots.slt_lock); > - seq->target_highest_slotid = session->se_slots.slt_target_highest_slotid; > - seq->highest_slotid = session->se_slots.slt_highest_slotid; > - spin_unlock(&session->se_slots.slt_lock); > + /* Adjust slot table, and retrieve new target/highest slotid values */ > + nfsd4_sequence_adjust_slot_table(session, slot, > + seq->highest_slotid, seq); This is allocating under a spin lock (see the client_lock that most of nfsd4_sequence is under.) (But I'm not sure what we should be doing instead.) --b. > > cstate->slot = slot; > cstate->session = session; > diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h > index 7a0d8e4..c2f897f 100644 > --- a/fs/nfsd/state.h > +++ b/fs/nfsd/state.h > @@ -129,6 +129,7 @@ static inline struct nfs4_delegation *delegstateid(struct nfs4_stid *s) > > struct nfsd4_slot { > struct list_head sl_list; > + unsigned long sl_generation; > u32 sl_slotid; > u32 sl_seqid; > __be32 sl_status; > @@ -146,6 +147,7 @@ struct nfsd4_slot_table { > spinlock_t slt_lock; > size_t slt_slotsize; > unsigned long slt_memused; > + unsigned long slt_generation; > u32 slt_target_highest_slotid; > u32 slt_highest_slotid; > }; > -- > 1.7.11.7 > ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [PATCH 4/4] NFSD: Add support for dynamic slot changes 2012-12-06 21:29 ` J. Bruce Fields @ 2012-12-06 23:20 ` Myklebust, Trond 2012-12-07 13:58 ` J. Bruce Fields 0 siblings, 1 reply; 8+ messages in thread From: Myklebust, Trond @ 2012-12-06 23:20 UTC (permalink / raw) To: J. Bruce Fields; +Cc: linux-nfs@vger.kernel.org > -----Original Message----- > From: J. Bruce Fields [mailto:bfields@fieldses.org] > Sent: Thursday, December 06, 2012 10:29 PM > To: Myklebust, Trond > Cc: linux-nfs@vger.kernel.org > Subject: Re: [PATCH 4/4] NFSD: Add support for dynamic slot changes > > On Wed, Nov 28, 2012 at 10:17:43PM -0500, Trond Myklebust wrote: > > The algorithm chosen for growing/shrinking the dynamic slot table size > > is as follows: > > > > If the client is sending a sa_highest_slotid value that is greater > > than or equal to the our server target_highest_slotid, then multiply > > the value of the latter by 5/4. > > > > If, on the other hand, the sa_highest_slotid value is smaller than our > > target_highest_slotid, then try adjusting the latter to half its > > current value, or to the value of sa_highest_slotid+1, whichever is larger. > > > > Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com> > > --- > > fs/nfsd/nfs4state.c | 72 > +++++++++++++++++++++++++++++++++++++++++++++++++---- > > fs/nfsd/state.h | 2 ++ > > 2 files changed, 69 insertions(+), 5 deletions(-) > > > > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index > > a1208cc..88d0de9 100644 > > --- a/fs/nfsd/nfs4state.c > > +++ b/fs/nfsd/nfs4state.c > > @@ -807,6 +807,7 @@ static bool nfsd4_grow_slot_table_locked(struct > nfsd4_slot_table *tbl, > > if (slot != NULL){ > > if (!nfsd4_get_drc_mem(tbl)) > > break; > > + slot->sl_generation = tbl->slt_generation - 1; > > slot->sl_slotid = next_slotid; > > list_add_tail(&slot->sl_list, &tbl->slt_head); > > } > > @@ -853,6 +854,30 @@ static void nfsd4_free_slot_table(struct > nfsd4_slot_table *tbl) > > spin_unlock(&tbl->slt_lock); > > } > > > > +static void nfsd4_fc_set_highest_slotid_locked(struct nfsd4_session > *session, > > + struct nfsd4_slot_table *tbl, > > + u32 target_highest_slotid, > > + u32 highest_slotid) > > +{ > > + /* Lower limit for target_highest_slotid is negotiated limit */ > > + target_highest_slotid = max(target_highest_slotid, > > + session->se_fchannel.maxreqs - 1); > > + /* Lower limit for highest_slotid is target_highest_slotid */ > > + highest_slotid = max(highest_slotid, target_highest_slotid); > > + > > + if (target_highest_slotid == tbl->slt_target_highest_slotid && > > + highest_slotid == tbl->slt_highest_slotid) > > + return; > > + > > + if (!nfsd4_grow_slot_table_locked(tbl, highest_slotid, GFP_KERNEL)) > > + return; > > + > > + tbl->slt_target_highest_slotid = target_highest_slotid; > > + tbl->slt_highest_slotid = highest_slotid; > > + tbl->slt_generation++; > > + nfsd4_truncate_slot_table_locked(tbl, tbl->slt_highest_slotid); } > > + > > static void nfsd4_init_slot_table(struct nfsd4_slot_table *tbl, > > size_t slotsize, u32 highest_slotid) { @@ -2039,6 +2064,45 > @@ out: > > return status; > > } > > > > +static void nfsd4_sequence_adjust_slot_table(struct nfsd4_session > *session, > > + struct nfsd4_slot *slot, u32 sa_highest_slotid, > > + struct nfsd4_sequence *res) > > +{ > > + struct nfsd4_slot_table *tbl = &session->se_slots; > > + u32 next_target; > > + u32 next_highest; > > + > > + spin_lock(&tbl->slt_lock); > > + /* > > + * If this slot hasn't seen our previous values, then don't trust > > + * that the client has seen them. Don't adjust the slot table yet. > > + */ > > + if (slot->sl_generation != tbl->slt_generation) > > + goto out; > > + > > + next_target = tbl->slt_target_highest_slotid; > > + > > + /* Is the client bumping against our current window limits? */ > > + if (sa_highest_slotid >= tbl->slt_target_highest_slotid) { > > + /* Yes! Grow the window size by 1/4 */ > > + next_target += 1 + (tbl->slt_target_highest_slotid >> 2); > > + next_highest = next_target; > > + } else { > > + /* No! Try to shrink the window size by 1/2 */ > > + next_target >>= 1; > > + if (sa_highest_slotid + 1 > next_target) > > + next_target = sa_highest_slotid + 1; > > + next_highest = tbl->slt_target_highest_slotid; > > + } > > + nfsd4_fc_set_highest_slotid_locked(session, tbl, > > + next_target, next_highest); > > +out: > > + slot->sl_generation = tbl->slt_generation; > > + res->highest_slotid = tbl->slt_highest_slotid; > > + res->target_highest_slotid = tbl->slt_target_highest_slotid; > > + spin_unlock(&tbl->slt_lock); > > +} > > + > > static struct nfsd4_conn *__nfsd4_find_conn(struct svc_xprt *xpt, > > struct nfsd4_session *s) { > > struct nfsd4_conn *c; > > @@ -2159,11 +2223,9 @@ nfsd4_sequence(struct svc_rqst *rqstp, > > else > > slot->sl_flags &= ~NFSD4_SLOT_CACHETHIS; > > > > - /* Retrieve new target/highest slotid values */ > > - spin_lock(&session->se_slots.slt_lock); > > - seq->target_highest_slotid = session- > >se_slots.slt_target_highest_slotid; > > - seq->highest_slotid = session->se_slots.slt_highest_slotid; > > - spin_unlock(&session->se_slots.slt_lock); > > + /* Adjust slot table, and retrieve new target/highest slotid values */ > > + nfsd4_sequence_adjust_slot_table(session, slot, > > + seq->highest_slotid, seq); > > This is allocating under a spin lock (see the client_lock that most of > nfsd4_sequence is under.) > > (But I'm not sure what we should be doing instead.) I've already fixed this issue in the latest patches: the right thing is simply to do all this after we drop the client lock. I'll send the updated patches when I get a moment: probably Saturday... Cheers Trond ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 4/4] NFSD: Add support for dynamic slot changes 2012-12-06 23:20 ` Myklebust, Trond @ 2012-12-07 13:58 ` J. Bruce Fields 0 siblings, 0 replies; 8+ messages in thread From: J. Bruce Fields @ 2012-12-07 13:58 UTC (permalink / raw) To: Myklebust, Trond; +Cc: linux-nfs@vger.kernel.org On Thu, Dec 06, 2012 at 11:20:15PM +0000, Myklebust, Trond wrote: > > -----Original Message----- > > From: J. Bruce Fields [mailto:bfields@fieldses.org] > > Sent: Thursday, December 06, 2012 10:29 PM > > To: Myklebust, Trond > > Cc: linux-nfs@vger.kernel.org > > Subject: Re: [PATCH 4/4] NFSD: Add support for dynamic slot changes > > > > On Wed, Nov 28, 2012 at 10:17:43PM -0500, Trond Myklebust wrote: > > > @@ -2159,11 +2223,9 @@ nfsd4_sequence(struct svc_rqst *rqstp, > > > else > > > slot->sl_flags &= ~NFSD4_SLOT_CACHETHIS; > > > > > > - /* Retrieve new target/highest slotid values */ > > > - spin_lock(&session->se_slots.slt_lock); > > > - seq->target_highest_slotid = session- > > >se_slots.slt_target_highest_slotid; > > > - seq->highest_slotid = session->se_slots.slt_highest_slotid; > > > - spin_unlock(&session->se_slots.slt_lock); > > > + /* Adjust slot table, and retrieve new target/highest slotid values */ > > > + nfsd4_sequence_adjust_slot_table(session, slot, > > > + seq->highest_slotid, seq); > > > > This is allocating under a spin lock (see the client_lock that most of > > nfsd4_sequence is under.) > > > > (But I'm not sure what we should be doing instead.) > > I've already fixed this issue in the latest patches: the right thing > is simply to do all this after we drop the client lock. Oh, sounds reasonable. > I'll send the > updated patches when I get a moment: probably Saturday... Thanks! --b. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-12-07 13:58 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-11-29 3:17 [PATCH 0/4] NFSD: Add support for session dynamic slot management Trond Myklebust 2012-11-29 3:17 ` [PATCH 1/4] NFSD: Convert the slot table to use a linked list Trond Myklebust 2012-11-29 3:17 ` [PATCH 2/4] NFSD: Make DRC memory limits work with dynamic slot allocation Trond Myklebust 2012-11-29 3:17 ` [PATCH 3/4] NFSD: Add slot table support for target_highest_slotid and highest_slotid Trond Myklebust 2012-11-29 3:17 ` [PATCH 4/4] NFSD: Add support for dynamic slot changes Trond Myklebust 2012-12-06 21:29 ` J. Bruce Fields 2012-12-06 23:20 ` Myklebust, Trond 2012-12-07 13:58 ` J. Bruce Fields
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).