Re: [PATCH 0/11] nfsd: Summary of "Improve NFS server performance"

["J. Bruce Fields" <bfields@fieldses.org>]

On Wed, Mar 25, 2009 at 07:06:07PM +0530, Krishna Kumar wrote:
> From: Krishna Kumar <krkumar2@in.ibm.com>
> 
> 			Patch summary:
> 			--------------
> Change the caching from ino/dev to a file handle model. Advantages:
> 
> 1. Since file handles are unique, this patch removes all dependencies on the
>    kernel readahead parameters and it's implementation; and instead caches
>    files based on file handles.
> 2. Allows the server to not have to open/close a file multiple times when the
>    client reads it, and results in faster lookup times.
> 3. Readahead is automatically taken care of since the file is not closed while
>    it is getting read (quickly) by the client.
> 4. Another optimization is to avoid getting the cache lock twice for each read
>    operation (after the first time it is taken to update the cache).
> 5. (ra_size, ra_depth retain the "ra" prefix for now, since I have not checked
>    whether changing that will break any user programs)
> 
> 
> 		Patches are described as:
> 		--------------------------
> 	1. nfsd: ADD data structure infrastructure
> 	2. nfsd: ADD new function infrastructure
> 	3. nfsd: CHANGE old function calls to new calls
> 	4. nfsd: ADD client "rm" support
> 	5. nfsd: ADD nfsd_fhcache_shutdown and nfsd_fhcache_init
> 	6. nfsd: CHANGE _ra* calls to _fh* calls in nfssvc
> 	7. nfsd: REMOVE old infrastructure's functions
> 	8. nfsd: REMOVE old infrastructure's data structures
> 
> 
> 			List of changes from Ver1:
> 			--------------------------
> 1. Implement entire logic of either updating-cache or do-nothing or close-file
>    in fh_cache_upd. This simplifies the caller (nfsd_read).
> 2. nfsd_get_fhcache doesn't overwrite existing entries which would require the
>    existing entries to be freed up - that is done by the daemon exclusively.
>    This saves time at lookup by avoiding freeing entries (fh_cache_put).
>    Another change is to optimize the logic of selecting a free slot.
> 3. Due to #2, fh_cache_upd doesn't have to test whether the entry is already
>    on the daemon list (it never is, plus list_del_init is changed to list_del).
> 4. As a result of #2, the daemon becomes simpler - there is no race to handle
>    where there is an entry on the list but it has no cached file/dentry, etc.
> 5. Made some comments clearer and easier to understand.
> 6. Jeff: Changed NFSD_CACHE_JIFFIES to use HZ.
> 7. Jeff: Changed nfsd_daemon_list to nfsd_daemon_free_list; and changed ra_init
> 	  and ra_shutdown prefixes.
> 8. Jeff: Split patch into smaller patches. Tested each with successful builds.
> 9. Pending:
> 	- Bruce: But I think I'd prefer some separate operation (probably just
> 	  triggered by a write to a some new file in the nfsd filesystem) that
> 	  told nfsd to release all its references to a given filesystem.  An
> 	  administrator would have to know to do this before unmounting (or
> 	  maybe mount could be patched to do this).
> 
> 
> 			Performance:
> 			-------------
> 
> This patch was tested with clients running 2, 4, 8, 16 --- 256 test processes,
> each doing reads of different files. Each test includes different I/O sizes.
> 31/77 tests got more than 5% improvement; with 5 tests getting more than 10%
> gain (full results at the end of this post).

The numbers look promising, but very noisy.  Maybe an average of a few
tests would give more stable numbers?

--b.

> Please review. Any comments or improvement ideas are appreciated.
> 
> Signed-off-by: Krishna Kumar <krkumar2@in.ibm.com>
> ---
> 		(#Test Processes on Client == #NFSD's on Server)
> -----------------------------------------------------------------------
> #Test Processes Bufsize		Org BW KB/s	New BW KB/s	%
> -----------------------------------------------------------------------
> 2		256		68022.46	71094.64	4.51
> 2		4096		67833.74	70726.38	4.26
> 2		8192		64541.14	69635.93	7.89
> 2		16384		65708.86	68994.88	5.00
> 2		32768		64272.28	68525.36	6.61
> 2		65536		64684.13	69103.28	6.83
> 2		131072		64765.67	68855.57	6.31
> 
> 4		256		60849.48	64702.04	6.33
> 4		4096		60660.67	64309.37	6.01
> 4		8192		60506.00	61142.84	1.05
> 4		16384		60796.86	64069.82	5.38
> 4		32768		60947.07	64648.57	6.07
> 4		65536		60774.45	63735.24	4.87
> 4		131072		61369.66	65261.85	6.34
> 
> 8		256		49239.57	54467.33	10.61
> 8		4096		50650.45	55400.01	9.37
> 8		8192		50661.58	51732.16	2.11
> 8		16384		51114.76	56025.31	9.60
> 8		32768		52367.20	54348.05	3.78
> 8		65536		51000.23	54285.63	6.44
> 8		131072		52996.73	54021.11	1.93
> 
> 16		256		44534.67	45478.60	2.11
> 16		4096		43897.69	46519.89	5.97
> 16		8192		43787.87	44083.61	.67
> 16		16384		43883.62	46726.03	6.47
> 16		32768		44284.96	44035.86	-.56
> 16		65536		43804.33	44865.20	2.42
> 16		131072		44525.30	43752.62	-1.73
> 
> 32		256		40420.30	42575.30	5.33
> 32		4096		39913.14	42279.21	5.92
> 32		8192		40261.19	42399.93	5.31
> 32		16384		38094.95	42645.32	11.94
> 32		32768		40610.27	43015.37	5.92
> 32		65536		41438.05	41794.76	.86
> 32		131072		41869.06	43644.07	4.23
> 
> 48		256		36986.45	40185.34	8.64
> 48		4096		36585.79	38227.38	4.48
> 48		8192		38406.78	38055.91	-.91
> 48		16384		34950.05	36688.86	4.97
> 48		32768		38908.71	37900.33	-2.59
> 48		65536		39364.64	40036.67	1.70
> 48		131072		40391.56	40887.11	1.22
> 
> 64		256		32821.89	34568.06	5.32
> 64		4096		35468.42	35529.29	.17
> 64		8192		34135.44	36462.31	6.81
> 64		16384		31926.51	32694.91	2.40
> 64		32768		35527.69	35234.60	-.82
> 64		65536		36066.08	36359.77	.81
> 64		131072		35969.04	37462.86	4.15
> 
> 96		256		30032.74	29973.45	-.19
> 96		4096		29687.06	30881.64	4.02
> 96		8192		31142.51	32504.66	4.37
> 96		16384		29546.77	30663.39	3.77
> 96		32768		32458.94	32797.36	1.04
> 96		65536		32826.99	33451.66	1.90
> 96		131072		33601.46	34177.39	1.71
> 
> 128		256		28584.59	29092.11	1.77
> 128		4096		29311.11	30097.65	2.68
> 128		8192		31398.87	33154.63	5.59
> 128		16384		28283.58	29071.45	2.78
> 128		32768		32819.93	33654.11	2.54
> 128		65536		32617.13	33778.46	3.56
> 128		131072		32972.71	34160.82	3.60
> 
> 192		256		25245.92	26331.48	4.29
> 192		4096		27368.48	29318.49	7.12
> 192		8192		30173.74	31477.35	4.32
> 192		16384		26388.54	29719.15	12.62
> 192		32768		31840.91	33606.17	5.54
> 192		65536		33374.85	33607.14	.69
> 192		131072		33523.48	32601.93	-2.74
> 
> 256		256		22256.91	21139.79	-5.01
> 256		4096		25192.75	24281.51	-3.61
> 256		8192		26534.95	28100.59	5.90
> 256		16384		24162.85	25607.86	5.98
> 256		32768		29218.38	29417.28	.68
> 256		65536		29609.59	30049.79	1.48
> 256		131072		30014.29	30132.33	.39
> -----------------------------------------------------------------------