[PATCH 00/09] cifs: local caching support using FS-Cache

[PATCH 00/09] cifs: local caching support using FS-Cache

[Suresh Jayaraman]

This patchset is a second try at adding persistent, local caching facility for
CIFS using the FS-Cache interface.

The cache index hierarchy which is mainly used to locate a file object or
discard a certain subset of the files cached, currently has three levels:
	- Server
	- Share 
	- File

The server index object is keyed by IPaddress of the server, socket family
and the port. The superblock index object is keyed by the sharename and the
inode object is keyed by the UniqueId. The cache coherency is ensured by
checking the LastWriteTime, LastChangeTime and end of file (eof) reported by
the server.

Changes since last post:
-------------------------
   - fix a bug during registration with FS-Cache
   - fix a bug while storing pages to the cache. The earlier set needed an rsize
     of 4096 to make caching working properly due to this bug.
   - server index key uses {IPaddress,family,port} tuple instead of servername
   - root dir of the share is validated by UniqueId/IndexNumber. Almost all
     servers seem to provide one of them and it's unique.
   - we now check LastWriteTime, LastChangeTime and eof of server for data
     coherency. CreateTime could be considered once some of related development
     effort advances
   - dropped the patch to guard cifsglob.h against multiple inclusion as it
     has been included in Jeff Layton's tree
   - some cleanups

To try these patches:

   - apply this patchset in order
   - mount the share, for e.g. mount -t cifs //server/share -o user=guest
   - try copying a huge file (say few hundred MBs) from mount point to local
     filesystem (during the first time, the cache will be initialized)
   - when you copy the second time, it should be read from the local cache (you
     could unmount and remount to reduce the page cache impact).


Known issues
--------------
   - when the 'noserverino' mount option is used, the client generates
     UniqueId itself rather than using the UniqueId from the server. As
     we use UniqueId to ensure that the root directory did not change
     under the hood, we won't be able to benefit from the cache.
   - the cache coherency check may not be reliable always as some
     CIFS servers are known not to update mtime until the filehandle is closed.

Suresh Jayaraman (09):
  cifs: add kernel config option for CIFS Client caching support
  cifs: register CIFS for caching
  cifs: define server-level cache index objects and register them with FS-Cache
  cifs: define superblock-level cache index objects and register them
  cifs: define inode-level cache object and register them
  cifs: FS-Cache page management
  cifs: store pages into local cache
  cifs: read pages from FS-Cache
  cifs: add mount option to enable local caching


 Kconfig      |    9 +
 Makefile     |    2 
 cache.c      |  331 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 cifs_fs_sb.h |    1 
 cifsfs.c     |   15 ++
 cifsglob.h   |   10 +
 connect.c    |   16 ++
 file.c       |   50 ++++++++
 fscache.c    |  236 ++++++++++++++++++++++++++++++++++++++++++
 fscache.h    |  136 ++++++++++++++++++++++++
 inode.c      |    7 +
 11 files changed, 813 insertions(+)

[PATCH 01/09] cifs: add kernel config option for CIFS Client caching support

[Suresh Jayaraman]

Add a kernel config option to enable local caching for CIFS.

Signed-off-by: Suresh Jayaraman <sjayaraman@suse.de>
---
 fs/cifs/Kconfig |    9 +++++++++
 1 file changed, 9 insertions(+)

Index: cifs-2.6/fs/cifs/Kconfig
===================================================================
--- cifs-2.6.orig/fs/cifs/Kconfig
+++ cifs-2.6/fs/cifs/Kconfig
@@ -131,6 +131,15 @@ config CIFS_DFS_UPCALL
 	    IP addresses) which is needed for implicit mounts of DFS junction
 	    points. If unsure, say N.
 
+config CIFS_FSCACHE
+	  bool "Provide CIFS client caching support (EXPERIMENTAL)"
+	  depends on EXPERIMENTAL
+	  depends on CIFS=m && FSCACHE || CIFS=y && FSCACHE=y
+	  help
+	    Makes CIFS FS-Cache capable. Say Y here if you want your CIFS data
+	    to be cached locally on disk through the general filesystem cache
+	    manager. If unsure, say N.
+
 config CIFS_EXPERIMENTAL
 	  bool "CIFS Experimental Features (EXPERIMENTAL)"
 	  depends on CIFS && EXPERIMENTAL

[PATCH 02/09] cifs: register CIFS for caching

[Suresh Jayaraman]

Define CIFS for FS-Cache and register for caching. Upon registration the
top-level index object cookie will be stuck to the netfs definition by
FS-Cache.

Signed-off-by: Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org>
---
 fs/cifs/Makefile  |    2 ++
 fs/cifs/cache.c   |   46 ++++++++++++++++++++++++++++++++++++++++++++++
 fs/cifs/cifsfs.c  |    8 ++++++++
 fs/cifs/fscache.h |   39 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 95 insertions(+)
 create mode 100644 fs/cifs/cache.c
 create mode 100644 fs/cifs/fscache.h

Index: cifs-2.6/fs/cifs/Makefile
===================================================================
--- cifs-2.6.orig/fs/cifs/Makefile
+++ cifs-2.6/fs/cifs/Makefile
@@ -11,3 +11,5 @@ cifs-y := cifsfs.o cifssmb.o cifs_debug.
 cifs-$(CONFIG_CIFS_UPCALL) += cifs_spnego.o
 
 cifs-$(CONFIG_CIFS_DFS_UPCALL) += dns_resolve.o cifs_dfs_ref.o
+
+cifs-$(CONFIG_CIFS_FSCACHE) += cache.o
Index: cifs-2.6/fs/cifs/cache.c
===================================================================
--- /dev/null
+++ cifs-2.6/fs/cifs/cache.c
@@ -0,0 +1,46 @@
+/*
+ *   fs/cifs/cache.c - CIFS filesystem cache index structure definitions
+ *
+ *   Copyright (c) 2010 Novell, Inc.
+ *   Authors(s): Suresh Jayaraman (sjayaraman-l3A5Bk7waGM@public.gmane.org>
+ *
+ *   This library is free software; you can redistribute it and/or modify
+ *   it under the terms of the GNU Lesser General Public License as published
+ *   by the Free Software Foundation; either version 2.1 of the License, or
+ *   (at your option) any later version.
+ *
+ *   This library is distributed in the hope that it will be useful,
+ *   but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+ *   the GNU Lesser General Public License for more details.
+ *
+ *   You should have received a copy of the GNU Lesser General Public License
+ *   along with this library; if not, write to the Free Software
+ *   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+#include "fscache.h"
+
+/*
+ * CIFS filesystem definition for FS-Cache
+ */
+struct fscache_netfs cifs_fscache_netfs = {
+	.name = "cifs",
+	.version = 0,
+};
+
+/*
+ * Register CIFS for caching with FS-Cache
+ */
+int cifs_fscache_register(void)
+{
+	return fscache_register_netfs(&cifs_fscache_netfs);
+}
+
+/*
+ * Unregister CIFS for caching
+ */
+void cifs_fscache_unregister(void)
+{
+	fscache_unregister_netfs(&cifs_fscache_netfs);
+}
+
Index: cifs-2.6/fs/cifs/cifsfs.c
===================================================================
--- cifs-2.6.orig/fs/cifs/cifsfs.c
+++ cifs-2.6/fs/cifs/cifsfs.c
@@ -47,6 +47,7 @@
 #include <linux/key-type.h>
 #include "dns_resolve.h"
 #include "cifs_spnego.h"
+#include "fscache.h"
 #define CIFS_MAGIC_NUMBER 0xFF534D42	/* the first four bytes of SMB PDUs */
 
 int cifsFYI = 0;
@@ -902,6 +903,10 @@ init_cifs(void)
 		cFYI(1, "cifs_max_pending set to max of 256");
 	}
 
+	rc = cifs_fscache_register();
+	if (rc)
+		goto out;
+
 	rc = cifs_init_inodecache();
 	if (rc)
 		goto out_clean_proc;
@@ -951,6 +956,8 @@ init_cifs(void)
 	cifs_destroy_inodecache();
  out_clean_proc:
 	cifs_proc_clean();
+	cifs_fscache_unregister();
+ out:
 	return rc;
 }
 
@@ -959,6 +966,7 @@ exit_cifs(void)
 {
 	cFYI(DBG2, "exit_cifs");
 	cifs_proc_clean();
+	cifs_fscache_unregister();
 #ifdef CONFIG_CIFS_DFS_UPCALL
 	cifs_dfs_release_automount_timer();
 	unregister_key_type(&key_type_dns_resolver);
Index: cifs-2.6/fs/cifs/fscache.h
===================================================================
--- /dev/null
+++ cifs-2.6/fs/cifs/fscache.h
@@ -0,0 +1,39 @@
+/*
+ *   fs/cifs/fscache.h - CIFS filesystem cache interface definitions
+ *
+ *   Copyright (c) 2010 Novell, Inc.
+ *   Authors(s): Suresh Jayaraman (sjayaraman-l3A5Bk7waGM@public.gmane.org>
+ *
+ *   This library is free software; you can redistribute it and/or modify
+ *   it under the terms of the GNU Lesser General Public License as published
+ *   by the Free Software Foundation; either version 2.1 of the License, or
+ *   (at your option) any later version.
+ *
+ *   This library is distributed in the hope that it will be useful,
+ *   but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+ *   the GNU Lesser General Public License for more details.
+ *
+ *   You should have received a copy of the GNU Lesser General Public License
+ *   along with this library; if not, write to the Free Software
+ *   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+#ifndef _CIFS_FSCACHE_H
+#define _CIFS_FSCACHE_H
+
+#include <linux/fscache.h>
+
+#ifdef CONFIG_CIFS_FSCACHE
+
+extern struct fscache_netfs cifs_fscache_netfs;
+
+extern int cifs_fscache_register(void);
+extern void cifs_fscache_unregister(void);
+
+#else /* CONFIG_CIFS_FSCACHE */
+static inline int cifs_fscache_register(void) { return 0; }
+static inline void cifs_fscache_unregister(void) {}
+
+#endif /* CONFIG_CIFS_FSCACHE */
+
+#endif /* _CIFS_FSCACHE_H */

[PATCH 03/09] cifs: define server-level cache index objects and register them

[Suresh Jayaraman]

Define server-level cache index objects (as managed by TCP_ServerInfo structs)
and register then with FS-Cache. Each server object is created in the CIFS
top-level index object and is itself an index into which superblock-level
objects are inserted.

The server objects are now keyed by {IPaddress,family,port} tuple.

Signed-off-by: Suresh Jayaraman <sjayaraman@suse.de>
---
 fs/cifs/Makefile   |    2 -
 fs/cifs/cache.c    |   62 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/cifs/cifsglob.h |    3 ++
 fs/cifs/connect.c  |    5 ++++
 fs/cifs/fscache.c  |   41 +++++++++++++++++++++++++++++++++++
 fs/cifs/fscache.h  |   14 +++++++++++
 6 files changed, 126 insertions(+), 1 deletion(-)
 create mode 100644 fs/cifs/fscache.c

Index: cifs-2.6/fs/cifs/Makefile
===================================================================
--- cifs-2.6.orig/fs/cifs/Makefile
+++ cifs-2.6/fs/cifs/Makefile
@@ -12,4 +12,4 @@ cifs-$(CONFIG_CIFS_UPCALL) += cifs_spneg
 
 cifs-$(CONFIG_CIFS_DFS_UPCALL) += dns_resolve.o cifs_dfs_ref.o
 
-cifs-$(CONFIG_CIFS_FSCACHE) += cache.o
+cifs-$(CONFIG_CIFS_FSCACHE) += fscache.o cache.o
Index: cifs-2.6/fs/cifs/cache.c
===================================================================
--- cifs-2.6.orig/fs/cifs/cache.c
+++ cifs-2.6/fs/cifs/cache.c
@@ -19,6 +19,7 @@
  *   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
  */
 #include "fscache.h"
+#include "cifs_debug.h"
 
 /*
  * CIFS filesystem definition for FS-Cache
@@ -44,3 +45,64 @@ void cifs_fscache_unregister(void)
 	fscache_unregister_netfs(&cifs_fscache_netfs);
 }
 
+/*
+ * Key layout of CIFS server cache index object
+ */
+struct cifs_server_key {
+	uint16_t	family;		/* address family */
+	uint16_t	port;		/* IP port */
+	union {
+		struct in_addr	ipv4_addr;
+		struct in6_addr	ipv6_addr;
+	} addr[0];
+};
+
+/*
+ * Server object keyed by {IPaddress,port,family} tuple
+ */
+static uint16_t cifs_server_get_key(const void *cookie_netfs_data,
+				   void *buffer, uint16_t maxbuf)
+{
+	const struct TCP_Server_Info *server = cookie_netfs_data;
+	const struct sockaddr *sa = (struct sockaddr *) &server->addr.sockAddr;
+	struct cifs_server_key *key = buffer;
+	uint16_t key_len = sizeof(struct cifs_server_key);
+
+	memset(key, 0, key_len);
+
+	/*
+	 * Should not be a problem as sin_family/sin6_family overlays
+	 * sa_family field
+	 */
+	switch (sa->sa_family) {
+	case AF_INET:
+		key->family = server->addr.sockAddr.sin_family;
+		key->port = server->addr.sockAddr.sin_port;
+		key->addr[0].ipv4_addr = server->addr.sockAddr.sin_addr;
+		key_len += sizeof(key->addr[0].ipv4_addr);
+		break;
+
+	case AF_INET6:
+		key->family = server->addr.sockAddr6.sin6_family;
+		key->port = server->addr.sockAddr6.sin6_port;
+		key->addr[0].ipv6_addr = server->addr.sockAddr6.sin6_addr;
+		key_len += sizeof(key->addr[0].ipv6_addr);
+		break;
+
+	default:
+		cERROR(1, "CIFS: Unknown network family '%d'", sa->sa_family);
+		key_len = 0;
+		break;
+	}
+
+	return key_len;
+}
+
+/*
+ * Server object for FS-Cache
+ */
+const struct fscache_cookie_def cifs_fscache_server_index_def = {
+	.name = "CIFS.server",
+	.type = FSCACHE_COOKIE_TYPE_INDEX,
+	.get_key = cifs_server_get_key,
+};
Index: cifs-2.6/fs/cifs/cifsglob.h
===================================================================
--- cifs-2.6.orig/fs/cifs/cifsglob.h
+++ cifs-2.6/fs/cifs/cifsglob.h
@@ -190,6 +190,9 @@ struct TCP_Server_Info {
 	bool	sec_mskerberos;		/* supports legacy MS Kerberos */
 	bool	sec_kerberosu2u;	/* supports U2U Kerberos */
 	bool	sec_ntlmssp;		/* supports NTLMSSP */
+#ifdef CONFIG_CIFS_FSCACHE
+	struct fscache_cookie   *fscache; /* client index cache cookie */
+#endif
 };
 
 /*
Index: cifs-2.6/fs/cifs/connect.c
===================================================================
--- cifs-2.6.orig/fs/cifs/connect.c
+++ cifs-2.6/fs/cifs/connect.c
@@ -48,6 +48,7 @@
 #include "nterr.h"
 #include "rfc1002pdu.h"
 #include "cn_cifs.h"
+#include "fscache.h"
 
 #define CIFS_PORT 445
 #define RFC1001_PORT 139
@@ -1460,6 +1461,8 @@ cifs_put_tcp_session(struct TCP_Server_I
 	server->tcpStatus = CifsExiting;
 	spin_unlock(&GlobalMid_Lock);
 
+	cifs_fscache_release_client_cookie(server);
+
 	task = xchg(&server->tsk, NULL);
 	if (task)
 		force_sig(SIGKILL, task);
@@ -1577,6 +1580,8 @@ cifs_get_tcp_session(struct smb_vol *vol
 	list_add(&tcp_ses->tcp_ses_list, &cifs_tcp_ses_list);
 	write_unlock(&cifs_tcp_ses_lock);
 
+	cifs_fscache_get_client_cookie(tcp_ses);
+
 	return tcp_ses;
 
 out_err:
Index: cifs-2.6/fs/cifs/fscache.c
===================================================================
--- /dev/null
+++ cifs-2.6/fs/cifs/fscache.c
@@ -0,0 +1,41 @@
+/*
+ *   fs/cifs/fscache.c - CIFS filesystem cache interface
+ *
+ *   Copyright (c) 2010 Novell, Inc.
+ *   Author(s): Suresh Jayaraman (sjayaraman@suse.de>
+ *
+ *   This library is free software; you can redistribute it and/or modify
+ *   it under the terms of the GNU Lesser General Public License as published
+ *   by the Free Software Foundation; either version 2.1 of the License, or
+ *   (at your option) any later version.
+ *
+ *   This library is distributed in the hope that it will be useful,
+ *   but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See
+ *   the GNU Lesser General Public License for more details.
+ *
+ *   You should have received a copy of the GNU Lesser General Public License
+ *   along with this library; if not, write to the Free Software
+ *   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+#include "fscache.h"
+#include "cifsglob.h"
+#include "cifs_debug.h"
+
+void cifs_fscache_get_client_cookie(struct TCP_Server_Info *server)
+{
+	server->fscache =
+		fscache_acquire_cookie(cifs_fscache_netfs.primary_index,
+				&cifs_fscache_server_index_def, server);
+	cFYI(1, "CIFS: get client cookie (0x%p/0x%p)", server,
+				server->fscache);
+}
+
+void cifs_fscache_release_client_cookie(struct TCP_Server_Info *server)
+{
+	cFYI(1, "CIFS: release client cookie (0x%p/0x%p)", server,
+				server->fscache);
+	fscache_relinquish_cookie(server->fscache, 0);
+	server->fscache = NULL;
+}
+
Index: cifs-2.6/fs/cifs/fscache.h
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.h
+++ cifs-2.6/fs/cifs/fscache.h
@@ -23,17 +23,31 @@
 
 #include <linux/fscache.h>
 
+#include "cifsglob.h"
+
 #ifdef CONFIG_CIFS_FSCACHE
 
 extern struct fscache_netfs cifs_fscache_netfs;
+extern const struct fscache_cookie_def cifs_fscache_server_index_def;
 
 extern int cifs_fscache_register(void);
 extern void cifs_fscache_unregister(void);
 
+/*
+ * fscache.c
+ */
+extern void cifs_fscache_get_client_cookie(struct TCP_Server_Info *);
+extern void cifs_fscache_release_client_cookie(struct TCP_Server_Info *);
+
 #else /* CONFIG_CIFS_FSCACHE */
 static inline int cifs_fscache_register(void) { return 0; }
 static inline void cifs_fscache_unregister(void) {}
 
+static inline void
+cifs_fscache_get_client_cookie(struct TCP_Server_Info *server) {}
+static inline void
+cifs_fscache_get_client_cookie(struct TCP_Server_Info *server); {}
+
 #endif /* CONFIG_CIFS_FSCACHE */
 
 #endif /* _CIFS_FSCACHE_H */

[PATCH 04/09] cifs: define superblock-level cache index objects and register them

[Suresh Jayaraman]

Define superblock-level cache index objects (managed by cifsTconInfo structs).
Each superblock object is created in a server-level index object and in itself
an index into which inode-level objects are inserted.

The superblock object is keyed by sharename. The UniqueId/IndexNumber is used to
validate that the exported share is the same since we accessed it last time.

Signed-off-by: Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org>
---
 fs/cifs/cache.c    |  109 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/cifs/cifsglob.h |    4 +
 fs/cifs/connect.c  |    3 +
 fs/cifs/fscache.c  |   17 ++++++++
 fs/cifs/fscache.h  |    6 ++
 fs/cifs/inode.c    |    3 +
 6 files changed, 142 insertions(+)

Index: cifs-2.6/fs/cifs/cache.c
===================================================================
--- cifs-2.6.orig/fs/cifs/cache.c
+++ cifs-2.6/fs/cifs/cache.c
@@ -106,3 +106,112 @@ const struct fscache_cookie_def cifs_fsc
 	.type = FSCACHE_COOKIE_TYPE_INDEX,
 	.get_key = cifs_server_get_key,
 };
+
+/*
+ * Auxiliary data attached to CIFS superblock within the cache
+ */
+struct cifs_fscache_super_auxdata {
+	u64	resource_id;		/* unique server resource id */
+};
+
+static char *extract_sharename(const char *treename)
+{
+	const char *src;
+	char *delim, *dst;
+	int len;
+
+	/* skip double chars at the beginning */
+	src = treename + 2;
+
+	/* share name is always preceded by '\\' now */
+	delim = strchr(src, '\\');
+	if (!delim)
+		return ERR_PTR(-EINVAL);
+	delim++;
+	len = strlen(delim);
+
+	/* caller has to free the memory */
+	dst = kstrndup(delim, len, GFP_KERNEL);
+	if (!dst)
+		return ERR_PTR(-ENOMEM);
+
+	return dst;
+}
+
+/*
+ * Superblock object currently keyed by share name
+ */
+static uint16_t cifs_super_get_key(const void *cookie_netfs_data, void *buffer,
+				   uint16_t maxbuf)
+{
+	const struct cifsTconInfo *tcon = cookie_netfs_data;
+	char *sharename;
+	uint16_t len;
+
+	sharename = extract_sharename(tcon->treeName);
+	if (IS_ERR(sharename)) {
+		cFYI(1, "CIFS: couldn't extract sharename\n");
+		sharename = NULL;
+		return 0;
+	}
+
+	len = strlen(sharename);
+	if (len > maxbuf)
+		return 0;
+
+	memcpy(buffer, sharename, len);
+
+	kfree(sharename);
+
+	return len;
+}
+
+static uint16_t
+cifs_fscache_super_get_aux(const void *cookie_netfs_data, void *buffer,
+			   uint16_t maxbuf)
+{
+	struct cifs_fscache_super_auxdata auxdata;
+	const struct cifsTconInfo *tcon = cookie_netfs_data;
+
+	memset(&auxdata, 0, sizeof(auxdata));
+	auxdata.resource_id = tcon->resource_id;
+
+	if (maxbuf > sizeof(auxdata))
+		maxbuf = sizeof(auxdata);
+
+	memcpy(buffer, &auxdata, maxbuf);
+
+	return maxbuf;
+}
+
+static enum
+fscache_checkaux cifs_fscache_super_check_aux(void *cookie_netfs_data,
+					      const void *data,
+					      uint16_t datalen)
+{
+	struct cifs_fscache_super_auxdata auxdata;
+	const struct cifsTconInfo *tcon = cookie_netfs_data;
+
+	if (datalen != sizeof(auxdata))
+		return FSCACHE_CHECKAUX_OBSOLETE;
+
+	memset(&auxdata, 0, sizeof(auxdata));
+	auxdata.resource_id = tcon->resource_id;
+
+	if (memcmp(data, &auxdata, datalen) != 0)
+		return FSCACHE_CHECKAUX_OBSOLETE;
+
+	return FSCACHE_CHECKAUX_OKAY;
+}
+
+/*
+ * Superblock object for FS-Cache
+ */
+const struct fscache_cookie_def cifs_fscache_super_index_def = {
+	.name = "CIFS.super",
+	.type = FSCACHE_COOKIE_TYPE_INDEX,
+	.get_key = cifs_super_get_key,
+	.get_aux = cifs_fscache_super_get_aux,
+	.check_aux = cifs_fscache_super_check_aux,
+};
+
Index: cifs-2.6/fs/cifs/fscache.c
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.c
+++ cifs-2.6/fs/cifs/fscache.c
@@ -39,3 +39,20 @@ void cifs_fscache_release_client_cookie(
 	server->fscache = NULL;
 }
 
+void cifs_fscache_get_super_cookie(struct cifsTconInfo *tcon)
+{
+	struct TCP_Server_Info *server = tcon->ses->server;
+
+	tcon->fscache =
+		fscache_acquire_cookie(server->fscache,
+				&cifs_fscache_super_index_def, tcon);
+	cFYI(1, "CIFS: get superblock cookie (0x%p/0x%p)",
+				server->fscache, tcon->fscache);
+}
+
+void cifs_fscache_release_super_cookie(struct cifsTconInfo *tcon)
+{
+	cFYI(1, "CIFS: releasing superblock cookie (0x%p)", tcon->fscache);
+	fscache_relinquish_cookie(tcon->fscache, 0);
+	tcon->fscache = NULL;
+}
Index: cifs-2.6/fs/cifs/fscache.h
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.h
+++ cifs-2.6/fs/cifs/fscache.h
@@ -29,6 +29,7 @@
 
 extern struct fscache_netfs cifs_fscache_netfs;
 extern const struct fscache_cookie_def cifs_fscache_server_index_def;
+extern const struct fscache_cookie_def cifs_fscache_super_index_def;
 
 extern int cifs_fscache_register(void);
 extern void cifs_fscache_unregister(void);
@@ -38,6 +39,8 @@ extern void cifs_fscache_unregister(void
  */
 extern void cifs_fscache_get_client_cookie(struct TCP_Server_Info *);
 extern void cifs_fscache_release_client_cookie(struct TCP_Server_Info *);
+extern void cifs_fscache_get_super_cookie(struct cifsTconInfo *);
+extern void cifs_fscache_release_super_cookie(struct cifsTconInfo *);
 
 #else /* CONFIG_CIFS_FSCACHE */
 static inline int cifs_fscache_register(void) { return 0; }
@@ -47,6 +50,9 @@ static inline void
 cifs_fscache_get_client_cookie(struct TCP_Server_Info *server) {}
 static inline void
 cifs_fscache_get_client_cookie(struct TCP_Server_Info *server); {}
+static inline void cifs_fscache_get_super_cookie(struct cifsTconInfo *tcon) {}
+static inline void
+cifs_fscache_release_super_cookie(struct cifsTconInfo *tcon) {}
 
 #endif /* CONFIG_CIFS_FSCACHE */
 
Index: cifs-2.6/fs/cifs/cifsglob.h
===================================================================
--- cifs-2.6.orig/fs/cifs/cifsglob.h
+++ cifs-2.6/fs/cifs/cifsglob.h
@@ -314,6 +314,10 @@ struct cifsTconInfo {
 	bool local_lease:1; /* check leases (only) on local system not remote */
 	bool broken_posix_open; /* e.g. Samba server versions < 3.3.2, 3.2.9 */
 	bool need_reconnect:1; /* connection reset, tid now invalid */
+#ifdef CONFIG_CIFS_FSCACHE
+	u64 resource_id;		/* server resource id */
+	struct fscache_cookie *fscache;	/* cookie for share */
+#endif
 	/* BB add field for back pointer to sb struct(s)? */
 };
 
Index: cifs-2.6/fs/cifs/connect.c
===================================================================
--- cifs-2.6.orig/fs/cifs/connect.c
+++ cifs-2.6/fs/cifs/connect.c
@@ -1779,6 +1779,7 @@ cifs_put_tcon(struct cifsTconInfo *tcon)
 	_FreeXid(xid);
 
 	tconInfoFree(tcon);
+	cifs_fscache_release_super_cookie(tcon);
 	cifs_put_smb_ses(ses);
 }
 
@@ -1848,6 +1849,8 @@ cifs_get_tcon(struct cifsSesInfo *ses, s
 	list_add(&tcon->tcon_list, &ses->tcon_list);
 	write_unlock(&cifs_tcp_ses_lock);
 
+	cifs_fscache_get_super_cookie(tcon);
+
 	return tcon;
 
 out_fail:
Index: cifs-2.6/fs/cifs/inode.c
===================================================================
--- cifs-2.6.orig/fs/cifs/inode.c
+++ cifs-2.6/fs/cifs/inode.c
@@ -807,6 +807,9 @@ struct inode *cifs_root_iget(struct supe
 	if (!inode)
 		return ERR_PTR(-ENOMEM);
 
+	/* populate tcon->resource_id */
+	cifs_sb->tcon->resource_id = CIFS_I(inode)->uniqueid;
+
 	if (rc && cifs_sb->tcon->ipc) {
 		cFYI(1, "ipc connection - fake read inode");
 		inode->i_mode |= S_IFDIR;

[PATCH 05/09] cifs: define inode-level cache object and register them

[Suresh Jayaraman]

Define inode-level data storage objects (managed by cifsInodeInfo structs).
Each inode-level object is created in a super-block level object and is itself
a data storage object in to which pages from the inode are stored.

The inode object is keyed by UniqueId. The coherency data being used is
LastWriteTime, LastChangeTime and end of file reported by the server.

Signed-off-by: Suresh Jayaraman <sjayaraman@suse.de>
---
 fs/cifs/cache.c    |   83 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/cifs/cifsfs.c   |    7 ++++
 fs/cifs/cifsglob.h |    3 +
 fs/cifs/file.c     |    6 +++
 fs/cifs/fscache.c  |   68 +++++++++++++++++++++++++++++++++++++++++++
 fs/cifs/fscache.h  |   12 +++++++
 fs/cifs/inode.c    |    4 ++
 7 files changed, 183 insertions(+)

Index: cifs-2.6/fs/cifs/cache.c
===================================================================
--- cifs-2.6.orig/fs/cifs/cache.c
+++ cifs-2.6/fs/cifs/cache.c
@@ -215,3 +215,86 @@ const struct fscache_cookie_def cifs_fsc
 	.check_aux = cifs_fscache_super_check_aux,
 };
 
+/*
+ * Auxiliary data attached to CIFS inode within the cache
+ */
+struct cifs_fscache_inode_auxdata {
+	struct timespec	last_write_time;
+	struct timespec	last_change_time;
+	u64		eof;
+};
+
+static uint16_t cifs_fscache_inode_get_key(const void *cookie_netfs_data,
+					   void *buffer, uint16_t maxbuf)
+{
+	const struct cifsInodeInfo *cifsi = cookie_netfs_data;
+	uint16_t keylen;
+
+	/* use the UniqueId as the key */
+	keylen = sizeof(cifsi->uniqueid);
+	if (keylen > maxbuf)
+		keylen = 0;
+	else
+		memcpy(buffer, &cifsi->uniqueid, keylen);
+
+	return keylen;
+}
+
+static void
+cifs_fscache_inode_get_attr(const void *cookie_netfs_data, uint64_t *size)
+{
+	const struct cifsInodeInfo *cifsi = cookie_netfs_data;
+
+	*size = cifsi->vfs_inode.i_size;
+}
+
+static uint16_t
+cifs_fscache_inode_get_aux(const void *cookie_netfs_data, void *buffer,
+			   uint16_t maxbuf)
+{
+	struct cifs_fscache_inode_auxdata auxdata;
+	const struct cifsInodeInfo *cifsi = cookie_netfs_data;
+
+	memset(&auxdata, 0, sizeof(auxdata));
+	auxdata.eof = cifsi->server_eof;
+	auxdata.last_write_time = cifsi->vfs_inode.i_mtime;
+	auxdata.last_change_time = cifsi->vfs_inode.i_ctime;
+
+	if (maxbuf > sizeof(auxdata))
+		maxbuf = sizeof(auxdata);
+
+	memcpy(buffer, &auxdata, maxbuf);
+
+	return maxbuf;
+}
+
+static enum
+fscache_checkaux cifs_fscache_inode_check_aux(void *cookie_netfs_data,
+					      const void *data,
+					      uint16_t datalen)
+{
+	struct cifs_fscache_inode_auxdata auxdata;
+	struct cifsInodeInfo *cifsi = cookie_netfs_data;
+
+	if (datalen != sizeof(auxdata))
+		return FSCACHE_CHECKAUX_OBSOLETE;
+
+	memset(&auxdata, 0, sizeof(auxdata));
+	auxdata.eof = cifsi->server_eof;
+	auxdata.last_write_time = cifsi->vfs_inode.i_mtime;
+	auxdata.last_change_time = cifsi->vfs_inode.i_ctime;
+
+	if (memcmp(data, &auxdata, datalen) != 0)
+		return FSCACHE_CHECKAUX_OBSOLETE;
+
+	return FSCACHE_CHECKAUX_OKAY;
+}
+
+const struct fscache_cookie_def cifs_fscache_inode_object_def = {
+	.name		= "CIFS.uniqueid",
+	.type		= FSCACHE_COOKIE_TYPE_DATAFILE,
+	.get_key	= cifs_fscache_inode_get_key,
+	.get_attr	= cifs_fscache_inode_get_attr,
+	.get_aux	= cifs_fscache_inode_get_aux,
+	.check_aux	= cifs_fscache_inode_check_aux,
+};
Index: cifs-2.6/fs/cifs/cifsfs.c
===================================================================
--- cifs-2.6.orig/fs/cifs/cifsfs.c
+++ cifs-2.6/fs/cifs/cifsfs.c
@@ -330,6 +330,12 @@ cifs_destroy_inode(struct inode *inode)
 }
 
 static void
+cifs_clear_inode(struct inode *inode)
+{
+	cifs_fscache_release_inode_cookie(inode);
+}
+
+static void
 cifs_show_address(struct seq_file *s, struct TCP_Server_Info *server)
 {
 	seq_printf(s, ",addr=");
@@ -490,6 +496,7 @@ static const struct super_operations cif
 	.alloc_inode = cifs_alloc_inode,
 	.destroy_inode = cifs_destroy_inode,
 	.drop_inode	= cifs_drop_inode,
+	.clear_inode	= cifs_clear_inode,
 /*	.delete_inode	= cifs_delete_inode,  */  /* Do not need above
 	function unless later we add lazy close of inodes or unless the
 	kernel forgets to call us with the same number of releases (closes)
Index: cifs-2.6/fs/cifs/cifsglob.h
===================================================================
--- cifs-2.6.orig/fs/cifs/cifsglob.h
+++ cifs-2.6/fs/cifs/cifsglob.h
@@ -405,6 +405,9 @@ struct cifsInodeInfo {
 	bool invalid_mapping:1;		/* pagecache is invalid */
 	u64  server_eof;		/* current file size on server */
 	u64  uniqueid;			/* server inode number */
+#ifdef CONFIG_CIFS_FSCACHE
+	struct fscache_cookie *fscache;
+#endif
 	struct inode vfs_inode;
 };
 
Index: cifs-2.6/fs/cifs/file.c
===================================================================
--- cifs-2.6.orig/fs/cifs/file.c
+++ cifs-2.6/fs/cifs/file.c
@@ -40,6 +40,7 @@
 #include "cifs_unicode.h"
 #include "cifs_debug.h"
 #include "cifs_fs_sb.h"
+#include "fscache.h"
 
 static inline int cifs_convert_flags(unsigned int flags)
 {
@@ -282,6 +283,9 @@ int cifs_open(struct inode *inode, struc
 				CIFSSMBClose(xid, tcon, netfid);
 				rc = -ENOMEM;
 			}
+
+			cifs_fscache_set_inode_cookie(inode, file);
+
 			goto out;
 		} else if ((rc == -EINVAL) || (rc == -EOPNOTSUPP)) {
 			if (tcon->ses->serverNOS)
@@ -373,6 +377,8 @@ int cifs_open(struct inode *inode, struc
 		goto out;
 	}
 
+	cifs_fscache_set_inode_cookie(inode, file);
+
 	if (oplock & CIFS_CREATE_ACTION) {
 		/* time to set mode which we can not set earlier due to
 		   problems creating new read-only files */
Index: cifs-2.6/fs/cifs/fscache.h
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.h
+++ cifs-2.6/fs/cifs/fscache.h
@@ -30,6 +30,8 @@
 extern struct fscache_netfs cifs_fscache_netfs;
 extern const struct fscache_cookie_def cifs_fscache_server_index_def;
 extern const struct fscache_cookie_def cifs_fscache_super_index_def;
+extern const struct fscache_cookie_def cifs_fscache_inode_object_def;
+
 
 extern int cifs_fscache_register(void);
 extern void cifs_fscache_unregister(void);
@@ -42,6 +44,10 @@ extern void cifs_fscache_release_client_
 extern void cifs_fscache_get_super_cookie(struct cifsTconInfo *);
 extern void cifs_fscache_release_super_cookie(struct cifsTconInfo *);
 
+extern void cifs_fscache_release_inode_cookie(struct inode *);
+extern void cifs_fscache_set_inode_cookie(struct inode *, struct file *);
+extern void cifs_fscache_reset_inode_cookie(struct inode *);
+
 #else /* CONFIG_CIFS_FSCACHE */
 static inline int cifs_fscache_register(void) { return 0; }
 static inline void cifs_fscache_unregister(void) {}
@@ -54,6 +60,12 @@ static inline void cifs_fscache_get_supe
 static inline void
 cifs_fscache_release_super_cookie(struct cifsTconInfo *tcon) {}
 
+static inline void cifs_fscache_release_inode_cookie(struct inode *inode) {}
+static inline void cifs_fscache_set_inode_cookie(struct inode *inode,
+						 struct file *filp) {}
+static inline void cifs_fscache_reset_inode_cookie(struct inode *inode) {}
+
+
 #endif /* CONFIG_CIFS_FSCACHE */
 
 #endif /* _CIFS_FSCACHE_H */
Index: cifs-2.6/fs/cifs/inode.c
===================================================================
--- cifs-2.6.orig/fs/cifs/inode.c
+++ cifs-2.6/fs/cifs/inode.c
@@ -29,6 +29,7 @@
 #include "cifsproto.h"
 #include "cifs_debug.h"
 #include "cifs_fs_sb.h"
+#include "fscache.h"
 
 
 static void cifs_set_ops(struct inode *inode, const bool is_dfs_referral)
@@ -776,6 +777,8 @@ retry_iget5_locked:
 			inode->i_flags |= S_NOATIME | S_NOCMTIME;
 		if (inode->i_state & I_NEW) {
 			inode->i_ino = hash;
+			/* initialize per-inode cache cookie pointer */
+			CIFS_I(inode)->fscache = NULL;
 			unlock_new_inode(inode);
 		}
 	}
@@ -1571,6 +1574,7 @@ cifs_invalidate_mapping(struct inode *in
 			cifs_i->write_behind_rc = rc;
 	}
 	invalidate_remote_inode(inode);
+	cifs_fscache_reset_inode_cookie(inode);
 }
 
 int cifs_revalidate_file(struct file *filp)
Index: cifs-2.6/fs/cifs/fscache.c
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.c
+++ cifs-2.6/fs/cifs/fscache.c
@@ -21,6 +21,7 @@
 #include "fscache.h"
 #include "cifsglob.h"
 #include "cifs_debug.h"
+#include "cifs_fs_sb.h"
 
 void cifs_fscache_get_client_cookie(struct TCP_Server_Info *server)
 {
@@ -56,3 +57,70 @@ void cifs_fscache_release_super_cookie(s
 	fscache_relinquish_cookie(tcon->fscache, 0);
 	tcon->fscache = NULL;
 }
+
+static void cifs_fscache_enable_inode_cookie(struct inode *inode)
+{
+	struct cifsInodeInfo *cifsi = CIFS_I(inode);
+	struct cifs_sb_info *cifs_sb = CIFS_SB(inode->i_sb);
+
+	if (cifsi->fscache)
+		return;
+
+	cifsi->fscache = fscache_acquire_cookie(cifs_sb->tcon->fscache,
+				&cifs_fscache_inode_object_def,
+				cifsi);
+	cFYI(1, "CIFS: got FH cookie (0x%p/0x%p)",
+			cifs_sb->tcon->fscache, cifsi->fscache);
+}
+
+void cifs_fscache_release_inode_cookie(struct inode *inode)
+{
+	struct cifsInodeInfo *cifsi = CIFS_I(inode);
+
+	if (cifsi->fscache) {
+		cFYI(1, "CIFS releasing inode cookie (0x%p)",
+				cifsi->fscache);
+		fscache_relinquish_cookie(cifsi->fscache, 0);
+		cifsi->fscache = NULL;
+	}
+}
+
+static void cifs_fscache_disable_inode_cookie(struct inode *inode)
+{
+	struct cifsInodeInfo *cifsi = CIFS_I(inode);
+
+	if (cifsi->fscache) {
+		cFYI(1, "CIFS disabling inode cookie (0x%p)",
+				cifsi->fscache);
+		fscache_relinquish_cookie(cifsi->fscache, 1);
+		cifsi->fscache = NULL;
+	}
+}
+
+void cifs_fscache_set_inode_cookie(struct inode *inode, struct file *filp)
+{
+	if ((filp->f_flags & O_ACCMODE) != O_RDONLY)
+		cifs_fscache_disable_inode_cookie(inode);
+	else {
+		cifs_fscache_enable_inode_cookie(inode);
+		cFYI(1, "CIFS: fscache inode cookie set");
+	}
+}
+
+void cifs_fscache_reset_inode_cookie(struct inode *inode)
+{
+	struct cifsInodeInfo *cifsi = CIFS_I(inode);
+	struct cifs_sb_info *cifs_sb = CIFS_SB(inode->i_sb);
+	struct fscache_cookie *old = cifsi->fscache;
+
+	if (cifsi->fscache) {
+		/* retire the current fscache cache and get a new one */
+		fscache_relinquish_cookie(cifsi->fscache, 1);
+
+		cifsi->fscache = fscache_acquire_cookie(cifs_sb->tcon->fscache,
+					&cifs_fscache_inode_object_def,
+					cifsi);
+		cFYI(1, "CIFS: new cookie 0x%p oldcookie 0x%p",
+				cifsi->fscache, old);
+	}
+}

[PATCH 06/09] cifs: FS-Cache page management

[Suresh Jayaraman]

Takes care of invalidation and release of FS-Cache marked pages and also
invalidation of the FsCache page flag when the inode is removed.

Signed-off-by: Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org>
Acked-by: David Howells <dhowells-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 fs/cifs/cache.c   |   31 +++++++++++++++++++++++++++++++
 fs/cifs/file.c    |   20 ++++++++++++++++++++
 fs/cifs/fscache.c |   26 ++++++++++++++++++++++++++
 fs/cifs/fscache.h |   16 ++++++++++++++++
 4 files changed, 93 insertions(+)

Index: cifs-2.6/fs/cifs/cache.c
===================================================================
--- cifs-2.6.orig/fs/cifs/cache.c
+++ cifs-2.6/fs/cifs/cache.c
@@ -290,6 +290,36 @@ fscache_checkaux cifs_fscache_inode_chec
 	return FSCACHE_CHECKAUX_OKAY;
 }
 
+static void cifs_fscache_inode_now_uncached(void *cookie_netfs_data)
+{
+	struct cifsInodeInfo *cifsi = cookie_netfs_data;
+	struct pagevec pvec;
+	pgoff_t first;
+	int loop, nr_pages;
+
+	pagevec_init(&pvec, 0);
+	first = 0;
+
+	cFYI(1, "cifs inode 0x%p now uncached", cifsi);
+
+	for (;;) {
+		nr_pages = pagevec_lookup(&pvec,
+					  cifsi->vfs_inode.i_mapping, first,
+					  PAGEVEC_SIZE - pagevec_count(&pvec));
+		if (!nr_pages)
+			break;
+
+		for (loop = 0; loop < nr_pages; loop++)
+			ClearPageFsCache(pvec.pages[loop]);
+
+		first = pvec.pages[nr_pages - 1]->index + 1;
+
+		pvec.nr = nr_pages;
+		pagevec_release(&pvec);
+		cond_resched();
+	}
+}
+
 const struct fscache_cookie_def cifs_fscache_inode_object_def = {
 	.name		= "CIFS.uniqueid",
 	.type		= FSCACHE_COOKIE_TYPE_DATAFILE,
@@ -297,4 +327,5 @@ const struct fscache_cookie_def cifs_fsc
 	.get_attr	= cifs_fscache_inode_get_attr,
 	.get_aux	= cifs_fscache_inode_get_aux,
 	.check_aux	= cifs_fscache_inode_check_aux,
+	.now_uncached	= cifs_fscache_inode_now_uncached,
 };
Index: cifs-2.6/fs/cifs/file.c
===================================================================
--- cifs-2.6.orig/fs/cifs/file.c
+++ cifs-2.6/fs/cifs/file.c
@@ -2271,6 +2271,22 @@ out:
 	return rc;
 }
 
+static int cifs_release_page(struct page *page, gfp_t gfp)
+{
+	if (PagePrivate(page))
+		return 0;
+
+	return cifs_fscache_release_page(page, gfp);
+}
+
+static void cifs_invalidate_page(struct page *page, unsigned long offset)
+{
+	struct cifsInodeInfo *cifsi = CIFS_I(page->mapping->host);
+
+	if (offset == 0)
+		cifs_fscache_invalidate_page(page, &cifsi->vfs_inode);
+}
+
 static void
 cifs_oplock_break(struct slow_work *work)
 {
@@ -2344,6 +2360,8 @@ const struct address_space_operations ci
 	.write_begin = cifs_write_begin,
 	.write_end = cifs_write_end,
 	.set_page_dirty = __set_page_dirty_nobuffers,
+	.releasepage = cifs_release_page,
+	.invalidatepage = cifs_invalidate_page,
 	/* .sync_page = cifs_sync_page, */
 	/* .direct_IO = */
 };
@@ -2360,6 +2378,8 @@ const struct address_space_operations ci
 	.write_begin = cifs_write_begin,
 	.write_end = cifs_write_end,
 	.set_page_dirty = __set_page_dirty_nobuffers,
+	.releasepage = cifs_release_page,
+	.invalidatepage = cifs_invalidate_page,
 	/* .sync_page = cifs_sync_page, */
 	/* .direct_IO = */
 };
Index: cifs-2.6/fs/cifs/fscache.c
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.c
+++ cifs-2.6/fs/cifs/fscache.c
@@ -124,3 +124,29 @@ void cifs_fscache_reset_inode_cookie(str
 				cifsi->fscache, old);
 	}
 }
+
+int cifs_fscache_release_page(struct page *page, gfp_t gfp)
+{
+	if (PageFsCache(page)) {
+		struct inode *inode = page->mapping->host;
+		struct cifsInodeInfo *cifsi = CIFS_I(inode);
+
+		cFYI(1, "CIFS: fscache release page (0x%p/0x%p)",
+				page, cifsi->fscache);
+		if (!fscache_maybe_release_page(cifsi->fscache, page, gfp))
+			return 0;
+	}
+
+	return 1;
+}
+
+void __cifs_fscache_invalidate_page(struct page *page, struct inode *inode)
+{
+	struct cifsInodeInfo *cifsi = CIFS_I(inode);
+	struct fscache_cookie *cookie = cifsi->fscache;
+
+	cFYI(1, "CIFS: fscache invalidatepage (0x%p/0x%p)", page, cookie);
+	fscache_wait_on_page_write(cookie, page);
+	fscache_uncache_page(cookie, page);
+}
+
Index: cifs-2.6/fs/cifs/fscache.h
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.h
+++ cifs-2.6/fs/cifs/fscache.h
@@ -48,6 +48,16 @@ extern void cifs_fscache_release_inode_c
 extern void cifs_fscache_set_inode_cookie(struct inode *, struct file *);
 extern void cifs_fscache_reset_inode_cookie(struct inode *);
 
+extern void __cifs_fscache_invalidate_page(struct page *, struct inode *);
+extern int cifs_fscache_release_page(struct page *page, gfp_t gfp);
+
+static inline void cifs_fscache_invalidate_page(struct page *page,
+					       struct inode *inode)
+{
+	if (PageFsCache(page))
+		__cifs_fscache_invalidate_page(page, inode);
+}
+
 #else /* CONFIG_CIFS_FSCACHE */
 static inline int cifs_fscache_register(void) { return 0; }
 static inline void cifs_fscache_unregister(void) {}
@@ -64,7 +74,13 @@ static inline void cifs_fscache_release_
 static inline void cifs_fscache_set_inode_cookie(struct inode *inode,
 						 struct file *filp) {}
 static inline void cifs_fscache_reset_inode_cookie(struct inode *inode) {}
+static inline void cifs_fscache_release_page(struct page *page, gfp_t gfp)
+{
+	return 1; /* May release page */
+}
 
+static inline int cifs_fscache_invalidate_page(struct page *page,
+			struct inode *) {}
 
 #endif /* CONFIG_CIFS_FSCACHE */
 

[PATCH 07/09] cifs: store pages into local cache

[Suresh Jayaraman]

Store pages from an CIFS inode into the data storage object associated with
that inode.

Signed-off-by: Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org>
---
 fs/cifs/file.c    |    7 +++++++
 fs/cifs/fscache.c |   11 +++++++++++
 fs/cifs/fscache.h |   11 +++++++++++
 3 files changed, 29 insertions(+)

Index: cifs-2.6/fs/cifs/file.c
===================================================================
--- cifs-2.6.orig/fs/cifs/file.c
+++ cifs-2.6/fs/cifs/file.c
@@ -1948,6 +1948,9 @@ static void cifs_copy_cache_pages(struct
 		SetPageUptodate(page);
 		unlock_page(page);
 		data += PAGE_CACHE_SIZE;
+
+		/* add page to FS-Cache */
+		cifs_readpage_to_fscache(mapping->host, page);
 	}
 	return;
 }
@@ -2117,6 +2120,10 @@ static int cifs_readpage_worker(struct f
 
 	flush_dcache_page(page);
 	SetPageUptodate(page);
+
+	/* send this page to the cache */
+	cifs_readpage_to_fscache(file->f_path.dentry->d_inode, page);
+
 	rc = 0;
 
 io_error:
Index: cifs-2.6/fs/cifs/fscache.c
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.c
+++ cifs-2.6/fs/cifs/fscache.c
@@ -140,6 +140,17 @@ int cifs_fscache_release_page(struct pag
 	return 1;
 }
 
+void __cifs_readpage_to_fscache(struct inode *inode, struct page *page)
+{
+	int ret;
+
+	cFYI(1, "CIFS: readpage_to_fscache(fsc: %p, p: %p, i: %p",
+			CIFS_I(inode)->fscache, page, inode);
+	ret = fscache_write_page(CIFS_I(inode)->fscache, page, GFP_KERNEL);
+	if (ret != 0)
+		fscache_uncache_page(CIFS_I(inode)->fscache, page);
+}
+
 void __cifs_fscache_invalidate_page(struct page *page, struct inode *inode)
 {
 	struct cifsInodeInfo *cifsi = CIFS_I(inode);
Index: cifs-2.6/fs/cifs/fscache.h
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.h
+++ cifs-2.6/fs/cifs/fscache.h
@@ -51,6 +51,8 @@ extern void cifs_fscache_reset_inode_coo
 extern void __cifs_fscache_invalidate_page(struct page *, struct inode *);
 extern int cifs_fscache_release_page(struct page *page, gfp_t gfp);
 
+extern void __cifs_readpage_to_fscache(struct inode *, struct page *);
+
 static inline void cifs_fscache_invalidate_page(struct page *page,
 					       struct inode *inode)
 {
@@ -58,6 +60,13 @@ static inline void cifs_fscache_invalida
 		__cifs_fscache_invalidate_page(page, inode);
 }
 
+static inline void cifs_readpage_to_fscache(struct inode *inode,
+					    struct page *page)
+{
+	if (PageFsCache(page))
+		__cifs_readpage_to_fscache(inode, page);
+}
+
 #else /* CONFIG_CIFS_FSCACHE */
 static inline int cifs_fscache_register(void) { return 0; }
 static inline void cifs_fscache_unregister(void) {}
@@ -81,6 +90,8 @@ static inline void cifs_fscache_release_
 
 static inline int cifs_fscache_invalidate_page(struct page *page,
 			struct inode *) {}
+static inline void cifs_readpage_to_fscache(struct inode *inode,
+			struct page *page) {}
 
 #endif /* CONFIG_CIFS_FSCACHE */
 

[PATCH 08/09] cifs: read pages from FS-Cache

[Suresh Jayaraman]

Read pages from a FS-Cache data storage object into a CIFS inode.

Signed-off-by: Suresh Jayaraman <sjayaraman@suse.de>
Acked-by: David Howells <dhowells@redhat.com>
---
 fs/cifs/file.c    |   17 ++++++++++++
 fs/cifs/fscache.c |   73 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/cifs/fscache.h |   40 ++++++++++++++++++++++++++++-
 3 files changed, 129 insertions(+), 1 deletion(-)

Index: cifs-2.6/fs/cifs/file.c
===================================================================
--- cifs-2.6.orig/fs/cifs/file.c
+++ cifs-2.6/fs/cifs/file.c
@@ -1981,6 +1981,15 @@ static int cifs_readpages(struct file *f
 	cifs_sb = CIFS_SB(file->f_path.dentry->d_sb);
 	pTcon = cifs_sb->tcon;
 
+	/*
+	 * Reads as many pages as possible from fscache. Returns -ENOBUFS
+	 * immediately if the cookie is negative
+	 */
+	rc = cifs_readpages_from_fscache(mapping->host, mapping, page_list,
+					 &num_pages);
+	if (rc == 0)
+		goto read_complete;
+
 	cFYI(DBG2, "rpages: num pages %d", num_pages);
 	for (i = 0; i < num_pages; ) {
 		unsigned contig_pages;
@@ -2091,6 +2100,7 @@ static int cifs_readpages(struct file *f
 		smb_read_data = NULL;
 	}
 
+read_complete:
 	FreeXid(xid);
 	return rc;
 }
@@ -2101,6 +2111,11 @@ static int cifs_readpage_worker(struct f
 	char *read_data;
 	int rc;
 
+	/* Is the page cached? */
+	rc = cifs_readpage_from_fscache(file->f_path.dentry->d_inode, page);
+	if (rc == 0)
+		goto read_complete;
+
 	page_cache_get(page);
 	read_data = kmap(page);
 	/* for reads over a certain size could initiate async read ahead */
@@ -2129,6 +2144,8 @@ static int cifs_readpage_worker(struct f
 io_error:
 	kunmap(page);
 	page_cache_release(page);
+
+read_complete:
 	return rc;
 }
 
Index: cifs-2.6/fs/cifs/fscache.c
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.c
+++ cifs-2.6/fs/cifs/fscache.c
@@ -140,6 +140,79 @@ int cifs_fscache_release_page(struct pag
 	return 1;
 }
 
+static void cifs_readpage_from_fscache_complete(struct page *page, void *ctx,
+						int error)
+{
+	cFYI(1, "CFS: readpage_from_fscache_complete (0x%p/%d)",
+			page, error);
+	if (!error)
+		SetPageUptodate(page);
+	unlock_page(page);
+}
+
+/*
+ * Retrieve a page from FS-Cache
+ */
+int __cifs_readpage_from_fscache(struct inode *inode, struct page *page)
+{
+	int ret;
+
+	cFYI(1, "CIFS: readpage_from_fscache(fsc:%p, p:%p, i:0x%p",
+			CIFS_I(inode)->fscache, page, inode);
+	ret = fscache_read_or_alloc_page(CIFS_I(inode)->fscache, page,
+					 cifs_readpage_from_fscache_complete,
+					 NULL,
+					 GFP_KERNEL);
+	switch (ret) {
+
+	case 0: /* page found in fscache, read submitted */
+		cFYI(1, "CIFS: readpage_from_fscache: submitted");
+		return ret;
+	case -ENOBUFS:	/* page won't be cached */
+	case -ENODATA:	/* page not in cache */
+		cFYI(1, "CIFS: readpage_from_fscache %d", ret);
+		return 1;
+
+	default:
+		cERROR(1, "unknown error ret = %d", ret);
+	}
+	return ret;
+}
+
+/*
+ * Retrieve a set of pages from FS-Cache
+ */
+int __cifs_readpages_from_fscache(struct inode *inode,
+				struct address_space *mapping,
+				struct list_head *pages,
+				unsigned *nr_pages)
+{
+	int ret;
+
+	cFYI(1, "CIFS: __cifs_readpages_from_fscache (0x%p/%u/0x%p)",
+			CIFS_I(inode)->fscache, *nr_pages, inode);
+	ret = fscache_read_or_alloc_pages(CIFS_I(inode)->fscache, mapping,
+					  pages, nr_pages,
+					  cifs_readpage_from_fscache_complete,
+					  NULL,
+					  mapping_gfp_mask(mapping));
+	switch (ret) {
+	case 0:	/* read submitted to the cache for all pages */
+		cFYI(1, "CIFS: readpages_from_fscache: submitted");
+		return ret;
+
+	case -ENOBUFS:	/* some pages are not cached and can't be */
+	case -ENODATA:	/* some pages are not cached */
+		cFYI(1, "CIFS: readpages_from_fscache: no page");
+		return 1;
+
+	default:
+		cFYI(1, "unknown error ret = %d", ret);
+	}
+
+	return ret;
+}
+
 void __cifs_readpage_to_fscache(struct inode *inode, struct page *page)
 {
 	int ret;
Index: cifs-2.6/fs/cifs/fscache.h
===================================================================
--- cifs-2.6.orig/fs/cifs/fscache.h
+++ cifs-2.6/fs/cifs/fscache.h
@@ -32,7 +32,6 @@ extern const struct fscache_cookie_def c
 extern const struct fscache_cookie_def cifs_fscache_super_index_def;
 extern const struct fscache_cookie_def cifs_fscache_inode_object_def;
 
-
 extern int cifs_fscache_register(void);
 extern void cifs_fscache_unregister(void);
 
@@ -50,6 +49,11 @@ extern void cifs_fscache_reset_inode_coo
 
 extern void __cifs_fscache_invalidate_page(struct page *, struct inode *);
 extern int cifs_fscache_release_page(struct page *page, gfp_t gfp);
+extern int __cifs_readpage_from_fscache(struct inode *, struct page *);
+extern int __cifs_readpages_from_fscache(struct inode *,
+					 struct address_space *,
+					 struct list_head *,
+					 unsigned *);
 
 extern void __cifs_readpage_to_fscache(struct inode *, struct page *);
 
@@ -60,6 +64,26 @@ static inline void cifs_fscache_invalida
 		__cifs_fscache_invalidate_page(page, inode);
 }
 
+static inline int cifs_readpage_from_fscache(struct inode *inode,
+					     struct page *page)
+{
+	if (CIFS_I(inode)->fscache)
+		return __cifs_readpage_from_fscache(inode, page);
+
+	return -ENOBUFS;
+}
+
+static inline int cifs_readpages_from_fscache(struct inode *inode,
+					      struct address_space *mapping,
+					      struct list_head *pages,
+					      unsigned *nr_pages)
+{
+	if (CIFS_I(inode)->fscache)
+		return __cifs_readpages_from_fscache(inode, mapping, pages,
+						     nr_pages);
+	return -ENOBUFS;
+}
+
 static inline void cifs_readpage_to_fscache(struct inode *inode,
 					    struct page *page)
 {
@@ -90,6 +114,20 @@ static inline void cifs_fscache_release_
 
 static inline int cifs_fscache_invalidate_page(struct page *page,
 			struct inode *) {}
+static inline int
+cifs_readpage_from_fscache(struct inode *inode, struct page *page)
+{
+	return -ENOBUFS;
+}
+
+static inline int cifs_readpages_from_fscache(struct inode *inode,
+					      struct address_space *mapping,
+					      struct list_head *pages,
+					      unsigned *nr_pages)
+{
+	return -ENOBUFS;
+}
+
 static inline void cifs_readpage_to_fscache(struct inode *inode,
 			struct page *page) {}
 

[PATCH 09/09] cifs: add mount option to enable local caching

[Suresh Jayaraman]

Add a mount option 'fsc' to enable local caching on CIFS.

I considered adding a separate debug bit for caching, but it appears that
debugging would be relatively easier with the normal CIFS_INFO level.

As the cifs-utils (userspace) changes are not done yet, this patch enables
'fsc' by default to enable testing.

Signed-off-by: Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org>
Acked-by: David Howells <dhowells-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 fs/cifs/cifs_fs_sb.h |    1 +
 fs/cifs/connect.c    |    8 ++++++++
 2 files changed, 9 insertions(+)

Index: cifs-2.6/fs/cifs/cifs_fs_sb.h
===================================================================
--- cifs-2.6.orig/fs/cifs/cifs_fs_sb.h
+++ cifs-2.6/fs/cifs/cifs_fs_sb.h
@@ -35,6 +35,7 @@
 #define CIFS_MOUNT_DYNPERM      0x1000 /* allow in-memory only mode setting   */
 #define CIFS_MOUNT_NOPOSIXBRL   0x2000 /* mandatory not posix byte range lock */
 #define CIFS_MOUNT_NOSSYNC      0x4000 /* don't do slow SMBflush on every sync*/
+#define CIFS_MOUNT_FSCACHE	0x8000 /* local caching enabled */
 
 struct cifs_sb_info {
 	struct cifsTconInfo *tcon;	/* primary mount */
Index: cifs-2.6/fs/cifs/connect.c
===================================================================
--- cifs-2.6.orig/fs/cifs/connect.c
+++ cifs-2.6/fs/cifs/connect.c
@@ -98,6 +98,7 @@ struct smb_vol {
 	bool noblocksnd:1;
 	bool noautotune:1;
 	bool nostrictsync:1; /* do not force expensive SMBflush on every sync */
+	bool fsc:1;	/* enable fscache */
 	unsigned int rsize;
 	unsigned int wsize;
 	bool sockopt_tcp_nodelay:1;
@@ -843,6 +844,9 @@ cifs_parse_mount_options(char *options,
 	/* default to using server inode numbers where available */
 	vol->server_ino = 1;
 
+	/* XXX: default to fsc for testing until mount.cifs pieces are done */
+	vol->fsc = 1;
+
 	if (!options)
 		return 1;
 
@@ -1332,6 +1336,8 @@ cifs_parse_mount_options(char *options,
 			printk(KERN_WARNING "CIFS: Mount option noac not "
 				"supported. Instead set "
 				"/proc/fs/cifs/LookupCacheEnabled to 0\n");
+		} else if (strnicmp(data, "fsc", 3) == 0) {
+			vol->fsc = true;
 		} else
 			printk(KERN_WARNING "CIFS: Unknown mount option %s\n",
 						data);
@@ -2405,6 +2411,8 @@ static void setup_cifs_sb(struct smb_vol
 		cifs_sb->mnt_cifs_flags |= CIFS_MOUNT_OVERR_GID;
 	if (pvolume_info->dynperm)
 		cifs_sb->mnt_cifs_flags |= CIFS_MOUNT_DYNPERM;
+	if (pvolume_info->fsc)
+		cifs_sb->mnt_cifs_flags |= CIFS_MOUNT_FSCACHE;
 	if (pvolume_info->direct_io) {
 		cFYI(1, "mounting share using direct i/o");
 		cifs_sb->mnt_cifs_flags |= CIFS_MOUNT_DIRECT_IO;

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Scott Lovenberg]

On 7/5/2010 8:41 AM, Suresh Jayaraman wrote:
> This patchset is a second try at adding persistent, local caching facility for
> CIFS using the FS-Cache interface.
>
>    
Just wondering, have you bench marked this at all?  I'd be interested to 
see how this compares (performance and scaling) to an oplock-centric 
design.

I'd hazard a guess that with pipelining support in SMB2 the performance 
will be even better since you can have a hot cache and more requests in 
flight.

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Steve French]

On Wed, Jul 14, 2010 at 12:41 PM, Scott Lovenberg
<scott.lovenberg-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On 7/5/2010 8:41 AM, Suresh Jayaraman wrote:
>>
>> This patchset is a second try at adding persistent, local caching facility
>> for
>> CIFS using the FS-Cache interface.
>>
>>
>
> Just wondering, have you bench marked this at all?  I'd be interested to see
> how this compares (performance and scaling) to an oplock-centric design.
>
> I'd hazard a guess that with pipelining support in SMB2 the performance will
> be even better since you can have a hot cache and more requests in flight.

Yes - very plausibly



-- 
Thanks,

Steve

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Suresh Jayaraman]

On 07/14/2010 08:09 PM, Steve French wrote:
> On Wed, Jul 14, 2010 at 12:41 PM, Scott Lovenberg
> <scott.lovenberg-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On 7/5/2010 8:41 AM, Suresh Jayaraman wrote:
>>>
>>> This patchset is a second try at adding persistent, local caching facility
>>> for
>>> CIFS using the FS-Cache interface.
>>>
>>>
>>
>> Just wondering, have you bench marked this at all? �I'd be interested to see
>> how this compares (performance and scaling) to an oplock-centric design.
>>

Yes, I have done a few performance benchmarks with the cifs client (and
not SMB2) and I'll post them early nextweek when I'm back (as I'm
travelling now).

However, I have never done scalability tests (not sure whether there is
a way to simulate a number of cifs clients).

>> I'd hazard a guess that with pipelining support in SMB2 the performance will
>> be even better since you can have a hot cache and more requests in flight.
> 
> Yes - very plausibly
> 

I have not tried the new SMB2 client. But, it seems the pipelining
support, Oplocks (only Level II kind) could help improve performance.


Thanks,

-- 
Suresh Jayaraman

Re: [PATCH 04/09] cifs: define superblock-level cache index objects and register them

[Jeff Layton]

On Mon,  5 Jul 2010 18:12:27 +0530
Suresh Jayaraman <sjayaraman@suse.de> wrote:

> Define superblock-level cache index objects (managed by cifsTconInfo structs).
> Each superblock object is created in a server-level index object and in itself
> an index into which inode-level objects are inserted.
> 
> The superblock object is keyed by sharename. The UniqueId/IndexNumber is used to
> validate that the exported share is the same since we accessed it last time.
> 
> Signed-off-by: Suresh Jayaraman <sjayaraman@suse.de>

Hmm...Steve started merging these already but I've just now had the
chance to review them.

This approach may be a problem. It seems to make the assumption that
there is only a single tcon per superblock. How exactly will this work
when there are multiple tcons per superblock as will be the case with
multisession mounts?

By having a cache cookie per tcon, will that mean that you'll
potentially have multiple versions of cached inodes (one for each tcon)?

-- 
Jeff Layton <jlayton@samba.org>

Re: [PATCH 04/09] cifs: define superblock-level cache index objects and register them

[Jeff Layton]

On Tue, 20 Jul 2010 08:53:27 -0400
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:

> On Mon,  5 Jul 2010 18:12:27 +0530
> Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org> wrote:
> 
> > Define superblock-level cache index objects (managed by cifsTconInfo structs).
> > Each superblock object is created in a server-level index object and in itself
> > an index into which inode-level objects are inserted.
> > 
> > The superblock object is keyed by sharename. The UniqueId/IndexNumber is used to
> > validate that the exported share is the same since we accessed it last time.
> > 
> > Signed-off-by: Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org>
> 
> Hmm...Steve started merging these already but I've just now had the
> chance to review them.
> 
> This approach may be a problem. It seems to make the assumption that
> there is only a single tcon per superblock. How exactly will this work
> when there are multiple tcons per superblock as will be the case with
> multisession mounts?
> 
> By having a cache cookie per tcon, will that mean that you'll
> potentially have multiple versions of cached inodes (one for each tcon)?
> 

Ahh nm. This shouldn't be a problem since this key is based on the
sharename only and that will be the same between the multiple tcons.

Please disregard!
-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: [PATCH 04/09] cifs: define superblock-level cache index objects and register them

[Suresh Jayaraman]

On 07/20/2010 07:07 PM, Jeff Layton wrote:
> On Tue, 20 Jul 2010 08:53:27 -0400
> Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> 
>> On Mon,  5 Jul 2010 18:12:27 +0530
>> Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org> wrote:
>>
>>> Define superblock-level cache index objects (managed by cifsTconInfo structs).
>>> Each superblock object is created in a server-level index object and in itself
>>> an index into which inode-level objects are inserted.
>>>
>>> The superblock object is keyed by sharename. The UniqueId/IndexNumber is used to
>>> validate that the exported share is the same since we accessed it last time.
>>>
>>> Signed-off-by: Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org>
>>
>> Hmm...Steve started merging these already but I've just now had the
>> chance to review them.
>>
>> This approach may be a problem. It seems to make the assumption that
>> there is only a single tcon per superblock. How exactly will this work
>> when there are multiple tcons per superblock as will be the case with
>> multisession mounts?
>>
>> By having a cache cookie per tcon, will that mean that you'll
>> potentially have multiple versions of cached inodes (one for each tcon)?
>>

In case of multisession mounts, there is a cache cookie per tcon, but
they will point to the same cached inodes.

> Ahh nm. This shouldn't be a problem since this key is based on the
> sharename only and that will be the same between the multiple tcons.

yeah. I think the description could be a bit more clear..


Thanks,

-- 
Suresh Jayaraman

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Suresh Jayaraman]

On 07/15/2010 09:53 PM, Suresh Jayaraman wrote:
> On 07/14/2010 08:09 PM, Steve French wrote:
>> On Wed, Jul 14, 2010 at 12:41 PM, Scott Lovenberg
>> <scott.lovenberg-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> On 7/5/2010 8:41 AM, Suresh Jayaraman wrote:
>>>>
>>>> This patchset is a second try at adding persistent, local caching facility
>>>> for
>>>> CIFS using the FS-Cache interface.
>>>>
>>>>
>>>
>>> Just wondering, have you bench marked this at all? �I'd be interested to see
>>> how this compares (performance and scaling) to an oplock-centric design.
>>>
> 
> Yes, I have done a few performance benchmarks with the cifs client (and
> not SMB2) and I'll post them early nextweek when I'm back (as I'm
> travelling now).
> 

Here are some results from my benchmarking:

Environment
------------

I'm using my T60p laptop as the CIFS server (running Samba) and one of
my test machines as CIFS client, connected over an ethernet of reported
speed 1000 Mb/s. The TCP bandwidth as seen by a pair of netcats between
the client and the server is about 786.24 Mb/s.

Client has a 2.8 GHz Pentium D CPU with 2GB RAM
Server has a 2.33GHz Core2 CPU (T7600) with 2GB RAM


Test
-----
The benchmark involves pulling a 200 MB file over CIFS to the client
using cat to /dev/zero under `time'. The wall clock time reported was
recorded.

Note
----
   - The client was rebooted after each test, but the server was not.
   - The entire file was loaded into RAM on the server before each test
     to eliminate disk I/O latencies on that end.
   - A seperate partition of size 4GB has been dedicated for the cache.
   - There were no other CIFS client that was accessing the Server when
     the tests were performed.


First, the test was run on the server twice and the second result was
recorded (noted as Server below).

Secondly, the client was rebooted and the test was run with cachefiled
not running and was recorded (noted as None below).

Next, the client was rebooted, the cache contents (if any) were erased
with mkfs.ext3 and test was run again with cachefilesd running (noted as
COLD)

Next the client was rebooted, tests were run with cachefilesd running
this time with a populated cache (noted as HOT).

Finally, the test was run again without unmounting, stopping cachefiled
or rebooting to ensure pagecache is valid (noted as PGCACHE).

The benchmark was repeated twice:

Cache (state)	Run #1		Run#2
=============  =======		=======
Server		0.107 s		0.090 s
None		6.497 s		6.440 s
COLD		6.707 s		6.635 s
HOT		5.286 s		5.078 s
PGCACHE		0.090 s		0.091 s

As it can been seen, the performance while reading when data is cache
hot (disk) is not great as the network link is a Gigabit ethernet (with
server having working set in memory) which is mostly expected. (I could
not get access to a slower network (say 100 Mb/s) where the real
performance boost could be evident).


Thanks,


-- 
Suresh Jayaraman

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[David Howells]

Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org> wrote:

> As it can been seen, the performance while reading when data is cache
> hot (disk) is not great as the network link is a Gigabit ethernet (with
> server having working set in memory) which is mostly expected.

That's what I see with NFS and AFS too.

> (I could not get access to a slower network (say 100 Mb/s) where the real
> performance boost could be evident).

ethtool?

David

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Jeff Layton]

On Thu, 22 Jul 2010 18:40:00 +0100
David Howells <dhowells-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org> wrote:
> 
> > As it can been seen, the performance while reading when data is cache
> > hot (disk) is not great as the network link is a Gigabit ethernet (with
> > server having working set in memory) which is mostly expected.
> 
> That's what I see with NFS and AFS too.
> 
> > (I could not get access to a slower network (say 100 Mb/s) where the real
> > performance boost could be evident).
> 
> ethtool?
> 

There's also the "netem" thingy that allows you to add arbitrary
latencies to packets (for simulating long-haul network links).

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Andreas Dilger]

On 2010-07-22, at 11:40, David Howells wrote:
> Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org> wrote:
>> As it can been seen, the performance while reading when data is cache
>> hot (disk) is not great as the network link is a Gigabit ethernet (with
>> server having working set in memory) which is mostly expected.
> 
> That's what I see with NFS and AFS too.
> 
>> (I could not get access to a slower network (say 100 Mb/s) where the real
>> performance boost could be evident).

More interesting than a slow network is testing with more clients.  10 clients should be able to get 10x the read performance from the client-local cache, probably more than the server's peak disk/network bandwidth.

Cheers, Andreas

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Stef Bon]

In my opinion there should be article published about this, describing
fs-cache generally, and these kinds of benchmarks!

Using fs-cache for network filesystem is an important issue, and
should get "exposure".

Stef Bon

2010/7/23 Andreas Dilger <adilger@dilger.ca>:
> On 2010-07-22, at 11:40, David Howells wrote:
>> Suresh Jayaraman <sjayaraman@suse.de> wrote:
>>> As it can been seen, the performance while reading when data is cache
>>> hot (disk) is not great as the network link is a Gigabit ethernet (with
>>> server having working set in memory) which is mostly expected.
>>
>> That's what I see with NFS and AFS too.
>>
>>> (I could not get access to a slower network (say 100 Mb/s) where the real
>>> performance boost could be evident).
>
> More interesting than a slow network is testing with more clients.  10 clients should be able to get 10x the read performance from the client-local cache, probably more than the server's peak disk/network bandwidth.
>
> Cheers, Andreas
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Suresh Jayaraman]

On 07/22/2010 11:10 PM, David Howells wrote:
> Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org> wrote:
> 
>> As it can been seen, the performance while reading when data is cache
>> hot (disk) is not great as the network link is a Gigabit ethernet (with
>> server having working set in memory) which is mostly expected.
> 
> That's what I see with NFS and AFS too.
> 
>> (I could not get access to a slower network (say 100 Mb/s) where the real
>> performance boost could be evident).
> 
> ethtool?
> 

Thanks for the pointer. Here are the results on a 100Mb/s network:


Environment
------------

I'm using my T60p laptop as the CIFS server (running Samba) and one of
my test machines as CIFS client, connected over an ethernet of reported
speed 1000 Mb/s. ethtool was used to throttle the speed to 100 Mb/s. The
TCP bandwidth as seen by a pair of netcats between the client and the
server is about 89.555 Mb/s.

Client has a 2.8 GHz Pentium D CPU with 2GB RAM
Server has a 2.33GHz Core2 CPU (T7600) with 2GB RAM


Test
-----
The benchmark involves pulling a 200 MB file over CIFS to the client
using cat to /dev/zero under `time'. The wall clock time reported was
recorded.

Note
----
   - The client was rebooted after each test, but the server was not.
   - The entire file was loaded into RAM on the server before each test
     to eliminate disk I/O latencies on that end.
   - A seperate partition of size 4GB has been dedicated for the cache.
   - There were no other CIFS client that was accessing the Server when
     the tests were performed.


First, the test was run on the server twice and the second result was
recorded (noted as Server below).

Secondly, the client was rebooted and the test was run with cachefiled
not running and was recorded (noted as None below).

Next, the client was rebooted, the cache contents (if any) were erased
with mkfs.ext3 and test was run again with cachefilesd running (noted as
COLD)

Next the client was rebooted, tests were run with cachefilesd running
this time with a populated disk cache (noted as HOT).

Finally, the test was run again without unmounting, stopping cachefiled
or rebooting to ensure pagecache is valid (noted as PGCACHE).

The benchmark was repeated twice:

Cache (state)	Run #1		Run#2
=============  =======		=======
Server		 0.104 s	 0.107 s
None		26.042 s	26.576 s
COLD		26.703 s        26.787 s
HOT		 5.115 s	 5.147 s
PGCACHE		 0.091 s	 0.092 s

I think the results are inline with the expectations given the speed
reported by netcat.

As noted by Andreas, the read performance with more number of clients
would be more interesting as the cache can positively impact the
scalability. However, I don't have a number of clients or know a way to
simulate a large number of cifs clients. The cache also can positively
impact the performance on heavily loaded network and/or server due to
reduction of network calls to the server.

Also, it should be noted that local caching is not for all workloads and
a few workloads could suffer (for e.g. read-once type workloads).


Thanks,

-- 
Suresh Jayaraman

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Steve French]

On Fri, Jul 23, 2010 at 6:57 AM, Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org> wrote:
> On 07/22/2010 11:10 PM, David Howells wrote:
>> Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org> wrote:
>>
>>> As it can been seen, the performance while reading when data is cache
>>> hot (disk) is not great as the network link is a Gigabit ethernet (with
>>> server having working set in memory) which is mostly expected.
>>
>> That's what I see with NFS and AFS too.
>>
>>> (I could not get access to a slower network (say 100 Mb/s) where the real
>>> performance boost could be evident).
>>
>> ethtool?
>>
>
> Thanks for the pointer. Here are the results on a 100Mb/s network:
 <snip>
Excellent data - thx

> As noted by Andreas, the read performance with more number of clients
> would be more interesting as the cache can positively impact the
> scalability. However, I don't have a number of clients or know a way to
> simulate a large number of cifs clients.

You could simulate increased load by running multiple smbtorture
instances from each real client, and perhaps some local dbench like
activity run locally on the server.

> The cache also can positively
> impact the performance on heavily loaded network and/or server due to
> reduction of network calls to the server.

Reminds me a little about the discussions during the last few SMB2 plugfests:

http://channel9.msdn.com/posts/Darryl/Peer-Content-Branch-Caching-and-Retrieval-Presentation/

and an earlier one (I couldn't find the newer version of this talk)
http://channel9.msdn.com/pdc2008/ES23/

and


-- 
Thanks,

Steve

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Suresh Jayaraman]

On 07/23/2010 02:05 PM, Stef Bon wrote:
> In my opinion there should be article published about this, describing
> fs-cache generally, and these kinds of benchmarks!

FS-Cache is nicely documented on
Documentation/filesystems/caching/fscache.txt present in the kernel
source. IIRC, LWN - http://lwn.net/ had published a couple of articles
about FS-Cache. I agree that there is little information available about
the use-cases and benchmarks.

What do you mean exactly by article? More blog entries or article in
magazines or whitepaper sort? Which one do you think has a good chance
of reaching the target audience?

Thanks,

> Using fs-cache for network filesystem is an important issue, and
> should get "exposure".
> 
> Stef Bon
> 
> 2010/7/23 Andreas Dilger <adilger-m1MBpc4rdrD3fQ9qLvQP4Q@public.gmane.org>:
>> On 2010-07-22, at 11:40, David Howells wrote:
>>> Suresh Jayaraman <sjayaraman-l3A5Bk7waGM@public.gmane.org> wrote:
>>>> As it can been seen, the performance while reading when data is cache
>>>> hot (disk) is not great as the network link is a Gigabit ethernet (with
>>>> server having working set in memory) which is mostly expected.
>>>
>>> That's what I see with NFS and AFS too.
>>>
>>>> (I could not get access to a slower network (say 100 Mb/s) where the real
>>>> performance boost could be evident).
>>
>> More interesting than a slow network is testing with more clients.  10 clients should be able to get 10x the read performance from the client-local cache, probably more than the server's peak disk/network bandwidth.
>>

-- 
Suresh Jayaraman

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Stef Bon]

2010/7/23 Suresh Jayaraman <sjayaraman@suse.de>:
> On 07/23/2010 02:05 PM, Stef Bon wrote:
>> In my opinion there should be article published about this, describing
>> fs-cache generally, and these kinds of benchmarks!
>
> FS-Cache is nicely documented on
> Documentation/filesystems/caching/fscache.txt present in the kernel
> source. IIRC, LWN - http://lwn.net/ had published a couple of articles
> about FS-Cache. I agree that there is little information available about
> the use-cases and benchmarks.
>
> What do you mean exactly by article? More blog entries or article in
> magazines or whitepaper sort? Which one do you think has a good chance
> of reaching the target audience?
>

Well,

I do not have an answer right now, I guess articles in general
magazines, not exactly the Linux Community,
but ICT in general.

What strikes me most, is that there are a lot of people, you, me,
probably everyone reading this, are working on a lot of projects,
creating excellent software, making the Linux better and better.

But does it get exposure? Is the public aware of what happens with
Linux, does "they" know Linux or at least heard of?
No! That's a pity.
Everybody knows the iPad, why not Linux?

Linux needs better marketing!!


Stef Bon

Re: [PATCH 00/09] cifs: local caching support using FS-Cache

[Scott Lovenberg]

On 7/22/2010 5:28 AM, Suresh Jayaraman wrote:
> Here are some results from my benchmarking:
>
> Environment
> ------------
>
> I'm using my T60p laptop as the CIFS server (running Samba) and one of
> my test machines as CIFS client, connected over an ethernet of reported
> speed 1000 Mb/s. The TCP bandwidth as seen by a pair of netcats between
> the client and the server is about 786.24 Mb/s.
>
> Client has a 2.8 GHz Pentium D CPU with 2GB RAM
> Server has a 2.33GHz Core2 CPU (T7600) with 2GB RAM
>
>
> Test
> -----
> The benchmark involves pulling a 200 MB file over CIFS to the client
> using cat to /dev/zero under `time'. The wall clock time reported was
> recorded.
>
> Note
> ----
>     - The client was rebooted after each test, but the server was not.
>     - The entire file was loaded into RAM on the server before each test
>       to eliminate disk I/O latencies on that end.
>     - A seperate partition of size 4GB has been dedicated for the cache.
>     - There were no other CIFS client that was accessing the Server when
>       the tests were performed.
>
>
> First, the test was run on the server twice and the second result was
> recorded (noted as Server below).
>
> Secondly, the client was rebooted and the test was run with cachefiled
> not running and was recorded (noted as None below).
>
> Next, the client was rebooted, the cache contents (if any) were erased
> with mkfs.ext3 and test was run again with cachefilesd running (noted as
> COLD)
>
> Next the client was rebooted, tests were run with cachefilesd running
> this time with a populated cache (noted as HOT).
>
> Finally, the test was run again without unmounting, stopping cachefiled
> or rebooting to ensure pagecache is valid (noted as PGCACHE).
>
> The benchmark was repeated twice:
>
> Cache (state)	Run #1		Run#2
> =============  =======		=======
> Server		0.107 s		0.090 s
> None		6.497 s		6.440 s
> COLD		6.707 s		6.635 s
> HOT		5.286 s		5.078 s
> PGCACHE		0.090 s		0.091 s
>
> As it can been seen, the performance while reading when data is cache
> hot (disk) is not great as the network link is a Gigabit ethernet (with
> server having working set in memory) which is mostly expected. (I could
> not get access to a slower network (say 100 Mb/s) where the real
> performance boost could be evident).
>
>
> Thanks,
>
>
>    

Suresh, thanks for taking the time to run these bench marks. :)