[lustre-devel] [PATCH 00/14] lustre: fixes for many test failures

[lustre-devel] [PATCH 00/14] lustre: fixes for many test failures

[James Simmons]

This patch series contains mostly fixes to resolve the majority of
the testing failures seen. Besides that are UAPI headers fixes to
prepare for when lustre will leave the staging tree. Include Neil's
dump_page_cache patch as well since it was landed in the OpenSFS
branch and the migration of lu_objects to rhashtables depended on
it.

James Simmons (9):
  lustre: uapi: replace cfs_size_* macros with __ALIGN_KERNEL
  lustre: uapi: final fixes for UAPI support.
  lustre: llite: op_data->op_valid is incorrect in ll_dir_getstripe
  lustre: llite: return correct amount of bytes for short operations
  lustre: llite: user xattr_prefix() to create xattr full name
  lustre: llite: conditionally set it_flags in ll_file_open
  lustre: llite: fix file migration
  lustre: sysfs: temporary work around for sysfs naming
  lustre: libcfs: hide struct cfs_cpt_table internals

John L. Hammond (1):
  lustre: llite: add LL_IOC_FUTIMES_3

Lai Siyao (1):
  lustre: llite: improve getdirstripe interface

Minh Diep (1):
  lustre: uapi: fix lustre_user.h to C++ compatible

NeilBrown (1):
  lustre: llite: change how "dump_page_cache" walks a hash table

Patrick Farrell (1):
  lustre: llite: Add S_NOSEC support

 .../lustre/include/linux/libcfs/libcfs_cpu.h       |  39 +----
 .../staging/lustre/include/linux/lnet/lib-types.h  |   1 +
 .../lustre/include/uapi/linux/lnet/lnet-types.h    |   1 -
 .../lustre/include/uapi/linux/lustre/lustre_idl.h  |  14 +-
 .../lustre/include/uapi/linux/lustre/lustre_user.h |  57 ++++---
 drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c    |  34 ++++
 .../staging/lustre/lustre/include/obd_support.h    |   2 +
 drivers/staging/lustre/lustre/llite/dir.c          |   3 +-
 drivers/staging/lustre/lustre/llite/file.c         |  53 ++++++-
 drivers/staging/lustre/lustre/llite/llite_lib.c    |  25 ++-
 drivers/staging/lustre/lustre/llite/namei.c        |   1 +
 drivers/staging/lustre/lustre/llite/vvp_dev.c      | 173 ++++++++++-----------
 drivers/staging/lustre/lustre/llite/vvp_io.c       |  10 +-
 drivers/staging/lustre/lustre/llite/xattr.c        |   4 +-
 drivers/staging/lustre/lustre/mdc/mdc_lib.c        |   1 -
 drivers/staging/lustre/lustre/mgc/mgc_request.c    |   2 +-
 .../staging/lustre/lustre/obdclass/obd_config.c    |   2 +-
 17 files changed, 249 insertions(+), 173 deletions(-)

-- 
1.8.3.1

[lustre-devel] [PATCH 01/14] lustre: llite: Add S_NOSEC support

[James Simmons]

From: Patrick Farrell <paf@cray.com>

We must use the i_mutex to protect permission changes,
which means we need to take it when we write to a file with
the setuid or setgid bit set (as this removes those bits).

LU-8025 attempted to use IS_NOSEC to check for this case,
but did not actually add support for the S_NOSEC flag to
Lustre.

S_NOSEC was added in upstream kernel commit:
69b4573296469fd3f70cf7044693074980517067
But a key change requiring parallel filesystems to opt in
with a superblock flag was added in:
9e1f1de02c2275d7172e18dc4e7c2065777611bf

This patch adds the required support.

Specifically, Lustre should set S_NOSEC correctly when an
inode is created (ll_iget), but only for new inodes.
Setting it for existing inodes requires taking the i_mutex,
creating an unacceptable metadata performance impact in the
lookup code.

Existing inodes get S_NOSEC set either by a setattr call
(see below), or by the first writer to write to the node,
in file_remove_privs/file_remove_suid.  So it's OK not to
set S_NOSEC on all inodes in ll_iget.

Also, Lustre must clear S_NOSEC when it gets a metadata
update because another node could have changed permissions.
i_flags is already cleared in ll_update_inode, but we would
prefer to have S_NOSEC set whenever possible, so we want to
re-do the check after the update.

This requires holding the i_mutex to avoid a check/set race
with permissions changes.

We cannot easily take the i_mutex in ll_update_inode (it is
called from too many places, some of which already hold the
i_mutex).

It is acceptable to sometimes not have S_NOSEC set (since
occasionally taking the i_mutex when not needed is OK), so
we look at getting it set most of the time.

It looks like the primary concern is ll_md_setattr.  The
caller (ll_setattr_raw) takes the i_mutex before returning,
so we do the relevant call there.

This opens a window during which S_NOSEC is not set, but
again, this merely creates a situation where we take the
i_mutex unnecessarily, and should be rare in practice.

Testing with multiple writers shows that we only very
rarely attempt to take the i_mutex, so the performance
impact is minimal.

Signed-off-by: Patrick Farrell <paf@cray.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-8656
Reviewed-on: https://review.whamcloud.com/22853
Reviewed-by: James Simmons <uja.ornl@yahoo.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 drivers/staging/lustre/lustre/include/obd_support.h |  2 ++
 drivers/staging/lustre/lustre/llite/llite_lib.c     | 19 +++++++++++++++++--
 drivers/staging/lustre/lustre/llite/namei.c         |  1 +
 drivers/staging/lustre/lustre/llite/vvp_io.c        | 10 +++++++++-
 4 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/include/obd_support.h b/drivers/staging/lustre/lustre/include/obd_support.h
index 1832193..79875fa 100644
--- a/drivers/staging/lustre/lustre/include/obd_support.h
+++ b/drivers/staging/lustre/lustre/include/obd_support.h
@@ -442,6 +442,8 @@
 #define OBD_FAIL_LLITE_LOST_LAYOUT		    0x1407
 #define OBD_FAIL_GETATTR_DELAY			    0x1409
 #define OBD_FAIL_LLITE_CREATE_NODE_PAUSE	    0x140c
+#define OBD_FAIL_LLITE_IMUTEX_SEC		    0x140e
+#define OBD_FAIL_LLITE_IMUTEX_NOSEC		    0x140f
 
 #define OBD_FAIL_FID_INDIR	0x1501
 #define OBD_FAIL_FID_INLMA	0x1502
diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c
index cd169c1..203a1f7 100644
--- a/drivers/staging/lustre/lustre/llite/llite_lib.c
+++ b/drivers/staging/lustre/lustre/llite/llite_lib.c
@@ -227,6 +227,11 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt)
 	if (sbi->ll_flags & LL_SBI_USER_XATTR)
 		data->ocd_connect_flags |= OBD_CONNECT_XATTR;
 
+	/* Setting this indicates we correctly support S_NOSEC (See kernel
+	 * commit 9e1f1de02c2275d7172e18dc4e7c2065777611bf)
+	 */
+	sb->s_flags |= MS_NOSEC;
+
 	if (sbi->ll_flags & LL_SBI_FLOCK)
 		sbi->ll_fop = &ll_file_operations_flock;
 	else if (sbi->ll_flags & LL_SBI_LOCALFLOCK)
@@ -1638,6 +1643,13 @@ int ll_setattr_raw(struct dentry *dentry, struct iattr *attr,
 		inode_lock(inode);
 		if ((attr->ia_valid & ATTR_SIZE) && !hsm_import)
 			inode_dio_wait(inode);
+		/* Once we've got the i_mutex, it's safe to set the S_NOSEC
+		 * flag.  ll_update_inode (called from ll_md_setattr), clears
+		 * inode flags, so there is a gap where S_NOSEC is not set.
+		 * This can cause a writer to take the i_mutex unnecessarily,
+		 * but this is safe to do and should be rare.
+		 */
+		inode_has_no_xattr(inode);
 	}
 
 	ll_stats_ops_tally(ll_i2sbi(inode), (attr->ia_valid & ATTR_SIZE) ?
@@ -1847,6 +1859,11 @@ int ll_update_inode(struct inode *inode, struct lustre_md *md)
 			inode->i_ctime.tv_sec = body->mbo_ctime;
 		lli->lli_ctime = body->mbo_ctime;
 	}
+
+	/* Clear i_flags to remove S_NOSEC before permissions are updated */
+	if (body->mbo_valid & OBD_MD_FLFLAGS)
+		ll_update_inode_flags(inode, body->mbo_flags);
+
 	if (body->mbo_valid & OBD_MD_FLMODE)
 		inode->i_mode = (inode->i_mode & S_IFMT) |
 				(body->mbo_mode & ~S_IFMT);
@@ -1865,8 +1882,6 @@ int ll_update_inode(struct inode *inode, struct lustre_md *md)
 		inode->i_gid = make_kgid(&init_user_ns, body->mbo_gid);
 	if (body->mbo_valid & OBD_MD_FLPROJID)
 		lli->lli_projid = body->mbo_projid;
-	if (body->mbo_valid & OBD_MD_FLFLAGS)
-		ll_update_inode_flags(inode, body->mbo_flags);
 	if (body->mbo_valid & OBD_MD_FLNLINK)
 		set_nlink(inode, body->mbo_nlink);
 	if (body->mbo_valid & OBD_MD_FLRDEV)
diff --git a/drivers/staging/lustre/lustre/llite/namei.c b/drivers/staging/lustre/lustre/llite/namei.c
index f2bd57e..b5b46f7 100644
--- a/drivers/staging/lustre/lustre/llite/namei.c
+++ b/drivers/staging/lustre/lustre/llite/namei.c
@@ -130,6 +130,7 @@ struct inode *ll_iget(struct super_block *sb, ino_t hash,
 			iput(inode);
 			inode = ERR_PTR(rc);
 		} else {
+			inode_has_no_xattr(inode);
 			unlock_new_inode(inode);
 		}
 	} else if (!(inode->i_state & (I_FREEING | I_CLEAR))) {
diff --git a/drivers/staging/lustre/lustre/llite/vvp_io.c b/drivers/staging/lustre/lustre/llite/vvp_io.c
index d6b27ba..b772e25 100644
--- a/drivers/staging/lustre/lustre/llite/vvp_io.c
+++ b/drivers/staging/lustre/lustre/llite/vvp_io.c
@@ -925,9 +925,10 @@ static int vvp_io_write_start(const struct lu_env *env,
 	struct cl_object   *obj   = io->ci_obj;
 	struct inode       *inode = vvp_object_inode(obj);
 	struct ll_inode_info *lli = ll_i2info(inode);
-	ssize_t result = 0;
+	bool lock_inode = !inode_is_locked(inode) && !IS_NOSEC(inode);
 	loff_t pos = io->u.ci_wr.wr.crw_pos;
 	size_t cnt = io->u.ci_wr.wr.crw_count;
+	ssize_t result = 0;
 
 	down_read(&lli->lli_trunc_sem);
 
@@ -963,6 +964,13 @@ static int vvp_io_write_start(const struct lu_env *env,
 		return -EFBIG;
 	}
 
+	/* Tests to verify we take the i_mutex correctly */
+	if (OBD_FAIL_CHECK(OBD_FAIL_LLITE_IMUTEX_SEC) && !lock_inode)
+		return -EINVAL;
+
+	if (OBD_FAIL_CHECK(OBD_FAIL_LLITE_IMUTEX_NOSEC) && lock_inode)
+		return -EINVAL;
+
 	if (!vio->vui_iter) {
 		/* from a temp io in ll_cl_init(). */
 		result = 0;
-- 
1.8.3.1

[lustre-devel] [PATCH 02/14] lustre: llite: change how "dump_page_cache" walks a hash table

[James Simmons]

From: NeilBrown <neilb@suse.com>

The "dump_page_cache" seq_file currently tries to encode
a location in the hash table into a 64bit file index so that
the seq_file can seek to any location.

This is not necessary with the current implementation of seq_file.
seq_file performs any seeks needed itself by rewinding and calling
->next and ->show until the required index is reached.

The required behaviour of ->next is that it always returns the next
object after the last one returned by either ->start or ->next.
It can ignore the ppos, but should increment it.

The required behaviour of ->start is one of:
1/ if *ppos is 0, then return the first object
2/ if *ppos is the same value that was passed to the most recent call
   to either ->start or ->next, then return the same object again
3/ if *ppos is anything else, return the next object after the most
   recently returned one.

To implement this we store a vvp_pgcache_id (index into hash table)
in the seq_private data structure, and also store 'prev_pos' as the
last value passed to either ->start or ->next.

We remove all converstion of an id to a pos, and any limits on the
size of the vpi_depth.

vvp_pgcache_obj_get() is changed to ignore dying objects so that
vvp_pgcache_obj only returns NULL when it reaches the end of a hash
chain, and so vpi_bucket needs to be incremented.

A reference to the current ->clob pointer is now kept as long as we
are iterating over the pages in a given object, so we don't have to
try to find it again (and possibly fail) for each page.

And the ->start and ->next functions are changed as described above.

Signed-off-by: NeilBrown <neilb@suse.com>
Signed-off-by: James Simmons <uja.ornl@yahoo.com>
WC-bug-id: https://jira.whamcloud.com/browse/LU-8066
Reviewed-on: https://review.whamcloud.com/33011
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 drivers/staging/lustre/lustre/llite/vvp_dev.c | 173 ++++++++++++--------------
 1 file changed, 78 insertions(+), 95 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/vvp_dev.c b/drivers/staging/lustre/lustre/llite/vvp_dev.c
index 8cc981b..4e55599 100644
--- a/drivers/staging/lustre/lustre/llite/vvp_dev.c
+++ b/drivers/staging/lustre/lustre/llite/vvp_dev.c
@@ -366,22 +366,6 @@ int cl_sb_fini(struct super_block *sb)
  *
  ****************************************************************************/
 
-/*
- * To represent contents of a page cache as a byte stream, following
- * information if encoded in 64bit offset:
- *
- *       - file hash bucket in lu_site::ls_hash[]       28bits
- *
- *       - how far file is from bucket head	      4bits
- *
- *       - page index				   32bits
- *
- * First two data identify a file in the cache uniquely.
- */
-
-#define PGC_OBJ_SHIFT (32 + 4)
-#define PGC_DEPTH_SHIFT (32)
-
 struct vvp_pgcache_id {
 	unsigned int		 vpi_bucket;
 	unsigned int		 vpi_depth;
@@ -396,37 +380,26 @@ struct vvp_seq_private {
 	struct lu_env		*vsp_env;
 	u16			vsp_refcheck;
 	struct cl_object	*vsp_clob;
+	struct vvp_pgcache_id	vsp_id;
+	/*
+	 * prev_pos is the 'pos' of the last object returned
+	 * by ->start of ->next.
+	 */
+	loff_t			vsp_prev_pos;
 };
 
-static void vvp_pgcache_id_unpack(loff_t pos, struct vvp_pgcache_id *id)
-{
-	BUILD_BUG_ON(sizeof(pos) != sizeof(__u64));
-
-	id->vpi_index  = pos & 0xffffffff;
-	id->vpi_depth  = (pos >> PGC_DEPTH_SHIFT) & 0xf;
-	id->vpi_bucket = (unsigned long long)pos >> PGC_OBJ_SHIFT;
-}
-
-static loff_t vvp_pgcache_id_pack(struct vvp_pgcache_id *id)
-{
-	return
-		((__u64)id->vpi_index) |
-		((__u64)id->vpi_depth  << PGC_DEPTH_SHIFT) |
-		((__u64)id->vpi_bucket << PGC_OBJ_SHIFT);
-}
-
 static int vvp_pgcache_obj_get(struct cfs_hash *hs, struct cfs_hash_bd *bd,
 			       struct hlist_node *hnode, void *data)
 {
 	struct vvp_pgcache_id   *id  = data;
 	struct lu_object_header *hdr = cfs_hash_object(hs, hnode);
 
-	if (id->vpi_curdep-- > 0)
-		return 0; /* continue */
-
 	if (lu_object_is_dying(hdr))
 		return 1;
 
+	if (id->vpi_curdep-- > 0)
+		return 0; /* continue */
+
 	cfs_hash_get(hs, hnode);
 	id->vpi_obj = hdr;
 	return 1;
@@ -438,7 +411,6 @@ static struct cl_object *vvp_pgcache_obj(const struct lu_env *env,
 {
 	LASSERT(lu_device_is_cl(dev));
 
-	id->vpi_depth &= 0xf;
 	id->vpi_obj    = NULL;
 	id->vpi_curdep = id->vpi_depth;
 
@@ -453,55 +425,42 @@ static struct cl_object *vvp_pgcache_obj(const struct lu_env *env,
 			return lu2cl(lu_obj);
 		}
 		lu_object_put(env, lu_object_top(id->vpi_obj));
-
-	} else if (id->vpi_curdep > 0) {
-		id->vpi_depth = 0xf;
 	}
 	return NULL;
 }
 
-static struct page *vvp_pgcache_find(const struct lu_env *env,
-				     struct lu_device *dev,
-				     struct cl_object **clobp, loff_t *pos)
+static struct page *vvp_pgcache_current(struct vvp_seq_private *priv)
 {
-	struct cl_object     *clob;
-	struct lu_site       *site;
-	struct vvp_pgcache_id id;
-
-	site = dev->ld_site;
-	vvp_pgcache_id_unpack(*pos, &id);
+	struct lu_device *dev = &priv->vsp_sbi->ll_cl->cd_lu_dev;
 
 	while (1) {
-		if (id.vpi_bucket >= CFS_HASH_NHLIST(site->ls_obj_hash))
-			return NULL;
-		clob = vvp_pgcache_obj(env, dev, &id);
-		if (clob) {
-			struct inode *inode = vvp_object_inode(clob);
-			struct page *vmpage;
-			int nr;
-
-			nr = find_get_pages_contig(inode->i_mapping,
-						   id.vpi_index, 1, &vmpage);
-			if (nr > 0) {
-				id.vpi_index = vmpage->index;
-				/* Cant support over 16T file */
-				if (vmpage->index <= 0xffffffff) {
-					*clobp = clob;
-					*pos = vvp_pgcache_id_pack(&id);
-					return vmpage;
-				}
-				put_page(vmpage);
-			}
-
-			lu_object_ref_del(&clob->co_lu, "dump", current);
-			cl_object_put(env, clob);
+		struct inode *inode;
+		struct page *vmpage;
+		int nr;
+
+		if (!priv->vsp_clob) {
+			struct cl_object *clob;
+
+			while ((clob = vvp_pgcache_obj(priv->vsp_env, dev, &priv->vsp_id)) == NULL &&
+			       ++(priv->vsp_id.vpi_bucket) < CFS_HASH_NHLIST(dev->ld_site->ls_obj_hash))
+				priv->vsp_id.vpi_depth = 0;
+			if (!clob)
+				return NULL;
+			priv->vsp_clob = clob;
+			priv->vsp_id.vpi_index = 0;
+		}
+
+		inode = vvp_object_inode(priv->vsp_clob);
+		nr = find_get_pages_contig(inode->i_mapping, priv->vsp_id.vpi_index, 1, &vmpage);
+		if (nr > 0) {
+			priv->vsp_id.vpi_index = vmpage->index;
+			return vmpage;
 		}
-		/* to the next object. */
-		++id.vpi_depth;
-		id.vpi_depth &= 0xf;
-		if (id.vpi_depth == 0 && ++id.vpi_bucket == 0)
-			return NULL;
-		id.vpi_index = 0;
+		lu_object_ref_del(&priv->vsp_clob->co_lu, "dump", current);
+		cl_object_put(priv->vsp_env, priv->vsp_clob);
+		priv->vsp_clob = NULL;
+		priv->vsp_id.vpi_index = 0;
+		priv->vsp_id.vpi_depth++;
 	}
 }
 
@@ -559,38 +518,55 @@ static int vvp_pgcache_show(struct seq_file *f, void *v)
 	} else {
 		seq_puts(f, "missing\n");
 	}
-	lu_object_ref_del(&priv->vsp_clob->co_lu, "dump", current);
-	cl_object_put(priv->vsp_env, priv->vsp_clob);
 
 	return 0;
 }
 
+static void vvp_pgcache_rewind(struct vvp_seq_private *priv)
+{
+	if (priv->vsp_prev_pos) {
+		memset(&priv->vsp_id, 0, sizeof(priv->vsp_id));
+		priv->vsp_prev_pos = 0;
+		if (priv->vsp_clob) {
+			lu_object_ref_del(&priv->vsp_clob->co_lu, "dump",
+					  current);
+			cl_object_put(priv->vsp_env, priv->vsp_clob);
+		}
+		priv->vsp_clob = NULL;
+	}
+}
+
+static struct page *vvp_pgcache_next_page(struct vvp_seq_private *priv)
+{
+	priv->vsp_id.vpi_index += 1;
+	return vvp_pgcache_current(priv);
+}
+
 static void *vvp_pgcache_start(struct seq_file *f, loff_t *pos)
 {
 	struct vvp_seq_private *priv = f->private;
-	struct page *ret;
 
-	if (priv->vsp_sbi->ll_site->ls_obj_hash->hs_cur_bits >
-	    64 - PGC_OBJ_SHIFT)
-		ret = ERR_PTR(-EFBIG);
-	else
-		ret = vvp_pgcache_find(priv->vsp_env,
-				       &priv->vsp_sbi->ll_cl->cd_lu_dev,
-				       &priv->vsp_clob, pos);
+	if (*pos == 0) {
+		vvp_pgcache_rewind(priv);
+	} else if (*pos == priv->vsp_prev_pos) {
+		/* Return the current item */;
+	} else {
+		WARN_ON(*pos != priv->vsp_prev_pos + 1);
+		priv->vsp_id.vpi_index += 1;
+	}
 
-	return ret;
+	priv->vsp_prev_pos = *pos;
+	return vvp_pgcache_current(priv);
 }
 
 static void *vvp_pgcache_next(struct seq_file *f, void *v, loff_t *pos)
 {
 	struct vvp_seq_private *priv = f->private;
-	struct page *ret;
 
-	*pos += 1;
-	ret = vvp_pgcache_find(priv->vsp_env,
-			       &priv->vsp_sbi->ll_cl->cd_lu_dev,
-			       &priv->vsp_clob, pos);
-	return ret;
+	WARN_ON(*pos != priv->vsp_prev_pos);
+
+	priv->vsp_prev_pos = *pos;
+	return vvp_pgcache_next_page(priv);
 }
 
 static void vvp_pgcache_stop(struct seq_file *f, void *v)
@@ -615,6 +591,8 @@ static int vvp_dump_pgcache_seq_open(struct inode *inode, struct file *filp)
 
 	priv->vsp_sbi = inode->i_private;
 	priv->vsp_env = cl_env_get(&priv->vsp_refcheck);
+	priv->vsp_clob = NULL;
+	memset(&priv->vsp_id, 0, sizeof(priv->vsp_id));
 	if (IS_ERR(priv->vsp_env)) {
 		int err = PTR_ERR(priv->vsp_env);
 
@@ -629,6 +607,11 @@ static int vvp_dump_pgcache_seq_release(struct inode *inode, struct file *file)
 	struct seq_file *seq = file->private_data;
 	struct vvp_seq_private *priv = seq->private;
 
+	if (priv->vsp_clob) {
+		lu_object_ref_del(&priv->vsp_clob->co_lu, "dump", current);
+		cl_object_put(priv->vsp_env, priv->vsp_clob);
+	}
+
 	cl_env_put(priv->vsp_env, &priv->vsp_refcheck);
 	return seq_release_private(inode, file);
 }
-- 
1.8.3.1

[lustre-devel] [PATCH 03/14] lustre: llite: add LL_IOC_FUTIMES_3

[James Simmons]

From: "John L. Hammond" <jhammond@whamcloud.com>

Add a new regular file ioctl LL_IOC_FUTIMES_3 similar to futimes() but
which allows setting of all three inode timestamps. Use this ioctl
during HSM restore to ensure that the volatile file has the same
timestamps as the file to be restored. Strengthen sanity-hsm test_24a
to check that archive, release, and restore do not change a file's
ctime. Add sanity-hsm test_24e to check that tar will succeed when it
encounters a HSM released file.

This is a hold over until a generic solution is worked out.

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
WC-bug-id: https://jira.hpdd.intel.com/browse/LU-6213
Reviewed-on: http://review.whamcloud.com/13665
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@gmail.com>
Reviewed-by: frank zago <fzago@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 .../lustre/include/uapi/linux/lustre/lustre_user.h | 10 +++++
 drivers/staging/lustre/lustre/llite/file.c         | 46 ++++++++++++++++++++++
 2 files changed, 56 insertions(+)

diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
index 9d553ce6..6904d6d 100644
--- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
+++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
@@ -215,6 +215,15 @@ struct ost_id {
 #define DOSTID "%#llx:%llu"
 #define POSTID(oi) ostid_seq(oi), ostid_id(oi)
 
+struct ll_futimes_3 {
+	__u64 lfu_atime_sec;
+	__u64 lfu_atime_nsec;
+	__u64 lfu_mtime_sec;
+	__u64 lfu_mtime_nsec;
+	__u64 lfu_ctime_sec;
+	__u64 lfu_ctime_nsec;
+};
+
 /*
  * The ioctl naming rules:
  * LL_*     - works on the currently opened filehandle instead of parent dir
@@ -251,6 +260,7 @@ struct ost_id {
 #define LL_IOC_PATH2FID		 _IOR('f', 173, long)
 #define LL_IOC_GET_CONNECT_FLAGS	_IOWR('f', 174, __u64 *)
 #define LL_IOC_GET_MDTIDX	       _IOR('f', 175, int)
+#define LL_IOC_FUTIMES_3		_IOWR('f', 176, struct ll_futimes_3)
 
 /*	lustre_ioctl.h			177-210 */
 #define LL_IOC_HSM_STATE_GET		_IOR('f', 211, struct hsm_user_state)
diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
index a6f149c..5c6ebfa 100644
--- a/drivers/staging/lustre/lustre/llite/file.c
+++ b/drivers/staging/lustre/lustre/llite/file.c
@@ -2098,6 +2098,42 @@ static inline long ll_lease_type_from_fmode(fmode_t fmode)
 	       ((fmode & FMODE_WRITE) ? LL_LEASE_WRLCK : 0);
 }
 
+static int ll_file_futimes_3(struct file *file, const struct ll_futimes_3 *lfu)
+{
+	struct inode *inode = file_inode(file);
+	struct iattr ia = {
+		.ia_valid = ATTR_ATIME | ATTR_ATIME_SET |
+			    ATTR_MTIME | ATTR_MTIME_SET |
+			    ATTR_CTIME,
+		.ia_atime = {
+			.tv_sec = lfu->lfu_atime_sec,
+			.tv_nsec = lfu->lfu_atime_nsec,
+		},
+		.ia_mtime = {
+			.tv_sec = lfu->lfu_mtime_sec,
+			.tv_nsec = lfu->lfu_mtime_nsec,
+		},
+		.ia_ctime = {
+			.tv_sec = lfu->lfu_ctime_sec,
+			.tv_nsec = lfu->lfu_ctime_nsec,
+		},
+	};
+	int rc;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	if (!S_ISREG(inode->i_mode))
+		return -EINVAL;
+
+	inode_lock(inode);
+	rc = ll_setattr_raw(file_dentry(file), &ia, OP_XVALID_CTIME_SET,
+			    false);
+	inode_unlock(inode);
+
+	return rc;
+}
+
 /*
  * Give file access advices
  *
@@ -2552,6 +2588,16 @@ int ll_ioctl_fssetxattr(struct inode *inode, unsigned int cmd,
 		kfree(hui);
 		return rc;
 	}
+	case LL_IOC_FUTIMES_3: {
+		const struct ll_futimes_3 __user *lfu_user;
+		struct ll_futimes_3 lfu;
+
+		lfu_user = (const struct ll_futimes_3 __user *)arg;
+		if (copy_from_user(&lfu, lfu_user, sizeof(lfu)))
+			return -EFAULT;
+
+		return ll_file_futimes_3(file, &lfu);
+        }
 	case LL_IOC_LADVISE: {
 		struct llapi_ladvise_hdr *ladvise_hdr;
 		int alloc_size = sizeof(*ladvise_hdr);
-- 
1.8.3.1

[lustre-devel] [PATCH 04/14] lustre: llite: improve getdirstripe interface

[James Simmons]

From: Lai Siyao <lai.siyao@whamcloud.com>

The user land lustre tools added the ability to choose the DNE2
hashing algorithm. Handle the hash values coming from user land.

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
WC-bug-id: https://jira.hpdd.intel.com/browse/LU-8314
Reviewed-on: https://review.whamcloud.com/24319
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 .../staging/lustre/include/uapi/linux/lustre/lustre_idl.h    | 12 ++++++++++++
 .../staging/lustre/include/uapi/linux/lustre/lustre_user.h   |  2 ++
 drivers/staging/lustre/lustre/llite/dir.c                    |  1 +
 3 files changed, 15 insertions(+)

diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_idl.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_idl.h
index 26646f9..a42ce9d 100644
--- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_idl.h
+++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_idl.h
@@ -1968,6 +1968,18 @@ static inline int lmv_mds_md_stripe_count_get(const union lmv_mds_md *lmm)
 	}
 }
 
+static inline int lmv_mds_md_hash_type_get(const union lmv_mds_md *lmm)
+{
+	switch (__le32_to_cpu(lmm->lmv_magic)) {
+	case LMV_MAGIC_V1:
+		return __le32_to_cpu(lmm->lmv_md_v1.lmv_hash_type);
+	case LMV_USER_MAGIC:
+		return __le32_to_cpu(lmm->lmv_user_md.lum_hash_type);
+	default:
+		return -EINVAL;
+	}
+}
+
 enum fld_rpc_opc {
 	FLD_QUERY	= 900,
 	FLD_READ	= 901,
diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
index 6904d6d..4412dc8 100644
--- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
+++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
@@ -443,6 +443,8 @@ enum lmv_hash_type {
 	LMV_HASH_TYPE_FNV_1A_64 = 2,
 };
 
+#define LMV_HASH_TYPE_MAX	LMV_HASH_TYPE_FNV_1A_64 + 1
+
 #define LMV_HASH_NAME_ALL_CHARS		"all_char"
 #define LMV_HASH_NAME_FNV_1A_64		"fnv_1a_64"
 
diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c
index f1c1c9c..86b5c7a 100644
--- a/drivers/staging/lustre/lustre/llite/dir.c
+++ b/drivers/staging/lustre/lustre/llite/dir.c
@@ -1315,6 +1315,7 @@ static long ll_dir_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 		tmp->lum_magic = LMV_MAGIC_V1;
 		tmp->lum_stripe_count = 0;
 		tmp->lum_stripe_offset = mdt_index;
+		tmp->lum_hash_type = lmv_mds_md_hash_type_get(lmm);
 		for (i = 0; i < stripe_count; i++) {
 			struct lu_fid fid;
 
-- 
1.8.3.1

[lustre-devel] [PATCH 05/14] lustre: uapi: fix lustre_user.h to C++ compatible

[James Simmons]

From: Minh Diep <mdiep@whamcloud.com>

Add missing user land headers. constifiy some inline
functions.

Signed-off-by: Minh Diep <mdiep@whamcloud.com>
WC-bug-id: https://jira.hpdd.intel.com/browse/LU-4107
Reviewed-on: http://review.whamcloud.com/8033
Reviewed-by: John L. Hammond <jhammond@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 .../staging/lustre/include/uapi/linux/lustre/lustre_user.h  | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
index 4412dc8..fef53b1 100644
--- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
+++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
@@ -50,6 +50,8 @@
 # include <linux/version.h>
 #else /* !__KERNEL__ */
 # define NEED_QUOTA_DEFS
+# include <limits.h>
+# include <stdbool.h>
 # include <stdio.h> /* snprintf() */
 # include <string.h>
 # include <sys/quota.h>
@@ -1052,7 +1054,7 @@ enum hsm_progress_states {
 
 #define HPS_NONE	0
 
-static inline char *hsm_progress_state2name(enum hsm_progress_states s)
+static inline const char *hsm_progress_state2name(enum hsm_progress_states s)
 {
 	switch  (s) {
 	case HPS_WAITING:	return "waiting";
@@ -1114,7 +1116,7 @@ enum hsm_user_action {
 	HUA_CANCEL  = 14  /* cancel a request */
 };
 
-static inline char *hsm_user_action2name(enum hsm_user_action  a)
+static inline const char *hsm_user_action2name(enum hsm_user_action  a)
 {
 	switch  (a) {
 	case HUA_NONE:    return "NOOP";
@@ -1201,7 +1203,7 @@ enum hsm_copytool_action {
 	HSMA_CANCEL  = 23
 };
 
-static inline char *hsm_copytool_action2name(enum hsm_copytool_action  a)
+static inline const char *hsm_copytool_action2name(enum hsm_copytool_action  a)
 {
 	switch  (a) {
 	case HSMA_NONE:    return "NOOP";
@@ -1294,9 +1296,10 @@ static inline struct hsm_action_item *hai_next(struct hsm_action_item *hai)
 }
 
 /* Return size of an hsm_action_list */
-static inline int hal_size(struct hsm_action_list *hal)
+static inline size_t hal_size(struct hsm_action_list *hal)
 {
-	int i, sz;
+	__u32 i;
+	size_t sz;
 	struct hsm_action_item *hai;
 
 	sz = sizeof(*hal) + cfs_size_round(strlen(hal->hal_fsname) + 1);
-- 
1.8.3.1

[lustre-devel] [PATCH 06/14] lustre: uapi: replace cfs_size_* macros with __ALIGN_KERNEL

[James Simmons]

The lustre specific cfs_size_* macros can be easily replaced with
the __ALIGN_KERNEL macro provided by the linux kernel for our
user land code. This brings us closer to building against the
upstream client.

Signed-off-by: James Simmons <uja.ornl@yahoo.com>
WC-bug-id: https://jira.hpdd.intel.com/browse/LU-6142
Reviewed-on: https://review.whamcloud.com/30379
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Ben Evans <bevans@cray.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 .../lustre/include/uapi/linux/lustre/lustre_user.h | 32 +++++++++-------------
 1 file changed, 13 insertions(+), 19 deletions(-)

diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
index fef53b1..aac91a1 100644
--- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
+++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_user.h
@@ -42,6 +42,9 @@
  * @{
  */
 
+#include <linux/kernel.h>
+#include <linux/types.h>
+
 #ifdef __KERNEL__
 # include <linux/fs.h>
 # include <linux/quota.h>
@@ -835,8 +838,8 @@ enum changelog_send_flag {
 	CHANGELOG_FLAG_JOBID	= 0x04,
 };
 
-#define CR_MAXSIZE cfs_size_round(2 * NAME_MAX + 2 + \
-				  changelog_rec_offset(CLF_SUPPORTED))
+#define CR_MAXSIZE __ALIGN_KERNEL(2 * NAME_MAX + 2 + \
+				  changelog_rec_offset(CLF_SUPPORTED), 8)
 
 /* 31 usable bytes string + null terminator. */
 #define LUSTRE_JOBID_SIZE	32
@@ -1270,29 +1273,20 @@ struct hsm_action_list {
 	 */
 } __packed;
 
-#ifndef HAVE_CFS_SIZE_ROUND
-static inline int cfs_size_round(int val)
-{
-	return (val + 7) & (~0x7);
-}
-
-#define HAVE_CFS_SIZE_ROUND
-#endif
-
 /* Return pointer to first hai in action list */
 static inline struct hsm_action_item *hai_first(struct hsm_action_list *hal)
 {
-	return (struct hsm_action_item *)(hal->hal_fsname +
-					  cfs_size_round(strlen(hal-> \
-								hal_fsname)
-							 + 1));
+	size_t offset = __ALIGN_KERNEL(strlen(hal->hal_fsname) + 1, 8);
+
+	return (struct hsm_action_item *)(hal->hal_fsname + offset);
 }
 
 /* Return pointer to next hai */
 static inline struct hsm_action_item *hai_next(struct hsm_action_item *hai)
 {
-	return (struct hsm_action_item *)((char *)hai +
-					  cfs_size_round(hai->hai_len));
+	size_t offset = __ALIGN_KERNEL(hai->hai_len, 8);
+
+	return (struct hsm_action_item *)((char *)hai + offset);
 }
 
 /* Return size of an hsm_action_list */
@@ -1302,10 +1296,10 @@ static inline size_t hal_size(struct hsm_action_list *hal)
 	size_t sz;
 	struct hsm_action_item *hai;
 
-	sz = sizeof(*hal) + cfs_size_round(strlen(hal->hal_fsname) + 1);
+	sz = sizeof(*hal) + __ALIGN_KERNEL(strlen(hal->hal_fsname) + 1, 8);
 	hai = hai_first(hal);
 	for (i = 0; i < hal->hal_count; i++, hai = hai_next(hai))
-		sz += cfs_size_round(hai->hai_len);
+		sz += __ALIGN_KERNEL(hai->hai_len, 8);
 
 	return sz;
 }
-- 
1.8.3.1

[lustre-devel] [PATCH 07/14] lustre: uapi: final fixes for UAPI support.

[James Simmons]

Two problems exist that can prevent lustre UAPI headers from
installing properly into /usr/include/linux/lustre. The first
problem is bvec.h is only for kernel space. So make lib-types.h
include bvec.h instead of the UAPI header lnet-types.h. Last
problem was having __uXX inside a comment for lustre_idl.h
confused make install.

Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 drivers/staging/lustre/include/linux/lnet/lib-types.h         | 1 +
 drivers/staging/lustre/include/uapi/linux/lnet/lnet-types.h   | 1 -
 drivers/staging/lustre/include/uapi/linux/lustre/lustre_idl.h | 2 +-
 3 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/lustre/include/linux/lnet/lib-types.h b/drivers/staging/lustre/include/linux/lnet/lib-types.h
index 8951a53..3a54e06 100644
--- a/drivers/staging/lustre/include/linux/lnet/lib-types.h
+++ b/drivers/staging/lustre/include/linux/lnet/lib-types.h
@@ -36,6 +36,7 @@
 #ifndef __LNET_LIB_TYPES_H__
 #define __LNET_LIB_TYPES_H__
 
+#include <linux/bvec.h>
 #include <linux/kthread.h>
 #include <linux/net.h>
 #include <linux/uio.h>
diff --git a/drivers/staging/lustre/include/uapi/linux/lnet/lnet-types.h b/drivers/staging/lustre/include/uapi/linux/lnet/lnet-types.h
index e440100..4fcc533 100644
--- a/drivers/staging/lustre/include/uapi/linux/lnet/lnet-types.h
+++ b/drivers/staging/lustre/include/uapi/linux/lnet/lnet-types.h
@@ -34,7 +34,6 @@
 #define __LNET_TYPES_H__
 
 #include <linux/types.h>
-#include <linux/bvec.h>
 
 /** \addtogroup lnet
  * @{
diff --git a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_idl.h b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_idl.h
index a42ce9d..1d6faf5 100644
--- a/drivers/staging/lustre/include/uapi/linux/lustre/lustre_idl.h
+++ b/drivers/staging/lustre/include/uapi/linux/lustre/lustre_idl.h
@@ -40,7 +40,7 @@
  * that are used in interfaces with userspace should go in lustre_user.h.
  *
  * All structs being declared here should be built from simple fixed-size
- * types (__u8, __u16, __u32, __u64) or be built from other types or
+ * types defined in linux/types.h or be built from other types or
  * structs also declared in this file.  Similarly, all flags and magic
  * values in those structs should also be declared here.  This ensures
  * that the Lustre wire protocol is not influenced by external dependencies.
-- 
1.8.3.1

[lustre-devel] [PATCH 08/14] lustre: llite: op_data->op_valid is incorrect in ll_dir_getstripe

[James Simmons]

The function ll_dir_getstripe() passes in valid but it was never
used to properly setup op_data->op_valid which prevented user
land applications from getting real directory striping
information. Apply the passed in valid parameter to the
op_data->op_valid field.

Fixes: 6e23ea986bf ("staging: lustre: llite: fix "getdirstripe" to show stripe info")
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 drivers/staging/lustre/lustre/llite/dir.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/llite/dir.c b/drivers/staging/lustre/lustre/llite/dir.c
index 86b5c7a..0a7330d 100644
--- a/drivers/staging/lustre/lustre/llite/dir.c
+++ b/drivers/staging/lustre/lustre/llite/dir.c
@@ -631,7 +631,7 @@ int ll_dir_getstripe(struct inode *inode, void **plmm, int *plmm_size,
 	if (IS_ERR(op_data))
 		return PTR_ERR(op_data);
 
-	op_data->op_valid = OBD_MD_FLEASIZE | OBD_MD_FLDIREA;
+	op_data->op_valid = valid | OBD_MD_FLEASIZE | OBD_MD_FLDIREA;
 	rc = md_getattr(sbi->ll_md_exp, op_data, &req);
 	ll_finish_md_op_data(op_data);
 	if (rc < 0) {
-- 
1.8.3.1

[lustre-devel] [PATCH 09/14] lustre: llite: return correct amount of bytes for short operations

[James Simmons]

The incorrect number of bytes processed by ll_file_io_generic() is
returned. We need to add io->ci_nob to result.

Fixes: 623f55c4510 ("staging: lustre: llite: restart short read/write for normal IO")
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 drivers/staging/lustre/lustre/llite/file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
index 5c6ebfa..b390971 100644
--- a/drivers/staging/lustre/lustre/llite/file.c
+++ b/drivers/staging/lustre/lustre/llite/file.c
@@ -1195,7 +1195,7 @@ static void ll_io_init(struct cl_io *io, const struct file *file, int write)
 	}
 
 	if (io->ci_nob > 0) {
-		result = io->ci_nob;
+		result += io->ci_nob;
 		count -= io->ci_nob;
 		*ppos = io->u.ci_wr.wr.crw_pos;
 
-- 
1.8.3.1

[lustre-devel] [PATCH 10/14] lustre: llite: user xattr_prefix() to create xattr full name

[James Simmons]

Some xattrs use the prefix field and some like ACLs use the name
field. The function xattr_prefix() was created to handle these
cases. A patch was landed to properly handle this but a part of
the patch was missed in the merger. For the creation of fulname
use xattr_prefix() instead of the handler prefix field.

Fixes: ab42bc01284 ("staging: lustre: llite: use xattr_handler name for ACLs")
Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 drivers/staging/lustre/lustre/llite/xattr.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/xattr.c b/drivers/staging/lustre/lustre/llite/xattr.c
index 5e27c85..aeaa04a 100644
--- a/drivers/staging/lustre/lustre/llite/xattr.c
+++ b/drivers/staging/lustre/lustre/llite/xattr.c
@@ -140,7 +140,7 @@ static int ll_xattr_set_common(const struct xattr_handler *handler,
 			return -EPERM;
 	}
 
-	fullname = kasprintf(GFP_KERNEL, "%s%s", handler->prefix, name);
+	fullname = kasprintf(GFP_KERNEL, "%s%s", xattr_prefix(handler), name);
 	if (!fullname)
 		return -ENOMEM;
 
@@ -443,7 +443,7 @@ static int ll_xattr_get_common(const struct xattr_handler *handler,
 	if (handler->flags == XATTR_ACL_DEFAULT_T && !S_ISDIR(inode->i_mode))
 		return -ENODATA;
 #endif
-	fullname = kasprintf(GFP_KERNEL, "%s%s", handler->prefix, name);
+	fullname = kasprintf(GFP_KERNEL, "%s%s", xattr_prefix(handler), name);
 	if (!fullname)
 		return -ENOMEM;
 
-- 
1.8.3.1

[lustre-devel] [PATCH 11/14] lustre: llite: conditionally set it_flags in ll_file_open

[James Simmons]

For ll_file_open the it->it_flags is always setting the
MDS_OPEN_LOCK bit but it should only be set when
lld_nfs_dentry is set.

Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 drivers/staging/lustre/lustre/llite/file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
index b390971..3647618 100644
--- a/drivers/staging/lustre/lustre/llite/file.c
+++ b/drivers/staging/lustre/lustre/llite/file.c
@@ -620,7 +620,7 @@ int ll_file_open(struct inode *inode, struct file *file)
 			 * Always specify MDS_OPEN_BY_FID because we don't want
 			 * to get file with different fid.
 			 */
-			it->it_flags |= MDS_OPEN_LOCK | MDS_OPEN_BY_FID;
+			it->it_flags |= MDS_OPEN_BY_FID;
 			rc = ll_intent_file_open(file->f_path.dentry,
 						 NULL, 0, it);
 			if (rc)
-- 
1.8.3.1

[lustre-devel] [PATCH 12/14] lustre: llite: fix file migration

[James Simmons]

Several small bugs have appeared over the years that broke file
migration. The first bug is that the hash was created using the
parent inode instead of using file_dentry(file). The second
problem occured with op_fid3 being set when the child dentry
was found. The last problem was rn_opcode was always being set
to REINT_RENAME in mdc_rename_pack(). Remove that extra
nr_opcode setting.

Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 drivers/staging/lustre/lustre/llite/file.c  | 3 +--
 drivers/staging/lustre/lustre/mdc/mdc_lib.c | 1 -
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/file.c b/drivers/staging/lustre/lustre/llite/file.c
index 3647618..c018c5f 100644
--- a/drivers/staging/lustre/lustre/llite/file.c
+++ b/drivers/staging/lustre/lustre/llite/file.c
@@ -3003,12 +3003,11 @@ int ll_migrate(struct inode *parent, struct file *file, int mdtidx,
 		return PTR_ERR(op_data);
 
 	/* Get child FID first */
-	qstr.hash = full_name_hash(parent, name, namelen);
+	qstr.hash = full_name_hash(file_dentry(file), name, namelen);
 	qstr.name = name;
 	qstr.len = namelen;
 	dchild = d_lookup(file_dentry(file), &qstr);
 	if (dchild) {
-		op_data->op_fid3 = *ll_inode2fid(dchild->d_inode);
 		if (dchild->d_inode)
 			child_inode = igrab(dchild->d_inode);
 		dput(dchild);
diff --git a/drivers/staging/lustre/lustre/mdc/mdc_lib.c b/drivers/staging/lustre/lustre/mdc/mdc_lib.c
index 415127f..a1b1e75 100644
--- a/drivers/staging/lustre/lustre/mdc/mdc_lib.c
+++ b/drivers/staging/lustre/lustre/mdc/mdc_lib.c
@@ -457,7 +457,6 @@ void mdc_rename_pack(struct ptlrpc_request *req, struct md_op_data *op_data,
 	/* XXX do something about time, uid, gid */
 	rec->rn_opcode	 = op_data->op_cli_flags & CLI_MIGRATE ?
 				REINT_MIGRATE : REINT_RENAME;
-	rec->rn_opcode   = REINT_RENAME;
 	rec->rn_fsuid    = op_data->op_fsuid;
 	rec->rn_fsgid    = op_data->op_fsgid;
 	rec->rn_cap      = op_data->op_cap.cap[0];
-- 
1.8.3.1

[lustre-devel] [PATCH 13/14] lustre: sysfs: temporary work around for sysfs naming

[James Simmons]

Lustre uses the internal addressing of data structures to name
the sysfs objects. With newer kernels hashing is done on the
data structure addresses to make it harder to bypass address
space randomization. This breaks auto tuning with lustre. The
temporary work around is to use "%px" for the naming just to
make things work again. The long term solution is to move to
proper uuids for the name which is being worked on.

Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 drivers/staging/lustre/lustre/llite/llite_lib.c     | 6 +++---
 drivers/staging/lustre/lustre/mgc/mgc_request.c     | 2 +-
 drivers/staging/lustre/lustre/obdclass/obd_config.c | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c
index 203a1f7..b258c80 100644
--- a/drivers/staging/lustre/lustre/llite/llite_lib.c
+++ b/drivers/staging/lustre/lustre/llite/llite_lib.c
@@ -959,7 +959,7 @@ int ll_fill_super(struct super_block *sb)
 		len -= 7;
 
 	/* Mount info */
-	snprintf(name, MAX_STRING_SIZE, "%.*s-%p", len,
+	snprintf(name, MAX_STRING_SIZE, "%.*s-%px", len,
 		 lsi->lsi_lmd->lmd_profile, sb);
 
 	/* Call ll_debugsfs_register_super() before lustre_process_log()
@@ -997,13 +997,13 @@ int ll_fill_super(struct super_block *sb)
 	CDEBUG(D_CONFIG, "Found profile %s: mdc=%s osc=%s\n", profilenm,
 	       lprof->lp_md, lprof->lp_dt);
 
-	dt = kasprintf(GFP_NOFS, "%s-%p", lprof->lp_dt, cfg->cfg_instance);
+	dt = kasprintf(GFP_NOFS, "%s-%px", lprof->lp_dt, cfg->cfg_instance);
 	if (!dt) {
 		err = -ENOMEM;
 		goto out_debugfs;
 	}
 
-	md = kasprintf(GFP_NOFS, "%s-%p", lprof->lp_md, cfg->cfg_instance);
+	md = kasprintf(GFP_NOFS, "%s-%px", lprof->lp_md, cfg->cfg_instance);
 	if (!md) {
 		err = -ENOMEM;
 		goto out_debugfs;
diff --git a/drivers/staging/lustre/lustre/mgc/mgc_request.c b/drivers/staging/lustre/lustre/mgc/mgc_request.c
index a6be2ca..ca74c75 100644
--- a/drivers/staging/lustre/lustre/mgc/mgc_request.c
+++ b/drivers/staging/lustre/lustre/mgc/mgc_request.c
@@ -1142,7 +1142,7 @@ static int mgc_apply_recover_logs(struct obd_device *mgc,
 	if (!inst)
 		return -ENOMEM;
 
-	pos = snprintf(inst, PAGE_SIZE, "%p", cfg->cfg_instance);
+	pos = snprintf(inst, PAGE_SIZE, "%px", cfg->cfg_instance);
 	if (pos >= PAGE_SIZE) {
 		kfree(inst);
 		return -E2BIG;
diff --git a/drivers/staging/lustre/lustre/obdclass/obd_config.c b/drivers/staging/lustre/lustre/obdclass/obd_config.c
index 8be8751..7d00ca4 100644
--- a/drivers/staging/lustre/lustre/obdclass/obd_config.c
+++ b/drivers/staging/lustre/lustre/obdclass/obd_config.c
@@ -1323,7 +1323,7 @@ int class_config_llog_handler(const struct lu_env *env,
 		    LUSTRE_CFG_BUFLEN(lcfg, 0) > 0) {
 			inst_len = LUSTRE_CFG_BUFLEN(lcfg, 0) +
 				   sizeof(clli->cfg_instance) * 2 + 4;
-			inst_name = kasprintf(GFP_NOFS, "%s-%p",
+			inst_name = kasprintf(GFP_NOFS, "%s-%px",
 					      lustre_cfg_string(lcfg, 0),
 					      clli->cfg_instance);
 			if (!inst_name) {
-- 
1.8.3.1

[lustre-devel] [PATCH 14/14] lustre: libcfs: hide struct cfs_cpt_table internals

[James Simmons]

Exposing the internals of struct cfs_cpt_table in the libcfs
header used by lustre makes it too tempting to use but those
fields are not available for UP systems. Make cfs_cpt_table
abstract so the lustre and LNet code will never use the
internals and instead use the functions provided in
libcfs_cpu.h.

Signed-off-by: James Simmons <jsimmons@infradead.org>
---
 .../lustre/include/linux/libcfs/libcfs_cpu.h       | 39 ++--------------------
 drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c    | 34 +++++++++++++++++++
 2 files changed, 37 insertions(+), 36 deletions(-)

diff --git a/drivers/staging/lustre/include/linux/libcfs/libcfs_cpu.h b/drivers/staging/lustre/include/linux/libcfs/libcfs_cpu.h
index 50eb484..3e51752 100644
--- a/drivers/staging/lustre/include/linux/libcfs/libcfs_cpu.h
+++ b/drivers/staging/lustre/include/linux/libcfs/libcfs_cpu.h
@@ -79,42 +79,9 @@
 /* any CPU partition */
 #define CFS_CPT_ANY		(-1)
 
-#ifdef CONFIG_SMP
-/** virtual processing unit */
-struct cfs_cpu_partition {
-	/* CPUs mask for this partition */
-	cpumask_var_t			cpt_cpumask;
-	/* nodes mask for this partition */
-	nodemask_t			*cpt_nodemask;
-	/* NUMA distance between CPTs */
-	unsigned int			*cpt_distance;
-	/* spread rotor for NUMA allocator */
-	unsigned int			cpt_spread_rotor;
-	/* NUMA node if cpt_nodemask is empty */
-	int				cpt_node;
-};
-
-
-/** descriptor for CPU partitions */
-struct cfs_cpt_table {
-	/* spread rotor for NUMA allocator */
-	unsigned int			ctb_spread_rotor;
-	/* maximum NUMA distance between all nodes in table */
-	unsigned int			ctb_distance;
-	/* # of CPU partitions */
-	unsigned int			ctb_nparts;
-	/* partitions tables */
-	struct cfs_cpu_partition	*ctb_parts;
-	/* shadow HW CPU to CPU partition ID */
-	int				*ctb_cpu2cpt;
-	/* all cpus in this partition table */
-	cpumask_var_t			ctb_cpumask;
-	/* shadow HW node to CPU partition ID */
-	int				*ctb_node2cpt;
-	/* all nodes in this partition table */
-	nodemask_t			*ctb_nodemask;
-};
+struct cfs_cpt_table;
 
+#ifdef CONFIG_SMP
 extern struct cfs_cpt_table	*cfs_cpt_tab;
 
 /**
@@ -215,7 +182,7 @@ void cfs_cpt_unset_nodemask(struct cfs_cpt_table *cptab,
 void cfs_cpu_fini(void);
 
 #else /* !CONFIG_SMP */
-struct cfs_cpt_table;
+
 #define cfs_cpt_tab ((struct cfs_cpt_table *)NULL)
 
 static inline int cfs_cpt_table_print(struct cfs_cpt_table *cptab,
diff --git a/drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c b/drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c
index a9b59cc..a384a73 100644
--- a/drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c
+++ b/drivers/staging/lustre/lnet/libcfs/libcfs_cpu.c
@@ -41,6 +41,40 @@
 #include <linux/libcfs/libcfs_string.h>
 #include <linux/libcfs/libcfs.h>
 
+/** virtual processing unit */
+struct cfs_cpu_partition {
+	/* CPUs mask for this partition */
+	cpumask_var_t			cpt_cpumask;
+	/* nodes mask for this partition */
+	nodemask_t			*cpt_nodemask;
+	/* NUMA distance between CPTs */
+	unsigned int			*cpt_distance;
+	/* spread rotor for NUMA allocator */
+	unsigned int			cpt_spread_rotor;
+	/* NUMA node if cpt_nodemask is empty */
+	int				cpt_node;
+};
+
+/** descriptor for CPU partitions */
+struct cfs_cpt_table {
+	/* spread rotor for NUMA allocator */
+	unsigned int			ctb_spread_rotor;
+	/* maximum NUMA distance between all nodes in table */
+	unsigned int			ctb_distance;
+	/* # of CPU partitions */
+	unsigned int			ctb_nparts;
+	/* partitions tables */
+	struct cfs_cpu_partition	*ctb_parts;
+	/* shadow HW CPU to CPU partition ID */
+	int				*ctb_cpu2cpt;
+	/* all cpus in this partition table */
+	cpumask_var_t			ctb_cpumask;
+	/* shadow HW node to CPU partition ID */
+	int				*ctb_node2cpt;
+	/* all nodes in this partition table */
+	nodemask_t			*ctb_nodemask;
+};
+
 /** Global CPU partition table */
 struct cfs_cpt_table   *cfs_cpt_tab __read_mostly;
 EXPORT_SYMBOL(cfs_cpt_tab);
-- 
1.8.3.1

[lustre-devel] [PATCH 00/14] lustre: fixes for many test failures

[NeilBrown]

On Sun, Jan 06 2019, James Simmons wrote:

> This patch series contains mostly fixes to resolve the majority of
> the testing failures seen. Besides that are UAPI headers fixes to
> prepare for when lustre will leave the staging tree. Include Neil's
> dump_page_cache patch as well since it was landed in the OpenSFS
> branch and the migration of lu_objects to rhashtables depended on
> it.

Thanks - I've applied these.
I haven't received all of the PFL series yet ... 16,17,18 still in limbo
- so I haven't applied those yet.
When the stragglers here here, I suspect they'll all go in.

Thanks,
NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20190107/694eb512/attachment.sig>

[lustre-devel] [PATCH 00/14] lustre: fixes for many test failures

[James Simmons]

> On Sun, Jan 06 2019, James Simmons wrote:
> 
> > This patch series contains mostly fixes to resolve the majority of
> > the testing failures seen. Besides that are UAPI headers fixes to
> > prepare for when lustre will leave the staging tree. Include Neil's
> > dump_page_cache patch as well since it was landed in the OpenSFS
> > branch and the migration of lu_objects to rhashtables depended on
> > it.
> 
> Thanks - I've applied these.
> I haven't received all of the PFL series yet ... 16,17,18 still in limbo
> - so I haven't applied those yet.
> When the stragglers here here, I suspect they'll all go in.

Do you recieve patch 16,17,18 ? I do see they made it to the mailing list
archive :

http://lists.lustre.org/pipermail/lustre-devel-lustre.org/2019-January/thread.html

Also they are showing up at the patchwork site:

https://patchwork.kernel.org/project/lustre-devel/list

which is much easier to work with as a patch. Lastly if you need I can 
send you those patches directly to you again. Let me know what you want.

[lustre-devel] [PATCH 00/14] lustre: fixes for many test failures

[NeilBrown]

On Mon, Jan 07 2019, James Simmons wrote:

>> On Sun, Jan 06 2019, James Simmons wrote:
>> 
>> > This patch series contains mostly fixes to resolve the majority of
>> > the testing failures seen. Besides that are UAPI headers fixes to
>> > prepare for when lustre will leave the staging tree. Include Neil's
>> > dump_page_cache patch as well since it was landed in the OpenSFS
>> > branch and the migration of lu_objects to rhashtables depended on
>> > it.
>> 
>> Thanks - I've applied these.
>> I haven't received all of the PFL series yet ... 16,17,18 still in limbo
>> - so I haven't applied those yet.
>> When the stragglers here here, I suspect they'll all go in.
>
> Do you recieve patch 16,17,18 ? I do see they made it to the mailing list
> archive :

Yes, they arrived eventually, thanks.  The whole series seemed to be
coming in dribs and drabs ... some mail server somewhere must have been
a bit sick.

I'll have a look through them all today.

Thanks,
NeilBrown


>
> http://lists.lustre.org/pipermail/lustre-devel-lustre.org/2019-January/thread.html
>
> Also they are showing up at the patchwork site:
>
> https://patchwork.kernel.org/project/lustre-devel/list
>
> which is much easier to work with as a patch. Lastly if you need I can 
> send you those patches directly to you again. Let me know what you want.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20190108/e46e461e/attachment.sig>