[PATCH v5 0/6] nilfs-utils: shortcut for certain GC operations

[PATCH v5 0/6] nilfs-utils: shortcut for certain GC operations

[Andreas Rohner]

Hi,

This patch set adds an optimized version of 
nilfs_xreclaim_segments, which has an additional parameter 
min_reclaimable_blks. This parameter specifies the minimum number of 
reclaimable blocks in a segment, before it can be cleaned. If a 
segment is below this threshold, it is considered to be not worth 
cleaning, because all the live blocks would need to be moved to a 
new segment, which is expensive, and the number of reclaimable 
blocks is too low. But it is still necessary to update the segment 
usage information to turn the old segment into a new one.
    
This is basically a shortcut to cleaning the segment. It is still 
necessary to read the segment summary information, but the writing 
of the live blocks can be skipped if it's not worth it.

Additionally new options are introduced for the configuration file 
and nilfs-clean to allow the user to specify the threshold. 

This is potentially useful for all gc policies, but it is especially
beneficial for the timestamp policy. Lets assume for example a NILFS2
volume with 20% static files and lets assume these static files 
are in the oldest segments. The current timestamp policy will 
select the oldest segments and, since the data is static, move 
them mostly unchanged to new segments. After a while they will 
become the oldest segments again. Then timestamp will move them 
again. These moving operations are expensive and unnecessary.

Regards,
Andreas Rohner

---
v4->v5 (based on review by Ryusuke Konishi)
* Add missing comments and documentation
* Fix nilfs_set_suinfo return value
* Fix coding style issues
* Return error if invalid min_reclaimable_blocks is sent 
  from nilfs-clean command
* Use nilfs_vector instead of malloc in nilfs_xreclaim_segment
* Rename nilfs_reclaim_params flag to
  NILFS_RECLAIM_PARAM_MIN_RECLAIMABLE_BLKS
* Changed the interpretation of deferred_segs in 
  nilfs_reclaim_stat

v3->v4 (based on review by Ryusuke Konishi)
* nilfs-utils: Rebase on refactored nilfs_reclaim_segments 
  (by Ryusuke Konishi)
* nilfs-utils: Add no_timeout if cleaning was deferred

v2->v3 (based on review by Ryusuke Konishi)
* Alignment in nilfs_suinfo_update
* Add missing comments
* nilfs2: Extra validity checks in nilfs_sufile_set_suinfo
* nilfs2: Update sh_ncleansegs, sh_ndirtysegs if flags change
* nilfs2: Fix bugs in nilfs_sufile_set_suinfo
* nilfs2: Use vmalloc instead of ioctl_wrap_copy
* nilfs-utils: Separate nilfs_relaim_segments_with_threshold
* nilfs-utils: Allow different units for min_reclaimable_blocks
* nilfs-utils: Introduce flag to disable the optimization
* nilfs-utils: Disable optimization if kernel retruns ENOTTY
* nilfs-utils: Remove EGCTRYAGAIN error return code
* nilfs-utils: Rename option to min_reclaimable_blocks

v1->v2 (based on review by Ryusuke Konishi)
* Implementation of NILFS_IOCTL_SET_SUINFO
* Added mc_min_free_blocks_threshold config option
  (if clean segments < min_clean_segments)
* Added new command line param for nilfs-clean
* Update man- and config-file
--
Andreas Rohner (6):
  nilfs-utils: cldconfig add an option to set min. reclaimable blocks
  nilfs-utils: add NILFS_OPT_SET_SUINFO
  nilfs-utils: nilfs-clean add cmdline param min-reclaimable-blocks
  nilfs-utils: add suport for NILFS_IOCTL_SET_SUINFO ioctl
  nilfs-utils: add optimized version of nilfs_xreclaim_segments
  nilfs-utils: add a no_timeout flag to enable faster loop

 include/nilfs.h                   |   9 +-
 include/nilfs2_fs.h               |  44 ++++++++++
 include/nilfs_cleaner.h           |  19 +++--
 include/nilfs_gc.h                |  11 +--
 lib/gc.c                          |  73 ++++++++++++++--
 lib/nilfs.c                       |  59 +++++++++++++
 man/nilfs-clean.8                 |   4 +
 man/nilfs_cleanerd.conf.5         |  22 +++++
 sbin/cleanerd/cldconfig.c         | 169 ++++++++++++++++++++++++++++----------
 sbin/cleanerd/cldconfig.h         |  12 +++
 sbin/cleanerd/cleanerd.c          |  68 +++++++++++++--
 sbin/cleanerd/nilfs_cleanerd.conf |  20 +++++
 sbin/nilfs-clean/nilfs-clean.c    |  44 ++++++++--
 13 files changed, 480 insertions(+), 74 deletions(-)

-- 
1.8.5.3

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v5 1/6] nilfs-utils: cldconfig add an option to set min. reclaimable blocks

[Andreas Rohner]

With this option the user can specify the minimum number of
reclaimable blocks of a segment, before it can be cleaned. This is
a threshold for the GC to prevent needless moving of data. If
there are less reclaimable blocks in a segment than the specified
number, the GC will abort and try again with a different segment.

If there are less clean segments than min_clean_segments,
mc_min_reclaimable_blocks is used instead of
min_reclaimable_blocks. This allows for more flexibility in
configuring the GC.

The number of blocks can be specified in percent of a segment or
in bytes.

If the use_set_suinfo switch is not set, the optimization is
completely disabled.

Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
---
 man/nilfs_cleanerd.conf.5         |  22 +++++
 sbin/cleanerd/cldconfig.c         | 169 ++++++++++++++++++++++++++++----------
 sbin/cleanerd/cldconfig.h         |  12 +++
 sbin/cleanerd/cleanerd.c          |  10 +++
 sbin/cleanerd/nilfs_cleanerd.conf |  20 +++++
 5 files changed, 191 insertions(+), 42 deletions(-)

diff --git a/man/nilfs_cleanerd.conf.5 b/man/nilfs_cleanerd.conf.5
index 8f3fcd2..4315ecd 100644
--- a/man/nilfs_cleanerd.conf.5
+++ b/man/nilfs_cleanerd.conf.5
@@ -79,6 +79,28 @@ Specify whether to use \fBmmap\fP(2) for reading segments.  At
 present, this option is enabled if supported regardless of this
 directive.
 .TP
+.B use_set_suinfo
+Specify whether to use the set_suinfo ioctl if it is supported. This is
+necessary for the \fBmin_reclaimable_blocks\fP feature. By disabling this
+switch \fBmin_reclaimable_blocks\fP is also disabled.
+.TP
+.B min_reclaimable_blocks
+Specify the minimum number of reclaimable blocks in a segment before
+it can be cleaned.
+.TP
+.B mc_min_reclaimable_blocks
+Specify the minimum number of reclaimable blocks in a segment before
+it can be cleaned. if clean segments < min_clean_segments.
+.PP
+\fBmin_reclaimable_blocks\fP and \fBmc_min_reclaimable_blocks\fP may
+be followed by a percent sign or the following multiplicative suffixes:
+kB 1000, K 1024, MB 1000*1000, M 1024*1024, GB 1000*1000*1000, G
+1024*1024*1024, and so on for T, P, E.  If the argument is followed by
+a percent sign, it represents the ratio of blocks in a segment.
+.PP
+The default values of \fBmin_reclaimable_blocks\fP and
+\fBmc_min_reclaimable_blocks\fP are 5 percent and 1 percent respectively.
+.TP
 .B log_priority
 Gives the verbosity level that is used when logging messages from
 \fBnilfs_cleanerd\fP(8).  The possible values are: \fBemerg\fP,
diff --git a/sbin/cleanerd/cldconfig.c b/sbin/cleanerd/cldconfig.c
index 270d360..bdd8c85 100644
--- a/sbin/cleanerd/cldconfig.c
+++ b/sbin/cleanerd/cldconfig.c
@@ -278,6 +278,55 @@ nilfs_cldconfig_handle_protection_period(struct nilfs_cldconfig *config,
 }
 
 static unsigned long long
+nilfs_convert_units_to_bytes(const struct nilfs_param *param)
+{
+	unsigned long long bytes = param->num;
+
+	switch (param->unit) {
+	case NILFS_SIZE_UNIT_KB:
+		bytes *= 1000ULL;
+		break;
+	case NILFS_SIZE_UNIT_KIB:
+		bytes <<= 10;
+		break;
+	case NILFS_SIZE_UNIT_MB:
+		bytes *= 1000000ULL;
+		break;
+	case NILFS_SIZE_UNIT_MIB:
+		bytes <<= 20;
+		break;
+	case NILFS_SIZE_UNIT_GB:
+		bytes *= 1000000000ULL;
+		break;
+	case NILFS_SIZE_UNIT_GIB:
+		bytes <<= 30;
+		break;
+	case NILFS_SIZE_UNIT_TB:
+		bytes *= 1000000000000ULL;
+		break;
+	case NILFS_SIZE_UNIT_TIB:
+		bytes <<= 40;
+		break;
+	case NILFS_SIZE_UNIT_PB:
+		bytes *= 1000000000000000ULL;
+		break;
+	case NILFS_SIZE_UNIT_PIB:
+		bytes <<= 50;
+		break;
+	case NILFS_SIZE_UNIT_EB:
+		bytes *= 1000000000000000000ULL;
+		break;
+	case NILFS_SIZE_UNIT_EIB:
+		bytes <<= 60;
+		break;
+	default:
+		assert(0);
+	}
+
+	return bytes;
+}
+
+static unsigned long long
 nilfs_convert_size_to_nsegments(struct nilfs *nilfs, struct nilfs_param *param)
 {
 	unsigned long long ret, segment_size, bytes;
@@ -287,48 +336,7 @@ nilfs_convert_size_to_nsegments(struct nilfs *nilfs, struct nilfs_param *param)
 	} else if (param->unit == NILFS_SIZE_UNIT_PERCENT) {
 		ret = (nilfs_get_nsegments(nilfs) * param->num + 99) / 100;
 	} else {
-		bytes = param->num;
-
-		switch (param->unit) {
-		case NILFS_SIZE_UNIT_KB:
-			bytes *= 1000ULL;
-			break;
-		case NILFS_SIZE_UNIT_KIB:
-			bytes <<= 10;
-			break;
-		case NILFS_SIZE_UNIT_MB:
-			bytes *= 1000000ULL;
-			break;
-		case NILFS_SIZE_UNIT_MIB:
-			bytes <<= 20;
-			break;
-		case NILFS_SIZE_UNIT_GB:
-			bytes *= 1000000000ULL;
-			break;
-		case NILFS_SIZE_UNIT_GIB:
-			bytes <<= 30;
-			break;
-		case NILFS_SIZE_UNIT_TB:
-			bytes *= 1000000000000ULL;
-			break;
-		case NILFS_SIZE_UNIT_TIB:
-			bytes <<= 40;
-			break;
-		case NILFS_SIZE_UNIT_PB:
-			bytes *= 1000000000000000ULL;
-			break;
-		case NILFS_SIZE_UNIT_PIB:
-			bytes <<= 50;
-			break;
-		case NILFS_SIZE_UNIT_EB:
-			bytes *= 1000000000000000000ULL;
-			break;
-		case NILFS_SIZE_UNIT_EIB:
-			bytes <<= 60;
-			break;
-		default:
-			assert(0);
-		}
+		bytes = nilfs_convert_units_to_bytes(param);
 		segment_size = nilfs_get_block_size(nilfs) *
 			nilfs_get_blocks_per_segment(nilfs);
 		ret = (bytes + segment_size - 1) / segment_size;
@@ -455,6 +463,52 @@ nilfs_cldconfig_handle_mc_nsegments_per_clean(struct nilfs_cldconfig *config,
 	return 0;
 }
 
+static unsigned long long
+nilfs_convert_size_to_blocks_per_segment(struct nilfs *nilfs,
+					 struct nilfs_param *param)
+{
+	unsigned long long ret, segment_size, block_size, bytes;
+
+	if (param->unit == NILFS_SIZE_UNIT_NONE) {
+		ret = param->num;
+	} else if (param->unit == NILFS_SIZE_UNIT_PERCENT) {
+		ret = (nilfs_get_blocks_per_segment(nilfs) * param->num) / 100;
+	} else {
+		block_size = nilfs_get_block_size(nilfs);
+		segment_size = block_size *
+				nilfs_get_blocks_per_segment(nilfs);
+		bytes = nilfs_convert_units_to_bytes(param) % segment_size;
+		ret = (bytes + block_size - 1) / block_size;
+	}
+	return ret;
+}
+
+static int
+nilfs_cldconfig_handle_min_reclaimable_blocks(struct nilfs_cldconfig *config,
+					      char **tokens, size_t ntoks,
+					      struct nilfs *nilfs)
+{
+	struct nilfs_param param;
+
+	if (nilfs_cldconfig_get_size_argument(tokens, ntoks, &param) == 0)
+		config->cf_min_reclaimable_blocks =
+			nilfs_convert_size_to_blocks_per_segment(nilfs, &param);
+	return 0;
+}
+
+static int
+nilfs_cldconfig_handle_mc_min_reclaimable_blocks(struct nilfs_cldconfig *config,
+						 char **tokens, size_t ntoks,
+						 struct nilfs *nilfs)
+{
+	struct nilfs_param param;
+
+	if (nilfs_cldconfig_get_size_argument(tokens, ntoks, &param) == 0)
+		config->cf_mc_min_reclaimable_blocks =
+			nilfs_convert_size_to_blocks_per_segment(nilfs, &param);
+	return 0;
+}
+
 static int
 nilfs_cldconfig_handle_cleaning_interval(struct nilfs_cldconfig *config,
 					 char **tokens, size_t ntoks,
@@ -490,6 +544,14 @@ static int nilfs_cldconfig_handle_use_mmap(struct nilfs_cldconfig *config,
 	return 0;
 }
 
+static int nilfs_cldconfig_handle_use_set_suinfo(struct nilfs_cldconfig *config,
+						 char **tokens, size_t ntoks,
+						 struct nilfs *nilfs)
+{
+	config->cf_use_set_suinfo = 1;
+	return 0;
+}
+
 static const struct nilfs_cldconfig_log_priority
 nilfs_cldconfig_log_priority_table[] = {
 	{"emerg",	LOG_EMERG},
@@ -576,6 +638,18 @@ nilfs_cldconfig_keyword_table[] = {
 		"log_priority", 2, 2,
 		nilfs_cldconfig_handle_log_priority
 	},
+	{
+		"min_reclaimable_blocks", 2, 2,
+		nilfs_cldconfig_handle_min_reclaimable_blocks
+	},
+	{
+		"mc_min_reclaimable_blocks", 2, 2,
+		nilfs_cldconfig_handle_mc_min_reclaimable_blocks
+	},
+	{
+		"use_set_suinfo", 1, 1,
+		nilfs_cldconfig_handle_use_set_suinfo
+	},
 };
 
 #define NILFS_CLDCONFIG_NKEYWORDS			\
@@ -640,7 +714,18 @@ static void nilfs_cldconfig_set_default(struct nilfs_cldconfig *config,
 	config->cf_retry_interval.tv_sec = NILFS_CLDCONFIG_RETRY_INTERVAL;
 	config->cf_retry_interval.tv_usec = 0;
 	config->cf_use_mmap = NILFS_CLDCONFIG_USE_MMAP;
+	config->cf_use_set_suinfo = NILFS_CLDCONFIG_USE_SET_SUINFO;
 	config->cf_log_priority = NILFS_CLDCONFIG_LOG_PRIORITY;
+
+	param.num = NILFS_CLDCONFIG_MIN_RECLAIMABLE_BLOCKS;
+	param.unit = NILFS_CLDCONFIG_MIN_RECLAIMABLE_BLOCKS_UNIT;
+	config->cf_min_reclaimable_blocks =
+		nilfs_convert_size_to_blocks_per_segment(nilfs, &param);
+
+	param.num = NILFS_CLDCONFIG_MC_MIN_RECLAIMABLE_BLOCKS;
+	param.unit = NILFS_CLDCONFIG_MC_MIN_RECLAIMABLE_BLOCKS_UNIT;
+	config->cf_mc_min_reclaimable_blocks =
+		nilfs_convert_size_to_blocks_per_segment(nilfs, &param);
 }
 
 static inline int iseol(int c)
diff --git a/sbin/cleanerd/cldconfig.h b/sbin/cleanerd/cldconfig.h
index 188ce9b..0a598d5 100644
--- a/sbin/cleanerd/cldconfig.h
+++ b/sbin/cleanerd/cldconfig.h
@@ -87,7 +87,11 @@ enum nilfs_size_unit {
  * if clean segments < min_clean_segments
  * @cf_retry_interval: retry interval
  * @cf_use_mmap: flag that indicate using mmap
+ * @cf_use_set_suinfo: flag that indicates the use of the set_suinfo ioctl
  * @cf_log_priority: log priority level
+ * @cf_min_reclaimable_blocks: minimum reclaimable blocks for cleaning
+ * @cf_mc_min_reclaimable_blocks: minimum reclaimable blocks for cleaning
+ * if clean segments < min_clean_segments
  */
 struct nilfs_cldconfig {
 	struct nilfs_selection_policy cf_selection_policy;
@@ -101,7 +105,10 @@ struct nilfs_cldconfig {
 	struct timeval cf_mc_cleaning_interval;
 	struct timeval cf_retry_interval;
 	int cf_use_mmap;
+	int cf_use_set_suinfo;
 	int cf_log_priority;
+	unsigned long cf_min_reclaimable_blocks;
+	unsigned long cf_mc_min_reclaimable_blocks;
 };
 
 #define NILFS_CLDCONFIG_SELECTION_POLICY_IMPORTANCE	\
@@ -119,7 +126,12 @@ struct nilfs_cldconfig {
 #define NILFS_CLDCONFIG_MC_CLEANING_INTERVAL		1
 #define NILFS_CLDCONFIG_RETRY_INTERVAL			60
 #define NILFS_CLDCONFIG_USE_MMAP			1
+#define NILFS_CLDCONFIG_USE_SET_SUINFO			0
 #define NILFS_CLDCONFIG_LOG_PRIORITY			LOG_INFO
+#define NILFS_CLDCONFIG_MIN_RECLAIMABLE_BLOCKS		5
+#define NILFS_CLDCONFIG_MIN_RECLAIMABLE_BLOCKS_UNIT	NILFS_SIZE_UNIT_PERCENT
+#define NILFS_CLDCONFIG_MC_MIN_RECLAIMABLE_BLOCKS	1
+#define NILFS_CLDCONFIG_MC_MIN_RECLAIMABLE_BLOCKS_UNIT	NILFS_SIZE_UNIT_PERCENT
 
 #define NILFS_CLDCONFIG_NSEGMENTS_PER_CLEAN_MAX	32
 
diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
index 1d09b5b..c58a542 100644
--- a/sbin/cleanerd/cleanerd.c
+++ b/sbin/cleanerd/cleanerd.c
@@ -145,6 +145,7 @@ const static struct option long_option[] = {
  * @cleaning_interval: cleaning interval
  * @target: target time for sleeping
  * @timeout: timeout value for sleeping
+ * @min_reclaimable_blocks: min. number of reclaimable blocks
  * @prev_nongc_ctime: previous nongc ctime
  * @recvq: receive queue
  * @recvq_name: receive queue name
@@ -172,6 +173,7 @@ struct nilfs_cleanerd {
 	struct timeval cleaning_interval;
 	struct timeval target;
 	struct timeval timeout;
+	unsigned long min_reclaimable_blocks;
 	__u64 prev_nongc_ctime;
 	mqd_t recvq;
 	char *recvq_name;
@@ -273,6 +275,8 @@ static int nilfs_cleanerd_reconfig(struct nilfs_cleanerd *cleanerd,
 	} else {
 		cleanerd->ncleansegs = config->cf_nsegments_per_clean;
 		cleanerd->cleaning_interval = config->cf_cleaning_interval;
+		cleanerd->min_reclaimable_blocks =
+				config->cf_min_reclaimable_blocks;
 		syslog(LOG_INFO, "configuration file reloaded");
 	}
 	return ret;
@@ -1238,10 +1242,14 @@ static int nilfs_cleanerd_handle_clean_check(struct nilfs_cleanerd *cleanerd,
 		/* disk space is close to limit -- accelerate cleaning */
 		cleanerd->ncleansegs = config->cf_mc_nsegments_per_clean;
 		cleanerd->cleaning_interval = config->cf_mc_cleaning_interval;
+		cleanerd->min_reclaimable_blocks =
+				config->cf_mc_min_reclaimable_blocks;
 	} else {
 		/* continue to run */
 		cleanerd->ncleansegs = config->cf_nsegments_per_clean;
 		cleanerd->cleaning_interval = config->cf_cleaning_interval;
+		cleanerd->min_reclaimable_blocks =
+				config->cf_min_reclaimable_blocks;
 	}
 
 	return 0; /* do gc */
@@ -1442,6 +1450,8 @@ static int nilfs_cleanerd_clean_loop(struct nilfs_cleanerd *cleanerd)
 
 	cleanerd->ncleansegs = cleanerd->config.cf_nsegments_per_clean;
 	cleanerd->cleaning_interval = cleanerd->config.cf_cleaning_interval;
+	cleanerd->min_reclaimable_blocks =
+			cleanerd->config.cf_min_reclaimable_blocks;
 
 
 	if (nilfs_cleanerd_automatic_suspend(cleanerd))
diff --git a/sbin/cleanerd/nilfs_cleanerd.conf b/sbin/cleanerd/nilfs_cleanerd.conf
index 26872aa..79d3fce 100644
--- a/sbin/cleanerd/nilfs_cleanerd.conf
+++ b/sbin/cleanerd/nilfs_cleanerd.conf
@@ -51,6 +51,26 @@ mc_cleaning_interval	1
 # Retry interval in seconds.
 retry_interval		60
 
+# Specify the minimum number of reclaimable blocks in a segment
+# before it can be cleaned.
+min_reclaimable_blocks	5%
+
+# Specify the minimum number of reclaimable blocks in a segment
+# before it can be cleaned.
+# if clean segments < min_clean_segments
+mc_min_reclaimable_blocks	1%
+
+# The argument of min_reclaimable_blocks and mc_min_reclaimable_blocks
+# can be followed by a percent sign (%) or one of the following
+# multiplicative suffixes similar to min_clean_segments.
+#
+# If the argument is followed by "%", it represents a ratio for the
+# number of blocks per segment.
+
+# enable set_suinfo ioctl if supported
+# (needed for min_reclaimable_blocks)
+#use_set_suinfo
+
 # Use mmap when reading segments if supported.
 use_mmap
 
-- 
1.8.5.3

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v5 2/6] nilfs-utils: add NILFS_OPT_SET_SUINFO

[Andreas Rohner]

This option for the nilfs object corresponds to the use_set_suinfo
flag in the configuration file. This patch adds the flag itself
plus some convenience functions.

The option is set after the configuration file was read in.

Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
---
 include/nilfs.h          |  7 ++++++-
 lib/nilfs.c              | 28 ++++++++++++++++++++++++++++
 sbin/cleanerd/cleanerd.c |  6 ++++++
 3 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/include/nilfs.h b/include/nilfs.h
index 56286a9..b5f85d3 100644
--- a/include/nilfs.h
+++ b/include/nilfs.h
@@ -128,7 +128,8 @@ struct nilfs {
 #define NILFS_OPEN_RDWR		0x0008	/* Open NILFS API in read/write mode */
 #define NILFS_OPEN_GCLK		0x1000	/* Open GC lock primitive */
 
-#define NILFS_OPT_MMAP	0x01
+#define NILFS_OPT_MMAP		0x01
+#define NILFS_OPT_SET_SUINFO	0x02
 
 
 struct nilfs *nilfs_open(const char *, const char *, int);
@@ -140,6 +141,10 @@ void nilfs_opt_clear_mmap(struct nilfs *);
 int nilfs_opt_set_mmap(struct nilfs *);
 int nilfs_opt_test_mmap(struct nilfs *);
 
+void nilfs_opt_clear_set_suinfo(struct nilfs *);
+int nilfs_opt_set_set_suinfo(struct nilfs *);
+int nilfs_opt_test_set_suinfo(struct nilfs *);
+
 nilfs_cno_t nilfs_get_oldest_cno(struct nilfs *);
 
 struct nilfs_super_block *nilfs_get_sb(struct nilfs *);
diff --git a/lib/nilfs.c b/lib/nilfs.c
index c1771e8..93822de 100644
--- a/lib/nilfs.c
+++ b/lib/nilfs.c
@@ -288,6 +288,34 @@ int nilfs_opt_test_mmap(struct nilfs *nilfs)
 	return !!(nilfs->n_opts & NILFS_OPT_MMAP);
 }
 
+/**
+ * nilfs_opt_set_set_suinfo - set set_suinfo option
+ * @nilfs: nilfs object
+ */
+int nilfs_opt_set_set_suinfo(struct nilfs *nilfs)
+{
+	nilfs->n_opts |= NILFS_OPT_SET_SUINFO;
+	return 0;
+}
+
+/**
+ * nilfs_opt_clear_set_suinfo - clear set_suinfo option
+ * @nilfs: nilfs object
+ */
+void nilfs_opt_clear_set_suinfo(struct nilfs *nilfs)
+{
+	nilfs->n_opts &= ~NILFS_OPT_SET_SUINFO;
+}
+
+/**
+ * nilfs_opt_test_set_suinfo - test whether set_suinfo option is set or not
+ * @nilfs: nilfs object
+ */
+int nilfs_opt_test_set_suinfo(struct nilfs *nilfs)
+{
+	return !!(nilfs->n_opts & NILFS_OPT_SET_SUINFO);
+}
+
 static int nilfs_open_sem(struct nilfs *nilfs)
 {
 	char semnambuf[NAME_MAX - 4];
diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
index c58a542..4624f92 100644
--- a/sbin/cleanerd/cleanerd.c
+++ b/sbin/cleanerd/cleanerd.c
@@ -248,6 +248,12 @@ static int nilfs_cleanerd_config(struct nilfs_cleanerd *cleanerd,
 	else
 		nilfs_opt_clear_mmap(cleanerd->nilfs);
 #endif	/* HAVE_MMAP */
+
+	if (cleanerd->config.cf_use_set_suinfo)
+		nilfs_opt_set_set_suinfo(cleanerd->nilfs);
+	else
+		nilfs_opt_clear_set_suinfo(cleanerd->nilfs);
+
 	nilfs_cleanerd_set_log_priority(cleanerd);
 
 	if (protection_period != ULONG_MAX) {
-- 
1.8.5.3

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v5 3/6] nilfs-utils: nilfs-clean add cmdline param min-reclaimable-blocks

[Andreas Rohner]

This patch adds a command line parameter for nilfs-clean to allow
the manual override of the min_reclaimable_blocks option.

Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
---
 include/nilfs_cleaner.h        | 19 +++++++++---------
 man/nilfs-clean.8              |  4 ++++
 sbin/cleanerd/cleanerd.c       | 21 ++++++++++++++++++++
 sbin/nilfs-clean/nilfs-clean.c | 44 ++++++++++++++++++++++++++++++++++++------
 4 files changed, 73 insertions(+), 15 deletions(-)

diff --git a/include/nilfs_cleaner.h b/include/nilfs_cleaner.h
index 0bf02aa..874c17a 100644
--- a/include/nilfs_cleaner.h
+++ b/include/nilfs_cleaner.h
@@ -46,17 +46,18 @@ struct nilfs_cleaner_args {
 	uint64_t start_segnum;	/* start segment number */
 	uint64_t nsegs;		/* number of segments */
 	uint32_t runtime; /* runtime in seconds */
-	uint32_t pad2;
+	uint32_t min_reclaimable_blocks;
 };
 /* valid flags */
-#define NILFS_CLEANER_ARG_PROTECTION_PERIOD	(1 << 0)
-#define NILFS_CLEANER_ARG_NSEGMENTS_PER_CLEAN	(1 << 1)
-#define NILFS_CLEANER_ARG_CLEANING_INTERVAL	(1 << 2)
-#define NILFS_CLEANER_ARG_USAGE_RATE_THRESHOLD	(1 << 3) /* reserved */
-#define NILFS_CLEANER_ARG_START_SEGNUM		(1 << 4) /* reserved */
-#define NILFS_CLEANER_ARG_NSEGS			(1 << 5) /* reserved */
-#define NILFS_CLEANER_ARG_NPASSES		(1 << 6) /* reserved */
-#define NILFS_CLEANER_ARG_RUNTIME		(1 << 7) /* reserved */
+#define NILFS_CLEANER_ARG_PROTECTION_PERIOD		(1 << 0)
+#define NILFS_CLEANER_ARG_NSEGMENTS_PER_CLEAN		(1 << 1)
+#define NILFS_CLEANER_ARG_CLEANING_INTERVAL		(1 << 2)
+#define NILFS_CLEANER_ARG_USAGE_RATE_THRESHOLD		(1 << 3) /* reserved */
+#define NILFS_CLEANER_ARG_START_SEGNUM			(1 << 4) /* reserved */
+#define NILFS_CLEANER_ARG_NSEGS				(1 << 5) /* reserved */
+#define NILFS_CLEANER_ARG_NPASSES			(1 << 6) /* reserved */
+#define NILFS_CLEANER_ARG_RUNTIME			(1 << 7) /* reserved */
+#define NILFS_CLEANER_ARG_MIN_RECLAIMABLE_BLOCKS	(1 << 8)
 
 enum {
 	NILFS_CLEANER_STATUS_IDLE,
diff --git a/man/nilfs-clean.8 b/man/nilfs-clean.8
index 04e11c6..94267e1 100644
--- a/man/nilfs-clean.8
+++ b/man/nilfs-clean.8
@@ -43,6 +43,10 @@ Display cleaner status.
 \fB\-h\fR, \fB\-\-help\fR
 Display help message and exit.
 .TP
+\fB\-m\fR, \fB\-\-min\-reclaimable\-blocks=\fICOUNT\fR
+Specify the minimum number of reclaimable blocks in a segment before
+it can be cleaned.
+.TP
 \fB\-p\fR, \fB\-\-protection-period=\fIinterval\fR
 Set protection period for a cleaner run.  The \fIinterval\fR parameter
 is an integer value and specifies the minimum time that deleted or
diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
index 4624f92..2896af8 100644
--- a/sbin/cleanerd/cleanerd.c
+++ b/sbin/cleanerd/cleanerd.c
@@ -159,6 +159,7 @@ const static struct option long_option[] = {
  * @mm_ncleansegs: number of segmetns cleaned per cycle (manual mode)
  * @mm_protection_period: protection period (manual mode)
  * @mm_cleaning_interval: cleaning interval (manual mode)
+ * @mm_min_reclaimable_blocks: min. number of reclaimable blocks (manual mode)
  */
 struct nilfs_cleanerd {
 	struct nilfs *nilfs;
@@ -186,6 +187,7 @@ struct nilfs_cleanerd {
 	long mm_ncleansegs;
 	struct timeval mm_protection_period;
 	struct timeval mm_cleaning_interval;
+	unsigned long mm_min_reclaimable_blocks;
 };
 
 /**
@@ -469,6 +471,14 @@ nilfs_cleanerd_protection_period(struct nilfs_cleanerd *cleanerd)
 		&cleanerd->config.cf_protection_period;
 }
 
+static unsigned long
+nilfs_cleanerd_min_reclaimable_blocks(struct nilfs_cleanerd *cleanerd)
+{
+	return cleanerd->running == 2 ?
+		cleanerd->mm_min_reclaimable_blocks :
+		cleanerd->min_reclaimable_blocks;
+}
+
 static void
 nilfs_cleanerd_reduce_ncleansegs_for_retry(struct nilfs_cleanerd *cleanerd)
 {
@@ -1011,6 +1021,17 @@ static int nilfs_cleanerd_cmd_run(struct nilfs_cleanerd *cleanerd,
 		cleanerd->mm_cleaning_interval =
 			cleanerd->cleaning_interval;
 	}
+	/* minimal reclaimable blocks */
+	if (req2->args.valid & NILFS_CLEANER_ARG_MIN_RECLAIMABLE_BLOCKS) {
+		if (req2->args.min_reclaimable_blocks >
+		    nilfs_get_blocks_per_segment(cleanerd->nilfs))
+			goto error_inval;
+		cleanerd->mm_min_reclaimable_blocks =
+			req2->args.min_reclaimable_blocks;
+	} else {
+		cleanerd->mm_min_reclaimable_blocks =
+			cleanerd->min_reclaimable_blocks;
+	}
 	/* number of passes */
 	if (req2->args.valid & NILFS_CLEANER_ARG_NPASSES) {
 		if (!req2->args.npasses)
diff --git a/sbin/nilfs-clean/nilfs-clean.c b/sbin/nilfs-clean/nilfs-clean.c
index 55abd55..f77fdf7 100644
--- a/sbin/nilfs-clean/nilfs-clean.c
+++ b/sbin/nilfs-clean/nilfs-clean.c
@@ -77,6 +77,7 @@ const static struct option long_option[] = {
 	{"stop", no_argument, NULL, 'b'},
 	{"suspend", no_argument, NULL, 's'},
 	{"speed", required_argument, NULL, 'S'},
+	{"min-reclaimable-blocks", required_argument, NULL, 'm'},
 	{"verbose", no_argument, NULL, 'v'},
 	{"version", no_argument, NULL, 'V'},
 	{NULL, 0, NULL, 0}
@@ -90,6 +91,9 @@ const static struct option long_option[] = {
 	"  -l, --status\t\tdisplay cleaner status\n"			\
 	"  -p, --protection-period=SECONDS\n"				\
 	"               \t\tspecify protection period\n"		\
+	"  -m, --min-reclaimable-blocks=COUNT\n"			\
+	"               \t\tset minimum number of reclaimable blocks\n"	\
+	"               \t\tbefore a segment can be cleaned\n"		\
 	"  -q, --quit\t\tshutdown cleaner\n"				\
 	"  -r, --resume\t\tresume cleaner\n"				\
 	"  -s, --suspend\t\tsuspend cleaner\n"				\
@@ -98,9 +102,10 @@ const static struct option long_option[] = {
 	"  -v, --verbose\t\tverbose mode\n"				\
 	"  -V, --version\t\tdisplay version and exit\n"
 #else
-#define NILFS_CLEAN_USAGE						\
-	"Usage: %s [-b] [-c [conffile]] [-h] [-l] [-p protection-period]" \
-	"          [-q] [-r] [-s] [-S gc-speed] [-v] [-V] [device]\n"
+#define NILFS_CLEAN_USAGE						  \
+	"Usage: %s [-b] [-c [conffile]] [-h] [-l] [-m blocks]\n"	  \
+	"          [-p protection-period] [-q] [-r] [-s] [-S gc-speed]\n" \
+	"          [-v] [-V] [device]\n"
 #endif	/* _GNU_SOURCE */
 
 
@@ -124,6 +129,7 @@ static const char *conffile = NULL;
 static unsigned long protection_period = ULONG_MAX;
 static int nsegments_per_clean = 2;
 static struct timespec cleaning_interval = { 0, 100000000 };   /* 100 msec */
+static unsigned long min_reclaimable_blocks = 20;  /* about 1% with 8MB segs */
 
 static sigjmp_buf nilfs_clean_env;
 static struct nilfs_cleaner *nilfs_cleaner;
@@ -164,9 +170,11 @@ static int nilfs_clean_do_run(struct nilfs_cleaner *cleaner)
 	args.nsegments_per_clean = nsegments_per_clean;
 	args.cleaning_interval = cleaning_interval.tv_sec;
 	args.cleaning_interval_nsec = cleaning_interval.tv_nsec;
+	args.min_reclaimable_blocks = min_reclaimable_blocks;
 	args.valid = (NILFS_CLEANER_ARG_NPASSES |
 		      NILFS_CLEANER_ARG_CLEANING_INTERVAL |
-		      NILFS_CLEANER_ARG_NSEGMENTS_PER_CLEAN);
+		      NILFS_CLEANER_ARG_NSEGMENTS_PER_CLEAN |
+		      NILFS_CLEANER_ARG_MIN_RECLAIMABLE_BLOCKS);
 
 	if (protection_period != ULONG_MAX) {
 		args.protection_period = protection_period;
@@ -426,6 +434,26 @@ failed_too_large:
 	return -1;
 }
 
+static int nilfs_clean_parse_min_reclaimable(const char *arg)
+{
+	unsigned long blocks;
+	char *endptr;
+
+	blocks = strtoul(arg, &endptr, 10);
+	if (endptr == arg || endptr[0] != '\0') {
+		myprintf(_("Error: invalid reclaimable blocks: %s\n"), arg);
+		return -1;
+	}
+
+	if (blocks == ULONG_MAX) {
+		myprintf(_("Error: value too large: %s\n"), arg);
+		return -1;
+	}
+
+	min_reclaimable_blocks = blocks;
+	return 0;
+}
+
 static void nilfs_clean_parse_options(int argc, char *argv[])
 {
 #ifdef _GNU_SOURCE
@@ -434,10 +462,10 @@ static void nilfs_clean_parse_options(int argc, char *argv[])
 	int c;
 
 #ifdef _GNU_SOURCE
-	while ((c = getopt_long(argc, argv, "bc::hlp:qrsS:vV",
+	while ((c = getopt_long(argc, argv, "bc::hlm:p:qrsS:vV",
 				long_option, &option_index)) >= 0) {
 #else
-	while ((c = getopt(argc, argv, "bc::hlp:qrsS:vV")) >= 0) {
+	while ((c = getopt(argc, argv, "bc::hlm:p:qrsS:vV")) >= 0) {
 #endif	/* _GNU_SOURCE */
 		switch (c) {
 		case 'b':
@@ -455,6 +483,10 @@ static void nilfs_clean_parse_options(int argc, char *argv[])
 		case 'l':
 			clean_cmd = NILFS_CLEAN_CMD_INFO;
 			break;
+		case 'm':
+			if (nilfs_clean_parse_min_reclaimable(optarg) < 0)
+				exit(EXIT_FAILURE);
+			break;
 		case 'p':
 			if (nilfs_clean_parse_protection_period(optarg) < 0)
 				exit(EXIT_FAILURE);
-- 
1.8.5.3

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v5 4/6] nilfs-utils: add suport for NILFS_IOCTL_SET_SUINFO ioctl

[Andreas Rohner]

This patch adds library support for the NILFS_IOCTL_SET_SUINFO
ioctl. A new function nilfs_set_suinfo and the structure
nilfs_suinfo_update, which contains the update information, are
added.

With this function the segment usage information can be updated
from userspace, which can be used as a shortcut for certain GC
operations.

Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
---
 include/nilfs.h     |  2 ++
 include/nilfs2_fs.h | 44 ++++++++++++++++++++++++++++++++++++++++++++
 lib/nilfs.c         | 31 +++++++++++++++++++++++++++++++
 3 files changed, 77 insertions(+)

diff --git a/include/nilfs.h b/include/nilfs.h
index b5f85d3..fab8ff2 100644
--- a/include/nilfs.h
+++ b/include/nilfs.h
@@ -304,6 +304,8 @@ int nilfs_delete_checkpoint(struct nilfs *, nilfs_cno_t);
 int nilfs_get_cpstat(const struct nilfs *, struct nilfs_cpstat *);
 ssize_t nilfs_get_suinfo(const struct nilfs *, __u64, struct nilfs_suinfo *,
 			 size_t);
+int nilfs_set_suinfo(const struct nilfs *, struct nilfs_suinfo_update *,
+		     size_t);
 int nilfs_get_sustat(const struct nilfs *, struct nilfs_sustat *);
 ssize_t nilfs_get_vinfo(const struct nilfs *, struct nilfs_vinfo *, size_t);
 ssize_t nilfs_get_bdescs(const struct nilfs *, struct nilfs_bdesc *, size_t);
diff --git a/include/nilfs2_fs.h b/include/nilfs2_fs.h
index e674f44..a16ad4c 100644
--- a/include/nilfs2_fs.h
+++ b/include/nilfs2_fs.h
@@ -714,6 +714,48 @@ static inline int nilfs_suinfo_clean(const struct nilfs_suinfo *si)
 }
 
 /* ioctl */
+/**
+ * nilfs_suinfo_update - segment usage information update
+ * @sup_segnum: segment number
+ * @sup_flags: flags for which fields are active in sup_sui
+ * @sup_reserved: reserved necessary for alignment
+ * @sup_sui: segment usage information
+ */
+struct nilfs_suinfo_update {
+	__u64 sup_segnum;
+	__u32 sup_flags;
+	__u32 sup_reserved;
+	struct nilfs_suinfo sup_sui;
+};
+
+enum {
+	NILFS_SUINFO_UPDATE_LASTMOD,
+	NILFS_SUINFO_UPDATE_NBLOCKS,
+	NILFS_SUINFO_UPDATE_FLAGS,
+	__NR_NILFS_SUINFO_UPDATE_FIELDS,
+};
+
+#define NILFS_SUINFO_UPDATE_FNS(flag, name)				\
+static inline void							\
+nilfs_suinfo_update_set_##name(struct nilfs_suinfo_update *sup)		\
+{									\
+	sup->sup_flags |= 1UL << NILFS_SUINFO_UPDATE_##flag;		\
+}									\
+static inline void							\
+nilfs_suinfo_update_clear_##name(struct nilfs_suinfo_update *sup)	\
+{									\
+	sup->sup_flags &= ~(1UL << NILFS_SUINFO_UPDATE_##flag);		\
+}									\
+static inline int							\
+nilfs_suinfo_update_##name(const struct nilfs_suinfo_update *sup)	\
+{									\
+	return !!(sup->sup_flags & (1UL << NILFS_SUINFO_UPDATE_##flag));\
+}
+
+NILFS_SUINFO_UPDATE_FNS(LASTMOD, lastmod)
+NILFS_SUINFO_UPDATE_FNS(NBLOCKS, nblocks)
+NILFS_SUINFO_UPDATE_FNS(FLAGS, flags)
+
 enum {
 	NILFS_CHECKPOINT,
 	NILFS_SNAPSHOT,
@@ -867,5 +909,7 @@ struct nilfs_bdesc {
 	_IOW(NILFS_IOCTL_IDENT, 0x8B, __u64)
 #define NILFS_IOCTL_SET_ALLOC_RANGE  \
 	_IOW(NILFS_IOCTL_IDENT, 0x8C, __u64[2])
+#define NILFS_IOCTL_SET_SUINFO  \
+	_IOW(NILFS_IOCTL_IDENT, 0x8D, struct nilfs_argv)
 
 #endif	/* _LINUX_NILFS_FS_H */
diff --git a/lib/nilfs.c b/lib/nilfs.c
index 93822de..7317205 100644
--- a/lib/nilfs.c
+++ b/lib/nilfs.c
@@ -596,6 +596,37 @@ ssize_t nilfs_get_suinfo(const struct nilfs *nilfs, __u64 segnum,
 }
 
 /**
+ * nilfs_set_suinfo - sets segment usage info
+ * @nilfs: nilfs object
+ * @sup: an array of nilfs_suinfo_update structs
+ * @nsup: number of elements in sup
+ *
+ * Description: Takes an array of nilfs_suinfo_update structs and updates
+ * segment usage info accordingly. Only the fields indicated by sup_flags
+ * are updated.
+ *
+ * Return Value: On success, 0 is returned. On error, -1 is returned.
+ */
+int nilfs_set_suinfo(const struct nilfs *nilfs,
+		     struct nilfs_suinfo_update *sup, size_t nsup)
+{
+	struct nilfs_argv argv;
+
+	if (nilfs->n_iocfd < 0) {
+		errno = EBADF;
+		return -1;
+	}
+
+	argv.v_base = (unsigned long)sup;
+	argv.v_nmembs = nsup;
+	argv.v_size = sizeof(struct nilfs_suinfo_update);
+	argv.v_index = 0;
+	argv.v_flags = 0;
+
+	return ioctl(nilfs->n_iocfd, NILFS_IOCTL_SET_SUINFO, &argv);
+}
+
+/**
  * nilfs_get_sustat - get segment usage statistics
  * @nilfs: nilfs object
  * @sustat: buffer of a nilfs_sustat struct to store statistics in
-- 
1.8.5.3

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v5 5/6] nilfs-utils: add optimized version of nilfs_xreclaim_segments

[Andreas Rohner]

This patch adds an additional parameter min_reclaimable_blks to
the nilfs_reclaim_params structure. This parameter specifies the
minimum number of reclaimable blocks in a segment, before it can be
cleaned. If a segment is below this threshold, it is considered to
be not worth cleaning, because all the live blocks would need to be
moved to a new segment, which is expensive, and the number of
reclaimable blocks is too low. But it is still necessary to update
the segment usage information to turn the old segment into a new
one.

This is basically a shortcut to cleaning the segment. It is still
necessary to read the segment summary information, but the writing
of the live blocks can be skipped if it's not worth it.

This optimization can be enabled and disabled by the
use_set_suinfo switch in the configuration file. If the kernel
doesn't support the set_suinfo ioctl it returns the error
number ENOTTY, which also permanently disables the optimization.

Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
---
 include/nilfs_gc.h       | 11 ++++----
 lib/gc.c                 | 73 ++++++++++++++++++++++++++++++++++++++++++++----
 sbin/cleanerd/cleanerd.c | 24 +++++++++++++---
 3 files changed, 93 insertions(+), 15 deletions(-)

diff --git a/include/nilfs_gc.h b/include/nilfs_gc.h
index e49cbf4..8bd83d7 100644
--- a/include/nilfs_gc.h
+++ b/include/nilfs_gc.h
@@ -15,20 +15,21 @@
 #include "nilfs.h"
 
 /* flags for nilfs_reclaim_params struct */
-#define NILFS_RECLAIM_PARAM_PROTSEQ	(1UL << 0)
-#define NILFS_RECLAIM_PARAM_PROTCNO	(1UL << 1)
-#define __NR_NILFS_RECLAIM_PARAMS	2
+#define NILFS_RECLAIM_PARAM_PROTSEQ			(1UL << 0)
+#define NILFS_RECLAIM_PARAM_PROTCNO			(1UL << 1)
+#define NILFS_RECLAIM_PARAM_MIN_RECLAIMABLE_BLKS	(1UL << 2)
+#define __NR_NILFS_RECLAIM_PARAMS	3
 
 /**
  * struct nilfs_reclaim_params - structure to specify GC parameters
  * @flags: flags of valid fields
- * @reserved: padding bytes
+ * @min_reclaimable_blks: minimum number of reclaimable blocks
  * @protseq: start of sequence number of protected segments
  * @protcno: start number of checkpoint to be protected
  */
 struct nilfs_reclaim_params {
 	unsigned long flags;
-	unsigned long reserved;
+	unsigned long min_reclaimable_blks;
 	__u64 protseq;
 	nilfs_cno_t protcno;
 };
diff --git a/lib/gc.c b/lib/gc.c
index 71c7307..5a1c2fe 100644
--- a/lib/gc.c
+++ b/lib/gc.c
@@ -29,6 +29,10 @@
 #include <syslog.h>
 #endif	/* HAVE_SYSLOG_H */
 
+#if HAVE_SYS_TIME_H
+#include <sys/time.h>
+#endif	/* HAVE_SYS_TIME */
+
 #include <errno.h>
 #include <assert.h>
 #include <stdarg.h>
@@ -614,11 +618,14 @@ int nilfs_xreclaim_segment(struct nilfs *nilfs,
 			   const struct nilfs_reclaim_params *params,
 			   struct nilfs_reclaim_stat *stat)
 {
-	struct nilfs_vector *vdescv, *bdescv, *periodv, *vblocknrv;
+	struct nilfs_vector *vdescv, *bdescv, *periodv, *vblocknrv, *supv;
 	sigset_t sigset, oldset, waitset;
 	nilfs_cno_t protcno;
-	ssize_t n, ret = -1;
+	ssize_t n, i, ret = -1;
 	size_t nblocks;
+	__u32 reclaimable_blocks;
+	struct nilfs_suinfo_update *sup;
+	struct timeval tv;
 
 	if (!(params->flags & NILFS_RECLAIM_PARAM_PROTSEQ) ||
 	    (params->flags & (~0UL << __NR_NILFS_RECLAIM_PARAMS))) {
@@ -637,7 +644,8 @@ int nilfs_xreclaim_segment(struct nilfs *nilfs,
 	bdescv = nilfs_vector_create(sizeof(struct nilfs_bdesc));
 	periodv = nilfs_vector_create(sizeof(struct nilfs_period));
 	vblocknrv = nilfs_vector_create(sizeof(__u64));
-	if (!vdescv || !bdescv || !periodv || !vblocknrv)
+	supv = nilfs_vector_create(sizeof(struct nilfs_suinfo_update));
+	if (!vdescv || !bdescv || !periodv || !vblocknrv || !supv)
 		goto out_vec;
 
 	sigemptyset(&sigset);
@@ -703,13 +711,16 @@ int nilfs_xreclaim_segment(struct nilfs *nilfs,
 	if (ret < 0)
 		goto out_lock;
 
+	reclaimable_blocks = (nilfs_get_blocks_per_segment(nilfs) * n) -
+			(nilfs_vector_get_size(vdescv) +
+			nilfs_vector_get_size(bdescv));
+
 	if (stat) {
 		stat->live_pblks = nilfs_vector_get_size(bdescv);
 		stat->defunct_pblks = nblocks - stat->live_pblks;
 
 		stat->live_blks = stat->live_vblks + stat->live_pblks;
-		stat->defunct_blks = n * nilfs_get_blocks_per_segment(nilfs) -
-			stat->live_blks;
+		stat->defunct_blks = reclaimable_blocks;
 	}
 	if (dryrun)
 		goto out_lock;
@@ -725,6 +736,54 @@ int nilfs_xreclaim_segment(struct nilfs *nilfs,
 		goto out_lock;
 	}
 
+	/*
+	 * if there are less reclaimable blocks than the minimal
+	 * threshold try to update suinfo instead of cleaning
+	 */
+	if ((params->flags & NILFS_RECLAIM_PARAM_MIN_RECLAIMABLE_BLKS) &&
+			reclaimable_blocks < params->min_reclaimable_blks * n) {
+		if (stat) {
+			stat->deferred_segs = n;
+			stat->cleaned_segs = 0;
+		}
+
+		ret = gettimeofday(&tv, NULL);
+		if (ret < 0)
+			goto out_lock;
+
+		for (i = 0; i < n; ++i) {
+			sup = nilfs_vector_get_new_element(supv);
+			if (!sup)
+				goto out_lock;
+
+			sup->sup_segnum = segnums[i];
+			sup->sup_flags = 0;
+			nilfs_suinfo_update_set_lastmod(sup);
+			sup->sup_sui.sui_lastmod = tv.tv_sec;
+		}
+
+		ret = nilfs_set_suinfo(nilfs, nilfs_vector_get_data(supv), n);
+
+		if (ret == 0)
+			goto out_lock;
+
+		if (ret < 0 && errno != ENOTTY) {
+			nilfs_gc_logger(LOG_ERR, "cannot set suinfo: %s",
+					strerror(errno));
+			goto out_lock;
+		}
+
+		/* errno == ENOTTY */
+		nilfs_gc_logger(LOG_WARNING,
+				"set_suinfo ioctl is not supported");
+		nilfs_opt_clear_set_suinfo(nilfs);
+		if (stat) {
+			stat->deferred_segs = 0;
+			stat->cleaned_segs = n;
+		}
+		/* Try nilfs_clean_segments */
+	}
+
 	ret = nilfs_clean_segments(nilfs,
 				   nilfs_vector_get_data(vdescv),
 				   nilfs_vector_get_size(vdescv),
@@ -759,6 +818,8 @@ out_vec:
 		nilfs_vector_destroy(periodv);
 	if (vblocknrv != NULL)
 		nilfs_vector_destroy(vblocknrv);
+	if (supv != NULL)
+		nilfs_vector_destroy(supv);
 	/*
 	 * Flags of valid fields in stat->exflags must be unset.
 	 */
@@ -783,7 +844,7 @@ ssize_t nilfs_reclaim_segment(struct nilfs *nilfs,
 
 	params.flags =
 		NILFS_RECLAIM_PARAM_PROTSEQ | NILFS_RECLAIM_PARAM_PROTCNO;
-	params.reserved = 0;
+	params.min_reclaimable_blks = 0;
 	params.protseq = protseq;
 	params.protcno = protcno;
 	memset(&stat, 0, sizeof(stat));
diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
index 2896af8..4c9f69b 100644
--- a/sbin/cleanerd/cleanerd.c
+++ b/sbin/cleanerd/cleanerd.c
@@ -1376,7 +1376,7 @@ static void nilfs_cleanerd_progress(struct nilfs_cleanerd *cleanerd, int nsegs)
 static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
 					 __u64 *segnums, size_t nsegs,
 					 __u64 protseq, __u64 prottime,
-					 size_t *ncleaned)
+					 size_t *ndone)
 {
 	struct nilfs_reclaim_params params;
 	struct nilfs_reclaim_stat stat;
@@ -1384,7 +1384,11 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
 
 	params.flags =
 		NILFS_RECLAIM_PARAM_PROTSEQ | NILFS_RECLAIM_PARAM_PROTCNO;
-	params.reserved = 0;
+	if (nilfs_opt_test_set_suinfo(cleanerd->nilfs))
+		params.flags |= NILFS_RECLAIM_PARAM_MIN_RECLAIMABLE_BLKS;
+
+	params.min_reclaimable_blks =
+			nilfs_cleanerd_min_reclaimable_blocks(cleanerd);
 	params.protseq = protseq;
 
 	ret = nilfs_cnoconv_time2cno(cleanerd->cnoconv, prottime,
@@ -1402,7 +1406,7 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
 		if (errno == ENOMEM) {
 			nilfs_cleanerd_reduce_ncleansegs_for_retry(cleanerd);
 			cleanerd->fallback = 1;
-			*ncleaned = 0;
+			*ndone = 0;
 			ret = 0;
 		}
 		goto out;
@@ -1415,6 +1419,17 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
 		nilfs_cleanerd_progress(cleanerd, stat.cleaned_segs);
 		cleanerd->fallback = 0;
 		cleanerd->retry_cleaning = 0;
+
+		*ndone = stat.cleaned_segs;
+	} else if (stat.deferred_segs > 0) {
+		for (i = 0; i < stat.deferred_segs; i++)
+			syslog(LOG_DEBUG, "segment %llu deferred",
+			       (unsigned long long)segnums[i]);
+		nilfs_cleanerd_progress(cleanerd, stat.deferred_segs);
+		cleanerd->fallback = 0;
+		cleanerd->retry_cleaning = 0;
+
+		*ndone = stat.deferred_segs;
 	} else {
 		syslog(LOG_DEBUG, "no segments cleaned");
 
@@ -1428,8 +1443,9 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
 		} else {
 			cleanerd->retry_cleaning = 0;
 		}
+		*ndone = 0;
 	}
-	*ncleaned = stat.cleaned_segs;
+
 out:
 	return ret;
 }
-- 
1.8.5.3

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v5 6/6] nilfs-utils: add a no_timeout flag to enable faster loop

[Andreas Rohner]

This patch adds a flag, that enables the GC to skip the timeout in
certain situations. For example if the cleaning of some segments
was deferred to a later time, then no real progress has been
made. Apart from reading a few summary blocks, no data was read or
written to disk. In this situation it makes sense to skip the
normal timeout once and immediately try to clean the next set of
segments.

Unfortunately it is not possible, to directly continue the cleaning
loop, because this would lead to an unresponsive GC process.
Therefore the timeout is simply set to 0 seconds.

Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
---
 sbin/cleanerd/cleanerd.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
index 4c9f69b..3506c20 100644
--- a/sbin/cleanerd/cleanerd.c
+++ b/sbin/cleanerd/cleanerd.c
@@ -141,6 +141,7 @@ const static struct option long_option[] = {
  * @running: running state
  * @fallback: fallback state
  * @retry_cleaning: retrying reclamation for protected segments
+ * @no_timeout: the next timeout will be 0 seconds
  * @ncleansegs: number of semgents cleaned per cycle
  * @cleaning_interval: cleaning interval
  * @target: target time for sleeping
@@ -169,6 +170,7 @@ struct nilfs_cleanerd {
 	int running;
 	int fallback;
 	int retry_cleaning;
+	int no_timeout;
 	int shutdown;
 	long ncleansegs;
 	struct timeval cleaning_interval;
@@ -896,7 +898,7 @@ static int nilfs_cleanerd_recalc_interval(struct nilfs_cleanerd *cleanerd,
 	interval = nilfs_cleanerd_cleaning_interval(cleanerd);
 	/* timercmp() does not work for '>=' or '<='. */
 	/* curr >= target */
-	if (!timercmp(&curr, &cleanerd->target, <)) {
+	if (!timercmp(&curr, &cleanerd->target, <) || cleanerd->no_timeout) {
 		cleanerd->timeout.tv_sec = 0;
 		cleanerd->timeout.tv_usec = 0;
 		timeradd(&curr, interval, &cleanerd->target);
@@ -1428,6 +1430,7 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
 		nilfs_cleanerd_progress(cleanerd, stat.deferred_segs);
 		cleanerd->fallback = 0;
 		cleanerd->retry_cleaning = 0;
+		cleanerd->no_timeout = 1;
 
 		*ndone = stat.deferred_segs;
 	} else {
@@ -1501,6 +1504,8 @@ static int nilfs_cleanerd_clean_loop(struct nilfs_cleanerd *cleanerd)
 		nilfs_cleanerd_clean_check_pause(cleanerd);
 
 	while (!cleanerd->shutdown) {
+		cleanerd->no_timeout = 0;
+
 		if (sigprocmask(SIG_BLOCK, &sigset, NULL) < 0) {
 			syslog(LOG_ERR, "cannot set signal mask: %m");
 			return -1;
-- 
1.8.5.3

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v5 1/6] nilfs-utils: cldconfig add an option to set min. reclaimable blocks

[Ryusuke Konishi]

On Thu,  6 Feb 2014 15:17:21 +0100, Andreas Rohner wrote:
> With this option the user can specify the minimum number of
> reclaimable blocks of a segment, before it can be cleaned. This is
> a threshold for the GC to prevent needless moving of data. If
> there are less reclaimable blocks in a segment than the specified
> number, the GC will abort and try again with a different segment.
> 
> If there are less clean segments than min_clean_segments,
> mc_min_reclaimable_blocks is used instead of
> min_reclaimable_blocks. This allows for more flexibility in
> configuring the GC.
> 
> The number of blocks can be specified in percent of a segment or
> in bytes.
> 
> If the use_set_suinfo switch is not set, the optimization is
> completely disabled.
> 
> Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
> ---
>  man/nilfs_cleanerd.conf.5         |  22 +++++
>  sbin/cleanerd/cldconfig.c         | 169 ++++++++++++++++++++++++++++----------
>  sbin/cleanerd/cldconfig.h         |  12 +++
>  sbin/cleanerd/cleanerd.c          |  10 +++
>  sbin/cleanerd/nilfs_cleanerd.conf |  20 +++++
>  5 files changed, 191 insertions(+), 42 deletions(-)
> 
> diff --git a/man/nilfs_cleanerd.conf.5 b/man/nilfs_cleanerd.conf.5
> index 8f3fcd2..4315ecd 100644
> --- a/man/nilfs_cleanerd.conf.5
> +++ b/man/nilfs_cleanerd.conf.5
> @@ -79,6 +79,28 @@ Specify whether to use \fBmmap\fP(2) for reading segments.  At
>  present, this option is enabled if supported regardless of this
>  directive.
>  .TP
> +.B use_set_suinfo
> +Specify whether to use the set_suinfo ioctl if it is supported. This is
> +necessary for the \fBmin_reclaimable_blocks\fP feature. By disabling this
> +switch \fBmin_reclaimable_blocks\fP is also disabled.
> +.TP
> +.B min_reclaimable_blocks
> +Specify the minimum number of reclaimable blocks in a segment before
> +it can be cleaned.
> +.TP
> +.B mc_min_reclaimable_blocks
> +Specify the minimum number of reclaimable blocks in a segment before
> +it can be cleaned. if clean segments < min_clean_segments.
> +.PP
> +\fBmin_reclaimable_blocks\fP and \fBmc_min_reclaimable_blocks\fP may
> +be followed by a percent sign or the following multiplicative suffixes:
> +kB 1000, K 1024, MB 1000*1000, M 1024*1024, GB 1000*1000*1000, G
> +1024*1024*1024, and so on for T, P, E.  If the argument is followed by
> +a percent sign, it represents the ratio of blocks in a segment.
> +.PP
> +The default values of \fBmin_reclaimable_blocks\fP and
> +\fBmc_min_reclaimable_blocks\fP are 5 percent and 1 percent respectively.
> +.TP
>  .B log_priority
>  Gives the verbosity level that is used when logging messages from
>  \fBnilfs_cleanerd\fP(8).  The possible values are: \fBemerg\fP,
> diff --git a/sbin/cleanerd/cldconfig.c b/sbin/cleanerd/cldconfig.c
> index 270d360..bdd8c85 100644
> --- a/sbin/cleanerd/cldconfig.c
> +++ b/sbin/cleanerd/cldconfig.c
> @@ -278,6 +278,55 @@ nilfs_cldconfig_handle_protection_period(struct nilfs_cldconfig *config,
>  }
>  
>  static unsigned long long
> +nilfs_convert_units_to_bytes(const struct nilfs_param *param)
> +{
> +	unsigned long long bytes = param->num;
> +
> +	switch (param->unit) {
> +	case NILFS_SIZE_UNIT_KB:
> +		bytes *= 1000ULL;
> +		break;
> +	case NILFS_SIZE_UNIT_KIB:
> +		bytes <<= 10;
> +		break;
> +	case NILFS_SIZE_UNIT_MB:
> +		bytes *= 1000000ULL;
> +		break;
> +	case NILFS_SIZE_UNIT_MIB:
> +		bytes <<= 20;
> +		break;
> +	case NILFS_SIZE_UNIT_GB:
> +		bytes *= 1000000000ULL;
> +		break;
> +	case NILFS_SIZE_UNIT_GIB:
> +		bytes <<= 30;
> +		break;
> +	case NILFS_SIZE_UNIT_TB:
> +		bytes *= 1000000000000ULL;
> +		break;
> +	case NILFS_SIZE_UNIT_TIB:
> +		bytes <<= 40;
> +		break;
> +	case NILFS_SIZE_UNIT_PB:
> +		bytes *= 1000000000000000ULL;
> +		break;
> +	case NILFS_SIZE_UNIT_PIB:
> +		bytes <<= 50;
> +		break;
> +	case NILFS_SIZE_UNIT_EB:
> +		bytes *= 1000000000000000000ULL;
> +		break;
> +	case NILFS_SIZE_UNIT_EIB:
> +		bytes <<= 60;
> +		break;
> +	default:
> +		assert(0);
> +	}
> +
> +	return bytes;
> +}
> +
> +static unsigned long long
>  nilfs_convert_size_to_nsegments(struct nilfs *nilfs, struct nilfs_param *param)
>  {
>  	unsigned long long ret, segment_size, bytes;
> @@ -287,48 +336,7 @@ nilfs_convert_size_to_nsegments(struct nilfs *nilfs, struct nilfs_param *param)
>  	} else if (param->unit == NILFS_SIZE_UNIT_PERCENT) {
>  		ret = (nilfs_get_nsegments(nilfs) * param->num + 99) / 100;
>  	} else {
> -		bytes = param->num;
> -
> -		switch (param->unit) {
> -		case NILFS_SIZE_UNIT_KB:
> -			bytes *= 1000ULL;
> -			break;
> -		case NILFS_SIZE_UNIT_KIB:
> -			bytes <<= 10;
> -			break;
> -		case NILFS_SIZE_UNIT_MB:
> -			bytes *= 1000000ULL;
> -			break;
> -		case NILFS_SIZE_UNIT_MIB:
> -			bytes <<= 20;
> -			break;
> -		case NILFS_SIZE_UNIT_GB:
> -			bytes *= 1000000000ULL;
> -			break;
> -		case NILFS_SIZE_UNIT_GIB:
> -			bytes <<= 30;
> -			break;
> -		case NILFS_SIZE_UNIT_TB:
> -			bytes *= 1000000000000ULL;
> -			break;
> -		case NILFS_SIZE_UNIT_TIB:
> -			bytes <<= 40;
> -			break;
> -		case NILFS_SIZE_UNIT_PB:
> -			bytes *= 1000000000000000ULL;
> -			break;
> -		case NILFS_SIZE_UNIT_PIB:
> -			bytes <<= 50;
> -			break;
> -		case NILFS_SIZE_UNIT_EB:
> -			bytes *= 1000000000000000000ULL;
> -			break;
> -		case NILFS_SIZE_UNIT_EIB:
> -			bytes <<= 60;
> -			break;
> -		default:
> -			assert(0);
> -		}
> +		bytes = nilfs_convert_units_to_bytes(param);
>  		segment_size = nilfs_get_block_size(nilfs) *
>  			nilfs_get_blocks_per_segment(nilfs);
>  		ret = (bytes + segment_size - 1) / segment_size;
> @@ -455,6 +463,52 @@ nilfs_cldconfig_handle_mc_nsegments_per_clean(struct nilfs_cldconfig *config,
>  	return 0;
>  }
>  
> +static unsigned long long
> +nilfs_convert_size_to_blocks_per_segment(struct nilfs *nilfs,
> +					 struct nilfs_param *param)
> +{
> +	unsigned long long ret, segment_size, block_size, bytes;
> +
> +	if (param->unit == NILFS_SIZE_UNIT_NONE) {
> +		ret = param->num;
> +	} else if (param->unit == NILFS_SIZE_UNIT_PERCENT) {
> +		ret = (nilfs_get_blocks_per_segment(nilfs) * param->num) / 100;
> +	} else {
> +		block_size = nilfs_get_block_size(nilfs);
> +		segment_size = block_size *
> +				nilfs_get_blocks_per_segment(nilfs);
> +		bytes = nilfs_convert_units_to_bytes(param) % segment_size;
> +		ret = (bytes + block_size - 1) / block_size;
> +	}
> +	return ret;
> +}
> +
> +static int
> +nilfs_cldconfig_handle_min_reclaimable_blocks(struct nilfs_cldconfig *config,
> +					      char **tokens, size_t ntoks,
> +					      struct nilfs *nilfs)
> +{
> +	struct nilfs_param param;
> +
> +	if (nilfs_cldconfig_get_size_argument(tokens, ntoks, &param) == 0)
> +		config->cf_min_reclaimable_blocks =
> +			nilfs_convert_size_to_blocks_per_segment(nilfs, &param);
> +	return 0;
> +}
> +
> +static int
> +nilfs_cldconfig_handle_mc_min_reclaimable_blocks(struct nilfs_cldconfig *config,
> +						 char **tokens, size_t ntoks,
> +						 struct nilfs *nilfs)
> +{
> +	struct nilfs_param param;
> +
> +	if (nilfs_cldconfig_get_size_argument(tokens, ntoks, &param) == 0)
> +		config->cf_mc_min_reclaimable_blocks =
> +			nilfs_convert_size_to_blocks_per_segment(nilfs, &param);
> +	return 0;
> +}
> +
>  static int
>  nilfs_cldconfig_handle_cleaning_interval(struct nilfs_cldconfig *config,
>  					 char **tokens, size_t ntoks,
> @@ -490,6 +544,14 @@ static int nilfs_cldconfig_handle_use_mmap(struct nilfs_cldconfig *config,
>  	return 0;
>  }
>  
> +static int nilfs_cldconfig_handle_use_set_suinfo(struct nilfs_cldconfig *config,
> +						 char **tokens, size_t ntoks,
> +						 struct nilfs *nilfs)
> +{
> +	config->cf_use_set_suinfo = 1;
> +	return 0;
> +}
> +
>  static const struct nilfs_cldconfig_log_priority
>  nilfs_cldconfig_log_priority_table[] = {
>  	{"emerg",	LOG_EMERG},
> @@ -576,6 +638,18 @@ nilfs_cldconfig_keyword_table[] = {
>  		"log_priority", 2, 2,
>  		nilfs_cldconfig_handle_log_priority
>  	},
> +	{
> +		"min_reclaimable_blocks", 2, 2,
> +		nilfs_cldconfig_handle_min_reclaimable_blocks
> +	},
> +	{
> +		"mc_min_reclaimable_blocks", 2, 2,
> +		nilfs_cldconfig_handle_mc_min_reclaimable_blocks
> +	},
> +	{
> +		"use_set_suinfo", 1, 1,
> +		nilfs_cldconfig_handle_use_set_suinfo
> +	},
>  };
>  
>  #define NILFS_CLDCONFIG_NKEYWORDS			\
> @@ -640,7 +714,18 @@ static void nilfs_cldconfig_set_default(struct nilfs_cldconfig *config,
>  	config->cf_retry_interval.tv_sec = NILFS_CLDCONFIG_RETRY_INTERVAL;
>  	config->cf_retry_interval.tv_usec = 0;
>  	config->cf_use_mmap = NILFS_CLDCONFIG_USE_MMAP;
> +	config->cf_use_set_suinfo = NILFS_CLDCONFIG_USE_SET_SUINFO;
>  	config->cf_log_priority = NILFS_CLDCONFIG_LOG_PRIORITY;
> +
> +	param.num = NILFS_CLDCONFIG_MIN_RECLAIMABLE_BLOCKS;
> +	param.unit = NILFS_CLDCONFIG_MIN_RECLAIMABLE_BLOCKS_UNIT;
> +	config->cf_min_reclaimable_blocks =
> +		nilfs_convert_size_to_blocks_per_segment(nilfs, &param);
> +
> +	param.num = NILFS_CLDCONFIG_MC_MIN_RECLAIMABLE_BLOCKS;
> +	param.unit = NILFS_CLDCONFIG_MC_MIN_RECLAIMABLE_BLOCKS_UNIT;
> +	config->cf_mc_min_reclaimable_blocks =
> +		nilfs_convert_size_to_blocks_per_segment(nilfs, &param);
>  }
>  
>  static inline int iseol(int c)
> diff --git a/sbin/cleanerd/cldconfig.h b/sbin/cleanerd/cldconfig.h
> index 188ce9b..0a598d5 100644
> --- a/sbin/cleanerd/cldconfig.h
> +++ b/sbin/cleanerd/cldconfig.h
> @@ -87,7 +87,11 @@ enum nilfs_size_unit {
>   * if clean segments < min_clean_segments
>   * @cf_retry_interval: retry interval
>   * @cf_use_mmap: flag that indicate using mmap
> + * @cf_use_set_suinfo: flag that indicates the use of the set_suinfo ioctl
>   * @cf_log_priority: log priority level
> + * @cf_min_reclaimable_blocks: minimum reclaimable blocks for cleaning
> + * @cf_mc_min_reclaimable_blocks: minimum reclaimable blocks for cleaning
> + * if clean segments < min_clean_segments
>   */
>  struct nilfs_cldconfig {
>  	struct nilfs_selection_policy cf_selection_policy;
> @@ -101,7 +105,10 @@ struct nilfs_cldconfig {
>  	struct timeval cf_mc_cleaning_interval;
>  	struct timeval cf_retry_interval;
>  	int cf_use_mmap;
> +	int cf_use_set_suinfo;
>  	int cf_log_priority;
> +	unsigned long cf_min_reclaimable_blocks;
> +	unsigned long cf_mc_min_reclaimable_blocks;
>  };
>  
>  #define NILFS_CLDCONFIG_SELECTION_POLICY_IMPORTANCE	\
> @@ -119,7 +126,12 @@ struct nilfs_cldconfig {
>  #define NILFS_CLDCONFIG_MC_CLEANING_INTERVAL		1
>  #define NILFS_CLDCONFIG_RETRY_INTERVAL			60
>  #define NILFS_CLDCONFIG_USE_MMAP			1
> +#define NILFS_CLDCONFIG_USE_SET_SUINFO			0
>  #define NILFS_CLDCONFIG_LOG_PRIORITY			LOG_INFO
> +#define NILFS_CLDCONFIG_MIN_RECLAIMABLE_BLOCKS		5
> +#define NILFS_CLDCONFIG_MIN_RECLAIMABLE_BLOCKS_UNIT	NILFS_SIZE_UNIT_PERCENT
> +#define NILFS_CLDCONFIG_MC_MIN_RECLAIMABLE_BLOCKS	1
> +#define NILFS_CLDCONFIG_MC_MIN_RECLAIMABLE_BLOCKS_UNIT	NILFS_SIZE_UNIT_PERCENT
>  
>  #define NILFS_CLDCONFIG_NSEGMENTS_PER_CLEAN_MAX	32
>  
> diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
> index 1d09b5b..c58a542 100644
> --- a/sbin/cleanerd/cleanerd.c
> +++ b/sbin/cleanerd/cleanerd.c
> @@ -145,6 +145,7 @@ const static struct option long_option[] = {
>   * @cleaning_interval: cleaning interval
>   * @target: target time for sleeping
>   * @timeout: timeout value for sleeping
> + * @min_reclaimable_blocks: min. number of reclaimable blocks
>   * @prev_nongc_ctime: previous nongc ctime
>   * @recvq: receive queue
>   * @recvq_name: receive queue name
> @@ -172,6 +173,7 @@ struct nilfs_cleanerd {
>  	struct timeval cleaning_interval;
>  	struct timeval target;
>  	struct timeval timeout;
> +	unsigned long min_reclaimable_blocks;
>  	__u64 prev_nongc_ctime;
>  	mqd_t recvq;
>  	char *recvq_name;
> @@ -273,6 +275,8 @@ static int nilfs_cleanerd_reconfig(struct nilfs_cleanerd *cleanerd,
>  	} else {
>  		cleanerd->ncleansegs = config->cf_nsegments_per_clean;
>  		cleanerd->cleaning_interval = config->cf_cleaning_interval;
> +		cleanerd->min_reclaimable_blocks =
> +				config->cf_min_reclaimable_blocks;
>  		syslog(LOG_INFO, "configuration file reloaded");
>  	}
>  	return ret;
> @@ -1238,10 +1242,14 @@ static int nilfs_cleanerd_handle_clean_check(struct nilfs_cleanerd *cleanerd,
>  		/* disk space is close to limit -- accelerate cleaning */
>  		cleanerd->ncleansegs = config->cf_mc_nsegments_per_clean;
>  		cleanerd->cleaning_interval = config->cf_mc_cleaning_interval;
> +		cleanerd->min_reclaimable_blocks =
> +				config->cf_mc_min_reclaimable_blocks;
>  	} else {
>  		/* continue to run */
>  		cleanerd->ncleansegs = config->cf_nsegments_per_clean;
>  		cleanerd->cleaning_interval = config->cf_cleaning_interval;
> +		cleanerd->min_reclaimable_blocks =
> +				config->cf_min_reclaimable_blocks;
>  	}
>  
>  	return 0; /* do gc */
> @@ -1442,6 +1450,8 @@ static int nilfs_cleanerd_clean_loop(struct nilfs_cleanerd *cleanerd)
>  
>  	cleanerd->ncleansegs = cleanerd->config.cf_nsegments_per_clean;
>  	cleanerd->cleaning_interval = cleanerd->config.cf_cleaning_interval;
> +	cleanerd->min_reclaimable_blocks =
> +			cleanerd->config.cf_min_reclaimable_blocks;
>  
>  
>  	if (nilfs_cleanerd_automatic_suspend(cleanerd))
> diff --git a/sbin/cleanerd/nilfs_cleanerd.conf b/sbin/cleanerd/nilfs_cleanerd.conf
> index 26872aa..79d3fce 100644
> --- a/sbin/cleanerd/nilfs_cleanerd.conf
> +++ b/sbin/cleanerd/nilfs_cleanerd.conf
> @@ -51,6 +51,26 @@ mc_cleaning_interval	1
>  # Retry interval in seconds.
>  retry_interval		60
>  
> +# Specify the minimum number of reclaimable blocks in a segment
> +# before it can be cleaned.
> +min_reclaimable_blocks	5%
> +
> +# Specify the minimum number of reclaimable blocks in a segment
> +# before it can be cleaned.
> +# if clean segments < min_clean_segments
> +mc_min_reclaimable_blocks	1%
> +
> +# The argument of min_reclaimable_blocks and mc_min_reclaimable_blocks
> +# can be followed by a percent sign (%) or one of the following
> +# multiplicative suffixes similar to min_clean_segments.
> +#
> +# If the argument is followed by "%", it represents a ratio for the
> +# number of blocks per segment.
> +
> +# enable set_suinfo ioctl if supported
> +# (needed for min_reclaimable_blocks)
> +#use_set_suinfo
> +
>  # Use mmap when reading segments if supported.
>  use_mmap
>  
> -- 
> 1.8.5.3

Looks good to me.

Regards,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v5 2/6] nilfs-utils: add NILFS_OPT_SET_SUINFO

[Ryusuke Konishi]

On Thu,  6 Feb 2014 15:17:22 +0100, Andreas Rohner wrote:
> This option for the nilfs object corresponds to the use_set_suinfo
> flag in the configuration file. This patch adds the flag itself
> plus some convenience functions.
> 
> The option is set after the configuration file was read in.
> 
> Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
> ---
>  include/nilfs.h          |  7 ++++++-
>  lib/nilfs.c              | 28 ++++++++++++++++++++++++++++
>  sbin/cleanerd/cleanerd.c |  6 ++++++
>  3 files changed, 40 insertions(+), 1 deletion(-)
> 
> diff --git a/include/nilfs.h b/include/nilfs.h
> index 56286a9..b5f85d3 100644
> --- a/include/nilfs.h
> +++ b/include/nilfs.h
> @@ -128,7 +128,8 @@ struct nilfs {
>  #define NILFS_OPEN_RDWR		0x0008	/* Open NILFS API in read/write mode */
>  #define NILFS_OPEN_GCLK		0x1000	/* Open GC lock primitive */
>  
> -#define NILFS_OPT_MMAP	0x01
> +#define NILFS_OPT_MMAP		0x01
> +#define NILFS_OPT_SET_SUINFO	0x02
>  
>  
>  struct nilfs *nilfs_open(const char *, const char *, int);
> @@ -140,6 +141,10 @@ void nilfs_opt_clear_mmap(struct nilfs *);
>  int nilfs_opt_set_mmap(struct nilfs *);
>  int nilfs_opt_test_mmap(struct nilfs *);
>  
> +void nilfs_opt_clear_set_suinfo(struct nilfs *);
> +int nilfs_opt_set_set_suinfo(struct nilfs *);
> +int nilfs_opt_test_set_suinfo(struct nilfs *);
> +
>  nilfs_cno_t nilfs_get_oldest_cno(struct nilfs *);
>  
>  struct nilfs_super_block *nilfs_get_sb(struct nilfs *);
> diff --git a/lib/nilfs.c b/lib/nilfs.c
> index c1771e8..93822de 100644
> --- a/lib/nilfs.c
> +++ b/lib/nilfs.c
> @@ -288,6 +288,34 @@ int nilfs_opt_test_mmap(struct nilfs *nilfs)
>  	return !!(nilfs->n_opts & NILFS_OPT_MMAP);
>  }
>  
> +/**
> + * nilfs_opt_set_set_suinfo - set set_suinfo option
> + * @nilfs: nilfs object
> + */
> +int nilfs_opt_set_set_suinfo(struct nilfs *nilfs)
> +{
> +	nilfs->n_opts |= NILFS_OPT_SET_SUINFO;
> +	return 0;
> +}
> +
> +/**
> + * nilfs_opt_clear_set_suinfo - clear set_suinfo option
> + * @nilfs: nilfs object
> + */
> +void nilfs_opt_clear_set_suinfo(struct nilfs *nilfs)
> +{
> +	nilfs->n_opts &= ~NILFS_OPT_SET_SUINFO;
> +}
> +
> +/**
> + * nilfs_opt_test_set_suinfo - test whether set_suinfo option is set or not
> + * @nilfs: nilfs object
> + */
> +int nilfs_opt_test_set_suinfo(struct nilfs *nilfs)
> +{
> +	return !!(nilfs->n_opts & NILFS_OPT_SET_SUINFO);
> +}
> +
>  static int nilfs_open_sem(struct nilfs *nilfs)
>  {
>  	char semnambuf[NAME_MAX - 4];
> diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
> index c58a542..4624f92 100644
> --- a/sbin/cleanerd/cleanerd.c
> +++ b/sbin/cleanerd/cleanerd.c
> @@ -248,6 +248,12 @@ static int nilfs_cleanerd_config(struct nilfs_cleanerd *cleanerd,
>  	else
>  		nilfs_opt_clear_mmap(cleanerd->nilfs);
>  #endif	/* HAVE_MMAP */
> +
> +	if (cleanerd->config.cf_use_set_suinfo)
> +		nilfs_opt_set_set_suinfo(cleanerd->nilfs);
> +	else
> +		nilfs_opt_clear_set_suinfo(cleanerd->nilfs);
> +
>  	nilfs_cleanerd_set_log_priority(cleanerd);
>  
>  	if (protection_period != ULONG_MAX) {
> -- 
> 1.8.5.3
> 

Looks good to me.

Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v5 3/6] nilfs-utils: nilfs-clean add cmdline param min-reclaimable-blocks

[Ryusuke Konishi]

On Thu,  6 Feb 2014 15:17:23 +0100, Andreas Rohner wrote:
> This patch adds a command line parameter for nilfs-clean to allow
> the manual override of the min_reclaimable_blocks option.
> 
> Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
> ---
>  include/nilfs_cleaner.h        | 19 +++++++++---------
>  man/nilfs-clean.8              |  4 ++++
>  sbin/cleanerd/cleanerd.c       | 21 ++++++++++++++++++++
>  sbin/nilfs-clean/nilfs-clean.c | 44 ++++++++++++++++++++++++++++++++++++------
>  4 files changed, 73 insertions(+), 15 deletions(-)
> 
> diff --git a/include/nilfs_cleaner.h b/include/nilfs_cleaner.h
> index 0bf02aa..874c17a 100644
> --- a/include/nilfs_cleaner.h
> +++ b/include/nilfs_cleaner.h
> @@ -46,17 +46,18 @@ struct nilfs_cleaner_args {
>  	uint64_t start_segnum;	/* start segment number */
>  	uint64_t nsegs;		/* number of segments */
>  	uint32_t runtime; /* runtime in seconds */
> -	uint32_t pad2;
> +	uint32_t min_reclaimable_blocks;
>  };
>  /* valid flags */
> -#define NILFS_CLEANER_ARG_PROTECTION_PERIOD	(1 << 0)
> -#define NILFS_CLEANER_ARG_NSEGMENTS_PER_CLEAN	(1 << 1)
> -#define NILFS_CLEANER_ARG_CLEANING_INTERVAL	(1 << 2)
> -#define NILFS_CLEANER_ARG_USAGE_RATE_THRESHOLD	(1 << 3) /* reserved */
> -#define NILFS_CLEANER_ARG_START_SEGNUM		(1 << 4) /* reserved */
> -#define NILFS_CLEANER_ARG_NSEGS			(1 << 5) /* reserved */
> -#define NILFS_CLEANER_ARG_NPASSES		(1 << 6) /* reserved */
> -#define NILFS_CLEANER_ARG_RUNTIME		(1 << 7) /* reserved */
> +#define NILFS_CLEANER_ARG_PROTECTION_PERIOD		(1 << 0)
> +#define NILFS_CLEANER_ARG_NSEGMENTS_PER_CLEAN		(1 << 1)
> +#define NILFS_CLEANER_ARG_CLEANING_INTERVAL		(1 << 2)
> +#define NILFS_CLEANER_ARG_USAGE_RATE_THRESHOLD		(1 << 3) /* reserved */
> +#define NILFS_CLEANER_ARG_START_SEGNUM			(1 << 4) /* reserved */
> +#define NILFS_CLEANER_ARG_NSEGS				(1 << 5) /* reserved */
> +#define NILFS_CLEANER_ARG_NPASSES			(1 << 6) /* reserved */
> +#define NILFS_CLEANER_ARG_RUNTIME			(1 << 7) /* reserved */
> +#define NILFS_CLEANER_ARG_MIN_RECLAIMABLE_BLOCKS	(1 << 8)
>  
>  enum {
>  	NILFS_CLEANER_STATUS_IDLE,
> diff --git a/man/nilfs-clean.8 b/man/nilfs-clean.8
> index 04e11c6..94267e1 100644
> --- a/man/nilfs-clean.8
> +++ b/man/nilfs-clean.8
> @@ -43,6 +43,10 @@ Display cleaner status.
>  \fB\-h\fR, \fB\-\-help\fR
>  Display help message and exit.
>  .TP
> +\fB\-m\fR, \fB\-\-min\-reclaimable\-blocks=\fICOUNT\fR
> +Specify the minimum number of reclaimable blocks in a segment before
> +it can be cleaned.
> +.TP
>  \fB\-p\fR, \fB\-\-protection-period=\fIinterval\fR
>  Set protection period for a cleaner run.  The \fIinterval\fR parameter
>  is an integer value and specifies the minimum time that deleted or
> diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
> index 4624f92..2896af8 100644
> --- a/sbin/cleanerd/cleanerd.c
> +++ b/sbin/cleanerd/cleanerd.c
> @@ -159,6 +159,7 @@ const static struct option long_option[] = {
>   * @mm_ncleansegs: number of segmetns cleaned per cycle (manual mode)
>   * @mm_protection_period: protection period (manual mode)
>   * @mm_cleaning_interval: cleaning interval (manual mode)
> + * @mm_min_reclaimable_blocks: min. number of reclaimable blocks (manual mode)
>   */
>  struct nilfs_cleanerd {
>  	struct nilfs *nilfs;
> @@ -186,6 +187,7 @@ struct nilfs_cleanerd {
>  	long mm_ncleansegs;
>  	struct timeval mm_protection_period;
>  	struct timeval mm_cleaning_interval;
> +	unsigned long mm_min_reclaimable_blocks;
>  };
>  
>  /**
> @@ -469,6 +471,14 @@ nilfs_cleanerd_protection_period(struct nilfs_cleanerd *cleanerd)
>  		&cleanerd->config.cf_protection_period;
>  }
>  
> +static unsigned long
> +nilfs_cleanerd_min_reclaimable_blocks(struct nilfs_cleanerd *cleanerd)
> +{
> +	return cleanerd->running == 2 ?
> +		cleanerd->mm_min_reclaimable_blocks :
> +		cleanerd->min_reclaimable_blocks;
> +}
> +
>  static void
>  nilfs_cleanerd_reduce_ncleansegs_for_retry(struct nilfs_cleanerd *cleanerd)
>  {
> @@ -1011,6 +1021,17 @@ static int nilfs_cleanerd_cmd_run(struct nilfs_cleanerd *cleanerd,
>  		cleanerd->mm_cleaning_interval =
>  			cleanerd->cleaning_interval;
>  	}
> +	/* minimal reclaimable blocks */
> +	if (req2->args.valid & NILFS_CLEANER_ARG_MIN_RECLAIMABLE_BLOCKS) {
> +		if (req2->args.min_reclaimable_blocks >
> +		    nilfs_get_blocks_per_segment(cleanerd->nilfs))
> +			goto error_inval;
> +		cleanerd->mm_min_reclaimable_blocks =
> +			req2->args.min_reclaimable_blocks;
> +	} else {
> +		cleanerd->mm_min_reclaimable_blocks =
> +			cleanerd->min_reclaimable_blocks;
> +	}
>  	/* number of passes */
>  	if (req2->args.valid & NILFS_CLEANER_ARG_NPASSES) {
>  		if (!req2->args.npasses)
> diff --git a/sbin/nilfs-clean/nilfs-clean.c b/sbin/nilfs-clean/nilfs-clean.c
> index 55abd55..f77fdf7 100644
> --- a/sbin/nilfs-clean/nilfs-clean.c
> +++ b/sbin/nilfs-clean/nilfs-clean.c
> @@ -77,6 +77,7 @@ const static struct option long_option[] = {
>  	{"stop", no_argument, NULL, 'b'},
>  	{"suspend", no_argument, NULL, 's'},
>  	{"speed", required_argument, NULL, 'S'},
> +	{"min-reclaimable-blocks", required_argument, NULL, 'm'},
>  	{"verbose", no_argument, NULL, 'v'},
>  	{"version", no_argument, NULL, 'V'},
>  	{NULL, 0, NULL, 0}
> @@ -90,6 +91,9 @@ const static struct option long_option[] = {
>  	"  -l, --status\t\tdisplay cleaner status\n"			\
>  	"  -p, --protection-period=SECONDS\n"				\
>  	"               \t\tspecify protection period\n"		\
> +	"  -m, --min-reclaimable-blocks=COUNT\n"			\
> +	"               \t\tset minimum number of reclaimable blocks\n"	\
> +	"               \t\tbefore a segment can be cleaned\n"		\
>  	"  -q, --quit\t\tshutdown cleaner\n"				\
>  	"  -r, --resume\t\tresume cleaner\n"				\
>  	"  -s, --suspend\t\tsuspend cleaner\n"				\
> @@ -98,9 +102,10 @@ const static struct option long_option[] = {
>  	"  -v, --verbose\t\tverbose mode\n"				\
>  	"  -V, --version\t\tdisplay version and exit\n"
>  #else
> -#define NILFS_CLEAN_USAGE						\
> -	"Usage: %s [-b] [-c [conffile]] [-h] [-l] [-p protection-period]" \
> -	"          [-q] [-r] [-s] [-S gc-speed] [-v] [-V] [device]\n"
> +#define NILFS_CLEAN_USAGE						  \
> +	"Usage: %s [-b] [-c [conffile]] [-h] [-l] [-m blocks]\n"	  \
> +	"          [-p protection-period] [-q] [-r] [-s] [-S gc-speed]\n" \
> +	"          [-v] [-V] [device]\n"
>  #endif	/* _GNU_SOURCE */
>  
>  
> @@ -124,6 +129,7 @@ static const char *conffile = NULL;
>  static unsigned long protection_period = ULONG_MAX;
>  static int nsegments_per_clean = 2;
>  static struct timespec cleaning_interval = { 0, 100000000 };   /* 100 msec */
> +static unsigned long min_reclaimable_blocks = 20;  /* about 1% with 8MB segs */
>  
>  static sigjmp_buf nilfs_clean_env;
>  static struct nilfs_cleaner *nilfs_cleaner;
> @@ -164,9 +170,11 @@ static int nilfs_clean_do_run(struct nilfs_cleaner *cleaner)
>  	args.nsegments_per_clean = nsegments_per_clean;
>  	args.cleaning_interval = cleaning_interval.tv_sec;
>  	args.cleaning_interval_nsec = cleaning_interval.tv_nsec;
> +	args.min_reclaimable_blocks = min_reclaimable_blocks;
>  	args.valid = (NILFS_CLEANER_ARG_NPASSES |
>  		      NILFS_CLEANER_ARG_CLEANING_INTERVAL |
> -		      NILFS_CLEANER_ARG_NSEGMENTS_PER_CLEAN);
> +		      NILFS_CLEANER_ARG_NSEGMENTS_PER_CLEAN |
> +		      NILFS_CLEANER_ARG_MIN_RECLAIMABLE_BLOCKS);
>  
>  	if (protection_period != ULONG_MAX) {
>  		args.protection_period = protection_period;
> @@ -426,6 +434,26 @@ failed_too_large:
>  	return -1;
>  }
>  
> +static int nilfs_clean_parse_min_reclaimable(const char *arg)
> +{
> +	unsigned long blocks;
> +	char *endptr;
> +
> +	blocks = strtoul(arg, &endptr, 10);
> +	if (endptr == arg || endptr[0] != '\0') {
> +		myprintf(_("Error: invalid reclaimable blocks: %s\n"), arg);
> +		return -1;
> +	}
> +
> +	if (blocks == ULONG_MAX) {
> +		myprintf(_("Error: value too large: %s\n"), arg);
> +		return -1;
> +	}
> +
> +	min_reclaimable_blocks = blocks;
> +	return 0;
> +}
> +
>  static void nilfs_clean_parse_options(int argc, char *argv[])
>  {
>  #ifdef _GNU_SOURCE
> @@ -434,10 +462,10 @@ static void nilfs_clean_parse_options(int argc, char *argv[])
>  	int c;
>  
>  #ifdef _GNU_SOURCE
> -	while ((c = getopt_long(argc, argv, "bc::hlp:qrsS:vV",
> +	while ((c = getopt_long(argc, argv, "bc::hlm:p:qrsS:vV",
>  				long_option, &option_index)) >= 0) {
>  #else
> -	while ((c = getopt(argc, argv, "bc::hlp:qrsS:vV")) >= 0) {
> +	while ((c = getopt(argc, argv, "bc::hlm:p:qrsS:vV")) >= 0) {
>  #endif	/* _GNU_SOURCE */
>  		switch (c) {
>  		case 'b':
> @@ -455,6 +483,10 @@ static void nilfs_clean_parse_options(int argc, char *argv[])
>  		case 'l':
>  			clean_cmd = NILFS_CLEAN_CMD_INFO;
>  			break;
> +		case 'm':
> +			if (nilfs_clean_parse_min_reclaimable(optarg) < 0)
> +				exit(EXIT_FAILURE);
> +			break;
>  		case 'p':
>  			if (nilfs_clean_parse_protection_period(optarg) < 0)
>  				exit(EXIT_FAILURE);
> -- 
> 1.8.5.3

Uum, I think users usually prefer to specify a ratio of reclaimable
blocks instead of the count of blocks.  And, the default value of
min_reclaimable_blocks should be defined as a ratio.

How about adding a new field in nilfs_cleanerd_args struct to allow
cleanerd to interpret the field of min_reclaimable_blocks as a ratio
or other units ?

 struct nilfs_cleaner_args {
 	uint16_t valid;
 	uint16_t npasses; /* number of passes */
 	uint16_t usage_rate_threshold;
 	uint16_t nsegments_per_clean;
- 	uint16_t pad1;
+	uint8_t pad;
+	uint8_t min_reclaimable_blocks_unit;
 	uint16_t cleaning_interval;
 	uint32_t cleaning_interval_nsec;
 	uint64_t protection_period; /* protection period in seconds */
 	uint64_t start_segnum;	/* start segment number */
 	uint64_t nsegs;		/* number of segments */
 	uint32_t runtime; /* runtime in seconds */
 	uint32_t min_reclaimable_blocks;
 };
 
+enum nilfs_cleaner_args_unit {
+	NILFS_CLEANER_ARG_UNIT_NONE = 0,
+	NILFS_CLEANER_ARG_UNIT_PERCENT,
+	NILFS_CLEANER_ARG_UNIT_KB,     /* kilo-byte (kB) */
+	NILFS_CLEANER_ARG_UNIT_KIB,    /* kibi-byte (KiB) */
+	NILFS_CLEANER_ARG_UNIT_MB,     /* mega-byte (MB) */
+	NILFS_CLEANER_ARG_UNIT_MIB,    /* mebi-byte (MiB) */
+	NILFS_CLEANER_ARG_UNIT_GB,     /* giga-byte (GB) */
+	NILFS_CLEANER_ARG_UNIT_GIB,    /* gibi-byte (GiB) */
+	NILFS_CLEANER_ARG_UNIT_TB,     /* tera-byte (TB) */
+	NILFS_CLEANER_ARG_UNIT_TIB,    /* tebi-byte (TiB) */
+	NILFS_CLEANER_ARG_UNIT_PB,     /* peta-byte (PB) */
+	NILFS_CLEANER_ARG_UNIT_PIB,    /* pebi-byte (PiB) */
+	NILFS_CLEANER_ARG_UNIT_EB,     /* exa-byte (EB) */
+	NILFS_CLEANER_ARG_UNIT_EIB,    /* exbi-byte (EiB) */
+
+	NILFS_CLEANER_ARG_MIN_BINARY_SUFFIX = NILFS_CLEANER_ARG_UNIT_KB,
+	NILFS_CLEANER_ARG_MAX_BINARY_SUFFIX = NILFS_CLEANER_ARG_UNIT_EIB,
+};


Then, we can implement -m option so that it specifies a ratio if
it is terminated with "%" symbol.

Please consider adding a patch for that.  I am thinking of applying
this series this time regardless of this comment unless there is an
unignorable problem.

Thanks,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v5 5/6] nilfs-utils: add optimized version of nilfs_xreclaim_segments

[Ryusuke Konishi]

On Thu,  6 Feb 2014 15:17:25 +0100, Andreas Rohner wrote:
> This patch adds an additional parameter min_reclaimable_blks to
> the nilfs_reclaim_params structure. This parameter specifies the
> minimum number of reclaimable blocks in a segment, before it can be
> cleaned. If a segment is below this threshold, it is considered to
> be not worth cleaning, because all the live blocks would need to be
> moved to a new segment, which is expensive, and the number of
> reclaimable blocks is too low. But it is still necessary to update
> the segment usage information to turn the old segment into a new
> one.
> 
> This is basically a shortcut to cleaning the segment. It is still
> necessary to read the segment summary information, but the writing
> of the live blocks can be skipped if it's not worth it.
> 
> This optimization can be enabled and disabled by the
> use_set_suinfo switch in the configuration file. If the kernel
> doesn't support the set_suinfo ioctl it returns the error
> number ENOTTY, which also permanently disables the optimization.
> 
> Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
> ---
>  include/nilfs_gc.h       | 11 ++++----
>  lib/gc.c                 | 73 ++++++++++++++++++++++++++++++++++++++++++++----
>  sbin/cleanerd/cleanerd.c | 24 +++++++++++++---
>  3 files changed, 93 insertions(+), 15 deletions(-)
> 
> diff --git a/include/nilfs_gc.h b/include/nilfs_gc.h
> index e49cbf4..8bd83d7 100644
> --- a/include/nilfs_gc.h
> +++ b/include/nilfs_gc.h
> @@ -15,20 +15,21 @@
>  #include "nilfs.h"
>  
>  /* flags for nilfs_reclaim_params struct */
> -#define NILFS_RECLAIM_PARAM_PROTSEQ	(1UL << 0)
> -#define NILFS_RECLAIM_PARAM_PROTCNO	(1UL << 1)
> -#define __NR_NILFS_RECLAIM_PARAMS	2
> +#define NILFS_RECLAIM_PARAM_PROTSEQ			(1UL << 0)
> +#define NILFS_RECLAIM_PARAM_PROTCNO			(1UL << 1)
> +#define NILFS_RECLAIM_PARAM_MIN_RECLAIMABLE_BLKS	(1UL << 2)
> +#define __NR_NILFS_RECLAIM_PARAMS	3
>  
>  /**
>   * struct nilfs_reclaim_params - structure to specify GC parameters
>   * @flags: flags of valid fields
> - * @reserved: padding bytes
> + * @min_reclaimable_blks: minimum number of reclaimable blocks
>   * @protseq: start of sequence number of protected segments
>   * @protcno: start number of checkpoint to be protected
>   */
>  struct nilfs_reclaim_params {
>  	unsigned long flags;
> -	unsigned long reserved;
> +	unsigned long min_reclaimable_blks;
>  	__u64 protseq;
>  	nilfs_cno_t protcno;
>  };
> diff --git a/lib/gc.c b/lib/gc.c
> index 71c7307..5a1c2fe 100644
> --- a/lib/gc.c
> +++ b/lib/gc.c
> @@ -29,6 +29,10 @@
>  #include <syslog.h>
>  #endif	/* HAVE_SYSLOG_H */
>  
> +#if HAVE_SYS_TIME_H
> +#include <sys/time.h>
> +#endif	/* HAVE_SYS_TIME */
> +
>  #include <errno.h>
>  #include <assert.h>
>  #include <stdarg.h>
> @@ -614,11 +618,14 @@ int nilfs_xreclaim_segment(struct nilfs *nilfs,
>  			   const struct nilfs_reclaim_params *params,
>  			   struct nilfs_reclaim_stat *stat)
>  {
> -	struct nilfs_vector *vdescv, *bdescv, *periodv, *vblocknrv;
> +	struct nilfs_vector *vdescv, *bdescv, *periodv, *vblocknrv, *supv;
>  	sigset_t sigset, oldset, waitset;
>  	nilfs_cno_t protcno;
> -	ssize_t n, ret = -1;
> +	ssize_t n, i, ret = -1;
>  	size_t nblocks;
> +	__u32 reclaimable_blocks;
> +	struct nilfs_suinfo_update *sup;
> +	struct timeval tv;
>  
>  	if (!(params->flags & NILFS_RECLAIM_PARAM_PROTSEQ) ||
>  	    (params->flags & (~0UL << __NR_NILFS_RECLAIM_PARAMS))) {
> @@ -637,7 +644,8 @@ int nilfs_xreclaim_segment(struct nilfs *nilfs,
>  	bdescv = nilfs_vector_create(sizeof(struct nilfs_bdesc));
>  	periodv = nilfs_vector_create(sizeof(struct nilfs_period));
>  	vblocknrv = nilfs_vector_create(sizeof(__u64));
> -	if (!vdescv || !bdescv || !periodv || !vblocknrv)
> +	supv = nilfs_vector_create(sizeof(struct nilfs_suinfo_update));
> +	if (!vdescv || !bdescv || !periodv || !vblocknrv || !supv)
>  		goto out_vec;
>  
>  	sigemptyset(&sigset);
> @@ -703,13 +711,16 @@ int nilfs_xreclaim_segment(struct nilfs *nilfs,
>  	if (ret < 0)
>  		goto out_lock;
>  
> +	reclaimable_blocks = (nilfs_get_blocks_per_segment(nilfs) * n) -
> +			(nilfs_vector_get_size(vdescv) +
> +			nilfs_vector_get_size(bdescv));
> +
>  	if (stat) {
>  		stat->live_pblks = nilfs_vector_get_size(bdescv);
>  		stat->defunct_pblks = nblocks - stat->live_pblks;
>  
>  		stat->live_blks = stat->live_vblks + stat->live_pblks;
> -		stat->defunct_blks = n * nilfs_get_blocks_per_segment(nilfs) -
> -			stat->live_blks;
> +		stat->defunct_blks = reclaimable_blocks;
>  	}
>  	if (dryrun)
>  		goto out_lock;
> @@ -725,6 +736,54 @@ int nilfs_xreclaim_segment(struct nilfs *nilfs,
>  		goto out_lock;
>  	}
>  
> +	/*
> +	 * if there are less reclaimable blocks than the minimal
> +	 * threshold try to update suinfo instead of cleaning
> +	 */
> +	if ((params->flags & NILFS_RECLAIM_PARAM_MIN_RECLAIMABLE_BLKS) &&
> +			reclaimable_blocks < params->min_reclaimable_blks * n) {

I think nilfs_opt_test_set_suinfo(nilfs) should be tested here instead
of the nilfs_cleanerd_clean_segments() side.

Later, I will fix it.

> +		if (stat) {
> +			stat->deferred_segs = n;
> +			stat->cleaned_segs = 0;
> +		}
> +
> +		ret = gettimeofday(&tv, NULL);
> +		if (ret < 0)
> +			goto out_lock;
> +
> +		for (i = 0; i < n; ++i) {
> +			sup = nilfs_vector_get_new_element(supv);
> +			if (!sup)
> +				goto out_lock;
> +
> +			sup->sup_segnum = segnums[i];
> +			sup->sup_flags = 0;
> +			nilfs_suinfo_update_set_lastmod(sup);
> +			sup->sup_sui.sui_lastmod = tv.tv_sec;
> +		}
> +
> +		ret = nilfs_set_suinfo(nilfs, nilfs_vector_get_data(supv), n);
> +
> +		if (ret == 0)
> +			goto out_lock;
> +
> +		if (ret < 0 && errno != ENOTTY) {
> +			nilfs_gc_logger(LOG_ERR, "cannot set suinfo: %s",
> +					strerror(errno));
> +			goto out_lock;
> +		}
> +
> +		/* errno == ENOTTY */
> +		nilfs_gc_logger(LOG_WARNING,
> +				"set_suinfo ioctl is not supported");
> +		nilfs_opt_clear_set_suinfo(nilfs);
> +		if (stat) {
> +			stat->deferred_segs = 0;
> +			stat->cleaned_segs = n;
> +		}
> +		/* Try nilfs_clean_segments */
> +	}
> +
>  	ret = nilfs_clean_segments(nilfs,
>  				   nilfs_vector_get_data(vdescv),
>  				   nilfs_vector_get_size(vdescv),
> @@ -759,6 +818,8 @@ out_vec:
>  		nilfs_vector_destroy(periodv);
>  	if (vblocknrv != NULL)
>  		nilfs_vector_destroy(vblocknrv);
> +	if (supv != NULL)
> +		nilfs_vector_destroy(supv);
>  	/*
>  	 * Flags of valid fields in stat->exflags must be unset.
>  	 */
> @@ -783,7 +844,7 @@ ssize_t nilfs_reclaim_segment(struct nilfs *nilfs,
>  
>  	params.flags =
>  		NILFS_RECLAIM_PARAM_PROTSEQ | NILFS_RECLAIM_PARAM_PROTCNO;
> -	params.reserved = 0;
> +	params.min_reclaimable_blks = 0;
>  	params.protseq = protseq;
>  	params.protcno = protcno;
>  	memset(&stat, 0, sizeof(stat));
> diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
> index 2896af8..4c9f69b 100644
> --- a/sbin/cleanerd/cleanerd.c
> +++ b/sbin/cleanerd/cleanerd.c
> @@ -1376,7 +1376,7 @@ static void nilfs_cleanerd_progress(struct nilfs_cleanerd *cleanerd, int nsegs)
>  static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
>  					 __u64 *segnums, size_t nsegs,
>  					 __u64 protseq, __u64 prottime,
> -					 size_t *ncleaned)
> +					 size_t *ndone)
>  {
>  	struct nilfs_reclaim_params params;
>  	struct nilfs_reclaim_stat stat;
> @@ -1384,7 +1384,11 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
>  
>  	params.flags =
>  		NILFS_RECLAIM_PARAM_PROTSEQ | NILFS_RECLAIM_PARAM_PROTCNO;
> -	params.reserved = 0;
> +	if (nilfs_opt_test_set_suinfo(cleanerd->nilfs))
> +		params.flags |= NILFS_RECLAIM_PARAM_MIN_RECLAIMABLE_BLKS;
> +
> +	params.min_reclaimable_blks =
> +			nilfs_cleanerd_min_reclaimable_blocks(cleanerd);
>  	params.protseq = protseq;
>  
>  	ret = nilfs_cnoconv_time2cno(cleanerd->cnoconv, prottime,
> @@ -1402,7 +1406,7 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
>  		if (errno == ENOMEM) {
>  			nilfs_cleanerd_reduce_ncleansegs_for_retry(cleanerd);
>  			cleanerd->fallback = 1;
> -			*ncleaned = 0;
> +			*ndone = 0;
>  			ret = 0;
>  		}
>  		goto out;
> @@ -1415,6 +1419,17 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
>  		nilfs_cleanerd_progress(cleanerd, stat.cleaned_segs);
>  		cleanerd->fallback = 0;
>  		cleanerd->retry_cleaning = 0;
> +
> +		*ndone = stat.cleaned_segs;
> +	} else if (stat.deferred_segs > 0) {
> +		for (i = 0; i < stat.deferred_segs; i++)
> +			syslog(LOG_DEBUG, "segment %llu deferred",
> +			       (unsigned long long)segnums[i]);
> +		nilfs_cleanerd_progress(cleanerd, stat.deferred_segs);
> +		cleanerd->fallback = 0;
> +		cleanerd->retry_cleaning = 0;
> +
> +		*ndone = stat.deferred_segs;
>  	} else {
>  		syslog(LOG_DEBUG, "no segments cleaned");
>  
> @@ -1428,8 +1443,9 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
>  		} else {
>  			cleanerd->retry_cleaning = 0;
>  		}
> +		*ndone = 0;

This condition judgment supposes that stat.deferred_segs and
stat.cleaned_segs are exclusive.  It is correct in the current
implementation, but I don't want to define the semantics of
nilfs_xreclaim_segment() as such.

Can you rewrite this patch ?
Or, I will change this post handling.

>  	}
> -	*ncleaned = stat.cleaned_segs;
> +
>  out:
>  	return ret;
>  }
> -- 
> 1.8.5.3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v5 6/6] nilfs-utils: add a no_timeout flag to enable faster loop

[Ryusuke Konishi]

On Thu,  6 Feb 2014 15:17:26 +0100, Andreas Rohner wrote:
> This patch adds a flag, that enables the GC to skip the timeout in
> certain situations. For example if the cleaning of some segments
> was deferred to a later time, then no real progress has been
> made. Apart from reading a few summary blocks, no data was read or
> written to disk. In this situation it makes sense to skip the
> normal timeout once and immediately try to clean the next set of
> segments.
> 
> Unfortunately it is not possible, to directly continue the cleaning
> loop, because this would lead to an unresponsive GC process.
> Therefore the timeout is simply set to 0 seconds.
> 
> Signed-off-by: Andreas Rohner <andreas.rohner-hi6Y0CQ0nG0@public.gmane.org>
> ---
>  sbin/cleanerd/cleanerd.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
> index 4c9f69b..3506c20 100644
> --- a/sbin/cleanerd/cleanerd.c
> +++ b/sbin/cleanerd/cleanerd.c
> @@ -141,6 +141,7 @@ const static struct option long_option[] = {
>   * @running: running state
>   * @fallback: fallback state
>   * @retry_cleaning: retrying reclamation for protected segments
> + * @no_timeout: the next timeout will be 0 seconds
>   * @ncleansegs: number of semgents cleaned per cycle
>   * @cleaning_interval: cleaning interval
>   * @target: target time for sleeping
> @@ -169,6 +170,7 @@ struct nilfs_cleanerd {
>  	int running;
>  	int fallback;
>  	int retry_cleaning;
> +	int no_timeout;
>  	int shutdown;
>  	long ncleansegs;
>  	struct timeval cleaning_interval;
> @@ -896,7 +898,7 @@ static int nilfs_cleanerd_recalc_interval(struct nilfs_cleanerd *cleanerd,
>  	interval = nilfs_cleanerd_cleaning_interval(cleanerd);
>  	/* timercmp() does not work for '>=' or '<='. */
>  	/* curr >= target */
> -	if (!timercmp(&curr, &cleanerd->target, <)) {
> +	if (!timercmp(&curr, &cleanerd->target, <) || cleanerd->no_timeout) {
>  		cleanerd->timeout.tv_sec = 0;
>  		cleanerd->timeout.tv_usec = 0;
>  		timeradd(&curr, interval, &cleanerd->target);
> @@ -1428,6 +1430,7 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
>  		nilfs_cleanerd_progress(cleanerd, stat.deferred_segs);
>  		cleanerd->fallback = 0;
>  		cleanerd->retry_cleaning = 0;
> +		cleanerd->no_timeout = 1;
>  
>  		*ndone = stat.deferred_segs;
>  	} else {
> @@ -1501,6 +1504,8 @@ static int nilfs_cleanerd_clean_loop(struct nilfs_cleanerd *cleanerd)
>  		nilfs_cleanerd_clean_check_pause(cleanerd);
>  
>  	while (!cleanerd->shutdown) {
> +		cleanerd->no_timeout = 0;
> +
>  		if (sigprocmask(SIG_BLOCK, &sigset, NULL) < 0) {
>  			syslog(LOG_ERR, "cannot set signal mask: %m");
>  			return -1;
> -- 
> 1.8.5.3

This one looks good to me.

Regards,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v5 5/6] nilfs-utils: add optimized version of nilfs_xreclaim_segments

[Ryusuke Konishi]

On Fri, 07 Feb 2014 13:56:56 +0900 (JST), Ryusuke Konishi wrote:
> On Thu,  6 Feb 2014 15:17:25 +0100, Andreas Rohner wrote:
>> @@ -1415,6 +1419,17 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
>>  		nilfs_cleanerd_progress(cleanerd, stat.cleaned_segs);
>>  		cleanerd->fallback = 0;
>>  		cleanerd->retry_cleaning = 0;
>> +
>> +		*ndone = stat.cleaned_segs;
>> +	} else if (stat.deferred_segs > 0) {
>> +		for (i = 0; i < stat.deferred_segs; i++)
>> +			syslog(LOG_DEBUG, "segment %llu deferred",
>> +			       (unsigned long long)segnums[i]);
>> +		nilfs_cleanerd_progress(cleanerd, stat.deferred_segs);
>> +		cleanerd->fallback = 0;
>> +		cleanerd->retry_cleaning = 0;
>> +
>> +		*ndone = stat.deferred_segs;
>>  	} else {
>>  		syslog(LOG_DEBUG, "no segments cleaned");
>>  
>> @@ -1428,8 +1443,9 @@ static int nilfs_cleanerd_clean_segments(struct nilfs_cleanerd *cleanerd,
>>  		} else {
>>  			cleanerd->retry_cleaning = 0;
>>  		}
>> +		*ndone = 0;
> 
> This condition judgment supposes that stat.deferred_segs and
> stat.cleaned_segs are exclusive.  It is correct in the current
> implementation, but I don't want to define the semantics of
> nilfs_xreclaim_segment() as such.
> 
> Can you rewrite this patch ?
> Or, I will change this post handling.
> 

Uum, it needs to introduce another semantics or new arguments
to realize this since we don't know which segments are 
deferred.

I am thinking of doing this by sorting segnum[] array as follows:

 - cleand segs:
    segnum[0] .. segnum[cleaned_segs - 1]

 - deferred segs:
    segnum[cleaned_segs] .. segnum[cleaned_segs + deferred_segs - 1]

 - prorected segs:
    segnum[cleaned_segs + deferred_segs] ..
      segnum[cleaned_segs + deferred_segs + protected_segs - 1]

Actually, protected segments are moved behind segnums of cleaned
segments in the current implementation.


Regards,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html