[PATCH V3 0/5] makedumpfile: --split: assign fair I/O workloads in appropriate time

[PATCH V3 0/5] makedumpfile: --split: assign fair I/O workloads in appropriate time

[Zhou Wenjian]

v2->v3:
	1.remove the filtering for incomplete block (previous second pass) and adjust
	  relevant codes
	2.address HATAYAMA's comments about coding style

v1->v2:
	1.use splitblock instead of block
	2.add restriction (align to the page size) to splitblock size
	3.adjust the position of prepare_splitblock_table and check the return code
	4.use --splitblock-size to specify splitblock size and modify the print_info.c

the v2 test result is:

	cyc-buf	128		256		512		1024		2048		4096	
splblk-size
2M		24.05(0.65)	24.04(0.65)	24.58(0.65)	24.15(0.64)	24.14(0.64)	49.05(25.46)
4M		23.98(0.65)	24.01(0.65)	24.29(0.65)	24.20(0.64)	24.18(0.66)	49.04(25.46)
8M		24.02(0.65)	24.03(0.65)	24.25(0.65)	24.26(0.70)	24.15(0.64)	48.98(25.44)
16M		24.01(0.65)	24.01(0.65)	24.30(0.65)	24.19(0.64)	24.12(0.65)	48.99(25.45)
32M		23.97(0.65)	24.06(0.73)	24.23(0.65)	24.17(0.64)	24.19(0.64)	48.97(25.50)
64M		24.06(0.66)	24.07(0.66)	24.27(0.66)	24.16(0.65)	24.17(0.65)	48.98(25.49)
128M		24.03(0.67)	24.00(0.67)	24.27(0.66)	24.22(0.66)	24.19(0.66)	48.98(25.48)
256M		24.12(0.67)	23.99(0.67)	24.27(0.67)	24.17(0.66)	24.12(0.66)	49.04(25.49)
512M		24.06(0.70)	24.08(0.70)	24.26(0.70)	24.14(0.71)	24.19(0.70)	49.13(25.64)
1G		24.20(0.82)	24.13(0.81)	24.36(0.81)	24.31(0.80)	24.33(0.81)	49.28(25.75)
2G		24.19(0.81)	24.22(0.81)	24.37(0.81)	24.29(0.80)	24.28(0.82)	49.30(25.78)
4G		25.29(1.90)	25.26(1.91)	25.49(1.91)	25.41(1.89)	25.50(1.90)	49.99(26.45)
8G		25.33(1.90)	26.60(3.23)	26.87(3.21)	26.71(3.23)	26.64(3.22)	51.27(27.73)
16G		25.28(1.90)	26.52(3.21)	29.47(5.86)	29.34(5.84)	29.38(5.86)	53.99(30.40)


the latest test result turns to:

	cyc-buf	128		256		512		1024		2048		4096	
splblk-size
2M		23.34(0.00)	23.36(0.00)	23.51(0.00)	23.86(0.00)	23.48(0.00)	23.45(0.00)
4M		23.36(0.00)	23.32(0.00)	23.62(0.00)	23.48(0.00)	23.53(0.00)	23.45(0.00)
8M		23.33(0.00)	23.46(0.00)	23.61(0.00)	23.77(0.00)	23.52(0.00)	23.53(0.00)
16M		23.33(0.00)	23.38(0.00)	23.57(0.00)	23.47(0.00)	23.55(0.00)	23.49(0.00)
32M		23.40(0.00)	23.31(0.00)	23.64(0.00)	23.51(0.00)	23.50(0.00)	23.55(0.00)
64M		23.39(0.00)	23.41(0.00)	23.62(0.00)	23.47(0.00)	23.46(0.00)	23.50(0.00)
128M		23.36(0.00)	23.34(0.00)	23.59(0.00)	23.46(0.00)	23.46(0.00)	23.50(0.00)
256M		23.32(0.00)	23.36(0.00)	23.57(0.00)	23.49(0.00)	23.45(0.00)	23.52(0.00)
512M		23.32(0.00)	23.32(0.00)	23.63(0.00)	23.49(0.00)	23.52(0.00)	23.51(0.00)
1G		23.38(0.00)	23.32(0.00)	23.60(0.00)	23.54(0.00)	23.49(0.00)	23.54(0.00)
2G		23.34(0.00)	23.36(0.00)	23.62(0.00)	23.52(0.00)	23.52(0.00)	23.50(0.00)
4G		23.36(0.00)	23.37(0.00)	23.60(0.00)	23.48(0.00)	23.56(0.00)	23.55(0.00)
8G		23.37(0.00)	23.39(0.00)	23.52(0.00)	23.53(0.00)	23.46(0.00)	23.48(0.00)
16G		23.39(0.00)	23.41(0.00)	23.85(0.00)	23.48(0.00)	23.50(0.00)	23.45(0.00)

Zhou Wenjian (5):
  Add support for splitblock
  Add tools for reading and writing from splitblock table
  Add module of generating table
  Add module of calculating start_pfn and end_pfn in each dumpfile
  Add support for --splitblock-size

 makedumpfile.8 |   10 +++
 makedumpfile.c |  244 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 makedumpfile.h |   19 +++++
 print_info.c   |    5 +
 4 files changed, 272 insertions(+), 6 deletions(-)

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH V3 1/5] Add support for splitblock

[Zhou Wenjian]

When --split option is specified, fair I/O workloads shoud be assigned
for each process. So the start and end pfn of each dumpfile should be
calculated with excluding unnecessary pages. However, it costs a lot of
time to execute excluding for the whole memory. That is why struct
SplitBlock exists. Struct SplitBlock is designed to manage memory, mainly
for recording the number of dumpable pages. We can use the number of
dumpable pages to calculate start and end pfn instead of execute excluding
for the whole memory.

The char array *table in struct SplitBlock is used to record the number of
dumpable pages.
The table entry size is calculated as
                        divideup(log2(splitblock_size / page_size), 8) bytes
The table entry size is calculated, so that the
space table taken will be small enough. And the code will also have a
good performence when the number of pages in one splitblock is big enough.

Signed-off-by: Qiao Nuohan <qiaonuohan@cn.fujitsu.com>
Signed-off-by: Zhou Wenjian <zhouwj-fnst@cn.fujitsu.com>
---
 makedumpfile.c |   31 +++++++++++++++++++++++++++++++
 makedumpfile.h |   14 ++++++++++++++
 2 files changed, 45 insertions(+), 0 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index b4d43d8..7017943 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -34,6 +34,7 @@ struct srcfile_table	srcfile_table;
 
 struct vm_table		vt = { 0 };
 struct DumpInfo		*info = NULL;
+struct SplitBlock		*splitblock = NULL;
 
 char filename_stdout[] = FILENAME_STDOUT;
 
@@ -5685,6 +5686,36 @@ out:
 	return ret;
 }
 
+/*
+ * cyclic_split mode:
+ *	manage memory by splitblocks,
+ *	divide memory into splitblocks
+ *	use splitblock_table to record numbers of dumpable pages in each
+ *	splitblock
+ */
+
+/*
+ * calculate entry size based on the amount of pages in one splitblock
+ */
+int
+calculate_entry_size(void)
+{
+	int entry_num = 1;
+	int count = 1;
+	int entry_size;
+
+	while (entry_num < splitblock->page_per_splitblock){
+		entry_num = entry_num << 1;
+		count++;
+	}
+
+	entry_size = count / BITPERBYTE;
+	if (count % BITPERBYTE)
+		entry_size++;
+
+	return entry_size;
+}
+
 mdf_pfn_t
 get_num_dumpable(void)
 {
diff --git a/makedumpfile.h b/makedumpfile.h
index 96830b0..7d9c2e6 100644
--- a/makedumpfile.h
+++ b/makedumpfile.h
@@ -1168,10 +1168,24 @@ struct DumpInfo {
 	 */
 	int (*page_is_buddy)(unsigned long flags, unsigned int _mapcount,
 			     unsigned long private, unsigned int _count);
+	/*
+	 * for cyclic_splitting mode, setup splitblock_size
+	 */
+	long long splitblock_size;
 };
 extern struct DumpInfo		*info;
 
 /*
+ * for cyclic_splitting mode,Manage memory by splitblock
+ */
+struct SplitBlock{
+        char *table;
+        long long num;
+        long long page_per_splitblock;
+        int entry_size;                 /* counted by byte */
+};
+
+/*
  * kernel VM-related data
  */
 struct vm_table {
-- 
1.7.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH V3 2/5] Add tools for reading and writing from splitblock table

[Zhou Wenjian]

The function added in this patch, is used for writing and reading value
from the char array in struct SplitBlock.

Signed-off-by: Qiao Nuohan <qiaonuohan@cn.fujitsu.com>
Signed-off-by: Zhou Wenjian <zhouwj-fnst@cn.fujitsu.com>
---
 makedumpfile.c |   29 +++++++++++++++++++++++++++++
 1 files changed, 29 insertions(+), 0 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index 7017943..f86dfd2 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -5716,6 +5716,35 @@ calculate_entry_size(void)
 	return entry_size;
 }
 
+void
+write_into_splitblock_table(char *entry,
+				unsigned long long value)
+{
+	char temp;
+	int i = 0;
+
+	while (i++ < splitblock->entry_size) {
+		temp = value & 0xff;
+		value = value >> BITPERBYTE;
+		*entry = temp;
+		entry++;
+	}
+}
+
+unsigned long long
+read_from_splitblock_table(char *entry)
+{
+	unsigned long long value = 0;
+	int i;
+
+	for (i = splitblock->entry_size; i > 0; i--) {
+		value = value << BITPERBYTE;
+		value += *(entry + i - 1) & 0xff;
+	}
+
+	return value;
+}
+
 mdf_pfn_t
 get_num_dumpable(void)
 {
-- 
1.7.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH V3 3/5] Add module of generating table

[Zhou Wenjian]

set block size and generate basic information of block table

Signed-off-by: Qiao Nuohan <qiaonuohan@cn.fujitsu.com>
Signed-off-by: Zhou Wenjian <zhouwj-fnst@cn.fujitsu.com>
---
 makedumpfile.c |  109 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 makedumpfile.h |    4 ++
 2 files changed, 112 insertions(+), 1 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index f86dfd2..9ead207 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -5208,7 +5208,14 @@ create_dump_bitmap(void)
 	if (info->flag_cyclic) {
 		if (!prepare_bitmap2_buffer_cyclic())
 			goto out;
-		info->num_dumpable = get_num_dumpable_cyclic();
+		if (info->flag_split) {
+			if(!prepare_splitblock_table())
+				goto out;
+
+			info->num_dumpable = get_num_dumpable_cyclic_withsplit();
+		}else {
+			info->num_dumpable = get_num_dumpable_cyclic();
+		}
 
 		if (!info->flag_elf_dumpfile)
 			free_bitmap2_buffer_cyclic();
@@ -5745,6 +5752,61 @@ read_from_splitblock_table(char *entry)
 	return value;
 }
 
+/*
+ * The splitblock size is specified as Kbyte with --splitblock-size <size> option.
+ * If not specified, set default value.
+ */
+int
+check_splitblock_size(void)
+{
+	if (info->splitblock_size) {
+		info->splitblock_size <<= 10;
+		if (info->splitblock_size == 0) {
+			ERRMSG("The splitblock size could not be 0. %s.\n",
+				strerror(errno));
+			return FALSE;
+		}
+		if (info->splitblock_size % info->page_size != 0) {
+			ERRMSG("The splitblock size must be align to page_size. %s.\n",
+				strerror(errno));
+			return FALSE;
+		}
+	} else {
+		info->splitblock_size = DEFAULT_SPLITBLOCK_SIZE;
+	}
+
+	return TRUE;
+}
+
+int
+prepare_splitblock_table(void)
+{
+	size_t table_size;
+
+	if(!check_splitblock_size())
+		return FALSE;
+	if ((splitblock = calloc(1, sizeof(struct SplitBlock))) == NULL) {
+		ERRMSG("Can't allocate memory for the splitblock. %s.\n",
+			strerror(errno));
+		return FALSE;
+	}
+
+	splitblock->page_per_splitblock = info->splitblock_size / info->page_size;
+	splitblock->num = (info->max_mapnr + splitblock->page_per_splitblock -1) /
+			  splitblock->page_per_splitblock;
+	splitblock->entry_size = calculate_entry_size();
+	table_size = splitblock->entry_size * splitblock->num;
+
+	splitblock->table = (char *)calloc(sizeof(char), table_size);
+	if (!splitblock->table) {
+		ERRMSG("Can't allocate memory for the splitblock_table. %s.\n",
+			 strerror(errno));
+		return FALSE;
+	}
+
+	return TRUE;
+}
+
 mdf_pfn_t
 get_num_dumpable(void)
 {
@@ -5760,6 +5822,45 @@ get_num_dumpable(void)
 	return num_dumpable;
 }
 
+/*
+ * generate splitblock_table
+ * modified from function get_num_dumpable_cyclic
+ */
+mdf_pfn_t
+get_num_dumpable_cyclic_withsplit(void)
+{
+	mdf_pfn_t pfn, num_dumpable = 0;
+	mdf_pfn_t dumpable_pfn_num = 0, pfn_num = 0;
+	struct cycle cycle = {0};
+	int pos = 0;
+
+	pfn_memhole = info->max_mapnr;
+
+	for_each_cycle(0, info->max_mapnr, &cycle) {
+		if (!exclude_unnecessary_pages_cyclic(&cycle))
+			return FALSE;
+
+		if (info->flag_mem_usage)
+			exclude_zero_pages_cyclic(&cycle);
+
+		for (pfn = cycle.start_pfn; pfn < cycle.end_pfn; pfn++) {
+			if (is_dumpable_cyclic(info->partial_bitmap2, pfn, &cycle)) {
+				num_dumpable++;
+				dumpable_pfn_num++;
+			}
+			if (++pfn_num >= splitblock->page_per_splitblock) {
+				write_into_splitblock_table(splitblock->table + pos,
+							    dumpable_pfn_num);
+				pos += splitblock->entry_size;
+				pfn_num = 0;
+				dumpable_pfn_num = 0;
+			}
+		}
+	}
+
+	return num_dumpable;
+}
+
 mdf_pfn_t
 get_num_dumpable_cyclic(void)
 {
@@ -9717,6 +9818,12 @@ out:
 		if (info->page_buf != NULL)
 			free(info->page_buf);
 		free(info);
+
+		if (splitblock) {
+			if (splitblock->table)
+				free(splitblock->table);
+			free(splitblock);
+		}
 	}
 	free_elf_info();
 
diff --git a/makedumpfile.h b/makedumpfile.h
index 7d9c2e6..39c49b8 100644
--- a/makedumpfile.h
+++ b/makedumpfile.h
@@ -1178,6 +1178,8 @@ extern struct DumpInfo		*info;
 /*
  * for cyclic_splitting mode,Manage memory by splitblock
  */
+#define DEFAULT_SPLITBLOCK_SIZE (1LL << 30)
+
 struct SplitBlock{
         char *table;
         long long num;
@@ -1888,9 +1890,11 @@ struct elf_prstatus {
  * Function Prototype.
  */
 mdf_pfn_t get_num_dumpable_cyclic(void);
+mdf_pfn_t get_num_dumpable_cyclic_withsplit(void);
 int get_loads_dumpfile_cyclic(void);
 int initial_xen(void);
 unsigned long long get_free_memory_size(void);
 int calculate_cyclic_buffer_size(void);
+int prepare_splitblock_table(void);
 
 #endif /* MAKEDUMPFILE_H */
-- 
1.7.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH V3 4/5] Add module of calculating start_pfn and end_pfn in each dumpfile

[Zhou Wenjian]

When --split is specified in cyclic mode, start_pfn and end_pfn of each dumpfile
will be calculated to make each dumpfile have the same size.

Signed-off-by: Qiao Nuohan <qiaonuohan@cn.fujitsu.com>
Signed-off-by: Zhou Wenjian <zhouwj-fnst@cn.fujitsu.com>
---
 makedumpfile.c |   71 ++++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 files changed, 66 insertions(+), 5 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index 9ead207..1468566 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -8218,6 +8218,63 @@ out:
 		return ret;
 }
 
+/*
+ * calculate end_pfn of one dumpfile.
+ * try to make every output file have the same size.
+ * splitblock_table is used to reduce calculate time.
+ */
+
+#define CURRENT_SPLITBLOCK_PFN_NUM (*cur_splitblock_num * splitblock->page_per_splitblock)
+mdf_pfn_t
+calculate_end_pfn_by_splitblock(mdf_pfn_t start_pfn,
+				 int *cur_splitblock_num)
+{
+	if (start_pfn >= info->max_mapnr)
+		return info->max_mapnr;
+
+	mdf_pfn_t end_pfn;
+	long long pfn_needed, offset;
+	char *splitblock_value_offset;
+
+	pfn_needed = info->num_dumpable / info->num_dumpfile;
+	offset = *cur_splitblock_num * splitblock->entry_size;
+	splitblock_value_offset = splitblock->table + offset;
+	end_pfn = start_pfn;
+
+	while (*cur_splitblock_num < splitblock->num && pfn_needed > 0) {
+		pfn_needed -= read_from_splitblock_table(splitblock_value_offset);
+		splitblock_value_offset += splitblock->entry_size;
+		++*cur_splitblock_num;
+	}
+
+	end_pfn = CURRENT_SPLITBLOCK_PFN_NUM;
+	if (end_pfn > info->max_mapnr)
+		end_pfn = info->max_mapnr;
+
+	return end_pfn;
+}
+
+/*
+ * calculate start_pfn and end_pfn in each output file.
+ */
+static int setup_splitting_cyclic(void)
+{
+	int i;
+	mdf_pfn_t start_pfn, end_pfn;
+	int cur_splitblock_num = 0;
+	start_pfn = end_pfn = 0;
+
+	for (i = 0; i < info->num_dumpfile; i++) {
+		start_pfn = end_pfn;
+		end_pfn = calculate_end_pfn_by_splitblock(start_pfn,
+							  &cur_splitblock_num);
+		SPLITTING_START_PFN(i) = start_pfn;
+		SPLITTING_END_PFN(i) = end_pfn;
+	}
+
+	return TRUE;
+}
+
 int
 setup_splitting(void)
 {
@@ -8231,12 +8288,16 @@ setup_splitting(void)
 		return FALSE;
 
 	if (info->flag_cyclic) {
-		for (i = 0; i < info->num_dumpfile; i++) {
-			SPLITTING_START_PFN(i) = divideup(info->max_mapnr, info->num_dumpfile) * i;
-			SPLITTING_END_PFN(i)   = divideup(info->max_mapnr, info->num_dumpfile) * (i + 1);
+		int ret = FALSE;
+
+		if(!prepare_bitmap2_buffer_cyclic()){
+			free_bitmap_buffer();
+			return ret;
 		}
-		if (SPLITTING_END_PFN(i-1) > info->max_mapnr)
-			SPLITTING_END_PFN(i-1) = info->max_mapnr;
+		ret = setup_splitting_cyclic();
+		free_bitmap2_buffer_cyclic();
+
+		return ret;
         } else {
 		initialize_2nd_bitmap(&bitmap2);
 
-- 
1.7.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH V3 5/5] Add support for --splitblock-size

[Zhou Wenjian]

Use --splitblock-size to specify splitblock size (KB)
When --split is specified in cyclic mode,splitblock table will be
generated in create_dump_bitmap().

Signed-off-by: Qiao Nuohan <qiaonuohan@cn.fujitsu.com>
Signed-off-by: Zhou Wenjian <zhouwj-fnst@cn.fujitsu.com>
---
 makedumpfile.8 |   10 ++++++++++
 makedumpfile.c |    4 ++++
 makedumpfile.h |    1 +
 print_info.c   |    5 +++++
 4 files changed, 20 insertions(+), 0 deletions(-)

diff --git a/makedumpfile.8 b/makedumpfile.8
index 9cb12c0..5e121fd 100644
--- a/makedumpfile.8
+++ b/makedumpfile.8
@@ -386,6 +386,16 @@ size, so ordinary users don't need to specify this option.
 # makedumpfile \-\-cyclic\-buffer 1024 \-d 31 \-x vmlinux /proc/vmcore dumpfile
 
 .TP
+\fB\-\-splitblock\-size\fR \fIsplitblock_size\fR
+Specify the splitblock size in kilo bytes for analysis in the cyclic mode with --split.
+If --splitblock N is specified, difference of each splitted dumpfile size is at most N
+kilo bytes.
+.br
+.B Example:
+.br
+# makedumpfile \-\-splitblock\-size 1024 \-d 31 \-x vmlinux \-\-split /proc/vmcore dumpfile1 dumpfile2
+
+.TP
 \fB\-\-non\-cyclic\fR
 Running in the non-cyclic mode, this mode uses the old filtering logic same as v1.4.4 or before.
 If you feel the cyclic mode is too slow, please try this mode.
diff --git a/makedumpfile.c b/makedumpfile.c
index 1468566..ec712d2 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -9568,6 +9568,7 @@ static struct option longopts[] = {
 	{"eppic", required_argument, NULL, OPT_EPPIC},
 	{"non-mmap", no_argument, NULL, OPT_NON_MMAP},
 	{"mem-usage", no_argument, NULL, OPT_MEM_USAGE},
+	{"splitblock-size", required_argument, NULL, OPT_SPLITBLOCK_SIZE},
 	{0, 0, 0, 0}
 };
 
@@ -9708,6 +9709,9 @@ main(int argc, char *argv[])
 		case OPT_CYCLIC_BUFFER:
 			info->bufsize_cyclic = atoi(optarg);
 			break;
+		case OPT_SPLITBLOCK_SIZE:
+			info->splitblock_size = atoi(optarg);
+			break;
 		case '?':
 			MSG("Commandline parameter is invalid.\n");
 			MSG("Try `makedumpfile --help' for more information.\n");
diff --git a/makedumpfile.h b/makedumpfile.h
index 39c49b8..d195069 100644
--- a/makedumpfile.h
+++ b/makedumpfile.h
@@ -1885,6 +1885,7 @@ struct elf_prstatus {
 #define OPT_EPPIC               OPT_START+12
 #define OPT_NON_MMAP            OPT_START+13
 #define OPT_MEM_USAGE            OPT_START+14
+#define OPT_SPLITBLOCK_SIZE		OPT_START+15
 
 /*
  * Function Prototype.
diff --git a/print_info.c b/print_info.c
index f6342d3..16830b2 100644
--- a/print_info.c
+++ b/print_info.c
@@ -203,6 +203,11 @@ print_usage(void)
 	MSG("      By default, BUFFER_SIZE will be calculated automatically depending on\n");
 	MSG("      system memory size, so ordinary users don't need to specify this option.\n");
 	MSG("\n");
+	MSG("  [--splitblock-size SPLITBLOCK_SIZE]:\n");
+	MSG("      Specify the splitblock size in kilo bytes for analysis in the cyclic mode\n");
+	MSG("      with --split.If --splitblock N is specified, difference of each splitted\n");
+	MSG("	   dumpfile size is at most N kilo bytes.\n");
+	MSG("\n");
 	MSG("  [--non-cyclic]:\n");
 	MSG("      Running in the non-cyclic mode, this mode uses the old filtering logic\n");
 	MSG("      same as v1.4.4 or before.\n");
-- 
1.7.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH V3 0/5] makedumpfile: --split: assign fair I/O workloads in appropriate time

["Zhou, Wenjian/周文剑"]

Ping...

On 10/30/2014 05:45 PM, Zhou Wenjian wrote:
> v2->v3:
> 	1.remove the filtering for incomplete block (previous second pass) and adjust
> 	  relevant codes
> 	2.address HATAYAMA's comments about coding style
>
> v1->v2:
> 	1.use splitblock instead of block
> 	2.add restriction (align to the page size) to splitblock size
> 	3.adjust the position of prepare_splitblock_table and check the return code
> 	4.use --splitblock-size to specify splitblock size and modify the print_info.c
>
> the v2 test result is:
>
> 	cyc-buf	128		256		512		1024		2048		4096	
> splblk-size
> 2M		24.05(0.65)	24.04(0.65)	24.58(0.65)	24.15(0.64)	24.14(0.64)	49.05(25.46)
> 4M		23.98(0.65)	24.01(0.65)	24.29(0.65)	24.20(0.64)	24.18(0.66)	49.04(25.46)
> 8M		24.02(0.65)	24.03(0.65)	24.25(0.65)	24.26(0.70)	24.15(0.64)	48.98(25.44)
> 16M		24.01(0.65)	24.01(0.65)	24.30(0.65)	24.19(0.64)	24.12(0.65)	48.99(25.45)
> 32M		23.97(0.65)	24.06(0.73)	24.23(0.65)	24.17(0.64)	24.19(0.64)	48.97(25.50)
> 64M		24.06(0.66)	24.07(0.66)	24.27(0.66)	24.16(0.65)	24.17(0.65)	48.98(25.49)
> 128M		24.03(0.67)	24.00(0.67)	24.27(0.66)	24.22(0.66)	24.19(0.66)	48.98(25.48)
> 256M		24.12(0.67)	23.99(0.67)	24.27(0.67)	24.17(0.66)	24.12(0.66)	49.04(25.49)
> 512M		24.06(0.70)	24.08(0.70)	24.26(0.70)	24.14(0.71)	24.19(0.70)	49.13(25.64)
> 1G		24.20(0.82)	24.13(0.81)	24.36(0.81)	24.31(0.80)	24.33(0.81)	49.28(25.75)
> 2G		24.19(0.81)	24.22(0.81)	24.37(0.81)	24.29(0.80)	24.28(0.82)	49.30(25.78)
> 4G		25.29(1.90)	25.26(1.91)	25.49(1.91)	25.41(1.89)	25.50(1.90)	49.99(26.45)
> 8G		25.33(1.90)	26.60(3.23)	26.87(3.21)	26.71(3.23)	26.64(3.22)	51.27(27.73)
> 16G		25.28(1.90)	26.52(3.21)	29.47(5.86)	29.34(5.84)	29.38(5.86)	53.99(30.40)
>
>
> the latest test result turns to:
>
> 	cyc-buf	128		256		512		1024		2048		4096	
> splblk-size
> 2M		23.34(0.00)	23.36(0.00)	23.51(0.00)	23.86(0.00)	23.48(0.00)	23.45(0.00)
> 4M		23.36(0.00)	23.32(0.00)	23.62(0.00)	23.48(0.00)	23.53(0.00)	23.45(0.00)
> 8M		23.33(0.00)	23.46(0.00)	23.61(0.00)	23.77(0.00)	23.52(0.00)	23.53(0.00)
> 16M		23.33(0.00)	23.38(0.00)	23.57(0.00)	23.47(0.00)	23.55(0.00)	23.49(0.00)
> 32M		23.40(0.00)	23.31(0.00)	23.64(0.00)	23.51(0.00)	23.50(0.00)	23.55(0.00)
> 64M		23.39(0.00)	23.41(0.00)	23.62(0.00)	23.47(0.00)	23.46(0.00)	23.50(0.00)
> 128M		23.36(0.00)	23.34(0.00)	23.59(0.00)	23.46(0.00)	23.46(0.00)	23.50(0.00)
> 256M		23.32(0.00)	23.36(0.00)	23.57(0.00)	23.49(0.00)	23.45(0.00)	23.52(0.00)
> 512M		23.32(0.00)	23.32(0.00)	23.63(0.00)	23.49(0.00)	23.52(0.00)	23.51(0.00)
> 1G		23.38(0.00)	23.32(0.00)	23.60(0.00)	23.54(0.00)	23.49(0.00)	23.54(0.00)
> 2G		23.34(0.00)	23.36(0.00)	23.62(0.00)	23.52(0.00)	23.52(0.00)	23.50(0.00)
> 4G		23.36(0.00)	23.37(0.00)	23.60(0.00)	23.48(0.00)	23.56(0.00)	23.55(0.00)
> 8G		23.37(0.00)	23.39(0.00)	23.52(0.00)	23.53(0.00)	23.46(0.00)	23.48(0.00)
> 16G		23.39(0.00)	23.41(0.00)	23.85(0.00)	23.48(0.00)	23.50(0.00)	23.45(0.00)
>
> Zhou Wenjian (5):
>    Add support for splitblock
>    Add tools for reading and writing from splitblock table
>    Add module of generating table
>    Add module of calculating start_pfn and end_pfn in each dumpfile
>    Add support for --splitblock-size
>
>   makedumpfile.8 |   10 +++
>   makedumpfile.c |  244 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
>   makedumpfile.h |   19 +++++
>   print_info.c   |    5 +
>   4 files changed, 272 insertions(+), 6 deletions(-)
>
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: [PATCH V3 3/5] Add module of generating table

[Atsushi Kumagai]

>set block size and generate basic information of block table
>
>Signed-off-by: Qiao Nuohan <qiaonuohan@cn.fujitsu.com>
>Signed-off-by: Zhou Wenjian <zhouwj-fnst@cn.fujitsu.com>
>---
> makedumpfile.c |  109 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> makedumpfile.h |    4 ++
> 2 files changed, 112 insertions(+), 1 deletions(-)
>
>diff --git a/makedumpfile.c b/makedumpfile.c
>index f86dfd2..9ead207 100644
>--- a/makedumpfile.c
>+++ b/makedumpfile.c
>@@ -5208,7 +5208,14 @@ create_dump_bitmap(void)
> 	if (info->flag_cyclic) {
> 		if (!prepare_bitmap2_buffer_cyclic())
> 			goto out;
>-		info->num_dumpable = get_num_dumpable_cyclic();
>+		if (info->flag_split) {
>+			if(!prepare_splitblock_table())
>+				goto out;
>+
>+			info->num_dumpable = get_num_dumpable_cyclic_withsplit();
>+		}else {
>+			info->num_dumpable = get_num_dumpable_cyclic();
>+		}
>
> 		if (!info->flag_elf_dumpfile)
> 			free_bitmap2_buffer_cyclic();
>@@ -5745,6 +5752,61 @@ read_from_splitblock_table(char *entry)
> 	return value;
> }
>
>+/*
>+ * The splitblock size is specified as Kbyte with --splitblock-size <size> option.
>+ * If not specified, set default value.
>+ */
>+int
>+check_splitblock_size(void)
>+{
>+	if (info->splitblock_size) {
>+		info->splitblock_size <<= 10;
>+		if (info->splitblock_size == 0) {
>+			ERRMSG("The splitblock size could not be 0. %s.\n",
>+				strerror(errno));
>+			return FALSE;
>+		}
>+		if (info->splitblock_size % info->page_size != 0) {
>+			ERRMSG("The splitblock size must be align to page_size. %s.\n",
>+				strerror(errno));
>+			return FALSE;
>+		}
>+	} else {
>+		info->splitblock_size = DEFAULT_SPLITBLOCK_SIZE;
>+	}
>+
>+	return TRUE;
>+}
>+
>+int
>+prepare_splitblock_table(void)
>+{
>+	size_t table_size;
>+
>+	if(!check_splitblock_size())
>+		return FALSE;
>+	if ((splitblock = calloc(1, sizeof(struct SplitBlock))) == NULL) {
>+		ERRMSG("Can't allocate memory for the splitblock. %s.\n",
>+			strerror(errno));
>+		return FALSE;
>+	}
>+
>+	splitblock->page_per_splitblock = info->splitblock_size / info->page_size;
>+	splitblock->num = (info->max_mapnr + splitblock->page_per_splitblock -1) /
>+			  splitblock->page_per_splitblock;

Please use "divideup(info->max_mapnr, splitblock->page_per_splitblock)".


Thanks,
Atsushi Kumagai

>+	splitblock->entry_size = calculate_entry_size();
>+	table_size = splitblock->entry_size * splitblock->num;
>+
>+	splitblock->table = (char *)calloc(sizeof(char), table_size);
>+	if (!splitblock->table) {
>+		ERRMSG("Can't allocate memory for the splitblock_table. %s.\n",
>+			 strerror(errno));
>+		return FALSE;
>+	}
>+
>+	return TRUE;
>+}
>+
> mdf_pfn_t
> get_num_dumpable(void)
> {
>@@ -5760,6 +5822,45 @@ get_num_dumpable(void)
> 	return num_dumpable;
> }
>
>+/*
>+ * generate splitblock_table
>+ * modified from function get_num_dumpable_cyclic
>+ */
>+mdf_pfn_t
>+get_num_dumpable_cyclic_withsplit(void)
>+{
>+	mdf_pfn_t pfn, num_dumpable = 0;
>+	mdf_pfn_t dumpable_pfn_num = 0, pfn_num = 0;
>+	struct cycle cycle = {0};
>+	int pos = 0;
>+
>+	pfn_memhole = info->max_mapnr;
>+
>+	for_each_cycle(0, info->max_mapnr, &cycle) {
>+		if (!exclude_unnecessary_pages_cyclic(&cycle))
>+			return FALSE;
>+
>+		if (info->flag_mem_usage)
>+			exclude_zero_pages_cyclic(&cycle);
>+
>+		for (pfn = cycle.start_pfn; pfn < cycle.end_pfn; pfn++) {
>+			if (is_dumpable_cyclic(info->partial_bitmap2, pfn, &cycle)) {
>+				num_dumpable++;
>+				dumpable_pfn_num++;
>+			}
>+			if (++pfn_num >= splitblock->page_per_splitblock) {
>+				write_into_splitblock_table(splitblock->table + pos,
>+							    dumpable_pfn_num);
>+				pos += splitblock->entry_size;
>+				pfn_num = 0;
>+				dumpable_pfn_num = 0;
>+			}
>+		}
>+	}
>+
>+	return num_dumpable;
>+}
>+
> mdf_pfn_t
> get_num_dumpable_cyclic(void)
> {
>@@ -9717,6 +9818,12 @@ out:
> 		if (info->page_buf != NULL)
> 			free(info->page_buf);
> 		free(info);
>+
>+		if (splitblock) {
>+			if (splitblock->table)
>+				free(splitblock->table);
>+			free(splitblock);
>+		}
> 	}
> 	free_elf_info();
>
>diff --git a/makedumpfile.h b/makedumpfile.h
>index 7d9c2e6..39c49b8 100644
>--- a/makedumpfile.h
>+++ b/makedumpfile.h
>@@ -1178,6 +1178,8 @@ extern struct DumpInfo		*info;
> /*
>  * for cyclic_splitting mode,Manage memory by splitblock
>  */
>+#define DEFAULT_SPLITBLOCK_SIZE (1LL << 30)
>+
> struct SplitBlock{
>         char *table;
>         long long num;
>@@ -1888,9 +1890,11 @@ struct elf_prstatus {
>  * Function Prototype.
>  */
> mdf_pfn_t get_num_dumpable_cyclic(void);
>+mdf_pfn_t get_num_dumpable_cyclic_withsplit(void);
> int get_loads_dumpfile_cyclic(void);
> int initial_xen(void);
> unsigned long long get_free_memory_size(void);
> int calculate_cyclic_buffer_size(void);
>+int prepare_splitblock_table(void);
>
> #endif /* MAKEDUMPFILE_H */
>--
>1.7.1
>
>
>_______________________________________________
>kexec mailing list
>kexec@lists.infradead.org
>http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: [PATCH V3 2/5] Add tools for reading and writing from splitblock table

[Atsushi Kumagai]

>The function added in this patch, is used for writing and reading value
>from the char array in struct SplitBlock.
>
>Signed-off-by: Qiao Nuohan <qiaonuohan@cn.fujitsu.com>
>Signed-off-by: Zhou Wenjian <zhouwj-fnst@cn.fujitsu.com>
>---
> makedumpfile.c |   29 +++++++++++++++++++++++++++++
> 1 files changed, 29 insertions(+), 0 deletions(-)
>
>diff --git a/makedumpfile.c b/makedumpfile.c
>index 7017943..f86dfd2 100644
>--- a/makedumpfile.c
>+++ b/makedumpfile.c
>@@ -5716,6 +5716,35 @@ calculate_entry_size(void)
> 	return entry_size;
> }
>
>+void
>+write_into_splitblock_table(char *entry,
>+				unsigned long long value)
>+{
>+	char temp;
>+	int i = 0;
>+
>+	while (i++ < splitblock->entry_size) {
>+		temp = value & 0xff;
>+		value = value >> BITPERBYTE;
>+		*entry = temp;
>+		entry++;
>+	}
>+}
>+
>+unsigned long long
>+read_from_splitblock_table(char *entry)
>+{
>+	unsigned long long value = 0;
>+	int i;
>+
>+	for (i = splitblock->entry_size; i > 0; i--) {
>+		value = value << BITPERBYTE;
>+		value += *(entry + i - 1) & 0xff;
>+	}

Did you forget to reflect the HATAYAMA-san's comment ?

>> +	for (i = splitblock->entry_size; i > 0; i--) {
>> +		ret = ret << BITPERBYTE;
>> +		ret += *(splitblock_inner + i - 1) & 0xff;
>
> The & 0xff is necessary? because splitblock_inner is of type char *.


Thanks,
Atsushi Kumagai

>+
>+	return value;
>+}
>+
> mdf_pfn_t
> get_num_dumpable(void)
> {
>--
>1.7.1
>
>
>_______________________________________________
>kexec mailing list
>kexec@lists.infradead.org
>http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH V3 2/5] Add tools for reading and writing from splitblock table

["Zhou, Wenjian/周文剑"]

On 11/05/2014 12:18 PM, Atsushi Kumagai wrote:
>> +unsigned long long
>> >+read_from_splitblock_table(char *entry)
>> >+{
>> >+	unsigned long long value = 0;
>> >+	int i;
>> >+
>> >+	for (i = splitblock->entry_size; i>  0; i--) {
>> >+		value = value<<  BITPERBYTE;
>> >+		value += *(entry + i - 1)&  0xff;
>> >+	}
> Did you forget to reflect the HATAYAMA-san's comment ?
>

I think it is needed.
For example:
	if *(entry + i - 1) == 0xfa
	without & 0xff:
		it will turn to: value += 0xfffffffffffffffa
	with & 0xff:
		it will be: value += 0xfffffffffffffffa & 0xff

Thanks
Zhou Wenjian

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: [PATCH V3 2/5] Add tools for reading and writing from splitblock table

[Atsushi Kumagai]

>On 11/05/2014 12:18 PM, Atsushi Kumagai wrote:
>>> +unsigned long long
>>> >+read_from_splitblock_table(char *entry)
>>> >+{
>>> >+	unsigned long long value = 0;
>>> >+	int i;
>>> >+
>>> >+	for (i = splitblock->entry_size; i>  0; i--) {
>>> >+		value = value<<  BITPERBYTE;
>>> >+		value += *(entry + i - 1)&  0xff;
>>> >+	}
>> Did you forget to reflect the HATAYAMA-san's comment ?
>>
>
>I think it is needed.
>For example:
>	if *(entry + i - 1) == 0xfa
>	without & 0xff:
>		it will turn to: value += 0xfffffffffffffffa
>	with & 0xff:
>		it will be: value += 0xfffffffffffffffa & 0xff

Ah, I see, you meant "sign extension".
OK, I understand.


Thanks,
Atsushi Kumagai

>
>Thanks
>Zhou Wenjian
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec