Repeat given pattern with buffer_compress_percentage

Repeat given pattern with buffer_compress_percentage

[Karthick Srinivasachary]

Hi Folks,

I have question. Any feedback appreciated..!

With buffer_compress_percentage, fio writes random data + zeros.

Is there a way to write random data + repeat_pattern (instead of
zero's). Pattern can be any user given pattern.

Thanks in advance.
-Karthick

Re: Repeat given pattern with buffer_compress_percentage

[Andrey Kuzmin]

On Wed, Dec 3, 2014 at 9:48 PM, Karthick Srinivasachary
<karthick@pernixdata.com> wrote:
> Hi Folks,
>
> I have question. Any feedback appreciated..!
>
> With buffer_compress_percentage, fio writes random data + zeros.
>
> Is there a way to write random data + repeat_pattern (instead of
> zero's). Pattern can be any user given pattern.

fio has a verification mode based on user-specified pattern (in
verify_pattern mode). May be you can merge this with compression, to
get you where you want to be.

HTH,
Andrey

>
> Thanks in advance.
> -Karthick
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Repeat given pattern with buffer_compress_percentage

[Karthick Srinivasachary]

Thanks. I tried that.
With verify_pattern='"pattern", pattern is not repeated matching
buffer_compression_percentage.

-Karthick

On Wed, Dec 3, 2014 at 11:45 AM, Andrey Kuzmin
<andrey.v.kuzmin@gmail.com> wrote:
> On Wed, Dec 3, 2014 at 9:48 PM, Karthick Srinivasachary
> <karthick@pernixdata.com> wrote:
>> Hi Folks,
>>
>> I have question. Any feedback appreciated..!
>>
>> With buffer_compress_percentage, fio writes random data + zeros.
>>
>> Is there a way to write random data + repeat_pattern (instead of
>> zero's). Pattern can be any user given pattern.
>
> fio has a verification mode based on user-specified pattern (in
> verify_pattern mode). May be you can merge this with compression, to
> get you where you want to be.
>
> HTH,
> Andrey
>
>>
>> Thanks in advance.
>> -Karthick
>> --
>> To unsubscribe from this list: send the line "unsubscribe fio" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Repeat given pattern with buffer_compress_percentage

[Andrey Kuzmin]

On Wed, Dec 3, 2014 at 10:49 PM, Karthick Srinivasachary
<karthick@pernixdata.com> wrote:
> Thanks. I tried that.
> With verify_pattern='"pattern", pattern is not repeated matching
> buffer_compression_percentage.

It won't work out-of-the-box as it serves a totally different purpose,
needs some extra code.

Regards,
Andrey
>
> -Karthick
>
> On Wed, Dec 3, 2014 at 11:45 AM, Andrey Kuzmin
> <andrey.v.kuzmin@gmail.com> wrote:
>> On Wed, Dec 3, 2014 at 9:48 PM, Karthick Srinivasachary
>> <karthick@pernixdata.com> wrote:
>>> Hi Folks,
>>>
>>> I have question. Any feedback appreciated..!
>>>
>>> With buffer_compress_percentage, fio writes random data + zeros.
>>>
>>> Is there a way to write random data + repeat_pattern (instead of
>>> zero's). Pattern can be any user given pattern.
>>
>> fio has a verification mode based on user-specified pattern (in
>> verify_pattern mode). May be you can merge this with compression, to
>> get you where you want to be.
>>
>> HTH,
>> Andrey
>>
>>>
>>> Thanks in advance.
>>> -Karthick
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe fio" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Repeat given pattern with buffer_compress_percentage

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

On 12/03/2014 11:48 AM, Karthick Srinivasachary wrote:
> Hi Folks,
>
> I have question. Any feedback appreciated..!
>
> With buffer_compress_percentage, fio writes random data + zeros.
>
> Is there a way to write random data + repeat_pattern (instead of
> zero's). Pattern can be any user given pattern.

We can just make it fill with buffer_pattern instead of zeroing. Given 
that the pattern is short enough, it should not skew the compression 
rate significantly.

OK, did a quick patch and ran a quick test, looks like it's still within 
half a percent.

Pull the latest -git, then apply the attached patch. That should do what 
you want. Please report back.

-- 
Jens Axboe


[-- Attachment #2: refill-compress-pattern.patch --]
[-- Type: text/x-patch, Size: 5087 bytes --]

diff --git a/init.c b/init.c
index 4f66759b59b5..82371044c1f5 100644
--- a/init.c
+++ b/init.c
@@ -724,7 +724,8 @@ static int fixup_options(struct thread_data *td)
 	if (td->o.compress_percentage == 100) {
 		td->o.zero_buffers = 1;
 		td->o.compress_percentage = 0;
-	}
+	} else if (td->o.compress_percentage)
+		td->o.refill_buffers = 1;
 
 	/*
 	 * Using a non-uniform random distribution excludes usage of
diff --git a/io_u.c b/io_u.c
index 33c82f2c4e9e..963e8cb764b9 100644
--- a/io_u.c
+++ b/io_u.c
@@ -1859,9 +1859,9 @@ static void save_buf_state(struct thread_data *td, struct frand_state *rs)
 void fill_io_buffer(struct thread_data *td, void *buf, unsigned int min_write,
 		    unsigned int max_bs)
 {
-	if (td->o.buffer_pattern_bytes)
-		fill_buffer_pattern(td, buf, max_bs);
-	else if (!td->o.zero_buffers) {
+	struct thread_options *o = &td->o;
+
+	if (o->compress_percentage && !o->zero_buffers) {
 		unsigned int perc = td->o.compress_percentage;
 		struct frand_state *rs;
 		unsigned int left = max_bs;
@@ -1879,7 +1879,8 @@ void fill_io_buffer(struct thread_data *td, void *buf, unsigned int min_write,
 					seg = min_write;
 
 				fill_random_buf_percentage(rs, buf, perc, seg,
-								min_write);
+					min_write, o->buffer_pattern,
+						   o->buffer_pattern_bytes);
 			} else
 				fill_random_buf(rs, buf, min_write);
 
@@ -1887,7 +1888,9 @@ void fill_io_buffer(struct thread_data *td, void *buf, unsigned int min_write,
 			left -= min_write;
 			save_buf_state(td, rs);
 		} while (left);
-	} else
+	} else if (o->buffer_pattern_bytes && !o->zero_buffers)
+		fill_buffer_pattern(td, buf, max_bs);
+	else
 		memset(buf, 0, max_bs);
 }
 
diff --git a/lib/rand.c b/lib/rand.c
index a79fb9c17c32..5235c7f5b3f2 100644
--- a/lib/rand.c
+++ b/lib/rand.c
@@ -34,6 +34,7 @@
 */
 
 #include <string.h>
+#include <assert.h>
 #include "rand.h"
 #include "../hash.h"
 
@@ -90,15 +91,45 @@ unsigned long fill_random_buf(struct frand_state *fs, void *buf,
 	return r;
 }
 
+void fill_pattern(void *p, unsigned int len, char *pattern,
+		  unsigned int pattern_bytes)
+{
+	switch (pattern_bytes) {
+	case 0:
+		assert(0);
+		break;
+	case 1:
+		memset(p, pattern[0], len);
+		break;
+	default: {
+		unsigned int i = 0, size = 0;
+		unsigned char *b = p;
+
+		while (i < len) {
+			size = pattern_bytes;
+			if (size > (len - i))
+				size = len - i;
+			memcpy(b+i, pattern, size);
+			i += size;
+		}
+		break;
+		}
+	}
+}
+
 unsigned long fill_random_buf_percentage(struct frand_state *fs, void *buf,
 					 unsigned int percentage,
-					 unsigned int segment, unsigned int len)
+					 unsigned int segment, unsigned int len,
+					 char *pattern, unsigned int pbytes)
 {
 	unsigned long r = __rand(fs);
 	unsigned int this_len;
 
 	if (percentage == 100) {
-		memset(buf, 0, len);
+		if (pattern)
+			fill_pattern(buf, len, pattern, pbytes);
+		else
+			memset(buf, 0, len);
 		return 0;
 	}
 
@@ -124,7 +155,10 @@ unsigned long fill_random_buf_percentage(struct frand_state *fs, void *buf,
 		if (this_len > len)
 			this_len = len;
 
-		memset(buf, 0, this_len);
+		if (pattern)
+			fill_pattern(buf, this_len, pattern, pbytes);
+		else
+			memset(buf, 0, this_len);
 		len -= this_len;
 		buf += this_len;
 	}
diff --git a/lib/rand.h b/lib/rand.h
index 8c35ab1fa263..803bea484757 100644
--- a/lib/rand.h
+++ b/lib/rand.h
@@ -30,6 +30,7 @@ extern void init_rand(struct frand_state *);
 extern void init_rand_seed(struct frand_state *, unsigned int seed);
 extern void __fill_random_buf(void *buf, unsigned int len, unsigned long seed);
 extern unsigned long fill_random_buf(struct frand_state *, void *buf, unsigned int len);
-extern unsigned long fill_random_buf_percentage(struct frand_state *, void *buf, unsigned int percentage, unsigned int segment, unsigned int len);
+extern unsigned long fill_random_buf_percentage(struct frand_state *, void *, unsigned int, unsigned int, unsigned int, char *, unsigned int);
+extern void fill_pattern(void *p, unsigned int len, char *pattern, unsigned int pattern_bytes);
 
 #endif
diff --git a/verify.c b/verify.c
index c2b3c40edef8..c1791fc69743 100644
--- a/verify.c
+++ b/verify.c
@@ -29,36 +29,6 @@ static void populate_hdr(struct thread_data *td, struct io_u *io_u,
 			 struct verify_header *hdr, unsigned int header_num,
 			 unsigned int header_len);
 
-static void fill_pattern(void *p, unsigned int len, char *pattern,
-			 unsigned int pattern_bytes)
-{
-	switch (pattern_bytes) {
-	case 0:
-		assert(0);
-		break;
-	case 1:
-		dprint(FD_VERIFY, "fill verify pattern b=0 len=%u\n", len);
-		memset(p, pattern[0], len);
-		break;
-	default: {
-		unsigned int i = 0, size = 0;
-		unsigned char *b = p;
-
-		dprint(FD_VERIFY, "fill verify pattern b=%d len=%u\n",
-					pattern_bytes, len);
-
-		while (i < len) {
-			size = pattern_bytes;
-			if (size > (len - i))
-				size = len - i;
-			memcpy(b+i, pattern, size);
-			i += size;
-		}
-		break;
-		}
-	}
-}
-
 void fill_buffer_pattern(struct thread_data *td, void *p, unsigned int len)
 {
 	fill_pattern(p, len, td->o.buffer_pattern, td->o.buffer_pattern_bytes);

Re: Repeat given pattern with buffer_compress_percentage

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

On 12/03/2014 07:42 PM, Jens Axboe wrote:
> On 12/03/2014 11:48 AM, Karthick Srinivasachary wrote:
>> Hi Folks,
>>
>> I have question. Any feedback appreciated..!
>>
>> With buffer_compress_percentage, fio writes random data + zeros.
>>
>> Is there a way to write random data + repeat_pattern (instead of
>> zero's). Pattern can be any user given pattern.
>
> We can just make it fill with buffer_pattern instead of zeroing. Given
> that the pattern is short enough, it should not skew the compression
> rate significantly.
>
> OK, did a quick patch and ran a quick test, looks like it's still within
> half a percent.
>
> Pull the latest -git, then apply the attached patch. That should do what
> you want. Please report back.

Use this -v2 instead, it fixes a crash if you _don't_ set buffer_pattern...

-- 
Jens Axboe


[-- Attachment #2: refill-compress-pattern-v2.patch --]
[-- Type: text/x-patch, Size: 4986 bytes --]

diff --git a/io_u.c b/io_u.c
index 33c82f2c4e9e..963e8cb764b9 100644
--- a/io_u.c
+++ b/io_u.c
@@ -1859,9 +1859,9 @@ static void save_buf_state(struct thread_data *td, struct frand_state *rs)
 void fill_io_buffer(struct thread_data *td, void *buf, unsigned int min_write,
 		    unsigned int max_bs)
 {
-	if (td->o.buffer_pattern_bytes)
-		fill_buffer_pattern(td, buf, max_bs);
-	else if (!td->o.zero_buffers) {
+	struct thread_options *o = &td->o;
+
+	if (o->compress_percentage && !o->zero_buffers) {
 		unsigned int perc = td->o.compress_percentage;
 		struct frand_state *rs;
 		unsigned int left = max_bs;
@@ -1879,7 +1879,8 @@ void fill_io_buffer(struct thread_data *td, void *buf, unsigned int min_write,
 					seg = min_write;
 
 				fill_random_buf_percentage(rs, buf, perc, seg,
-								min_write);
+					min_write, o->buffer_pattern,
+						   o->buffer_pattern_bytes);
 			} else
 				fill_random_buf(rs, buf, min_write);
 
@@ -1887,7 +1888,9 @@ void fill_io_buffer(struct thread_data *td, void *buf, unsigned int min_write,
 			left -= min_write;
 			save_buf_state(td, rs);
 		} while (left);
-	} else
+	} else if (o->buffer_pattern_bytes && !o->zero_buffers)
+		fill_buffer_pattern(td, buf, max_bs);
+	else
 		memset(buf, 0, max_bs);
 }
 
diff --git a/lib/rand.c b/lib/rand.c
index a79fb9c17c32..e5332bfe7924 100644
--- a/lib/rand.c
+++ b/lib/rand.c
@@ -34,6 +34,7 @@
 */
 
 #include <string.h>
+#include <assert.h>
 #include "rand.h"
 #include "../hash.h"
 
@@ -90,15 +91,45 @@ unsigned long fill_random_buf(struct frand_state *fs, void *buf,
 	return r;
 }
 
+void fill_pattern(void *p, unsigned int len, char *pattern,
+		  unsigned int pattern_bytes)
+{
+	switch (pattern_bytes) {
+	case 0:
+		assert(0);
+		break;
+	case 1:
+		memset(p, pattern[0], len);
+		break;
+	default: {
+		unsigned int i = 0, size = 0;
+		unsigned char *b = p;
+
+		while (i < len) {
+			size = pattern_bytes;
+			if (size > (len - i))
+				size = len - i;
+			memcpy(b+i, pattern, size);
+			i += size;
+		}
+		break;
+		}
+	}
+}
+
 unsigned long fill_random_buf_percentage(struct frand_state *fs, void *buf,
 					 unsigned int percentage,
-					 unsigned int segment, unsigned int len)
+					 unsigned int segment, unsigned int len,
+					 char *pattern, unsigned int pbytes)
 {
 	unsigned long r = __rand(fs);
 	unsigned int this_len;
 
 	if (percentage == 100) {
-		memset(buf, 0, len);
+		if (pbytes)
+			fill_pattern(buf, len, pattern, pbytes);
+		else
+			memset(buf, 0, len);
 		return 0;
 	}
 
@@ -124,7 +155,10 @@ unsigned long fill_random_buf_percentage(struct frand_state *fs, void *buf,
 		if (this_len > len)
 			this_len = len;
 
-		memset(buf, 0, this_len);
+		if (pbytes)
+			fill_pattern(buf, this_len, pattern, pbytes);
+		else
+			memset(buf, 0, this_len);
 		len -= this_len;
 		buf += this_len;
 	}
diff --git a/lib/rand.h b/lib/rand.h
index 8c35ab1fa263..803bea484757 100644
--- a/lib/rand.h
+++ b/lib/rand.h
@@ -30,6 +30,7 @@ extern void init_rand(struct frand_state *);
 extern void init_rand_seed(struct frand_state *, unsigned int seed);
 extern void __fill_random_buf(void *buf, unsigned int len, unsigned long seed);
 extern unsigned long fill_random_buf(struct frand_state *, void *buf, unsigned int len);
-extern unsigned long fill_random_buf_percentage(struct frand_state *, void *buf, unsigned int percentage, unsigned int segment, unsigned int len);
+extern unsigned long fill_random_buf_percentage(struct frand_state *, void *, unsigned int, unsigned int, unsigned int, char *, unsigned int);
+extern void fill_pattern(void *p, unsigned int len, char *pattern, unsigned int pattern_bytes);
 
 #endif
diff --git a/options.c b/options.c
index 23469d8d793c..75cfe6df4b06 100644
--- a/options.c
+++ b/options.c
@@ -1048,6 +1048,7 @@ static int str_buffer_compress_cb(void *data, unsigned long long *il)
 
 	td->flags |= TD_F_COMPRESS;
 	td->o.compress_percentage = *il;
+	td->o.refill_buffers = 1;
 	return 0;
 }
 
diff --git a/verify.c b/verify.c
index c2b3c40edef8..c1791fc69743 100644
--- a/verify.c
+++ b/verify.c
@@ -29,36 +29,6 @@ static void populate_hdr(struct thread_data *td, struct io_u *io_u,
 			 struct verify_header *hdr, unsigned int header_num,
 			 unsigned int header_len);
 
-static void fill_pattern(void *p, unsigned int len, char *pattern,
-			 unsigned int pattern_bytes)
-{
-	switch (pattern_bytes) {
-	case 0:
-		assert(0);
-		break;
-	case 1:
-		dprint(FD_VERIFY, "fill verify pattern b=0 len=%u\n", len);
-		memset(p, pattern[0], len);
-		break;
-	default: {
-		unsigned int i = 0, size = 0;
-		unsigned char *b = p;
-
-		dprint(FD_VERIFY, "fill verify pattern b=%d len=%u\n",
-					pattern_bytes, len);
-
-		while (i < len) {
-			size = pattern_bytes;
-			if (size > (len - i))
-				size = len - i;
-			memcpy(b+i, pattern, size);
-			i += size;
-		}
-		break;
-		}
-	}
-}
-
 void fill_buffer_pattern(struct thread_data *td, void *p, unsigned int len)
 {
 	fill_pattern(p, len, td->o.buffer_pattern, td->o.buffer_pattern_bytes);

Re: Repeat given pattern with buffer_compress_percentage

[Jens Axboe]

On 12/03/2014 07:49 PM, Jens Axboe wrote:
> On 12/03/2014 07:42 PM, Jens Axboe wrote:
>> On 12/03/2014 11:48 AM, Karthick Srinivasachary wrote:
>>> Hi Folks,
>>>
>>> I have question. Any feedback appreciated..!
>>>
>>> With buffer_compress_percentage, fio writes random data + zeros.
>>>
>>> Is there a way to write random data + repeat_pattern (instead of
>>> zero's). Pattern can be any user given pattern.
>>
>> We can just make it fill with buffer_pattern instead of zeroing. Given
>> that the pattern is short enough, it should not skew the compression
>> rate significantly.
>>
>> OK, did a quick patch and ran a quick test, looks like it's still within
>> half a percent.
>>
>> Pull the latest -git, then apply the attached patch. That should do what
>> you want. Please report back.
>
> Use this -v2 instead, it fixes a crash if you _don't_ set buffer_pattern...

Updated documentation, everything is now in -git. So forget the patch, 
just git pull and try it out.

Note the distinction on using refill_buffers=1 or not. If you don't set 
this, then each block will be compressible to the extent specified. But 
buffers are recycled, so whole device/file ratios might look different. 
If you want the compression percentage to apply across all blocks 
written, then you set refill_buffers=1 as well.

-- 
Jens Axboe

Re: Repeat given pattern with buffer_compress_percentage

[Karthick Srinivasachary]

Hi Jens,

Thanks for your response and quick fix.. Much appreciated.
Tried few things and saw couple issues, summarizing below. Please let
me know if I am missing something.

1. Looks like buffer_compress_percentage with verify option is not
giving expected compression percentage

fio --name=global --rw=write --buffer_compress_percentage=50
--name=job1 --filename=/root/test1 --size=100m --refill_buffers
--verify=md5

lz4 could compress fio data file

lz4 /root/test1
Compressed 104857600 bytes into 104857715 bytes ==> 100.00%

Fio writes give correct compression result without verify option (with
refill_buffers)

2. buffer_compress_percentage with buffer_pattern + refill_buffers is
not giving expected compression percentage (Data is highly
compressible).

fio --name=global --rw=write --buffer_compress_percentage=50
--name=job1 --filename=/root/test1 --size=100m --refill_buffers
--buffer_pattern=111

root@ubuntu-base:~/fio-master# lz4 /root/test1
Compressed filename will be : /root/test1.lz4
Compressed 104857600 bytes into 463065 bytes ==> 0.44%

Regards,
Karthick


On Wed, Dec 3, 2014 at 6:58 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 12/03/2014 07:49 PM, Jens Axboe wrote:
>>
>> On 12/03/2014 07:42 PM, Jens Axboe wrote:
>>>
>>> On 12/03/2014 11:48 AM, Karthick Srinivasachary wrote:
>>>>
>>>> Hi Folks,
>>>>
>>>> I have question. Any feedback appreciated..!
>>>>
>>>> With buffer_compress_percentage, fio writes random data + zeros.
>>>>
>>>> Is there a way to write random data + repeat_pattern (instead of
>>>> zero's). Pattern can be any user given pattern.
>>>
>>>
>>> We can just make it fill with buffer_pattern instead of zeroing. Given
>>> that the pattern is short enough, it should not skew the compression
>>> rate significantly.
>>>
>>> OK, did a quick patch and ran a quick test, looks like it's still within
>>> half a percent.
>>>
>>> Pull the latest -git, then apply the attached patch. That should do what
>>> you want. Please report back.
>>
>>
>> Use this -v2 instead, it fixes a crash if you _don't_ set
>> buffer_pattern...
>
>
> Updated documentation, everything is now in -git. So forget the patch, just
> git pull and try it out.
>
> Note the distinction on using refill_buffers=1 or not. If you don't set
> this, then each block will be compressible to the extent specified. But
> buffers are recycled, so whole device/file ratios might look different. If
> you want the compression percentage to apply across all blocks written, then
> you set refill_buffers=1 as well.
>
> --
> Jens Axboe
>

Re: Repeat given pattern with buffer_compress_percentage

[Jens Axboe]

On 12/04/2014 12:53 AM, Karthick Srinivasachary wrote:
> Hi Jens,
>
> Thanks for your response and quick fix.. Much appreciated.
> Tried few things and saw couple issues, summarizing below. Please let
> me know if I am missing something.
>
> 1. Looks like buffer_compress_percentage with verify option is not
> giving expected compression percentage
>
> fio --name=global --rw=write --buffer_compress_percentage=50
> --name=job1 --filename=/root/test1 --size=100m --refill_buffers
> --verify=md5
>
> lz4 could compress fio data file
>
> lz4 /root/test1
> Compressed 104857600 bytes into 104857715 bytes ==> 100.00%

The verify buffers are filled differently. It will take a bit more code 
juggling to make that honor the buffer_pattern as a fill pattern. I'll 
take a look at it.

> 2. buffer_compress_percentage with buffer_pattern + refill_buffers is
> not giving expected compression percentage (Data is highly
> compressible).
>
> fio --name=global --rw=write --buffer_compress_percentage=50
> --name=job1 --filename=/root/test1 --size=100m --refill_buffers
> --buffer_pattern=111
>
> root@ubuntu-base:~/fio-master# lz4 /root/test1
> Compressed filename will be : /root/test1.lz4
> Compressed 104857600 bytes into 463065 bytes ==> 0.44%

That's a bug in how we clear refill_buffers when we set a buffer 
pattern, that's not correct anymore with the combination of buffer 
compression and pattern. Fixed and pushed.

-- 
Jens Axboe

Re: Repeat given pattern with buffer_compress_percentage

[Jens Axboe]

On 12/04/2014 08:27 AM, Jens Axboe wrote:
> On 12/04/2014 12:53 AM, Karthick Srinivasachary wrote:
>> Hi Jens,
>>
>> Thanks for your response and quick fix.. Much appreciated.
>> Tried few things and saw couple issues, summarizing below. Please let
>> me know if I am missing something.
>>
>> 1. Looks like buffer_compress_percentage with verify option is not
>> giving expected compression percentage
>>
>> fio --name=global --rw=write --buffer_compress_percentage=50
>> --name=job1 --filename=/root/test1 --size=100m --refill_buffers
>> --verify=md5
>>
>> lz4 could compress fio data file
>>
>> lz4 /root/test1
>> Compressed 104857600 bytes into 104857715 bytes ==> 100.00%
>
> The verify buffers are filled differently. It will take a bit more code
> juggling to make that honor the buffer_pattern as a fill pattern. I'll
> take a look at it.

Try current -git, should do the right thing for this case now too. And 
it works for the buffer content regeneration, when corruption is 
detected and verify_dump=1 is set.

-- 
Jens Axboe

Re: Repeat given pattern with buffer_compress_percentage

[Karthick Srinivasachary]

Hi Jens,

Awesome.. Thanks for the fix..!
With compress_percentage 80 or more, fio verify is failing. I don't
think there any issue with storage in my test environment. Do you
think we have a bug here. Complete command and output below.

/usr/local/bin/fio --name=global --ioengine=libaio --bsrange=16k-64k
--rw=randrw --iodepth=16 --direct=1 --verify=md5 --name=job0
--size=100m --filename=/fio/fio.dat --rw=randrw --rate=20m
--buffer_compress_percentage=80 --refill_buffers --buffer_pattern=123
--verify_dump=1

job0: (g=0): rw=randrw, bs=16K-64K/16K-64K/16K-64K, ioengine=libaio, iodepth=16
fio-2.1.14
Starting 1 process
verify: bad magic header a498, wanted acca at file /fio/fio.dat offset
3751936, length 16384ps] [eta 00m:01s]
fio: pid=2310, err=84/file:io_u.c:1803, func=io_u_queued_complete,
error=Invalid or incomplete multibyte or wide character

job0: (groupid=0, jobs=1): err=84 (file:io_u.c:1803,
func=io_u_queued_complete, error=Invalid or incomplete multibyte or
wide character): pid=2310: Thu Dec  4 23:21:24 2014
  read : io=52768KB, bw=21485KB/s, iops=654, runt=  2456msec
    slat (usec): min=13, max=16556, avg=397.53, stdev=1965.21
    clat (usec): min=807, max=10720, avg=3090.70, stdev=1278.73
     lat (usec): min=822, max=18879, avg=3489.21, stdev=2323.61
    clat percentiles (usec):
     |  1.00th=[ 1272],  5.00th=[ 1624], 10.00th=[ 1784], 20.00th=[ 2064],
     | 30.00th=[ 2288], 40.00th=[ 2512], 50.00th=[ 2800], 60.00th=[ 3152],
     | 70.00th=[ 3536], 80.00th=[ 4048], 90.00th=[ 4704], 95.00th=[ 5344],
     | 99.00th=[ 7392], 99.50th=[ 8384], 99.90th=[ 9536], 99.95th=[10688],
     | 99.99th=[10688]
    bw (KB  /s): min=20336, max=20963, per=95.66%, avg=20552.75, stdev=282.50
  write: io=51920KB, bw=21270KB/s, iops=648, runt=  2441msec
    slat (usec): min=18, max=259, avg=31.17, stdev=16.34
    clat (usec): min=827, max=6207, avg=2740.55, stdev=998.80
     lat (usec): min=858, max=6243, avg=2772.83, stdev=998.17
    clat percentiles (usec):
     |  1.00th=[ 1144],  5.00th=[ 1432], 10.00th=[ 1608], 20.00th=[ 1864],
     | 30.00th=[ 2064], 40.00th=[ 2288], 50.00th=[ 2544], 60.00th=[ 2832],
     | 70.00th=[ 3216], 80.00th=[ 3664], 90.00th=[ 4192], 95.00th=[ 4576],
     | 99.00th=[ 5408], 99.50th=[ 5856], 99.90th=[ 6176], 99.95th=[ 6176],
     | 99.99th=[ 6176]
    bw (KB  /s): min=18843, max=22700, per=99.54%, avg=21170.50, stdev=1853.97
    lat (usec) : 1000=0.22%
    lat (msec) : 2=21.29%, 4=61.79%, 10=16.68%, 20=0.03%
  cpu          : usr=10.42%, sys=2.77%, ctx=792, majf=0, minf=120
  IO depths    : 1=1.3%, 2=2.5%, 4=5.0%, 8=10.0%, 16=81.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=1608/w=1582/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: io=52768KB, aggrb=21485KB/s, minb=21485KB/s, maxb=21485KB/s,
mint=2456msec, maxt=2456msec
  WRITE: io=51920KB, aggrb=21269KB/s, minb=21269KB/s, maxb=21269KB/s,
mint=2441msec, maxt=2441msec

Disk stats (read/write):
    dm-0: ios=1521/1553, merge=0/0, ticks=4392/3884, in_queue=8276,
util=26.68%, aggrios=1608/1588, aggrmerge=0/6, aggrticks=4640/3944,
aggrin_queue=8584, aggrutil=27.16%
  sda: ios=1608/1588, merge=0/6, ticks=4640/3944, in_queue=8584, util=27.16%

Regards,
Karthick

On Thu, Dec 4, 2014 at 2:41 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 12/04/2014 08:27 AM, Jens Axboe wrote:
>>
>> On 12/04/2014 12:53 AM, Karthick Srinivasachary wrote:
>>>
>>> Hi Jens,
>>>
>>> Thanks for your response and quick fix.. Much appreciated.
>>> Tried few things and saw couple issues, summarizing below. Please let
>>> me know if I am missing something.
>>>
>>> 1. Looks like buffer_compress_percentage with verify option is not
>>> giving expected compression percentage
>>>
>>> fio --name=global --rw=write --buffer_compress_percentage=50
>>> --name=job1 --filename=/root/test1 --size=100m --refill_buffers
>>> --verify=md5
>>>
>>> lz4 could compress fio data file
>>>
>>> lz4 /root/test1
>>> Compressed 104857600 bytes into 104857715 bytes ==> 100.00%
>>
>>
>> The verify buffers are filled differently. It will take a bit more code
>> juggling to make that honor the buffer_pattern as a fill pattern. I'll
>> take a look at it.
>
>
> Try current -git, should do the right thing for this case now too. And it
> works for the buffer content regeneration, when corruption is detected and
> verify_dump=1 is set.
>
> --
> Jens Axboe
>

Re: Repeat given pattern with buffer_compress_percentage

[Jens Axboe]

On 12/05/2014 12:25 AM, Karthick Srinivasachary wrote:
> Hi Jens,
>
> Awesome.. Thanks for the fix..!
> With compress_percentage 80 or more, fio verify is failing. I don't
> think there any issue with storage in my test environment. Do you
> think we have a bug here. Complete command and output below.
>
> /usr/local/bin/fio --name=global --ioengine=libaio --bsrange=16k-64k
> --rw=randrw --iodepth=16 --direct=1 --verify=md5 --name=job0
> --size=100m --filename=/fio/fio.dat --rw=randrw --rate=20m
> --buffer_compress_percentage=80 --refill_buffers --buffer_pattern=123
> --verify_dump=1

You can't do a random read/write test, it'll fail on the reads that have 
not been populated. It might sometimes work if the file had been laid 
out prior with the right contents, but generally you cannot depend upon 
it. You want to do just --rw=randwrite and then the verify phase will 
take place after the writes are done, verifying those.

Try and delete fio.dat prior to running this job, and it should fail 
consistently with or without buffer_compress_percentage.

-- 
Jens Axboe

Re: Repeat given pattern with buffer_compress_percentage

[Jens Axboe]

On 12/05/2014 10:02 AM, Jens Axboe wrote:
> On 12/05/2014 12:25 AM, Karthick Srinivasachary wrote:
>> Hi Jens,
>>
>> Awesome.. Thanks for the fix..!
>> With compress_percentage 80 or more, fio verify is failing. I don't
>> think there any issue with storage in my test environment. Do you
>> think we have a bug here. Complete command and output below.
>>
>> /usr/local/bin/fio --name=global --ioengine=libaio --bsrange=16k-64k
>> --rw=randrw --iodepth=16 --direct=1 --verify=md5 --name=job0
>> --size=100m --filename=/fio/fio.dat --rw=randrw --rate=20m
>> --buffer_compress_percentage=80 --refill_buffers --buffer_pattern=123
>> --verify_dump=1
>
> You can't do a random read/write test, it'll fail on the reads that have
> not been populated. It might sometimes work if the file had been laid
> out prior with the right contents, but generally you cannot depend upon
> it. You want to do just --rw=randwrite and then the verify phase will
> take place after the writes are done, verifying those.
>
> Try and delete fio.dat prior to running this job, and it should fail
> consistently with or without buffer_compress_percentage.

I take it back, there is a bug there. Fio does skip verifying the reads 
we generate initially, and verifies the written data after the 
read/write phase. So it _should_ work, the fact that it doesn't is a 
bug. I'll take a look at it.

-- 
Jens Axboe

Re: Repeat given pattern with buffer_compress_percentage

[Jens Axboe]

On 12/05/2014 10:20 AM, Jens Axboe wrote:
> On 12/05/2014 10:02 AM, Jens Axboe wrote:
>> On 12/05/2014 12:25 AM, Karthick Srinivasachary wrote:
>>> Hi Jens,
>>>
>>> Awesome.. Thanks for the fix..!
>>> With compress_percentage 80 or more, fio verify is failing. I don't
>>> think there any issue with storage in my test environment. Do you
>>> think we have a bug here. Complete command and output below.
>>>
>>> /usr/local/bin/fio --name=global --ioengine=libaio --bsrange=16k-64k
>>> --rw=randrw --iodepth=16 --direct=1 --verify=md5 --name=job0
>>> --size=100m --filename=/fio/fio.dat --rw=randrw --rate=20m
>>> --buffer_compress_percentage=80 --refill_buffers --buffer_pattern=123
>>> --verify_dump=1
>>
>> You can't do a random read/write test, it'll fail on the reads that have
>> not been populated. It might sometimes work if the file had been laid
>> out prior with the right contents, but generally you cannot depend upon
>> it. You want to do just --rw=randwrite and then the verify phase will
>> take place after the writes are done, verifying those.
>>
>> Try and delete fio.dat prior to running this job, and it should fail
>> consistently with or without buffer_compress_percentage.
>
> I take it back, there is a bug there. Fio does skip verifying the reads
> we generate initially, and verifies the written data after the
> read/write phase. So it _should_ work, the fact that it doesn't is a
> bug. I'll take a look at it.

Fixed, pull and retry please.

-- 
Jens Axboe

Re: Repeat given pattern with buffer_compress_percentage

[Karthick Srinivasachary]

Great.. Works now..! Thank you for considering my request and
providing new feature in quick time. Appreciated..!

Regards,
Karthick

On Fri, Dec 5, 2014 at 9:46 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 12/05/2014 10:20 AM, Jens Axboe wrote:
>>
>> On 12/05/2014 10:02 AM, Jens Axboe wrote:
>>>
>>> On 12/05/2014 12:25 AM, Karthick Srinivasachary wrote:
>>>>
>>>> Hi Jens,
>>>>
>>>> Awesome.. Thanks for the fix..!
>>>> With compress_percentage 80 or more, fio verify is failing. I don't
>>>> think there any issue with storage in my test environment. Do you
>>>> think we have a bug here. Complete command and output below.
>>>>
>>>> /usr/local/bin/fio --name=global --ioengine=libaio --bsrange=16k-64k
>>>> --rw=randrw --iodepth=16 --direct=1 --verify=md5 --name=job0
>>>> --size=100m --filename=/fio/fio.dat --rw=randrw --rate=20m
>>>> --buffer_compress_percentage=80 --refill_buffers --buffer_pattern=123
>>>> --verify_dump=1
>>>
>>>
>>> You can't do a random read/write test, it'll fail on the reads that have
>>> not been populated. It might sometimes work if the file had been laid
>>> out prior with the right contents, but generally you cannot depend upon
>>> it. You want to do just --rw=randwrite and then the verify phase will
>>> take place after the writes are done, verifying those.
>>>
>>> Try and delete fio.dat prior to running this job, and it should fail
>>> consistently with or without buffer_compress_percentage.
>>
>>
>> I take it back, there is a bug there. Fio does skip verifying the reads
>> we generate initially, and verifies the written data after the
>> read/write phase. So it _should_ work, the fact that it doesn't is a
>> bug. I'll take a look at it.
>
>
> Fixed, pull and retry please.
>
> --
> Jens Axboe
>