dedupe and replay

dedupe and replay

[Alket Memushaj]

Hi,

I am playing with the dedupe option and while it works well under
synthetic tests, it doesn't seem to work when replaying a trace.

My test jobs are simple:

fio --name=replay --ioengine=libaio --direct=1 --iodepth=32
--read_iolog=/path/to/trace --dedupe_percentage=80

or

fio --name=synth --ioengine=libaio --direct=1 --iodepth=32
--dedupe_percentage=80 --rw=write


I understand that block sizes would be variable during a trace replay,
but is it possible to specify a base block size and calculate dedupe
based on that for blocks that are multiples of the base size?

Thanks,
Alket

Re: dedupe and replay

[Jens Axboe]

On 09/26/2014 12:30 PM, Alket Memushaj wrote:
> Hi,
> 
> I am playing with the dedupe option and while it works well under
> synthetic tests, it doesn't seem to work when replaying a trace.
> 
> My test jobs are simple:
> 
> fio --name=replay --ioengine=libaio --direct=1 --iodepth=32
> --read_iolog=/path/to/trace --dedupe_percentage=80

That should work, as far as I can tell, since we do the buffer fill
after the iolog retrieval has filled out an IO unit for us. Can you say
more about what doesn't appear to work?

> I understand that block sizes would be variable during a trace replay,
> but is it possible to specify a base block size and calculate dedupe
> based on that for blocks that are multiples of the base size?

Ah, that might be the key element here. Yes, the dedupe part might not
really work well for multiple write sizes. Nothing that inherently
prevents that from working, just didn't add that bit yet. Will do that.


-- 
Jens Axboe

Re: dedupe and replay

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]

On 09/26/2014 01:08 PM, Jens Axboe wrote:
> On 09/26/2014 12:30 PM, Alket Memushaj wrote:
>> Hi,
>>
>> I am playing with the dedupe option and while it works well under
>> synthetic tests, it doesn't seem to work when replaying a trace.
>>
>> My test jobs are simple:
>>
>> fio --name=replay --ioengine=libaio --direct=1 --iodepth=32
>> --read_iolog=/path/to/trace --dedupe_percentage=80
> 
> That should work, as far as I can tell, since we do the buffer fill
> after the iolog retrieval has filled out an IO unit for us. Can you say
> more about what doesn't appear to work?
> 
>> I understand that block sizes would be variable during a trace replay,
>> but is it possible to specify a base block size and calculate dedupe
>> based on that for blocks that are multiples of the base size?
> 
> Ah, that might be the key element here. Yes, the dedupe part might not
> really work well for multiple write sizes. Nothing that inherently
> prevents that from working, just didn't add that bit yet. Will do that.

Does it work better with this patch?

-- 
Jens Axboe


[-- Attachment #2: dedupe-multiple.patch --]
[-- Type: text/x-patch, Size: 1654 bytes --]

diff --git a/io_u.c b/io_u.c
index eac871bfe9d9..db0695817b94 100644
--- a/io_u.c
+++ b/io_u.c
@@ -1485,9 +1485,13 @@ struct io_u *get_io_u(struct thread_data *td)
 		f->last_pos = io_u->offset + io_u->buflen;
 
 		if (io_u->ddir == DDIR_WRITE) {
+			unsigned int min_write;
+
+			min_write = td->o.min_bs[DDIR_WRITE];
+
 			if (td->flags & TD_F_REFILL_BUFFERS) {
-				io_u_fill_buffer(td, io_u,
-					io_u->xfer_buflen, io_u->xfer_buflen);
+				io_u_fill_buffer(td, io_u, min_write,
+							io_u->xfer_buflen);
 			} else if ((td->flags & TD_F_SCRAMBLE_BUFFERS) &&
 				   !(td->flags & TD_F_COMPRESS))
 				do_scramble = 1;
@@ -1864,22 +1868,29 @@ void fill_io_buffer(struct thread_data *td, void *buf, unsigned int min_write,
 	else if (!td->o.zero_buffers) {
 		unsigned int perc = td->o.compress_percentage;
 		struct frand_state *rs;
+		unsigned int left = max_bs;
 
-		rs = get_buf_state(td);
+		do {
+			rs = get_buf_state(td);
 
-		if (perc) {
-			unsigned int seg = min_write;
+			min_write = min(min_write, left);
 
-			seg = min(min_write, td->o.compress_chunk);
-			if (!seg)
-				seg = min_write;
+			if (perc) {
+				unsigned int seg = min_write;
 
-			fill_random_buf_percentage(rs, buf, perc, seg,max_bs);
-			save_buf_state(td, rs);
-		} else {
-			fill_random_buf(rs, buf, max_bs);
+				seg = min(min_write, td->o.compress_chunk);
+				if (!seg)
+					seg = min_write;
+
+				fill_random_buf_percentage(rs, buf, perc, seg,
+								min_write);
+			} else
+				fill_random_buf(rs, buf, min_write);
+
+			buf += min_write;
+			left -= min_write;
 			save_buf_state(td, rs);
-		}
+		} while (left);
 	} else
 		memset(buf, 0, max_bs);
 }

Re: dedupe and replay

[Alket Memushaj]

thanks for the quick response - but unfortunately this patch does not
seem to have changed the result. I managed to get a 1.4:1 dedupe when
setting dedupe_percentage to 99%. The test rig I have is like this: an
array w/ inline dedupe @ 4K and I am the only user on it - I have a
single machine mounting a single lun from the array (the whole space
in the array is assigned to this lun). The synthetic test at
dedupe_percentage=80 results in a 4:1 dedupe ratio from the array. The
replay at 80% comes out as undedupable (ratio = 1.0:1) even with the
latest patch. The replay generates about 1.7 million I/Os. ~1 million
are 4K, 80 thousand are 8K, 70 thousand are 32K, etc... there's a
decent number of I/O that is not a multiple of 4K, but the vast
majority of the I/O is a multiple of 4K. The offset for each I/O is 4K
aligned in the vast majority of I/Os.

On Fri, Sep 26, 2014 at 8:38 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 09/26/2014 01:08 PM, Jens Axboe wrote:
>> On 09/26/2014 12:30 PM, Alket Memushaj wrote:
>>> Hi,
>>>
>>> I am playing with the dedupe option and while it works well under
>>> synthetic tests, it doesn't seem to work when replaying a trace.
>>>
>>> My test jobs are simple:
>>>
>>> fio --name=replay --ioengine=libaio --direct=1 --iodepth=32
>>> --read_iolog=/path/to/trace --dedupe_percentage=80
>>
>> That should work, as far as I can tell, since we do the buffer fill
>> after the iolog retrieval has filled out an IO unit for us. Can you say
>> more about what doesn't appear to work?
>>
>>> I understand that block sizes would be variable during a trace replay,
>>> but is it possible to specify a base block size and calculate dedupe
>>> based on that for blocks that are multiples of the base size?
>>
>> Ah, that might be the key element here. Yes, the dedupe part might not
>> really work well for multiple write sizes. Nothing that inherently
>> prevents that from working, just didn't add that bit yet. Will do that.
>
> Does it work better with this patch?
>
> --
> Jens Axboe
>

Re: dedupe and replay

[Jens Axboe]

On 09/27/2014 05:19 AM, Alket Memushaj wrote:
> thanks for the quick response - but unfortunately this patch does not
> seem to have changed the result. I managed to get a 1.4:1 dedupe when
> setting dedupe_percentage to 99%. The test rig I have is like this: an
> array w/ inline dedupe @ 4K and I am the only user on it - I have a
> single machine mounting a single lun from the array (the whole space
> in the array is assigned to this lun). The synthetic test at
> dedupe_percentage=80 results in a 4:1 dedupe ratio from the array. The
> replay at 80% comes out as undedupable (ratio = 1.0:1) even with the
> latest patch. The replay generates about 1.7 million I/Os. ~1 million
> are 4K, 80 thousand are 8K, 70 thousand are 32K, etc... there's a
> decent number of I/O that is not a multiple of 4K, but the vast
> majority of the I/O is a multiple of 4K. The offset for each I/O is 4K
> aligned in the vast majority of I/Os.

(please don't top post)

I'll try with a trace replay here. If you can, would be nice to get your
trace.


-- 
Jens Axboe

Re: dedupe and replay

[Jens Axboe]

On 2014-09-27 07:45, Jens Axboe wrote:
> On 09/27/2014 05:19 AM, Alket Memushaj wrote:
>> thanks for the quick response - but unfortunately this patch does not
>> seem to have changed the result. I managed to get a 1.4:1 dedupe when
>> setting dedupe_percentage to 99%. The test rig I have is like this: an
>> array w/ inline dedupe @ 4K and I am the only user on it - I have a
>> single machine mounting a single lun from the array (the whole space
>> in the array is assigned to this lun). The synthetic test at
>> dedupe_percentage=80 results in a 4:1 dedupe ratio from the array. The
>> replay at 80% comes out as undedupable (ratio = 1.0:1) even with the
>> latest patch. The replay generates about 1.7 million I/Os. ~1 million
>> are 4K, 80 thousand are 8K, 70 thousand are 32K, etc... there's a
>> decent number of I/O that is not a multiple of 4K, but the vast
>> majority of the I/O is a multiple of 4K. The offset for each I/O is 4K
>> aligned in the vast majority of I/Os.
>
> (please don't top post)
>
> I'll try with a trace replay here. If you can, would be nice to get your
> trace.

So ran a quick test here. The original file:

[dedupe]
filename=test
bssplit=4k/58:8k/5:16k/5:32k/5:64k/5:128k/5
rw=write
size=1g
dedupe_percentage=80
write_iolog=dedupe.log

Ran this and check how well 'test' dedupes:

axboe@nelson:/home/axboe/git/fio $ t/dedupe test
Will check <test>, size <1073741824>, using 8 threads
Threads(8): 262144 items processed
Extents=262144, Unique extents=51955
De-dupe factor: 5.05
Fio setting: dedupe_percentage=80

So far, so good, we get expected results (1:4, or 80%). The dedupe.log 
looks fine, lets replay that with:

[dedupe]
filename=test
rw=write
read_iolog=dedupe.log
size=1g
dedupe_percentage=80
write_iolog=dedupe.log

Ran this, and then check how well that dedupes:

axboe@nelson:/home/axboe/git/fio $ t/dedupe test
Will check <test>, size <1073741824>, using 8 threads
Threads(8): 262144 items processed
Extents=262144, Unique extents=51953
De-dupe factor: 5.05
Fio setting: dedupe_percentage=80

Pretty much identical. So it seems to work fine for me, generating 
appropriately random buffers.

I might need your trace to figure out what's going on for you.

-- 
Jens Axboe