Re: dedupe and replay

[Jens Axboe <axboe@kernel.dk>]

[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]

On 09/26/2014 01:08 PM, Jens Axboe wrote:
> On 09/26/2014 12:30 PM, Alket Memushaj wrote:
>> Hi,
>>
>> I am playing with the dedupe option and while it works well under
>> synthetic tests, it doesn't seem to work when replaying a trace.
>>
>> My test jobs are simple:
>>
>> fio --name=replay --ioengine=libaio --direct=1 --iodepth=32
>> --read_iolog=/path/to/trace --dedupe_percentage=80
> 
> That should work, as far as I can tell, since we do the buffer fill
> after the iolog retrieval has filled out an IO unit for us. Can you say
> more about what doesn't appear to work?
> 
>> I understand that block sizes would be variable during a trace replay,
>> but is it possible to specify a base block size and calculate dedupe
>> based on that for blocks that are multiples of the base size?
> 
> Ah, that might be the key element here. Yes, the dedupe part might not
> really work well for multiple write sizes. Nothing that inherently
> prevents that from working, just didn't add that bit yet. Will do that.

Does it work better with this patch?

-- 
Jens Axboe


[-- Attachment #2: dedupe-multiple.patch --]
[-- Type: text/x-patch, Size: 1654 bytes --]

diff --git a/io_u.c b/io_u.c
index eac871bfe9d9..db0695817b94 100644
--- a/io_u.c
+++ b/io_u.c
@@ -1485,9 +1485,13 @@ struct io_u *get_io_u(struct thread_data *td)
 		f->last_pos = io_u->offset + io_u->buflen;
 
 		if (io_u->ddir == DDIR_WRITE) {
+			unsigned int min_write;
+
+			min_write = td->o.min_bs[DDIR_WRITE];
+
 			if (td->flags & TD_F_REFILL_BUFFERS) {
-				io_u_fill_buffer(td, io_u,
-					io_u->xfer_buflen, io_u->xfer_buflen);
+				io_u_fill_buffer(td, io_u, min_write,
+							io_u->xfer_buflen);
 			} else if ((td->flags & TD_F_SCRAMBLE_BUFFERS) &&
 				   !(td->flags & TD_F_COMPRESS))
 				do_scramble = 1;
@@ -1864,22 +1868,29 @@ void fill_io_buffer(struct thread_data *td, void *buf, unsigned int min_write,
 	else if (!td->o.zero_buffers) {
 		unsigned int perc = td->o.compress_percentage;
 		struct frand_state *rs;
+		unsigned int left = max_bs;
 
-		rs = get_buf_state(td);
+		do {
+			rs = get_buf_state(td);
 
-		if (perc) {
-			unsigned int seg = min_write;
+			min_write = min(min_write, left);
 
-			seg = min(min_write, td->o.compress_chunk);
-			if (!seg)
-				seg = min_write;
+			if (perc) {
+				unsigned int seg = min_write;
 
-			fill_random_buf_percentage(rs, buf, perc, seg,max_bs);
-			save_buf_state(td, rs);
-		} else {
-			fill_random_buf(rs, buf, max_bs);
+				seg = min(min_write, td->o.compress_chunk);
+				if (!seg)
+					seg = min_write;
+
+				fill_random_buf_percentage(rs, buf, perc, seg,
+								min_write);
+			} else
+				fill_random_buf(rs, buf, min_write);
+
+			buf += min_write;
+			left -= min_write;
 			save_buf_state(td, rs);
-		}
+		} while (left);
 	} else
 		memset(buf, 0, max_bs);
 }