[RFC] [PATCH] Allow verification of random overwrites w/ba < bs

[RFC] [PATCH] Allow verification of random overwrites w/ba < bs

[Justin Eno (jeno)]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

Hi Jens,

This is a two patch set.

blockalign.patch:
This patch modifies the behavior of write logging such that *any* overlap
between a newly-submitted io_piece and elements in the hist_tree results
in the stale element(s) being evicted and replaced by the new io_piece.
The current implementation performs replacement only for totally overlapping
(i.e., aligned) io_pieces.  This restriction causes verification failures
for random overwrite workloads if blockalign is smaller than blocksize, and
the patch allows such workloads to succeed.

Note that this changes verification semantics for the specified workload.
Instead of verifying each written byte, verification for this case covers
only fully-intact blocks, i.e., remnants of partially-overwritten blocks
are left unverified.

bsrange.patch:
This takes advantage of the behavior introduced by blockalign.patch to
unlock verification of random overwrite workloads with multiple blocksizes.
The caveat about verification applies here as well, so I've left the warning
in init.c.

Thanks,
Justin

[-- Attachment #2: blockalign.patch --]
[-- Type: application/octet-stream, Size: 1576 bytes --]

From 17a236040f841665ca29724a5b65266f642ecab8 Mon Sep 17 00:00:00 2001
From: Justin Eno <jeno@micron.com>
Date: Thu, 29 Jan 2015 12:59:22 -0800
Subject: [PATCH] Allow verification of random overwrites w/ba < bs

When blockalign is less than blocksize, random overwrite
workloads may partially overwrite blocks.  This change
evicts partially-overwritten blocks from the iolog tree
so subsequent verification targets only intact blocks.

This change also allows verification with norandommap
and bsrange, though that is left disabled.
Signed-off-by: Justin Eno <jeno@micron.com>
---
 iolog.c |   14 +++++++++++---
 1 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/iolog.c b/iolog.c
index 99f8bc1..b867583 100644
--- a/iolog.c
+++ b/iolog.c
@@ -250,6 +250,7 @@ restart:
 	p = &td->io_hist_tree.rb_node;
 	parent = NULL;
 	while (*p) {
+		int overlap = 0;
 		parent = *p;
 
 		__ipo = rb_entry(parent, struct io_piece, rb_node);
@@ -257,11 +258,18 @@ restart:
 			p = &(*p)->rb_left;
 		else if (ipo->file > __ipo->file)
 			p = &(*p)->rb_right;
-		else if (ipo->offset < __ipo->offset)
+		else if (ipo->offset < __ipo->offset) {
 			p = &(*p)->rb_left;
-		else if (ipo->offset > __ipo->offset)
+			overlap = ipo->offset + ipo->len > __ipo->offset;
+		}
+		else if (ipo->offset > __ipo->offset) {
 			p = &(*p)->rb_right;
-		else {
+			overlap = __ipo->offset + __ipo->len > ipo->offset;
+		}
+		else
+			overlap = 1;
+
+		if (overlap) {
 			dprint(FD_IO, "iolog: overlap %llu/%lu, %llu/%lu",
 				__ipo->offset, __ipo->len,
 				ipo->offset, ipo->len);
-- 
1.7.1


[-- Attachment #3: bsrange.patch --]
[-- Type: application/octet-stream, Size: 1841 bytes --]

From 3c4675657ef285a51b36f24231bb94cf6412b2a7 Mon Sep 17 00:00:00 2001
From: Justin Eno <jeno@micron.com>
Date: Thu, 29 Jan 2015 14:28:38 -0800
Subject: [PATCH] Allow verify w/norandommap and bsrange

Only intact blocks are verified; partially-overwritten blocks
are ignored
Signed-off-by: Justin Eno <jeno@micron.com>
---
 HOWTO  |    8 ++++----
 init.c |    3 +--
 2 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/HOWTO b/HOWTO
index 8da5527..0f7909d 100644
--- a/HOWTO
+++ b/HOWTO
@@ -919,10 +919,10 @@ norandommap	Normally fio will cover every block of the file when doing
 		random IO. If this option is given, fio will just get a
 		new random offset without looking at past io history. This
 		means that some blocks may not be read or written, and that
-		some blocks may be read/written more than once. This option
-		is mutually exclusive with verify= if and only if multiple
-		blocksizes (via bsrange=) are used, since fio only tracks
-		complete rewrites of blocks.
+		some blocks may be read/written more than once. If this option
+		is used with verify= and multiple blocksizes (via bsrange=),
+		only intact blocks are verified, i.e., partially-overwritten
+		blocks are ignored.
 
 softrandommap=bool See norandommap. If fio runs with the random block map
 		enabled and it fails to allocate the map, if this option is
diff --git a/init.c b/init.c
index bb53a1e..c210ad2 100644
--- a/init.c
+++ b/init.c
@@ -596,8 +596,7 @@ static int fixup_options(struct thread_data *td)
 	if (o->norandommap && o->verify != VERIFY_NONE
 	    && !fixed_block_size(o))  {
 		log_err("fio: norandommap given for variable block sizes, "
-			"verify disabled\n");
-		o->verify = VERIFY_NONE;
+			"verify limited\n");
 		ret = warnings_fatal;
 	}
 	if (o->bs_unaligned && (o->odirect || td->io_ops->flags & FIO_RAWIO))
-- 
1.7.1

Re: [RFC] [PATCH] Allow verification of random overwrites w/ba < bs

[Jens Axboe]

On 02/17/2015 04:36 PM, Justin Eno (jeno) wrote:
> Hi Jens,
>
> This is a two patch set.
>
> blockalign.patch:
> This patch modifies the behavior of write logging such that *any* overlap
> between a newly-submitted io_piece and elements in the hist_tree results
> in the stale element(s) being evicted and replaced by the new io_piece.
> The current implementation performs replacement only for totally overlapping
> (i.e., aligned) io_pieces.  This restriction causes verification failures
> for random overwrite workloads if blockalign is smaller than blocksize, and
> the patch allows such workloads to succeed.
>
> Note that this changes verification semantics for the specified workload.
> Instead of verifying each written byte, verification for this case covers
> only fully-intact blocks, i.e., remnants of partially-overwritten blocks
> are left unverified.
>
> bsrange.patch:
> This takes advantage of the behavior introduced by blockalign.patch to
> unlock verification of random overwrite workloads with multiple blocksizes.
> The caveat about verification applies here as well, so I've left the warning
> in init.c.

Both of these look good, applied, thanks!

-- 
Jens Axboe