* Disabling verify when norandommap is given
@ 2009-02-18 23:12 Radha Ramachandran
2009-02-19 7:47 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Radha Ramachandran @ 2009-02-18 23:12 UTC (permalink / raw)
To: fio
Hi,
Is there a reason why there is a hard rule to disable verify when
norandommap option is also set?
Thanks
-radha
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disabling verify when norandommap is given
2009-02-18 23:12 Disabling verify when norandommap is given Radha Ramachandran
@ 2009-02-19 7:47 ` Jens Axboe
2009-02-19 18:10 ` Radha Ramachandran
0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2009-02-19 7:47 UTC (permalink / raw)
To: Radha Ramachandran; +Cc: fio
On Wed, Feb 18 2009, Radha Ramachandran wrote:
> Hi,
> Is there a reason why there is a hard rule to disable verify when
> norandommap option is also set?
Without a map of the blocks that have been written already, you might
overwrite full or partial blocks before you are done. That would cause
verify to fail. See the 'norandommap' explanation in the HOWTO as well.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disabling verify when norandommap is given
2009-02-19 7:47 ` Jens Axboe
@ 2009-02-19 18:10 ` Radha Ramachandran
2009-02-19 18:26 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Radha Ramachandran @ 2009-02-19 18:10 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
Yes, that makes sense for a randwrite operation.
But is it acceptable to allow norandommap with verify for
read/randomread only operations with no writes involved. The case Iam
specifically interested in is doing sequential writes followed by
sequential reads (with verify) and then randomaccess reads (with
verify). For a small enough block size on a large enough drive the
norandommap takes up too much memory.
Thanks
-radha
On Wed, Feb 18, 2009 at 11:47 PM, Jens Axboe <jens.axboe@oracle.com> wrote:
> On Wed, Feb 18 2009, Radha Ramachandran wrote:
>> Hi,
>> Is there a reason why there is a hard rule to disable verify when
>> norandommap option is also set?
>
> Without a map of the blocks that have been written already, you might
> overwrite full or partial blocks before you are done. That would cause
> verify to fail. See the 'norandommap' explanation in the HOWTO as well.
>
> --
> Jens Axboe
>
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disabling verify when norandommap is given
2009-02-19 18:10 ` Radha Ramachandran
@ 2009-02-19 18:26 ` Jens Axboe
2009-02-19 19:56 ` Radha Ramachandran
0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2009-02-19 18:26 UTC (permalink / raw)
To: Radha Ramachandran; +Cc: fio
On Thu, Feb 19 2009, Radha Ramachandran wrote:
> Yes, that makes sense for a randwrite operation.
> But is it acceptable to allow norandommap with verify for
> read/randomread only operations with no writes involved. The case Iam
> specifically interested in is doing sequential writes followed by
> sequential reads (with verify) and then randomaccess reads (with
> verify). For a small enough block size on a large enough drive the
> norandommap takes up too much memory.
The memory usage is indeed problematic, that is why the option exists.
I'd be very happy to slim that down...
What you suggest will work, but only for fixed size blocks. Once you
have varying block sizes, a given random minimum block offset may be
inside a verify block, not at the start of it.
So I guess the 'norandommap disables verify' can be relaxed to
'norandommap disables verify IFF non-fixed block sizes are used'. I'm
assuming that would work for you? Please try the below patch. If
succesful, I'll augment it with an updated HOWTO description.
diff --git a/init.c b/init.c
index 95c282a..001e5c4 100644
--- a/init.c
+++ b/init.c
@@ -206,6 +206,13 @@ static int setup_rate(struct thread_data *td)
return 0;
}
+static int fixed_block_size(struct thread_options *o)
+{
+ return o->min_bs[DDIR_READ] == o->max_bs[DDIR_READ] &&
+ o->min_bs[DDIR_WRITE] == o->max_bs[DDIR_WRITE] &&
+ o->min_bs[DDIR_READ] == o->min_bs[DDIR_WRITE];
+}
+
/*
* Lazy way of fixing up options that depend on each other. We could also
* define option callback handlers, but this is easier.
@@ -269,8 +276,10 @@ static int fixup_options(struct thread_data *td)
if (!o->file_size_high)
o->file_size_high = o->file_size_low;
- if (o->norandommap && o->verify != VERIFY_NONE) {
- log_err("fio: norandommap given, verify disabled\n");
+ if (o->norandommap && o->verify != VERIFY_NONE
+ && !fixed_block_size(o)) {
+ log_err("fio: norandommap given for variable block sizes, "
+ "verify disabled\n");
o->verify = VERIFY_NONE;
}
if (o->bs_unaligned && (o->odirect || td->io_ops->flags & FIO_RAWIO))
--
Jens Axboe
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: Disabling verify when norandommap is given
2009-02-19 18:26 ` Jens Axboe
@ 2009-02-19 19:56 ` Radha Ramachandran
2009-02-19 20:12 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Radha Ramachandran @ 2009-02-19 19:56 UTC (permalink / raw)
To: Jens Axboe; +Cc: fio
Yes, this works for me.
Thanks for looking into this.
-radha
On Thu, Feb 19, 2009 at 10:26 AM, Jens Axboe <jens.axboe@oracle.com> wrote:
> On Thu, Feb 19 2009, Radha Ramachandran wrote:
>> Yes, that makes sense for a randwrite operation.
>> But is it acceptable to allow norandommap with verify for
>> read/randomread only operations with no writes involved. The case Iam
>> specifically interested in is doing sequential writes followed by
>> sequential reads (with verify) and then randomaccess reads (with
>> verify). For a small enough block size on a large enough drive the
>> norandommap takes up too much memory.
>
> The memory usage is indeed problematic, that is why the option exists.
> I'd be very happy to slim that down...
>
> What you suggest will work, but only for fixed size blocks. Once you
> have varying block sizes, a given random minimum block offset may be
> inside a verify block, not at the start of it.
>
> So I guess the 'norandommap disables verify' can be relaxed to
> 'norandommap disables verify IFF non-fixed block sizes are used'. I'm
> assuming that would work for you? Please try the below patch. If
> succesful, I'll augment it with an updated HOWTO description.
>
> diff --git a/init.c b/init.c
> index 95c282a..001e5c4 100644
> --- a/init.c
> +++ b/init.c
> @@ -206,6 +206,13 @@ static int setup_rate(struct thread_data *td)
> return 0;
> }
>
> +static int fixed_block_size(struct thread_options *o)
> +{
> + return o->min_bs[DDIR_READ] == o->max_bs[DDIR_READ] &&
> + o->min_bs[DDIR_WRITE] == o->max_bs[DDIR_WRITE] &&
> + o->min_bs[DDIR_READ] == o->min_bs[DDIR_WRITE];
> +}
> +
> /*
> * Lazy way of fixing up options that depend on each other. We could also
> * define option callback handlers, but this is easier.
> @@ -269,8 +276,10 @@ static int fixup_options(struct thread_data *td)
> if (!o->file_size_high)
> o->file_size_high = o->file_size_low;
>
> - if (o->norandommap && o->verify != VERIFY_NONE) {
> - log_err("fio: norandommap given, verify disabled\n");
> + if (o->norandommap && o->verify != VERIFY_NONE
> + && !fixed_block_size(o)) {
> + log_err("fio: norandommap given for variable block sizes, "
> + "verify disabled\n");
> o->verify = VERIFY_NONE;
> }
> if (o->bs_unaligned && (o->odirect || td->io_ops->flags & FIO_RAWIO))
>
> --
> Jens Axboe
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disabling verify when norandommap is given
2009-02-19 19:56 ` Radha Ramachandran
@ 2009-02-19 20:12 ` Jens Axboe
0 siblings, 0 replies; 6+ messages in thread
From: Jens Axboe @ 2009-02-19 20:12 UTC (permalink / raw)
To: Radha Ramachandran; +Cc: fio
On Thu, Feb 19 2009, Radha Ramachandran wrote:
> Yes, this works for me.
> Thanks for looking into this.
Thanks for testing, I'll double check and test myself tomorrow and then
integrate.
> -radha
>
> On Thu, Feb 19, 2009 at 10:26 AM, Jens Axboe <jens.axboe@oracle.com> wrote:
> > On Thu, Feb 19 2009, Radha Ramachandran wrote:
> >> Yes, that makes sense for a randwrite operation.
> >> But is it acceptable to allow norandommap with verify for
> >> read/randomread only operations with no writes involved. The case Iam
> >> specifically interested in is doing sequential writes followed by
> >> sequential reads (with verify) and then randomaccess reads (with
> >> verify). For a small enough block size on a large enough drive the
> >> norandommap takes up too much memory.
> >
> > The memory usage is indeed problematic, that is why the option exists.
> > I'd be very happy to slim that down...
> >
> > What you suggest will work, but only for fixed size blocks. Once you
> > have varying block sizes, a given random minimum block offset may be
> > inside a verify block, not at the start of it.
> >
> > So I guess the 'norandommap disables verify' can be relaxed to
> > 'norandommap disables verify IFF non-fixed block sizes are used'. I'm
> > assuming that would work for you? Please try the below patch. If
> > succesful, I'll augment it with an updated HOWTO description.
> >
> > diff --git a/init.c b/init.c
> > index 95c282a..001e5c4 100644
> > --- a/init.c
> > +++ b/init.c
> > @@ -206,6 +206,13 @@ static int setup_rate(struct thread_data *td)
> > return 0;
> > }
> >
> > +static int fixed_block_size(struct thread_options *o)
> > +{
> > + return o->min_bs[DDIR_READ] == o->max_bs[DDIR_READ] &&
> > + o->min_bs[DDIR_WRITE] == o->max_bs[DDIR_WRITE] &&
> > + o->min_bs[DDIR_READ] == o->min_bs[DDIR_WRITE];
> > +}
> > +
> > /*
> > * Lazy way of fixing up options that depend on each other. We could also
> > * define option callback handlers, but this is easier.
> > @@ -269,8 +276,10 @@ static int fixup_options(struct thread_data *td)
> > if (!o->file_size_high)
> > o->file_size_high = o->file_size_low;
> >
> > - if (o->norandommap && o->verify != VERIFY_NONE) {
> > - log_err("fio: norandommap given, verify disabled\n");
> > + if (o->norandommap && o->verify != VERIFY_NONE
> > + && !fixed_block_size(o)) {
> > + log_err("fio: norandommap given for variable block sizes, "
> > + "verify disabled\n");
> > o->verify = VERIFY_NONE;
> > }
> > if (o->bs_unaligned && (o->odirect || td->io_ops->flags & FIO_RAWIO))
> >
> > --
> > Jens Axboe
> >
> >
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2009-02-19 20:15 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-02-18 23:12 Disabling verify when norandommap is given Radha Ramachandran
2009-02-19 7:47 ` Jens Axboe
2009-02-19 18:10 ` Radha Ramachandran
2009-02-19 18:26 ` Jens Axboe
2009-02-19 19:56 ` Radha Ramachandran
2009-02-19 20:12 ` Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox