* Disabling verify when norandommap is given @ 2009-02-18 23:12 Radha Ramachandran 2009-02-19 7:47 ` Jens Axboe 0 siblings, 1 reply; 6+ messages in thread From: Radha Ramachandran @ 2009-02-18 23:12 UTC (permalink / raw) To: fio Hi, Is there a reason why there is a hard rule to disable verify when norandommap option is also set? Thanks -radha ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disabling verify when norandommap is given 2009-02-18 23:12 Disabling verify when norandommap is given Radha Ramachandran @ 2009-02-19 7:47 ` Jens Axboe 2009-02-19 18:10 ` Radha Ramachandran 0 siblings, 1 reply; 6+ messages in thread From: Jens Axboe @ 2009-02-19 7:47 UTC (permalink / raw) To: Radha Ramachandran; +Cc: fio On Wed, Feb 18 2009, Radha Ramachandran wrote: > Hi, > Is there a reason why there is a hard rule to disable verify when > norandommap option is also set? Without a map of the blocks that have been written already, you might overwrite full or partial blocks before you are done. That would cause verify to fail. See the 'norandommap' explanation in the HOWTO as well. -- Jens Axboe ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disabling verify when norandommap is given 2009-02-19 7:47 ` Jens Axboe @ 2009-02-19 18:10 ` Radha Ramachandran 2009-02-19 18:26 ` Jens Axboe 0 siblings, 1 reply; 6+ messages in thread From: Radha Ramachandran @ 2009-02-19 18:10 UTC (permalink / raw) To: Jens Axboe; +Cc: fio Yes, that makes sense for a randwrite operation. But is it acceptable to allow norandommap with verify for read/randomread only operations with no writes involved. The case Iam specifically interested in is doing sequential writes followed by sequential reads (with verify) and then randomaccess reads (with verify). For a small enough block size on a large enough drive the norandommap takes up too much memory. Thanks -radha On Wed, Feb 18, 2009 at 11:47 PM, Jens Axboe <jens.axboe@oracle.com> wrote: > On Wed, Feb 18 2009, Radha Ramachandran wrote: >> Hi, >> Is there a reason why there is a hard rule to disable verify when >> norandommap option is also set? > > Without a map of the blocks that have been written already, you might > overwrite full or partial blocks before you are done. That would cause > verify to fail. See the 'norandommap' explanation in the HOWTO as well. > > -- > Jens Axboe > > -- > To unsubscribe from this list: send the line "unsubscribe fio" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disabling verify when norandommap is given 2009-02-19 18:10 ` Radha Ramachandran @ 2009-02-19 18:26 ` Jens Axboe 2009-02-19 19:56 ` Radha Ramachandran 0 siblings, 1 reply; 6+ messages in thread From: Jens Axboe @ 2009-02-19 18:26 UTC (permalink / raw) To: Radha Ramachandran; +Cc: fio On Thu, Feb 19 2009, Radha Ramachandran wrote: > Yes, that makes sense for a randwrite operation. > But is it acceptable to allow norandommap with verify for > read/randomread only operations with no writes involved. The case Iam > specifically interested in is doing sequential writes followed by > sequential reads (with verify) and then randomaccess reads (with > verify). For a small enough block size on a large enough drive the > norandommap takes up too much memory. The memory usage is indeed problematic, that is why the option exists. I'd be very happy to slim that down... What you suggest will work, but only for fixed size blocks. Once you have varying block sizes, a given random minimum block offset may be inside a verify block, not at the start of it. So I guess the 'norandommap disables verify' can be relaxed to 'norandommap disables verify IFF non-fixed block sizes are used'. I'm assuming that would work for you? Please try the below patch. If succesful, I'll augment it with an updated HOWTO description. diff --git a/init.c b/init.c index 95c282a..001e5c4 100644 --- a/init.c +++ b/init.c @@ -206,6 +206,13 @@ static int setup_rate(struct thread_data *td) return 0; } +static int fixed_block_size(struct thread_options *o) +{ + return o->min_bs[DDIR_READ] == o->max_bs[DDIR_READ] && + o->min_bs[DDIR_WRITE] == o->max_bs[DDIR_WRITE] && + o->min_bs[DDIR_READ] == o->min_bs[DDIR_WRITE]; +} + /* * Lazy way of fixing up options that depend on each other. We could also * define option callback handlers, but this is easier. @@ -269,8 +276,10 @@ static int fixup_options(struct thread_data *td) if (!o->file_size_high) o->file_size_high = o->file_size_low; - if (o->norandommap && o->verify != VERIFY_NONE) { - log_err("fio: norandommap given, verify disabled\n"); + if (o->norandommap && o->verify != VERIFY_NONE + && !fixed_block_size(o)) { + log_err("fio: norandommap given for variable block sizes, " + "verify disabled\n"); o->verify = VERIFY_NONE; } if (o->bs_unaligned && (o->odirect || td->io_ops->flags & FIO_RAWIO)) -- Jens Axboe ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: Disabling verify when norandommap is given 2009-02-19 18:26 ` Jens Axboe @ 2009-02-19 19:56 ` Radha Ramachandran 2009-02-19 20:12 ` Jens Axboe 0 siblings, 1 reply; 6+ messages in thread From: Radha Ramachandran @ 2009-02-19 19:56 UTC (permalink / raw) To: Jens Axboe; +Cc: fio Yes, this works for me. Thanks for looking into this. -radha On Thu, Feb 19, 2009 at 10:26 AM, Jens Axboe <jens.axboe@oracle.com> wrote: > On Thu, Feb 19 2009, Radha Ramachandran wrote: >> Yes, that makes sense for a randwrite operation. >> But is it acceptable to allow norandommap with verify for >> read/randomread only operations with no writes involved. The case Iam >> specifically interested in is doing sequential writes followed by >> sequential reads (with verify) and then randomaccess reads (with >> verify). For a small enough block size on a large enough drive the >> norandommap takes up too much memory. > > The memory usage is indeed problematic, that is why the option exists. > I'd be very happy to slim that down... > > What you suggest will work, but only for fixed size blocks. Once you > have varying block sizes, a given random minimum block offset may be > inside a verify block, not at the start of it. > > So I guess the 'norandommap disables verify' can be relaxed to > 'norandommap disables verify IFF non-fixed block sizes are used'. I'm > assuming that would work for you? Please try the below patch. If > succesful, I'll augment it with an updated HOWTO description. > > diff --git a/init.c b/init.c > index 95c282a..001e5c4 100644 > --- a/init.c > +++ b/init.c > @@ -206,6 +206,13 @@ static int setup_rate(struct thread_data *td) > return 0; > } > > +static int fixed_block_size(struct thread_options *o) > +{ > + return o->min_bs[DDIR_READ] == o->max_bs[DDIR_READ] && > + o->min_bs[DDIR_WRITE] == o->max_bs[DDIR_WRITE] && > + o->min_bs[DDIR_READ] == o->min_bs[DDIR_WRITE]; > +} > + > /* > * Lazy way of fixing up options that depend on each other. We could also > * define option callback handlers, but this is easier. > @@ -269,8 +276,10 @@ static int fixup_options(struct thread_data *td) > if (!o->file_size_high) > o->file_size_high = o->file_size_low; > > - if (o->norandommap && o->verify != VERIFY_NONE) { > - log_err("fio: norandommap given, verify disabled\n"); > + if (o->norandommap && o->verify != VERIFY_NONE > + && !fixed_block_size(o)) { > + log_err("fio: norandommap given for variable block sizes, " > + "verify disabled\n"); > o->verify = VERIFY_NONE; > } > if (o->bs_unaligned && (o->odirect || td->io_ops->flags & FIO_RAWIO)) > > -- > Jens Axboe > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disabling verify when norandommap is given 2009-02-19 19:56 ` Radha Ramachandran @ 2009-02-19 20:12 ` Jens Axboe 0 siblings, 0 replies; 6+ messages in thread From: Jens Axboe @ 2009-02-19 20:12 UTC (permalink / raw) To: Radha Ramachandran; +Cc: fio On Thu, Feb 19 2009, Radha Ramachandran wrote: > Yes, this works for me. > Thanks for looking into this. Thanks for testing, I'll double check and test myself tomorrow and then integrate. > -radha > > On Thu, Feb 19, 2009 at 10:26 AM, Jens Axboe <jens.axboe@oracle.com> wrote: > > On Thu, Feb 19 2009, Radha Ramachandran wrote: > >> Yes, that makes sense for a randwrite operation. > >> But is it acceptable to allow norandommap with verify for > >> read/randomread only operations with no writes involved. The case Iam > >> specifically interested in is doing sequential writes followed by > >> sequential reads (with verify) and then randomaccess reads (with > >> verify). For a small enough block size on a large enough drive the > >> norandommap takes up too much memory. > > > > The memory usage is indeed problematic, that is why the option exists. > > I'd be very happy to slim that down... > > > > What you suggest will work, but only for fixed size blocks. Once you > > have varying block sizes, a given random minimum block offset may be > > inside a verify block, not at the start of it. > > > > So I guess the 'norandommap disables verify' can be relaxed to > > 'norandommap disables verify IFF non-fixed block sizes are used'. I'm > > assuming that would work for you? Please try the below patch. If > > succesful, I'll augment it with an updated HOWTO description. > > > > diff --git a/init.c b/init.c > > index 95c282a..001e5c4 100644 > > --- a/init.c > > +++ b/init.c > > @@ -206,6 +206,13 @@ static int setup_rate(struct thread_data *td) > > return 0; > > } > > > > +static int fixed_block_size(struct thread_options *o) > > +{ > > + return o->min_bs[DDIR_READ] == o->max_bs[DDIR_READ] && > > + o->min_bs[DDIR_WRITE] == o->max_bs[DDIR_WRITE] && > > + o->min_bs[DDIR_READ] == o->min_bs[DDIR_WRITE]; > > +} > > + > > /* > > * Lazy way of fixing up options that depend on each other. We could also > > * define option callback handlers, but this is easier. > > @@ -269,8 +276,10 @@ static int fixup_options(struct thread_data *td) > > if (!o->file_size_high) > > o->file_size_high = o->file_size_low; > > > > - if (o->norandommap && o->verify != VERIFY_NONE) { > > - log_err("fio: norandommap given, verify disabled\n"); > > + if (o->norandommap && o->verify != VERIFY_NONE > > + && !fixed_block_size(o)) { > > + log_err("fio: norandommap given for variable block sizes, " > > + "verify disabled\n"); > > o->verify = VERIFY_NONE; > > } > > if (o->bs_unaligned && (o->odirect || td->io_ops->flags & FIO_RAWIO)) > > > > -- > > Jens Axboe > > > > -- Jens Axboe ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2009-02-19 20:15 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-02-18 23:12 Disabling verify when norandommap is given Radha Ramachandran 2009-02-19 7:47 ` Jens Axboe 2009-02-19 18:10 ` Radha Ramachandran 2009-02-19 18:26 ` Jens Axboe 2009-02-19 19:56 ` Radha Ramachandran 2009-02-19 20:12 ` Jens Axboe
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox