* memory leak bug? when do write with verify [not found] <CADson1wk9astwdroeAtPXb4A3+yrAcDcCXaEPvEGJPhM4TBfqQ@mail.gmail.com> @ 2011-09-14 9:53 ` Zhang Taile 2011-09-14 9:56 ` Jiri Horky 2011-09-14 17:39 ` Jens Axboe 0 siblings, 2 replies; 6+ messages in thread From: Zhang Taile @ 2011-09-14 9:53 UTC (permalink / raw) To: fio HI When I do RandWrite with verify, fio will use a lot of memory. The fio will increase memory about 10MB in every second. And, the system will become very slow, after fio start to use swap partition. For example,� if use fio with following configuration, [rand-write] rw=randwrite numjobs=32 loops=10 thread=1 filename=/dev/sdb verify=md5 verify_interval=4k after� minutes, top will Like this : �PID USER����� PR� NI� VIRT� RES� SHR S %CPU %MEM��� TIME+� COMMAND �3749 root����� 15�� 0 2672m 1.5g 304m S 169.4 40.4�� 4:10.04 fio If remove the verify from the configuration, there will be no the memory problem. I guess it is a memory leak problem? Thanks! -- Taile ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory leak bug? when do write with verify 2011-09-14 9:53 ` memory leak bug? when do write with verify Zhang Taile @ 2011-09-14 9:56 ` Jiri Horky 2011-09-14 13:13 ` Zhang Taile 2011-09-14 17:39 ` Jens Axboe 1 sibling, 1 reply; 6+ messages in thread From: Jiri Horky @ 2011-09-14 9:56 UTC (permalink / raw) To: Zhang Taile; +Cc: fio [-- Attachment #1: Type: text/plain, Size: 1223 bytes --] Hi, this sounds like the problem addressed by this thread [1]. Have you tried to use the latest (trunk) version that includes the mentioned patch? Jiri Horky [1] http://www.spinics.net/lists/fio/msg00843.html On 09/14/2011 11:53 AM, Zhang Taile wrote: > HI > When I do RandWrite with verify, fio will use a lot of memory. The fio > will increase memory about 10MB in every second. And, the system will > become very slow, after fio start to use swap partition. > > For example, if use fio with following configuration, > [rand-write] > rw=randwrite > numjobs=32 > loops=10 > thread=1 > filename=/dev/sdb > verify=md5 > verify_interval=4k > > after minutes, top will Like this : > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 3749 root 15 0 2672m 1.5g 304m S 169.4 40.4 4:10.04 fio > > > If remove the verify from the configuration, there will be no the > memory problem. > > I guess it is a memory leak problem? > > > Thanks! > -- > Taile > -- > To unsubscribe from this list: send the line "unsubscribe fio" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 6917 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory leak bug? when do write with verify 2011-09-14 9:56 ` Jiri Horky @ 2011-09-14 13:13 ` Zhang Taile 0 siblings, 0 replies; 6+ messages in thread From: Zhang Taile @ 2011-09-14 13:13 UTC (permalink / raw) To: Jiri Horky; +Cc: fio Hi, Seems not same bug. I have tested latest (trunk), still has the memory leak issue. Taile On Wed, Sep 14, 2011 at 5:56 PM, Jiri Horky <jiri.horky@cesnet.cz> wrote: > Hi, > > this sounds like the problem addressed by this thread [1]. Have you tried to > use the latest (trunk) version that includes the mentioned patch? > > Jiri Horky > > [1] http://www.spinics.net/lists/fio/msg00843.html > > On 09/14/2011 11:53 AM, Zhang Taile wrote: >> >> HI >> When I do RandWrite with verify, fio will use a lot of memory. The fio >> will increase memory about 10MB in every second. And, the system will >> become very slow, after fio start to use swap partition. >> >> For example, if use fio with following configuration, >> [rand-write] >> rw=randwrite >> numjobs=32 >> loops=10 >> thread=1 >> filename=/dev/sdb >> verify=md5 >> verify_interval=4k >> >> after minutes, top will Like this : >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 3749 root 15 0 2672m 1.5g 304m S 169.4 40.4 4:10.04 fio >> >> >> If remove the verify from the configuration, there will be no the >> memory problem. >> >> I guess it is a memory leak problem? >> >> >> Thanks! >> -- >> Taile >> -- >> To unsubscribe from this list: send the line "unsubscribe fio" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- Zhang, Taile 张泰乐 zhangtl@gmail.com MP:18611745290 msn:zhangtaile@hotmail.com skype: zhangtaile ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory leak bug? when do write with verify 2011-09-14 9:53 ` memory leak bug? when do write with verify Zhang Taile 2011-09-14 9:56 ` Jiri Horky @ 2011-09-14 17:39 ` Jens Axboe 2011-09-15 2:49 ` Zhang Taile 1 sibling, 1 reply; 6+ messages in thread From: Jens Axboe @ 2011-09-14 17:39 UTC (permalink / raw) To: Zhang Taile; +Cc: fio On 2011-09-14 11:53, Zhang Taile wrote: > HI > When I do RandWrite with verify, fio will use a lot of memory. The fio > will increase memory about 10MB in every second. And, the system will > become very slow, after fio start to use swap partition. > > For example, if use fio with following configuration, > [rand-write] > rw=randwrite > numjobs=32 > loops=10 > thread=1 > filename=/dev/sdb > verify=md5 > verify_interval=4k > > after minutes, top will Like this : > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 3749 root 15 0 2672m 1.5g 304m S 169.4 40.4 4:10.04 fio > > > If remove the verify from the configuration, there will be no the > memory problem. > > I guess it is a memory leak problem? It's probably not a leak, fio will use a lot of memory for your job file. Each job will maintain a full backlog of meta data for blocks written, so it can safely verify it. And you have 32 jobs, so it'll be a lot of memory. -- Jens Axboe ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory leak bug? when do write with verify 2011-09-14 17:39 ` Jens Axboe @ 2011-09-15 2:49 ` Zhang Taile 2011-09-15 7:24 ` Jens Axboe 0 siblings, 1 reply; 6+ messages in thread From: Zhang Taile @ 2011-09-15 2:49 UTC (permalink / raw) To: Jens Axboe; +Cc: fio Hi , Understand now. One more question, what backlog for? Why read only with verify do not need it? Thanks! On Thu, Sep 15, 2011 at 1:39 AM, Jens Axboe <axboe@kernel.dk> wrote: > On 2011-09-14 11:53, Zhang Taile wrote: >> HI >> When I do RandWrite with verify, fio will use a lot of memory. The fio >> will increase memory about 10MB in every second. And, the system will >> become very slow, after fio start to use swap partition. >> >> For example, if use fio with following configuration, >> [rand-write] >> rw=randwrite >> numjobs=32 >> loops=10 >> thread=1 >> filename=/dev/sdb >> verify=md5 >> verify_interval=4k >> >> after minutes, top will Like this : >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 3749 root 15 0 2672m 1.5g 304m S 169.4 40.4 4:10.04 fio >> >> >> If remove the verify from the configuration, there will be no the >> memory problem. >> >> I guess it is a memory leak problem? > > It's probably not a leak, fio will use a lot of memory for your job > file. Each job will maintain a full backlog of meta data for blocks > written, so it can safely verify it. And you have 32 jobs, so it'll be a > lot of memory. > > -- > Jens Axboe > > -- Zhang, Taile 张泰乐 zhangtl@gmail.com MP:18611745290 msn:zhangtaile@hotmail.com skype: zhangtaile ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: memory leak bug? when do write with verify 2011-09-15 2:49 ` Zhang Taile @ 2011-09-15 7:24 ` Jens Axboe 0 siblings, 0 replies; 6+ messages in thread From: Jens Axboe @ 2011-09-15 7:24 UTC (permalink / raw) To: Zhang Taile; +Cc: fio On 2011-09-15 04:49, Zhang Taile wrote: > Hi , > Understand now. > One more question, what backlog for? Why read only with verify do not > need it? It's not strictly necessary for most cases. For random writes, we can verify faster by doing it in disk order instead of in write order. If you are using trim to discard parts of the ranges, then we need those appropriately marked in memory. But we can/could relax it a bit and save the memory. I don't think I've had this complaint before, I guess people generally have plenty of RAM these days :-) -- Jens Axboe ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-09-15 7:24 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CADson1wk9astwdroeAtPXb4A3+yrAcDcCXaEPvEGJPhM4TBfqQ@mail.gmail.com>
2011-09-14 9:53 ` memory leak bug? when do write with verify Zhang Taile
2011-09-14 9:56 ` Jiri Horky
2011-09-14 13:13 ` Zhang Taile
2011-09-14 17:39 ` Jens Axboe
2011-09-15 2:49 ` Zhang Taile
2011-09-15 7:24 ` Jens Axboe
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox