A lot of data written by cleaner for specific workload

A lot of data written by cleaner for specific workload

[Clemens Eisserer]

Hi,

I hope it is ok to write this question to the developer mailing list,
I didn't find any user-related one.

I am using f2fs on my raspberry pi (linux-3.10.23) to store sensor
values every 10s in a postgresql database (a workload that killed a
32gb microSD card formatted with ext4 within 4 days, it developed bad
blocks where the journal was located).
With f2fs I noticed the cleaner is writing a lot, often as much as
80-90% of the total data written:

Iotop accumulated results for a few minutes are:

                              READ:       WRITE
01313 be/4 root         35.19 M     70.12 M  0.00 %  0.09 % [f2fs_gc-8:0]
01863 be/4 root          0.00 B   1240.00 K  0.00 %  0.02 % [kworker/u2:1]
02290 be/4 root          0.00 B   1100.00 K  0.00 %  0.42 % java
-Djava.util.logging.config.file=/root/apache~7/temp
org.apache.catalina.startup.Bootstrap start
02289 be/4 root          0.00 B   1092.00 K  0.00 %  0.41 % java -jar
/root/WPO.jar
12346 be/4 root          0.00 B   1052.00 K  0.00 %  0.02 % [kworker/u2:0]
02147 be/4 postgres   0.00 B    768.00 K  0.00 %  0.00 % postgres: wal
writer process
02529 be/4 root          0.00 B    532.00 K  0.00 %  0.08 % [kworker/u2:2]
02146 be/4 postgres   8.00 K    464.00 K  0.00 %  0.00 % postgres:
writer process
02447 be/4 root          8.00 K    384.00 K  0.00 %  0.01 % java
-Djava.util.logging.config.file=/root/apache~7/temp
org.apache.catalina.startup.Bootstrap start
rest < 100 KB

Is this to be expected for this kind of workload?

Thank you in advance, Clemens

------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today. 
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk

Re: A lot of data written by cleaner for specific workload

[Jaegeuk Kim]

Hi,

2014-01-23 (목), 13:42 +0100, Clemens Eisserer:
> Hi,
> 
> I hope it is ok to write this question to the developer mailing list,
> I didn't find any user-related one.
> 
> I am using f2fs on my raspberry pi (linux-3.10.23) to store sensor
> values every 10s in a postgresql database (a workload that killed a
> 32gb microSD card formatted with ext4 within 4 days, it developed bad
> blocks where the journal was located).
> With f2fs I noticed the cleaner is writing a lot, often as much as
> 80-90% of the total data written:
> 
> Iotop accumulated results for a few minutes are:
> 
>                               READ:       WRITE
> 01313 be/4 root         35.19 M     70.12 M  0.00 %  0.09 % [f2fs_gc-8:0]

The f2fs_gc can be controlled by user through a sysfs entry.
So, could you look at the amount of written data by varying the gc
interval?

cat /sys/fs/f2fs/$dev/gc_min_sleep_time
30000
cat /sys/fs/f2fs/$dev/gc_max_sleep_time
60000

These values are represented in milliseconds.
If you don't want to write data as much as possibe and don't need a high
performance as the partition becomes full, you can just grow the
interval times.
The other way is that you can just add a mount option, "-o
background_gc=off", which turns off the f2fs_gc.

Thanks,

> 01863 be/4 root          0.00 B   1240.00 K  0.00 %  0.02 % [kworker/u2:1]
> 02290 be/4 root          0.00 B   1100.00 K  0.00 %  0.42 % java
> -Djava.util.logging.config.file=/root/apache~7/temp
> org.apache.catalina.startup.Bootstrap start
> 02289 be/4 root          0.00 B   1092.00 K  0.00 %  0.41 % java -jar
> /root/WPO.jar
> 12346 be/4 root          0.00 B   1052.00 K  0.00 %  0.02 % [kworker/u2:0]
> 02147 be/4 postgres   0.00 B    768.00 K  0.00 %  0.00 % postgres: wal
> writer process
> 02529 be/4 root          0.00 B    532.00 K  0.00 %  0.08 % [kworker/u2:2]
> 02146 be/4 postgres   8.00 K    464.00 K  0.00 %  0.00 % postgres:
> writer process
> 02447 be/4 root          8.00 K    384.00 K  0.00 %  0.01 % java
> -Djava.util.logging.config.file=/root/apache~7/temp
> org.apache.catalina.startup.Bootstrap start
> rest < 100 KB
> 
> Is this to be expected for this kind of workload?
> 
> Thank you in advance, Clemens
> 
> ------------------------------------------------------------------------------
> CenturyLink Cloud: The Leader in Enterprise Cloud Services.
> Learn Why More Businesses Are Choosing CenturyLink Cloud For
> Critical Workloads, Development Environments & Everything In Between.
> Get a Quote or Start a Free Trial Today. 
> http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
> _______________________________________________
> Linux-f2fs-devel mailing list
> Linux-f2fs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

-- 
Jaegeuk Kim
Samsung


------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today. 
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: A lot of data written by cleaner for specific workload

[Clemens Eisserer]

Hi Jaegeuk,

Thanks a lot for your reply.

> The f2fs_gc can be controlled by user through a sysfs entry.
> So, could you look at the amount of written data by varying the gc
> interval?
>
> cat /sys/fs/f2fs/$dev/gc_min_sleep_time
> 30000
> cat /sys/fs/f2fs/$dev/gc_max_sleep_time
> 60000
>
> These values are represented in milliseconds.
> If you don't want to write data as much as possibe and don't need a high
> performance as the partition becomes full, you can just grow the
> interval times.
> The other way is that you can just add a mount option, "-o
> background_gc=off", which turns off the f2fs_gc.

I'll try turning off the background-gc, as I can't find the
sysfs-knobs in my system.

What I still wonder is, how the cleaner can write twice the data it
reads (as in the above example).

Regards, Clemens

------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today. 
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk

Re: A lot of data written by cleaner for specific workload

[Clemens Eisserer]

Hi again and sorry for the traffic,

I tried to mount with background_gc=off, but in this case f2fs complains:
> [  145.052343] F2FS-fs (mmcblk0p3): Unrecognized mount option "background_gc=off" or missing value

Also, there is no f2fs entry in /sys/fs :/

I guess my kernel (3.10.24+) is too old...

Regards, Clemens

------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today. 
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk

Re: A lot of data written by cleaner for specific workload

[Jaegeuk Kim]

Hi,

2014-01-24 (금), 23:07 +0100, Clemens Eisserer:
> Hi Jaegeuk,
> 
> Thanks a lot for your reply.
> 
> > The f2fs_gc can be controlled by user through a sysfs entry.
> > So, could you look at the amount of written data by varying the gc
> > interval?
> >
> > cat /sys/fs/f2fs/$dev/gc_min_sleep_time
> > 30000
> > cat /sys/fs/f2fs/$dev/gc_max_sleep_time
> > 60000
> >
> > These values are represented in milliseconds.
> > If you don't want to write data as much as possibe and don't need a high
> > performance as the partition becomes full, you can just grow the
> > interval times.
> > The other way is that you can just add a mount option, "-o
> > background_gc=off", which turns off the f2fs_gc.
> 
> I'll try turning off the background-gc, as I can't find the
> sysfs-knobs in my system.
> 
> What I still wonder is, how the cleaner can write twice the data it
> reads (as in the above example).

If a data is selected to be moved, its node block, an index block,
should be read too.
So, I think it can be double.

I think the mount option would be "background_gc_off" in v3.10.
Thanks,

> 
> Regards, Clemens
> 
> ------------------------------------------------------------------------------
> CenturyLink Cloud: The Leader in Enterprise Cloud Services.
> Learn Why More Businesses Are Choosing CenturyLink Cloud For
> Critical Workloads, Development Environments & Everything In Between.
> Get a Quote or Start a Free Trial Today. 
> http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
> _______________________________________________
> Linux-f2fs-devel mailing list
> Linux-f2fs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

-- 
Jaegeuk Kim
Samsung


------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today. 
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel