Re: video archive on a microSD card

[Jaegeuk Kim <jaegeuk@kernel.org>]

On Mon, Aug 22, 2016 at 11:52:12PM +0300, Alexander Gordeev wrote:
> Hi,
> 
> I ran the test over weekend and I think I have some interesting results.
> I used 1 new SD card in one device and two fully utilized SD cards,
> that have problems with write latency, on two oter devices.
> I mounted all the cards with mode=lfs. The new SD card got utilized at 95% after some time.
> Here is the current status after the archive is rotated for some time:
> 
> =====[ partition info(sda1). #0, RW]=====
> [SB: 1] [CP: 2] [SIT: 2] [NAT: 68] [SSA: 30] [MAIN: 15112(OverProv:803 Resv:50)]
> 
> Utilization: 94% (6929763 valid blocks)
>   - Node: 8113 (Inode: 1255, Other: 6858)
>   - Data: 6921650
>   - Inline_xattr Inode: 0
>   - Inline_data Inode: 1
>   - Inline_dentry Inode: 0
>   - Orphan Inode: 0
> 
> Main area: 15112 segs, 7556 secs 7556 zones
>   - COLD  data: 5306, 2653, 2653
>   - WARM  data: 5233, 2616, 2616
>   - HOT   data: 15100, 7550, 7550
>   - Dir   dnode: 15097, 7548, 7548
>   - File   dnode: 4701, 2350, 2350
>   - Indir nodes: 15105, 7552, 7552
> 
>   - Valid: 97
>   - Dirty: 13798
>   - Prefree: 0
>   - Free: 1217 (604)
> 
> CP calls: 282 (BG: 0)
> GC calls: 0 (BG: 0)
>   - data segments : 0 (0)
>   - node segments : 0 (0)
> Try to move 0 blocks (BG: 0)
>   - data blocks : 0 (0)
>   - node blocks : 0 (0)
> 
> Extent Cache:
>   - Hit Count: L1-1:3084 L1-2:456 L2:0
>   - Hit Ratio: 4% (3540 / 84026)
>   - Inner Struct Count: tree: 1252(0), node: 0
> 
> Balancing F2FS Async:
>   - inmem:    0, wb_bios:    0
>   - nodes:   12 in   30
>   - dents:    3 in dirs:   2 (   2)
>   - datas:   48 in files:   0
>   - meta:    9 in   34
>   - NATs:        10/      249
>   - SITs:        13/    15112
>   - free_nids:      1797
> 
> Distribution of User Blocks: [ valid | invalid | free ]
>   [-----------------------------------------------|--|-]
> 
> IPU: 0 blocks
> SSR: 0 blocks in 0 segments
> LFS: 912188 blocks in 1781 segments
> 
> BDF: 94, avg. vblocks: 996
> 
> Memory: 3604 KB
>   - static: 3270 KB
>   - cached: 78 KB
>   - paged : 256 KB
> 
> 
> The interesting thing here is the very small number of valid and
> a huge number of dirty sections. I don't understand this at all.

This is the number of dirty segments, so it needs to consider section and
segment at the same time; a dirty section can consist of valid and free
segments.
How abouting testing 2MB-sized section which is the default option?

> Still the archive is working perfectly. Also I see, that there GC is never
> called, which is probably an indication of FS working exactly as
> we expected.
> Also the small number of cold sections does not make problems.
> So, well, it works perfect so fat. But I don't understand everything here.
> Is this expected?

So, I'm in doubt that dirty sections consist of entirely valid or free segments.

> 
> The other two SD cards were tested differently. On one of them I called
> ioctl(F2FS_IOC_GARBAGE_COLLECT) for several hours. And indeed the number
> of dirty sectoins dropped considerably. It works fine so far.

It makes sense that valid segments in dirty sections would be migrated to
different free sections.

> On the other SD card I called ioctl(F2FS_IOC_DEFRAGMENT) for every
> file in the archive. It works fine as well now. But the number of dirty sections
> was still very high at the end of defragmentation. I don't understand this
> as well.

This is for defragementation to the given file, which would not move blocks in
order to decrease the number of dirty sections.
I think it's not necessary for this workload.

Thanks,

> 
> Thanks!
> 
> 19.08.2016, 14:56, "Alexander Gordeev" <alex@gordick.net>:
> >  Hi Jaegeuk,
> >
> >  19.08.2016, 05:41, "Jaegeuk Kim" <jaegeuk@kernel.org>:
> >>    Hello,
> >>
> >>    On Thu, Aug 18, 2016 at 02:04:55PM +0300, Alexander Gordeev wrote:
> >>
> >>    ...
> >>
> >>>     >>>>>>>   Here is also /sys/kernel/debug/f2fs/status for reference:
> >>>     >>>>>>>   =====[ partition info(sda). #0 ]=====
> >>>     >>>>>>>   [SB: 1] [CP: 2] [SIT: 4] [NAT: 118] [SSA: 60] [MAIN: 29646(OverProv:1529
> >>>     >>>>>>>  Resv:50)]
> >>>     >>>>>>>
> >>>     >>>>>>>   Utilization: 94% (13597314 valid blocks)
> >>>     >>>>>>>     - Node: 16395 (Inode: 2913, Other: 13482)
> >>>     >>>>>>>     - Data: 13580919
> >>>     >>>>>>>
> >>>     >>>>>>>   Main area: 29646 segs, 14823 secs 14823 zones
> >>>     >>>>>>>     - COLD data: 3468, 1734, 1734
> >>>     >>>>>>>     - WARM data: 12954, 6477, 6477
> >>>     >>>>>>>     - HOT data: 28105, 14052, 14052
> >>>     >>>>>>>     - Dir dnode: 29204, 14602, 14602
> >>>     >>>>>>>     - File dnode: 19960, 9980, 9980
> >>>     >>>>>>>     - Indir nodes: 29623, 14811, 14811
> >>>     >>>>>>>
> >>>     >>>>>>>     - Valid: 13615
> >>>     >>>>>>>     - Dirty: 13309
> >>>     >>>>>>>     - Prefree: 0
> >>>     >>>>>>>     - Free: 2722 (763)
> >>>     >>>>>>>
> >>>     >>>>>>>   GC calls: 8622 (BG: 4311)
> >>>     >>>>>>>     - data segments : 8560
> >>>     >>>>>>>     - node segments : 62
> >>>     >>>>>>>   Try to move 3552161 blocks
> >>>     >>>>>>>     - data blocks : 3540278
> >>>     >>>>>>>     - node blocks : 11883
> >>>     >>>>>>>
> >>>     >>>>>>>   Extent Hit Ratio: 49 / 4171
> >>>     >>>>>>>
> >>>     >>>>>>>   Balancing F2FS Async:
> >>>     >>>>>>>     - nodes 6 in 141
> >>>     >>>>>>>     - dents 0 in dirs: 0
> >>>     >>>>>>>     - meta 13 in 346
> >>>     >>>>>>>     - NATs 16983 > 29120
> >>>     >>>>>>>     - SITs: 17
> >>>     >>>>>>>     - free_nids: 1861
> >>>     >>>>>>>
> >>>     >>>>>>>   Distribution of User Blocks: [ valid | invalid | free ]
> >>>     >>>>>>>     [-----------------------------------------------|-|--]
> >>>     >>>>>>>
> >>>     >>>>>>>   SSR: 1230719 blocks in 14834 segments
> >>>     >>>>>>>   LFS: 15150190 blocks in 29589 segments
> >>>     >>>>>>>
> >>>     >>>>>>>   BDF: 89, avg. vblocks: 949
> >>>     >>>>>>>
> >>>     >>>>>>>   Memory: 6754 KB = static: 4763 + cached: 1990
> >>
> >>    ...
> >>
> >>>     >>  Per my understanding of f2fs internals, it should write these "cold" files and
> >>>     >>  usual "hot" files to different sections (that should map internally to
> >>>     >>  different allocation units). So the sections used by "cold" data should almost
> >>>     >>  never get "dirty" because most of the time all their blocks become free at
> >>>     >>  the same time. Of course, the files are not exactly 4MB in size so the last
> >>>     >>  section of the deleted file will become dirty. If it is moved by garbage
> >>>     >>  collector and becomes mixed with fresh "cold" data, then indeed it might cause
> >>>     >>  some problems, I think. What is your opinion?
> >>>     >
> >>>     > If your fs is not fragmented, it may be as what you said, otherwise, SSR will
> >>>     > still try to reuse invalid block of other temperture segments, then your cold
> >>>     > data will be fixed with warm data too.
> >>>     >
> >>>     > I guess, what you are facing is the latter one:
> >>>     > SSR: 1230719 blocks in 14834 segments
> >>>
> >>>     I guess, I need to somehow disable any cleaning or SSR for my archive and index
> >>>     files. But keep the cleaning for other data and nodes.
> >>
> >>    Could you test a mount option, "mode=lfs", to disable SSR?
> >>    (I guess sqlite may suffer from logner latency due to GC though.)
> >>
> >>    Seems like it's caused by SSR starting to make worse before 95% as you described
> >>    below.
> >
> >  Thanks, I'll run a test with a couple of SD cards over weekend.
> >  So if I understand it correctly, GC will not cause the problems described below, right?
> >  I.e. it will not mix the new data with old data from dirty sections?
> >  Longer SQLite latencies should not be a problem because the database is written not
> >  frequently and also it is about 200-250KB in size usually. Maybe forcing IPU as
> >  suggested by Chao would help sqlite, no?
> >  However looks like setting ipu_policy to 1 has no effect when mode=lfs.
> >  The IPU counter is still zero on my test system.
> > ...
> -- 
>  Alexander

------------------------------------------------------------------------------