Re: [oops, 4.4-rc8] warn + oops during generic/204

[Chao Yu <chao2.yu@samsung.com>]

> -----Original Message-----
> From: Jaegeuk Kim [mailto:jaegeuk@kernel.org]
> Sent: Friday, January 22, 2016 9:53 AM
> To: Chao Yu
> Cc: 'Dave Chinner'; linux-f2fs-devel@lists.sourceforge.net
> Subject: Re: [f2fs-dev] [oops, 4.4-rc8] warn + oops during generic/204
> 
> On Thu, Jan 21, 2016 at 07:46:10PM +0800, Chao Yu wrote:
> > Hi Dave,
> >
> > > -----Original Message-----
> > > From: Dave Chinner [mailto:david@fromorbit.com]
> > > Sent: Thursday, January 21, 2016 5:31 AM
> > > To: linux-f2fs-devel@lists.sourceforge.net
> > > Subject: [f2fs-dev] [oops, 4.4-rc8] warn + oops during generic/204
> > >
> > > Hi f2fs folks,
> > >
> > > I just ran xfstests on f2fs using defaults and a pair of 4GB ram
> > > disks for the test and scratch devices, and it hard locked the
> > > machine with this failure in generic/204:
> >
> > Thanks for your report! :)
> >
> > Hi all,
> >
> > We didn't handle well with the case of inline data storm which floods
> > full disk. Actually the reason is: if we have 10M free space, and user
> > fillings the disk with ~10M inline data, then in memory there are ~10M
> > dirty inline datas and ~10M dirty inodes, once inodes were writebacked
> > before inline datas, all free space will be occupied, then we have to
> > write these dirty inline datas to ovp area which doesn't have enough
> > space there normally.
> 
> Well, I think the user block count was wrong which is determined by mkfs.f2fs.
> When writing inline_data, gc should activate to reclaim the space, but it
> seems it couldn't do that cause there is no victim.
> So, when taking a look at mkfs, I suspect that the number of total user blocks
> does not consider our current segments, which is 6 by default.
> IOWs, we gave more blocks to users, which turns out actually gc couldn't get a
> victim from them in this case.

Hmm, I don't think this could fix similar inline storm issue, I change some
parameters in generic/204 like below patch, and do the test with your patch
based on commit 9b72a388f586 ("f2fs: skip releasing nodes in chindless extent
tree"), oops occurs again, still the same problem. :(

---
 tests/generic/204 | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/generic/204 b/tests/generic/204
index 24a2a94..4db1533 100755
--- a/tests/generic/204
+++ b/tests/generic/204
@@ -56,7 +56,7 @@ _scratch_mkfs 2> /dev/null | _filter_mkfs 2> $tmp.mkfs > /dev/null
 # And v4/512 v5/1k xfs don't have enough free inodes, set imaxpct=50 at mkfs
 # time solves this problem.
 [ $FSTYP = "xfs" ] && MKFS_OPTIONS="$MKFS_OPTIONS -l size=7m -i maxpct=50"
-SIZE=`expr 106 \* 1024 \* 1024`
+SIZE=`expr 406 \* 1024 \* 1024`
 
 _scratch_mkfs_sized $SIZE $dbsize 2> /dev/null \
 		| _filter_mkfs 2> $tmp.mkfs > /dev/null
@@ -69,7 +69,7 @@ _scratch_mount
 # work out correctly. Space usages is based 22500 files and 1024 reserved blocks
 # on a 4k block size 256 byte inode size filesystem.
 resv_blks=1024
-space=97920000
+space=397920000
 
 # decrease files for inode size.
 #	22500 * (256 + 4k) = ~97MB
-- 
2.6.3

I traced the details of this issue, we can get some hints with below datas:

steps in this issue:
a) generic/204: submit 22 * 512 inline data
b) f2fs_write_node_pages: f2fs_balance_fs_bg->f2fs_sync_fs
c) f2fs_write_data_pages: write inline data page into inode page
d) f2fs_write_node_pages: persist inode pages


Test #1, mkfs.f2fs: not patched,  generic/204: not patched:
[MAIN: 45(OverProv:23 Resv:16)]
valid user segment = 45 - 23 = 22
	inline data	dentry		node		free segment	prefree+dirty
a)	22		1		22		39		0
b)	22		0		0		18		0
c)	0		0		22		18		0
d)	0		0		4		0		19

Test #2, mkfs.f2fs: patched,  generic/204: not patched:
[MAIN: 45(OverProv:23 Resv:16)]
valid user segment = 45 - 23 - 6 = 16
	inline data	dentry		node		free segment	prefree+dirty
a)	16		1		16		39		0
b)	16		0		0		23		0
c)	0		0		16		23		0
d)	0		0		0		7		16

Test #3, mkfs.f2fs: patched,  generic/204: patched:
[MAIN: 195(OverProv:44 Resv:28)]
valid user segment = 195 - 44 - 6 = 145
	inline data	dentry		node		free segment	prefree+dirty
a)	145		1		145		189		0
b)	145		0		0		45		0
c)	0		0		145		45		0
d)	0		0		99		0		46

Once we arrive step c), we would face the danger, because writebacking
of dirty node page will exhaust free segments, result in oops.
During step c) to step d), inode pages were COWed, prefree will increase,
it's a pity these prefree segments could only be reused after a checkpoint,
however checkpoint needs more free space for flushing dirty nodes, that
also will exhaust free segments, this likes a chicken and egg problem.

Any idea to solve this?

Thanks,

> 
> I could reproduce this issue, and once I reduced the number by 6 segments,
> I could avoid the issue.
> 
> Here the change of f2fs-tools.
> 
> From 411166a44009bb138eca937376863e9ce673278a Mon Sep 17 00:00:00 2001
> From: Jaegeuk Kim <jaegeuk@kernel.org>
> Date: Thu, 21 Jan 2016 05:16:37 -0800
> Subject: [PATCH] mkfs.f2fs: adjust current segments for total usable blocks
> 
> When calculating total number of user blocks, we should reserve the current
> segments.
> 
> Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
> ---
>  mkfs/f2fs_format.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/mkfs/f2fs_format.c b/mkfs/f2fs_format.c
> index 2c81ecc..b25a490 100644
> --- a/mkfs/f2fs_format.c
> +++ b/mkfs/f2fs_format.c
> @@ -474,7 +474,12 @@ static int f2fs_write_check_point_pack(void)
> 
>  	/* main segments - reserved segments - (node + data segments) */
>  	set_cp(free_segment_count, get_sb(segment_count_main) - 6);
> -	set_cp(user_block_count, ((get_cp(free_segment_count) + 6 -
> +	if (get_cp(free_segment_count) <= get_cp(overprov_segment_count)) {
> +		MSG(0, "Error: Too small size\n");
> +		goto free_sum_compact;
> +	}
> +
> +	set_cp(user_block_count, ((get_cp(free_segment_count) -
>  			get_cp(overprov_segment_count)) * config.blks_per_seg));
>  	/* cp page (2), data summaries (1), node summaries (3) */
>  	set_cp(cp_pack_total_block_count, 6 + get_sb(cp_payload));
> --
> 2.6.3



------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140