[PATCH 0/2] btrfs-progs: add check and repair ability for super num devices mismatch

[PATCH 0/2] btrfs-progs: add check and repair ability for super num devices mismatch

[Qu Wenruo]

The patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/super_num_devs

The 2nd patch contains a compressed raw image, thus it may be a little
too large for the mail list.

This patchset will allow btrfs check to detect and repair super num devices
mismatch.

The detect part is to iterate through chunk tree to grab the correct
number of devices.
This is more reliable than counting devices in fs_devices, since seed
device is in another fs_devices.

The repair is more straightforward, just reset the value in superblock
and commit a transaction.

Qu Wenruo (2):
  btrfs-progs: check: add check and repair ability for super num devs
    mismatch
  btrfs-progs: tests/fsck: add test case for super num devs mismatch

 check/main.c                                  |   1 +
 check/mode-common.c                           |  88 ++++++++++++++++++
 check/mode-common.h                           |   2 +
 .../default.raw.xz                            | Bin 0 -> 22028 bytes
 4 files changed, 91 insertions(+)
 create mode 100644 tests/fsck-tests/054-super-num-devs-mismatch/default.raw.xz

-- 
2.35.1

[PATCH 1/2] btrfs-progs: check: add check and repair ability for super num devs mismatch

[Qu Wenruo]

[BUG]
There is a bug report of kernel rejecting fs which has a mismatch in
super num devices and num devices found in chunk tree.

But btrfs-check reports no problem about the fs.

[CAUSE]
We just didn't verify super num devices against the result found in
chunk tree.

[FIX]
Add such check and repair ability for btrfs-check.

The ability is mode independent.

Reported-by: Luca Béla Palkovics <luca.bela.palkovics@gmail.com>
Link: https://lore.kernel.org/linux-btrfs/CA+8xDSpvdm_U0QLBAnrH=zqDq_cWCOH5TiV46CKmp3igr44okQ@mail.gmail.com/
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 check/main.c        |  1 +
 check/mode-common.c | 88 +++++++++++++++++++++++++++++++++++++++++++++
 check/mode-common.h |  2 ++
 3 files changed, 91 insertions(+)

diff --git a/check/main.c b/check/main.c
index 8ccba4478de8..b29f6266b974 100644
--- a/check/main.c
+++ b/check/main.c
@@ -9140,6 +9140,7 @@ static int do_check_chunks_and_extents(void)
 		if (ret > 0)
 			ret = 0;
 	}
+	ret = check_and_repair_super_num_devs(gfs_info);
 	return ret;
 }
 
diff --git a/check/mode-common.c b/check/mode-common.c
index c3d8bb45c6b6..a2c6c7732f4e 100644
--- a/check/mode-common.c
+++ b/check/mode-common.c
@@ -1583,3 +1583,91 @@ int fill_csum_tree(struct btrfs_trans_handle *trans, bool search_fs_tree)
 	}
 	return ret;
 }
+
+static int get_num_devs_in_chunk_tree(struct btrfs_fs_info *fs_info)
+{
+	struct btrfs_root *chunk_root = fs_info->chunk_root;
+	struct btrfs_path path = {0};
+	struct btrfs_key key = {0};
+	int found_devs = 0;
+	int ret;
+
+	ret = btrfs_search_slot(NULL, chunk_root, &key, &path, 0, 0);
+	if (ret < 0)
+		return ret;
+
+	/* We should be the first slot, and chunk tree should not be empty*/
+	ASSERT(path.slots[0] == 0 && btrfs_header_nritems(path.nodes[0]));
+
+	btrfs_item_key_to_cpu(path.nodes[0], &key, path.slots[0]);
+
+	while (key.objectid == BTRFS_DEV_ITEMS_OBJECTID &&
+	       key.type == BTRFS_DEV_ITEM_KEY) {
+		found_devs++;
+
+		ret = btrfs_next_item(chunk_root, &path);
+		if (ret < 0)
+			break;
+
+		/*
+		 * This should not happen, as we should have CHUNK items after
+		 * dev items, but since we're only to get the num devices,
+		 * no need to bother the problem.
+		 */
+		if (ret > 0) {
+			ret = 0;
+			break;
+		}
+		btrfs_item_key_to_cpu(path.nodes[0], &key, path.slots[0]);
+	}
+	btrfs_release_path(&path);
+	if (ret < 0)
+		return ret;
+	return found_devs;
+}
+
+int check_and_repair_super_num_devs(struct btrfs_fs_info *fs_info)
+{
+	struct btrfs_trans_handle *trans;
+	int found_devs;
+	int ret;
+
+	ret = get_num_devs_in_chunk_tree(fs_info);
+	if (ret < 0)
+		return ret;
+
+	found_devs = ret;
+
+	if (found_devs == btrfs_super_num_devices(fs_info->super_copy))
+		return 0;
+
+	/* Now the found devs in chunk tree mismatch with super block*/
+	error("super num devices mismatch, have %llu expect %u",
+	      btrfs_super_num_devices(fs_info->super_copy),
+	      found_devs);
+
+	if (!repair)
+		return -EUCLEAN;
+
+	/*
+	 * Repair is pretty simple, just reset the super block value and
+	 * commit a new transaction.
+	 */
+	trans = btrfs_start_transaction(fs_info->tree_root, 0);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		errno = -ret;
+		error("failed to start trans: %m");
+		return ret;
+	}
+	btrfs_set_super_num_devices(fs_info->super_copy, found_devs);
+	ret = btrfs_commit_transaction(trans, fs_info->tree_root);
+	if (ret < 0) {
+		errno = -ret;
+		error("failed to commit trans: %m");
+		btrfs_abort_transaction(trans, ret);
+		return ret;
+	}
+	printf("Successfully reset super num devices to %u\n", found_devs);
+	return 0;
+}
diff --git a/check/mode-common.h b/check/mode-common.h
index b5e6b727fe73..d5bab85a4f5e 100644
--- a/check/mode-common.h
+++ b/check/mode-common.h
@@ -201,4 +201,6 @@ int repair_dev_item_bytes_used(struct btrfs_fs_info *fs_info,
 
 int fill_csum_tree(struct btrfs_trans_handle *trans, bool search_fs_tree);
 
+int check_and_repair_super_num_devs(struct btrfs_fs_info *fs_info);
+
 #endif
-- 
2.35.1

[PATCH 2/2] btrfs-progs: tests/fsck: add test case for super num devs mismatch

[Qu Wenruo]

The new image will has incorrect super num devices (have 8 expect 1).

The image has to be raw format, as btrfs-image restore will reset super
num devices to correct value.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 .../054-super-num-devs-mismatch/default.raw.xz  | Bin 0 -> 22028 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 tests/fsck-tests/054-super-num-devs-mismatch/default.raw.xz

diff --git a/tests/fsck-tests/054-super-num-devs-mismatch/default.raw.xz b/tests/fsck-tests/054-super-num-devs-mismatch/default.raw.xz
new file mode 100644
index 0000000000000000000000000000000000000000..af0306bedbd0bd45dfd2d65a0f0a9f6bf662dfa8
GIT binary patch
literal 22028
zcmeHPXH=8vwoas@6sc0BNCas?L8=&vREg3F0s#a8=}bTvX@U|ugdzw^FQIoOf*4VH
z5oywkB8Uc%CP>SfIdkT&`(vD|&c(xz?{8MJ-j#Pf``P<>-o5v(^sgqSAQ0*7V71yI
z5HFY$1Om~&H0dM~Pmng6gFvpkL}K<MqPn66vxlseZf%_4Y<396m&F<Gpg!>?IR&tG
zg@0RkxQ=)>&2T{8+erSS`csHwO`hRzZfrhG2sm;bKY`M;G0V8hMaK3fLfiMj?R7HR
z{0mR({jztiJw*s5s80C=@O-Z16qMvjZu2W3S;^^9Tn~_JFIMQv8Q&rANbSgEI1fg;
zCB{~dFdzDWW^^>NL=Nba_1PC$1i&>lT8?&q=4hEZHV1R%3Nup<b<>h*Afyh&Vis@o
zoD^IZ7%UKSg&yUb)LQLqh`*d!Wc3A`ovO(HaDA&c5|-YVuRX(|KG@H!(LfDx5~c{>
zup7Oe+mxT~=$mYf%T?Nx^*65G4kDi@NL*f65L<+uT5fMoPPLO<UF9mWEgzCfOG{-7
zauuT5LVRp?8^1(0G%T(m|H^h=%zdwbQJA4_&8Mm$_4ezqIJHmJRj@A(icI>}Ud4!+
zcUQ<LzqE{$>}=}^qq?}S@F;&7*FU!@WJ4t%*m25fvPS;Xizi~pLOMn5Ys#^Vv_$&&
zO?Pj^=HV`4qM#aG!H)05b_!I?2p@B2HTV|Z_Jx4{<~ruA1X*tRqU$p^9rYuxFh>hZ
zEV&+cSDZX%h6qe50?Ru0dN$aD#PB+VFwHFaId#3BJa-0Bo7`7PO2{a7Qg0s}qkalW
z7=={x<fXT}Y~Dn!lQi9K-~odEr4#gm1&q}Lhj!GuuXPoyh6Qp7E2@ceQ~Q*yTd*uz
z`e@d5k7z!4c4*$xhw4!^k1B*kW&wP}Q7nY>bX*6S+1l8KwN`~&n+Vr~;SD|Q6r6Tn
zy<AD&{e<p{&v6EKMXoZ(`*88Q6&pTPl3#MObgp_-SR&us0hg#$j$A3nPtKgM^RxR1
zMK(1<@^R2otZ}WeHeY;Vk9*{W!ji-WEm4I`;Q_4)QmGJ~Q{o%p`pIcJS2n5{sZZWJ
zs~wd<lX|yQasWKn)cs*XM@24Tb<av5VtY1Nb@`SqBQ2C3!I~{@SI2T?J&c_#Wy%A;
zrn-`GB>)atnxte2xo~IL5b2N^!pW_CRF$C)hGe)&c(HytHgojWWhQcrxvY44kr74l
zW-}9I#b<%1>=uQvi9&PPtW1wU>DPKngo{^(wx^I?iioH?{pFZFmK9SSi<<U=8P<V^
zK2)bk6Iw>;cqK?zwq6c4tFJ9B)1c{w>^w0?kaz0+tudlwQEc;vIP5cLCV5Vtjzmoj
z`o~K^qapZUH3fp#C>cGU>KS$S0_gJ;CiX66hYo9;;Z*AIy)&Ht<}$~p6qc$DvP_GG
zwiih@NNnn^pUr)Gxbefb85!J+t`|<h-g$S^ov=Ep-xM%Tf+`UQKhF#svM6(?XY9DU
zurnGFPQ2!nFu{>j?}Q(q%ssS_3cpn`-%MIsR9J5P7EVg}^2-vMirLZXc?z5g!Xlk=
zSj8RX@vaxqYhB&pZCz4#-VHi$J5nOzemZ-vNFnA5YE4>N$e4_{S3+jhb>0M;G23CL
z!_ja2i*w#|>z)s94fafo^=&^>UF$fMrr8VU%DB0V)@};Y#q^0><&MzvW}uiJco{@6
z!TtETm=n`(Y5Dv}f?&s;SBc5t{--c2Cp|aV@Z`xl$-6^k1F>{31f)D8%+rbVU||s9
z7ybO*hEiV%_Vh<l5-$JuPyareLgugOnKaD*Hv)NIkJhC-UztwqA(c3Y7uHEgoh?O|
zz#R&=2U|H*J3R9E*3fhFAUe)5WtCkgrmeh%VnZd`<DP|{?;oF_)0!Rf9^*Z)a+lLY
zaxcV~JXTiwDOKN9?+ur1n0N^>%ukxTmT;dItp(32mkQkEJ9kQi^;7}c!jTG(eZIrC
zUWv(~{v{_-X4B?$-O@s<ux<tR3NnI5H*a(mVd&v&lo1jHIfg#^mJ4Af%ExSW;hhE@
zHw5*;k5jAIM5Z5y@>GgLpQ|lxt7(@}GG$CVsvPT1)?-82(A)q!L_s)S+4`_ibREXH
z@MO1~9eYV|G_g-l{#?@uDO~j8IA2F#d<VIZ=3x}))s-)(Q#n&IIE=wn>fxM+;|uSN
zD%!els7W=Bm8A~DF|kXLBA#d21F6q2Xy%>m6f>SN-7D8$)wbyBH0he{Osc4_kys2;
zYj1NY(-C`;7Ue9+C}5Z%z;7Uoyj$L=it$frv|d5kI#HT8JNkZJXB9J|BwE!vvWC@*
zbI3P6NotyBEeOC;cBRXJV=V@U+9;JR$rg;t3}w5oL{V5p$A!l;kARPA?Zz5%Mo+j!
z+I#@j^VBXZU#DjM=&v$Ejqb)q@8;b(qH4n-W6OW7doFhLH3{{~G#pG(v@z=GY-h}9
zqcZ0`RUjWNI^N^?$XQ?51mBID&^F)k&RDPAoCGtWZ)x{w+rcg@Inr2uoB*S2{auOC
zUy|kDI|g}8J;_)-9-m(z%9OW9&CF`gmmD_i>s&lUh)LI>NvhXe51FUpe^vPc{+ch-
zXKJiuV>{BE;W{EeCiO7c(1ebZX~G1RlNtB>%%JAE0o<pTg{zO6pPH){VWHj45-hrE
zJ#0;8vL_w~|9ZcakA*e8fL#igS5b%Co~rU?*K33_GUrUMX^0jXIN|C0GzP*8Xz}_`
zO;{wg3-*K@3G0S9jNZ$Nn3G<2>lHaF%5auNLEzK*lKE?H>c(%z64g@W1^ZWA7MtL;
zg(CMG)oaVL+<s+po#b@5sHis<hpOQTEIUKj<4<Nh=5o0)oDfe<WLR=RRQcTyu0wkf
zZsJ{#Dq&0~d2E%o4ah^n_e!%ieOmqt<%KSm#uwn(@2J)O;8}XG$bsV71k&E>P+9s5
zY>dI-4;JDx2b|;jqerey!#|w6r^zCm_(gpSKBxC4j@*bXO*QRJ?~K}KgHNHJBGA#s
z%LW*}BytjDo0~in{<L&*E;BgyeKyr8LLt!<QtWYrB`&`~_M39{R}$}|x+*sSVm|;e
zp!WGj{q$!{0v85c*!~pu0K&f0()i=T%E71ul_vi*Vro9_F4{6aU~*cz2=lIdODX7p
z)``m0wQP6y?*P5wX|KK?6<Et{Jf%zy8F?!Q0ri?QELc~IDvq4-sbvZ|Glo&_ylrP?
zl>z&Bc~%6G*58F2;xHV}P~DUkwH*KGQTa2|>wjFL-^Jy9ifNKmght;bAl(A#cK=)s
zfDC_#3|GOY4<s25@d!AWX^23uE$BTbni$L8U<X9b#pCKf&+Pk(aClrj=HLKO89-$K
zm3_~9(ND0M+ep$5B$a(of!wH&+F9nOM|23;FSgR2iJlHR#yxw1)igSuZ#&P{De)A1
z)}dHMWhK2%sK@$d)|f0mCp(E(@bL@DT1a|^aPR&qYU&J{*(5CqNvfnub~cNyi|8~N
zn#Qup5%pV@k#KK_NA^Whvpkb!^=DO<H&mdn6=?AolUb!~N6r_+fh)X(o2hrJP~E$c
zrhBiQOv!`fc4JJHq3OtLB^ZwBGwLvs#7e<JXn06nOMw@E%KKbr3@<SRif#}eV4&ks
z$?uk~X|dMFq8do*Az^qom}n`T@${uRj9TVNBvyLCEmUR6^-Xn~oc(6KNxP2K+$vkG
zo~c0N+2&G#wpS{VuAM%&`5mjDEnjzPfV`#O9!t#Q%&8I*aj`XGYHhWbjh~xTe90nH
z=(W;r_pawbT}5|qlm&IBX@C0!2VvkviZ<)~yB32Y78aWF@rLC$-2~XL`7B1iKpDg3
zXA-#`$vbD`4^=WhGTFHhK?0Ge?8(}Ma2pC=q;kZ|HDqP>vB%@qQbcbaos~RyIs(Gb
zy45=qenZi-wj`-m$cUE@>#)NB=RKSk!92JXx>BI`fR-dGd9uUf@=`1}L&c2ukn^N?
zg{Z@*r1)8!mG7Xm&HD9FjR$BW5_?Lvwh^<s7VJB=dO{XnI8_UV)xX2-#BgMgCL@J)
zjc+w6f^b(6^<t6T;}$aLFfJ6~DI_NOP;h}*wt!h`@YU*LIbt}2{-lHt-js~CM{Tj2
zmA!c<9h;?T`0B<Td+U3N2#WXFkyg)&__-Eqr~8o%{cWtqk6YpW;e+PLP^)bXofXdQ
zKFS%s*9`W%a`<VxqKkN)Eu8669F8B(((EGf>5`Lw=m}|YnVW}>l-W9DN28sb>Ws^0
zFD*pV@jZ^WHp05(<}FFT43;yp_19SIK^X<#Cw1w3GW<TgBuSJg_A%wp2j71hz+U8^
zI)Staq)i}g9<a0tJZFD%OAMefe@J5j2m=tdU(GE*NCF{=_@}k$-}XTQ2m=rXAnbel
zgMpCz&W7rraAA&MMbJlwR%SN&Oc$%MD5dI3Eu)Q-(h7E|0XF=rw`xZJA6sR==AQr2
zJ^qzH>*vlX_S7c2G0W+P+H-&C%x`X%*Ai$M01S-pHZYP8=mXQ9k_=9GZMbWz{J8w=
zUye9@1*G2$n7^$mzK&H0ksdJ!0PVYh2B=IxWdbVm+b6j8mEHi+1c)X;Gy$S%pQlg(
z4m)@%6R5+0It-}8fI4j7RuB9X(*JFe4)q-6)p&33{s>611cHZwVrtk-4P~dFWGU`<
zugniR2fy!S*w@SXOVh%_NB%ruSNwooA$p*?6;tlbi2H3V{<iag!|&P3lF<r%5KfUk
z^5${U|GoC(=fmsY8%G3qBXa?a{Q$;*;X`2f5EwrE)}==K%5H!q23TVITb=<30}uuv
z3@}{*)AcVF7Vc|q|4U)~Kf4n5^=N?f0n*=p(hrpG1W|wp9s&NoSmWPTx`I-aL_b~D
gt#}9mPZxm$0|Q~deb(-YLNT$*LrH%B0x81uU)=*YbpQYW

literal 0
HcmV?d00001

-- 
2.35.1

Re: [PATCH 0/2] btrfs-progs: add check and repair ability for super num devices mismatch

[David Sterba]

On Mon, Feb 28, 2022 at 08:50:06AM +0800, Qu Wenruo wrote:
> The patchset can be fetched from github:
> https://github.com/adam900710/btrfs-progs/tree/super_num_devs
> 
> The 2nd patch contains a compressed raw image, thus it may be a little
> too large for the mail list.
> 
> This patchset will allow btrfs check to detect and repair super num devices
> mismatch.
> 
> The detect part is to iterate through chunk tree to grab the correct
> number of devices.
> This is more reliable than counting devices in fs_devices, since seed
> device is in another fs_devices.
> 
> The repair is more straightforward, just reset the value in superblock
> and commit a transaction.

As repair it should work and be safe, also it should be reasonably safe
to be part of --repair. In addition to that we may also want the rescue
subcommand as it's meant to be for such one-time fixes for specific
errors.

Re: [PATCH 0/2] btrfs-progs: add check and repair ability for super num devices mismatch

[David Sterba]

On Mon, Feb 28, 2022 at 08:50:06AM +0800, Qu Wenruo wrote:
> The patchset can be fetched from github:
> https://github.com/adam900710/btrfs-progs/tree/super_num_devs
> 
> The 2nd patch contains a compressed raw image, thus it may be a little
> too large for the mail list.

The compressed size is 22K, that's fine. I've recompressed it with 'xz
--best -e' and it shaved a few hundred bytes but that was just out of
curiosity.

Re: [PATCH 1/2] btrfs-progs: check: add check and repair ability for super num devs mismatch

[David Sterba]

On Mon, Feb 28, 2022 at 08:50:07AM +0800, Qu Wenruo wrote:
> [BUG]
> There is a bug report of kernel rejecting fs which has a mismatch in
> super num devices and num devices found in chunk tree.
> 
> But btrfs-check reports no problem about the fs.
> 
> [CAUSE]
> We just didn't verify super num devices against the result found in
> chunk tree.
> 
> [FIX]
> Add such check and repair ability for btrfs-check.
> 
> The ability is mode independent.
> 
> Reported-by: Luca Béla Palkovics <luca.bela.palkovics@gmail.com>
> Link: https://lore.kernel.org/linux-btrfs/CA+8xDSpvdm_U0QLBAnrH=zqDq_cWCOH5TiV46CKmp3igr44okQ@mail.gmail.com/
> Signed-off-by: Qu Wenruo <wqu@suse.com>

With this patch applied there's a new test failure:

=== START TEST .../tests/fsck-tests/014-no-extent-info
testing image no_extent.raw.restored
====== RUN MUSTFAIL .../btrfs check ./no_extent.raw.restored
Opening filesystem to check...
Checking filesystem on ./no_extent.raw.restored
UUID: aeee7297-37a4-4547-a8a9-7b870d58d31f
cache and super generation don't match, space cache will be invalidated
found 180224 bytes used, no error found
total csum bytes: 0
total tree bytes: 180224
total fs tree bytes: 81920
total extent tree bytes: 16384
btree space waste bytes: 138540
file data blocks allocated: 0
 referenced 0
succeeded (unexpected!): .../btrfs check ./no_extent.raw.restored
unexpected success: btrfs check should have detected corruption

Re: [PATCH 1/2] btrfs-progs: check: add check and repair ability for super num devs mismatch

[Qu Wenruo]

On 2022/3/24 01:43, David Sterba wrote:
> On Mon, Feb 28, 2022 at 08:50:07AM +0800, Qu Wenruo wrote:
>> [BUG]
>> There is a bug report of kernel rejecting fs which has a mismatch in
>> super num devices and num devices found in chunk tree.
>>
>> But btrfs-check reports no problem about the fs.
>>
>> [CAUSE]
>> We just didn't verify super num devices against the result found in
>> chunk tree.
>>
>> [FIX]
>> Add such check and repair ability for btrfs-check.
>>
>> The ability is mode independent.
>>
>> Reported-by: Luca Béla Palkovics <luca.bela.palkovics@gmail.com>
>> Link: https://lore.kernel.org/linux-btrfs/CA+8xDSpvdm_U0QLBAnrH=zqDq_cWCOH5TiV46CKmp3igr44okQ@mail.gmail.com/
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>
> With this patch applied there's a new test failure:
>
> === START TEST .../tests/fsck-tests/014-no-extent-info
> testing image no_extent.raw.restored
> ====== RUN MUSTFAIL .../btrfs check ./no_extent.raw.restored
> Opening filesystem to check...
> Checking filesystem on ./no_extent.raw.restored
> UUID: aeee7297-37a4-4547-a8a9-7b870d58d31f
> cache and super generation don't match, space cache will be invalidated
> found 180224 bytes used, no error found
> total csum bytes: 0
> total tree bytes: 180224
> total fs tree bytes: 81920
> total extent tree bytes: 16384
> btree space waste bytes: 138540
> file data blocks allocated: 0
>   referenced 0
> succeeded (unexpected!): .../btrfs check ./no_extent.raw.restored
> unexpected success: btrfs check should have detected corruption


Weird, the problem is, for the restored image, if I run ./btrfs check
manually it can detects the problem, but still go "no error found" path.

So it must be my patch overriding the return value.

Will fix it soon.

Thanks,
Qu

Re: [PATCH 0/2] btrfs-progs: add check and repair ability for super num devices mismatch

[Qu Wenruo]

On 2022/3/24 01:17, David Sterba wrote:
> On Mon, Feb 28, 2022 at 08:50:06AM +0800, Qu Wenruo wrote:
>> The patchset can be fetched from github:
>> https://github.com/adam900710/btrfs-progs/tree/super_num_devs
>>
>> The 2nd patch contains a compressed raw image, thus it may be a little
>> too large for the mail list.
>
> The compressed size is 22K, that's fine. I've recompressed it with 'xz
> --best -e' and it shaved a few hundred bytes but that was just out of
> curiosity.

I'm also a little interested in why things like zstd/xz is not that
efficient in compressing raw btrfs images.

Especially when creating the image I have created the image with all
zero (fallocated) file in the first place, thus most space should just
be all zero, and any compression method should be super efficient on them.

My current guesses are:

- DUP metadata is not fully detected by the algo
   Especially when they are mostly quite some ranges away

- Some older tree blocks from mkfs

So for further raw image size reduction, we may want to:

- Use SINGLE metadata/data

- Fstrim the image first

Thanks,
Qu

Re: [PATCH 1/2] btrfs-progs: check: add check and repair ability for super num devs mismatch

[Qu Wenruo]

On 2022/3/24 07:15, Qu Wenruo wrote:
>
>
> On 2022/3/24 01:43, David Sterba wrote:
>> On Mon, Feb 28, 2022 at 08:50:07AM +0800, Qu Wenruo wrote:
>>> [BUG]
>>> There is a bug report of kernel rejecting fs which has a mismatch in
>>> super num devices and num devices found in chunk tree.
>>>
>>> But btrfs-check reports no problem about the fs.
>>>
>>> [CAUSE]
>>> We just didn't verify super num devices against the result found in
>>> chunk tree.
>>>
>>> [FIX]
>>> Add such check and repair ability for btrfs-check.
>>>
>>> The ability is mode independent.
>>>
>>> Reported-by: Luca Béla Palkovics <luca.bela.palkovics@gmail.com>
>>> Link:
>>> https://lore.kernel.org/linux-btrfs/CA+8xDSpvdm_U0QLBAnrH=zqDq_cWCOH5TiV46CKmp3igr44okQ@mail.gmail.com/
>>>
>>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>>
>> With this patch applied there's a new test failure:
>>
>> === START TEST .../tests/fsck-tests/014-no-extent-info
>> testing image no_extent.raw.restored
>> ====== RUN MUSTFAIL .../btrfs check ./no_extent.raw.restored
>> Opening filesystem to check...
>> Checking filesystem on ./no_extent.raw.restored
>> UUID: aeee7297-37a4-4547-a8a9-7b870d58d31f
>> cache and super generation don't match, space cache will be invalidated
>> found 180224 bytes used, no error found
>> total csum bytes: 0
>> total tree bytes: 180224
>> total fs tree bytes: 81920
>> total extent tree bytes: 16384
>> btree space waste bytes: 138540
>> file data blocks allocated: 0
>>   referenced 0
>> succeeded (unexpected!): .../btrfs check ./no_extent.raw.restored
>> unexpected success: btrfs check should have detected corruption
>
>
> Weird, the problem is, for the restored image, if I run ./btrfs check
> manually it can detects the problem, but still go "no error found" path.
>
> So it must be my patch overriding the return value.

Indeed that's the case.

Fix upon current devel branch is:
https://lore.kernel.org/linux-btrfs/f576d7a548b91d42d02b17d2dc52ee04d943a57d.1648077794.git.wqu@suse.com/T/#u

Please fold the fix into the patch.

Thanks,
Qu
>
> Will fix it soon.
>
> Thanks,
> Qu

Re: [PATCH 0/2] btrfs-progs: add check and repair ability for super num devices mismatch

[David Sterba]

On Thu, Mar 24, 2022 at 07:30:31AM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/3/24 01:17, David Sterba wrote:
> > On Mon, Feb 28, 2022 at 08:50:06AM +0800, Qu Wenruo wrote:
> >> The patchset can be fetched from github:
> >> https://github.com/adam900710/btrfs-progs/tree/super_num_devs
> >>
> >> The 2nd patch contains a compressed raw image, thus it may be a little
> >> too large for the mail list.
> >
> > The compressed size is 22K, that's fine. I've recompressed it with 'xz
> > --best -e' and it shaved a few hundred bytes but that was just out of
> > curiosity.
> 
> I'm also a little interested in why things like zstd/xz is not that
> efficient in compressing raw btrfs images.

I think it is efficient, unpacked image was 128M and getting that to 22K
is a good result.

> Especially when creating the image I have created the image with all
> zero (fallocated) file in the first place, thus most space should just
> be all zero, and any compression method should be super efficient on them.
> 
> My current guesses are:
> 
> - DUP metadata is not fully detected by the algo
>    Especially when they are mostly quite some ranges away
> 
> - Some older tree blocks from mkfs
> 
> So for further raw image size reduction, we may want to:
> 
> - Use SINGLE metadata/data
> 
> - Fstrim the image first

Yeah that could remove any stale blocks.