From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB2863A1A2D for ; Fri, 10 Apr 2026 15:41:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.100 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775835686; cv=none; b=h/QINIGIDNTmQRK+GSRuiT4QH3/mkZPvepyWnO5KEJwaxFj1hMom3hCJCne5p6r+nYxqKFDSgFtFefjozb3VA1zUygK21565G66Sc9DYQ93hhHbigmIUXt9TkT2I5gDmoiUoa84165debXskUY2pNJ39pOfYGxvWFtXnbos1aYg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775835686; c=relaxed/simple; bh=h/s/ez2NcK8y1rxwkhec1BIoTJGci5947SV3NtJicoI=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=c9WUXYEPUQiwueJRv7su87QgwdR6xBwn6Zt5/l7WQXhhTpi6yWJtXgybP2TPu263I/+5sYMtta8MWDtuWagIW+QSHRLuMEArCouK+s2TUyiDTiCH6p2VNZSiVIlxIhZ4vWvBFN1Pn4dHUC+Obst66hoIlQrhksZJdh9l1fgTuTo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=Nu1vDjGu; arc=none smtp.client-ip=115.124.30.100 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="Nu1vDjGu" DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1775835676; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; bh=qpSbhb9yAAn81Tu6HnLT64SJVfnfTLUbKpOYf/dcx6U=; b=Nu1vDjGuO/5rasFhHKaGt/O1jvriw9qyCpC6nxIFg4b34MtFgeC0BGHSD2BkqDPBwjZJvcElkSqi/iz91Ii0RuKIsMVoZUEGEKMpF6wyNBNZ4SZPrepk/z2orj3HPT044FLoCiAJNv1eL5uWFnW79jGob7LQjPjBShunw8snmJY= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032089153;MF=hsiangkao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0X0lb6.n_1775835675; Received: from 30.41.54.139(mailfrom:hsiangkao@linux.alibaba.com fp:SMTPD_---0X0lb6.n_1775835675 cluster:ay36) by smtp.aliyun-inc.com; Fri, 10 Apr 2026 23:41:15 +0800 Message-ID: <8c0bdfab-dbf2-4f1e-8e2a-ce18f166d841@linux.alibaba.com> Date: Fri, 10 Apr 2026 23:41:14 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: erofs pointer corruption and kernel crash To: Arseniy Krasnov Cc: oxffffaa@gmail.com, linux-erofs@lists.ozlabs.org, linux-kernel@vger.kernel.org, kernel@salutedevices.com, Gao Xiang References: <4a2f3801-fac1-42fe-ae75-da315822e088@salutedevices.com> <2e916997-0557-45e7-831a-b436c07c5ba4@salutedevices.com> <97ca00c7-822d-4b57-9dc0-9b396049adc9@salutedevices.com> From: Gao Xiang In-Reply-To: <97ca00c7-822d-4b57-9dc0-9b396049adc9@salutedevices.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi Arseniy, On 2026/4/10 21:27, Arseniy Krasnov wrote: > > > 10.04.2026 15:20, Gao Xiang пишет: >> >> >> On 2026/4/10 19:37, Arseniy Krasnov wrote: >> >> (drop unrelated folks since they all subscribed erofs mailing list) >> >>> >>> >>> 10.04.2026 11:31, Gao Xiang wrote: >>>> Hi, >>>> >>>> On 2026/4/10 16:13, Arseniy Krasnov wrote: >>>>> Hi, >>>>> >>>>> We found unexpected behaviour of erofs: >>>>> >>>>> There is function in erofs - 'erofs_onlinefolio_end()'. It has pointer to >>>>> 'struct folio' as first argument, and there is loop inside this function, >>>>> which updates 'private' field of provided folio: >>>>> >>>>>     do { >>>>>             orig = atomic_read((atomic_t *)&folio->private); >>>>>             DBG_BUGON(orig <= 0); >>>>>             v = dirty << EROFS_ONLINEFOLIO_DIRTY; >>>>>             v |= (orig - 1) | (!!err << EROFS_ONLINEFOLIO_EIO); >>>>>     } while (atomic_cmpxchg((atomic_t *)&folio->private, orig, v) != orig); >>>>> >>>>> Now, we see that in some rare case, this function processes folio, where >>>>> 'private' is pointer, and thus this loop will update some bits in this >>>>> pointer. Then later kernel dereferences such pointer and crashes. >>>>> >>>>> To catch this, the following small debug patch was used (e.g. we check that 'private' field is pointer): >>>>> >>>>> diff --git a/fs/erofs/data.c b/fs/erofs/data.c >>>>> index 33cb0a7330d2..b1d8deffec4d 100644 >>>>> --- a/fs/erofs/data.c >>>>> +++ b/fs/erofs/data.c >>>>> @@ -238,6 +238,11 @@ void erofs_onlinefolio_end(struct folio *folio, int err, bool dirty) >>>>>    { >>>>>        int orig, v; >>>>>    +    if (((uintptr_t)folio->private) & 0xffff000000000000) { >>>> >>>> No, if erofs_onlinefolio_end() is called, `folio->private` >>>> shouldn't be a pointer, it's just a counter inside, and >>>> storing a pointer is unexpected. >>>> >>>> And since the folio is locked, it shouldn't call into >>>> try_to_free_buffers(). >>>> >>>> Is it easy to reproduce? if yes, can you print other >>>> values like `folio->mapping` and `folio->index` as >>>> well? >>>> >>>> I need more informations to find some clues. >>> >>> >>> >>> So reproduced again with this debug patch which adds magic to 'struct z_erofs_pcluster' and prints 'struct folio' >>> when pointer in 'private' is passed to 'erofs_onlinefolio_end()'. In short - 'private' points to 'struct z_erofs_pcluster'. >> First, erofs-utils 1.8.10 doesn't support `-E48bit`: >> only erofs-utils 1.9+ ship it as an experimental >> feature, see Changelog; so I think you're using >> modified erofs-utils 1.8.10: >> https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/tree/ChangeLog >> >> ``` >> erofs-utils 1.9 >> >>  * This release includes the following updates: >>    - Add 48-bit layout support for larger filesystems (EXPERIMENTAL); >> ``` >> >> Second, I'm pretty sure this issue is related to >> experimenal `-E48bit`, and those information is >> not enough for me to find the root cause, so I >> need to find a way to reproduce myself: It may >> take time; you could debug yourself but I don't >> think it's an easy task if you don't quite familiar >> with the EROFS codebase. >> >> Anyway I really suggest if you need a rush solution >> for production, don't use `-E48bit + zstd` like >> this for now: try to use other options like >> `-zzstd -C65536 -Efragments` instead since those >> are common production choices. > > Ok thanks for this advice! One more question: currently we use this options: > "zstd,22 --max-extent-bytes 65536 -E48bit". Ok we remove "zstd,22" and "E48bit", > but what about "--max-extent-bytes 65536" - is it considered stable option? > Or it is better to use your version: "-zzstd -C65536 -Efragments" ? I'm not sure how you find this "zstd,22 --max-extent-bytes 65536 -E48bit" combination. My suggestion based on production is that as long as you don't use `-zzstd` ++ `-E48bit`, it should be fine. If you need smaller images, I suggest: `-zlzma,9 -C65536 -Efragments` Or like Android, they all use `-zlz4hc`, Or zstd, but don't add `-E48bit`. As for "--max-extent-bytes 65536", it can be dropped since if `-E48bit` is not used, it only has negative impacts. In short, `-E48bit` + `-zzstd` + `--max-extent-bytes` enables new unaligned compression for zstd, but it's a relatively new feature, I still still some time to stablize it but my own time is limited and all things are always prioritized. Thanks, Gao Xiang > > Thanks > >> >> Thanks, >> Gao Xiang