From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out30-99.freemail.mail.aliyun.com (out30-99.freemail.mail.aliyun.com [115.124.30.99]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8615FC for ; Sun, 3 Dec 2023 08:52:36 -0800 (PST) X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R411e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=hsiangkao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VxgSxzN_1701622352; Received: from 30.27.64.151(mailfrom:hsiangkao@linux.alibaba.com fp:SMTPD_---0VxgSxzN_1701622352) by smtp.aliyun-inc.com; Mon, 04 Dec 2023 00:52:34 +0800 Message-ID: <5a0e8b44-6feb-b489-cdea-e3be3811804a@linux.alibaba.com> Date: Mon, 4 Dec 2023 00:52:31 +0800 Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: Weird EROFS data corruption To: Juhyung Park , Gao Xiang , linux-erofs@lists.ozlabs.org Cc: linux-f2fs-devel@lists.sourceforge.net, linux-crypto@vger.kernel.org, Yann Collet References: From: Gao Xiang In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi Juhyung, On 2023/12/4 00:22, Juhyung Park wrote: > (Cc'ing f2fs and crypto as I've noticed something similar with f2fs a > while ago, which may mean that this is not specific to EROFS: > https://lore.kernel.org/all/CAD14+f2nBZtLfLC6CwNjgCOuRRRjwzttp3D3iK4Of+1EEjK+cw@mail.gmail.com/ > ) > > Hi. > > I'm encountering a very weird EROFS data corruption. > > I noticed when I build an EROFS image for AOSP development, the device > would randomly not boot from a certain build. > After inspecting the log, I noticed that a file got corrupted. Is it observed on your laptop (i7-1185G7), yes? or some other arm64 device? > > After adding a hash check during the build flow, I noticed that EROFS > would randomly read data wrong. > > I now have a reliable method of reproducing the issue, but here's the > funny/weird part: it's only happening on my laptop (i7-1185G7). This > is not happening with my 128 cores buildfarm machine (Threadripper > 3990X).> > I first suspected a hardware issue, but: > a. The laptop had its motherboard replaced recently (due to a failing > physical Type-C port). > b. The laptop passes memory test (memtest86). > c. This happens on all kernel versions from v5.4 to the latest v6.6 > including my personal custom builds and Canonical's official Ubuntu > kernels. > d. This happens on different host SSDs and file-system combinations. > e. This only happens on LZ4. LZ4HC doesn't trigger the issue. > f. This only happens when mounting the image natively by the kernel. > Using fuse with erofsfuse is fine. I think it's a weird issue with inplace decompression because you said it depends on the hardware. In addition, with your dataset sadly I cannot reproduce on my local server (Xeon(R) CPU E5-2682 v4). What is the difference between these two machines? just different CPU or they have some other difference like different compliers? Thanks, Gao Xiang From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ECD5AC4167B for ; Sun, 3 Dec 2023 16:52:54 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4SjtC94Mdbz3cST for ; Mon, 4 Dec 2023 03:52:53 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.alibaba.com (client-ip=115.124.30.118; helo=out30-118.freemail.mail.aliyun.com; envelope-from=hsiangkao@linux.alibaba.com; receiver=lists.ozlabs.org) Received: from out30-118.freemail.mail.aliyun.com (out30-118.freemail.mail.aliyun.com [115.124.30.118]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4SjtC407hWz2yhT for ; Mon, 4 Dec 2023 03:52:44 +1100 (AEDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R411e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=hsiangkao@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VxgSxzN_1701622352; Received: from 30.27.64.151(mailfrom:hsiangkao@linux.alibaba.com fp:SMTPD_---0VxgSxzN_1701622352) by smtp.aliyun-inc.com; Mon, 04 Dec 2023 00:52:34 +0800 Message-ID: <5a0e8b44-6feb-b489-cdea-e3be3811804a@linux.alibaba.com> Date: Mon, 4 Dec 2023 00:52:31 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: Weird EROFS data corruption To: Juhyung Park , Gao Xiang , linux-erofs@lists.ozlabs.org References: From: Gao Xiang In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: linux-erofs@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development of Linux EROFS file system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Yann Collet , linux-crypto@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net Errors-To: linux-erofs-bounces+linux-erofs=archiver.kernel.org@lists.ozlabs.org Sender: "Linux-erofs" Hi Juhyung, On 2023/12/4 00:22, Juhyung Park wrote: > (Cc'ing f2fs and crypto as I've noticed something similar with f2fs a > while ago, which may mean that this is not specific to EROFS: > https://lore.kernel.org/all/CAD14+f2nBZtLfLC6CwNjgCOuRRRjwzttp3D3iK4Of+1EEjK+cw@mail.gmail.com/ > ) > > Hi. > > I'm encountering a very weird EROFS data corruption. > > I noticed when I build an EROFS image for AOSP development, the device > would randomly not boot from a certain build. > After inspecting the log, I noticed that a file got corrupted. Is it observed on your laptop (i7-1185G7), yes? or some other arm64 device? > > After adding a hash check during the build flow, I noticed that EROFS > would randomly read data wrong. > > I now have a reliable method of reproducing the issue, but here's the > funny/weird part: it's only happening on my laptop (i7-1185G7). This > is not happening with my 128 cores buildfarm machine (Threadripper > 3990X).> > I first suspected a hardware issue, but: > a. The laptop had its motherboard replaced recently (due to a failing > physical Type-C port). > b. The laptop passes memory test (memtest86). > c. This happens on all kernel versions from v5.4 to the latest v6.6 > including my personal custom builds and Canonical's official Ubuntu > kernels. > d. This happens on different host SSDs and file-system combinations. > e. This only happens on LZ4. LZ4HC doesn't trigger the issue. > f. This only happens when mounting the image natively by the kernel. > Using fuse with erofsfuse is fine. I think it's a weird issue with inplace decompression because you said it depends on the hardware. In addition, with your dataset sadly I cannot reproduce on my local server (Xeon(R) CPU E5-2682 v4). What is the difference between these two machines? just different CPU or they have some other difference like different compliers? Thanks, Gao Xiang From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EBF9EC4167B for ; Sun, 3 Dec 2023 17:08:52 +0000 (UTC) Received: from [127.0.0.1] (helo=sfs-ml-2.v29.lw.sourceforge.com) by sfs-ml-2.v29.lw.sourceforge.com with esmtp (Exim 4.95) (envelope-from ) id 1r9pxa-0001ft-I1; Sun, 03 Dec 2023 17:08:51 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-2.v29.lw.sourceforge.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1r9pxX-0001fk-GQ for linux-f2fs-devel@lists.sourceforge.net; Sun, 03 Dec 2023 17:08:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To: From:References:Cc:To:Subject:MIME-Version:Date:Message-ID:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=FowQ/5sWrP2WT5J/Mx1Lbri0pmb1bWbatWSIzB8moBY=; b=T0vR7V1eRQjpiXUP34JYavdzNX xAQocrLSNtQA8jzs6i3hiaQ9wbWkcZr2LGDsxh/cww0Lb5DfN/whc48Z2tn0lFOgbgyY5IY1OkXCO y8ULrFSqm5mfLCok3F50xJwHR82DBUOgG+T6LqseQMuHlt4wGJaoA0n1pYvCABkAy9ok=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:From:References:Cc:To: Subject:MIME-Version:Date:Message-ID:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=FowQ/5sWrP2WT5J/Mx1Lbri0pmb1bWbatWSIzB8moBY=; b=GXizPz2/lUVTnj8iJh+TI9xEbO jX42s1zfgiNc+bNyLpUmpfWh262lk4faWLXUcMEgzrzLk8RwODbq/WpaDWeOW+FephgdliaYGZ2Cs BzeUg7+FSyAh2z/1iNtz4Aa0DpXGNseAVi+ovHPT4hz0VVkXJjZLa5jOOVPuCOSc4a7k=; Received: from out30-118.freemail.mail.aliyun.com ([115.124.30.118]) by sfi-mx-2.v28.lw.sourceforge.com with esmtps (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.95) id 1r9pxW-0002LF-FU for linux-f2fs-devel@lists.sourceforge.net; Sun, 03 Dec 2023 17:08:48 +0000 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R411e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=ay29a033018045192; MF=hsiangkao@linux.alibaba.com; NM=1; PH=DS; RN=6; SR=0; TI=SMTPD_---0VxgSxzN_1701622352; Received: from 30.27.64.151(mailfrom:hsiangkao@linux.alibaba.com fp:SMTPD_---0VxgSxzN_1701622352) by smtp.aliyun-inc.com; Mon, 04 Dec 2023 00:52:34 +0800 Message-ID: <5a0e8b44-6feb-b489-cdea-e3be3811804a@linux.alibaba.com> Date: Mon, 4 Dec 2023 00:52:31 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 To: Juhyung Park , Gao Xiang , linux-erofs@lists.ozlabs.org References: From: Gao Xiang In-Reply-To: X-Headers-End: 1r9pxW-0002LF-FU Subject: Re: [f2fs-dev] Weird EROFS data corruption X-BeenThere: linux-f2fs-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Yann Collet , linux-crypto@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: linux-f2fs-devel-bounces@lists.sourceforge.net Hi Juhyung, On 2023/12/4 00:22, Juhyung Park wrote: > (Cc'ing f2fs and crypto as I've noticed something similar with f2fs a > while ago, which may mean that this is not specific to EROFS: > https://lore.kernel.org/all/CAD14+f2nBZtLfLC6CwNjgCOuRRRjwzttp3D3iK4Of+1EEjK+cw@mail.gmail.com/ > ) > > Hi. > > I'm encountering a very weird EROFS data corruption. > > I noticed when I build an EROFS image for AOSP development, the device > would randomly not boot from a certain build. > After inspecting the log, I noticed that a file got corrupted. Is it observed on your laptop (i7-1185G7), yes? or some other arm64 device? > > After adding a hash check during the build flow, I noticed that EROFS > would randomly read data wrong. > > I now have a reliable method of reproducing the issue, but here's the > funny/weird part: it's only happening on my laptop (i7-1185G7). This > is not happening with my 128 cores buildfarm machine (Threadripper > 3990X).> > I first suspected a hardware issue, but: > a. The laptop had its motherboard replaced recently (due to a failing > physical Type-C port). > b. The laptop passes memory test (memtest86). > c. This happens on all kernel versions from v5.4 to the latest v6.6 > including my personal custom builds and Canonical's official Ubuntu > kernels. > d. This happens on different host SSDs and file-system combinations. > e. This only happens on LZ4. LZ4HC doesn't trigger the issue. > f. This only happens when mounting the image natively by the kernel. > Using fuse with erofsfuse is fine. I think it's a weird issue with inplace decompression because you said it depends on the hardware. In addition, with your dataset sadly I cannot reproduce on my local server (Xeon(R) CPU E5-2682 v4). What is the difference between these two machines? just different CPU or they have some other difference like different compliers? Thanks, Gao Xiang _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel