From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mtafr.prnet.org (mtafr.prnet.org [54.38.152.168]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 50E2A261B92; Thu, 12 Feb 2026 22:23:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=54.38.152.168 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770935028; cv=none; b=XCeAeC0M3HcnMFoQBFtE7LJ3udVTLqbvyUd4pjMZiJlVfFm9gvp9fc+sQ7d3NUXR23cxNWszr1uBHIriQa/YJVqv6wu5jn9jDxR6GTDxFAbv5OI9aSV+kGdQJVrte8huUnJ1ZdJbJuE/WgiiazgCRIBnoL5bhl8b0ef73FaD7/s= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770935028; c=relaxed/simple; bh=1lJZTEsVYEFdbl8KMEFBkfZm9PQynTuS91keAuDmlMg=; h=Message-ID:Date:MIME-Version:Subject:To:References:From: In-Reply-To:Content-Type; b=jPKW7fZ+j7htGyJu8SKtInyw2/js8nI1GGnJo4RdlyGOSNzgFnKjkuok9Tt0mtzB1W9qmqgCAAdQdp0YlMeqwGMCjwgKLyJNBBug3a4wyiy5pfVYETb2OskvPN7ldXnpMC2769cJYe6NgNt37spiKOpE8+2dGhF0VmXaaBL9VB8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=prnet.org; spf=pass smtp.mailfrom=prnet.org; arc=none smtp.client-ip=54.38.152.168 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=prnet.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=prnet.org Received: from secure.prnet.org (unknown [10.1.0.1]) by mtafr.prnet.org (Postfix) with ESMTP id 5409B2004D; Thu, 12 Feb 2026 23:24:10 +0100 (CET) Received: from [IPV6:2001:7e8:cf00:bc01:8cac:7fff:fe9e:52] (unknown [IPv6:2001:7e8:cf00:bc01:8cac:7fff:fe9e:52]) by secure.prnet.org (Postfix) with ESMTPSA id 86CC810003F; Thu, 12 Feb 2026 23:23:44 +0100 (CET) Message-ID: Date: Thu, 12 Feb 2026 23:23:44 +0100 Precedence: bulk X-Mailing-List: linux-btrfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Orange PI 5 MAX: very unstable using kernel 6.19.0 and 6.18.10, 6.18.9 perfectly stable To: Qu Wenruo , LKML , linux-rockchip@lists.infradead.org, Btrfs BTRFS References: <8ebe4d76-eb07-499b-b140-1f300c1b8d7e@prnet.org> Content-Language: en-US From: David Arendt In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/12/26 10:05 PM, Qu Wenruo wrote: > > > 在 2026/2/13 06:41, David Arendt 写道: >> Hello, >> >> I am using a Kubernetes Cluster with 3 Orange PI5 MAX nodes. The data >> is stored using a btrfs filesystem as backend. If using kernel 6.19.0 >> or kernel 6.18.10 I have experienced many crashes during high IO load >> on all 3 nodes. Reverting back to 6.18.9 solves the problems >> completely. Unfortunately the crashes are spontaneous reboots without >> leaving a trace in any logfile, so I have no stacktrace of them. >> After the crashes I have sometimes incorrect btrfs csums for a file >> but these may also be a result of a partial write due to the crash. >> On one node I had a btrfs error logged without crashing, but I am not >> sure if this is the root cause or a result of a prior crash. A scrub >> after reboot returned no error with 6.19.0. > > The offending tree dump items are: > > Feb 10 13:31:07 opi02 kernel:  item 92 key (13218356101120 > Feb 10 13:31:07 opi02 kernel:  item 93 key (13216208642048 > Feb 10 13:31:07 opi02 kernel:  item 94 key (13218356162560 > > Obviously item 93 is smaller than all its previous and next item keys. > > hex(13218356101120) = 0xc05a36b8000 > hex(13216208642048) = 0xc05236be000 > hex(13218356162560) = 0xc05a36c7000 > > It looks like something fliped, "0xc05a3" -> "0xc0523" > > 0xa -> 0x2 is exactly one bit flipped. > > So either the memory hardware has something wrong and resulting a > sticking bit (always 0), or there is something inside the kernel > touching memory it shouldn't. > > And this exactly matches the symptom, changing random bit of your > kernel, crash always expected. > > > Can you run a memtest to make sure it is not hardware problems first? Hello, I don't know of anything like memtest86 for the arm64 platform for testing the whole memory, so I used the user space memtester to check the 14G of unused ram on all 3 machines while using kernel 6.18.10. Here is the result of the first iteration (same on every machine): memtester version 4.7.1 (64-bit) Copyright (C) 2001-2024 Charles Cazabon. Licensed under the GNU General Public License version 2 (only). pagesize is 4096 pagesizemask is 0xfffffffffffff000 want 14000MB (14680064000 bytes) got  14000MB (14680064000 bytes), trying mlock ...locked. Loop 1:   Stuck Address       : ok   Random Value        : ok   Compare XOR         : ok   Compare SUB         : ok   Compare MUL         : ok   Compare DIV         : ok   Compare OR          : ok   Compare AND         : ok   Sequential Increment: ok   Solid Bits          : ok   Block Sequential    : ok   Checkerboard        : ok   Bit Spread          : ok   Bit Flip            : ok   Walking Ones        : ok   Walking Zeroes      : ok I don't think it is hardware a failure as it is happening on 3 different machines. Crashes occur somewhere between 30 minutes and 12 hours on all 3 machines that have been running without a single crash for more than a year now with older kernel versions including 4 days with 6.18.9 and all version from 6.18.0 to 6.18.9, so it seems to be caused by something that has changed between 6.18.9 and 6.18.10. Thanks, David Arendt > > Thanks, > Qu > > >> >> Unfortunately I don't have more information at the moment. >> >> Thanks in advance, >> >> David Arendt >> >> > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3D3E3EEA867 for ; Thu, 12 Feb 2026 22:23:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:To:Subject:MIME-Version: Date:Message-ID:Reply-To:Cc:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=25ZwekMEms8K9REdIoeTPfA3lkTplqas6L6Fclm5JRw=; b=Ne5fXfeoO0/qim3r+omfeyplMl 94R+0WpYX7zFEb/Kg95fHYAQoHKaL9darpFhUR4yHvl608l52wl2TPW7CGujaL+61PMFo30u/3dtX HrE/o7XZYs+V3rAoYOrKBsMN44Gh/khq+fd/odtW1Ugz2XkiYj+vKllQ8pcBUG9VJ7wpQYWF+23sp pLFgRjYJNCUFtllo09gilmVQ+bAvwhH+nqCxfVizOlKg1zbzkA4l63HLdNmkW/AiLorNPZy1fINGa 0Vz0+PhUzljHuFcYuaGT/PHjhyswxbxQrXAmBnTGmeN4yc962QcXXjitqWBMfatF4RY3bluGGu03f giJ8PEBA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1vqf6E-00000002hPP-2Kpo; Thu, 12 Feb 2026 22:23:50 +0000 Received: from mtafr.prnet.org ([54.38.152.168]) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1vqf6B-00000002hP3-2oV9 for linux-rockchip@lists.infradead.org; Thu, 12 Feb 2026 22:23:49 +0000 Received: from secure.prnet.org (unknown [10.1.0.1]) by mtafr.prnet.org (Postfix) with ESMTP id 5409B2004D; Thu, 12 Feb 2026 23:24:10 +0100 (CET) Received: from [IPV6:2001:7e8:cf00:bc01:8cac:7fff:fe9e:52] (unknown [IPv6:2001:7e8:cf00:bc01:8cac:7fff:fe9e:52]) by secure.prnet.org (Postfix) with ESMTPSA id 86CC810003F; Thu, 12 Feb 2026 23:23:44 +0100 (CET) Message-ID: Date: Thu, 12 Feb 2026 23:23:44 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Orange PI 5 MAX: very unstable using kernel 6.19.0 and 6.18.10, 6.18.9 perfectly stable To: Qu Wenruo , LKML , linux-rockchip@lists.infradead.org, Btrfs BTRFS References: <8ebe4d76-eb07-499b-b140-1f300c1b8d7e@prnet.org> Content-Language: en-US From: David Arendt In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260212_142347_892369_B0A7A00B X-CRM114-Status: GOOD ( 25.09 ) X-BeenThere: linux-rockchip@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Upstream kernel work for Rockchip platforms List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "Linux-rockchip" Errors-To: linux-rockchip-bounces+linux-rockchip=archiver.kernel.org@lists.infradead.org T24gMi8xMi8yNiAxMDowNSBQTSwgUXUgV2VucnVvIHdyb3RlOgo+Cj4KPiDlnKggMjAyNi8yLzEz IDA2OjQxLCBEYXZpZCBBcmVuZHQg5YaZ6YGTOgo+PiBIZWxsbywKPj4KPj4gSSBhbSB1c2luZyBh IEt1YmVybmV0ZXMgQ2x1c3RlciB3aXRoIDMgT3JhbmdlIFBJNSBNQVggbm9kZXMuIFRoZSBkYXRh IAo+PiBpcyBzdG9yZWQgdXNpbmcgYSBidHJmcyBmaWxlc3lzdGVtIGFzIGJhY2tlbmQuIElmIHVz aW5nIGtlcm5lbCA2LjE5LjAgCj4+IG9yIGtlcm5lbCA2LjE4LjEwIEkgaGF2ZSBleHBlcmllbmNl ZCBtYW55IGNyYXNoZXMgZHVyaW5nIGhpZ2ggSU8gbG9hZCAKPj4gb24gYWxsIDMgbm9kZXMuIFJl dmVydGluZyBiYWNrIHRvIDYuMTguOSBzb2x2ZXMgdGhlIHByb2JsZW1zIAo+PiBjb21wbGV0ZWx5 LiBVbmZvcnR1bmF0ZWx5IHRoZSBjcmFzaGVzIGFyZSBzcG9udGFuZW91cyByZWJvb3RzIHdpdGhv dXQgCj4+IGxlYXZpbmcgYSB0cmFjZSBpbiBhbnkgbG9nZmlsZSwgc28gSSBoYXZlIG5vIHN0YWNr dHJhY2Ugb2YgdGhlbS4gCj4+IEFmdGVyIHRoZSBjcmFzaGVzIEkgaGF2ZSBzb21ldGltZXMgaW5j b3JyZWN0IGJ0cmZzIGNzdW1zIGZvciBhIGZpbGUgCj4+IGJ1dCB0aGVzZSBtYXkgYWxzbyBiZSBh IHJlc3VsdCBvZiBhIHBhcnRpYWwgd3JpdGUgZHVlIHRvIHRoZSBjcmFzaC4gCj4+IE9uIG9uZSBu b2RlIEkgaGFkIGEgYnRyZnMgZXJyb3IgbG9nZ2VkIHdpdGhvdXQgY3Jhc2hpbmcsIGJ1dCBJIGFt IG5vdCAKPj4gc3VyZSBpZiB0aGlzIGlzIHRoZSByb290IGNhdXNlIG9yIGEgcmVzdWx0IG9mIGEg cHJpb3IgY3Jhc2guIEEgc2NydWIgCj4+IGFmdGVyIHJlYm9vdCByZXR1cm5lZCBubyBlcnJvciB3 aXRoIDYuMTkuMC4KPgo+IFRoZSBvZmZlbmRpbmcgdHJlZSBkdW1wIGl0ZW1zIGFyZToKPgo+IEZl YiAxMCAxMzozMTowNyBvcGkwMiBrZXJuZWw6wqAgaXRlbSA5MiBrZXkgKDEzMjE4MzU2MTAxMTIw Cj4gRmViIDEwIDEzOjMxOjA3IG9waTAyIGtlcm5lbDrCoCBpdGVtIDkzIGtleSAoMTMyMTYyMDg2 NDIwNDgKPiBGZWIgMTAgMTM6MzE6MDcgb3BpMDIga2VybmVsOsKgIGl0ZW0gOTQga2V5ICgxMzIx ODM1NjE2MjU2MAo+Cj4gT2J2aW91c2x5IGl0ZW0gOTMgaXMgc21hbGxlciB0aGFuIGFsbCBpdHMg cHJldmlvdXMgYW5kIG5leHQgaXRlbSBrZXlzLgo+Cj4gaGV4KDEzMjE4MzU2MTAxMTIwKSA9IDB4 YzA1YTM2YjgwMDAKPiBoZXgoMTMyMTYyMDg2NDIwNDgpID0gMHhjMDUyMzZiZTAwMAo+IGhleCgx MzIxODM1NjE2MjU2MCkgPSAweGMwNWEzNmM3MDAwCj4KPiBJdCBsb29rcyBsaWtlIHNvbWV0aGlu ZyBmbGlwZWQsICIweGMwNWEzIiAtPiAiMHhjMDUyMyIKPgo+IDB4YSAtPiAweDIgaXMgZXhhY3Rs eSBvbmUgYml0IGZsaXBwZWQuCj4KPiBTbyBlaXRoZXIgdGhlIG1lbW9yeSBoYXJkd2FyZSBoYXMg c29tZXRoaW5nIHdyb25nIGFuZCByZXN1bHRpbmcgYSAKPiBzdGlja2luZyBiaXQgKGFsd2F5cyAw KSwgb3IgdGhlcmUgaXMgc29tZXRoaW5nIGluc2lkZSB0aGUga2VybmVsIAo+IHRvdWNoaW5nIG1l bW9yeSBpdCBzaG91bGRuJ3QuCj4KPiBBbmQgdGhpcyBleGFjdGx5IG1hdGNoZXMgdGhlIHN5bXB0 b20sIGNoYW5naW5nIHJhbmRvbSBiaXQgb2YgeW91ciAKPiBrZXJuZWwsIGNyYXNoIGFsd2F5cyBl eHBlY3RlZC4KPgo+Cj4gQ2FuIHlvdSBydW4gYSBtZW10ZXN0IHRvIG1ha2Ugc3VyZSBpdCBpcyBu b3QgaGFyZHdhcmUgcHJvYmxlbXMgZmlyc3Q/CgpIZWxsbywKCkkgZG9uJ3Qga25vdyBvZiBhbnl0 aGluZyBsaWtlIG1lbXRlc3Q4NiBmb3IgdGhlIGFybTY0IHBsYXRmb3JtIGZvciAKdGVzdGluZyB0 aGUgd2hvbGUgbWVtb3J5LCBzbyBJIHVzZWQgdGhlIHVzZXIgc3BhY2UgbWVtdGVzdGVyIHRvIGNo ZWNrIAp0aGUgMTRHIG9mIHVudXNlZCByYW0gb24gYWxsIDMgbWFjaGluZXMgd2hpbGUgdXNpbmcg a2VybmVsIDYuMTguMTAuCgpIZXJlIGlzIHRoZSByZXN1bHQgb2YgdGhlIGZpcnN0IGl0ZXJhdGlv biAoc2FtZSBvbiBldmVyeSBtYWNoaW5lKToKCm1lbXRlc3RlciB2ZXJzaW9uIDQuNy4xICg2NC1i aXQpCkNvcHlyaWdodCAoQykgMjAwMS0yMDI0IENoYXJsZXMgQ2F6YWJvbi4KTGljZW5zZWQgdW5k ZXIgdGhlIEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNlIHZlcnNpb24gMiAob25seSkuCgpwYWdl c2l6ZSBpcyA0MDk2CnBhZ2VzaXplbWFzayBpcyAweGZmZmZmZmZmZmZmZmYwMDAKd2FudCAxNDAw ME1CICgxNDY4MDA2NDAwMCBieXRlcykKZ290wqAgMTQwMDBNQiAoMTQ2ODAwNjQwMDAgYnl0ZXMp LCB0cnlpbmcgbWxvY2sgLi4ubG9ja2VkLgpMb29wIDE6CiDCoCBTdHVjayBBZGRyZXNzwqAgwqAg wqAgwqA6IG9rCiDCoCBSYW5kb20gVmFsdWXCoCDCoCDCoCDCoCA6IG9rCiDCoCBDb21wYXJlIFhP UsKgIMKgIMKgIMKgIMKgOiBvawogwqAgQ29tcGFyZSBTVULCoCDCoCDCoCDCoCDCoDogb2sKIMKg IENvbXBhcmUgTVVMwqAgwqAgwqAgwqAgwqA6IG9rCiDCoCBDb21wYXJlIERJVsKgIMKgIMKgIMKg IMKgOiBvawogwqAgQ29tcGFyZSBPUsKgIMKgIMKgIMKgIMKgIDogb2sKIMKgIENvbXBhcmUgQU5E wqAgwqAgwqAgwqAgwqA6IG9rCiDCoCBTZXF1ZW50aWFsIEluY3JlbWVudDogb2sKIMKgIFNvbGlk IEJpdHPCoCDCoCDCoCDCoCDCoCA6IG9rCiDCoCBCbG9jayBTZXF1ZW50aWFswqAgwqAgOiBvawog wqAgQ2hlY2tlcmJvYXJkwqAgwqAgwqAgwqAgOiBvawogwqAgQml0IFNwcmVhZMKgIMKgIMKgIMKg IMKgIDogb2sKIMKgIEJpdCBGbGlwwqAgwqAgwqAgwqAgwqAgwqAgOiBvawogwqAgV2Fsa2luZyBP bmVzwqAgwqAgwqAgwqAgOiBvawogwqAgV2Fsa2luZyBaZXJvZXPCoCDCoCDCoCA6IG9rCgpJIGRv bid0IHRoaW5rIGl0IGlzIGhhcmR3YXJlIGEgZmFpbHVyZSBhcyBpdCBpcyBoYXBwZW5pbmcgb24g MyBkaWZmZXJlbnQgCm1hY2hpbmVzLiBDcmFzaGVzIG9jY3VyIHNvbWV3aGVyZSBiZXR3ZWVuIDMw IG1pbnV0ZXMgYW5kIDEyIGhvdXJzIG9uIGFsbCAKMyBtYWNoaW5lcyB0aGF0IGhhdmUgYmVlbiBy dW5uaW5nIHdpdGhvdXQgYSBzaW5nbGUgY3Jhc2ggZm9yIG1vcmUgdGhhbiBhIAp5ZWFyIG5vdyB3 aXRoIG9sZGVyIGtlcm5lbCB2ZXJzaW9ucyBpbmNsdWRpbmcgNCBkYXlzIHdpdGggNi4xOC45IGFu ZCBhbGwgCnZlcnNpb24gZnJvbSA2LjE4LjAgdG8gNi4xOC45LCBzbyBpdCBzZWVtcyB0byBiZSBj YXVzZWQgYnkgc29tZXRoaW5nIAp0aGF0IGhhcyBjaGFuZ2VkIGJldHdlZW4gNi4xOC45IGFuZCA2 LjE4LjEwLgoKVGhhbmtzLAoKRGF2aWQgQXJlbmR0Cgo+Cj4gVGhhbmtzLAo+IFF1Cj4KPgo+Pgo+ PiBVbmZvcnR1bmF0ZWx5IEkgZG9uJ3QgaGF2ZSBtb3JlIGluZm9ybWF0aW9uIGF0IHRoZSBtb21l bnQuCj4+Cj4+IFRoYW5rcyBpbiBhZHZhbmNlLAo+Pgo+PiBEYXZpZCBBcmVuZHQKPj4KPj4KPgoK Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCkxpbnV4LXJv Y2tjaGlwIG1haWxpbmcgbGlzdApMaW51eC1yb2NrY2hpcEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0 dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtcm9ja2NoaXAK