From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC349C2FC14 for ; Thu, 17 Aug 2023 01:24:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347497AbjHQBYE (ORCPT ); Wed, 16 Aug 2023 21:24:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347501AbjHQBXc (ORCPT ); Wed, 16 Aug 2023 21:23:32 -0400 Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31E722721 for ; Wed, 16 Aug 2023 18:23:31 -0700 (PDT) Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-565f8b23151so202567a12.0 for ; Wed, 16 Aug 2023 18:23:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1692235410; x=1692840210; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=KSFMIKhRz8y1LlVzN4JBlWQlvqVf2idLZ8u0F6cnYUU=; b=gUoyYERSC36L4OWS0q5oV2c+ccT1RLS0gROqEwlAZPupSl2eFCUV4arQuYIEI12uY/ +jLZvPdh4KcgWqK1v7DfSF+5XT4PTN3tgfOzxKjtWL8G1MjiRPJblHYsJdptMX2EBXnQ k+B3yE6qSkNwqlq9+Q8ulsX+N2w374JxD8NW2qDlQOnWOgefs9hc68pin7cwZXB3OrY9 4/x5Iwmw7BA81x3V2HEKbyv6/4nHrkjB4WbuWHchdVaQW0Ligb+4C3SaXtdQNdTj/giR Vrpexu6pT2+Kn9MGPdmeBnHfes8zxvRVVrw+DRyxGhoZera+u4XqpQ1vUorUu9nhXTr1 EaWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692235410; x=1692840210; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KSFMIKhRz8y1LlVzN4JBlWQlvqVf2idLZ8u0F6cnYUU=; b=G36p+uuYMX9L7uYkWLxX/gdL7rsZ2apEdt8W9ttJhpKr/rqciyVh5P9KG0EXyy+hJP tv8Oz7C5co1k2llVBHMQkbuKJHUgYRaarMAUrV0kg7mat/AdCCN3h5ZzVLMEojPWYrWY jnnT/nQ3Pw+mD3vOBN7TOjvvkgavPhEmNJpZofWlvn6G6eQ2v9mttb86HCnY1MKdKI6i f0yeytNwGh8jpLZ+0bkd4AuQj0NekZqJF4ZnTa9o1lHtHAAg/H42NzxDS9EZXtN6G3oo gRlc+KIYVv8UCPpJ5yn/1gTp33PnCwNKR0ylGHGS9Tkobo6ljPnl8B+UAkmqaKF5FYL7 BSEw== X-Gm-Message-State: AOJu0YzgxAaAUyXppA+2A3RwX07O509QP/6plXI+h/6/UwKKaNl4CF7T 8ekJxZqJieWIz/UCJ5QS8gjvrw== X-Google-Smtp-Source: AGHT+IGaocOGI4L8ILTJCB0o1yEm6B9ZDaPg0s0JuiYtCAKrJwtnlnMNA9N3EiydHlS3UzO+6QXbGA== X-Received: by 2002:a17:90a:3d0d:b0:269:60ed:d493 with SMTP id h13-20020a17090a3d0d00b0026960edd493mr3097421pjc.4.1692235410586; Wed, 16 Aug 2023 18:23:30 -0700 (PDT) Received: from [192.168.1.136] ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 9-20020a17090a030900b0026b420ae167sm391727pje.17.2023.08.16.18.23.29 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 16 Aug 2023 18:23:30 -0700 (PDT) Message-ID: Date: Wed, 16 Aug 2023 19:23:28 -0600 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Possible io_uring related race leads to btrfs data csum mismatch Content-Language: en-US To: Qu Wenruo , "linux-btrfs@vger.kernel.org" , Linux FS Devel , io-uring@vger.kernel.org References: <95600f18-5fd1-41c8-b31b-14e7f851e8bc@gmx.com> <51945229-5b35-4191-a3f3-16cf4b3ffce6@kernel.dk> <2b3d6880-59c7-4483-9e08-3b10ac936d04@gmx.com> From: Jens Axboe In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On 8/16/23 7:19 PM, Qu Wenruo wrote: > On 2023/8/17 09:12, Jens Axboe wrote: >> On 8/16/23 7:05 PM, Qu Wenruo wrote: >>> >>> >>> On 2023/8/17 06:28, Jens Axboe wrote: >>> [...] >>>> >>>>>> 2) What's the .config you are using? >>>>> >>>>> Pretty common config, no heavy debug options (KASAN etc). >>>> >>>> Please just send the .config, I'd rather not have to guess. Things like >>>> preempt etc may make a difference in reproducing this. >>> >>> Sure, please see the attached config.gz >> >> Thanks >> >>>> And just to be sure, this is not mixing dio and buffered, right? >>> >>> I'd say it's mixing, there are dwrite() and writev() for the same file, >>> but at least not overlapping using this particular seed, nor they are >>> concurrent (all inside the same process sequentially). >>> >>> But considering if only uring_write is disabled, then no more reproduce, >>> thus there must be some untested btrfs path triggered by uring_write. >> >> That would be one conclusion, another would be that timing is just >> different and that triggers and issue. Or it could of course be a bug in >> io_uring, perhaps a short write that gets retried or something like >> that. I've run the tests for hours here and don't hit anything, I've >> pulled in the for-next branch for btrfs and see if that'll make a >> difference. I'll check your .config too. > > Just to mention, the problem itself was pretty hard to hit before if > using any debug kernel configs. The kernels I'm testing with don't have any debug options enabled, outside of the basic cheap stuff. I do notice you have all btrfs debug stuff enabled, I'll try and do that too. > Not sure why but later I switched both my CPUs (from a desktop i7-13700K > but with limited 160W power, to a laptop 7940HS), dropping all heavy > debug kernel configs, then it's 100% reproducible here. > > So I guess a faster CPU is also one factor? I've run this on kvm on an apple m1 max, no luck there. Ran it on a 7950X, no luck there. Fiddling config options on the 7950 and booting up the 7763 two socket box. Both that and the 7950 are using gen4 optane, should be plenty beefy. But if it's timing related, well... >> Might not be a bad idea to have the writes contain known data, and when >> you hit the failure to verify the csum, dump the data where the csum >> says it's wrong and figure out at what offset, what content, etc it is? >> If that can get correlated to the log of what happened, that might shed >> some light on this. >> > Thanks for the advice, would definitely try this method, would keep you > updated when I found something valuable. If I can't reproduce this, then this seems like the best way forward indeed. -- Jens Axboe