From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 655DBC4363D for ; Fri, 25 Sep 2020 16:47:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D874621D42 for ; Fri, 25 Sep 2020 16:47:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mVEYFWer" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D874621D42 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 266F36B0078; Fri, 25 Sep 2020 12:47:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 219136B0081; Fri, 25 Sep 2020 12:47:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 109BD8E0001; Fri, 25 Sep 2020 12:47:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0242.hostedemail.com [216.40.44.242]) by kanga.kvack.org (Postfix) with ESMTP id EC9196B0078 for ; Fri, 25 Sep 2020 12:47:57 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B7B605003 for ; Fri, 25 Sep 2020 16:47:57 +0000 (UTC) X-FDA: 77302165794.13.kick05_410c73827169 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id 965E8181415A0 for ; Fri, 25 Sep 2020 16:47:57 +0000 (UTC) X-HE-Tag: kick05_410c73827169 X-Filterd-Recvd-Size: 6595 Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Fri, 25 Sep 2020 16:47:56 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id y4so3003225ljk.8 for ; Fri, 25 Sep 2020 09:47:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=rzPzwRJveHSctwZp3wrT4wsJ+vKh/2WbdUB6RmJgiVQ=; b=mVEYFWer0WJ/V7bBTxWewvWswsOXfCvL4L67MAfo71n9MaavrHxr+gH++cMXtE0aCc tjiNi8kvFv/h+VZhpRaOEIH4QFJvGTljbwrgnPn2eKNoBqS+R4vlb9jJ/fq1Ivf4sidk AMv0GQD4cHBrQczMW5xZ6oBfg/2a6KMKfxVF4iBnkFG9ZA6OievpgshAx1tehNUJOpf/ LhhyvgYhZuJICk7pnWzPUqH00A89KidWyOpxfGK/awCwLm35sfl0uF+Sqlgx1dmO9TQq xP88/mqY+5yYrhRlYUiFgXui98azowaWrX+h/Uhq7Pjj2ZyDWT+cl370XigGl2Q+uKUs kwzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=rzPzwRJveHSctwZp3wrT4wsJ+vKh/2WbdUB6RmJgiVQ=; b=kn1YXhPJlc6zZVETKWZpm3vsSQRwmlvhccM1lUJdDNyDs7bPjbBTu/usR5H5zVBHaH ok1HZ/CXpEtOw3P752/T8uWegLnNQFzfHzeS47A9u9Q+oe6yut9ayHvyRfhzf/QBg/iE blx8FfY9J4m+u1qi8Km0i2gzIOi8fAif2xReLtIi5Bv3AX3iZSdWjWCdKUfI3ROTm+/i Wt5oAdx3naeRqjKXgxBB8AUq/e3uaMA7yrvrCfZBOTDPvvH6qBzd2LbH4MsWuBFQV6SL AfOE3l1LdLuqBgPvgmmwzi0HalY53YUQFX5zJt1lCHJQy/J+D86Di5Lgal20vjE2BmAf yTHw== X-Gm-Message-State: AOAM530dbWMD+QUxosgZHaHIEAPg9IiCGmrf1jfhVK7YThyqEBz5yj8D TUC3iUkgv6APA2UCg9q6WmVqCk7rPpLw52vriRjNvA== X-Google-Smtp-Source: ABdhPJwJx0cGU4uB4elPs7ZF82yVS4VWNXuQL+KAAN+7j7jvra3jAOYCv6pmtiH02Yvg+WFL6sMc1f67KGybV/4BiBs= X-Received: by 2002:a2e:3511:: with SMTP id z17mr1525178ljz.58.1601052475022; Fri, 25 Sep 2020 09:47:55 -0700 (PDT) MIME-Version: 1.0 References: <20200915073303.GA754106@T590> <20200915224541.GB38283@mit.edu> <20200915230941.GA791425@T590> <20200916202026.GC38283@mit.edu> <20200917022051.GA1004828@T590> <20200917143012.GF38283@mit.edu> <20200924005901.GB1806978@T590> <20200924143345.GD482521@mit.edu> <20200925011311.GJ482521@mit.edu> <20200925073145.GC2388140@T590> <20200925161918.GD2388140@T590> In-Reply-To: From: Shakeel Butt Date: Fri, 25 Sep 2020 09:47:43 -0700 Message-ID: Subject: Re: REGRESSION: 37f4a24c2469: blk-mq: centralise related handling into blk_mq_get_driver_tag To: Ming Lei Cc: "Theodore Y. Ts'o" , Jens Axboe , linux-ext4@vger.kernel.org, "linux-kernel@vger.kernel.org" , "open list:BLOCK LAYER" , Linus Torvalds , Linux MM , Roman Gushchin , Andrew Morton , Johannes Weiner , Vlastimil Babka Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Sep 25, 2020 at 9:32 AM Shakeel Butt wrote: > > On Fri, Sep 25, 2020 at 9:19 AM Ming Lei wrote: > > > > On Fri, Sep 25, 2020 at 03:31:45PM +0800, Ming Lei wrote: > > > On Thu, Sep 24, 2020 at 09:13:11PM -0400, Theodore Y. Ts'o wrote: > > > > On Thu, Sep 24, 2020 at 10:33:45AM -0400, Theodore Y. Ts'o wrote: > > > > > HOWEVER, thanks to a hint from a colleague at $WORK, and realizing > > > > > that one of the stack traces had virtio balloon in the trace, I > > > > > realized that when I switched the GCE VM type from e1-standard-2 to > > > > > n1-standard-2 (where e1 VM's are cheaper because they use > > > > > virtio-balloon to better manage host OS memory utilization), problem > > > > > has become, much, *much* rarer (and possibly has gone away, although > > > > > I'm going to want to run a lot more tests before I say that > > > > > conclusively) on my test setup. At the very least, using an n1 VM > > > > > (which doesn't have virtio-balloon enabled in the hypervisor) is > > > > > enough to unblock ext4 development. > > > > > > > > .... and I spoke too soon. A number of runs using -rc6 are now > > > > failing even with the n1-standard-2 VM, so virtio-ballon may not be an > > > > indicator. > > > > > > > > This is why debugging this is frustrating; it is very much a heisenbug > > > > --- although 5.8 seems to work completely reliably, as does commits > > > > before 37f4a24c2469. Anything after that point will show random > > > > failures. :-( > > > > > > It does not make sense to mention 37f4a24c2469, which is reverted in > > > 4e2f62e566b5. Later the patch in 37f4a24c2469 is fixed and re-commited > > > as 568f27006577. > > > > > > However, I can _not_ reproduce the issue by running the same test on > > > kernel built from 568f27006577 directly. > > > > > > Also you have confirmed that the issue can't be fixed after reverting > > > 568f27006577 against v5.9-rc4. > > > > > > Looks the real issue(slab list corruption) should be introduced between > > > 568f27006577 and v5.9-rc4. > > > > git bisect shows the first bad commit: > > > > [10befea91b61c4e2c2d1df06a2e978d182fcf792] mm: memcg/slab: use a single set of > > kmem_caches for all allocations > > > > And I have double checked that the above commit is really the first bad > > commit for the list corruption issue of 'list_del corruption, ffffe1c241b00408->next > > is LIST_POISON1 (dead000000000100)', see the detailed stack trace and > > kernel oops log in the following link: > > > > https://lore.kernel.org/lkml/20200916202026.GC38283@mit.edu/ > > The failure signature is similar to > https://lore.kernel.org/lkml/20200901075321.GL4299@shao2-debian/ > > > > > And the kernel config is the one(without KASAN) used by Theodore in GCE VM, see > > the following link: > > > > https://lore.kernel.org/lkml/20200917143012.GF38283@mit.edu/ > > > > The reproducer is xfstests generic/038. In my setting, test device is virtio-scsi, and > > scratch device is virtio-blk. Is it possible to check SLUB as well to confirm that the issue is only happening on SLAB?