From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9F6EC433E1 for ; Tue, 19 May 2020 08:45:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B5268204EF for ; Tue, 19 May 2020 08:45:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1589877943; bh=vxc7D6Zk6ZmGktlWR27RKzhjbNWsFvuijN1fpLUdzJw=; h=Date:From:To:Cc:Subject:References:In-Reply-To:List-ID:From; b=JaT9Dn3I4Q6pHSERGjYP+dwv9v2sJ2Y6zUU7PXU2HcqIEFG2JxBh2QC8NGpscU/WZ VkHaWvEd6JxDpCyS3RbZlyb195KT25VMIAucSIjZJn9vUo9p39uUe3BeVCB7NWOuFJ uARTYBA9Zn5oKZW6hujoQKPWUsBwuhr92U+HW2WI= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726880AbgESIpk (ORCPT ); Tue, 19 May 2020 04:45:40 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:35916 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726121AbgESIpk (ORCPT ); Tue, 19 May 2020 04:45:40 -0400 Received: by mail-wr1-f65.google.com with SMTP id k13so12815342wrx.3; Tue, 19 May 2020 01:45:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=VLlTscmlbpCKZ4nor9NSeLRkKjW9ch72hYbKNbV2XL8=; b=NfXM7akETbG7URg31ox9h9K/LBeQ+CVi20pudJwQP7mXavEN86P21iihMnby12++/N E2nDKFayhnaDUUQW7DT0ymnYMjuQJlHgFGQ1eaLeRgS3dmm2bQSgN+/KUNfHqQy8aWx6 XX+SGglZRC2fu+vNYNYQiLi6EyqiVgcr6BszPP71J1qCGT7EUYEeErChicBrVuaMQoCC I11kLWXWQjXLgkKYBJboxJkeEpRyLIoimKTamuvPPWjzPofeLXTVABDfsuu3HNbMt/Fo KKEy7yMKE2aXHRmdP+ivr7SstvnulZaFw94K3JfSTIMFqSvadC9qXHjC4UttCtWgYZe2 xzGw== X-Gm-Message-State: AOAM531WUQnf1kbpPc7YUUjYzA8dvorXoavmjg+Ulk4xHZmaBQf3XtVp XQcSRXZ3ziv1jHVZe2K6bFw= X-Google-Smtp-Source: ABdhPJwX9uniXfChgB/t9L9lwBsmYEKUSMJMO45yp0yCMYOVHIGMwp7lYEjNDLfh+Oj66I63vmhbKw== X-Received: by 2002:adf:ed06:: with SMTP id a6mr23890678wro.8.1589877937465; Tue, 19 May 2020 01:45:37 -0700 (PDT) Received: from localhost (ip-37-188-176-234.eurotel.cz. [37.188.176.234]) by smtp.gmail.com with ESMTPSA id j2sm19737914wrp.47.2020.05.19.01.45.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 01:45:36 -0700 (PDT) Date: Tue, 19 May 2020 10:45:35 +0200 From: Michal Hocko To: Arnd Bergmann Cc: Naresh Kamboju , "Linux F2FS DEV, Mailing List" , linux-ext4 , linux-block , Andrew Morton , open list , Linux-Next Mailing List , linux-mm , Andreas Dilger , Jaegeuk Kim , Theodore Ts'o , Chao Yu , Hugh Dickins , Andrea Arcangeli , Matthew Wilcox , Chao Yu , lkft-triage@lists.linaro.org Subject: Re: mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page Message-ID: <20200519084535.GG32497@dhcp22.suse.cz> References: <20200501135806.4eebf0b92f84ab60bba3e1e7@linux-foundation.org> <20200519075213.GF32497@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On Tue 19-05-20 10:11:25, Arnd Bergmann wrote: > On Tue, May 19, 2020 at 9:52 AM Michal Hocko wrote: > > > > On Mon 18-05-20 19:40:55, Naresh Kamboju wrote: > > > Thanks for looking into this problem. > > > > > > On Sat, 2 May 2020 at 02:28, Andrew Morton wrote: > > > > > > > > On Fri, 1 May 2020 18:08:28 +0530 Naresh Kamboju wrote: > > > > > > > > > mkfs -t ext4 invoked oom-killer on i386 kernel running on x86_64 device > > > > > and started happening on linux -next master branch kernel tag next-20200430 > > > > > and next-20200501. We did not bisect this problem. > > [...] > > > Creating journal (131072 blocks): [ 31.251333] mkfs.ext4 invoked > > > oom-killer: gfp_mask=0x101cc0(GFP_USER|__GFP_WRITE), order=0, > > > oom_score_adj=0 > > [...] > > > [ 31.500943] DMA free:187396kB min:22528kB low:28160kB high:33792kB > > > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > > > active_file:4736kB inactive_file:431688kB unevictable:0kB > > > writepending:62020kB present:783360kB managed:668264kB mlocked:0kB > > > kernel_stack:888kB pagetables:0kB bounce:0kB free_pcp:880kB > > > local_pcp:216kB free_cma:163840kB > > > > This is really unexpected. You are saying this is a regular i386 and DMA > > should be bottom 16MB while yours is 780MB and the rest of the low mem > > is in the Normal zone which is completely missing here. How have you got > > to that configuration? I have to say I haven't seen anything like that > > on i386. > > I think that line comes from an ARM32 beaglebone-X15 machine showing > the same symptom. The i386 line from the log file that Naresh linked to at > https://lkft.validation.linaro.org/scheduler/job/1406110#L1223 is less > unusual: OK, that makes more sense! At least for the memory layout. > [ 34.931663] Node 0 active_anon:21464kB inactive_anon:8688kB > active_file:16604kB inactive_file:849976kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB mapped:25284kB dirty:58952kB > writeback:27772kB shmem:8944kB writeback_tmp:0kB unstable:0kB > all_unreclaimable? yes > [ 34.955523] DMA free:3356kB min:68kB low:84kB high:100kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > active_file:0kB inactive_file:11964kB unevictable:0kB > writepending:11980kB present:15964kB managed:15876kB mlocked:0kB > kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB > free_cma:0kB > [ 34.983385] lowmem_reserve[]: 0 825 1947 825 > [ 34.987678] Normal free:3948kB min:7732kB low:8640kB high:9548kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > active_file:1096kB inactive_file:786400kB unevictable:0kB > writepending:65432kB present:884728kB managed:845576kB mlocked:0kB > kernel_stack:1112kB pagetables:0kB bounce:0kB free_pcp:2908kB > local_pcp:500kB free_cma:0kB The lowmem is really low (way below the min watermark so even memory reserves for high priority and atomic requests are depleted. There is still 786MB of inactive page cache to be reclaimed. It doesn't seem to be dirty or under the writeback but it still might be pinned by the filesystem. I would suggest watching vmscan reclaim tracepoints and check why the reclaim fails to reclaim anything. -- Michal Hocko SUSE Labs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80047C433DF for ; Tue, 19 May 2020 08:45:49 +0000 (UTC) Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 454B9204EF for ; Tue, 19 May 2020 08:45:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=sourceforge.net header.i=@sourceforge.net header.b="IshEd9zq"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=sf.net header.i=@sf.net header.b="F97gN1Bu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 454B9204EF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linux-f2fs-devel-bounces@lists.sourceforge.net Received: from [127.0.0.1] (helo=sfs-ml-1.v29.lw.sourceforge.com) by sfs-ml-1.v29.lw.sourceforge.com with esmtp (Exim 4.90_1) (envelope-from ) id 1jaxsg-0005H2-U9; Tue, 19 May 2020 08:45:46 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-1.v29.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jaxsf-0005GI-88 for linux-f2fs-devel@lists.sourceforge.net; Tue, 19 May 2020 08:45:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=In-Reply-To:Content-Type:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=VLlTscmlbpCKZ4nor9NSeLRkKjW9ch72hYbKNbV2XL8=; b=IshEd9zqvpvOLW4kMhpgrZZ/G2 /jv2aOuCMzayO7gs4KMM65us5tK8rO5OiaZUYujKE/vVzMI5QsS3uifdPeuQ85kVPRFQcsoH1lxlP sUO3J9zEXudw/0OKvUqMyvsGEWBo0hm7MqVM0EmzY3yGWmPISxicr7Avtql8hjoQcGsE=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To :From:Date:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=VLlTscmlbpCKZ4nor9NSeLRkKjW9ch72hYbKNbV2XL8=; b=F97gN1BuSYqWscmk+odqHQ/gAC EBz7v2bYu7FCXJxEvxlxhQpdXWEfgqZL1AKeb/EugZlXyCW1L+xoMyqWLA+Th+Hmf7RInqPjm1pLj 7my7isZSAov3npKUA7sGlJ9JyhGnr/ZJ8y+v/eTvrd0HEpVaWohzGjClGuO55bX8Nrh4=; Received: from mail-wr1-f66.google.com ([209.85.221.66]) by sfi-mx-1.v28.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.92.2) id 1jaxsd-0073ag-PP for linux-f2fs-devel@lists.sourceforge.net; Tue, 19 May 2020 08:45:45 +0000 Received: by mail-wr1-f66.google.com with SMTP id e16so14888807wra.7 for ; Tue, 19 May 2020 01:45:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=VLlTscmlbpCKZ4nor9NSeLRkKjW9ch72hYbKNbV2XL8=; b=MXikr5gctooh8966E5YZSZKZCyAoYrhxFqXXnYx4X8O8DpZ1beMW+Ak7SCSnWm9e16 wnm4N37kFvsLneulfh7Lpsl4rou6kvh5/42T8PatDf/oy4EA4RIMPY0te4+TelY3pLiL bnY+0I9ZIQOICDrRXnJw3yJ+PL+lGSlsUTtnG9IkcPy6YGLwyOixQeynUafzyLvISP9t 8Y8Zae/yI6IByOX+u2InLIHr0FrClp98X7b5nzafsVgFwy9QEHq6xJVsYXkeU10r+W+D x8f0AZ4QjAZ4yRws7aqsHSAAlMblUVQBKGQXc9b+XWDXCZkTrlsiQcz5bKBiSHCozUKs 5kCA== X-Gm-Message-State: AOAM531Jlyv9V3qqOeHRxYOSHhTZOfUGBJKror3K8c4E8mBH0v3OsNRP eRKYvlZsBGnSLH40xa4u0nE= X-Google-Smtp-Source: ABdhPJwX9uniXfChgB/t9L9lwBsmYEKUSMJMO45yp0yCMYOVHIGMwp7lYEjNDLfh+Oj66I63vmhbKw== X-Received: by 2002:adf:ed06:: with SMTP id a6mr23890678wro.8.1589877937465; Tue, 19 May 2020 01:45:37 -0700 (PDT) Received: from localhost (ip-37-188-176-234.eurotel.cz. [37.188.176.234]) by smtp.gmail.com with ESMTPSA id j2sm19737914wrp.47.2020.05.19.01.45.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 01:45:36 -0700 (PDT) Date: Tue, 19 May 2020 10:45:35 +0200 From: Michal Hocko To: Arnd Bergmann Message-ID: <20200519084535.GG32497@dhcp22.suse.cz> References: <20200501135806.4eebf0b92f84ab60bba3e1e7@linux-foundation.org> <20200519075213.GF32497@dhcp22.suse.cz> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Headers-End: 1jaxsd-0073ag-PP Subject: Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page X-BeenThere: linux-f2fs-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrea Arcangeli , Theodore Ts'o , Naresh Kamboju , Hugh Dickins , open list , Matthew Wilcox , "Linux F2FS DEV, Mailing List" , linux-block , linux-mm , Linux-Next Mailing List , Andreas Dilger , lkft-triage@lists.linaro.org, Jaegeuk Kim , Andrew Morton , linux-ext4 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-f2fs-devel-bounces@lists.sourceforge.net On Tue 19-05-20 10:11:25, Arnd Bergmann wrote: > On Tue, May 19, 2020 at 9:52 AM Michal Hocko wrote: > > > > On Mon 18-05-20 19:40:55, Naresh Kamboju wrote: > > > Thanks for looking into this problem. > > > > > > On Sat, 2 May 2020 at 02:28, Andrew Morton wrote: > > > > > > > > On Fri, 1 May 2020 18:08:28 +0530 Naresh Kamboju wrote: > > > > > > > > > mkfs -t ext4 invoked oom-killer on i386 kernel running on x86_64 device > > > > > and started happening on linux -next master branch kernel tag next-20200430 > > > > > and next-20200501. We did not bisect this problem. > > [...] > > > Creating journal (131072 blocks): [ 31.251333] mkfs.ext4 invoked > > > oom-killer: gfp_mask=0x101cc0(GFP_USER|__GFP_WRITE), order=0, > > > oom_score_adj=0 > > [...] > > > [ 31.500943] DMA free:187396kB min:22528kB low:28160kB high:33792kB > > > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > > > active_file:4736kB inactive_file:431688kB unevictable:0kB > > > writepending:62020kB present:783360kB managed:668264kB mlocked:0kB > > > kernel_stack:888kB pagetables:0kB bounce:0kB free_pcp:880kB > > > local_pcp:216kB free_cma:163840kB > > > > This is really unexpected. You are saying this is a regular i386 and DMA > > should be bottom 16MB while yours is 780MB and the rest of the low mem > > is in the Normal zone which is completely missing here. How have you got > > to that configuration? I have to say I haven't seen anything like that > > on i386. > > I think that line comes from an ARM32 beaglebone-X15 machine showing > the same symptom. The i386 line from the log file that Naresh linked to at > https://lkft.validation.linaro.org/scheduler/job/1406110#L1223 is less > unusual: OK, that makes more sense! At least for the memory layout. > [ 34.931663] Node 0 active_anon:21464kB inactive_anon:8688kB > active_file:16604kB inactive_file:849976kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB mapped:25284kB dirty:58952kB > writeback:27772kB shmem:8944kB writeback_tmp:0kB unstable:0kB > all_unreclaimable? yes > [ 34.955523] DMA free:3356kB min:68kB low:84kB high:100kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > active_file:0kB inactive_file:11964kB unevictable:0kB > writepending:11980kB present:15964kB managed:15876kB mlocked:0kB > kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB > free_cma:0kB > [ 34.983385] lowmem_reserve[]: 0 825 1947 825 > [ 34.987678] Normal free:3948kB min:7732kB low:8640kB high:9548kB > reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB > active_file:1096kB inactive_file:786400kB unevictable:0kB > writepending:65432kB present:884728kB managed:845576kB mlocked:0kB > kernel_stack:1112kB pagetables:0kB bounce:0kB free_pcp:2908kB > local_pcp:500kB free_cma:0kB The lowmem is really low (way below the min watermark so even memory reserves for high priority and atomic requests are depleted. There is still 786MB of inactive page cache to be reclaimed. It doesn't seem to be dirty or under the writeback but it still might be pinned by the filesystem. I would suggest watching vmscan reclaim tracepoints and check why the reclaim fails to reclaim anything. -- Michal Hocko SUSE Labs _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel