From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.2 required=3.0 tests=DKIMWL_WL_HIGH,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5EC5C43381 for ; Tue, 19 Feb 2019 07:14:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 78FBA21903 for ; Tue, 19 Feb 2019 07:14:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="rIluS3pB"; dkim=pass (1024-bit key) header.d=fb.onmicrosoft.com header.i=@fb.onmicrosoft.com header.b="CsHp1jPz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725802AbfBSHOB (ORCPT ); Tue, 19 Feb 2019 02:14:01 -0500 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:38564 "EHLO mx0b-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725730AbfBSHOB (ORCPT ); Tue, 19 Feb 2019 02:14:01 -0500 Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x1J7DQXD014220; Mon, 18 Feb 2019 23:13:52 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : content-type : content-id : content-transfer-encoding : mime-version; s=facebook; bh=ZPpCqQtFwqciHf/SkNd587ZGZa/yYn3cppiHex2bE04=; b=rIluS3pBCUVF5hKeCA7vI6ppuC92uQFFG+nppgIG9vqpy+X0IbLEzsZ3FfusZxTSYl+1 hinYCrl9HiS3SLt7oixqVbpNe10kqx4O6IiYai6FmBmg/oKUpIKt8iEiZ2KpgDO2DwzQ oGbxDFfqi8Avt/4K5laPUKduMRIzqcuvo58= Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0a-00082601.pphosted.com with ESMTP id 2qrct481ur-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Mon, 18 Feb 2019 23:13:51 -0800 Received: from frc-mbx01.TheFacebook.com (2620:10d:c0a1:f82::25) by frc-hub05.TheFacebook.com (2620:10d:c021:18::175) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.1.1531.3; Mon, 18 Feb 2019 23:13:51 -0800 Received: from frc-hub04.TheFacebook.com (2620:10d:c021:18::174) by frc-mbx01.TheFacebook.com (2620:10d:c0a1:f82::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.1.1531.3; Mon, 18 Feb 2019 23:13:50 -0800 Received: from NAM01-SN1-obe.outbound.protection.outlook.com (192.168.183.28) by o365-in.thefacebook.com (192.168.177.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.1.1531.3 via Frontend Transport; Mon, 18 Feb 2019 23:13:50 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector1-fb-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZPpCqQtFwqciHf/SkNd587ZGZa/yYn3cppiHex2bE04=; b=CsHp1jPzknxBCvMAuWW6cF12gUT60uFB1BrIRGXUrL+az31kMF8gvZXBSwv7b70jBfIyL0rfqcTsPnXK+sc7oJ/0KFleyPMk7sObOx+JSx/v11WOaJDV89tRGDOEICPOS7vLPGtO0z8thG3jOv18sDE7UAlwPbT1ckFVdJvzAQU= Received: from BYAPR15MB2631.namprd15.prod.outlook.com (20.179.156.24) by BYASPR01MB0012.namprd15.prod.outlook.com (52.135.241.97) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1622.16; Tue, 19 Feb 2019 07:13:33 +0000 Received: from BYAPR15MB2631.namprd15.prod.outlook.com ([fe80::ecc7:1a8c:289f:df92]) by BYAPR15MB2631.namprd15.prod.outlook.com ([fe80::ecc7:1a8c:289f:df92%3]) with mapi id 15.20.1601.016; Tue, 19 Feb 2019 07:13:33 +0000 From: Roman Gushchin To: "lsf-pc@lists.linux-foundation.org" CC: "linux-fsdevel@vger.kernel.org" , "linux-mm@kvack.org" , "riel@surriel.com" , "dchinner@redhat.com" , "guroan@gmail.com" , Kernel Team , "hannes@cmpxchg.org" Subject: [LSF/MM TOPIC] dying memory cgroups and slab reclaim issues Thread-Topic: [LSF/MM TOPIC] dying memory cgroups and slab reclaim issues Thread-Index: AQHUyCKfeT5lZyV8PESiIPUMxsb/VQ== Date: Tue, 19 Feb 2019 07:13:33 +0000 Message-ID: <20190219071329.GA7827@castle.DHCP.thefacebook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BYAPR08CA0064.namprd08.prod.outlook.com (2603:10b6:a03:117::41) To BYAPR15MB2631.namprd15.prod.outlook.com (2603:10b6:a03:152::24) x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [2620:10d:c090:180::1:753b] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 72a1185d-9415-4e27-7c68-08d69639c1fe x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600110)(711020)(4605104)(2017052603328)(7153060)(7193020);SRVR:BYASPR01MB0012; x-ms-traffictypediagnostic: BYASPR01MB0012: x-ms-exchange-purlcount: 4 x-microsoft-exchange-diagnostics: 1;BYASPR01MB0012;20:Yu5iKlZpalGPZ8L2Wfw3VpYg4kYT/hZyZ/0tTuDyIBY8pwfPzkIkNHks0aClkoUJCp5nXuSj8vJ987G6E3J3EjfxtN0X99ZlZQx1gVqgfyCkcj6N6dxuyB5lY5VC1j1WngP5GtP813V6RT/hQqj9NPeHEBfLMtNOIWR1n7PwGvI= x-microsoft-antispam-prvs: x-forefront-prvs: 09538D3531 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(39860400002)(136003)(366004)(396003)(346002)(376002)(199004)(189003)(256004)(81166006)(486006)(6486002)(33896004)(966005)(5660300002)(97736004)(5640700003)(6436002)(71190400001)(8676002)(71200400001)(2351001)(476003)(478600001)(7736002)(14454004)(33656002)(25786009)(105586002)(6116002)(46003)(106356001)(4326008)(6506007)(386003)(2501003)(305945005)(102836004)(316002)(186003)(99286004)(53936002)(14444005)(1076003)(54906003)(8936002)(52116002)(86362001)(81156014)(68736007)(6916009)(6306002)(6512007)(9686003)(2906002);DIR:OUT;SFP:1102;SCL:1;SRVR:BYASPR01MB0012;H:BYAPR15MB2631.namprd15.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: fb.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: uIU+CNNbUdwhKtBS8tZjFlEk8yKH3GiMuGviIYegZ8kV4wkBM1aXNtcyY9JM/jTEC49LVuszv4WjM0+1pOQAg5RvMlxGhak5eG9FOFFOZHTsGf99vf9OWfX9QJzihDQd1iBPK/WE49pe7htNuH3aOwDR4j7qjPpLmEJ00BGobDTXey8vJYXfI/Z/2nM+XJfad2rGln1La9o+e52sz7KIiysHIj4beZig7V7gDtRdrR+O3kbRupD2PDd9YDDTYTE92JZaCjm74OUJfEZnc2uEqrBzhyH5KKD5aSL3iy963C/CVvnYYKPtiNDOoSTm9rfYFE484hko2QkFw+g3tSLoyVXFK+AW9GRwXCnTkpwD+vddRk5LD0CCghgSm6fO9Kd/GKBjI0VqsxXZpEITC0gc9RN+RAjQb6FA/ZMnTqTiT3k= Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 72a1185d-9415-4e27-7c68-08d69639c1fe X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Feb 2019 07:13:32.5392 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYASPR01MB0012 X-OriginatorOrg: fb.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-02-19_05:,, signatures=0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Sorry, once more, now with fsdevel@ in cc, asked by Dave. -- Recent reverts of memcg leak fixes [1, 2] reintroduced the problem with accumulating of dying memory cgroups. This is a serious problem: on most of our machines we've seen thousands on dying cgroups, and the corresponding memory footprint was measured in hundreds of megabytes. The problem was also independently discovered by other companies. The fixes were reverted due to xfs regression investigated by Dave Chinner. Simultaneously we've seen a very small (0.18%) cpu regression on some hosts= , which caused Rik van Riel to propose a patch [3], which aimed to fix the regression. The idea is to accumulate small memory pressure and apply it periodically, so that we don't overscan small shrinker lists. According to Jan Kara's data [4], Rik's patch partially fixed the regression, but not entirely. The path forward isn't entirely clear now, and the status quo isn't accepta= ble due to memcg leak bug. Dave and Michal's position is to focus on dying memo= ry cgroup case and apply some artificial memory pressure on corresponding slab= s (probably, during cgroup deletion process). This approach can theoretically be less harmful for the subtle scanning balance, and not cause any regressi= ons. In my opinion, it's not necessarily true. Slab objects can be shared betwee= n cgroups, and often can't be reclaimed on cgroup removal without an impact o= n the rest of the system. Applying constant artificial memory pressure precisely = only on objects accounted to dying cgroups is challenging and will likely cause a quite significant overhead. Also, by "forgetting" of some slab obje= cts under light or even moderate memory pressure, we're wasting memory, which c= an be used for something useful. Dying cgroups are just making this problem more obvious because of their size. So, using "natural" memory pressure in a way, that all slabs objects are sc= anned periodically, seems to me as the best solution. The devil is in details, an= d how to do it without causing any regressions, is an open question now. Also, completely re-parenting slabs to parent cgroup (not only shrinker lis= ts) is a potential option to consider. It will be nice to discuss the problem on LSF/MM, agree on general path and make a potential list of benchmarks, which can be used to prove the solutio= n. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/comm= it/?id=3Da9a238e83fbb0df31c3b9b67003f8f9d1d1b6c96 [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/comm= it/?id=3D69056ee6a8a3d576ed31e38b3b14c70d6c74edcc [3] https://lkml.org/lkml/2019/1/28/1865 [4] https://lkml.org/lkml/2019/2/8/336