From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6A48C3ABDD for ; Mon, 19 May 2025 16:08:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73E7C6B008C; Mon, 19 May 2025 12:08:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C5A66B00CC; Mon, 19 May 2025 12:08:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 519086B00D1; Mon, 19 May 2025 12:08:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 300796B008C for ; Mon, 19 May 2025 12:08:50 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 797441409D0 for ; Mon, 19 May 2025 16:08:51 +0000 (UTC) X-FDA: 83460140862.12.934253C Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by imf03.hostedemail.com (Postfix) with ESMTP id 3F18820008 for ; Mon, 19 May 2025 16:08:49 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=gwVcOngO; spf=pass (imf03.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.194 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=gwVcOngO; spf=pass (imf03.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.194 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747670929; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=upX6IpvOqfloTUvspsgVLqbnsnYfH2bP1tZTrioWapk=; b=7qUqG9W3szZEyLaWlF/EePzbOZo+Vn1OZnEuGYmkDYtwBMN7zbdf4n9kPJr2yBavt70zCo OcKWjcn3L6vAbwXFkHM/DUk1DamzW85dM8SRdVR9kXFaqE/nWmjllU9NLcy6hNaJoAFbm3 prnW6jMeCdtBuAe10F5Zb0YI6DSgkz8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747670929; a=rsa-sha256; cv=none; b=axj+gJXjEel/pwKWLiEREXMLWNqzl9BFSpg0342a461QBp3TbLJUfznD4k3Wt/HaZDeq8X ev3gMqoNcqlSRcsLyfQFfpnQaeShiZ1qT8UKXjwHKdxZCaqGek2vU3FQ9Q3gngUDPpAdDX PdnIhwVbLGMQcnXdl8kOXlxINjbIIw0= Received: by mail-qk1-f194.google.com with SMTP id af79cd13be357-7c9376c4dbaso523551185a.0 for ; Mon, 19 May 2025 09:08:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1747670928; x=1748275728; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=upX6IpvOqfloTUvspsgVLqbnsnYfH2bP1tZTrioWapk=; b=gwVcOngO+ZcyTwD6qfnqBk8sN87mf7NbHyNmoxKK1t6HVns8czw6NMb3WCZbvS8AVU fxxyZVm+io4Qz1B+Xnaq28ge+V8TLhFZyLa96In4lFP+Q1ZFNw+Pub/qxwSS7jQa4FiL jkeUIMAE4YuyJuDtJbEuFIHj8D7GZ5RrccQRgPN0ksr2K17DNY/LIy1AiIYTI+aCDosr 7Zq7rNsUFSDh9sn+rmj/ts4tCDNuE+znHij92fd89gjpU6BJl0qz5CKhvUZ004cbN6q8 0sH+QBC22oCwbi7D+4cTEVqgDLojnufmokwrmY6LRDFWutAjB2JDnHptIFnU/E2FQsTc k1eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747670928; x=1748275728; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=upX6IpvOqfloTUvspsgVLqbnsnYfH2bP1tZTrioWapk=; b=WzK84Wu/ami4O7bwmkgIyU+7ooKcnnmsmlsOoN6bQyjgcob6DP70ywPohBkA0AsT+u WEihkijHR3911ST/SMWbSbtUusyZ4HLFnWff3APgCRnxA3SyKVtEa1l4umpqq5RDgc85 OXyLqvHKWD7iw4TpcYvjufREpN7h/kYTurwQ3PhuZOxoUkH8LFenFY9m+Rro6CpH/9ye 6ijLROOEBht0ZSWibO6PIYB/uUno/b/G/1Y7rklDCmahpfSYy+CsWlNWmw3CkyurSiWy enslVtBZl2f21IZOaHxQCjl+md7UdejKfj3pIjt7zAvoo5JIc9SRnFT9w1EJFXj1D56K qKJw== X-Forwarded-Encrypted: i=1; AJvYcCVe4W4gMqw2VH5wGwYFHfbvcNdUr7y0Nq4ngsNLD/RMdaqzllQq9fyuxloqjZcHMXWY4O+/VsL9hg==@kvack.org X-Gm-Message-State: AOJu0YwIwC7/W/fBm0dQkwa6j9kLQw8LAQrwYAmPxqZ8kDuAz0T3CwYj fiz75NIqNaXFNJWTOpMnAWaeYk3GHaFUlDCZRNmPYB3R/zXP8uomjh3dptNQHeWsl+s= X-Gm-Gg: ASbGnculerAg/8rno2LtCdpQ0b72xd7Qyx2orgEEGbCHYTKGDafeDDw9r6emoLyFJ7E HhUzeWodXZqPVgXtXrlKMlQhpQAVg+wt8ttcaAvWwp2H8YTOfviTxb5OfefzuxlX5GZ5Qw3URoL sWCfYfSMgX/jN+1RsmPbm667glb4HgEecHJaHVNwAe0h7Lu/4jkhSZo5pwP1ZW5IFdl0STuHvQ2 ftgxBjt7rdhrfdd9SaL49T5mP3tR6kfpgMRfHQUhOIZUP5UC2ylTuSRuiwLRXoDTEjXCiLnKpN/ qdOqXs5z21CTzFCv+wQMPhzEDC1us15isPoFoM+ARwkojpOjqw== X-Google-Smtp-Source: AGHT+IFCpCXmmAG/Ctq2VEZt3jMlYTRvOR82GvLrrjQCgiUqpgo9pFiwiQ/IWBkiALA2IWv9m8mbtA== X-Received: by 2002:a05:620a:bd5:b0:7c4:bca3:6372 with SMTP id af79cd13be357-7cd469dad0amr2180895385a.0.1747670927943; Mon, 19 May 2025 09:08:47 -0700 (PDT) Received: from localhost ([2603:7000:c01:2716:365a:60ff:fe62:ff29]) by smtp.gmail.com with UTF8SMTPSA id af79cd13be357-7cd467d93d5sm597913185a.31.2025.05.19.09.08.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 May 2025 09:08:46 -0700 (PDT) Date: Mon, 19 May 2025 12:08:46 -0400 From: Johannes Weiner To: Suren Baghdasaryan Cc: Usama Arif , Shakeel Butt , Linux Memory Management List , kent.overstreet@linux.dev, vlad.wing@gmail.com Subject: Re: Memory allocation profiling warnings in memory bound systems Message-ID: <20250519160846.GA773385@cmpxchg.org> References: <17fab2d6-5a74-4573-bcc3-b75951508f0a@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3F18820008 X-Stat-Signature: wu3s6r85bhnt16nc1jc13oe1bmf134bp X-Rspam-User: X-HE-Tag: 1747670929-433545 X-HE-Meta: U2FsdGVkX1/i2zIPLcPEPK2ajjYNSWRYEEii55h+aIti8mUdbtZbOrcG9MNCAo2AhZqGawZUtaqUGEMRZf6P/HSD5hAaO9768EGu3mB2KAw3DWUGBncSzxt+40JyUu4IhY7Cpk5z4QyHotMisVbfqBWH5FYbBHw7FdkQCv6eyc/6b3PO3adQiVxBbQKbk6QXoFF8/29XWumkVyga/KlXSBj+6EcxhvsGVWW9s9yeOTXt4KVju7f828pRcT+31K5XEraSJ3QRa2OSlrLC5z1py2WmSMrsBApSljqc+gFlAu66BDoaXaqDds/M4LXHL7jrQ6uENogQk+Nitu+RvCtXKavE6f+12JNS+GkODaX7Bbr59pw4+yWcZGv0nAMShKJNFAQeAyC5cQSBO6ihYF1LAqUzsSTWKxS8Jnd7i2vQX4xGzY+gvk7eKMURRk5t58TYdIBiUKsb87/v6v+qEfBTIo2r2ikOuGf2z4JhYAwtMwpaKfFbO2kmh4MX+kQzqMhMDWG67VMTmKvwxIeRuZtqieHqpHlmYg+K6dihWS9NSiuVXa3qR08otU5VszOP/VfarQPPy3ZriLL8MTyFoQViWtq5ERQGodqUkAjW/ufnKYgmtZiVT8LaOqszgVKnogF/KzTDBgZMEBPAmJ3PMTWCt/hGc/jBZqU8P51eVe2jsfkbBeovjt7SbgB1eEkoKbV+ga2TgBa4Yh5sc5gbvrmht9pHIaT0Hsut00uIpAE5pzfTbhbcn9OTzIoRJOQdnJ061Xy7KOy2m/5F+OOi07cpghZ6quZbT0h1qQPAqQXVuqZXW81a1eKlhJjsn9rtbQrVSx0xF46cC8DqC97CGR5u1G5Sw0PL1AtD6gPm3Ft+wJkScF+t++2Sw00t8i7TXCQq/L1RRHyBztVvfXBAhcNgFhg1mXxM7c6v4DaVKrLs1QdcdKVbajtXicVqSUL2ywKDrbxoK5/lJp8U0KTWIBe WhBKAUs0 EXB/4zN4gz3DpSt/pK28EoKHEbFDApQZFFY4pPfDH5swoxjB4nBVBPsUck1M15LnE63kIsea29R8nDzqlIrTfpSO7nVyy3YuhPA8JRqoVZWE4dJC6BW17jTkmJztNtphTS/nalxDvJnzkPGGFwOgWjLkkBEmUEUc5+hmc3lmi2tbbVlOSSWEoCoFQP7il6SAqzLAYtoKZH6yytTv6h3PWpgFqNjHUzCsbL5fEKuMe/SiwjHh0uXBWf3v/ugxYbopbJOgI6OtaPNFyhkXSCooeEha5NFkVGHir+1AC672v87bSNkO76YVlxrQ/s4tFxKm6egpsOy2uctDMt/1pqfPjlAzlLPTxDR5AQP20BOLQjgMpxgZ2uMBQMx321mph2xFFyCnYxJE3f3tA7AC/3sH0jBQOQNGWOUp5ZI3pOTJc+NCdLa9KR0VzserlCLZHZPJ0Fivl21Lh1Dw+ZoEuRZPccfKlJ9AYp25w5dUg7pIHu5tjIoUOR2TenOhxHT9LmiSdko8Ox56f4130HnkR6d3nT+TlBr+PVyLY+DCCILlt9x7AoVZWIyb8s0RN9rOD6ic2as+rvgcLXoPCSBnadtQObRRMyaQyl7+8x3TkGpPb03xrN2oTsKJi/ihKkHjVawUR/zEG7J3bM5jueFo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, May 19, 2025 at 08:50:28AM -0700, Suren Baghdasaryan wrote: > On Mon, May 19, 2025 at 6:33 AM Usama Arif wrote: > > > > > > +cc Vlad > > > > On 19/05/2025 14:31, Usama Arif wrote: > > > Hi, > > > > > > We have started enabling memory allocation profiling (with kernel 6.13) in our fleet > > > and are seeing a large number of warnings (reported by Vlad Poenaru) due to failure > > > in allocation of slab object extensions on services that are memory bound. I have attached > > > one of the logs at the end. > > > > > > Does it make sense to change the slabobj_ext to be allocated via kvcalloc and also change > > > the WARN to WARN_ONCE (or maybe even pr_debug?) like the diff below? A large number of > > > prints for this in a short time may mask any real issues in the system during memory > > > pressure being reported in dmesg. I tried to see if there were any changes after 6.13 > > > to this code but didn't find any, but thought will check before sending below as a patch. > > > > > > diff --git a/mm/slub.c b/mm/slub.c > > > index c2151c9fee22..4595ca190cd9 100644 > > > --- a/mm/slub.c > > > +++ b/mm/slub.c > > > @@ -1961,7 +1961,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, > > > gfp &= ~OBJCGS_CLEAR_MASK; > > > /* Prevent recursive extension vector allocation */ > > > gfp |= __GFP_NO_OBJ_EXT; > > > - vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, > > > + vec = kvcalloc_node(objects, sizeof(struct slabobj_ext), gfp, > > Hi Usama, > Is the allocation larger than page size? IIUC, unless allocation size > is over PAGE_SIZE, kvcalloc_node() will not fall back to vmalloc (see: > https://elixir.bootlin.com/linux/v6.14.7/source/mm/util.c#L668). How > big is the allocation when it fails in your case? Digging through the reports, it appears we're encountering both. We've seen a zswap slab where the slab is order-0 and slabext is higher-order (8 byte objects, 512 objsperslab, 1 pageperslab), but also biovec-max where it's the other way round (4k byte objects, 8 objsperslab, 8 pagesperslab). In the first case, vmalloc would help. In the second it wouldn't. The second case is interesting. The higher-order slab succeeds because bios use a mempool; but the system is so depleted that the order-0 for the slabext fails. I'm not sure there is much we can do about this tbh. It would seem overkill to add a mempool or grant the tracking access to system-wide emergency reserves. A warn-once would probably make sense nonetheless. It might also make sense to flag the line item for that callsite in the reporting file, to make it obvious that the counter is compromised and is missing allocations?