From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA2F8C433F5 for ; Tue, 5 Oct 2021 08:19:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 095C36136F for ; Tue, 5 Oct 2021 08:19:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 095C36136F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 151C56B006C; Tue, 5 Oct 2021 04:19:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 101C1900002; Tue, 5 Oct 2021 04:19:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F32926B0072; Tue, 5 Oct 2021 04:19:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0027.hostedemail.com [216.40.44.27]) by kanga.kvack.org (Postfix) with ESMTP id E03A56B006C for ; Tue, 5 Oct 2021 04:19:34 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 815152BFCC for ; Tue, 5 Oct 2021 08:19:34 +0000 (UTC) X-FDA: 78661684668.19.CF7D408 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf19.hostedemail.com (Postfix) with ESMTP id 053A2B001CE2 for ; Tue, 5 Oct 2021 08:19:33 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 840E221BCF; Tue, 5 Oct 2021 08:19:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1633421972; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hNwtCpdH8x2CNs+9d+1b7WmfBaR40WkJXzrsENFs2vM=; b=zAGJFrmv1FOiyXoFqNB0ypj1Y3YTOhyuVNz7C4UHlMtew8JEa0+P71+bV//pWqvlGQt988 Fsmt9GPgEJ3OSZn5+JRgtFFeRUfrC9+U64Zcr8y+8I/W+nLUQ6BfPO5sFH3oX/8vdsMROQ ebR1BILzeHWQ68d7r1Un0dbUM9gn5cc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1633421972; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hNwtCpdH8x2CNs+9d+1b7WmfBaR40WkJXzrsENFs2vM=; b=V0qXyZbuxwcXCTo6US79LggLmLKiA8/mOtGFypTLwltDGuE1xBO3iZOZeq98jXhkJ3RTtr TA6QYATx2OJVnIBg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 6093B13A78; Tue, 5 Oct 2021 08:19:32 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 6iTIFpQKXGE/YQAAMHmgww (envelope-from ); Tue, 05 Oct 2021 08:19:32 +0000 Message-ID: <09ca489a-ecfb-dd5e-b057-dc9c59c8585e@suse.cz> Date: Tue, 5 Oct 2021 10:19:32 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.1.2 Content-Language: en-US To: Christoph Lameter , Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org, Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , linux-kernel@vger.kernel.org References: <20210927090347.GA2533@linux.asia-northeast3-a.c.our-ratio-313919.internal> <8aa15f4b-71de-5283-5ebc-d8d1a323473d@suse.cz> <20211001003908.GA2657@linux.asia-northeast3-a.c.our-ratio-313919.internal> From: Vlastimil Babka Subject: Re: Queueing is outside of SLUB nowdays In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 053A2B001CE2 X-Stat-Signature: t8memia8h45nojj3h954qu7xxaea1uxo Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=zAGJFrmv; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=V0qXyZbu; dmarc=none; spf=pass (imf19.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-HE-Tag: 1633421973-596206 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 10/4/21 16:56, Christoph Lameter wrote: > On Fri, 1 Oct 2021, Hyeonggon Yoo wrote: > >> Looking at other layers, they implemented queuing layer outside of SLUB. >> See commit 795bb1c00dd ("net: bulk free infrastructure for NAPI context, >> use napi_consume_skb") for example. They made skb cache because SLUB is >> not suitable for intensive alloc/free. >> >> And because the queue is outside of slab, it can go lockless >> depending on it's context. (But it's not easy to do so in slab because >> slab is general purpose allocator.) > > The queuing within in SLUB/SLAB is lockless. > >> So current approach on place where slab's performance is critical >> is implementing queuing layer on top of slab. > > If you have to use object specific characteristics to optimize then yes > you can optimize further. However, the slab allocators implement each > their own form of queuing that is generic. > >> Then new question arising: >> - Is that proper way to solve fundamental problem? > > There is a problem? If someone benefits from implementing a caching layer on top of SL*B, it probably indicates a problem. >> - why not use SLAB if they need queuing? > > SLAB is LIFO queuing whereas SLUB uses spatial considerations and queues > within a page before going outside. IIUC SLUB queueing works well for allocation (we just consume a per-cpu freelist that nobody else can touch) but freeing uses the corresponding page's freelist so the atomics are more expensive. In both cases the linked freelists might be also worse for cache locality than an array of pointers. So perhaps some workload still benefit from a array-based cache on top of SLUB and it would be great if they didn't have to implement own solutions? > Slab requires disabling interrupts, > SLUB is optimized to rely on per cpu atomics and there are numerous other > differences. > >> - how does this approach work on SLAB? > > SLAB has a lockless layer that is only requiring disabling interrupts. It > provides a generic queuing layer as well. > > See my talk on Slab allocators awhile back. > > https://www.youtube.com/watch?v=h0VMLXavx30 >