From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E705C02180 for ; Thu, 16 Jan 2025 09:23:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE183280002; Thu, 16 Jan 2025 04:23:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A915E280001; Thu, 16 Jan 2025 04:23:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95905280002; Thu, 16 Jan 2025 04:23:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 776C0280001 for ; Thu, 16 Jan 2025 04:23:02 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2D85E44626 for ; Thu, 16 Jan 2025 09:23:02 +0000 (UTC) X-FDA: 83012775804.01.71A030C Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf01.hostedemail.com (Postfix) with ESMTP id 5139E40009 for ; Thu, 16 Jan 2025 09:23:00 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=i79qKEZ8; spf=pass (imf01.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737019380; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=7xRWpo+T7uUV2bVwHTa7TpcAURmustR4SiBpyzSbmGY=; b=IOatudaIgrerYoomoSf+5l/e/V5FBZWvUczofIjTx0Sr+bDZRFP4C9J40BT0RW9EjQG25V jo6NYADkp4CWqN8TQGDYMjK6d67Ssbnb13wopnufpRcyWuuiF2C0K8qqbf8+7aCxqiDY4O GZWT1WdQ4WbeqGzaqbRHkqZcOnA2jZM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737019380; a=rsa-sha256; cv=none; b=PMovobUBplyckza/hzXfB+HbJeDQ1TnGfbwjwkj90J+2TTIsOkkx5ZIvQog5NOBFNBqQba xXbhN9/csc1hm5lMipVjurvgzb55bbYWqiRJaFoZAmvU0qLMLkOLCGkYfwrDvB0KBMPXJ1 WBE6RnY/aWclcrhZxwKf9ll8BNPXB+g= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=i79qKEZ8; spf=pass (imf01.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-21649a7bcdcso10452375ad.1 for ; Thu, 16 Jan 2025 01:23:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737019379; x=1737624179; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=7xRWpo+T7uUV2bVwHTa7TpcAURmustR4SiBpyzSbmGY=; b=i79qKEZ8zmsNv3unIKPyBUyvq3Wco7x00gMrOxk224S9CUesmclobwcbC2otV+xJu6 yCP5nuFL8M273MAp0rWHks+ghPkR6hy4B1wTt3/uDzjbkTfAm/qgu0fHSA5SJ/VpPQV9 bmfVgg2c1BLHN3ves7npkNK85eWt9h69184IogaPxGdVfR2KI1uNtSwxyZiJTX8yorSZ C5iEIU0eIiSvV2jLci6yg4LZv8yj+VupMKwn6i1ilZqmx4qQKEulbO4gDNoV/h8xJtrl cuDWkQZl3xUO1r7nDAnujbYFN1xksrWuhatrDLKkqfQRm+awH2WkyxmFQiWZqbZzNqAf IqQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737019379; x=1737624179; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=7xRWpo+T7uUV2bVwHTa7TpcAURmustR4SiBpyzSbmGY=; b=khsFuRT37nTj+PjeoCqFIn4ZZyUNwMme4ExNZsRSIdH3gl4Lf9Wvle8akZPVUB46k3 eA1UrujjyBT2JqPMVPJ8KOXKoo4SPFmUMgpPF3ed6HsUv1GuX6F8AR/gSnI/TV6kFbJj ECu0Hu5Pd1fD4VgRTa5QyiWsCAxZenzBXotFLi4KP1NLs2F2JomBi2ULDzvWtzKHGXXu ZRmpxv7T+aS13SH+4SMwljN62szvkI/Z8/h2XFVnp2gcnQ+arfUaBsvOymKDgKemt96/ MHpEqBhSBROLeWpOPo6K0tr5fzhXpBtxyIiZZK/NUjG9r4Kl8f/5Al1sfDLHjD/dMT8j k6Ng== X-Forwarded-Encrypted: i=1; AJvYcCXjWtifMA1MeNQBW871oyhqy8Vw8HF4yZL2XOjlqp5H3X40eJwmpmQJKazlV7Ww3NE+aVa80xY0mA==@kvack.org X-Gm-Message-State: AOJu0YzuZCxLl8+wXFm5O4nQNfjuEb7Xd0EGLTFIGbQcswcJX6NkcV8N HE1wAgZxa9ZN83ZbIRbKooD8rneUuPypqvMIuxyS7MpAT4yHKLS6 X-Gm-Gg: ASbGncvD5DUM9wzc54hL6eoE0Dq/sbiwCUuZ/ZMRQb85KFEyF3EErf+UOyDxs7Ak2WE Sgf1m+t+uFw0OEDZVJVwhYojPr4U8Q3yiw4mAODujw5k4lNK2tncbkAXmZV0jil8UEw0gzOGBCf wv+effc6nFh/ekf8Oo3g/19s1U1uL8pwPzkhyeifa4PnKdIoJcX/od/YPx08rNR/YhEVwH2QoPu LngB/E2CE4Xtqp5XobM4qwI8yxYqXjxVvSCJXUYP+dsmUU= X-Google-Smtp-Source: AGHT+IG+hGvwVWEKkEmMNIBmLVXYEOB7+s/MtGgovJ800P0ZGy4dP6w4jiNWq1tDZ3gCVZ2BRwyeMA== X-Received: by 2002:a05:6a00:3e01:b0:725:e37d:cd35 with SMTP id d2e1a72fcca58-72d21ff4af7mr50049961b3a.18.1737019378738; Thu, 16 Jan 2025 01:22:58 -0800 (PST) Received: from localhost ([1.54.215.56]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72d4054942dsm10368393b3a.21.2025.01.16.01.22.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 16 Jan 2025 01:22:58 -0800 (PST) From: Nhat Pham To: lsf-pc@lists.linux-foundation.org, akpm@linux-foundation.org, hannes@cmpxchg.org Cc: ryncsn@gmail.com, chengming.zhou@linux.dev, yosryahmed@google.com, chrisl@kernel.org, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, shakeel.butt@linux.dev, hch@infradead.org, hughd@google.com, 21cnbao@gmail.com, usamaarif642@gmail.com Subject: [LSF/MM/BPF TOPIC] Virtual Swap Space Date: Thu, 16 Jan 2025 01:22:54 -0800 Message-Id: <20250116092254.204549-1-nphamcs@gmail.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Stat-Signature: iu7ob6u7eqkbo3ona6nqcihj5rrhjt14 X-Rspam-User: X-Rspamd-Queue-Id: 5139E40009 X-Rspamd-Server: rspam03 X-HE-Tag: 1737019380-842551 X-HE-Meta: U2FsdGVkX19HwxeIVa9ldqShrrdNlOEt9XhEmtYcG9LWQpLtBz9J6fo96QdvpYLJD3S2IfcxIlVEdeB3KHh4klxU4gJq71XJHXF6tFlZO3N/AG5WOEuvrycbjsM85/Y8DxnAf7PbijpUJFjuLi81HcDyCR4BGhF8R+0v13hUh9nYZvfyyo3QEzdq7Z+60sZc6MoiNrxd/adxS88LlU5C96bwyPTK8eroU5bAGVLgbqQOFeSbFzPOtw21vkEs7ypupHIGFrF8JC1ANbtdbgUis3WNRwpBVIAUZ812VpwvLKBEuSWfxnFDCuvwoGJL+E9s4OKrZiDOkCTxtEpewCcNCWljgljs6g5FsPt32jdCvb2SkuRs9ibVa3lgKO5oj/LhdqyYAUo26+xgqRaQ0F7VDaaFVJSd70+jlLO1NudXuFHIJCWQrpZmYOSmVIuOPUaP0YKW0IQNAx0OtKZ+isLh4QWvZ1TlBdVTKeEwhn6Px5xGBV6lYVonLZBE8M/gbpJnYRS6f7q4ltI7SOJgOZwBGqPZraWIA5VJkpGdVA3/dtEZxCm6mIY4TpcEKn0Jm9kr6nm+ZK2ZQFjR/5my3UXSJY4pro1GEIB6DtPwp+suuYqKPJqtdMV0W3EFlIi323/aAlisXOVXrBeBuUHfSaPjw6rGkXeWUvyeQyciBmAHaquRdrDh+SxhlMVCpAtuxhOsMwVhkQYP/fdDzNRR+KlYN+cp5Qn2UlFsKMCkhbTRnMvCdKNZRaTu7j60k78UXw9PW/XPTMOsxsZaVCozn09qXrVYd24mYMxkgiI1DSdnaHamXD7zsStTV61TvS5PbthTffHieqOggIe8ORBJSJfFzy0JFjTxIj6T46k5NqQOq9sEfEo1+tLbFWGMVGwOwsxrmW+U9rSD8qe1U95jPCw4RwO2p5NGhJRItJ7kAXM7IqK3RZwLS3jnl0TA7H3X7VU3wp5+srJVWq10PQaiGkc k7B3MVjh pT9qWuiXHxyuFo7fc1u0sW7nsB1o5JvIrdN5X6KuD20GBepkVIOJls5RsyGbYqElob+mwQqpaMrOCNtTceDHu2wiel59wrRE8oNd3lOuw/Xw/I8TlqQbKZcAGuhX8IafnOWVU9wTfoFXShsp10pt38Knr1t4cfxYlk1FB3AezhMKrePXyL71qScnxHDqjfECE+wI08OLM0Nrbi6GP+KuSzeEf3h336NoKSNbnP3B/LnsTk6hu6Go5deeX2KETbyvKzW9N/4fHaPORBI53txE1lMV5WqpcSNdx6oSNsWekw2xiidWArqfUnwlPoZ+UE2CiZ4BNCYDz/W/QIVrpDZGhSXNy/pVGHtehATqwdDSrnCyQjDq4A6bys5K+U/MrYmaR9+NPbDVJ+/XkgpTXohfYnSvIFE6Pj2I8yq6AbdtzWUtVgQQD0MRQYsOW3/cbXjrY1zfuoWvfUHoNSdv5SI3wwjVokfx4VfQ7/LCHN2NWyw3RmX35c/xUoC1L4ZHZp5vujGhSFHhbrOsduVm+ie7DgNpNUN8bKfMhpn8flWN0imbHbkqZlDqN0rMtTW3zA9pDI0HU6zq6AzrNLHH26DFkXurCgnvMeKpr87Wd12BRIZgOfAsF8uswR5dCHaGYqLkqNi69 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: My apologies if I missed any interested party in the cc list - hopefully the mailing lists cc's suffice :) I'd like to (re-)propose the topic of swap abstraction layer for the conference, as a continuation of Yosry's proposals at LSFMMBPF 2023 (see [1], [2], [3]). (AFAICT, the same idea has been floated by Rik van Riel since at least 2011 - see [8]). I have a working(-ish) prototype, which hopefully will be submission-ready soon. For now, I'd like to give the motivation/context for the topic, as well as some high level design: I. Motivation Currently, when an anon page is swapped out, a slot in a backing swap device is allocated and stored in the page table entries that refer to the original page. This slot is also used as the "key" to find the swapped out content, as well as the index to swap data structures, such as the swap cache, or the swap cgroup mapping. Tying a swap entry to its backing slot in this way is performant and efficient when swap is purely just disk space, and swapoff is rare. However, the advent of many swap optimizations has exposed major drawbacks of this design. The first problem is that we occupy a physical slot in the swap space, even for pages that are NEVER expected to hit the disk: pages compressed and stored in the zswap pool, zero-filled pages, or pages rejected by both of these optimizations when zswap writeback is disabled. This is the arguably central shortcoming of zswap: * In deployments when no disk space can be afforded for swap (such as mobile and embedded devices), users cannot adopt zswap, and are forced to use zram. This is confusing for users, and creates extra burdens for developers, having to develop and maintain similar features for two separate swap backends (writeback, cgroup charging, THP support, etc.). For instance, see the discussion in [4]. * Resource-wise, it is hugely wasteful in terms of disk usage, and limits the memory saving potentials of these optimizations by the static size of the swapfile, especially in high memory systems that can have up to terabytes worth of memory. It also creates significant challenges for users who rely on swap utilization as an early OOM signal. Another motivation for a swap redesign is to simplify swapoff, which is complicated and expensive in the current design. Tight coupling between a swap entry and its backing storage means that it requires a whole page table walk to update all the page table entries that refer to this swap entry, as well as updating all the associated swap data structures (swap cache, etc.). II. High Level Design Overview To fix the aforementioned issues, we need an abstraction that separates a swap entry from its physical backing storage. IOW, we need to “virtualize” the swap space: swap clients will work with a virtual swap slot (that is dynamically allocated on-demand), storing it in page table entries, and using it to index into various swap-related data structures. The backing storage is decoupled from this slot, and the newly introduced layer will “resolve” the ID to the actual storage, as well as cooperating with the swap cache to handle all the required synchronization. This layer also manages other metadata of the swap entry, such as its lifetime information (swap count), via a dynamically allocated per-entry swap descriptor: struct swp_desc { swp_entry_t vswap; union { swp_slot_t slot; struct folio *folio; struct zswap_entry *zswap_entry; }; struct rcu_head rcu; rwlock_t lock; enum swap_type type; #ifdef CONFIG_MEMCG atomic_t memcgid; #endif atomic_t in_swapcache; struct kref refcnt; atomic_t swap_count; }; This design allows us to: * Decouple zswap (and zeromapped swap entry) from backing swapfile: simply associate the swap ID with one of the supported backends: a zswap entry, a zero-filled swap page, a slot on the swapfile, or a page in memory . * Simplify and optimize swapoff: we only have to fault the page in and have the swap ID points to the page instead of the on-disk swap slot. No need to perform any page table walking :) III. Future Use Cases Other than decoupling swap backends and optimizing swapoff, this new design allows us to implement the following more easily and efficiently: * Multi-tier swapping (as mentioned in [5]), with transparent transferring (promotion/demotion) of pages across tiers (see [8] and [9]). Similar to swapoff, with the old design we would need to perform the expensive page table walk. * Swapfile compaction to alleviate fragmentation (as proposed by Ying Huang in [6]). * Mixed backing THP swapin (see [7]): Once you have pinned down the backing store of THPs, then you can dispatch each range of subpages to appropriate pagein handler. [1]: https://lore.kernel.org/all/CAJD7tkbCnXJ95Qow_aOjNX6NOMU5ovMSHRC+95U4wtW6cM+puw@mail.gmail.com/ [2]: https://lwn.net/Articles/932077/ [3]: https://www.youtube.com/watch?v=Hwqw_TBGEhg [4]: https://lore.kernel.org/all/Zqe_Nab-Df1CN7iW@infradead.org/ [5]: https://lore.kernel.org/lkml/CAF8kJuN-4UE0skVHvjUzpGefavkLULMonjgkXUZSBVJrcGFXCA@mail.gmail.com/ [6]: https://lore.kernel.org/linux-mm/87o78mzp24.fsf@yhuang6-desk2.ccr.corp.intel.com/ [7]: https://lore.kernel.org/all/CAGsJ_4ysCN6f7qt=6gvee1x3ttbOnifGneqcRm9Hoeun=uFQ2w@mail.gmail.com/ [8]: https://lore.kernel.org/linux-mm/4DA25039.3020700@redhat.com/ [9]: https://lore.kernel.org/all/CA+ZsKJ7DCE8PMOSaVmsmYZL9poxK6rn0gvVXbjpqxMwxS2C9TQ@mail.gmail.com/