From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.216.53 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794910232; b=mEViA67Wo6ZYQdNuesex2W2FE2035NC5WgS7DPuqz3iNYdFTY9rYU+C1eh7fvGcaOkHK R0njT3X7hpo5ifiFULQqQVdYh4IDWV6r9MdzlR3D8ytmlCS1Qy+dqTbCnB+J2fBlXvqAf vpuVRriAZBVw1V89ER+rEjQsgvUtmv5jy0mDJRHcJKjnn5FFFW/XRTx6fmUI3NTZ/2QXc 2LKIdUzjW5LimxIkkVdH8qLG7fMls0vD4dDiASbNv7pbXLkc8dKFwp2vxigM4mRDZihlk VG7H6ywKcXO4GVbCQyHwf0wvoIPEofBUgNiltXSi+SvFr6OzBiESwjiBsETPk6t2e0A== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794910232; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: From:To:Cc:Subject:Date:Message-ID:X-Mailer:MIME-Version: Content-Transfer-Encoding; bh=q3GZzS1NlBQVXcKUhBOvFOkneqeR2LKFaenmjFFghE0=; b=WjCPA2mOexsUiaOo537HrQ/JThlkpT1S+VphjarHB+GZBvdI3ZmpIIjhy+dy1hj5F5oY T+CIBUYbGuGL5H5dUZE9aoi3ilf162pwcjm2lkXoGb5sNPZENny2vLBB3pE+rcVZCjInm gUnXbeEeY7asZ2HhfJGqwFvClfqz1KAl/QZpwfyzHyBKfjYY4cfFlQe9A6CbicsLH1qYb zHS7j14BKd691Af9UQe3uebJSfKWx9wAZ8EnNjexl5KpFMg9lQKz4rdRShqE7mqxuu/wS 2duOZ/iAIzRVuHtfa4hjf7EWqF7Q/mQuuFNrooL7NV3kSNn3OXDdaRKONHJu+JuHEmA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20260707 header.b=PnXCNeRt; arc=none smtp.remote-ip=209.85.216.53 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3C64606A329 for ; Tue, 17 Nov 2026 10:10:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org B3C64606A329 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20260707 header.b=PnXCNeRt Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-3b3b3c072eeso328079a91.3 for ; Tue, 17 Nov 2026 02:10:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794910231; x=1795515031; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=q3GZzS1NlBQVXcKUhBOvFOkneqeR2LKFaenmjFFghE0=; b=PnXCNeRt2TO75mOhTCfdg8r53WPIkxIEKrOtAJhItpL1dSp/KKvMKCzqGK3M37cBeF XFgScLjCPtrt3mJ3n2RgSgTLvx2w62ryTHU529xmsy7u++ukzF+QMzNDyNWdB4gCUsRr Z+ZznwUlsAcWtg+4GWfHxT9P/n6CIoR5RMudV4UctjHzeLj2LNu5fK5tysYu3W2elqG4 EhCgH2JQ04xuZls3Ld1rYDuvhyAeCS6E/sCgjhL0Huz87+F3c8S5YjwyYb9woU0Va9Oc iOd7CA2WfOzjuY32kz8Q6cVZiaUIB3dYyFVKhXoshoBrYAdcBgN7E9D48S1eIcyHQKmT cZ1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794910231; x=1795515031; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=q3GZzS1NlBQVXcKUhBOvFOkneqeR2LKFaenmjFFghE0=; b=byAXAwDIdUpmXlpTpJiuwSU3lnDS43Cr1iqKNzs+E3XsqZIobBNGNpElC3LQ+gre88 iYJmpCvweNudhwMLnQH1OvqH+Z4s3LaJ7h/ljXrg8OgMFdKt5c/9QHG0NLvMieh7EduU AxuPYr5C7Mo3ErEsmFWW+0iNa8CbegoYIXlVAAaozOiUl8NKKlUCm1HZkkT9bq/IcaeL p2m65297BAV271MD5O/y2sLw9oyw1SnvIF75dAAEeS7rjDsInRAgQeRyY7W4eTQG/v5k vKlSu3yDlMerXaGvl7+hN6U2VWdPOuKqzBVpC+KSr9pvyCjW/FE5NpNXPO0FbRzaIibo FUwA== X-Forwarded-Encrypted: i=1; AL+hErFp3rGDNzeXmWF0iqJW0n3FrUTfltKv/DESYhlC7iV00pi6VdukHm7nNxksz83FoniFNuTJbII=@vger.kernel.org X-Gm-Message-State: APuU4Z/YW8LqyJeILiVJUwewGasm0g8OjVxnpfp0Wsv1r4DiFRIMmBOX uTfF12y3l2tgdVCL+n7vOrB6HzLyG6bBTYmVY2fGIKDgswNarZm7wV+P X-Gm-Gg: AZFdTZctuPJicdMDyLhqiUdKhBDfDP+HrvJYxhMdDly+7+jpsz6qr6M0aXBxEpC78kU RIqP2Mu0yLFukzme6KJC2PeAiOfj1B8RsaI3suiH7oqfVnFtjk1iqYXb/EmcDYj1B1AFhpBjuqX fN40b/a023USAw47VoX6p23wG9BQoXtAnH4EcesJOV3PcXk52Z0h0d5d11UncfjiWMb1MVi+Kjj rM7p/9dgiNh/bwOLRCBWjq7/uDp3yUnXva1En5RNkGhzFwGIu8HrLxLaUzvSxnGdzD/mEmvMIZZ KSkHw26ntqjK0zkP29G76gv8sNql9BQ4Sl4WogOKRIEZD1US78xullYwLzlrOieCA5w+0/n8U41 UqDN//tTvMVpt0Y9RLDr2UpMILqj0SN8j2HQglkYtjUb3gfGtjfHEl3nRvVmbtsST3uMLPKEcG4 SxISxJJYRE18wpkzQTP0MFCFJjsmPwTDVmBz81WOb8GXgATCz8ZoQnyQFYjvU28pcZDbvRZOAE2 6rfhgsEBnKu5f0M5hTzuWDVeNBur46VbVGpcml0qnhez9R5EgFPDrT0z5X/sZDpf+2UTtKMhpEX J080pTPEGbhmYPXlXYGoChETs2BvoVYH49xaPn3BgDqj X-Received: by 2002:a17:90a:e18c:b0:3a0:ace4:d2b4 with SMTP id 98e67ed59e1d1-3b1b0a0d930mr11125522a91.23.1794910230152; Tue, 17 Nov 2026 02:10:30 -0800 (PST) Received: from gmail.com ([116.162.234.136]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3b3ab9f3c5dsm3534751a91.0.2026.11.17.02.10.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Nov 2026 02:10:29 -0800 (PST) From: Xue Boyang To: kadlec@netfilter.org, pablo@netfilter.org, fw@strlen.de Cc: netfilter-devel@vger.kernel.org, coreteam@netfilter.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, phil@nwl.cc, sunyiqixm@gmail.com, bsdhenrymartin@gmail.com Subject: [PATCH nf v4] netfilter: ipset: fix double free on kernel-side delete during resize Date: Tue, 17 Nov 2026 04:10:18 -0600 Message-ID: <20261117101018.523826-1-fuchen.dust@gmail.com> X-Mailer: git-send-email 2.53.0 X-Mailing-List: netdev@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit A kernel-side delete, i.e. one triggered by the SET target while the set is being resized, can free an element extension that the resize copy still references. The resize copies elements to the new table keeping the pointers to the extension data, e.g. to the comment. The delete removes the element from the current table and calls ip_set_ext_destroy(), releasing the shared extension; when the copy in the new table is later deleted or reclaimed by the garbage collector, the same rcu_head is queued twice. With CONFIG_DEBUG_OBJECTS_RCU_HEAD (and slub_debug=F, which keeps kfree_rcu() off the sheaf fast path): ODEBUG: activate active (active state 1) object type: rcu_head WARNING: lib/debugobjects.c:629 at debug_print_object+0x7b/0x100 kvfree_call_rcu(): Double-freed call. rcu_head ffff88810474ce40 WARNING: mm/slab_common.c:2041 at kvfree_call_rcu+0x10b/0x140 Call Trace: debug_object_activate (lib/debugobjects.c:905) kvfree_call_rcu (mm/slab_common.c:2039) hash_ip4_del (net/netfilter/ipset/ip_set_hash_gen.h:1226) hash_ip4_kadt (net/netfilter/ipset/ip_set_hash_ip.c:96) ip_set_del (net/netfilter/ipset/ip_set_core.c:820) set_target_v3 (net/netfilter/xt_set.c:411) ipt_do_table (net/ipv4/netfilter/ip_tables.c:345) nf_hook_slow (net/netfilter/core.c:619) raw_sendmsg (net/ipv4/raw.c:652) __sys_sendto (net/socket.c:2281) The double free is reachable from an unprivileged user namespace and permits local privilege escalation. The existing deferral of kernel-side adds into the per-table backlog cannot be extended to deletes soundly: the deferral decision reads t->resizing without a common lock with the copy loop, so a destructive writer can still pass the check before the copier reaches its bucket; the arrival order of deferred operations against operations executed directly cannot be reconstructed once the flag is cleared; and the forced garbage collection of a full set, the forceadd and the reuse of a live entry walk the same check-then-destroy pattern. Previous versions of this fix (v1-v3) tried to defer deletes into the backlog; review showed those windows cannot be closed within that design, so this version removes the backlog altogether. Packet-side writers now run directly and synchronize with the resize explicitly. Every writer (ADD/DEL/GC) holds a per-set rwlock for reading from table selection through its last modification and records completion in a per-set sequence counter. The resize copies each source region under the source region lock and publishes the new table under the write lock only if the sequence counter is unchanged: any writer that modified a region after it was copied invalidates the whole snapshot. A rejected or incomplete snapshot is discarded wholesale; it never owns an extension shared with the current table. The two tables no longer share destroyable extension objects: comments are cloned privately under the source region lock, and the destination slot is marked used only after the clone succeeds. Upstream resize shallow-copies comment pointers between the old and new tables, while a multi-message dump pins the old table across netlink messages without holding RCU continuously between them. After a new-table DEL or GC schedules the shared comment for reclamation, a later dump message using the pinned old table may access the same comment after it has been freed. Private cloning removes this shared ownership. This pre-existing UAF scenario follows from code analysis; the targeted control test did not force the shared comment through the lazy kfree_rcu reclamation pipeline, so the upstream UAF itself has not yet been reproduced at runtime. Expired-but-still-used entries are copied as well, so the new table owns every used entry's extension and the regular garbage collector reclaims them; this also fixes the pre-existing leak of comment extensions of entries dropped by the old copy loop. Global prefix bookkeeping (rnets) belongs to the set, not to any single table, and is now released only at set destruction. If a future destroyable extension is enabled for a set without a private-clone rule, the resize fails with -EOPNOTSUPP instead of silently reverting to shared ownership. After three consecutive snapshot conflicts the resize returns -EBUSY with the current table and all completed operations intact: under a sustained packet rate the triggering add fails visibly to userspace instead of deferring work unboundedly. This is an intentional behavior change. Tested on a KASAN + lockdep kernel in a VM. The SET --del-set reproducer completed 294 resize-race shots with err=0 and no ODEBUG, double-free, KASAN or ipset-related WARNING/BUG report in the post-run dmesg scan. The ordering regression test (packet del+add / add+del pairs racing continuous resizes) passed 12 rounds with 37 resize windows and zero lost or resurrected elements. Bounded fault injection at the snapshot bucket allocation, at the comment clone allocation and at the second clone (partial-clone cleanup), 30 failures each: every failed resize left the hashsize, the memsize accounting and the current table untouched, injection-free resizes afterwards converged, and no BUG, lockdep warning, ODEBUG or double-free report appeared in dmesg. A candidate-side multi-message dump test paused the dump after its first batch, published a resized table, deleted 75 of 150 commented entries from the new table, waited five seconds, and resumed the old-table dump: all 150 entries returned with byte-identical comments and no KASAN report. A scoped A/B benchmark in a 2-vCPU KASAN VM using netlink operations observed no material throughput regression; peak memsize during comment resize was about 4% higher. Under a sustained packet writer, 33 resizes returned -EBUSY with the current table remaining queryable and no missing final elements; after the writer stopped, all 33 operations succeeded on retry, the longest add taking 13.5 ms. Compatibility smoke tests passed 34/34 on each of two candidate kernel configurations, covering IPv4/IPv6, hash:net multi-CIDR resize, forceadd, restore-style bursts, timeout/GC accounting, repeated create/destroy, and full dumps. Fixes: f66ee0410b1c ("netfilter: ipset: Fix "INFO: rcu detected stall in hash_xxx" reports") Cc: stable@vger.kernel.org Link: https://lore.kernel.org/netfilter-devel/20261105045210.1179991-1-sunyiqixm@gmail.com/ Signed-off-by: Xue Boyang --- Changes in v4, prompted by Yiqi Sun's review of v3 (https://lore.kernel.org/netfilter-devel/20261105045210.1179991-1-sunyiqixm@gmail.com/) and by a self-review of the same family of paths: The review and the self-review confirmed that check-then-destroy patterns around the resizing flag leave double-free windows that the backlog cannot close (a writer passing the flag check before the copier reaches its bucket, the forced GC of a full set, forceadd/reuse of a live entry) and that the ordering of deferred operations cannot be preserved once the flag is cleared. Rather than patching each path, v4 removes the backlog and the resizing flag entirely and replaces them with the validated-snapshot protocol described in the commit message: writers run directly under a per-set read lock plus a sequence counter, resize publishes only a snapshot whose sequence is unchanged, failed snapshots are discarded wholesale, and comment extensions are cloned so the two tables never share a destroyable object. Consequences worth reviewers' attention: - Kernel-side DEL/ADD and GC now execute immediately during resizes; there is no backlog or replay, so the resize protocol does not discard operations from a deferred-operation queue. - Resize returns -EBUSY to the triggering userspace add after three consecutive snapshot conflicts under a sustained packet rate. The operation fails visibly instead of being deferred unboundedly; userspace can retry once the rate drops. Under a sustained packet writer, 33 resizes returned -EBUSY with the current table remaining queryable and no missing final elements. After the writer stopped, all 33 operations succeeded on retry; the longest add took 13.5 ms. - The resize briefly duplicates every comment extension (GFP_ATOMIC clone) and adds a per-set rwlock plus sequence counter to the packet path. A scoped A/B benchmark in a 2-vCPU KASAN VM using netlink operations observed no material throughput regression. The comment-resize case ranged from 3% slower to 12% faster, while an unaffected TEST control was 14-21% faster, indicating substantial code-layout or VM noise rather than an established speedup. Peak memsize during comment resize was about 4% higher. Production packet-path throughput has not been measured. Compatibility smoke tests passed 34/34 on each of two candidate kernel configurations, covering IPv4/IPv6, hash:net multi-CIDR resize, forceadd, restore-style bursts, timeout/GC accounting, repeated create/destroy, and full dumps. - Henry Martin tested v3 (Tested-by, msgid <20261111105833.2060684-1-bsdhenrymartin@gmail.com>): the fix has been reworked since, so his tag is not carried over. Henry, a retest of v4 would be much appreciated. Changes in v3 (https://lore.kernel.org/netfilter-devel/20261026102957.5530-1-fuchen.dust@gmail.com/): drop the non-standard "Assisted-by:" trailer which fails checkpatch; the AI assistance is documented here instead. Changes in v2, addressing the review of v1: defer the kernel-side add of an element still present in the current table; replay the backlog on all resize failure exits; queue deferred deletes FIFO to preserve the arrival order; report -ENOMEM when a deferred delete cannot allocate its backlog entry. The reproducer and the regression/fault-injection/dump tests are available on request. AI coding assistants, including Codex and GLM, were used for code analysis, protocol review, test-tool development, and test execution. The author reviewed the resulting patch and archived test results and takes responsibility for the submission. net/netfilter/ipset/ip_set_hash_gen.h | 309 +++++++++++++------------- 1 file changed, 150 insertions(+), 159 deletions(-) diff --git a/net/netfilter/ipset/ip_set_hash_gen.h b/net/netfilter/ipset/ip_set_hash_gen.h index 80ca523f304bb..30f7c0523d911 100644 --- a/net/netfilter/ipset/ip_set_hash_gen.h +++ b/net/netfilter/ipset/ip_set_hash_gen.h @@ -6,6 +6,7 @@ #include #include +#include #include #include #include @@ -33,9 +34,14 @@ * Readers and resizing * * Resizing can be triggered by userspace command only, and those - * are serialized by the nfnl mutex. During resizing the set is - * read-locked, so the only possible concurrent operations are - * the kernel side readers. Those must be protected by proper RCU locking. + * are serialized by the nfnl mutex. Packet-path writers and garbage + * collection hold resize_lock for reading from table selection through + * their last modification. They advance resize_seq before dropping it. + * Resize copies each source region under its lock, then takes resize_lock + * for writing and publishes only if resize_seq has not changed. A failed + * snapshot owns only private extension clones and can be discarded + * without replay or touching the source table. + * Lockless lookups continue to use RCU. */ /* Number of elements to store in an initial array block */ @@ -45,6 +51,8 @@ /* Max muber of elements in the array block when tuned */ #define AHASH_MAX_TUNED 64 #define AHASH_MAX(h) ((h)->bucketsize) +/* Limit snapshot retries under sustained packet-path writes. */ +#define AHASH_RESIZE_RETRIES 3 /* A hash bucket */ struct hbucket { @@ -76,17 +84,14 @@ struct hbucket { struct htable_gc { struct delayed_work dwork; struct ip_set *set; /* Set the gc belongs to */ - spinlock_t lock; /* Lock to exclude gc and resize */ u32 region; /* Last gc run position */ }; /* The hash table: the table size stored here in order to make resizing easy */ struct htable { - bool resizing; /* Mark ongoing resize */ - atomic_t uref; /* References for dumping and gc */ + atomic_t uref; /* Current-table ownership and dump references */ u8 htable_bits; /* size of hash table == 2^htable_bits */ u32 maxelem; /* Maxelem per region */ - struct list_head ad; /* Resize add|del backlist */ struct ip_set_region *hregion; /* Region locks and ext sizes */ struct hbucket __rcu *bucket[]; /* hashtable buckets */ }; @@ -217,6 +222,8 @@ static const union nf_inet_addr zeromask = {}; #undef mtype_ahash_destroy #undef mtype_ext_cleanup +#undef mtype_ext_clone +#undef mtype_ext_clone_supported #undef mtype_add_cidr #undef mtype_del_cidr #undef mtype_del_cidr_all @@ -236,7 +243,6 @@ static const union nf_inet_addr zeromask = {}; #undef mtype_uref #undef mtype_resize #undef mtype_ext_size -#undef mtype_resize_ad #undef mtype_head #undef mtype_list #undef mtype_gc_do @@ -265,6 +271,8 @@ static const union nf_inet_addr zeromask = {}; #define mtype_ahash_destroy IPSET_TOKEN(MTYPE, _ahash_destroy) #define mtype_ext_cleanup IPSET_TOKEN(MTYPE, _ext_cleanup) +#define mtype_ext_clone IPSET_TOKEN(MTYPE, _ext_clone) +#define mtype_ext_clone_supported IPSET_TOKEN(MTYPE, _ext_clone_supported) #define mtype_add_cidr IPSET_TOKEN(MTYPE, _add_cidr) #define mtype_del_cidr IPSET_TOKEN(MTYPE, _del_cidr) #define mtype_del_cidr_all IPSET_TOKEN(MTYPE, _del_cidr_all) @@ -284,7 +292,6 @@ static const union nf_inet_addr zeromask = {}; #define mtype_uref IPSET_TOKEN(MTYPE, _uref) #define mtype_resize IPSET_TOKEN(MTYPE, _resize) #define mtype_ext_size IPSET_TOKEN(MTYPE, _ext_size) -#define mtype_resize_ad IPSET_TOKEN(MTYPE, _resize_ad) #define mtype_head IPSET_TOKEN(MTYPE, _head) #define mtype_list IPSET_TOKEN(MTYPE, _list) #define mtype_gc_do IPSET_TOKEN(MTYPE, _gc_do) @@ -315,6 +322,8 @@ struct htype { struct htable __rcu *table; /* the hash table */ struct net_prefixes __rcu *rnets[IPSET_NET_COUNT]; /* cidr prefixes */ struct htable_gc gc; /* gc workqueue */ + rwlock_t resize_lock; /* Exclude writers while publishing a snapshot */ + atomic64_t resize_seq; /* Completed writer operations */ u32 maxelem; /* max elements in the hash */ u32 initval; /* random jhash init value */ #ifdef IP_SET_HASH_WITH_MARKMASK @@ -331,16 +340,6 @@ struct htype { struct mtype_elem next; /* temporary storage for uadd */ }; -/* ADD|DEL entries saved during resize */ -struct mtype_resize_ad { - struct list_head list; - enum ipset_adt ad; /* ADD|DEL element */ - struct mtype_elem d; /* Element value */ - struct ip_set_ext ext; /* Extensions for ADD */ - struct ip_set_ext mext; /* Target extensions for ADD */ - u32 flags; /* Flags for ADD */ -}; - #ifdef IP_SET_HASH_WITH_NETS /** * mtype_add_cidr - Add a CIDR entry to hash table bookkeeping @@ -504,6 +503,62 @@ mtype_ext_cleanup(struct ip_set *set, struct hbucket *n) ip_set_ext_destroy(set, ahash_data(n, i, set->dsize)); } +/* Fail closed if a future extension adds private allocations without teaching + * the resize snapshot how to acquire independent ownership of them. + */ +static bool +mtype_ext_clone_supported(const struct ip_set *set) +{ + enum ip_set_ext_id id; + + for (id = 0; id < IPSET_EXT_ID_MAX; id++) { + const struct ip_set_ext_type *ext = &ip_set_extensions[id]; + u8 type = ext->type & ~IPSET_EXT_DESTROY; + + if (!ext->destroy || !(set->extensions & type)) + continue; + if (id != IPSET_EXT_ID_COMMENT) + return false; + } + + return true; +} + +/* Clone extension allocations so retired tables and the published table do + * not share destroyable objects. The caller holds the source region lock. + */ +static int +mtype_ext_clone(struct ip_set *set, void *dst, const void *src, + spinlock_t *region_lock) +{ + const struct ip_set_comment *src_comment; + struct ip_set_comment *dst_comment; + struct ip_set_comment_rcu *comment, *clone; + size_t len; + + lockdep_assert_held(region_lock); + if (!SET_WITH_COMMENT(set)) + return 0; + + src_comment = ext_comment(src, set); + dst_comment = ext_comment(dst, set); + RCU_INIT_POINTER(dst_comment->c, NULL); + comment = rcu_dereference_protected(src_comment->c, + lockdep_is_held(region_lock)); + if (!comment) + return 0; + + len = strlen(comment->str) + 1; + clone = kmalloc(sizeof(*clone) + len, GFP_ATOMIC); + if (!clone) + return -ENOMEM; + memcpy(clone->str, comment->str, len); + atomic64_add(sizeof(*clone) + len, &set->ext_size); + RCU_INIT_POINTER(dst_comment->c, clone); + + return 0; +} + /* Flush a hash type of set: destroy all elements */ static void mtype_flush(struct ip_set *set) @@ -553,9 +608,6 @@ mtype_flush(struct ip_set *set) static void mtype_ahash_destroy(struct ip_set *set, struct htable *t, bool ext_destroy) { -#ifdef IP_SET_HASH_WITH_NETS - struct htype *h = set->data; -#endif struct hbucket *n; u32 i; @@ -569,11 +621,6 @@ mtype_ahash_destroy(struct ip_set *set, struct htable *t, bool ext_destroy) kfree(n); } -#ifdef IP_SET_HASH_WITH_NETS - if (ext_destroy) - for (i = 0; i < IPSET_NET_COUNT; i++) - kfree(rcu_dereference_raw(h->rnets[i])); -#endif ip_set_free(t->hregion); ip_set_free(t); } @@ -584,13 +631,15 @@ mtype_destroy(struct ip_set *set) { struct htype *h = set->data; struct htable *t = (__force struct htable *)h->table; - struct list_head *l, *lt; +#ifdef IP_SET_HASH_WITH_NETS + int i; +#endif - list_for_each_safe(l, lt, &t->ad) { - list_del(l); - kfree(l); - } mtype_ahash_destroy(set, t, true); +#ifdef IP_SET_HASH_WITH_NETS + for (i = 0; i < IPSET_NET_COUNT; i++) + kfree(rcu_dereference_raw(h->rnets[i])); +#endif kfree(h); set->data = NULL; @@ -694,9 +743,9 @@ mtype_gc(struct work_struct *work) set = gc->set; h = set->data; + read_lock_bh(&h->resize_lock); rcu_read_lock_bh(); t = rcu_dereference_bh(h->table); - atomic_inc(&t->uref); rcu_read_unlock_bh(); numof_locks = ahash_numof_locks(t->htable_bits); r = gc->region++; @@ -707,15 +756,9 @@ mtype_gc(struct work_struct *work) if (next_run < HZ/10) next_run = HZ/10; - spin_lock_bh(&gc->lock); - if (!t->resizing) - mtype_gc_do(set, h, t, r); - spin_unlock_bh(&gc->lock); - - if (atomic_dec_and_test(&t->uref) && t->resizing) { - pr_debug("Table destroy after resize by expire: %p\n", t); - mtype_ahash_destroy(set, t, false); - } + mtype_gc_do(set, h, t, r); + atomic64_inc(&h->resize_seq); + read_unlock_bh(&h->resize_lock); queue_delayed_work(system_power_efficient_wq, &gc->dwork, next_run); @@ -737,16 +780,9 @@ mtype_cancel_gc(struct ip_set *set) disable_delayed_work_sync(&h->gc.dwork); } -static int -mtype_add(struct ip_set *set, void *value, const struct ip_set_ext *ext, - struct ip_set_ext *mext, u32 flags); -static int -mtype_del(struct ip_set *set, void *value, const struct ip_set_ext *ext, - struct ip_set_ext *mext, u32 flags); - /* Resize a hash: create a new hash table with doubling the hashsize - * and inserting the elements to it. Repeat until we succeed or - * fail due to memory pressures. + * and inserting the elements to it. Failed snapshots own only private + * extension clones. Bound conflicts instead of draining an unbounded queue. */ static int mtype_resize(struct ip_set *set, bool retried) @@ -762,11 +798,14 @@ mtype_resize(struct ip_set *set, bool retried) struct mtype_elem *data; struct mtype_elem *d; struct hbucket *n, *m; - struct list_head *l, *lt; - struct mtype_resize_ad *x; u32 i, j, r, nr, key; + u64 seq; + unsigned int conflicts = 0; int ret; + if (unlikely(!mtype_ext_clone_supported(set))) + return -EOPNOTSUPP; + #ifdef IP_SET_HASH_WITH_NETS tmp = kmalloc(dsize, GFP_KERNEL); if (!tmp) @@ -775,14 +814,15 @@ mtype_resize(struct ip_set *set, bool retried) orig = ipset_dereference_bh_nfnl(h->table); htable_bits = orig->htable_bits; -retry: - ret = 0; +grow: htable_bits++; if (!htable_bits) goto hbwarn; hsize = htable_size(htable_bits); if (!hsize) goto hbwarn; +retry: + ret = 0; t = ip_set_alloc(hsize); if (!t) { ret = -ENOMEM; @@ -796,23 +836,19 @@ mtype_resize(struct ip_set *set, bool retried) } t->htable_bits = htable_bits; t->maxelem = h->maxelem / ahash_numof_locks(htable_bits); - INIT_LIST_HEAD(&t->ad); + atomic_set(&t->uref, 1); for (i = 0; i < ahash_numof_locks(htable_bits); i++) spin_lock_init(&t->hregion[i].lock); - /* There can't be another parallel resizing, - * but dumping and kernel side add/del are possible + /* A writer records completion before releasing resize_lock. The + * final write lock waits for in-flight writers, including ones + * which selected the source table before the first region copy. */ - orig = ipset_dereference_bh_nfnl(h->table); - atomic_inc(&orig->uref); - spin_lock_bh(&h->gc.lock); - orig->resizing = true; - spin_unlock_bh(&h->gc.lock); + seq = atomic64_read_acquire(&h->resize_seq); pr_debug("attempt to resize set %s from %u to %u, t %p\n", set->name, orig->htable_bits, htable_bits, orig); for (r = 0; r < ahash_numof_locks(orig->htable_bits); r++) { - /* Expire may replace a hbucket with another one */ - rcu_read_lock_bh(); + spin_lock_bh(&orig->hregion[r].lock); for (i = ahash_bucket_start(r, orig->htable_bits); i < ahash_bucket_end(r, orig->htable_bits); i++) { n = __ipset_dereference(hbucket(orig, i)); @@ -823,8 +859,10 @@ mtype_resize(struct ip_set *set, bool retried) if (!test_bit_acquire(j, n->used)) continue; data = ahash_data(n, j, dsize); - if (SET_ELEM_EXPIRED(set, data)) - continue; + /* Keep expired entries too: the new table takes + * ownership of every used entry's extensions. + * GC can reclaim them after publication. + */ #ifdef IP_SET_HASH_WITH_NETS /* We have readers running parallel with us, * so the live data cannot be modified. @@ -877,6 +915,10 @@ mtype_resize(struct ip_set *set, bool retried) } d = ahash_data(m, m->pos, dsize); memcpy(d, data, dsize); + ret = mtype_ext_clone(set, d, data, + &orig->hregion[r].lock); + if (ret < 0) + goto cleanup; set_bit(m->pos++, m->used); t->hregion[nr].elements++; #ifdef IP_SET_HASH_WITH_NETS @@ -884,36 +926,35 @@ mtype_resize(struct ip_set *set, bool retried) #endif } } - rcu_read_unlock_bh(); + spin_unlock_bh(&orig->hregion[r].lock); + cond_resched(); } - /* There can't be any other writer. */ - rcu_assign_pointer(h->table, t); - - /* Give time to other readers of the set */ - synchronize_rcu(); - - pr_debug("set %s resized from %u (%p) to %u (%p)\n", set->name, - orig->htable_bits, orig, t->htable_bits, t); - /* Add/delete elements processed by the SET target during resize. - * Kernel-side add cannot trigger a resize and userspace actions - * are serialized by the mutex. + /* No source writer may finish between validation and publication. + * Writers selecting a table after this point select the new one. */ - list_for_each_safe(l, lt, &orig->ad) { - x = list_entry(l, struct mtype_resize_ad, list); - if (x->ad == IPSET_ADD) { - mtype_add(set, &x->d, &x->ext, &x->mext, x->flags); - } else { - mtype_del(set, &x->d, NULL, NULL, 0); + write_lock_bh(&h->resize_lock); + if (atomic64_read(&h->resize_seq) != seq) { + write_unlock_bh(&h->resize_lock); + mtype_ahash_destroy(set, t, true); + if (++conflicts < AHASH_RESIZE_RETRIES) { + cond_resched(); + goto retry; } - list_del(l); - kfree(l); - } - /* If there's nobody else using the table, destroy it */ - if (atomic_dec_and_test(&orig->uref)) { - pr_debug("Table destroy by resize %p\n", orig); - mtype_ahash_destroy(set, orig, false); + ret = -EBUSY; + goto out; } + rcu_assign_pointer(h->table, t); + write_unlock_bh(&h->resize_lock); + + /* Readers may still be using the old buckets. Drop its current-table + * ownership only after they finish; dumps hold separate references. + * Extensions are private to each table, so the last reference destroys + * both retired buckets and their original extension allocations. + */ + synchronize_rcu(); + if (atomic_dec_and_test(&orig->uref)) + mtype_ahash_destroy(set, orig, true); out: #ifdef IP_SET_HASH_WITH_NETS @@ -922,25 +963,13 @@ mtype_resize(struct ip_set *set, bool retried) return ret; cleanup: - rcu_read_unlock_bh(); - spin_lock_bh(&h->gc.lock); - orig->resizing = false; - spin_unlock_bh(&h->gc.lock); - /* Make sure parallel readers see that orig->resizing is false - * before we decrement uref */ - synchronize_rcu(); - atomic_dec(&orig->uref); - mtype_ahash_destroy(set, t, false); - if (ret == -EAGAIN) - goto retry; - - /* Cleanup the backlog of ADD/DEL elements */ - spin_lock_bh(&set->lock); - list_for_each_safe(l, lt, &orig->ad) { - list_del(l); - kfree(l); + spin_unlock_bh(&orig->hregion[r].lock); + /* The draft owns only its cloned extensions. */ + mtype_ahash_destroy(set, t, true); + if (ret == -EAGAIN) { + cond_resched(); + goto grow; } - spin_unlock_bh(&set->lock); goto out; hbwarn: @@ -999,11 +1028,11 @@ mtype_add(struct ip_set *set, void *value, const struct ip_set_ext *ext, u32 r, key, multi = 0, elements, maxelem; u8 npos = 0; + read_lock_bh(&h->resize_lock); rcu_read_lock_bh(); t = rcu_dereference_bh(h->table); key = HKEY(value, h->initval, t->htable_bits); r = ahash_region(key); - atomic_inc(&t->uref); rcu_read_unlock_bh(); elements = t->hregion[r].elements; maxelem = t->maxelem; @@ -1139,23 +1168,6 @@ mtype_add(struct ip_set *set, void *value, const struct ip_set_ext *ext, ret = 0; resize: spin_unlock_bh(&t->hregion[r].lock); - if (t->resizing && ext && ext->target) { - /* Resize is in process and kernel side add, save values */ - struct mtype_resize_ad *x; - - x = kzalloc_obj(struct mtype_resize_ad, GFP_ATOMIC); - if (!x) - /* Don't bother */ - goto out; - x->ad = IPSET_ADD; - memcpy(&x->d, value, sizeof(struct mtype_elem)); - memcpy(&x->ext, ext, sizeof(struct ip_set_ext)); - memcpy(&x->mext, mext, sizeof(struct ip_set_ext)); - x->flags = flags; - spin_lock_bh(&set->lock); - list_add_tail(&x->list, &t->ad); - spin_unlock_bh(&set->lock); - } goto out; set_full: @@ -1166,10 +1178,8 @@ mtype_add(struct ip_set *set, void *value, const struct ip_set_ext *ext, unlock: spin_unlock_bh(&t->hregion[r].lock); out: - if (atomic_dec_and_test(&t->uref) && t->resizing) { - pr_debug("Table destroy after resize by add: %p\n", t); - mtype_ahash_destroy(set, t, false); - } + atomic64_inc(&h->resize_seq); + read_unlock_bh(&h->resize_lock); return ret; } @@ -1184,7 +1194,6 @@ mtype_del(struct ip_set *set, void *value, const struct ip_set_ext *ext, const struct mtype_elem *d = value; struct mtype_elem *data; struct hbucket *n; - struct mtype_resize_ad *x = NULL; int i, j, k, r, ret = -IPSET_ERR_EXIST; u32 key, multi = 0; size_t dsize = set->dsize; @@ -1193,11 +1202,11 @@ mtype_del(struct ip_set *set, void *value, const struct ip_set_ext *ext, /* Userspace add and resize is excluded by the mutex. * Kernespace add does not trigger resize. */ + read_lock_bh(&h->resize_lock); rcu_read_lock_bh(); t = rcu_dereference_bh(h->table); key = HKEY(value, h->initval, t->htable_bits); r = ahash_region(key); - atomic_inc(&t->uref); rcu_read_unlock_bh(); spin_lock_bh(&t->hregion[r].lock); @@ -1225,18 +1234,6 @@ mtype_del(struct ip_set *set, void *value, const struct ip_set_ext *ext, mtype_del_cidr_all(set, h, d); ip_set_ext_destroy(set, data); - if (t->resizing && ext && ext->target) { - /* Resize is in process and kernel side del, - * save values - */ - x = kzalloc_obj(struct mtype_resize_ad, GFP_ATOMIC); - if (x) { - x->ad = IPSET_DEL; - memcpy(&x->d, value, - sizeof(struct mtype_elem)); - x->flags = flags; - } - } for (; i < pos; i++) { if (!test_bit(i, n->used)) k++; @@ -1271,15 +1268,8 @@ mtype_del(struct ip_set *set, void *value, const struct ip_set_ext *ext, out: spin_unlock_bh(&t->hregion[r].lock); - if (x) { - spin_lock_bh(&set->lock); - list_add(&x->list, &t->ad); - spin_unlock_bh(&set->lock); - } - if (atomic_dec_and_test(&t->uref) && t->resizing) { - pr_debug("Table destroy after resize by del: %p\n", t); - mtype_ahash_destroy(set, t, false); - } + atomic64_inc(&h->resize_seq); + read_unlock_bh(&h->resize_lock); return ret; } @@ -1531,10 +1521,10 @@ mtype_uref(struct ip_set *set, struct netlink_callback *cb, bool start) rcu_read_unlock_bh(); } else if (cb->args[IPSET_CB_PRIVATE]) { t = (struct htable *)cb->args[IPSET_CB_PRIVATE]; - if (atomic_dec_and_test(&t->uref) && t->resizing) { + if (atomic_dec_and_test(&t->uref)) { pr_debug("Table destroy after resize " " by dump: %p\n", t); - mtype_ahash_destroy(set, t, false); + mtype_ahash_destroy(set, t, true); } cb->args[IPSET_CB_PRIVATE] = 0; } @@ -1785,7 +1775,8 @@ IPSET_TOKEN(HTYPE, _create)(struct net *net, struct ip_set *set, } #endif h->gc.set = set; - spin_lock_init(&h->gc.lock); + rwlock_init(&h->resize_lock); + atomic64_set(&h->resize_seq, 0); for (i = 0; i < ahash_numof_locks(hbits); i++) spin_lock_init(&t->hregion[i].lock); h->maxelem = maxelem; @@ -1812,7 +1803,7 @@ IPSET_TOKEN(HTYPE, _create)(struct net *net, struct ip_set *set, } t->htable_bits = hbits; t->maxelem = h->maxelem / ahash_numof_locks(hbits); - INIT_LIST_HEAD(&t->ad); + atomic_set(&t->uref, 1); RCU_INIT_POINTER(h->table, t); set->data = h; -- 2.53.0