From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from YT6PR01CU002.outbound.protection.outlook.com (mail-canadacentralazon11022092.outbound.protection.outlook.com [40.107.193.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 80ADF37DAB1 for ; Sat, 8 Aug 2026 17:34:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.193.92 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786210456; cv=fail; b=j1Juoo1MBz0ljwN2pb34LBhdX4mWEwz8L+7At6e0YXNwtk13nlHHLysBkxLAs62nmCZMMscIjUObIMubf9rPJI0H3G+2jusUM6/n2CV18aY46HerczVpalTvCngWFcFfVGG5VLIB2hpPlFIiIVfFAj7/Q3XAgPm4Rs+aTyMTekY= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786210456; c=relaxed/simple; bh=jr4631DA4tiUSzKRK+9rd5eoSruLekAYERQjiH706FY=; h=Message-ID:Date:Subject:To:Cc:References:From:In-Reply-To: Content-Type:MIME-Version; b=RvztSZt0zCuuhyuGJR+XJKte1pE8wY/lfbu7vvmbmpC5AbTvfYZVN3UO0/AuuSOYJxLs4Pv7tUGmrnb7VckILl87NkM7ZlaEroB66fFX/lgUmDDm5burmDPTOg1/VGqz1vF+VHBFZEMnM8dgj4YHEul8DAvm697Gecs7Z52zIIo= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com; spf=pass smtp.mailfrom=efficios.com; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b=JiWmkL+u; arc=fail smtp.client-ip=40.107.193.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=efficios.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b="JiWmkL+u" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UxwRJ3Lldh9kQsVxYNxVaaA+DGwEOjzNBybxNo3LDSjgmgiRMQANokvQa4CSeuKCfg1JLS1YKgsHqjEUz20n0o4k91XTlMSOT79H4dfnaKdxXr2kkbfKj2jN5XMEeYVq4dUrilauWLZ1zI1Wjatt3WuBApyBzFmjMYx5bQO8H1gceWlKbeug5D8NNrw5n/vQD1Uw+rujxmxCvrO4/pGTUJTt7XWphqdad+BBwV74uzarvfN88BaT7+22uNI8nvtaGstQj0hhinX2my9X94ed7k9c0p9k4IHrNutg5WaOh11KE1Yz/EhftPUpM7ggiMVqo8F13XwmtFMeZV/a+divzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=O1379EpXdMHZ34LTxmr3TmB5unPeI7c4wq7mvcCBCzI=; b=bKZRMV2gjAQOqm1Qj739QsVQqLhCqKcCXDu4YtFRLXHVUZsNGdVIrCR4aRM+1ARM8cginbNcviw0+veQw21bzjCkydJTJNuVLfGFLfNAiakecgCx3fC8bssJ+OAZTtofHGKoUPqJtnnxWPWHeFgq5GoxwjX930DAEMA0SixfUywy0NKkznwbbuMVlt3+v67ARvbAD/XmQ8ihq1+qa9DN8BMR0Gr5ycDa39nZmmbpVT7JPSohLCdL5nSOEHDrb7yDGRhieAy0j5tCwIluQPqiJWKjWgOO6RWB3j8bnzlPB8LAwB0KqnwwptySa3+D61V0phgNezm1Orvt7TmTDVfBiQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=efficios.com; dmarc=pass action=none header.from=efficios.com; dkim=pass header.d=efficios.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O1379EpXdMHZ34LTxmr3TmB5unPeI7c4wq7mvcCBCzI=; b=JiWmkL+uwzqO88cY2rN940vBhoM7+/vk1uvA4lpeXULb3AvkbuWzT4d5B3HMZMtf9jBEZFrcLc8wXjx8xxO9G2Ok5ZmIJblPiwCTl4dYe63HjN1Mz+jm5Euy2K16JtF9n1AkW8LYrvahcYWmBVzpr2Dy/2wRjkii+4KNcaeBiM5X81J3s+CZ6DnV9rFCQyiv8zAwTsEuBAN742q84oa54ExzgXAVkGlvl5Yr7HY2eytdKkt6J2GuOHr+N02+Skv3YchBnku2Po4LIM/ttOspWxOW6YLXURHWq/abQPWeBOF0NnjJyVAyo2BwdrkVmR6L/2UmZ8C+pxi2emVUJIyyYg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=efficios.com; Received: from YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:be::5) by YT2PR01MB9537.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:ae::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.292.25; Sat, 8 Aug 2026 17:34:10 +0000 Received: from YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM ([fe80::6004:a862:d45d:90c1]) by YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM ([fe80::6004:a862:d45d:90c1%4]) with mapi id 15.21.0292.024; Sat, 8 Aug 2026 17:34:10 +0000 Message-ID: <969d4cf6-cde2-4dd5-ab7a-3b9d252adbbf@efficios.com> Date: Sat, 8 Aug 2026 13:34:09 -0400 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH 1/1] hazptr: Implement two-phases wildcard scan To: "Paul E . McKenney" Cc: linux-kernel@vger.kernel.org References: <20260808173152.6137-1-mathieu.desnoyers@efficios.com> From: Mathieu Desnoyers Content-Language: en-US In-Reply-To: <20260808173152.6137-1-mathieu.desnoyers@efficios.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: YT4PR01CA0088.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:ff::21) To YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:be::5) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: YT2PR01MB9175:EE_|YT2PR01MB9537:EE_ X-MS-Office365-Filtering-Correlation-Id: 1bb88c91-53dd-4461-c32f-08def573419f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|23010399003|18002099003|22082099003|10067099003|56012099006; X-Microsoft-Antispam-Message-Info: eo4t6GQ/ZveRqeeAn9Xa4q/Rw94ZnzWcnYG4PWQUjh66+FCwAAsYUHIN54u9izWUxi7Wb9p47xovc11opsncSkwkZDlyQkzZ5zb3BKO/wPZaPRnuus6xu0LnGxyLg00QqxbNfOtfj5XVEoMb5SX3NOycjH3L0UTOfKMLzzBHhYVDfq8sIVBLJ2U7XgbV0D6k84eOhm89NMjCDZ97+Fe4FRxet1wZbipLxvxAhFgzrc3gKvcTetwVOfSWK5xHWYv+aMdZAS4Wn6y7eRGhvtOjlDRAaMMPHasxz+Y/VXxDhHGZIwC0FFZ6c7cDnLcbtKrEKRcUIsrbhWT2PCDGALDRe2hw4qeIlsobESmLHIir/7SIK7aeLsDrvskhKiNrYP0LRRpnJ1Oi+ob2bxF2j7SfPafuVuALoeMpzJDLKhi3U/qyAyRmfFScxbOx+9pp7rq4cbw/JODAuv5se2e6wTEut+sISG46CLZuNbDrfxKW6UZsB31bUCnp5nqPSb1h1u+ekGRXskSlpkQ0CqAsT9eJNwEZygIHiy1qB/zxfcYSD6g9IsQR4LsX8/F5i13PKSBQqFTIyJKeEYI2vF4FVevm28WD6e9ZgvJLfSH4+u4oz5Y= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(23010399003)(18002099003)(22082099003)(10067099003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SDlUeUU1MjRjWEpSYkpLWXJvMTVIQzQrNEo0SURBUVZLdHBXbWlidXlWdENN?= =?utf-8?B?enJRSXFQeEMyeWdXL2tUamNFLzlpU0l6RkZFNXl1RkZabnVQVDlDa3Zlb3lC?= =?utf-8?B?S3Nsbm5oNWFvM2V0N1B6UlB3bW4rb0I1ZXJ5WWUxbnNNSis0VnNMc3IraTR5?= =?utf-8?B?TlNGK3JNam9lL0pSUS9oK3JVdE1wY0FjaDcydDMyWEVZZmV1WkpEdVdwcGQy?= =?utf-8?B?ZXUxbENRa1R3WHVNWUFTazRCZVBYejFDSkVOMkVhdUc5M2pQZFh6TThyVkwv?= =?utf-8?B?YklrVVo4SDk2UG5TMWt0dnE2eVRBTDYxZnpGVTBlZHZYNzZnU1NUanl6RW0v?= =?utf-8?B?ODkxNjA1aVR4Qy8ycFM2QkFwU1czQVhnYVpUQ080MUROdG0zRnVsaVdTbk8w?= =?utf-8?B?cFFBck1oZXY0UVBlUHBRZC9qMDhrcDlUK3NXQUpkN2VhdU0rU1F4UjVZU3Rp?= =?utf-8?B?WEhwYkxVMmpjbXlXRnZvdlJhWkcrQzA0MXNuYldxYmVRUFRkdGJTRmMzWEZW?= =?utf-8?B?T3gxQWs2UXFhRktZUGtVTjJMTkF4R2RuYTRLT0hqd3J6eXp6c2hib2JJdURp?= =?utf-8?B?amJPWjJDSDVrTTduVE5FQ04rUFlTNWdjNXlZajZOMmU4V2FqTGh3RkFOd3BI?= =?utf-8?B?OTc0UlhqQnJsa0NDSERya3BWWVUxVlZoK2tyNzhsYnpSU3FYdWpVNUNjUXJL?= =?utf-8?B?TjNvU2lVMnBXcjdOendkWG43OFRKa0RRTzYzWGh0NjE3UnFMUnZuZm1CZjVt?= =?utf-8?B?bU5HUmlqb21Ra1psLzNwTGoxbG9HaFlFd2pLVm5MQzNaQzBVUjZZRjBRK1Q3?= =?utf-8?B?VkpaYnVibXh6cGp0dmVpWFFIaXVTS0MySllaaVVGZlFZTmJPT3VTeFluODJu?= =?utf-8?B?SkttSEphUXk4ck5XS3RtUEs3ZXZCbnFJMXBFK3h3ZWhEd1BYakV1NDJrcXpx?= =?utf-8?B?TlZFT3doU3M2bXFpeThJeU1WUVZWWGFrOEdpaUd2d2xYWE9sZi9NME1aQUcz?= =?utf-8?B?Z0JraUpyK24rZU1Qa0JxbkRWVWg0ajZ4VG50bmxtQmllMm5Ya0ovOHo2dE9L?= =?utf-8?B?UzlnSG9pd1B2Ym9WanFuOU95T3d5YmRTWTJUcVdVdFh2UFgrZFh0Y0N4UXV1?= =?utf-8?B?TUNxNjZ4N2FQYlFOMEkxWStzaU1zK3VJYUdYUTRwa2RSc2l0aEpVRHJta2p2?= =?utf-8?B?cVJOTkhtY3Q5S1l0SWJFVlA3L2p4eWR3UnlncnIwNjhMVXpkT1BhZVhVUUZ6?= =?utf-8?B?czZQbHY5MWI3dWM2MU1jdHE1QUNVTTcxaTJ4dlRZUXphRnB6Y0l6cTNtN1JL?= =?utf-8?B?NWxnSlZ2MnRYZDFLTW1qaGJ3UzEyNk10Q0ZvRFVBYm8vZktWNm5EcHFOUGhO?= =?utf-8?B?SFA4LzJSUGdVTldwQk00NUMwY0xXQjZrSEwzVzFvRjZOdUpJcnpqaTNQVFdY?= =?utf-8?B?NVNjbGhFcmJFNTc0QjRrRU1yZ2xROTlqc1dkNzhJUVJaMmUwVENIQTNpbW1F?= =?utf-8?B?MTQ2ekJjZmF3L2V5WkFlSnFnS0psYWQ5RXFRT0tBMDdKUkJEZ0ZoS25OT0VR?= =?utf-8?B?L3VtZDFPek13OWdVdjNjNmNYemN4bHJ3RVhVWFVDT2VZeVFMbFQ0cndzNmZJ?= =?utf-8?B?Q05KdituY25jV0Fock1KVEswT2NJNno3Nmc3dllxZ2lwVFFNdXMrQUdoMFpl?= =?utf-8?B?UXF6ZGR4UGg3VjB5NVpRaEpJSUJ2RjlqOUxQdGt1bWkxVnNrRDQ4aDA2VDVC?= =?utf-8?B?eXlkNldkSzlOc3dIV0JzYTk0bnZNSDdoTmVjVUVnWFpqYXExbUpvc20xU2RM?= =?utf-8?B?eVM2NGh6TXJkLys0S1NqVjlaSWR5S0pBa3hUZDRoWWtEZ0Z0NnBzcWxSd0ZW?= =?utf-8?B?Z0JvN0cxZ1R2K05tZzdXZnNRa043ckVJaXVrT0NrYlk5aElVc3hQZnV1YWpG?= =?utf-8?B?bmpqRzI1OElkeVhPeWRRWVg4RnVrMW5XWi9Dd2xmQ2pyTWdPYXlSZ0RydGNm?= =?utf-8?B?MFpkM1N1aG9jSTJIb21ROTlXNGZod3o0cThKZVRsa1U5UWhJOU1hS0dYUDlW?= =?utf-8?B?cVRXYzBjUDdEbURlSnNoNVB4ejlTSDBvNU9QcXR1a2dXM3kvWGN4ZkI2UkRo?= =?utf-8?B?ckh4WlJpOWdmZUhYT2tIZ3JoNUxrLytJNTJNRHd5UzEwRHB1RHRyakh0aHJu?= =?utf-8?B?TWl0aTEyYlk0T2VCWjNYZDNGNHcvNTN4K3VQM2w4dHlxVXlZZmJ3UEdLN0pa?= =?utf-8?B?Tlo5T1loOE9kUTRYYWpUd28vdmJIQzJJQlhLZUlHVVlWbDFaSWszczBENURF?= =?utf-8?B?QUlkZmRlNkpCNE5Fb000c2JmWEpTeGdWZlRkZUV5S3Nrd3ZURW1WeFR3S05x?= =?utf-8?Q?IwlnQ4Pw3C+wiTLA=3D?= X-OriginatorOrg: efficios.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1bb88c91-53dd-4461-c32f-08def573419f X-MS-Exchange-CrossTenant-AuthSource: YT2PR01MB9175.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Aug 2026 17:34:10.5348 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4f278736-4ab6-415c-957e-1f55336bd31e X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RKphU7XLB0p4EebVB32dCRpGu6AzvOVj+Rnjf2nMDNfM/smUK9B8VFhFnvOpMPgJ/YrREsWqLu0gIB/asO6QCQV6ANjBRx0K39n+nPrz2JI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT2PR01MB9537 On 2026-08-08 13:31, Mathieu Desnoyers wrote: > Implement a two-phases wildcard scan to guarantee forward progress of > synchronize_hazptr() even if there is a steady stream of ill-timed > readers which populate wildcards into per-CPU slots. > > This is performed by flipping between two wildcard values (1UL and 2UL), > and alternatively scanning for the opposite wildcard while newcoming > readers use the other one. > > There is no possibility to miss a reader because all slots for all > wildcards are accounted for during a synchronize. > > As a simplificaiton, use this period flip to drive the hazptr overflow typo: simplification Thanks, Mathieu > list selection as well, since there is really no point is making the > overflow list flip use a different state. > > Protect the wildcard flip with a mutex. > > Signed-off-by: Mathieu Desnoyers > --- > include/linux/hazptr.h | 6 ++- > kernel/hazptr.c | 98 ++++++++++++++++++++++++++++++------------ > 2 files changed, 74 insertions(+), 30 deletions(-) > > diff --git a/include/linux/hazptr.h b/include/linux/hazptr.h > index 43998bf43de4..43122c5673bd 100644 > --- a/include/linux/hazptr.h > +++ b/include/linux/hazptr.h > @@ -28,7 +28,9 @@ > > /* 4 slots (each sizeof(hazptr_slot_item)) fit in a single 64-byte cache line. */ > #define NR_HAZPTR_PERCPU_SLOTS 4 > -#define HAZPTR_WILDCARD ((void *) 0x1UL) > + > +/* The current hazard pointer wildcard. */ > +extern void *hazptr_wildcard; > > /* > * Hazard pointer slot. > @@ -243,7 +245,7 @@ void *hazptr_acquire(struct hazptr_ctx *ctx, void * const *addr_p) > #endif > if (unlikely(slot->addr)) > return __hazptr_acquire(ctx, addr_p); > - WRITE_ONCE(slot->addr, HAZPTR_WILDCARD); /* Store B */ > + WRITE_ONCE(slot->addr, READ_ONCE(hazptr_wildcard)); /* Store B */ > > /* Memory ordering: Store B before Load A. */ > smp_mb(); > diff --git a/kernel/hazptr.c b/kernel/hazptr.c > index a9d3d68a1525..d3d1050d92cf 100644 > --- a/kernel/hazptr.c > +++ b/kernel/hazptr.c > @@ -13,6 +13,17 @@ > #include > #include > > +/* > + * The current hazard pointer wildcard. Flips between 1UL and 2UL to guarantee > + * hazptr_synchronize forward progress even with a steady stream of readers. > + * This wildcard value is used by acquire to temporarily tag the per-CPU slots. > + * This also affects the overflow list selection: the current list used by > + * readers is array[(unsigned long) hazptr_wildcard - 1]. > + */ > +static DEFINE_MUTEX(hazptr_wildcard_lock); /* Protect the wildcard flip. */ > +void *hazptr_wildcard = (void *) 1UL; > +EXPORT_SYMBOL_GPL(hazptr_wildcard); > + > struct hazptr_overflow_list { > raw_spinlock_t lock; /* Lock protecting overflow list and list generation. */ > struct hlist_head head; /* Overflow list head. */ > @@ -28,8 +39,6 @@ struct hazptr_overflow_list { > * limited to the number of list elements. > */ > struct hazptr_overflow_list_flip { > - struct mutex lock; /* Mutex protecting add_idx from concurrent updates. */ > - unsigned int add_idx; /* Index of current flip-list to add to. */ > struct hazptr_overflow_list array[2]; > }; > > @@ -38,6 +47,20 @@ static DEFINE_PER_CPU(struct hazptr_overflow_list_flip, percpu_overflow_list_fli > DEFINE_PER_CPU(struct hazptr_percpu_slots, hazptr_percpu_slots); > EXPORT_PER_CPU_SYMBOL_GPL(hazptr_percpu_slots); > > +static > +void *flip_wildcard(void *wildcard) > +{ > + return ((unsigned long) wildcard == 1UL) ? (void *) 2UL : (void *) 1UL; > +} > + > +static > +bool is_wildcard(void *addr) > +{ > + if ((unsigned long) addr == 1UL || (unsigned long) addr == 2UL) > + return true; > + return false; > +} > + > static > struct hazptr_slot *hazptr_get_free_percpu_slot(struct hazptr_ctx *ctx) > { > @@ -72,7 +95,7 @@ void *__hazptr_acquire(struct hazptr_ctx *ctx, void * const *addr_p) > */ > if (unlikely(!slot)) > slot = hazptr_chain_backup_slot(ctx); > - WRITE_ONCE(slot->addr, HAZPTR_WILDCARD); /* Store B */ > + WRITE_ONCE(slot->addr, READ_ONCE(hazptr_wildcard)); /* Store B */ > > /* Memory ordering: Store B before Load A. */ > smp_mb(); > @@ -118,7 +141,9 @@ void hazptr_synchronize_overflow_list(struct hazptr_overflow_list *overflow_list > for (;;) { > void *load_addr = smp_load_acquire(&backup_slot->slot.addr); /* Load B */ > > - if (load_addr != addr && load_addr != HAZPTR_WILDCARD) > + /* We don't expect wildcards in overflow list. */ > + WARN_ON_ONCE(is_wildcard(load_addr)); > + if (load_addr != addr) > break; > raw_spin_unlock_irqrestore(&overflow_list->lock, flags); > cpu_relax(); > @@ -139,7 +164,7 @@ void hazptr_synchronize_overflow_list(struct hazptr_overflow_list *overflow_list > } > > static > -void hazptr_synchronize_cpu_slots(int cpu, void *addr) > +void hazptr_synchronize_cpu_slots(int cpu, void *addr, void *scan_wildcard) > { > struct hazptr_percpu_slots *percpu_slots = per_cpu_ptr(&hazptr_percpu_slots, cpu); > unsigned int idx; > @@ -148,7 +173,39 @@ void hazptr_synchronize_cpu_slots(int cpu, void *addr) > struct hazptr_slot_item *item = &percpu_slots->items[idx]; > > /* Busy-wait if node is found. */ > - smp_cond_load_acquire(&item->slot.addr, VAL != addr && VAL != HAZPTR_WILDCARD); /* Load B */ > + smp_cond_load_acquire(&item->slot.addr, VAL != addr && VAL != scan_wildcard); /* Load B */ > + } > +} > + > +static > +void hazptr_scan_period(void *addr, void *scan_wildcard) > +{ > + unsigned int scan_idx = (unsigned long) scan_wildcard - 1; > + int cpu; > + > + /* Scan all CPUs slots. */ > + for_each_possible_cpu(cpu) { > + struct hazptr_overflow_list_flip *overflow_list_flip = per_cpu_ptr(&percpu_overflow_list_flip, cpu); > + > + /* > + * Scan CPU slots. > + * Forward progress against recurring wildcards is guaranteed > + * by scanning for one wildcard while new elements use the > + * other wildcard value (1UL vs 2UL). > + * Forward progress against recurring single hazard pointer > + * values is guaranteed by the fact that a hazard pointer > + * is not reclaimed nor reused until the scan for that hazard > + * pointer completes, which prevents a steady flow of readers > + * to acquire that same hazard pointer value. > + */ > + hazptr_synchronize_cpu_slots(cpu, addr, scan_wildcard); > + > + /* > + * Scan backup slots in percpu overflow lists. > + * Forward progress is guaranteed by scanning one list > + * while new elements are added into the other list. > + */ > + hazptr_synchronize_overflow_list(&overflow_list_flip->array[scan_idx], addr); > } > } > > @@ -161,7 +218,7 @@ void hazptr_synchronize_cpu_slots(int cpu, void *addr) > */ > void hazptr_synchronize(void *addr) > { > - int cpu; > + void *scan_wildcard; > > /* > * Busy-wait should only be done from preemptible context. > @@ -177,33 +234,19 @@ void hazptr_synchronize(void *addr) > return; > /* Memory ordering: Store A before Load B. */ > smp_mb(); > - /* Scan all CPUs slots. */ > - for_each_possible_cpu(cpu) { > - struct hazptr_overflow_list_flip *overflow_list_flip = per_cpu_ptr(&percpu_overflow_list_flip, cpu); > - unsigned int scan_idx; > - > - /* Scan CPU slots. */ > - hazptr_synchronize_cpu_slots(cpu, addr); > > - /* > - * Scan backup slots in percpu overflow lists. > - * Forward progress is guaranteed by scanning one list > - * while new elements are added into the other list. > - */ > - guard(mutex)(&overflow_list_flip->lock); > - scan_idx = overflow_list_flip->add_idx ^ 1; > - hazptr_synchronize_overflow_list(&overflow_list_flip->array[scan_idx], addr); > - /* Flip current list. */ > - WRITE_ONCE(overflow_list_flip->add_idx, scan_idx); > - hazptr_synchronize_overflow_list(&overflow_list_flip->array[scan_idx ^ 1], addr); > - } > + guard(mutex)(&hazptr_wildcard_lock); > + scan_wildcard = flip_wildcard(hazptr_wildcard); > + hazptr_scan_period(addr, scan_wildcard); > + WRITE_ONCE(hazptr_wildcard, scan_wildcard); /* Flip the current wildcard. */ > + hazptr_scan_period(addr, flip_wildcard(scan_wildcard)); > } > EXPORT_SYMBOL_GPL(hazptr_synchronize); > > struct hazptr_slot *hazptr_chain_backup_slot(struct hazptr_ctx *ctx) > { > struct hazptr_overflow_list_flip *overflow_list_flip = this_cpu_ptr(&percpu_overflow_list_flip); > - unsigned int list_idx = READ_ONCE(overflow_list_flip->add_idx); > + unsigned int list_idx = (unsigned long) READ_ONCE(hazptr_wildcard) - 1; > struct hazptr_overflow_list *overflow_list = &overflow_list_flip->array[list_idx]; > struct hazptr_slot *slot = &ctx->backup_slot.slot; > > @@ -233,7 +276,6 @@ void __init hazptr_init(void) > for_each_possible_cpu(cpu) { > struct hazptr_overflow_list_flip *overflow_list_flip = per_cpu_ptr(&percpu_overflow_list_flip, cpu); > > - mutex_init(&overflow_list_flip->lock); > for (int i = 0; i < 2; i++) { > raw_spin_lock_init(&overflow_list_flip->array[i].lock); > INIT_HLIST_HEAD(&overflow_list_flip->array[i].head); -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com