From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1629EC83F22 for ; Tue, 15 Jul 2025 16:31:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABB876B0092; Tue, 15 Jul 2025 12:31:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A93106B0093; Tue, 15 Jul 2025 12:31:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A9316B0095; Tue, 15 Jul 2025 12:31:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8B6CF6B0092 for ; Tue, 15 Jul 2025 12:31:19 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0F5B31A04A2 for ; Tue, 15 Jul 2025 16:31:19 +0000 (UTC) X-FDA: 83667039078.09.16EB695 Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf18.hostedemail.com (Postfix) with ESMTP id C120D1C0017 for ; Tue, 15 Jul 2025 16:31:16 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hh7pQU4T; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of boqun.feng@gmail.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=boqun.feng@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752597076; a=rsa-sha256; cv=none; b=sIS9Db69B1MnyvxIegjf8PfSOFct//DydLOedqjD15PNtrKfARGuyaVm8kbZEztIbHv47W iFvTzxmRfYmdc8YdITa335mhjolAL0vjb7f7I3WaD/33rQ46NBrkUBXWsmRyhl7nLkqZ3m unK7A+NHzLQsyhm6iAFohyH1BLeKWy8= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=hh7pQU4T; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of boqun.feng@gmail.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=boqun.feng@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752597076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Mjhxjbj7iS8ZyBEjzsqSgbZseWbedQ/eZ617a3fR4vs=; b=OrviGXMg3WKwS4pxfQxuPYp7VE70Dib5LJwU8AtIaYHdVrL81WRUaBW/w7Ou6oxm980Ut0 ZGOSPAHdIqBWyHH8zwkucvXc7miNsZtqrUjy2wok2WzxYr3u3/kchNOVpoQgF3oZf+abnI LBFFDWXM3ooY/y3LGzJu13Yk0+ircVk= Received: by mail-qt1-f173.google.com with SMTP id d75a77b69052e-4ab8403b6daso7848241cf.2 for ; Tue, 15 Jul 2025 09:31:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1752597076; x=1753201876; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:feedback-id:from:to:cc:subject:date :message-id:reply-to; bh=Mjhxjbj7iS8ZyBEjzsqSgbZseWbedQ/eZ617a3fR4vs=; b=hh7pQU4TbryjVqXqgPgS+jcZeLdezS9gbXbu+T1Rte/HAHbrPlg8/NW8HOVZg5d1Iq wMRMTYiD5xN3LR7EIDNfQqc8mRJJS8+odoJMnKMktPSdy6CK1/KmM5Hc6YOXNFCqOfKK q3u/nbKH9GxPtjR2smreSElpCBkWrvAVMjxONtxn3PYLeTh3/k5xMYDAMT8DujTwTadT APOc1Pa/8zZzYCtZ0rUi3uzPmbtP0SYN2FVpfg8G/QZ6hT8vGAKBImLSwfMfyobDS9EA Z2YKeY7q7IubSJAyPeo8FREVNNdwlTreZI79saiDJctV425Nl8NXL5DrKShvj0v1ak0t y2WA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752597076; x=1753201876; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:feedback-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Mjhxjbj7iS8ZyBEjzsqSgbZseWbedQ/eZ617a3fR4vs=; b=EzrDtIxNFsWV3umzAePzGxJ7Wi/i9mJE+byrh+Z77OyNE8wwT67HzEh97jMMESo6eB CO63bJt85E36NpqyfYzmM8Az9hyTN0+swLP1hwXoDkS5n3XFwTgRdEV9RzEHtp7WjG1o g8Rw/ZJ6b72DKhDzhKg5hClFbFvIQHCrSSnZfeczkHtvb1ouyeRc2gQo+/MgYTsZ/xPm IU7NVkNFeimJFqLt+AIrGFrFryPAfrOvqyxJNDzvM0B99E2zfGdGWFAMS2pXT9CnHkkl PeyhNSajh8FwQWw2XESLbMIuO8dwZyTFutwndyn/WAwycC7z+n7d7MgNGMTS271+fAEp 1CBw== X-Forwarded-Encrypted: i=1; AJvYcCW5SZ6C65/J2BgeSMH8SzfMAnY9A16ft6yWSIawq1opbGVPgwydxLWtVUUgXIAA75t5UW/Qyg8MGg==@kvack.org X-Gm-Message-State: AOJu0YxOpjVoFlcDut7lGg4Cne3JHl7C+4xyLh3hqHwPafn7I4IfTI3N /pu46QTl3FZ618Ncc0XMybMemeLmY+p9d7NaLiv0t4YYZTTJDXaHYTnB X-Gm-Gg: ASbGncuUIidberqnPQ+N7udR2eB7qpoUKs39e8r6y1/6U2QgAF14C533beE97kJUDle Vd/z2hXp3Z6M6Sx5vwwnHFhVXSgYeN3ZWQjxnGgOM6su/LE/fyyLpJ57IrdrYjLyfun+W6iDCs7 J73kSl/ulVGa4GrcQGEyAhIeVl6+FbbtXascQbqYwlYXpEFEs4YsCuOklo/HYEdzgYMe6OJvRsG HxiuyrhpJM5LeM2KfniLZnGPbiL2eygMYk+akvZnIwC8jt7O8pckWdmfdCvLaIAnPIRL30BkEzd HL8xueSamwBni7wBIga+WqhLZu/KxcDdzYiC5Pg/ry7NKyYeAXeLAioj9W4aUdJVWTnyYd2bBPY NNrEwqvyheoFvhI2Li2c74HFO5k06B45f7dKhdINYAkync9wLBi2vV80wetU8nJK9zcXsu/BCOZ 4NBz8gGx9fa5qs X-Google-Smtp-Source: AGHT+IGKcT812uvfhN6lx3CaHfEFlXJDMWfENRiBLEgYlLqzk0l/AjvFrC+4wp7erxyiJHyRaZV26g== X-Received: by 2002:a05:622a:1449:b0:4ab:7cf5:877b with SMTP id d75a77b69052e-4ab909b6fc6mr2180011cf.4.1752597075580; Tue, 15 Jul 2025 09:31:15 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4ab7f75fcb5sm11048021cf.27.2025.07.15.09.31.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jul 2025 09:31:15 -0700 (PDT) Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfauth.phl.internal (Postfix) with ESMTP id 906D5F40066; Tue, 15 Jul 2025 12:31:14 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Tue, 15 Jul 2025 12:31:14 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdefgdehheefvdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhepvdegueegtdejkeehueekheeffeeiffeilefgteetlefhheegkeejtdeiveejheef necuffhomhgrihhnpehophgvnhhgrhhouhhprdhorhhgpdhruhhsthdqlhgrnhhgrdhorh hgpdhgihhthhhusgdrtghomhenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhep mhgrihhlfhhrohhmpegsohhquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhith ihqdeiledvgeehtdeigedqudejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgr ihhlrdgtohhmsehfihigmhgvrdhnrghmvgdpnhgspghrtghpthhtohepudekpdhmohguvg epshhmthhpohhuthdprhgtphhtthhopehlohhsshhinheskhgvrhhnvghlrdhorhhgpdhr tghpthhtoheplhgvvhihmhhithgthhgvlhhltdesghhmrghilhdrtghomhdprhgtphhtth hopehojhgvuggrsehkvghrnhgvlhdrohhrghdprhgtphhtthhopegrlhgvgidrghgrhihn ohhrsehgmhgrihhlrdgtohhmpdhrtghpthhtohepghgrrhihsehgrghrhihguhhordhnvg htpdhrtghpthhtohepsghjohhrnhefpghghhesphhrohhtohhnmhgrihhlrdgtohhmpdhr tghpthhtoheprgdrhhhinhgusghorhhgsehkvghrnhgvlhdrohhrghdprhgtphhtthhope grlhhitggvrhihhhhlsehgohhoghhlvgdrtghomhdprhgtphhtthhopehtmhhgrhhoshhs sehumhhitghhrdgvughu X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 15 Jul 2025 12:31:13 -0400 (EDT) Date: Tue, 15 Jul 2025 09:31:12 -0700 From: Boqun Feng To: Benno Lossin Cc: Mitchell Levy , Miguel Ojeda , Alex Gaynor , Gary Guo , =?iso-8859-1?Q?Bj=F6rn?= Roy Baron , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v2 3/5] rust: percpu: add a rust per-CPU variable test Message-ID: References: <20250712-rust-percpu-v2-0-826f2567521b@gmail.com> <20250712-rust-percpu-v2-3-826f2567521b@gmail.com> <68762e19.170a0220.33e203.a0b7@mx.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: C120D1C0017 X-Stat-Signature: sgwp198txi5sdiin4mogpypekgh1mp53 X-Rspam-User: X-HE-Tag: 1752597076-848191 X-HE-Meta: U2FsdGVkX1+7xTqj6yo74kyl15JJEDCaV9lz0tUdhkqVl5YnC7MBm8j9d1Cw9+9RyS3iL7J5EdEhNDG9LgbDlRgX50HyTGhgsasaa0G/fb9ZZ7h2J/bsUWzXX1EH0ocZ0zWv/4wWtE9XlmF2l827Expwz7F+AwQMcVG0K9AcgNlJB076oT6uJSJQGIMyPfRCXvR7dWthvmeRoJpyAFP0g6Kyo5hmaewovnYQB9GtCU4T6SvYHh5YXUbI3ANWhfpjr7QSvrvK5Jtp+nA2wW17JaXxe10QLacnzi4UBlMDGecztf9IH8B5bMNYwruEIcBQ0BLmjBgOqaQIS9cfBfCBgCpf9GHCthPS4Aww27+EVZQk4s/zmUDotleRDrNnMN1SOA2vYul7/g38eCVUoVzlc2vYmY2HYHwTdyxuw9OmevovSgbT6fMidFBVcVY6hQ1kCNBH9XAvL8gh8W/7Jlw52KtUw8jwdwDi58BbXJol4Zh5JXd9OeiqmZ9Ei+EMNyAyidd0gzHmgyJQiJ9Xs8KzQqWZnjiGjbmMV3YYpKRB3anfowOno66YGaMEYpIlLrvg6PGMCjpiDHl9loGA8ugEXAOfG8MCRpoZWhm2+gR2m6viJGEMVvRAN4izP64Of5gcxKfFE8xQPV7je1PvAFw+Zbmk1GdNmuLiRXaN7eIuKQfKjIQFdeQ69Fgv+74WXBhlvg6PukYOWZk9ksOwvIb97UeODW/BgmEPdk5nQt7qWuj4cQji76EzkwPynzr3YY3gXMSnk200IzCFHwliTJmdK/ZgyQFU5XNwBc/X1FG33cB96VsEf00PAA+RlUi/q3sgfCI3COqo4qS4A0lbm/raUN3FHSRTdhGk05e9gbnXlG4XuDonSrozQpT8BzOjZaIJsSokZYoK8yUhdo5nIfkvfYydwSrhbJruMc5YHV38ZXOn+rtPShhNR3ioC8/qWPeCAYVMpz8h/z3iv2eFAMa +qItMRoJ xNLrKJw0HaDFhknNSSiGcia6dknQ7Welb+i561VD3+gDLarM+FZh/1QM1NPrVud3BDpK8EZWVoNnW5B8wwmCWCveyRnA7pqg1uW8uTehN0r4fwNvdgxDy57OZyC2L3HuHjdK1sKQMOxjlRMcvqNgt+bFtbckV8lhES6yO1Ed5QgQpnsdQz6djvxL+Ntf6RIX+mghEy3yPoCHfHcABkxRSK8Gu7BdT8fkLWJw0l4pPy+3vUnXFlS3ckNzcHppzci29Cr7tJbugLUi9pKn57HjyQIA5I6Sm48Mr+7ATSj26rqP2YCWCNE6M9G8nhZozDu/p49fbENG6kjMO6XF5n/EqDWbrYVqh2x4XUWMd9Wlo6BWBxm13qkrOWbCuALuk7OuSN1rokXNkR1r7UyvT6lUx+4YW6X+X5ZWVC337ECSp/d4vQQoqXC6kX02VO5pp0/A8NUOayeECPMnsyoLLNrS26Dgiq/TuvV4Q0GQ5csZAA30SsN2YzGCOgagTQOL42Rt5gQWl9CILbyK01uL1EC7caeAjVsR2zt1rj9WYwC4B1OiYv6HsANqNYMMvO6D2dghgTiwJjglsrjll+mlHaiBwOuWMCgIvJ8atVpnGjMIdaojk65vtHWqCaCsAmHh+bsbU3qR/DciOln4Nu1kooEbJg8Fh4AbPGJiqNZjHGSpz3Rpw1so= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jul 15, 2025 at 05:55:13PM +0200, Benno Lossin wrote: > On Tue Jul 15, 2025 at 4:10 PM CEST, Boqun Feng wrote: > > On Tue, Jul 15, 2025 at 01:31:06PM +0200, Benno Lossin wrote: > > [...] > >> >> > +impl kernel::Module for PerCpuTestModule { > >> >> > + fn init(_module: &'static ThisModule) -> Result { > >> >> > + pr_info!("rust percpu test start\n"); > >> >> > + > >> >> > + let mut native: i64 = 0; > >> >> > + // SAFETY: PERCPU is properly defined > >> >> > + let mut pcpu: StaticPerCpu = unsafe { unsafe_get_per_cpu!(PERCPU) }; > >> >> > >> >> I don't understand why we need unsafe here, can't we just create > >> >> something specially in the `define_per_cpu` macro that is then confirmed > >> >> by the `get_per_cpu!` macro and thus it can be safe? > >> > > >> > As is, something like > >> > define_per_cpu!(PERCPU: i32 = 0); > >> > > >> > fn func() { > >> > let mut pcpu: StaticPerCpu = unsafe { unsafe_get_per_cpu!(PERCPU) }; > >> > } > >> > will compile, but any usage of `pcpu` will be UB. This is because > >> > `unsafe_get_per_cpu!` is just blindly casting pointers and, as far as I > >> > know, the compiler does not do any checking of pointer casts. If you > >> > have thoughts/ideas on how to get around this problem, I'd certainly > >> > *like* to provide a safe API here :) > >> > >> I haven't taken a look at your implementation, but you do have the type > >> declared in `define_per_cpu!`, so it's a bit of a mystery to me why you > >> can't get that out in `unsafe_get_per_cpu!`... > >> > >> Maybe in a few weeks I'll be able to take a closer look. > >> > >> >> > + // SAFETY: We only have one PerCpu that points at PERCPU > >> >> > + unsafe { pcpu.get(CpuGuard::new()) }.with(|val: &mut i64| { > >> >> > >> >> Hmm I also don't like the unsafe part here... > >> >> > >> >> Can't we use the same API that `thread_local!` in the standard library > > > > First of all, `thread_local!` has to be implemented by some sys-specific > > unsafe mechanism, right? For example on unix, I think it's using > > pthread_key_t: > > > > https://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_key_create.html > > > > what we are implementing (or wrapping) is the very basic unsafe > > mechanism for percpu here. Surely we can explore the design for a safe > > API, but the unsafe mechanism is probably necessary to look into at > > first. > > But this is intended to be used by drivers, right? If so, then we should Not necessarily only for drivers, we can also use it for implementing other safe abstraction (e.g. hazard pointers, percpu counters etc) > do our usual due diligence and work out a safe abstraction. Only fall > back to unsafe if it isn't possible. > All I'm saying is instead of figuring out a safe abstraction at first, we should probably focus on identifying how to implement it and which part is really unsafe and the safety requirement for that. > I'm not familiar with percpu, but from the name I assumed that it's > "just a variable for each cpu" so similar to `thread_local!`, but it's > bound to the specific cpu instead of the thread. > > That in my mind should be rather easy to support in Rust at least with > the thread_local-style API. You just need to ensure that no reference > can escape the cpu, so we can make it `!Send` & `!Sync` + rely on klint Not really, in kernel, we have plenty of use cases that we read the other CPU's percpu variables. For example, each CPU keeps it's own counter and we sum them other in another CPU. If we would like to model it conceptually, it's more like an array that's index by CpuId to me. > to detect context switches. > > >> >> has: > >> >> > >> >> https://doc.rust-lang.org/std/macro.thread_local.html > >> >> > >> >> So in this example you would store a `Cell` instead. > >> >> > >> >> I'm not familiar with per CPU variables, but if you're usually storing > >> >> `Copy` types, then this is much better wrt not having unsafe code > >> >> everywhere. > >> >> > >> >> If one also often stores `!Copy` types, then we might be able to get > >> >> away with `RefCell`, but that's a small runtime overhead -- which is > >> >> probably bad given that per cpu variables are most likely used for > >> >> performance reasons? In that case the user might just need to store > >> >> `UnsafeCell` and use unsafe regardless. (or we invent something > > > > This sounds reasonable to me. > > > >> >> specifically for that case, eg tokens that are statically known to be > >> >> unique etc) > >> > > >> > I'm open to including a specialization for `T: Copy` in a similar vein > >> > to what I have here for numeric types. Off the top of my head, that > >> > shouldn't require any user-facing `unsafe`. But yes, I believe there is > >> > a significant amount of interest in having `!Copy` per-CPU variables. > >> > (At least, I'm interested in having them around for experimenting with > >> > using Rust for HV drivers.) > >> > >> What kinds of types would you like to store? Allocations? Just integers > >> in bigger structs? Mutexes? > >> > > > > In the VMBus driver, there is a percpu work_struct. > > Do you have a link? Or better yet a Rust struct description of what you > think it will look like :) > Not Rust code yet, but here is the corresponding C code: https://github.com/Rust-for-Linux/linux/blob/rust-next/drivers/hv/vmbus_drv.c#L1396 But please note that we are not solely developing the abstraction for this usage, but more for generally understand how to wrap percpu functionality similar to the usage in C. > >> > I would definitely like to avoid *requiring* the use of `RefCell` since, > >> > as you mention, it does have a runtime overhead. Per-CPU variables can > >> > be used for "logical" reasons rather than just as a performance > >> > optimization, so there might be some cases where paying the runtime > >> > overhead is ok. But that's certainly not true in all cases. That said, > >> > perhaps there could be a safely obtainable token type that only passes a > >> > `&T` (rather than a `&mut T`) to its closure, and then if a user doesn't > >> > mind the runtime overhead, they can choose `T` to be a `RefCell`. > >> > Thoughts? > >> > >> So I think using an API similar to `thread_local!` will allow us to have > >> multiple other APIs that slot into that. `Cell` for `T: Copy`, > >> `RefCell` for cases where you don't care about the runtime overhead, > >> plain `T` for cases where you only need `&T`. For the case where you > >> need `&mut T`, we could have something like a `TokenCell` that gives > >> out a token that you need to mutably borrow in order to get `&mut T`. > >> Finally for anything else that is too restricted by this, users can also > >> use `UnsafeCell` although that requires `unsafe`. > >> > >> I think the advantage of this is that the common cases are all safe and > >> very idiomatic. In the current design, you *always* have to use unsafe. > >> > > > > I agree, but like I said, we need to figure out the unsafe interface > > that C already uses and build API upon it. I think focusing on the > > unsafe mechanism may be the way to start: you cannot implement something > > that cannot be implemented, and we don't have the magic pthread_key here > > ;-) > > Sure we can do some experimentation, but I don't think we should put > unsafe abstractions upstream that we intend to replace with a safe > abstraction later. Otherwise people are going to depend on it and it's I doubt we can replace the unsafe abstraction with a safe one, if users really care the performance then they would really need to use some unsafe API to build their safe abstraction. > going to be a mess. Do the experimenting out of tree and learn there. I disagree, Rust as a language its own should be able to do what C does including being able to implement the percpu functionality same as C, there is nothing wrong with a set of Rust primitives in the kernel that provides fundamental percpu functionality the other core facilities can rely on. The better part is that it will have all the safety requirement documented well. Regards, Boqun > > --- > Cheers, > Benno