* Re: Official Linux system wrapper library?
From: Dave Martin @ 2018-11-14 10:54 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Daniel Colascione, Florian Weimer, Michael Kerrisk (man-pages),
linux-kernel, Joel Fernandes, Linux API, Willy Tarreau,
Vlastimil Babka, Carlos O'Donell, libc-alpha@sourceware.org
In-Reply-To: <69B07026-5E8B-47FC-9313-E51E899FAFB0@amacapital.net>
On Tue, Nov 13, 2018 at 12:58:39PM -0800, Andy Lutomirski wrote:
>
> > On Nov 13, 2018, at 11:39 AM, Dave Martin <Dave.Martin@arm.com> wrote:
> >
> > On Mon, Nov 12, 2018 at 05:19:14AM -0800, Daniel Colascione wrote:
> >
> > [...]
> >
> >> We can learn something from how Windows does things. On that system,
> >> what we think of as "libc" is actually two parts. (More, actually, but
> >> I'm simplifying.) At the lowest level, you have the semi-documented
> >> ntdll.dll, which contains raw system call wrappers and arcane
> >> kernel-userland glue. On top of ntdll live the "real" libc
> >> (msvcrt.dll, kernel32.dll, etc.) that provide conventional
> >> application-level glue. The tight integration between ntdll.dll and
> >> the kernel allows Windows to do very impressive things. (For example,
> >> on x86_64, Windows has no 32-bit ABI as far as the kernel is
> >> concerned! You can still run 32-bit programs though, and that works
> >> via ntdll.dll essentially shimming every system call and switching the
> >> processor between long and compatibility mode as needed.) Normally,
> >> you'd use the higher-level capabilities, but if you need something in
> >> ntdll (e.g., if you're Cygwin) nothing stops your calling into the
> >> lower-level system facilities directly. ntdll is tightly bound to the
> >> kernel; the higher-level libc, not so.
> >>
> >> We should adopt a similar approach. Shipping a lower-level
> >> "liblinux.so" tightly bound to the kernel would not only let the
> >> kernel bypass glibc's "editorial discretion" in exposing new
> >> facilities to userspace, but would also allow for tighter user-kernel
> >> integration that one can achieve with a simplistic syscall(2)-style
> >> escape hatch. (For example, for a long time now, I've wanted to go
> >> beyond POSIX and improve the system's signal handling API, and this
> >> improvement requires userspace cooperation.) The vdso is probably too
> >> small and simplistic to serve in this role; I'd want a real library.
> >
> > Can you expand on your reasoning here?
> >
> > Playing devil's advocate:
> >
> > If the library is just exposing the syscall interface, I don't see
> > why it _couldn't_ fit into the vdso (or something vdso-like).
> >
> > If a separate library, I'd be concerned that it would accumulate
> > value-add bloat over time, and the kernel ABI may start to creep since
> > most software wouldn't invoke the kernel directly any more. Even if
> > it's maintained in the kernel tree, its existence as an apparently
> > standalone component may encourage forking, leading to a potential
> > compatibility mess.
> >
> > The vdso approach would mean we can guarantee that the library is
> > available and up to date at runtime, and may make it easier to keep
> > what's in it down to sane essentials.
>
> Hmm. Putting on my vDSO hat:
>
> The vDSO could provide all kinds of nifty things. Better exception
> handling comes to mind. But it has two major limitations that severely
> restrict what it can do:
>
> - It can’t allocate memory. We probably want to keep it that way.
>
> - It can’t use TLS. Solving this without genuinely awful ABI issues
> may be extremely hard. We *could* require callers to pass a thread
> pointer in, I suppose.
>
> Also, if we make the vDSO stateful, CRIU is going to have a blast. We
> might need to expose explicit save and restore abilities.
>
> As a straw man use case, it would be neat if DSOs (or the loader,
> maybe) could register a list of exception fixups per DSO. The kernel
> could consult these lists before delivering a signal. ISTM it wouldn’t
> be so crazy if the vDSO handled registration, although it could uses
> syscalls as well. If the vDSO did it, it would need somewhere to put
> the lists.
Fair points, though this is rather what I meant by "sane essentials".
Because there are strict limits on what can be done in the vDSO, it may
be more bloat-resistant and more conservatively maintained.
This might provide a way to push some dumb compatibility kludge code
that receives little ongoing maintenance outside the privilege wall,
whereas it has to sit in the kernel proper today.
In theory we could opt to advertise new syscalls only via vDSO entry
points, and not maintain __NR_xxx values for them (which may or may
not upset ptrace users.) Anyway, I digress...
Cheers
---Dave
^ permalink raw reply
* Re: [PATCH v2] UAPI: move RENAME_* definitions to separated file
From: Yury Norov @ 2018-11-14 7:26 UTC (permalink / raw)
To: Andrew Morton, Alexander Viro, Florian Weimer
Cc: Norov, Yuri, linux-kernel@vger.kernel.org,
linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org,
libc-alpha@sourceware.org
In-Reply-To: <20181114071346.31210-1-ynorov@caviumnetworks.com>
Hi all,
It seems that I forgot to rename guards in v2. This is the correct
version. My apologize for noise.
Yury
>From 61215b1cb559e69fb57f2a82864e72e3a1ef4b12 Mon Sep 17 00:00:00 2001
From: Yury Norov <ynorov@caviumnetworks.com>
Date: Wed, 14 Nov 2018 09:51:49 +0300
To: Andrew Morton <akpm@linux-foundation.org>,
Alexander Viro <viro@zeniv.linux.org.uk>,
Florian Weimer <fweimer@redhat.com>
CC: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org,
linux-api@vger.kernel.org, libc-alpha@sourceware.org
Subject: [PATCH v2] UAPI: move RENAME_* definitions to separated file
Discussion:
https://lore.kernel.org/lkml/20180702084622.GA15274@yury-thinkpad/
Although RENAME_* macros are exposed in kernel headers, they are not
used by glibc. That's because linux/fs.h which hosts RENAME_* is
considered unsuitable by glibc developers:
As Florian Weimer wrote:
> <linux/fs.h> undefines and defines macros not mentioned in the standards
> (and it even contains a few unrelated structs), so we cannot include it
> without _GNU_SOURCE.
>
> It might be possible to include it only for _GNU_SOURCE, but there are a
> lot of things in <linux/fs.h>, so that does not seem to be particularly
> advisable.
In this patch RENAME_* is removed to a separated header file that
conforms the standards and therefore is suitable for inclusion by glibc.
v2:
- rename dedicated file to 'rename-flags.h', as requested by Florian.
Signed-off-by: Yury Norov <ynorov@caviumnetworks.com>
---
include/uapi/linux/fs.h | 4 +---
include/uapi/linux/rename-flags.h | 12 ++++++++++++
2 files changed, 13 insertions(+), 3 deletions(-)
create mode 100644 include/uapi/linux/rename-flags.h
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index 73e01918f996..34c308203746 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -39,9 +39,7 @@
#define SEEK_HOLE 4 /* seek to the next hole */
#define SEEK_MAX SEEK_HOLE
-#define RENAME_NOREPLACE (1 << 0) /* Don't overwrite target */
-#define RENAME_EXCHANGE (1 << 1) /* Exchange source and dest */
-#define RENAME_WHITEOUT (1 << 2) /* Whiteout source */
+#include <linux/rename-flags.h>
struct file_clone_range {
__s64 src_fd;
diff --git a/include/uapi/linux/rename-flags.h b/include/uapi/linux/rename-flags.h
new file mode 100644
index 000000000000..7178f0565657
--- /dev/null
+++ b/include/uapi/linux/rename-flags.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _UAPI_LINUX_RENAME_FLAGS_H
+#define _UAPI_LINUX_RENAME_FLAGS_H
+
+/*
+ * Definitions for rename syscall family.
+ */
+#define RENAME_NOREPLACE (1 << 0) /* Don't overwrite target */
+#define RENAME_EXCHANGE (1 << 1) /* Exchange source and dest */
+#define RENAME_WHITEOUT (1 << 2) /* Whiteout source */
+
+#endif /* _UAPI_LINUX_RENAME_FLAGS_H */
--
2.17.1
^ permalink raw reply related
* [PATCH v2] UAPI: move RENAME_* definitions to separated file
From: Yury Norov @ 2018-11-14 7:13 UTC (permalink / raw)
To: Andrew Morton, Alexander Viro, Florian Weimer
Cc: Norov, Yuri, linux-kernel@vger.kernel.org,
linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org,
libc-alpha@sourceware.org
Discussion:
https://lore.kernel.org/lkml/20180702084622.GA15274@yury-thinkpad/
Although RENAME_* macros are exposed in kernel headers, they are not
used by glibc. That's because linux/fs.h which hosts RENAME_* is
considered unsuitable by glibc developers:
As Florian Weimer wrote:
> <linux/fs.h> undefines and defines macros not mentioned in the standards
> (and it even contains a few unrelated structs), so we cannot include it
> without _GNU_SOURCE.
>
> It might be possible to include it only for _GNU_SOURCE, but there are a
> lot of things in <linux/fs.h>, so that does not seem to be particularly
> advisable.
In this patch RENAME_* is removed to a separated header file that
conforms the standards and therefore is suitable for inclusion by glibc.
v2:
- rename dedicated file to 'rename-flags.h', as requested by Florian.
Signed-off-by: Yury Norov <ynorov@caviumnetworks.com>
---
include/uapi/linux/fs.h | 4 +---
include/uapi/linux/rename-flags.h | 12 ++++++++++++
2 files changed, 13 insertions(+), 3 deletions(-)
create mode 100644 include/uapi/linux/rename-flags.h
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index 73e01918f996..34c308203746 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -39,9 +39,7 @@
#define SEEK_HOLE 4 /* seek to the next hole */
#define SEEK_MAX SEEK_HOLE
-#define RENAME_NOREPLACE (1 << 0) /* Don't overwrite target */
-#define RENAME_EXCHANGE (1 << 1) /* Exchange source and dest */
-#define RENAME_WHITEOUT (1 << 2) /* Whiteout source */
+#include <linux/rename-flags.h>
struct file_clone_range {
__s64 src_fd;
diff --git a/include/uapi/linux/rename-flags.h b/include/uapi/linux/rename-flags.h
new file mode 100644
index 000000000000..7178f0565657
--- /dev/null
+++ b/include/uapi/linux/rename-flags.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _UAPI_LINUX_RENAME_H
+#define _UAPI_LINUX_RENAME_H
+
+/*
+ * Definitions for rename syscall family.
+ */
+#define RENAME_NOREPLACE (1 << 0) /* Don't overwrite target */
+#define RENAME_EXCHANGE (1 << 1) /* Exchange source and dest */
+#define RENAME_WHITEOUT (1 << 2) /* Whiteout source */
+
+#endif /* _UAPI_LINUX_RENAME_H */
--
2.17.1
^ permalink raw reply related
* Re: [PATCH v5 05/27] Documentation/x86: Add CET description
From: Yu-cheng Yu @ 2018-11-13 21:02 UTC (permalink / raw)
To: Borislav Petkov
Cc: x86, H. Peter Anvin, Thomas Gleixner, Ingo Molnar, linux-kernel,
linux-doc, linux-mm, linux-arch, linux-api, Arnd Bergmann,
Andy Lutomirski, Balbir Singh, Cyrill Gorcunov, Dave Hansen,
Eugene Syromiatnikov, Florian Weimer, H.J. Lu, Jann Horn,
Jonathan Corbet, Kees Cook, Mike Kravetz, Nadav Amit,
Oleg Nesterov, Pa
In-Reply-To: <20181113184337.GM10502@zn.tnic>
On Tue, 2018-11-13 at 19:43 +0100, Borislav Petkov wrote:
> On Thu, Oct 11, 2018 at 08:15:01AM -0700, Yu-cheng Yu wrote:
> > Explain how CET works and the no_cet_shstk/no_cet_ibt kernel
> > parameters.
> >
> > Signed-off-by: Yu-cheng Yu <yu-cheng.yu@intel.com>
> > ---
> > .../admin-guide/kernel-parameters.txt | 6 +
> > Documentation/index.rst | 1 +
> > Documentation/x86/index.rst | 11 +
> > Documentation/x86/intel_cet.rst | 266 ++++++++++++++++++
> > 4 files changed, 284 insertions(+)
> > create mode 100644 Documentation/x86/index.rst
> > create mode 100644 Documentation/x86/intel_cet.rst
>
> So this patch should probably come first in the series so that a reader
> can know what to expect...
>
> > diff --git a/Documentation/admin-guide/kernel-parameters.txt
> > b/Documentation/admin-guide/kernel-parameters.txt
> > index 92eb1f42240d..3854423f7c86 100644
> > --- a/Documentation/admin-guide/kernel-parameters.txt
> > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > @@ -2764,6 +2764,12 @@
> > noexec=on: enable non-executable mappings (default)
> > noexec=off: disable non-executable mappings
> >
> > + no_cet_ibt [X86-64] Disable indirect branch tracking for
> > user-mode
> > + applications
> > +
> > + no_cet_shstk [X86-64] Disable shadow stack support for user-
> > mode
> > + applications
> > +
> > nosmap [X86]
> > Disable SMAP (Supervisor Mode Access Prevention)
> > even if it is supported by processor.
> > diff --git a/Documentation/index.rst b/Documentation/index.rst
> > index 5db7e87c7cb1..1cdc139adb40 100644
> > --- a/Documentation/index.rst
> > +++ b/Documentation/index.rst
>
> Please integrate scripts/checkpatch.pl into your patch creation
> workflow. Some of the warnings/errors *actually* make sense:
>
> WARNING: Missing or malformed SPDX-License-Identifier tag in line 1
> #76: FILE: Documentation/x86/index.rst:1:
> +=======================
>
> WARNING: Missing or malformed SPDX-License-Identifier tag in line 1
> #93: FILE: Documentation/x86/intel_cet.rst:1:
> +=========================================
>
> > @@ -104,6 +104,7 @@ implementation.
> > :maxdepth: 2
> >
> > sh/index
> > + x86/index
> >
> > Filesystem Documentation
> > ------------------------
> > diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
> > new file mode 100644
> > index 000000000000..9c34d8cbc8f0
> > --- /dev/null
> > +++ b/Documentation/x86/index.rst
> > @@ -0,0 +1,11 @@
> > +=======================
> > +X86 Documentation
> > +=======================
> > +
> > +Control Flow Enforcement
> > +========================
> > +
> > +.. toctree::
> > + :maxdepth: 1
> > +
> > + intel_cet
> > diff --git a/Documentation/x86/intel_cet.rst
> > b/Documentation/x86/intel_cet.rst
> > new file mode 100644
> > index 000000000000..946f4802a51f
> > --- /dev/null
> > +++ b/Documentation/x86/intel_cet.rst
> > @@ -0,0 +1,266 @@
> > +=========================================
> > +Control Flow Enforcement Technology (CET)
> > +=========================================
> > +
> > +[1] Overview
> > +============
> > +
> > +Control Flow Enforcement Technology (CET) provides protection against
> > +return/jump-oriented programming (ROP) attacks. It can be implemented
> > +to protect both the kernel and applications. In the first phase,
> > +only the user-mode protection is implemented on the 64-bit kernel.
>
> s/the// is implemented in 64-bit mode.
>
> > +However, 32-bit applications are supported under the compatibility
> > +mode.
>
> Drop "However":
>
> "32-bit applications are, of course, supported in compatibility mode."
>
> > +
> > +CET includes shadow stack (SHSTK) and indirect branch tracking (IBT).
>
> "CET introduces two a shadow stack and an indirect branch tracking mechanism."
>
> > +The SHSTK is a secondary stack allocated from memory. The processor
>
> s/The//
>
> > +automatically pushes/pops a secure copy to the SHSTK every return
> > +address and,
>
> that reads funny - pls reorganize. Also, what is a "secure copy"?
>
> You mean a copy of every return address which software cannot access?
>
> > by comparing the secure copy to the program stack copy,
> > +verifies function returns are as intended.
>
> ... have not been corrupted/modified."
>
> > The IBT verifies all
> > +indirect CALL/JMP targets are intended and marked by the compiler with
> > +'ENDBR' op codes.
>
> "opcode" - one word. And before you use "ENDBR" you need to explain it
> above what it is.
>
> /me reads further... encounters ENDBR's definition...
>
> ah, ok, so you should say something like
>
> "... and marked by the compiler with the ENDBR opcode (see below)."
I will work on it. Thanks!
Yu-cheng
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Andy Lutomirski @ 2018-11-13 20:58 UTC (permalink / raw)
To: Dave Martin
Cc: Daniel Colascione, Florian Weimer, Michael Kerrisk (man-pages),
linux-kernel, Joel Fernandes, Linux API, Willy Tarreau,
Vlastimil Babka, Carlos O'Donell, libc-alpha@sourceware.org
In-Reply-To: <20181113193859.GJ3505@e103592.cambridge.arm.com>
> On Nov 13, 2018, at 11:39 AM, Dave Martin <Dave.Martin@arm.com> wrote:
>
> On Mon, Nov 12, 2018 at 05:19:14AM -0800, Daniel Colascione wrote:
>
> [...]
>
>> We can learn something from how Windows does things. On that system,
>> what we think of as "libc" is actually two parts. (More, actually, but
>> I'm simplifying.) At the lowest level, you have the semi-documented
>> ntdll.dll, which contains raw system call wrappers and arcane
>> kernel-userland glue. On top of ntdll live the "real" libc
>> (msvcrt.dll, kernel32.dll, etc.) that provide conventional
>> application-level glue. The tight integration between ntdll.dll and
>> the kernel allows Windows to do very impressive things. (For example,
>> on x86_64, Windows has no 32-bit ABI as far as the kernel is
>> concerned! You can still run 32-bit programs though, and that works
>> via ntdll.dll essentially shimming every system call and switching the
>> processor between long and compatibility mode as needed.) Normally,
>> you'd use the higher-level capabilities, but if you need something in
>> ntdll (e.g., if you're Cygwin) nothing stops your calling into the
>> lower-level system facilities directly. ntdll is tightly bound to the
>> kernel; the higher-level libc, not so.
>>
>> We should adopt a similar approach. Shipping a lower-level
>> "liblinux.so" tightly bound to the kernel would not only let the
>> kernel bypass glibc's "editorial discretion" in exposing new
>> facilities to userspace, but would also allow for tighter user-kernel
>> integration that one can achieve with a simplistic syscall(2)-style
>> escape hatch. (For example, for a long time now, I've wanted to go
>> beyond POSIX and improve the system's signal handling API, and this
>> improvement requires userspace cooperation.) The vdso is probably too
>> small and simplistic to serve in this role; I'd want a real library.
>
> Can you expand on your reasoning here?
>
> Playing devil's advocate:
>
> If the library is just exposing the syscall interface, I don't see
> why it _couldn't_ fit into the vdso (or something vdso-like).
>
> If a separate library, I'd be concerned that it would accumulate
> value-add bloat over time, and the kernel ABI may start to creep since
> most software wouldn't invoke the kernel directly any more. Even if
> it's maintained in the kernel tree, its existence as an apparently
> standalone component may encourage forking, leading to a potential
> compatibility mess.
>
> The vdso approach would mean we can guarantee that the library is
> available and up to date at runtime, and may make it easier to keep
> what's in it down to sane essentials.
Hmm. Putting on my vDSO hat:
The vDSO could provide all kinds of nifty things. Better exception handling comes to mind. But it has two major limitations that severely restrict what it can do:
- It can’t allocate memory. We probably want to keep it that way.
- It can’t use TLS. Solving this without genuinely awful ABI issues may be extremely hard. We *could* require callers to pass a thread pointer in, I suppose.
Also, if we make the vDSO stateful, CRIU is going to have a blast. We might need to expose explicit save and restore abilities.
As a straw man use case, it would be neat if DSOs (or the loader, maybe) could register a list of exception fixups per DSO. The kernel could consult these lists before delivering a signal. ISTM it wouldn’t be so crazy if the vDSO handled registration, although it could uses syscalls as well. If the vDSO did it, it would need somewhere to put the lists.
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Dave Martin @ 2018-11-13 19:39 UTC (permalink / raw)
To: Daniel Colascione
Cc: Florian Weimer, Michael Kerrisk (man-pages), linux-kernel,
Joel Fernandes, Linux API, Willy Tarreau, Vlastimil Babka,
Carlos O'Donell, libc-alpha@sourceware.org
In-Reply-To: <CAKOZuetdgk1QYhx1538-98rFpogMin=8DkPnCtU9_=ip23Vk7w@mail.gmail.com>
On Mon, Nov 12, 2018 at 05:19:14AM -0800, Daniel Colascione wrote:
[...]
> We can learn something from how Windows does things. On that system,
> what we think of as "libc" is actually two parts. (More, actually, but
> I'm simplifying.) At the lowest level, you have the semi-documented
> ntdll.dll, which contains raw system call wrappers and arcane
> kernel-userland glue. On top of ntdll live the "real" libc
> (msvcrt.dll, kernel32.dll, etc.) that provide conventional
> application-level glue. The tight integration between ntdll.dll and
> the kernel allows Windows to do very impressive things. (For example,
> on x86_64, Windows has no 32-bit ABI as far as the kernel is
> concerned! You can still run 32-bit programs though, and that works
> via ntdll.dll essentially shimming every system call and switching the
> processor between long and compatibility mode as needed.) Normally,
> you'd use the higher-level capabilities, but if you need something in
> ntdll (e.g., if you're Cygwin) nothing stops your calling into the
> lower-level system facilities directly. ntdll is tightly bound to the
> kernel; the higher-level libc, not so.
>
> We should adopt a similar approach. Shipping a lower-level
> "liblinux.so" tightly bound to the kernel would not only let the
> kernel bypass glibc's "editorial discretion" in exposing new
> facilities to userspace, but would also allow for tighter user-kernel
> integration that one can achieve with a simplistic syscall(2)-style
> escape hatch. (For example, for a long time now, I've wanted to go
> beyond POSIX and improve the system's signal handling API, and this
> improvement requires userspace cooperation.) The vdso is probably too
> small and simplistic to serve in this role; I'd want a real library.
Can you expand on your reasoning here?
Playing devil's advocate:
If the library is just exposing the syscall interface, I don't see
why it _couldn't_ fit into the vdso (or something vdso-like).
If a separate library, I'd be concerned that it would accumulate
value-add bloat over time, and the kernel ABI may start to creep since
most software wouldn't invoke the kernel directly any more. Even if
it's maintained in the kernel tree, its existence as an apparently
standalone component may encourage forking, leading to a potential
compatibility mess.
The vdso approach would mean we can guarantee that the library is
available and up to date at runtime, and may make it easier to keep
what's in it down to sane essentials.
Cheers
---Dave
^ permalink raw reply
* Re: [PATCH v5 05/27] Documentation/x86: Add CET description
From: Borislav Petkov @ 2018-11-13 18:43 UTC (permalink / raw)
To: Yu-cheng Yu
Cc: x86, H. Peter Anvin, Thomas Gleixner, Ingo Molnar, linux-kernel,
linux-doc, linux-mm, linux-arch, linux-api, Arnd Bergmann,
Andy Lutomirski, Balbir Singh, Cyrill Gorcunov, Dave Hansen,
Eugene Syromiatnikov, Florian Weimer, H.J. Lu, Jann Horn,
Jonathan Corbet, Kees Cook, Mike Kravetz, Nadav Amit,
Oleg Nesterov, Pa
In-Reply-To: <20181011151523.27101-6-yu-cheng.yu@intel.com>
On Thu, Oct 11, 2018 at 08:15:01AM -0700, Yu-cheng Yu wrote:
> Explain how CET works and the no_cet_shstk/no_cet_ibt kernel
> parameters.
>
> Signed-off-by: Yu-cheng Yu <yu-cheng.yu@intel.com>
> ---
> .../admin-guide/kernel-parameters.txt | 6 +
> Documentation/index.rst | 1 +
> Documentation/x86/index.rst | 11 +
> Documentation/x86/intel_cet.rst | 266 ++++++++++++++++++
> 4 files changed, 284 insertions(+)
> create mode 100644 Documentation/x86/index.rst
> create mode 100644 Documentation/x86/intel_cet.rst
So this patch should probably come first in the series so that a reader
can know what to expect...
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 92eb1f42240d..3854423f7c86 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -2764,6 +2764,12 @@
> noexec=on: enable non-executable mappings (default)
> noexec=off: disable non-executable mappings
>
> + no_cet_ibt [X86-64] Disable indirect branch tracking for user-mode
> + applications
> +
> + no_cet_shstk [X86-64] Disable shadow stack support for user-mode
> + applications
> +
> nosmap [X86]
> Disable SMAP (Supervisor Mode Access Prevention)
> even if it is supported by processor.
> diff --git a/Documentation/index.rst b/Documentation/index.rst
> index 5db7e87c7cb1..1cdc139adb40 100644
> --- a/Documentation/index.rst
> +++ b/Documentation/index.rst
Please integrate scripts/checkpatch.pl into your patch creation
workflow. Some of the warnings/errors *actually* make sense:
WARNING: Missing or malformed SPDX-License-Identifier tag in line 1
#76: FILE: Documentation/x86/index.rst:1:
+=======================
WARNING: Missing or malformed SPDX-License-Identifier tag in line 1
#93: FILE: Documentation/x86/intel_cet.rst:1:
+=========================================
> @@ -104,6 +104,7 @@ implementation.
> :maxdepth: 2
>
> sh/index
> + x86/index
>
> Filesystem Documentation
> ------------------------
> diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
> new file mode 100644
> index 000000000000..9c34d8cbc8f0
> --- /dev/null
> +++ b/Documentation/x86/index.rst
> @@ -0,0 +1,11 @@
> +=======================
> +X86 Documentation
> +=======================
> +
> +Control Flow Enforcement
> +========================
> +
> +.. toctree::
> + :maxdepth: 1
> +
> + intel_cet
> diff --git a/Documentation/x86/intel_cet.rst b/Documentation/x86/intel_cet.rst
> new file mode 100644
> index 000000000000..946f4802a51f
> --- /dev/null
> +++ b/Documentation/x86/intel_cet.rst
> @@ -0,0 +1,266 @@
> +=========================================
> +Control Flow Enforcement Technology (CET)
> +=========================================
> +
> +[1] Overview
> +============
> +
> +Control Flow Enforcement Technology (CET) provides protection against
> +return/jump-oriented programming (ROP) attacks. It can be implemented
> +to protect both the kernel and applications. In the first phase,
> +only the user-mode protection is implemented on the 64-bit kernel.
s/the// is implemented in 64-bit mode.
> +However, 32-bit applications are supported under the compatibility
> +mode.
Drop "However":
"32-bit applications are, of course, supported in compatibility mode."
> +
> +CET includes shadow stack (SHSTK) and indirect branch tracking (IBT).
"CET introduces two a shadow stack and an indirect branch tracking mechanism."
> +The SHSTK is a secondary stack allocated from memory. The processor
s/The//
> +automatically pushes/pops a secure copy to the SHSTK every return
> +address and,
that reads funny - pls reorganize. Also, what is a "secure copy"?
You mean a copy of every return address which software cannot access?
> by comparing the secure copy to the program stack copy,
> +verifies function returns are as intended.
... have not been corrupted/modified."
> The IBT verifies all
> +indirect CALL/JMP targets are intended and marked by the compiler with
> +'ENDBR' op codes.
"opcode" - one word. And before you use "ENDBR" you need to explain it
above what it is.
/me reads further... encounters ENDBR's definition...
ah, ok, so you should say something like
"... and marked by the compiler with the ENDBR opcode (see below)."
> +
> +There are two kernel configuration options:
> +
> + INTEL_X86_SHADOW_STACK_USER, and
> + INTEL_X86_BRANCH_TRACKING_USER.
> +
> +To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or later
> +are required. To build a CET-enabled application, GLIBC v2.28 or
> +later is also required.
> +
> +There are two command-line options for disabling CET features:
> +
> + no_cet_shstk - disables SHSTK, and
> + no_cet_ibt - disables IBT.
> +
> +At run time, /proc/cpuinfo shows the availability of SHSTK and IBT.
--
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Carlos O'Donell @ 2018-11-13 15:15 UTC (permalink / raw)
To: Joseph Myers, Florian Weimer
Cc: Michael Kerrisk (man-pages), Willy Tarreau, Daniel Colascione,
linux-kernel, Joel Fernandes, Linux API, Vlastimil Babka,
libc-alpha@sourceware.org
In-Reply-To: <alpine.DEB.2.21.1811121641040.6607@digraph.polyomino.org.uk>
On 11/12/18 11:43 AM, Joseph Myers wrote:
> On Sun, 11 Nov 2018, Florian Weimer wrote:
>
>> People may have disappeared from glibc development who have objected to
>> gettid. I thought this was the case with strlcpy/strlcat, but it was
>> not.
>
> Well, I know of two main people who were objecting to the notion of adding
> bindings for all non-obsolescent syscalls, Linux-specific if not suitable
> for adding to the OS-independent GNU API, and neither seems to have posted
> in the past year.
>
>> At present, it takes one semi-active glibc contributor to block addition
>> of a system call. The process to override a sustained objection has
>> never been used successfully, and it is a lot of work to get it even
>> started.
>
> We don't have such a process. (I've suggested, e.g. in conversation with
> Carlos at the Cauldron, that we should have something involving a
> supermajority vote of the GNU maintainers for glibc in cases where we're
> unable to reach a consensus in the community as a whole.)
... and I need a good excuse to propose such a process :-)
--
Cheers,
Carlos.
^ permalink raw reply
* Re: [PATCH RESEND] UAPI: move RENAME_* definitions to separated file
From: Florian Weimer @ 2018-11-13 11:09 UTC (permalink / raw)
To: Yury Norov
Cc: Andrew Morton, Alexander Viro, Norov, Yuri,
linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org,
linux-api@vger.kernel.org, libc-alpha@sourceware.org
In-Reply-To: <20181113085102.3079-1-ynorov@caviumnetworks.com>
* Yury Norov:
> diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
> index 53a22e8e0408..dbf58bbf5bad 100644
> --- a/include/uapi/linux/fs.h
> +++ b/include/uapi/linux/fs.h
Could you move it to a dedicated <linux/rename-flags.h> header? Or add
a comment that the header is only for rename flags?
Then we can change glibc to include it if it is present and when
compiling under _GNU_SOURCE.
Thanks,
Florian
^ permalink raw reply
* [PATCH RESEND] UAPI: move RENAME_* definitions to separated file
From: Yury Norov @ 2018-11-13 8:51 UTC (permalink / raw)
To: Andrew Morton, Alexander Viro, Florian Weimer
Cc: Norov, Yuri, linux-kernel@vger.kernel.org,
linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org,
libc-alpha@sourceware.org
Discussion:
https://lore.kernel.org/lkml/20180702084622.GA15274@yury-thinkpad/
Although RENAME_* macros are exposed in kernel headers, they are not
used by glibc. That's because linux/fs.h which hosts RENAME_* is
considered unsuitable by glibc developers:
As Florian Weimer wrote:
> <linux/fs.h> undefines and defines macros not mentioned in the standards
> (and it even contains a few unrelated structs), so we cannot include it
> without _GNU_SOURCE.
>
> It might be possible to include it only for _GNU_SOURCE, but there are a
> lot of things in <linux/fs.h>, so that does not seem to be particularly
> advisable.
In this patch RENAME_* is removed to separated header file that
conforms standards and therefore is suitable for inclusion by glibc.
Signed-off-by: Yury Norov <ynorov@caviumnetworks.com>
---
include/uapi/linux/fs.h | 4 +---
include/uapi/linux/rename.h | 12 ++++++++++++
2 files changed, 13 insertions(+), 3 deletions(-)
create mode 100644 include/uapi/linux/rename.h
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index 53a22e8e0408..dbf58bbf5bad 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -44,9 +44,7 @@
#define SEEK_HOLE 4 /* seek to the next hole */
#define SEEK_MAX SEEK_HOLE
-#define RENAME_NOREPLACE (1 << 0) /* Don't overwrite target */
-#define RENAME_EXCHANGE (1 << 1) /* Exchange source and dest */
-#define RENAME_WHITEOUT (1 << 2) /* Whiteout source */
+#include <linux/rename.h>
struct file_clone_range {
__s64 src_fd;
diff --git a/include/uapi/linux/rename.h b/include/uapi/linux/rename.h
new file mode 100644
index 000000000000..7178f0565657
--- /dev/null
+++ b/include/uapi/linux/rename.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _UAPI_LINUX_RENAME_H
+#define _UAPI_LINUX_RENAME_H
+
+/*
+ * Definitions for rename syscall family.
+ */
+#define RENAME_NOREPLACE (1 << 0) /* Don't overwrite target */
+#define RENAME_EXCHANGE (1 << 1) /* Exchange source and dest */
+#define RENAME_WHITEOUT (1 << 2) /* Whiteout source */
+
+#endif /* _UAPI_LINUX_RENAME_H */
--
2.17.1
^ permalink raw reply related
* Re: Official Linux system wrapper library?
From: Joseph Myers @ 2018-11-12 23:26 UTC (permalink / raw)
To: Daniel Colascione
Cc: Florian Weimer, Zack Weinberg, Michael Kerrisk (man-pages),
Linux Kernel Mailing List, Joel Fernandes, Linux API,
Willy Tarreau, Vlastimil Babka, Carlos O'Donell,
GNU C Library
In-Reply-To: <CAKOZueuM=3pxuXaJU4v571daWFCTJ1LYO-ApD6MrD7313RNYfQ@mail.gmail.com>
On Mon, 12 Nov 2018, Daniel Colascione wrote:
> I initially wanted to put the APIs in libc. I still do. But that's
> proving to be impractical, for the reasons we're discussing on this
> thread.
Well, your proposed APIs didn't attract consensus among libc developers.
> > (I can imagine *other* parts of the toolchain being involved, if e.g. you
> > want to have a good way of checking "is the address of the instruction
> > causing this signal in this library?" that works with static as well as
> > dynamic linking - for dynamic linking, I expect something could be done
> > using libc_nonshared and __dso_handle to identify code in the library
> > calling some registering function. And indeed there might also be new
> > kernel interfaces that help improve signal handling.)
>
> Again: you're blocking a practical solution for the sake of some
> elegant theoretical implementation that will never arrive, and so the
I'm not - I'm observing various areas that might be open to improvements
related to signal handling, not saying improvements in one area are a
prerequisite to improvements in another. I'm exploring the problem and
solution space, and collectively exploring the problem and solution space
is an important part of trying to work out where there might be useful
future improvements related to the general issue of signal handling.
Exploring the problem and solution space can include coming to the
conclusion that an idea that seems obvious is in fact a bad idea, or in
fact orthogonal to other ideas that are independently useful - those
things are still useful in yielding a better rationale for taking a given
approach.
> > In the absence of consensus for adding such a new API for signals to
> > glibc, it's unlikely one would get consensus for glibc to depend on some
> > other library providing such an API either.
>
> glibc would continue using an unsupported legacy system call
> interfaces in lieu of a supported low-level interface library?
The Linux kernel supports the interfaces that people actually use, on the
principle of not breaking userspace, not the interfaces that someone would
like to declare to be the supported ones. We'd use the interfaces that
seem suitable for use by glibc, and direct syscalls seem more suitable to
me than any kernel-provided userspace library.
Naturally a library invented in the kernel on the basis of not liking what
libc people are doing or not doing is unlikely to be suitable for use by
libc (and use together with libc of anything in it that interferes with
libc functionality such as sigaction might be explicitly discouraged by
libc maintainers, just as e.g. direct use of clone can be discouraged) -
whereas interfaces developed collaboratively with libc implementations and
getting consensus from those users are more likely to be of use to libc
implementations.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Daniel Colascione @ 2018-11-12 23:10 UTC (permalink / raw)
To: Joseph Myers
Cc: Florian Weimer, Zack Weinberg, Michael Kerrisk (man-pages),
Linux Kernel Mailing List, Joel Fernandes, Linux API,
Willy Tarreau, Vlastimil Babka, Carlos O'Donell,
GNU C Library
In-Reply-To: <alpine.DEB.2.21.1811122236240.18130@digraph.polyomino.org.uk>
On Mon, Nov 12, 2018 at 2:51 PM, Joseph Myers <joseph@codesourcery.com> wrote:
> I see no obvious reason why a kernel-provided
> library, with all the problems that entails, should need to be involved,
> rather than putting new APIs either in libc
I initially wanted to put the APIs in libc. I still do. But that's
proving to be impractical, for the reasons we're discussing on this
thread.
> or in a completely separate
> libsignal
A separate library can't prevent some use of sigaction elsewhere in
the program stomping on its handler. One of the key aspects of the
registered-handler design is that registered handlers get to run
*before* the legacy process-wide handler. The only non-hacky way to do
that is to put the signal handler registration logic in the same logic
component that houses the legacy signal registration machinery.
> (I can imagine *other* parts of the toolchain being involved, if e.g. you
> want to have a good way of checking "is the address of the instruction
> causing this signal in this library?" that works with static as well as
> dynamic linking - for dynamic linking, I expect something could be done
> using libc_nonshared and __dso_handle to identify code in the library
> calling some registering function. And indeed there might also be new
> kernel interfaces that help improve signal handling.)
Again: you're blocking a practical solution for the sake of some
elegant theoretical implementation that will never arrive, and so the
world remains in a poor state indefinitely. Incremental improvement is
good. Nothing about the registered signal handler approach precludes
this sort of enhancement in the future. The same goes for the system
call metadata database you've described: nice-to-have; shouldn't block
simpler and more immediately practical work.
> In the absence of consensus for adding such a new API for signals to
> glibc, it's unlikely one would get consensus for glibc to depend on some
> other library providing such an API either.
glibc would continue using an unsupported legacy system call
interfaces in lieu of a supported low-level interface library?
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Joseph Myers @ 2018-11-12 22:51 UTC (permalink / raw)
To: Daniel Colascione
Cc: Florian Weimer, Zack Weinberg, Michael Kerrisk (man-pages),
Linux Kernel Mailing List, Joel Fernandes, Linux API,
Willy Tarreau, Vlastimil Babka, Carlos O'Donell,
GNU C Library
In-Reply-To: <CAKOZues2bQo1y_1ynxFMHGTvtTjABsqVpKJt5MYMdZBq6-ikHA@mail.gmail.com>
On Mon, 12 Nov 2018, Daniel Colascione wrote:
> The two features *are* unrelated. The design I've proposed works
> equally well for synchronous and asynchronous signals, and limiting it
Whatever the design, I see no obvious reason why a kernel-provided
library, with all the problems that entails, should need to be involved,
rather than putting new APIs either in libc or in a completely separate
libsignal for libraries wanting to use such a system for cooperative
signal use.
(I can imagine *other* parts of the toolchain being involved, if e.g. you
want to have a good way of checking "is the address of the instruction
causing this signal in this library?" that works with static as well as
dynamic linking - for dynamic linking, I expect something could be done
using libc_nonshared and __dso_handle to identify code in the library
calling some registering function. And indeed there might also be new
kernel interfaces that help improve signal handling.)
In the absence of consensus for adding such a new API for signals to
glibc, it's unlikely one would get consensus for glibc to depend on some
other library providing such an API either. But you can always write a
library (which I think would most naturally be a completely separate
libsignal, not part of the kernel source tree) and seek to persuade
libraries they should be using it rather than interfering with global
state by registering normal signal handlers directly.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Joseph Myers @ 2018-11-12 22:34 UTC (permalink / raw)
To: Florian Weimer
Cc: Daniel Colascione, Zack Weinberg, Michael Kerrisk (man-pages),
Linux Kernel Mailing List, Joel Fernandes, Linux API,
Willy Tarreau, Vlastimil Babka, Carlos O'Donell,
GNU C Library
In-Reply-To: <87lg5ym7qi.fsf@oldenburg.str.redhat.com>
On Mon, 12 Nov 2018, Florian Weimer wrote:
> For that use case, a machine-readable system call ABI specification is
> the only reasonable approach: Some people want inline system calls,
I also think it's much more likely to be of use to glibc than any syscall
library provided by the kernel. I don't think a syscall library provided
by the kernel is likely to be of use for implementing glibc functions, but
some kind of textual ABI specification might at least be of use for
checking that syscall macro calls / syscalls.list entries are consistent
with what the kernel thinks its ABI is. (Hopefully there would be
automated tests on the kernel side as well of some kind of consistency
between the ABI specification and the kernel.)
strace is indeed a more obvious potential consumer of such a description
of syscall ABIs.
I'd think a syscall library would more likely be something a few
applications would use if they want to access a syscall that for whatever
reason glibc doesn't have a wrapper for yet - not something useful for
glibc itself to call or link against.
> and for application programmer's sanity, make sure that the kernel adds
> generic system calls in a single version, across all architectures.
That would be strongly desirable for glibc as well - a way of ensuring
that the kernel rapidly fails CI tests and does not get released if new
syscalls are only present on some architectures (including e.g. being
missing from some compat syscall tables, or defined in asm/unistd.h but
not in the actual syscall table, or vice versa - or some way of making
sure such inconsistencies cannot occur by eliminating duplicate copies of
the syscall list information in the sources).
When we have compatibility code in glibc for the absence of some syscall,
we can only eliminate that code when the oldest kernel version supported
by glibc is new enough to have the syscall on whichever glibc architecture
was slowest to introduce the syscall in the kernel - and that can often be
years after the first architectures gained support for that syscall in the
kernel.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Greg KH @ 2018-11-12 20:03 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Carlos O'Donell, Daniel Colascione, linux-kernel,
Joel Fernandes, Linux API
In-Reply-To: <20181112090828.2573987d@lwn.net>
On Mon, Nov 12, 2018 at 09:08:28AM -0700, Jonathan Corbet wrote:
> On Sun, 11 Nov 2018 18:36:30 -0800
> Greg KH <gregkh@linuxfoundation.org> wrote:
>
> > We should have a checklist. That's a great idea. Now to find someone
> > to write it... :)
>
> Do we think the LPC session might have the right people to create such a
> thing? If so, I can try to put together a coherent presentation of the
> result.
I do not know who will be there, sorry...
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Daniel Colascione @ 2018-11-12 19:26 UTC (permalink / raw)
To: Florian Weimer
Cc: Zack Weinberg, Michael Kerrisk (man-pages),
Linux Kernel Mailing List, Joel Fernandes, Linux API,
Willy Tarreau, Vlastimil Babka, Carlos O'Donell,
GNU C Library
In-Reply-To: <87lg5ym7qi.fsf@oldenburg.str.redhat.com>
On Mon, Nov 12, 2018 at 11:11 AM, Florian Weimer <fweimer@redhat.com> wrote:
> * Daniel Colascione:
>
>> What about off_t differences? Again, it doesn't matter. From the
>> *kernel's* point of view, there's one width of offset parameter per
>> system call per architecture. The library I'm proposing would expose
>> this parameter literally.
>
> Does this mean the application author needs to know when to split an
> off_t argument into two, and when to pass it as a single argument, and
> when to insert dummy arguments for alignment, depending on the
> architecture?
No, I wouldn't make callers go to that trouble. I don't see any
barrier to common-sense local data transformations. These
transformations don't have external dependencies, after all. I want a
thin interface to the kernel, but not so thin as to be a direct
mapping onto register locations. I don't see value in that level of
correspondence.
>>> And that means you wouldn't get as much
>>> decoupling from the C and POSIX standards -- both of which specify at
>>> least part of those semantics -- as you want, and we would still be
>>> having these arguments. For example, it would be every bit as
>>> troublesome for liblinuxabi.so.1 to export set_robust_list as it would
>>> be for libc.so.6 to do that.
>>
>> Why? Such an exported function would cause no trouble until called,
>> and there are legitimate reasons for calling such a function. Not
>> everyone, as mentioned, wants to write a program that relies on libc.
>
> For that use case, a machine-readable system call ABI specification is
> the only reasonable approach:
> The challenge here is to come up with a
> uniform description of the system call interface for all architectures,
This is another example in which we should remember the old aphorism
that the perfect is the enemy of the good. There's no reason that the
kernel couldn't simply provide a library with conventional functions
exported in the conventional way doing the conventional things that
functions do, one that would free users from relying on direct use of
syscall(2). If this library were to interact with errno and
cancelation properly, so much the better. There's no reason to avoid
this work in favor of some theoretically-elegant
abstract-function-description metadata-based approach that will likely
never materialize.
(Alternatively: just regard C as the uniform description language.)
>> This stance in the paragraph I've quoted is another example of glibc's
>> misplaced idealism. As I've elaborated elsewhere, people use signals
>> for many purposes today. The current signals API is extremely
>> difficult to use correctly in a process in which multiple unrelated
>> components want to take advantage of signal-handling functionality.
>> Users deserve a cleaner, modern, and safe API. It's not productive
>> withhold improvements to the signal API and gate them on unrelated
>> features like process handles merely because, in the personal
>> judgement of the glibc maintainers, developers should use signals for
>> fewer things.
>
> The two aren't unrelated. If you take asynchronous signals out of the
> picture, the design becomes simpler and easier to use.
The two features *are* unrelated. The design I've proposed works
equally well for synchronous and asynchronous signals, and limiting it
to synchronous signals doesn't simplify it. Even if it were the case
that the design were simpler and easier to use when limited to
synchronous signals --- which it isn't, unless you want to go in the
SEH direction, which is more, not less complicated --- that wouldn't
be a reason to block the work until some form of process handle
landed. The objections I've seen have all essentially amounted to "we
don't think people should use signals".
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Florian Weimer @ 2018-11-12 19:11 UTC (permalink / raw)
To: Daniel Colascione
Cc: Zack Weinberg, Michael Kerrisk (man-pages),
Linux Kernel Mailing List, Joel Fernandes, Linux API,
Willy Tarreau, Vlastimil Babka, Carlos O'Donell,
GNU C Library
In-Reply-To: <CAKOZuev7zqq+xpjyDA2mSdy-zwyNjECCzLsBELF6_v1rwar_mA@mail.gmail.com>
* Daniel Colascione:
> What about off_t differences? Again, it doesn't matter. From the
> *kernel's* point of view, there's one width of offset parameter per
> system call per architecture. The library I'm proposing would expose
> this parameter literally.
Does this mean the application author needs to know when to split an
off_t argument into two, and when to pass it as a single argument, and
when to insert dummy arguments for alignment, depending on the
architecture?
>> And that means you wouldn't get as much
>> decoupling from the C and POSIX standards -- both of which specify at
>> least part of those semantics -- as you want, and we would still be
>> having these arguments. For example, it would be every bit as
>> troublesome for liblinuxabi.so.1 to export set_robust_list as it would
>> be for libc.so.6 to do that.
>
> Why? Such an exported function would cause no trouble until called,
> and there are legitimate reasons for calling such a function. Not
> everyone, as mentioned, wants to write a program that relies on libc.
For that use case, a machine-readable system call ABI specification is
the only reasonable approach: Some people want inline system calls,
others want dedicated routines per system call. The calling convention
for the dedicated functions will vary, and the way errors are handled as
well. Some want connect calls to be handled by socketcall if possible,
others prefer the direct call.
The nice thing here is that once you settled for a particular approach,
the functions are really small and will not change, so there is no real
need for dynamic linking. The challenge here is to come up with a
uniform description of the system call interface for all architectures,
and for application programmer's sanity, make sure that the kernel adds
generic system calls in a single version, across all architectures.
> This stance in the paragraph I've quoted is another example of glibc's
> misplaced idealism. As I've elaborated elsewhere, people use signals
> for many purposes today. The current signals API is extremely
> difficult to use correctly in a process in which multiple unrelated
> components want to take advantage of signal-handling functionality.
> Users deserve a cleaner, modern, and safe API. It's not productive
> withhold improvements to the signal API and gate them on unrelated
> features like process handles merely because, in the personal
> judgement of the glibc maintainers, developers should use signals for
> fewer things.
The two aren't unrelated. If you take asynchronous signals out of the
picture, the design becomes simpler and easier to use.
Thanks,
Florian
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Daniel Colascione @ 2018-11-12 18:28 UTC (permalink / raw)
To: Zack Weinberg
Cc: Florian Weimer, Michael Kerrisk (man-pages),
Linux Kernel Mailing List, Joel Fernandes, Linux API,
Willy Tarreau, Vlastimil Babka, Carlos O'Donell,
GNU C Library
In-Reply-To: <CAKCAbMiHC9r54h=XeW7CkBZ1Z5eHr9MPH3Rn7KTc9DjoHG=8UA@mail.gmail.com>
On Mon, Nov 12, 2018 at 9:24 AM, Zack Weinberg <zackw@panix.com> wrote:
> Daniel Colascione <dancol@google.com> wrote:
>> >> If the kernel provides a system call, libc should provide a C wrapper
>> >> for it, even if in the opinion of the libc maintainers, that system
>> >> call is flawed.
>
> I would like to state general support for this principle; in fact, I
> seriously considered preparing patches that made exactly this change,
> about a year ago, posting them, and calling for objections. Then
> $dayjob ate all my hacking time (and is still doing so, alas).
>
> Nonetheless I do think there are exceptions, such as those that are
> completely obsolete (bdflush, socketcall) and those that cannot be
> used without stomping on glibc's own data structures (set_robust_list
> is the only one of these I know about off the top of my head, but
> there may well be others).
If people want to stomp over glibc's data structures, let them. Maybe
a particular program, for whatever reason, wants to avoid glibc
mutexes entirely and do its own synchronization. It should be possible
to cleanly separate the users on a per-thread basis.
Besides, adhering to the principle that all system functionality is
provided is worth it even if (in the case of bdflush) there's not a
compelling use right now.
Consider bdflush: in kernel debugging, hijacking "useless" system
calls and setting breakpoints on them or temporarily wiring them to
custom functionality is sometimes useful, and there's no particular
reason to *prevent* a program from calling one of these routines,
especially since there's little cost to providing a wrapper and
noticeable value in completeness itself.
> Daniel Colascione <dancol@google.com> wrote:
>> We can learn something from how Windows does things. On that system,
>> what we think of as "libc" is actually two parts. (More, actually, but
>> I'm simplifying.) At the lowest level, you have the semi-documented
>> ntdll.dll, which contains raw system call wrappers and arcane
>> kernel-userland glue. On top of ntdll live the "real" libc
>> (msvcrt.dll, kernel32.dll, etc.) that provide conventional
>> application-level glue.
>
> This is an appealing idea at first sight; there are several other
> constituencies for it besides frustrated kernel hackers, such as
> alternative system programming languages (Rust, Go) that want to
> minimize dependencies on legacy "C library" functionality. If we
> could find a clean way to do it, I would support it.
>
> The trouble is that "raw system call wrappers and arcane
> kernel-userland glue" turns out to be a lot more code, with a lot more
> tentacles in both directions, than you might think. If you compare
> the sizes of the text sections of `ntdll.dll` and `libc.so.6` you will
> notice that the former is _bigger_. The reason for this, as far as I
> can determine (without any access to Microsoft's internal
> documentation or source code ;-) is that ntdll.dll contains the
> dynamic linker-equivalent, a basic memory allocator, the stack
> unwinder, and a good chunk of the core thread library. (It also has
> stuff in it that's needed by programs that run early during boot and
> can't use kernel32.dll, but that's not our problem.) I don't think
> this is an accident or an engineering compromise. It is necessary for
> the dynamic loader to understand threads, and the thread library to
> understand shared library semantics.
Sure, but I'm not proposing talking about including threads or dynamic
library loading in the minimal kernel glue library we're discussing.
That ntdll includes this functionality (and a thread pool, and various
other gunk) works for Windows, but it's not a necessary consequence of
our adopting a layering model that the lowest of *our* layers include
what the lowest layer on Windows includes. As I mentioned above,
there's room for a "minimal" kernel interface library that actually
touches relatively little of glibc's concerns.
> A hypothetical equivalent liblinuxabi.so.1 would
> have to do the same.
It depends on what you put into the library. Basic system call
wrappers and potential future userspace glue. The ABI I'm proposing
doesn't have to look like POSIX --- for example, it can indicate error
returns via a separate out parameter. (This approach is cleaner
anyway.) As for pthread cancelation? All there's required is to mark a
range of PC values as "after cancel check, before syscall
instruction". The Linux ABI library could export a function that libc
could use, passing in a program counter value, to determine whether PC
(extracted from ucontext_t in a signal handler) were immediately
before a cancellation check.
What about off_t differences? Again, it doesn't matter. From the
*kernel's* point of view, there's one width of offset parameter per
system call per architecture. The library I'm proposing would expose
this parameter literally. If a higher-level libc wants to use a
preprocessor switch to conditionally support different offset widths,
that's fine, but there's no reason that a more literal kernel
interface library would have to do that.
> And that means you wouldn't get as much
> decoupling from the C and POSIX standards -- both of which specify at
> least part of those semantics -- as you want, and we would still be
> having these arguments. For example, it would be every bit as
> troublesome for liblinuxabi.so.1 to export set_robust_list as it would
> be for libc.so.6 to do that.
Why? Such an exported function would cause no trouble until called,
and there are legitimate reasons for calling such a function. Not
everyone, as mentioned, wants to write a program that relies on libc.
> You might be able to get out of most of the tangle by putting the
> dynamic loader in a separate process
I don't think that's a workable approach. The creation of a separate
process is a very observable side effect, and it seems unexpected that
something as simple as cat(1) would have this side effect. If
anything, parts of the dynamic linker should move into the *kernel* to
support things like applying relocations to clean pages, but that's a
separate discussion.
> and that's _also_ an appealing
> idea for several other reasons, but it would still need to understand
> some of the thread-related data structures within the processes it
> manipulated, so I don't think it would help enough to be worth it (in
> a complete greenfields design where I get to ignore POSIX and rewrite
> the kernel API from scratch, now, that might be a different story).
>
> On a larger note, the fundamental complaint here is a project process
> / communication complaint. We haven't been communicating enough with
> the kernel team, fair criticism. We can do better. But the
> communication has to go both ways. When, for instance, we tell you
> that membarrier needs to have its semantics nailed down in terms of
> the C++17 memory model, that actually needs to happen
I think you can think of membarrier as upgrading signal fences to thread fences.
> And, because this is a process / communication problem, you cannot
> expect there to be a purely technical fix. Your position appears,
> from where I'm sitting, to be something like "if we split glibc into
> two pieces, then you and us will never have to talk to each other
> again" which, I'm sorry, I can't see that working out in the long run.
>
>> (For example, for a long time now, I've wanted to go
>> beyond POSIX and improve the system's signal handling API, and this
>> improvement requires userspace cooperation.)
>
> This is also an appealing notion, but the first step should be to
> eliminate all of the remaining uses for asynchronous signals: for
> instance, give us process handles already! Once a program only ever
> needs to call sigaction() to deal with
> SIGSEGV/SIGBUS/SIGILL/SIGFPE/SIGTRAP, then we can think about
> inventing a better replacement for that scenario.
I too want process handles. (See my other patches.) But that's besides
the point.
This stance in the paragraph I've quoted is another example of glibc's
misplaced idealism. As I've elaborated elsewhere, people use signals
for many purposes today. The current signals API is extremely
difficult to use correctly in a process in which multiple unrelated
components want to take advantage of signal-handling functionality.
Users deserve a cleaner, modern, and safe API. It's not productive
withhold improvements to the signal API and gate them on unrelated
features like process handles merely because, in the personal
judgement of the glibc maintainers, developers should use signals for
fewer things. This attitude is an unwarranted imposition on the entire
ecosystem. It should be possible to innovate in this area without
these blockers, one way or another.
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Randy Dunlap @ 2018-11-12 18:14 UTC (permalink / raw)
To: Joseph Myers, Greg KH
Cc: Willy Tarreau, Florian Weimer, Michael Kerrisk (man-pages),
Daniel Colascione, linux-kernel, Joel Fernandes, Linux API,
Vlastimil Babka, Carlos O'Donell, libc-alpha@sourceware.org
In-Reply-To: <alpine.DEB.2.21.1811121803520.6607@digraph.polyomino.org.uk>
On 11/12/18 10:09 AM, Joseph Myers wrote:
> On Mon, 12 Nov 2018, Greg KH wrote:
>
>> If there are still problems with this, please let us know and we will be
>> glad to resolve them.
>
> With headers installed from Linus's latest tree, I retried (for x86_64)
> the case of a source file containing the single line
>
> #include <linux/elfcore.h>
>
> which (as previously discussed, and Arnd had an RFC patch) I want to use
> in a glibc test of header consistency. It gives errors "unknown type name
> 'elf_greg_t'" etc. (for lots more types as well) - but even before getting
> onto those errors, there's
>
> asm/signal.h:127:2: error: unknown type name 'size_t'
>
> from a header included from linux/elfcore.h. So this doesn't seem to be
> working as I'd expect yet.
Yes, someone from Google (iirc) and also David Howells had some tests
that would point out all of the problems. I thought (expected) more follow-up
from them with patches...
--
~Randy
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Joseph Myers @ 2018-11-12 18:09 UTC (permalink / raw)
To: Greg KH
Cc: Willy Tarreau, Florian Weimer, Michael Kerrisk (man-pages),
Daniel Colascione, linux-kernel, Joel Fernandes, Linux API,
Vlastimil Babka, Carlos O'Donell, libc-alpha@sourceware.org
In-Reply-To: <20181112175305.GB15484@kroah.com>
On Mon, 12 Nov 2018, Greg KH wrote:
> If there are still problems with this, please let us know and we will be
> glad to resolve them.
With headers installed from Linus's latest tree, I retried (for x86_64)
the case of a source file containing the single line
#include <linux/elfcore.h>
which (as previously discussed, and Arnd had an RFC patch) I want to use
in a glibc test of header consistency. It gives errors "unknown type name
'elf_greg_t'" etc. (for lots more types as well) - but even before getting
onto those errors, there's
asm/signal.h:127:2: error: unknown type name 'size_t'
from a header included from linux/elfcore.h. So this doesn't seem to be
working as I'd expect yet.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Greg KH @ 2018-11-12 17:53 UTC (permalink / raw)
To: Joseph Myers
Cc: Willy Tarreau, Florian Weimer, Michael Kerrisk (man-pages),
Daniel Colascione, linux-kernel, Joel Fernandes, Linux API,
Vlastimil Babka, Carlos O'Donell, libc-alpha@sourceware.org
In-Reply-To: <alpine.DEB.2.21.1811121723210.6607@digraph.polyomino.org.uk>
On Mon, Nov 12, 2018 at 05:36:11PM +0000, Joseph Myers wrote:
> What *is*, in my view, a bug in the uapi headers is that some of them
> don't work when included on their own. I'd expect #include
> <linux/whatever.h> or #include <asm/whatever.h>, for any such header
> installed by make headers_install, to compile on its own in userspace
> without needing any other headers to be included first, unless some header
> is specifically defined as being an internal part of another header which
> is the one that should be included.
Yes, that is a bug, and people have been working on fixing that. We now
have a new build target:
make headers_check
to keep this all working properly.
Right now on Linus's latest tree I only see one failure when running
this:
./usr/include/linux/v4l2-controls.h:1105: found __[us]{8,16,32,64} type without #include <linux/types.h>
so we are getting better.
If there are still problems with this, please let us know and we will be
glad to resolve them.
thanks,
greg k-h
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Joseph Myers @ 2018-11-12 17:36 UTC (permalink / raw)
To: Willy Tarreau
Cc: Florian Weimer, Michael Kerrisk (man-pages), Daniel Colascione,
linux-kernel, Joel Fernandes, Linux API, Vlastimil Babka,
Carlos O'Donell, libc-alpha@sourceware.org
In-Reply-To: <20181111120910.GA4425@1wt.eu>
On Sun, 11 Nov 2018, Willy Tarreau wrote:
> > The kernel developers care not, and the result is that we
> > copy definitions and declarations from the kernel header files, creating
> > additional problems.
>
> Probably that these standard compatibility issues should be addressed at
> their root in the kernel header definitions in fact. Working around issues
> always leads to a stall at some point, and it encourages the process not
> to change.
But it's not a bug in the Linux kernel header files. The set of feature
test macros supported is libc-specific and libc-version-specific. The
internal macros defined as a result of the feature test macros, that
determine what features to expose, are also libc-specific and
libc-version-specific. (The __USE_* macros in glibc are not a stable API.
For example, we might move to using __GLIBC_USE for more features in place
of the defined/undefined __USE_* internal macros.)
If a feature is Linux-specific, and the userspace header for it is also
Linux-specific (as opposed to constants in standard headers such as
sys/mman.h, where you get all the namespace issues), that userspace header
*can* include uapi headers in many cases to get constants and structures -
if those uapi headers actually work in userspace without defining things
conflicting with libc types. E.g. <sys/fanotify.h> includes
<linux/fanotify.h>.
What *is*, in my view, a bug in the uapi headers is that some of them
don't work when included on their own. I'd expect #include
<linux/whatever.h> or #include <asm/whatever.h>, for any such header
installed by make headers_install, to compile on its own in userspace
without needing any other headers to be included first, unless some header
is specifically defined as being an internal part of another header which
is the one that should be included.
In glibc we have scripts/check-installed-headers.sh which verifies that
installed headers work when included like that in various language
standard and feature test macro modes - and with my bots running
build-many-glibcs.py, this property is effectively verified every few
hours for (currently) 79 different glibc configurations covering all
supported glibc ABIs. If the uapi headers are fixed to work on their own,
there should be similar continuous integration to make sure that this
continues to be the case in future.
Simply having uapi headers that reliably work when included on their own
would help with adding further test automation in glibc to verify
consistency of constant and structure definitions between glibc and uapi
headers. We have a few such checks (e.g. for signal numbers), but now
that we require Python 3 to build glibc I hope to convert those into more
general infrastructure for extracting information from headers and running
checks on the extracted information.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Zack Weinberg @ 2018-11-12 17:24 UTC (permalink / raw)
To: dancol
Cc: Florian Weimer, Michael Kerrisk (man-pages),
Linux Kernel Mailing List, joelaf, linux-api, w, vbabka,
Carlos O'Donell, GNU C Library
In-Reply-To: <CAKOZuetdgk1QYhx1538-98rFpogMin=8DkPnCtU9_=ip23Vk7w@mail.gmail.com>
Daniel Colascione <dancol@google.com> wrote:
> >> If the kernel provides a system call, libc should provide a C wrapper
> >> for it, even if in the opinion of the libc maintainers, that system
> >> call is flawed.
I would like to state general support for this principle; in fact, I
seriously considered preparing patches that made exactly this change,
about a year ago, posting them, and calling for objections. Then
$dayjob ate all my hacking time (and is still doing so, alas).
Nonetheless I do think there are exceptions, such as those that are
completely obsolete (bdflush, socketcall) and those that cannot be
used without stomping on glibc's own data structures (set_robust_list
is the only one of these I know about off the top of my head, but
there may well be others).
Daniel Colascione <dancol@google.com> wrote:
> We can learn something from how Windows does things. On that system,
> what we think of as "libc" is actually two parts. (More, actually, but
> I'm simplifying.) At the lowest level, you have the semi-documented
> ntdll.dll, which contains raw system call wrappers and arcane
> kernel-userland glue. On top of ntdll live the "real" libc
> (msvcrt.dll, kernel32.dll, etc.) that provide conventional
> application-level glue.
This is an appealing idea at first sight; there are several other
constituencies for it besides frustrated kernel hackers, such as
alternative system programming languages (Rust, Go) that want to
minimize dependencies on legacy "C library" functionality. If we
could find a clean way to do it, I would support it.
The trouble is that "raw system call wrappers and arcane
kernel-userland glue" turns out to be a lot more code, with a lot more
tentacles in both directions, than you might think. If you compare
the sizes of the text sections of `ntdll.dll` and `libc.so.6` you will
notice that the former is _bigger_. The reason for this, as far as I
can determine (without any access to Microsoft's internal
documentation or source code ;-) is that ntdll.dll contains the
dynamic linker-equivalent, a basic memory allocator, the stack
unwinder, and a good chunk of the core thread library. (It also has
stuff in it that's needed by programs that run early during boot and
can't use kernel32.dll, but that's not our problem.) I don't think
this is an accident or an engineering compromise. It is necessary for
the dynamic loader to understand threads, and the thread library to
understand shared library semantics. It is necessary for both of
those components to allocate memory. And both of those components are
naturally tightly coupled to the kernel, and in particular they have
to be up and running from the first user-space instruction executed in
a new process, so it's natural to put them in the component that is
responsible for talking directly to the kernel.
But the _consequence_ of this design is, ntdll.dll defines the
semantics of shared library loading, and the semantics of threads, for
the entire system. A hypothetical equivalent liblinuxabi.so.1 would
have to do the same. And that means you wouldn't get as much
decoupling from the C and POSIX standards -- both of which specify at
least part of those semantics -- as you want, and we would still be
having these arguments. For example, it would be every bit as
troublesome for liblinuxabi.so.1 to export set_robust_list as it would
be for libc.so.6 to do that.
You might be able to get out of most of the tangle by putting the
dynamic loader in a separate process, and that's _also_ an appealing
idea for several other reasons, but it would still need to understand
some of the thread-related data structures within the processes it
manipulated, so I don't think it would help enough to be worth it (in
a complete greenfields design where I get to ignore POSIX and rewrite
the kernel API from scratch, now, that might be a different story).
On a larger note, the fundamental complaint here is a project process
/ communication complaint. We haven't been communicating enough with
the kernel team, fair criticism. We can do better. But the
communication has to go both ways. When, for instance, we tell you
that membarrier needs to have its semantics nailed down in terms of
the C++17 memory model, that actually needs to happen. When we tell
you that we can't use UAPI headers directly unless you commit to
honoring all of the standard-sourced namespace constraints on
user-visible headers, that needs to end the argument unless and until
someone does commit to doing all of that work on the kernel side. (We
could discuss things we could do to make that work easier from your
end -- the __USE macros could stand to be better documented, for
instance -- but ultimately someone has to do the work.)
And, because this is a process / communication problem, you cannot
expect there to be a purely technical fix. Your position appears,
from where I'm sitting, to be something like "if we split glibc into
two pieces, then you and us will never have to talk to each other
again" which, I'm sorry, I can't see that working out in the long run.
> (For example, for a long time now, I've wanted to go
> beyond POSIX and improve the system's signal handling API, and this
> improvement requires userspace cooperation.)
This is also an appealing notion, but the first step should be to
eliminate all of the remaining uses for asynchronous signals: for
instance, give us process handles already! Once a program only ever
needs to call sigaction() to deal with
SIGSEGV/SIGBUS/SIGILL/SIGFPE/SIGTRAP, then we can think about
inventing a better replacement for that scenario.
zw
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Joseph Myers @ 2018-11-12 16:59 UTC (permalink / raw)
To: Florian Weimer
Cc: Willy Tarreau, Michael Kerrisk (man-pages), Daniel Colascione,
linux-kernel, Joel Fernandes, Linux API, Vlastimil Babka,
Carlos O'Donell, libc-alpha@sourceware.org
In-Reply-To: <87zhufvntw.fsf@oldenburg.str.redhat.com>
On Sun, 11 Nov 2018, Florian Weimer wrote:
> The kernel does not know about TCB layout, so a lot of low-level
> threading aspects are defined by userspace.
>
> The kernel does not know about POSIX cancellation. Directly calling
> system calls breaks support for that.
Indeed. Where cancellation is involved, glibc needs to know exactly what
instructions might be calling a cancellable syscall and what instructions
are before or after the syscall (see Adhemerval's patches for bug 12683).
This involves an ABI that is not just specific to a particular libc, but
specific to a particular libc version. So it's inherently unsuitable to
put cancellable syscalls in libc_nonshared.a, as well as unsuitable to put
them in any kernel-provided library.
The interface for setting errno may also be libc-specific, for any
syscalls involving setting errno.
Syscalls often involve types in their interfaces such as off_t and struct
timespec. libcs may have multiple different variants of those types; the
variants available, and the ways of selecting them, are libc-specific and
libc-version-specific. So for any syscall for which the proper userspace
interface involves one of those types, wrappers for it are inherently
specific to a particular libc and libc version. (See e.g. how preadv2 and
pwritev2 syscalls also have preadv64v2 and pwritev64v2 APIs in glibc, with
appropriate redirections hased on __USE_FILE_OFFSET64, which is in turn
based on _FILE_OFFSET_BITS.)
There are many ABI variants that are relevant to glibc but not to the
kernel. Some of these involve ABI tagging of object files to indicate
which ABI variant an object is built for (and those that don't have such
tagging ought to have it), to prevent accidental linking of objects for
different ABIs. How to build objects for different userspace ABIs is not
something the kernel should need to know anything about; it's most
naturally dealt with at the level of building compiler multilibs and libc.
glibc deliberately avoids depending at compile time on the existence of
libgcc_s.so to facilitate bootstrap builds (a stripped glibc binary built
with a C-only static-only inhibit_libc GCC that was built without glibc
should be identical to the result of a longer alternating sequence of GCC
and glibc builds). I don't think any kernel-provided library would be any
better to depend on.
What one might suggest is that when new syscalls are added, kernel
developers should at least obtain agreement on linux-api from libc people
about what the userspace interface to the syscall should be. That means
the userspace-level types (such as off_t and struct timespec), and the
choice of error handling (returning error number or setting errno), and
the name of the header declaring the function, and the name of the
function, and how the syscall relates to thread cancellation, for example
- and whatever other issues may be raised.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply
* Re: Official Linux system wrapper library?
From: Joseph Myers @ 2018-11-12 16:43 UTC (permalink / raw)
To: Florian Weimer
Cc: Michael Kerrisk (man-pages), Willy Tarreau, Daniel Colascione,
linux-kernel, Joel Fernandes, Linux API, Vlastimil Babka,
Carlos O'Donell, libc-alpha@sourceware.org
In-Reply-To: <878t1zx4gj.fsf@oldenburg.str.redhat.com>
On Sun, 11 Nov 2018, Florian Weimer wrote:
> People may have disappeared from glibc development who have objected to
> gettid. I thought this was the case with strlcpy/strlcat, but it was
> not.
Well, I know of two main people who were objecting to the notion of adding
bindings for all non-obsolescent syscalls, Linux-specific if not suitable
for adding to the OS-independent GNU API, and neither seems to have posted
in the past year.
> At present, it takes one semi-active glibc contributor to block addition
> of a system call. The process to override a sustained objection has
> never been used successfully, and it is a lot of work to get it even
> started.
We don't have such a process. (I've suggested, e.g. in conversation with
Carlos at the Cauldron, that we should have something involving a
supermajority vote of the GNU maintainers for glibc in cases where we're
unable to reach a consensus in the community as a whole.)
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox