* [PATCHv6 08/10] acpi/hmat: Register performance attributes
From: Keith Busch @ 2019-02-14 17:10 UTC (permalink / raw)
To: linux-kernel, linux-acpi, linux-mm, linux-api
Cc: Greg Kroah-Hartman, Rafael Wysocki, Dave Hansen, Dan Williams,
Keith Busch
In-Reply-To: <20190214171017.9362-1-keith.busch@intel.com>
Save the best performance access attributes and register these with the
memory's node if HMAT provides the locality table. While HMAT does make
it possible to know performance for all possible initiator-target
pairings, we export only the local pairings at this time.
Signed-off-by: Keith Busch <keith.busch@intel.com>
---
drivers/acpi/hmat/hmat.c | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/drivers/acpi/hmat/hmat.c b/drivers/acpi/hmat/hmat.c
index b29f7160c7bb..6833c4897ff4 100644
--- a/drivers/acpi/hmat/hmat.c
+++ b/drivers/acpi/hmat/hmat.c
@@ -549,12 +549,27 @@ static __init void hmat_register_target_initiators(struct memory_target *target)
}
}
+static __init void hmat_register_target_perf(struct memory_target *target)
+{
+ unsigned mem_nid = pxm_to_node(target->memory_pxm);
+
+ if (!target->hmem_attrs.read_bandwidth &&
+ !target->hmem_attrs.read_latency &&
+ !target->hmem_attrs.write_bandwidth &&
+ !target->hmem_attrs.write_latency)
+ return;
+
+ node_set_perf_attrs(mem_nid, &target->hmem_attrs, 0);
+}
+
static __init void hmat_register_targets(void)
{
struct memory_target *target;
- list_for_each_entry(target, &targets, node)
+ list_for_each_entry(target, &targets, node) {
hmat_register_target_initiators(target);
+ hmat_register_target_perf(target);
+ }
}
static __init void hmat_free_structures(void)
--
2.14.4
^ permalink raw reply related
* [PATCHv6 09/10] acpi/hmat: Register memory side cache attributes
From: Keith Busch @ 2019-02-14 17:10 UTC (permalink / raw)
To: linux-kernel, linux-acpi, linux-mm, linux-api
Cc: Greg Kroah-Hartman, Rafael Wysocki, Dave Hansen, Dan Williams,
Keith Busch
In-Reply-To: <20190214171017.9362-1-keith.busch@intel.com>
Register memory side cache attributes with the memory's node if HMAT
provides the side cache iniformation table.
Signed-off-by: Keith Busch <keith.busch@intel.com>
---
drivers/acpi/hmat/hmat.c | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/drivers/acpi/hmat/hmat.c b/drivers/acpi/hmat/hmat.c
index 6833c4897ff4..e2a15f53fe45 100644
--- a/drivers/acpi/hmat/hmat.c
+++ b/drivers/acpi/hmat/hmat.c
@@ -314,6 +314,7 @@ static __init int hmat_parse_cache(union acpi_subtable_headers *header,
const unsigned long end)
{
struct acpi_hmat_cache *cache = (void *)header;
+ struct node_cache_attrs cache_attrs;
u32 attrs;
if (cache->header.length < sizeof(*cache)) {
@@ -327,6 +328,37 @@ static __init int hmat_parse_cache(union acpi_subtable_headers *header,
cache->memory_PD, cache->cache_size, attrs,
cache->number_of_SMBIOShandles);
+ cache_attrs.size = cache->cache_size;
+ cache_attrs.level = (attrs & ACPI_HMAT_CACHE_LEVEL) >> 4;
+ cache_attrs.line_size = (attrs & ACPI_HMAT_CACHE_LINE_SIZE) >> 16;
+
+ switch ((attrs & ACPI_HMAT_CACHE_ASSOCIATIVITY) >> 8) {
+ case ACPI_HMAT_CA_DIRECT_MAPPED:
+ cache_attrs.associativity = NODE_CACHE_DIRECT_MAP;
+ break;
+ case ACPI_HMAT_CA_COMPLEX_CACHE_INDEXING:
+ cache_attrs.associativity = NODE_CACHE_INDEXED;
+ break;
+ case ACPI_HMAT_CA_NONE:
+ default:
+ cache_attrs.associativity = NODE_CACHE_OTHER;
+ break;
+ }
+
+ switch ((attrs & ACPI_HMAT_WRITE_POLICY) >> 12) {
+ case ACPI_HMAT_CP_WB:
+ cache_attrs.write_policy = NODE_CACHE_WRITE_BACK;
+ break;
+ case ACPI_HMAT_CP_WT:
+ cache_attrs.write_policy = NODE_CACHE_WRITE_THROUGH;
+ break;
+ case ACPI_HMAT_CP_NONE:
+ default:
+ cache_attrs.write_policy = NODE_CACHE_WRITE_OTHER;
+ break;
+ }
+
+ node_add_cache(pxm_to_node(cache->memory_PD), &cache_attrs);
return 0;
}
--
2.14.4
^ permalink raw reply related
* [PATCHv6 10/10] doc/mm: New documentation for memory performance
From: Keith Busch @ 2019-02-14 17:10 UTC (permalink / raw)
To: linux-kernel, linux-acpi, linux-mm, linux-api
Cc: Greg Kroah-Hartman, Rafael Wysocki, Dave Hansen, Dan Williams,
Keith Busch
In-Reply-To: <20190214171017.9362-1-keith.busch@intel.com>
Platforms may provide system memory where some physical address ranges
perform differently than others, or is side cached by the system.
Add documentation describing a high level overview of such systems and the
perforamnce and caching attributes the kernel provides for applications
wishing to query this information.
Reviewed-by: Mike Rapoport <rppt@linux.ibm.com>
Signed-off-by: Keith Busch <keith.busch@intel.com>
---
Documentation/admin-guide/mm/numaperf.rst | 164 ++++++++++++++++++++++++++++++
1 file changed, 164 insertions(+)
create mode 100644 Documentation/admin-guide/mm/numaperf.rst
diff --git a/Documentation/admin-guide/mm/numaperf.rst b/Documentation/admin-guide/mm/numaperf.rst
new file mode 100644
index 000000000000..be8a23bb075d
--- /dev/null
+++ b/Documentation/admin-guide/mm/numaperf.rst
@@ -0,0 +1,164 @@
+.. _numaperf:
+
+=============
+NUMA Locality
+=============
+
+Some platforms may have multiple types of memory attached to a compute
+node. These disparate memory ranges may share some characteristics, such
+as CPU cache coherence, but may have different performance. For example,
+different media types and buses affect bandwidth and latency.
+
+A system supports such heterogeneous memory by grouping each memory type
+under different domains, or "nodes", based on locality and performance
+characteristics. Some memory may share the same node as a CPU, and others
+are provided as memory only nodes. While memory only nodes do not provide
+CPUs, they may still be local to one or more compute nodes relative to
+other nodes. The following diagram shows one such example of two compute
+nodes with local memory and a memory only node for each of compute node:
+
+ +------------------+ +------------------+
+ | Compute Node 0 +-----+ Compute Node 1 |
+ | Local Node0 Mem | | Local Node1 Mem |
+ +--------+---------+ +--------+---------+
+ | |
+ +--------+---------+ +--------+---------+
+ | Slower Node2 Mem | | Slower Node3 Mem |
+ +------------------+ +--------+---------+
+
+A "memory initiator" is a node containing one or more devices such as
+CPUs or separate memory I/O devices that can initiate memory requests.
+A "memory target" is a node containing one or more physical address
+ranges accessible from one or more memory initiators.
+
+When multiple memory initiators exist, they may not all have the same
+performance when accessing a given memory target. Each initiator-target
+pair may be organized into different ranked access classes to represent
+this relationship. The highest performing initiator to a given target
+is considered to be one of that target's local initiators, and given
+the highest access class, 0. Any given target may have one or more
+local initiators, and any given initiator may have multiple local
+memory targets.
+
+To aid applications matching memory targets with their initiators, the
+kernel provides symlinks to each other. The following example lists the
+relationship for the access class "0" memory initiators and targets, which is
+the of nodes with the highest performing access relationship::
+
+ # symlinks -v /sys/devices/system/node/nodeX/access0/targets/
+ relative: /sys/devices/system/node/nodeX/access0/targets/nodeY -> ../../nodeY
+
+ # symlinks -v /sys/devices/system/node/nodeY/access0/initiators/
+ relative: /sys/devices/system/node/nodeY/access0/initiators/nodeX -> ../../nodeX
+
+================
+NUMA Performance
+================
+
+Applications may wish to consider which node they want their memory to
+be allocated from based on the node's performance characteristics. If
+the system provides these attributes, the kernel exports them under the
+node sysfs hierarchy by appending the attributes directory under the
+memory node's access class 0 initiators as follows::
+
+ /sys/devices/system/node/nodeY/access0/initiators/
+
+These attributes apply only when accessed from nodes that have the
+are linked under the this access's inititiators.
+
+The performance characteristics the kernel provides for the local initiators
+are exported are as follows::
+
+ # tree -P "read*|write*" /sys/devices/system/node/nodeY/access0/initiators/
+ /sys/devices/system/node/nodeY/access0/initiators/
+ |-- read_bandwidth
+ |-- read_latency
+ |-- write_bandwidth
+ `-- write_latency
+
+The bandwidth attributes are provided in MiB/second.
+
+The latency attributes are provided in nanoseconds.
+
+The values reported here correspond to the rated latency and bandwidth
+for the platform.
+
+==========
+NUMA Cache
+==========
+
+System memory may be constructed in a hierarchy of elements with various
+performance characteristics in order to provide large address space of
+slower performing memory cached by a smaller higher performing memory. The
+system physical addresses memory initiators are aware of are provided
+by the last memory level in the hierarchy. The system meanwhile uses
+higher performing memory to transparently cache access to progressively
+slower levels.
+
+The term "far memory" is used to denote the last level memory in the
+hierarchy. Each increasing cache level provides higher performing
+initiator access, and the term "near memory" represents the fastest
+cache provided by the system.
+
+This numbering is different than CPU caches where the cache level (ex:
+L1, L2, L3) uses a CPU-side view where each increased level is lower
+performing. In contrast, the memory cache level is centric to the last
+level memory, so the higher numbered cache level denotes memory nearer
+to the CPU, and further from far memory.
+
+The memory-side caches are not directly addressable by software. When
+software accesses a system address, the system will return it from the
+near memory cache if it is present. If it is not present, the system
+accesses the next level of memory until there is either a hit in that
+cache level, or it reaches far memory.
+
+An application does not need to know about caching attributes in order
+to use the system. Software may optionally query the memory cache
+attributes in order to maximize the performance out of such a setup.
+If the system provides a way for the kernel to discover this information,
+for example with ACPI HMAT (Heterogeneous Memory Attribute Table),
+the kernel will append these attributes to the NUMA node memory target.
+
+When the kernel first registers a memory cache with a node, the kernel
+will create the following directory::
+
+ /sys/devices/system/node/nodeX/memory_side_cache/
+
+If that directory is not present, the system either does not not provide
+a memory-side cache, or that information is not accessible to the kernel.
+
+The attributes for each level of cache is provided under its cache
+level index::
+
+ /sys/devices/system/node/nodeX/memory_side_cache/indexA/
+ /sys/devices/system/node/nodeX/memory_side_cache/indexB/
+ /sys/devices/system/node/nodeX/memory_side_cache/indexC/
+
+Each cache level's directory provides its attributes. For example, the
+following shows a single cache level and the attributes available for
+software to query::
+
+ # tree sys/devices/system/node/node0/memory_side_cache/
+ /sys/devices/system/node/node0/memory_side_cache/
+ |-- index1
+ | |-- associativity
+ | |-- line_size
+ | |-- size
+ | `-- write_policy
+
+The "associativity" will be 0 if it is a direct-mapped cache, and non-zero
+for any other indexed based, multi-way associativity.
+
+The "line_size" is the number of bytes accessed from the next cache
+level on a miss.
+
+The "size" is the number of bytes provided by this cache level.
+
+The "write_policy" will be 0 for write-back, and non-zero for
+write-through caching.
+
+========
+See Also
+========
+.. [1] https://www.uefi.org/sites/default/files/resources/ACPI_6_2.pdf
+ Section 5.2.27
--
2.14.4
^ permalink raw reply related
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Dave Chinner @ 2019-02-14 22:06 UTC (permalink / raw)
To: Omar Sandoval
Cc: linux-fsdevel, Al Viro, kernel-team, linux-api, linux-btrfs,
linux-ext4, linux-f2fs-devel, linux-xfs
In-Reply-To: <cover.1550136164.git.osandov@fb.com>
On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
>
> Hi,
>
> Since statx was added in 4.11, userspace has had an interface for
> reading btime (file creation time), but no way to set it. This RFC patch
> series adds support for changing btime with utimensat(). Patch 1 adds
> the VFS infrastructure, patch 2 adds the support to utimensat() with a
> new flag, and the rest of the patches add filesystem support; I excluded
> CIFS for now because I don't have a CIFS setup to test it on.
>
> Updating btime is useful for at least a couple of use cases:
>
> - Backup/restore programs (my motivation for this feature is btrfs send)
> - File servers which interoperate with operating systems that allow
> updating file creation time, including Mac OS [1] and Windows [2]
So you're adding an interface that allows users to change the create
time of files without needing any privileges?
Inode create time is forensic metadata in XFS - information we use
for sequence of event and inode lifetime analysis during examination
of broken filesystem images and systems that have been broken into.
Just because it's exposed to userspace via statx(), it doesn't mean
that it is information that users should be allowed to change. i.e.
allowing users to be able to change the create time on files makes
it completely useless for the purpose it was added to XFS for...
And allowing root to change the create time doesn't really help,
because once you've broken into a system, this makes it really easy
to cover tracks (e.g. we can't find files that were created and
unlinked during the break in window anymore) and lay false
trails....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Omar Sandoval @ 2019-02-14 23:14 UTC (permalink / raw)
To: Dave Chinner
Cc: linux-fsdevel, Al Viro, kernel-team, linux-api, linux-btrfs,
linux-ext4, linux-f2fs-devel, linux-xfs
In-Reply-To: <20190214220626.GV14116@dastard>
On Fri, Feb 15, 2019 at 09:06:26AM +1100, Dave Chinner wrote:
> On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote:
> > From: Omar Sandoval <osandov@fb.com>
> >
> > Hi,
> >
> > Since statx was added in 4.11, userspace has had an interface for
> > reading btime (file creation time), but no way to set it. This RFC patch
> > series adds support for changing btime with utimensat(). Patch 1 adds
> > the VFS infrastructure, patch 2 adds the support to utimensat() with a
> > new flag, and the rest of the patches add filesystem support; I excluded
> > CIFS for now because I don't have a CIFS setup to test it on.
> >
> > Updating btime is useful for at least a couple of use cases:
> >
> > - Backup/restore programs (my motivation for this feature is btrfs send)
> > - File servers which interoperate with operating systems that allow
> > updating file creation time, including Mac OS [1] and Windows [2]
>
> So you're adding an interface that allows users to change the create
> time of files without needing any privileges?
I think it'd be reasonable to make this a privileged operation. I didn't
for this initial submission for a couple of reasons:
1. The precedent on Mac OS and Windows is that this isn't a privileged
operation.
2. I knew there would be different opinions on this either way I went.
> Inode create time is forensic metadata in XFS - information we use
> for sequence of event and inode lifetime analysis during examination
> of broken filesystem images and systems that have been broken into.
> Just because it's exposed to userspace via statx(), it doesn't mean
> that it is information that users should be allowed to change. i.e.
> allowing users to be able to change the create time on files makes
> it completely useless for the purpose it was added to XFS for...
>
> And allowing root to change the create time doesn't really help,
> because once you've broken into a system, this makes it really easy
> to cover tracks
If the threat model is that the attacker has root, then they can
overwrite the timestamp on disk anyways, no?
> (e.g. we can't find files that were created and
> unlinked during the break in window anymore) and lay false
> trails....
Fair point, although there's still ctime during the break-in window,
which I assume you'd be looking for anyways since files modified during
the break-in window are also of interest.
I see a few options, none of which are particularly nice:
1. Filesystems like XFS could choose not to support setting btime even
if they support reading it.
2. XFS could add a second, writeable btime which is used for
statx/utimes when available (it would fit in di_pad2...).
3. We could add a btime_writable sysctl/mount option/mkfs option.
Thanks!
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Dave Chinner @ 2019-02-15 0:16 UTC (permalink / raw)
To: Omar Sandoval
Cc: linux-fsdevel, Al Viro, kernel-team, linux-api, linux-btrfs,
linux-ext4, linux-f2fs-devel, linux-xfs
In-Reply-To: <20190214231429.GE9819@vader>
On Thu, Feb 14, 2019 at 03:14:29PM -0800, Omar Sandoval wrote:
> On Fri, Feb 15, 2019 at 09:06:26AM +1100, Dave Chinner wrote:
> > On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote:
> > > From: Omar Sandoval <osandov@fb.com>
> > >
> > > Hi,
> > >
> > > Since statx was added in 4.11, userspace has had an interface for
> > > reading btime (file creation time), but no way to set it. This RFC patch
> > > series adds support for changing btime with utimensat(). Patch 1 adds
> > > the VFS infrastructure, patch 2 adds the support to utimensat() with a
> > > new flag, and the rest of the patches add filesystem support; I excluded
> > > CIFS for now because I don't have a CIFS setup to test it on.
> > >
> > > Updating btime is useful for at least a couple of use cases:
> > >
> > > - Backup/restore programs (my motivation for this feature is btrfs send)
> > > - File servers which interoperate with operating systems that allow
> > > updating file creation time, including Mac OS [1] and Windows [2]
> >
> > So you're adding an interface that allows users to change the create
> > time of files without needing any privileges?
>
> I think it'd be reasonable to make this a privileged operation. I didn't
> for this initial submission for a couple of reasons:
>
> 1. The precedent on Mac OS and Windows is that this isn't a privileged
> operation.
Don't really care about them. Interop file servers that support these
operations on other OSs will need to be storing this info in xattrs
because they have to work on filesystems that don't support btime.
> 2. I knew there would be different opinions on this either way I went.
Yup.
> > Inode create time is forensic metadata in XFS - information we use
> > for sequence of event and inode lifetime analysis during examination
> > of broken filesystem images and systems that have been broken into.
> > Just because it's exposed to userspace via statx(), it doesn't mean
> > that it is information that users should be allowed to change. i.e.
> > allowing users to be able to change the create time on files makes
> > it completely useless for the purpose it was added to XFS for...
> >
> > And allowing root to change the create time doesn't really help,
> > because once you've broken into a system, this makes it really easy
> > to cover tracks
>
> If the threat model is that the attacker has root, then they can
> overwrite the timestamp on disk anyways, no?
Modifying the block devicee under an active filesystem is fraught
with danger, and there's no guarantee it will work if the metadata
being modified is still active in memory. Corrupting the filesystem
is a sure way to get noticed....
> > (e.g. we can't find files that were created and
> > unlinked during the break in window anymore) and lay false
> > trails....
>
> Fair point, although there's still ctime during the break-in window,
Unless you're smart enough to know how to trigger S_NOCMTIME or
FMODE_NOCMTIME....
> which I assume you'd be looking for anyways since files modified during
> the break-in window are also of interest.
... and then that also can't be guaranteed. :/
> I see a few options, none of which are particularly nice:
>
> 1. Filesystems like XFS could choose not to support setting btime even
> if they support reading it.
> 2. XFS could add a second, writeable btime which is used for
> statx/utimes when available (it would fit in di_pad2...).
> 3. We could add a btime_writable sysctl/mount option/mkfs option.
4. create time remains a read-only field, and btrfs grows its own
special interface to twiddle it in btrfs-recv if it really is
necessary.
I'm still not convinced that even backup/restore should be doing this,
because there's so much other metadata that is unique even on
restored files that it doesn't really make any sense to me to lie
about it being created in the past....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Hans van Kranenburg @ 2019-02-15 1:57 UTC (permalink / raw)
To: Omar Sandoval, linux-fsdevel@vger.kernel.org, Al Viro
Cc: kernel-team@fb.com, linux-api@vger.kernel.org,
linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org,
linux-f2fs-devel@lists.sourceforge.net, linux-xfs@vger.kernel.org
In-Reply-To: <cover.1550136164.git.osandov@fb.com>
Hi,
On 2/14/19 11:00 AM, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
>
> Since statx was added in 4.11, userspace has had an interface for
> reading btime (file creation time), but no way to set it. This RFC patch
> series adds support for changing btime with utimensat(). Patch 1 adds
> the VFS infrastructure, patch 2 adds the support to utimensat() with a
> new flag, and the rest of the patches add filesystem support; I excluded
> CIFS for now because I don't have a CIFS setup to test it on.
>
> Updating btime is useful for at least a couple of use cases:
>
> - Backup/restore programs (my motivation for this feature is btrfs send)
Can you give an example of such usefulness? What's the thing you run
into that you can't do without having this?
You guys are having a technical discussion about 'implementation or the
interface' in this thread, while I'm wondering what I'm missing as btrfs
send/receive user by not having this. I never needed it in my use cases.
There's two levels of use case hidden in the above line. So, I don't
mean why btrfs send/receive is useful (it is, for quick efficient
replication of changes) but, what's an important usecase for btime on
top of that?
> - File servers which interoperate with operating systems that allow
> updating file creation time, including Mac OS [1] and Windows [2]
>
> I've also included a man page patch, xfs_io support, and an xfstest.
>
> Thoughts on the implementation or the interface?
>
> Thanks!
>
> 1: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/setattrlist.2.html
> 2: https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-setfiletime
>
> Omar Sandoval (6):
> fs: add btime to struct iattr
> fs: add AT_UTIME_BTIME for utimensat()
> Btrfs: add support for setting btime
> ext4: add support for setting btime
> f2fs: add support for setting btime
> xfs: add support for setting btime
>
> fs/attr.c | 6 +++
> fs/btrfs/inode.c | 2 +
> fs/btrfs/super.c | 4 +-
> fs/ext4/inode.c | 15 +++++-
> fs/ext4/super.c | 2 +-
> fs/f2fs/file.c | 19 ++++++--
> fs/f2fs/super.c | 2 +-
> fs/utimes.c | 86 +++++++++++++++++++++-------------
> fs/xfs/libxfs/xfs_format.h | 2 +-
> fs/xfs/libxfs/xfs_log_format.h | 2 +-
> fs/xfs/xfs_iops.c | 11 ++++-
> fs/xfs/xfs_super.c | 2 +-
> include/linux/fs.h | 4 ++
> include/uapi/linux/fcntl.h | 2 +
> 14 files changed, 111 insertions(+), 48 deletions(-)
>
--
Hans van Kranenburg
^ permalink raw reply
* Re: [PATCH v7 1/2] signal: add pidfd_send_signal() syscall
From: Tycho Andersen @ 2019-02-15 5:00 UTC (permalink / raw)
To: Christian Brauner
Cc: linux-kernel, linux-api, luto, arnd, serge, keescook, akpm, jannh,
oleg, cyphar, viro, linux-fsdevel, dancol, timmurray, fweimer,
tglx, x86, ebiederm
In-Reply-To: <20190102161654.9093-1-christian@brauner.io>
On Wed, Jan 02, 2019 at 05:16:53PM +0100, Christian Brauner wrote:
> The kill() syscall operates on process identifiers (pid). After a process
> ...
Fashionably late to the party, but feel free to consider these two:
Reviewed-by: Tycho Andersen <tycho@tycho.ws>
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Omar Sandoval @ 2019-02-15 5:39 UTC (permalink / raw)
To: Hans van Kranenburg
Cc: linux-fsdevel@vger.kernel.org, Al Viro, kernel-team@fb.com,
linux-api@vger.kernel.org, linux-btrfs@vger.kernel.org,
linux-ext4@vger.kernel.org,
linux-f2fs-devel@lists.sourceforge.net, linux-xfs@vger.kernel.org
In-Reply-To: <1190dcc0-7054-72b4-2a88-33d0b97e332c@mendix.com>
On Fri, Feb 15, 2019 at 01:57:39AM +0000, Hans van Kranenburg wrote:
> Hi,
>
> On 2/14/19 11:00 AM, Omar Sandoval wrote:
> > From: Omar Sandoval <osandov@fb.com>
> >
> > Since statx was added in 4.11, userspace has had an interface for
> > reading btime (file creation time), but no way to set it. This RFC patch
> > series adds support for changing btime with utimensat(). Patch 1 adds
> > the VFS infrastructure, patch 2 adds the support to utimensat() with a
> > new flag, and the rest of the patches add filesystem support; I excluded
> > CIFS for now because I don't have a CIFS setup to test it on.
> >
> > Updating btime is useful for at least a couple of use cases:
> >
> > - Backup/restore programs (my motivation for this feature is btrfs send)
>
> Can you give an example of such usefulness? What's the thing you run
> into that you can't do without having this?
That boils down to what's useful about having the file creation time,
and it's really just another tidbit of information which you may or may
not care about. Maybe you have a document that you've been editing for
awhile, and you want to know when you started working on it. Or, you
want to know when a user created some directory that they keep adding
files to.
If the file creation time is useful to you, then you likely want it
preserved if you have to restore from backups. If I had to restore from
backups yesterday and I'm trying to figure out when I started that
document, I don't care that I restored that file yesterday, I want the
real creation date.
If you've never wondered when a file was created, then I'm sure you
won't care whether btrfs send/receive preserves it :)
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Omar Sandoval @ 2019-02-15 6:59 UTC (permalink / raw)
To: Dave Chinner
Cc: linux-fsdevel, Al Viro, kernel-team, linux-api, linux-btrfs,
linux-ext4, linux-f2fs-devel, linux-xfs, Theodore Ts'o,
Jaegeuk Kim, Steve French
In-Reply-To: <20190215001657.GY14116@dastard>
On Fri, Feb 15, 2019 at 11:16:57AM +1100, Dave Chinner wrote:
> On Thu, Feb 14, 2019 at 03:14:29PM -0800, Omar Sandoval wrote:
> > On Fri, Feb 15, 2019 at 09:06:26AM +1100, Dave Chinner wrote:
> > > On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote:
> > > > From: Omar Sandoval <osandov@fb.com>
> > > >
> > > > Hi,
> > > >
> > > > Since statx was added in 4.11, userspace has had an interface for
> > > > reading btime (file creation time), but no way to set it. This RFC patch
> > > > series adds support for changing btime with utimensat(). Patch 1 adds
> > > > the VFS infrastructure, patch 2 adds the support to utimensat() with a
> > > > new flag, and the rest of the patches add filesystem support; I excluded
> > > > CIFS for now because I don't have a CIFS setup to test it on.
> > > >
> > > > Updating btime is useful for at least a couple of use cases:
> > > >
> > > > - Backup/restore programs (my motivation for this feature is btrfs send)
> > > > - File servers which interoperate with operating systems that allow
> > > > updating file creation time, including Mac OS [1] and Windows [2]
> > >
> > > So you're adding an interface that allows users to change the create
> > > time of files without needing any privileges?
> >
> > I think it'd be reasonable to make this a privileged operation. I didn't
> > for this initial submission for a couple of reasons:
> >
> > 1. The precedent on Mac OS and Windows is that this isn't a privileged
> > operation.
>
> Don't really care about them. Interop file servers that support these
> operations on other OSs will need to be storing this info in xattrs
> because they have to work on filesystems that don't support btime.
>
> > 2. I knew there would be different opinions on this either way I went.
>
> Yup.
>
> > > Inode create time is forensic metadata in XFS - information we use
> > > for sequence of event and inode lifetime analysis during examination
> > > of broken filesystem images and systems that have been broken into.
> > > Just because it's exposed to userspace via statx(), it doesn't mean
> > > that it is information that users should be allowed to change. i.e.
> > > allowing users to be able to change the create time on files makes
> > > it completely useless for the purpose it was added to XFS for...
> > >
> > > And allowing root to change the create time doesn't really help,
> > > because once you've broken into a system, this makes it really easy
> > > to cover tracks
> >
> > If the threat model is that the attacker has root, then they can
> > overwrite the timestamp on disk anyways, no?
>
> Modifying the block devicee under an active filesystem is fraught
> with danger, and there's no guarantee it will work if the metadata
> being modified is still active in memory. Corrupting the filesystem
> is a sure way to get noticed....
>
> > > (e.g. we can't find files that were created and
> > > unlinked during the break in window anymore) and lay false
> > > trails....
> >
> > Fair point, although there's still ctime during the break-in window,
>
> Unless you're smart enough to know how to trigger S_NOCMTIME or
> FMODE_NOCMTIME....
>
> > which I assume you'd be looking for anyways since files modified during
> > the break-in window are also of interest.
>
> ... and then that also can't be guaranteed. :/
>
> > I see a few options, none of which are particularly nice:
> >
> > 1. Filesystems like XFS could choose not to support setting btime even
> > if they support reading it.
> > 2. XFS could add a second, writeable btime which is used for
> > statx/utimes when available (it would fit in di_pad2...).
> > 3. We could add a btime_writable sysctl/mount option/mkfs option.
>
> 4. create time remains a read-only field, and btrfs grows its own
> special interface to twiddle it in btrfs-recv if it really is
> necessary.
I'm curious to hear what the ext4/f2fs/CIFS developers think. If no one
else wants btime to be mutable, then I might as well make it
Btrfs-specific. That is, assuming we reach consensus on the Btrfs side
that btrfs receive should set btime.
> I'm still not convinced that even backup/restore should be doing this,
> because there's so much other metadata that is unique even on
> restored files that it doesn't really make any sense to me to lie
> about it being created in the past....
I suppose it depends on how you interpret btime: if it's strictly
filesystem metadata, then it makes sense that it should be immutable; if
it's metadata for the user's own purposes, then we should allow setting
it.
^ permalink raw reply
* Re: [PATCH] mmap.2: describe the 5level paging hack
From: Michael Ellerman @ 2019-02-15 9:13 UTC (permalink / raw)
To: Jann Horn, mtk.manpages
Cc: linux-arch, linux-man, linux-mm, Peter Zijlstra,
Benjamin Herrenschmidt, Aneesh Kumar K.V, Will Deacon,
linuxppc-dev, Andy Lutomirski, Dave Hansen, Paul Mackerras,
linux-arm-kernel, Catalin Marinas, Andrew Morton, linux-api,
Linus Torvalds, Thomas Gleixner, Kirill A . Shutemov
In-Reply-To: <20190211163653.97742-1-jannh@google.com>
Jann Horn <jannh@google.com> writes:
> The manpage is missing information about the compatibility hack for
> 5-level paging that went in in 4.14, around commit ee00f4a32a76 ("x86/mm:
> Allow userspace have mappings above 47-bit"). Add some information about
> that.
Thanks for doing this.
> While I don't think any hardware supporting this is shipping yet (?), I
> think it's useful to try to write a manpage for this API, partly to
> figure out how usable that API actually is, and partly because when this
> hardware does ship, it'd be nice if distro manpages had information about
> how to use it.
>
> Signed-off-by: Jann Horn <jannh@google.com>
> ---
> This patch goes on top of the patch "[PATCH] mmap.2: fix description of
> treatment of the hint" that I just sent, but I'm not sending them in a
> series because I want the first one to go in, and I think this one might
> be a bit more controversial.
>
> It would be nice if the architecture maintainers and mm folks could have
> a look at this and check that what I wrote is right - I only looked at
> the source for this, I haven't tried it.
>
> man2/mmap.2 | 15 +++++++++++++++
> 1 file changed, 15 insertions(+)
>
> diff --git a/man2/mmap.2 b/man2/mmap.2
> index 8556bbfeb..977782fa8 100644
> --- a/man2/mmap.2
> +++ b/man2/mmap.2
> @@ -67,6 +67,8 @@ is NULL,
> then the kernel chooses the (page-aligned) address
> at which to create the mapping;
> this is the most portable method of creating a new mapping.
> +On Linux, in this case, the kernel may limit the maximum address that can be
> +used for allocations to a legacy limit for compatibility reasons.
> If
> .I addr
> is not NULL,
> @@ -77,6 +79,19 @@ or equal to the value specified by
> and attempt to create the mapping there.
> If another mapping already exists there, the kernel picks a new
> address, independent of the hint.
> +However, if a hint above the architecture's legacy address limit is provided
> +(on x86-64: above 0x7ffffffff000, on arm64: above 0x1000000000000, on ppc64 with
> +book3s: above 0x7fffffffffff or 0x3fffffffffff, depending on page size), the
It doesn't depend on page size for ppc64(le). With 4K pages the user VM
is always 64TB.
So the only boundary for us is at 128T when using 64K pages.
cheers
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: David Disseldorp via Linux-f2fs-devel @ 2019-02-15 13:57 UTC (permalink / raw)
To: Omar Sandoval, Samba Technical
Cc: Steve French, Theodore Ts'o, linux-api, Dave Chinner,
linux-f2fs-devel, linux-xfs, Al Viro, linux-fsdevel, Jaegeuk Kim,
kernel-team, linux-ext4, linux-btrfs
In-Reply-To: <20190215065947.GG9819@vader>
On Thu, 14 Feb 2019 22:59:47 -0800, Omar Sandoval wrote:
> On Fri, Feb 15, 2019 at 11:16:57AM +1100, Dave Chinner wrote:
> > On Thu, Feb 14, 2019 at 03:14:29PM -0800, Omar Sandoval wrote:
> > > On Fri, Feb 15, 2019 at 09:06:26AM +1100, Dave Chinner wrote:
...
> > > > Inode create time is forensic metadata in XFS - information we use
> > > > for sequence of event and inode lifetime analysis during examination
> > > > of broken filesystem images and systems that have been broken into.
> > > > Just because it's exposed to userspace via statx(), it doesn't mean
> > > > that it is information that users should be allowed to change. i.e.
> > > > allowing users to be able to change the create time on files makes
> > > > it completely useless for the purpose it was added to XFS for...
> > > >
> > > > And allowing root to change the create time doesn't really help,
> > > > because once you've broken into a system, this makes it really easy
> > > > to cover tracks
> > >
> > > If the threat model is that the attacker has root, then they can
> > > overwrite the timestamp on disk anyways, no?
> >
> > Modifying the block devicee under an active filesystem is fraught
> > with danger, and there's no guarantee it will work if the metadata
> > being modified is still active in memory. Corrupting the filesystem
> > is a sure way to get noticed....
> >
> > > > (e.g. we can't find files that were created and
> > > > unlinked during the break in window anymore) and lay false
> > > > trails....
> > >
> > > Fair point, although there's still ctime during the break-in window,
> >
> > Unless you're smart enough to know how to trigger S_NOCMTIME or
> > FMODE_NOCMTIME....
> >
> > > which I assume you'd be looking for anyways since files modified during
> > > the break-in window are also of interest.
I'm not sure I follow the forensics use-case for immutable btime. I'd
expect dm-verity or selinux/apparmor audits to do a better job for those
worried about this kind of attack.
> > ... and then that also can't be guaranteed. :/
> >
> > > I see a few options, none of which are particularly nice:
> > >
> > > 1. Filesystems like XFS could choose not to support setting btime even
> > > if they support reading it.
> > > 2. XFS could add a second, writeable btime which is used for
> > > statx/utimes when available (it would fit in di_pad2...).
> > > 3. We could add a btime_writable sysctl/mount option/mkfs option.
> >
> > 4. create time remains a read-only field, and btrfs grows its own
> > special interface to twiddle it in btrfs-recv if it really is
> > necessary.
>
> I'm curious to hear what the ext4/f2fs/CIFS developers think. If no one
> else wants btime to be mutable, then I might as well make it
> Btrfs-specific. That is, assuming we reach consensus on the Btrfs side
> that btrfs receive should set btime.
Samba currently uses a user.DOSATTRIB xattr for tracking creation time.
IMO a mutable btime accessible via statx would be useful for
cross-protocol interoperability.
Cheers, David
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Hans van Kranenburg @ 2019-02-15 18:25 UTC (permalink / raw)
To: Omar Sandoval
Cc: linux-fsdevel@vger.kernel.org, Al Viro, kernel-team@fb.com,
linux-api@vger.kernel.org, linux-btrfs@vger.kernel.org,
linux-ext4@vger.kernel.org,
linux-f2fs-devel@lists.sourceforge.net, linux-xfs@vger.kernel.org
In-Reply-To: <20190215053950.GF9819@vader>
On 2/15/19 6:39 AM, Omar Sandoval wrote:
> On Fri, Feb 15, 2019 at 01:57:39AM +0000, Hans van Kranenburg wrote:
>> Hi,
>>
>> On 2/14/19 11:00 AM, Omar Sandoval wrote:
>>> From: Omar Sandoval <osandov@fb.com>
>>>
>>> Since statx was added in 4.11, userspace has had an interface for
>>> reading btime (file creation time), but no way to set it. This RFC patch
>>> series adds support for changing btime with utimensat(). Patch 1 adds
>>> the VFS infrastructure, patch 2 adds the support to utimensat() with a
>>> new flag, and the rest of the patches add filesystem support; I excluded
>>> CIFS for now because I don't have a CIFS setup to test it on.
>>>
>>> Updating btime is useful for at least a couple of use cases:
>>>
>>> - Backup/restore programs (my motivation for this feature is btrfs send)
>>
>> Can you give an example of such usefulness? What's the thing you run
>> into that you can't do without having this?
>
> That boils down to what's useful about having the file creation time,
> and it's really just another tidbit of information which you may or may
> not care about. Maybe you have a document that you've been editing for
> awhile, and you want to know when you started working on it. Or, you
> want to know when a user created some directory that they keep adding
> files to.
>
> If the file creation time is useful to you, then you likely want it
> preserved if you have to restore from backups. If I had to restore from
> backups yesterday and I'm trying to figure out when I started that
> document, I don't care that I restored that file yesterday, I want the
> real creation date.
>
> If you've never wondered when a file was created, then I'm sure you
> won't care whether btrfs send/receive preserves it :)
Thanks for the elaborate answer. I was just curious if this was to make
send "more feature complete" or if there was something else special to
it, and it seems to be the first one. ;]
And for myself, I mostly treat btrfs send/receive like a improved
replacement for rsync and usually don't care that much about btime indeed.
Thanks,
--
Hans van Kranenburg
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Andreas Dilger @ 2019-02-17 1:57 UTC (permalink / raw)
To: Omar Sandoval
Cc: Dave Chinner, linux-fsdevel, Al Viro, kernel-team, Linux API,
linux-btrfs, linux-ext4, linux-f2fs-devel, linux-xfs,
Theodore Ts'o, Jaegeuk Kim, Steve French
In-Reply-To: <20190215065947.GG9819@vader>
[-- Attachment #1: Type: text/plain, Size: 2264 bytes --]
On Feb 14, 2019, at 11:59 PM, Omar Sandoval <osandov@osandov.com> wrote:
>
> On Fri, Feb 15, 2019 at 11:16:57AM +1100, Dave Chinner wrote:
>> On Thu, Feb 14, 2019 at 03:14:29PM -0800, Omar Sandoval wrote:
>>> I see a few options, none of which are particularly nice:
>>>
>>> 1. Filesystems like XFS could choose not to support setting btime even
>>> if they support reading it.
>>> 2. XFS could add a second, writeable btime which is used for
>>> statx/utimes when available (it would fit in di_pad2...).
>>> 3. We could add a btime_writable sysctl/mount option/mkfs option.
>>
>> 4. create time remains a read-only field, and btrfs grows its own
>> special interface to twiddle it in btrfs-recv if it really is
>> necessary.
>
> I'm curious to hear what the ext4/f2fs/CIFS developers think. If no one
> else wants btime to be mutable, then I might as well make it
> Btrfs-specific. That is, assuming we reach consensus on the Btrfs side
> that btrfs receive should set btime.
>
>> I'm still not convinced that even backup/restore should be doing this,
>> because there's so much other metadata that is unique even on
>> restored files that it doesn't really make any sense to me to lie
>> about it being created in the past....
>
> I suppose it depends on how you interpret btime: if it's strictly
> filesystem metadata, then it makes sense that it should be immutable; if
> it's metadata for the user's own purposes, then we should allow setting
> it.
My personal preference is that crtime/btime be read-only information that
tells when the file itself was created in this filesystem, not some added
metadata that is managed by userspace. There have been many times when
I've needed to know when a file was _actually_ created in the filesystem,
not what mtime/ctime report.
While it may be a bit of a stretch to call this "forensic evidence", making
it hard to change from except via total root compromise by a skilled hacker
is very useful.
If this were to go in (which I'm not in favour of), then there would need to
be a CONFIG and/or runtime knob to turn it off (or better to only turn it on),
similar to how FIPS and other security options can only go in one direction.
Cheers, Andreas
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 873 bytes --]
^ permalink raw reply
* Re: [PATCH net-next] sock: consistent handling of extreme SO_SNDBUF/SO_RCVBUF values
From: David Miller @ 2019-02-17 2:10 UTC (permalink / raw)
To: gnault; +Cc: netdev, linux-api, edumazet, mleitner, pabeni
In-Reply-To: <2cb69b6987b6e23f8b8c1aa8dc524efa6bd53191.1550026643.git.gnault@redhat.com>
From: Guillaume Nault <gnault@redhat.com>
Date: Wed, 13 Feb 2019 04:30:34 +0100
> SO_SNDBUF and SO_RCVBUF (and their *BUFFORCE version) may overflow or
> underflow their input value. This patch aims at providing explicit
> handling of these extreme cases, to get a clear behaviour even with
> values bigger than INT_MAX / 2 or lower than INT_MIN / 2.
...
> Signed-off-by: Guillaume Nault <gnault@redhat.com>
Thank you so much for writing such a detailed commit message.
Applied.
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Boaz Harrosh @ 2019-02-17 16:35 UTC (permalink / raw)
To: Dave Chinner, Omar Sandoval
Cc: linux-fsdevel, Al Viro, kernel-team, linux-api, linux-btrfs,
linux-ext4, linux-f2fs-devel, linux-xfs
In-Reply-To: <20190214220626.GV14116@dastard>
On 15/02/19 00:06, Dave Chinner wrote:
> On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote:
>> From: Omar Sandoval <osandov@fb.com>
>>
>> Hi,
>>
>> Since statx was added in 4.11, userspace has had an interface for
>> reading btime (file creation time), but no way to set it. This RFC patch
>> series adds support for changing btime with utimensat(). Patch 1 adds
>> the VFS infrastructure, patch 2 adds the support to utimensat() with a
>> new flag, and the rest of the patches add filesystem support; I excluded
>> CIFS for now because I don't have a CIFS setup to test it on.
>>
>> Updating btime is useful for at least a couple of use cases:
>>
[1]
>> - Backup/restore programs (my motivation for this feature is btrfs send)
>> - File servers which interoperate with operating systems that allow
>> updating file creation time, including Mac OS [1] and Windows [2]
>
> So you're adding an interface that allows users to change the create
> time of files without needing any privileges?
>
[2]
> Inode create time is forensic metadata in XFS - information we use
> for sequence of event and inode lifetime analysis during examination
> of broken filesystem images and systems that have been broken into.
> Just because it's exposed to userspace via statx(), it doesn't mean
> that it is information that users should be allowed to change. i.e.
> allowing users to be able to change the create time on files makes
> it completely useless for the purpose it was added to XFS for...
>
<snap>
I think the difference in opinion here is that there are two totally
different BTIme out in the world. For two somewhat opposite motivations
and it seems they both try to be crammed into the same on disk space.
One - Author creation time
This is a Windows originated creature and later MAC (and all vendors who
make a living by serving cifs (hint see my email address))
This is a tag carried globally on the globe denoting the time of the
original creator of the file. copy, download, backup-restore and so
on preserve it from the very first original creation.
This creature is a user oriented information. That needs to be carefully
orchestrated by all parties
Two - Local creation time
This is an immutable local FS information that helps in debugging and
FS-checking / recovery of data. It is an information that kind of denotes
the order of creation of files on a local FS.
So it looks like both sides are correct trying to preserve their own guy?
XFS invented [2] I'd let it be. If you need [1] on XFS better push for
a well defined standardized xattr and be in peace.
BTRFS should decide which one of [2] or [1] it has space for in the inode
and commit to it. Documenting well what it is.
That said putting my Netapp hat. I would love to see an easy API
for Author-creation-time BTime type of value. That is accessed
uniformly by user-mode and/or Network file servers (NFS/CIFS).
And would love to see a generic implementation of that interface
that puts it into a standardized xattr if the FS in question does
not have a native support for it [1].
So I love these patches. And would want to see this through. But
let us understand each other?
Thanks
Boaz
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Adam Borowski @ 2019-02-17 17:54 UTC (permalink / raw)
To: Boaz Harrosh
Cc: Dave Chinner, Omar Sandoval, linux-fsdevel, Al Viro, kernel-team,
linux-api, linux-btrfs, linux-ext4, linux-f2fs-devel, linux-xfs
In-Reply-To: <6a9dc05a-0445-d0ab-0140-1de4fee7ba9b@gmail.com>
On Sun, Feb 17, 2019 at 06:35:25PM +0200, Boaz Harrosh wrote:
> On 15/02/19 00:06, Dave Chinner wrote:
> > So you're adding an interface that allows users to change the create
> > time of files without needing any privileges?
> > Inode create time is forensic metadata in XFS - information we use
> > for sequence of event and inode lifetime analysis during examination
> > of broken filesystem images and systems that have been broken into.
> I think the difference in opinion here is that there are two totally
> different BTIme out in the world. For two somewhat opposite motivations
> and it seems they both try to be crammed into the same on disk space.
>
> One - Author creation time
> Two - Local creation time
> So it looks like both sides are correct trying to preserve their own guy?
I'd say that [2] is too easily gameable to be worth the effort. You can
just change it on the disk. That right now it'd take some skill to find the
right place to edit doesn't matter -- a tool to update the btime against
your wishes would need to be written just once. Unlike btrfs, XFS doesn't
even have a chain of checksums all the way to the root.
On the other hand, [1] has a lot of uses. It can also be preserved in
backups and version control (svnt and git-restore-mtime could be easily
extended).
I'd thus go with [2] -- any uses for [1] are better delegated to filesystem
specific tools.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Have you accepted Khorne as your lord and saviour?
⠈⠳⣄⠀⠀⠀⠀
^ permalink raw reply
* Re: [RFC PATCH 0/6] Allow setting file birth time with utimensat()
From: Andy Lutomirski @ 2019-02-17 20:40 UTC (permalink / raw)
To: Adam Borowski
Cc: Boaz Harrosh, Dave Chinner, Omar Sandoval, Linux FS Devel,
Al Viro, kernel-team, Linux API, Linux btrfs Developers List,
Ext4 Developers List, linux-f2fs-devel, linux-xfs
In-Reply-To: <20190217175450.psaesabv3vlzvjv4@angband.pl>
On Sun, Feb 17, 2019 at 9:55 AM Adam Borowski <kilobyte@angband.pl> wrote:
>
> On Sun, Feb 17, 2019 at 06:35:25PM +0200, Boaz Harrosh wrote:
> > On 15/02/19 00:06, Dave Chinner wrote:
> > > So you're adding an interface that allows users to change the create
> > > time of files without needing any privileges?
>
> > > Inode create time is forensic metadata in XFS - information we use
> > > for sequence of event and inode lifetime analysis during examination
> > > of broken filesystem images and systems that have been broken into.
>
> > I think the difference in opinion here is that there are two totally
> > different BTIme out in the world. For two somewhat opposite motivations
> > and it seems they both try to be crammed into the same on disk space.
> >
> > One - Author creation time
> > Two - Local creation time
>
> > So it looks like both sides are correct trying to preserve their own guy?
>
> I'd say that [2] is too easily gameable to be worth the effort. You can
> just change it on the disk. That right now it'd take some skill to find the
> right place to edit doesn't matter -- a tool to update the btime against
> your wishes would need to be written just once. Unlike btrfs, XFS doesn't
> even have a chain of checksums all the way to the root.
>
> On the other hand, [1] has a lot of uses. It can also be preserved in
> backups and version control (svnt and git-restore-mtime could be easily
> extended).
>
> I'd thus go with [2] -- any uses for [1] are better delegated to filesystem
> specific tools.
>
I started out in the Windows world, and I found it quite handy to
right-click a file and see when it was created. When I started using
Linux, I saw things like this:
Access: 2019-02-16 22:19:32.024284060 -0800
Modify: 2016-02-02 19:26:47.901766778 -0800
Change: 2016-02-02 19:26:47.907766649 -0800
and my mind boggled a bit. Modify makes sense. Change? What's that?
Why do I care?
Adding "birth" makes sense, and I think that filesystem-agnostic
backup tools *should* be able to write it.
So I'm highly in favor of this patch. If XFS wants to disallow
writing the birth time, fine, but I think that behavior should be
overridable.
^ permalink raw reply
* mremap vs sysctl_max_map_count
From: Oscar Salvador @ 2019-02-18 8:33 UTC (permalink / raw)
To: linux-mm, linux-kernel, linux-api; +Cc: hughd, viro, torvalds
Hi all,
I would like to bring up a topic that comes from an issue a customer of ours
is facing with the mremap syscall + hitting the max_map_count threshold:
When passing the MREMAP_FIXED flag, mremap() calls mremap_to() which does the
following:
1) it unmaps the region where we want to put the new map:
(new_addr, new_addr + new_len] [1]
2) IFF old_len > new_len, it unmaps the region:
(old_addr + new_len, (old_addr + new_len) + (old_len - new_len)] [2]
Now, having gone through steps 1) and 2), we eventually call move_vma() to do
the actual move.
move_vma() checks if we are at least 4 maps below max_map_count, otherwise
it bails out with -ENOMEM [3].
The problem is that we might have already unmapped the vma's in steps 1) and 2),
so it is not possible for userspace to figure out the state of the vma's after
it gets -ENOMEM.
- Did new_addr got unmaped?
- Did part of the old_addr got unmaped?
Because of that, it gets tricky for userspace to clean up properly on error
path.
While it is true that we can return -ENOMEM for more reasons
(e.g: see vma_to_resize()->may_expand_vm()), I think that we might be able to
pre-compute the number of maps that we are going add/release during the first
two do_munmaps(), and check whether we are 4 maps below the threshold
(as move_vma() does).
Should not be the case, we can bail out early before we unmap anything, so we
make sure the vma's are left untouched in case we are going to be short of maps.
I am not sure if that is realistically doable, or there are limitations
I overlooked, or we simply do not want to do that.
Before investing more time and giving it a shoot, I just wanted to bring
this upstream to get feedback on this matter.
Thanks
[1] https://github.com/torvalds/linux/blob/master/mm/mremap.c#L519
[2] https://github.com/torvalds/linux/blob/master/mm/mremap.c#L523
[3] https://github.com/torvalds/linux/blob/master/mm/mremap.c#L338
--
Oscar Salvador
SUSE L3
^ permalink raw reply
* Re: mremap vs sysctl_max_map_count
From: Vlastimil Babka @ 2019-02-18 9:57 UTC (permalink / raw)
To: Oscar Salvador, linux-mm, linux-kernel, linux-api; +Cc: hughd, viro, torvalds
In-Reply-To: <20190218083326.xsnx7cx2lxurbmux@d104.suse.de>
On 2/18/19 9:33 AM, Oscar Salvador wrote:
>
> Hi all,
>
> I would like to bring up a topic that comes from an issue a customer of ours
> is facing with the mremap syscall + hitting the max_map_count threshold:
>
> When passing the MREMAP_FIXED flag, mremap() calls mremap_to() which does the
> following:
>
> 1) it unmaps the region where we want to put the new map:
> (new_addr, new_addr + new_len] [1]
> 2) IFF old_len > new_len, it unmaps the region:
> (old_addr + new_len, (old_addr + new_len) + (old_len - new_len)] [2]
>
> Now, having gone through steps 1) and 2), we eventually call move_vma() to do
> the actual move.
>
> move_vma() checks if we are at least 4 maps below max_map_count, otherwise
> it bails out with -ENOMEM [3].
> The problem is that we might have already unmapped the vma's in steps 1) and 2),
> so it is not possible for userspace to figure out the state of the vma's after
> it gets -ENOMEM.
>
> - Did new_addr got unmaped?
> - Did part of the old_addr got unmaped?
>
> Because of that, it gets tricky for userspace to clean up properly on error
> path.
>
> While it is true that we can return -ENOMEM for more reasons
> (e.g: see vma_to_resize()->may_expand_vm()), I think that we might be able to
> pre-compute the number of maps that we are going add/release during the first
> two do_munmaps(), and check whether we are 4 maps below the threshold
> (as move_vma() does).
> Should not be the case, we can bail out early before we unmap anything, so we
> make sure the vma's are left untouched in case we are going to be short of maps.
>
> I am not sure if that is realistically doable, or there are limitations
> I overlooked, or we simply do not want to do that.
IMHO it makes sense to do all such resource limit checks upfront. It
should all be protected by mmap_sem and thus stable, right? Even if it
was racy, I'd think it's better to breach the limit a bit due to a race
than bail out in the middle of operation. Being also resilient against
"real" ENOMEM's due to e.g. failure to alocate a vma would be much
harder perhaps (but maybe it's already mostly covered by the
too-small-to-fail in page allocator), but I'd try with the artificial
limits at least.
> Before investing more time and giving it a shoot, I just wanted to bring
> this upstream to get feedback on this matter.
>
> Thanks
>
> [1] https://github.com/torvalds/linux/blob/master/mm/mremap.c#L519
> [2] https://github.com/torvalds/linux/blob/master/mm/mremap.c#L523
> [3] https://github.com/torvalds/linux/blob/master/mm/mremap.c#L338
>
^ permalink raw reply
* Re: mremap vs sysctl_max_map_count
From: Mike Rapoport @ 2019-02-18 10:29 UTC (permalink / raw)
To: Vlastimil Babka
Cc: Oscar Salvador, linux-mm, linux-kernel, linux-api, hughd, viro,
torvalds
In-Reply-To: <a11a10b5-4a31-2537-7b14-83f4b22e5f6c@suse.cz>
On Mon, Feb 18, 2019 at 10:57:18AM +0100, Vlastimil Babka wrote:
> On 2/18/19 9:33 AM, Oscar Salvador wrote:
> >
> > Hi all,
> >
> > I would like to bring up a topic that comes from an issue a customer of ours
> > is facing with the mremap syscall + hitting the max_map_count threshold:
> >
> > When passing the MREMAP_FIXED flag, mremap() calls mremap_to() which does the
> > following:
> >
> > 1) it unmaps the region where we want to put the new map:
> > (new_addr, new_addr + new_len] [1]
> > 2) IFF old_len > new_len, it unmaps the region:
> > (old_addr + new_len, (old_addr + new_len) + (old_len - new_len)] [2]
> >
> > Now, having gone through steps 1) and 2), we eventually call move_vma() to do
> > the actual move.
> >
> > move_vma() checks if we are at least 4 maps below max_map_count, otherwise
> > it bails out with -ENOMEM [3].
> > The problem is that we might have already unmapped the vma's in steps 1) and 2),
> > so it is not possible for userspace to figure out the state of the vma's after
> > it gets -ENOMEM.
> >
> > - Did new_addr got unmaped?
> > - Did part of the old_addr got unmaped?
> >
> > Because of that, it gets tricky for userspace to clean up properly on error
> > path.
> >
> > While it is true that we can return -ENOMEM for more reasons
> > (e.g: see vma_to_resize()->may_expand_vm()), I think that we might be able to
> > pre-compute the number of maps that we are going add/release during the first
> > two do_munmaps(), and check whether we are 4 maps below the threshold
> > (as move_vma() does).
> > Should not be the case, we can bail out early before we unmap anything, so we
> > make sure the vma's are left untouched in case we are going to be short of maps.
> >
> > I am not sure if that is realistically doable, or there are limitations
> > I overlooked, or we simply do not want to do that.
>
> IMHO it makes sense to do all such resource limit checks upfront. It
> should all be protected by mmap_sem and thus stable, right? Even if it
> was racy, I'd think it's better to breach the limit a bit due to a race
> than bail out in the middle of operation. Being also resilient against
> "real" ENOMEM's due to e.g. failure to alocate a vma would be much
> harder perhaps (but maybe it's already mostly covered by the
> too-small-to-fail in page allocator), but I'd try with the artificial
> limits at least.
The mremap_to() is called with mmap_sem hold, so there won't be a race.
But it seems mremap_to() is not the only path to call do_munmap(). There is
also an unmap in shrinking remap and possible move_vma() even with
~MREMAP_FIXED.
Maybe it'd make sense to check the limits right after taking the mmap_sem?
> > Before investing more time and giving it a shoot, I just wanted to bring
> > this upstream to get feedback on this matter.
> >
> > Thanks
> >
> > [1] https://github.com/torvalds/linux/blob/master/mm/mremap.c#L519
> > [2] https://github.com/torvalds/linux/blob/master/mm/mremap.c#L523
> > [3] https://github.com/torvalds/linux/blob/master/mm/mremap.c#L338
> >
>
--
Sincerely yours,
Mike.
^ permalink raw reply
* Re: mremap vs sysctl_max_map_count
From: Kirill A. Shutemov @ 2019-02-18 11:15 UTC (permalink / raw)
To: Vlastimil Babka
Cc: Oscar Salvador, linux-mm, linux-kernel, linux-api, hughd, viro,
torvalds
In-Reply-To: <a11a10b5-4a31-2537-7b14-83f4b22e5f6c@suse.cz>
On Mon, Feb 18, 2019 at 10:57:18AM +0100, Vlastimil Babka wrote:
> On 2/18/19 9:33 AM, Oscar Salvador wrote:
> >
> > Hi all,
> >
> > I would like to bring up a topic that comes from an issue a customer of ours
> > is facing with the mremap syscall + hitting the max_map_count threshold:
> >
> > When passing the MREMAP_FIXED flag, mremap() calls mremap_to() which does the
> > following:
> >
> > 1) it unmaps the region where we want to put the new map:
> > (new_addr, new_addr + new_len] [1]
> > 2) IFF old_len > new_len, it unmaps the region:
> > (old_addr + new_len, (old_addr + new_len) + (old_len - new_len)] [2]
> >
> > Now, having gone through steps 1) and 2), we eventually call move_vma() to do
> > the actual move.
> >
> > move_vma() checks if we are at least 4 maps below max_map_count, otherwise
> > it bails out with -ENOMEM [3].
> > The problem is that we might have already unmapped the vma's in steps 1) and 2),
> > so it is not possible for userspace to figure out the state of the vma's after
> > it gets -ENOMEM.
> >
> > - Did new_addr got unmaped?
> > - Did part of the old_addr got unmaped?
> >
> > Because of that, it gets tricky for userspace to clean up properly on error
> > path.
> >
> > While it is true that we can return -ENOMEM for more reasons
> > (e.g: see vma_to_resize()->may_expand_vm()), I think that we might be able to
> > pre-compute the number of maps that we are going add/release during the first
> > two do_munmaps(), and check whether we are 4 maps below the threshold
> > (as move_vma() does).
> > Should not be the case, we can bail out early before we unmap anything, so we
> > make sure the vma's are left untouched in case we are going to be short of maps.
> >
> > I am not sure if that is realistically doable, or there are limitations
> > I overlooked, or we simply do not want to do that.
>
> IMHO it makes sense to do all such resource limit checks upfront. It
> should all be protected by mmap_sem and thus stable, right? Even if it
> was racy, I'd think it's better to breach the limit a bit due to a race
> than bail out in the middle of operation. Being also resilient against
> "real" ENOMEM's due to e.g. failure to alocate a vma would be much
> harder perhaps (but maybe it's already mostly covered by the
> too-small-to-fail in page allocator), but I'd try with the artificial
> limits at least.
There's slight chance of false-postive -ENOMEM with upfront approach:
unmapping can reduce number of VMAs so in some cases upfront check would
fail something that could succeed otherwise.
We could check also what number of VMA unmap would free (if any). But it
complicates the picture and I don't think worth it in the end.
--
Kirill A. Shutemov
^ permalink raw reply
* Re: [PATCHv6 00/10] Heterogenous memory node attributes
From: Brice Goglin @ 2019-02-18 14:25 UTC (permalink / raw)
To: Keith Busch, linux-kernel, linux-acpi, linux-mm, linux-api
Cc: Greg Kroah-Hartman, Rafael Wysocki, Dave Hansen, Dan Williams
In-Reply-To: <20190214171017.9362-1-keith.busch@intel.com>
Le 14/02/2019 à 18:10, Keith Busch a écrit :
> == Changes since v5 ==
>
> Updated HMAT parsing to account for the recently released ACPI 6.3
> changes.
>
> HMAT attribute calculation overflow checks.
>
> Fixed memory leak if HMAT parse fails.
>
> Minor change to the patch order. All the base node attributes occur
> before HMAT usage for these new node attributes to resolve a
> dependency on a new struct.
>
> Reporting failures to parse HMAT or allocate structures are elevated
> to a NOTICE level from DEBUG. Any failure will result in just one
> print so that it is obvious something may need to be investigated
> rather than silently fail, but also not to be too alarming either.
>
> Determining the cpu and memory node local relationships is quite
> different this time (PATCH 7/10). The local relationship to a memory
> target will be either *only* the node from the Initiator Proximity
> Domain if provided, or if it is not provided, all the nodes that have
> the same highest performance. Latency was chosen to take prioirty over
> bandwidth when ranking performance.
Hello Keith
I am trying to understand what this last paragraph means.
Let's say I have a machine with DDR and NVDIMM both attached to the same
socket, and I use Dave Hansen's kmem patchs to make the NVDIMM appear as
"normal memory" in an additional NUMA node. Let's call node0 the DDR and
node1 the NVDIMM kmem node.
Now user-space wants to find out which CPUs are actually close to the
NVDIMMs. My understanding is that SRAT says that CPUs are local to the
DDR only. Hence /sys/devices/system/node/node1/cpumap says there are no
CPU local to the NVDIMM. And HMAT won't change this, right?
Will node1 contain access0/initiators/node0 to clarify that CPUs local
to the NVDIMM are those of node0? Even if latency from node0 to node1
latency is higher than node0 to node0?
Another way to ask this: Is the latency/performance only used for
distinguishing the local initiator CPUs among multiple CPU nodes
accesing the same memory node? Or is it also used to distinguish the
local memory target among multiple memories access by a single CPU node?
The Intel machine I am currently testing patches on doesn't have a HMAT
in 1-level-memory unfortunately.
Thanks
Brice
^ permalink raw reply
* [PATCH 0/8] y2038: remove time32 ABI on rv32 and csky
From: Arnd Bergmann @ 2019-02-18 21:07 UTC (permalink / raw)
To: y2038
Cc: linux-arch, uclinux-h8-devel, arnd, yury.norov, linux-api,
Palmer Dabbelt, linux-kernel, linux-riscv, Vineet Gupta, Guo Ren,
Greentime Hu, linux-hexagon, Thomas Gleixner, Guan Xuetao,
Stafford Horne, linux-arm-kernel
This is a follow-up to the y2038 syscall patches that Thomas already
merged. Both the csky and riscv maintainers have requested that the
time_t syscall ABI should be completely converted before an official
glibc port is merged.
The series achieves this in a few steps:
- A couple of bug fixes for minor regressions I introduced
in the original series
- A couple of older patches from Yury Norov that I had never
merged in the past, these fix up the openat/open_by_handle_at
and getrlimit/setrlimit syscalls to disallow the old versions
of off_t and rlimit.
- Hiding the deprecated system calls behind an #ifdef in
include/uapi/asm-generic/unistd.h
- Changing the two architectures to use the new style
syscall ABI for all three types: loff_t, rusage, and
time_t.
Arnd Bergmann (5):
asm-generic: Make time32 syscall numbers optional
unicore32: Fix __ARCH_WANT_STAT64 definition
checksyscalls: fix up mq_timedreceive and stat exceptions
csky: Use latest system call ABI
riscv: Use latest system call ABI
Yury Norov (3):
compat ABI: use non-compat openat and open_by_handle_at variants
32-bit userspace ABI: introduce ARCH_32BIT_OFF_T config option
asm-generic: Drop getrlimit and setrlimit syscalls from default list
arch/Kconfig | 15 ++++++++
arch/arc/Kconfig | 1 +
arch/arc/include/uapi/asm/unistd.h | 2 ++
arch/arm/Kconfig | 1 +
arch/arm64/include/uapi/asm/unistd.h | 2 ++
arch/c6x/Kconfig | 1 +
arch/c6x/include/uapi/asm/unistd.h | 2 ++
arch/h8300/Kconfig | 1 +
arch/h8300/include/uapi/asm/unistd.h | 2 ++
arch/hexagon/Kconfig | 1 +
arch/hexagon/include/uapi/asm/unistd.h | 2 ++
arch/m68k/Kconfig | 1 +
arch/microblaze/Kconfig | 1 +
arch/mips/Kconfig | 1 +
arch/nds32/Kconfig | 1 +
arch/nds32/include/uapi/asm/unistd.h | 2 ++
arch/nios2/Kconfig | 1 +
arch/nios2/include/uapi/asm/unistd.h | 2 ++
arch/openrisc/Kconfig | 1 +
arch/openrisc/include/uapi/asm/unistd.h | 2 ++
arch/parisc/Kconfig | 1 +
arch/powerpc/Kconfig | 1 +
arch/riscv/include/uapi/asm/unistd.h | 1 +
arch/riscv/kernel/vdso/Makefile | 2 ++
arch/sh/Kconfig | 1 +
arch/sparc/Kconfig | 1 +
arch/unicore32/Kconfig | 1 +
arch/unicore32/include/uapi/asm/unistd.h | 4 ++-
arch/x86/Kconfig | 1 +
arch/x86/um/Kconfig | 1 +
arch/xtensa/Kconfig | 1 +
include/linux/fcntl.h | 2 +-
include/uapi/asm-generic/unistd.h | 46 ++++++++++++++++++++++--
scripts/checksyscalls.sh | 15 ++++++--
34 files changed, 112 insertions(+), 8 deletions(-)
--
2.20.0
^ permalink raw reply
* [PATCH 1/8] compat ABI: use non-compat openat and open_by_handle_at variants
From: Arnd Bergmann @ 2019-02-18 21:07 UTC (permalink / raw)
To: y2038
Cc: Thomas Gleixner, linux-arch, linux-api, linux-kernel, yury.norov,
linux-arm-kernel, linux-hexagon, uclinux-h8-devel, Stafford Horne,
Vineet Gupta, Palmer Dabbelt, Guo Ren, Greentime Hu, arnd,
linux-riscv, Guan Xuetao, Yury Norov, Yury Norov
In-Reply-To: <20190218210712.3503891-1-arnd@arndb.de>
From: Yury Norov <ynorov@caviumnetworks.com>
The only difference between native and compat openat and open_by_handle_at
is that non-compat version forces O_LARGEFILE, and it should be the
default behaviour for all architectures, as we are going to drop the
support of 32-bit userspace off_t.
Signed-off-by: Yury Norov <ynorov@caviumnetworks.com>
Signed-off-by: Yury Norov <ynorov@marvell.com>
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
---
include/uapi/asm-generic/unistd.h | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index acf9a07ab2ff..b928eff3bf92 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -179,7 +179,7 @@ __SYSCALL(__NR_fchownat, sys_fchownat)
#define __NR_fchown 55
__SYSCALL(__NR_fchown, sys_fchown)
#define __NR_openat 56
-__SC_COMP(__NR_openat, sys_openat, compat_sys_openat)
+__SYSCALL(__NR_openat, sys_openat)
#define __NR_close 57
__SYSCALL(__NR_close, sys_close)
#define __NR_vhangup 58
@@ -678,8 +678,7 @@ __SYSCALL(__NR_fanotify_mark, sys_fanotify_mark)
#define __NR_name_to_handle_at 264
__SYSCALL(__NR_name_to_handle_at, sys_name_to_handle_at)
#define __NR_open_by_handle_at 265
-__SC_COMP(__NR_open_by_handle_at, sys_open_by_handle_at, \
- compat_sys_open_by_handle_at)
+__SYSCALL(__NR_open_by_handle_at, sys_open_by_handle_at)
#define __NR_clock_adjtime 266
__SC_3264(__NR_clock_adjtime, sys_clock_adjtime32, sys_clock_adjtime)
#define __NR_syncfs 267
--
2.20.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).