* Re: [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Sasha Levin @ 2019-07-19 0:39 UTC (permalink / raw)
To: Kees Cook
Cc: corbet, solar, will, peterz, gregkh, tyhicks, linux-doc,
linux-kernel
In-Reply-To: <201907181457.D61AC061C@keescook>
On Thu, Jul 18, 2019 at 03:00:55PM -0700, Kees Cook wrote:
>On Wed, Jul 17, 2019 at 07:11:03PM -0400, Sasha Levin wrote:
>> Provide more information about how to interact with the linux-distros
>> mailing list for disclosing security bugs.
>>
>> Reference the linux-distros list policy and clarify that the reporter
>> must read and understand those policies as they differ from
>> security@kernel.org's policy.
>>
>> Suggested-by: Solar Designer <solar@openwall.com>
>> Signed-off-by: Sasha Levin <sashal@kernel.org>
>
>Sorry, but NACK, see below...
>
>> ---
>>
>> Changes in v2:
>> - Focus more on pointing to the linux-distros wiki and policies.
>
>I think this is already happening in the text. What specifically do you
>want described differently?
The main issue was that there isn't anything pointing to the
linux-distros policies. The current text outlines a few of them ("add
[vs]", and "there should be an embargo period"), but it effectively just
gives out the linux-distros mailing address and tells the reporter to
contact it.
>> - Remove explicit linux-distros email.
>
>I don't like this because we had past trouble with notifications going
>to the distros@ list and leaking Linux-only flaws to the BSDs. As there
>isn't a separate linux-distros wiki, the clarification of WHICH list is
>needed.
Why would removing the explicit linux-distros email encourage people to
send reports to it?
I also don't understand what you mean by "there isn't a separate
linux-distros wiki"? There is one, and I want to point the reporter
there.
>> - Remove various explanations of linux-distros policies.
>
>I don't think there's value in removing the Tue-Thu comment, nor
>providing context for why distros need time. This has been a regular
>thing we've had to explain to researchers that aren't familiar with
>update procedures and publication timing.
To be fair, the Tue-Thu comment is listed in the section describing how
to do coordination with linux-distros, and linux-distros don't have a
Tue-Thu policy. If it's a security@kernel.org policy then let's list it
elsewhere.
If you feel that there is a consensus around Tue-Thu let's just add it
to the linux-distros policy wiki, there's no point in listing random
policies from that wiki.
--
Thanks,
Sasha
^ permalink raw reply
* Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger
From: Brendan Higgins @ 2019-07-19 0:08 UTC (permalink / raw)
To: Stephen Boyd
Cc: Frank Rowand, Greg KH, Josh Poimboeuf, Kees Cook, Kieran Bingham,
Luis Chamberlain, Peter Zijlstra, Rob Herring, shuah,
Theodore Ts'o, Masahiro Yamada, devicetree, dri-devel,
kunit-dev, open list:DOCUMENTATION, linux-fsdevel, linux-kbuild,
Linux Kernel Mailing List, open list:KERNEL SELFTEST FRAMEWORK,
linux-nvdimm, linux-um, Sasha Levin, Bird, Timothy,
Amir Goldstein, Dan Carpenter, Daniel Vetter, Jeff Dike,
Joel Stanley, Julia Lawall, Kevin Hilman, Knut Omang,
Logan Gunthorpe, Michael Ellerman, Petr Mladek, Randy Dunlap,
Richard Weinberger, David Rientjes, Steven Rostedt, wfg
In-Reply-To: <CAFd5g46a7C1+R6ZcE_SkqaYqgrH5Rx3M=X7orFyaMgFLDbeYYA@mail.gmail.com>
On Thu, Jul 18, 2019 at 12:22:33PM -0700, Brendan Higgins wrote:
> On Thu, Jul 18, 2019 at 10:50 AM Stephen Boyd <sboyd@kernel.org> wrote:
> >
> > Quoting Brendan Higgins (2019-07-16 11:52:01)
> > > On Tue, Jul 16, 2019 at 10:50 AM Stephen Boyd <sboyd@kernel.org> wrote:
[...]
> > Do you have a link to those earlier patches?
>
> This is the first patchset:
>
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788057.html
>
> In particular you can see the code for matching functions here:
>
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788073.html
>
> And parameter matching code here:
>
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788072.html
>
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788086.html
>
> My apologies in advance, but the code at this early stage had not
> adopted the kunit_* prefix and was still using the test_* and mock_*
> prefix. (Hence, struct kunit_stream was known as struct test_stream).
[...]
> > The crux of my complaint is that the string stream API is too loosely
> > defined to be usable. It allows tests to build up a string of
> > unstructured information, but with certain calling constraints so we
> > have to tread carefully. If there was more structure to the data that's
> > being recorded then the test case runner could operate on the data
> > without having to do string/stream operations, allocations, etc. This
> > would make the assertion logic much more concrete and specific to kunit,
> > instead of this small kunit wrapper that's been placed on top of string
> > stream.
>
> Yeah, I can see the point of wanting something that provides more
> structure than the raw `struct kunit_stream` interface. In fact, it is
> something I had already started working on, when I had determined it
> would be a large effort to capture all the variations. I was further
> put off from the idea when I had been asked to convert the KUnit
> intermediate format from what I was using to TAP, because, as it is,
> the current data printed out by KUnit doesn't contain all the data I
> would like to put in it in a way that best takes advantage of the TAP
> specification. One problematic area in particular: TAP already
> provides a way to present a lot of the data I would like to export,
> but it involves JSON serialization which was an idea that some of the
> other reviewers understandably weren't too keen on. TAP also wants to
> report data some time after it is available, which is generally not a
> good idea for test debug information; you want to make it available as
> soon as you can or you risk crashing with the data still inside.
>
> Hence, I decided we could probably spend a good long while debating
> how I present the information. So the idea of having a loose
> definition seemed attractive to me in its own right since it would
> likely conform to whatever we ended up deciding in the long run. Also,
> all the better that it was what I already had and no one seemed to
> mind too much.
>
> The only constant I expect is that `struct kunit` will likely need to
> take an abstract object with a `commit` method, or a `format` method
> or whatever so it could control when data was going to be printed out
> to the user. We will probably also use a string builder in there
> somewhere.
>
> > TL;DR: If we can get rid of the string stream API I'd view that as an
> > improvement because building arbitrary strings in the kernel is complex,
> > error prone and has calling context concerns.
>
> True. No argument there.
>
> > Is the intention that other code besides unit tests will use this string
> > stream API to build up strings? Any targets in mind? This would be a
> > good way to get the API merged upstream given that its 2019 and we
> > haven't had such an API in the kernel so far.
>
> Someone, (was it you?) asked about code sharing with a string builder
> thingy that was used for creating structured human readable files, but
> that seemed like a pretty massive undertaking.
>
> Aside from that, no. I would kind of prefered that nobody used it for
> anything else because I the issues you described.
>
> Nevertheless, I think the debate over the usefulness of the
> string_stream and kunit_stream are separate topics. Even if we made
> kunit_stream more structured, I am pretty sure I would want to use
> string_stream or some variation for constructing the message.
>
> > An "object oriented" (strong quotes!) approach where kunit_fail_msg is
> > the innermost struct in some assertion specific structure might work
> > nicely and allow the test runner to call a generic 'format' function to
> > print out the message based on the type of assertion/expectation it is.
> > It probably would mean less code size too because the strings that are
> > common will be in the common printing function instead of created twice,
> > in the macros/code and then copied to the heap for the string stream.
> >
> > struct kunit_assert {
> > const char *line;
> > const char *file;
> > const char *func;
> > void (*format)(struct kunit_assert *assert);
> > };
> >
> > struct kunit_comparison_assert {
> > enum operator operator;
> > const char *left;
> > const char *right;
> > struct kunit_assert assert;
> > };
> >
> > struct kunit_bool_assert {
> > const char *truth;
> > const char *statement;
> > struct kunit_assert assert;
> > };
> >
> > void kunit_format_comparison(struct kunit_assert *assert)
> > {
> > struct kunit_comparison_assert *comp = container_of(assert, ...)
> >
> > kunit_printk(...)
> > }
I started poking around with your suggestion while we are waiting. A
couple early observations:
1) It is actually easier to do than I previously thought and will probably
help with getting more of the planned TAP output stuff working.
That being said, this is still a pretty substantial undertaking and
will likely take *at least* a week to implement and properly review.
Assuming everything goes extremely well (no unexpected issues on my
end, very responsive reviewers, etc).
2) It *will* eliminate the need for kunit_stream.
3) ...but, it *will not* eliminate the need for string_stream.
Based on my early observations, I do think it is worth doing, but I
don't think it is worth trying to make it in this patchset (unless I
have already missed the window, or it is going to be open for a while):
I do think it will make things much cleaner, but I don't think it will
achieve your desired goal of getting rid of an unstructured
{kunit|string}_stream style interface; it just adds a layer on top of it
that makes it harder to misuse.
I attached a patch of what I have so far at the end of this email so you
can see what I am talking about. And of course, if you agree with my
assessment, so we can start working on it as a future patch.
A couple things in regard to the patch I attached:
1) I wrote it pretty quickly so there are almost definitely mistakes.
You should consider it RFC. I did verify it compiles though.
2) Also, I did use kunit_stream in writing it: all occurences should be
pretty easy to replace with string_stream; nevertheless, the reason
for this is just to make it easier to play with the current APIs. I
wanted to have something working before I went through a big tedious
refactoring. So sorry if it causes any confusion.
3) I also based the patch on all the KUnit patches I have queued up
(includes things like mocking and such) since I want to see how this
serialization thing will work with mocks and matchers and things like
that.
> I started working on something similarish, but by the time I ended up
> coming up with a parent object whose definition was loose enough to
> satisfy all the properties required by the child classes it ended up
> basically being the same as what I have now just with a more complex
> hierarchy of message manipulation logic.
>
> On the other hand, I didn't have the idea of doing the parent object
> quite the way you did and that would clean up a lot of the duplicated
> first line logic.
>
> I would like to give it a try, but I am afraid I am going to get
> sucked down a really deep rabbit hole.
>
> > Maybe other people have opinions here on if you should do it now or
> > later. Future coding is not a great argument because it's hard to
> > predict the future. On the other hand, this patchset is in good shape to
>
> Yeah, that's kind of why I am afraid to go down this road when I have
> something that works now and I know works with the mocking stuff I
> want to do.
>
> I would like to try your suggestion, but I want to try to make it work
> with my mocking patches before I commit to it because otherwise I am
> just going to have to back it out in a follow up patchset.
>
> > merge and I'd like to use it to write unit tests for code I maintain so
> > I don't want to see this stall out. Sorry if I'm opening the can of
> > worms you're talking about.
>
> Don't be sorry. I agree with you that the kunit_stream stuff is not very pretty.
>
> Shuah, have we missed the merge window for 5.3?
>
> I saw you only sent one PR out so far for this release, and there
> wasn't much in it; I imagine you are going to send at least one more?
>
> I figure, if we still got time to try out your suggestion, Stephen, no
> harm in trying.
>
> Also if we missed it, then I have another couple months to play around with it.
>
> What do you think?
I attached the patch mentioned above below. Let me know what you think!
Cheers!
From 53d475d3d56afcf92b452c6d347dbedfa1a17d34 Mon Sep 17 00:00:00 2001
From: Brendan Higgins <brendanhiggins@google.com>
Date: Thu, 18 Jul 2019 16:08:52 -0700
Subject: [PATCH v1] DO NOT MERGE: started playing around with the
serialization api
---
include/kunit/assert.h | 130 ++++++++++++++++++++++++++++++
include/kunit/mock.h | 4 +
kunit/Makefile | 3 +-
kunit/assert.c | 179 +++++++++++++++++++++++++++++++++++++++++
kunit/mock.c | 6 +-
5 files changed, 318 insertions(+), 4 deletions(-)
create mode 100644 include/kunit/assert.h
create mode 100644 kunit/assert.c
diff --git a/include/kunit/assert.h b/include/kunit/assert.h
new file mode 100644
index 0000000000000..e054fdff4642f
--- /dev/null
+++ b/include/kunit/assert.h
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <brendanhiggins@google.com>
+ */
+
+#ifndef _KUNIT_ASSERT_H
+#define _KUNIT_ASSERT_H
+
+#include <kunit/test.h>
+#include <kunit/mock.h>
+
+enum kunit_assert_type {
+ KUNIT_ASSERTION,
+ KUNIT_EXPECTATION,
+};
+
+struct kunit_assert {
+ enum kunit_assert_type type;
+ const char *line;
+ const char *file;
+ struct va_format message;
+ void (*format)(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+};
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_unary_assert {
+ struct kunit_assert assert;
+ const char *condition;
+ bool expected_true;
+};
+
+void kunit_unary_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_ptr_not_err_assert {
+ struct kunit_assert assert;
+ const char *text;
+ void *value;
+};
+
+void kunit_ptr_not_err_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_binary_assert {
+ struct kunit_assert assert;
+ const char *operation;
+ const char *left_text;
+ long long left_value;
+ const char *right_text;
+ long long right_value;
+};
+
+void kunit_binary_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_binary_ptr_assert {
+ struct kunit_assert assert;
+ const char *operation;
+ const char *left_text;
+ void *left_value;
+ const char *right_text;
+ void *right_value;
+};
+
+void kunit_binary_ptr_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_binary_str_assert {
+ struct kunit_assert assert;
+ const char *operation;
+ const char *left_text;
+ const char *left_value;
+ const char *right_text;
+ const char *right_value;
+};
+
+void kunit_binary_str_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_mock_assert {
+ struct kunit_assert assert;
+};
+
+struct kunit_mock_no_expectations {
+ struct kunit_mock_assert assert;
+};
+
+struct kunit_mock_declaration {
+ const char *function_name;
+ const char **type_names;
+ const void **params;
+ int len;
+};
+
+void kunit_mock_declaration_format(struct kunit_mock_declaration *declaration,
+ struct kunit_stream *stream);
+
+struct kunit_matcher_result {
+ struct kunit_assert assert;
+};
+
+struct kunit_mock_failed_match {
+ struct list_head node;
+ const char *expectation_text;
+ struct kunit_matcher_result *matcher_list;
+ size_t matcher_list_len;
+};
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+ struct kunit_stream *stream);
+
+struct kunit_mock_no_match {
+ struct kunit_mock_assert assert;
+ struct kunit_mock_declaration declaration;
+ struct list_head failed_match_list;
+};
+
+void kunit_mock_no_match_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+#endif /* _KUNIT_ASSERT_H */
diff --git a/include/kunit/mock.h b/include/kunit/mock.h
index 001b96af62f1e..52c9e427c831b 100644
--- a/include/kunit/mock.h
+++ b/include/kunit/mock.h
@@ -144,6 +144,10 @@ void mock_register_formatter(struct mock_param_formatter *formatter);
void mock_unregister_formatter(struct mock_param_formatter *formatter);
+void mock_format_param(struct kunit_stream *stream,
+ const char *type_name,
+ const void *param);
+
struct mock *mock_get_global_mock(void);
#define MOCK(name) name##_mock
diff --git a/kunit/Makefile b/kunit/Makefile
index bbf43fcfb93a9..149d856a30f04 100644
--- a/kunit/Makefile
+++ b/kunit/Makefile
@@ -3,7 +3,8 @@ obj-$(CONFIG_KUNIT) += test.o \
common-mocks.o \
string-stream.o \
kunit-stream.o \
- try-catch.o
+ try-catch.o \
+ assert.o
obj-$(CONFIG_KUNIT_TEST) += test-test.o \
test-mock.o \
diff --git a/kunit/assert.c b/kunit/assert.c
new file mode 100644
index 0000000000000..75bb6922a994e
--- /dev/null
+++ b/kunit/assert.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <brendanhiggins@google.com>
+ */
+#include <kunit/assert.h>
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ const char *expect_or_assert;
+
+ if (assert->type == KUNIT_EXPECTATION)
+ expect_or_assert = "EXPECTATION";
+ else
+ expect_or_assert = "ASSERTION";
+
+ kunit_stream_add(stream, "%s FAILED at %s:%s\n",
+ expect_or_assert, assert->file, assert->line);
+}
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ if (assert->message.fmt)
+ kunit_stream_add(stream, "\n%pV", &assert->message);
+}
+
+void kunit_unary_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_unary_assert *unary_assert = container_of(
+ assert, struct kunit_unary_assert, assert);
+
+ kunit_base_assert_format(assert, stream);
+ if (unary_assert->expected_true)
+ kunit_stream_add(stream,
+ "\tExpected %s to be true, but is false\n",
+ unary_assert->condition);
+ else
+ kunit_stream_add(stream,
+ "\tExpected %s to be false, but is true\n",
+ unary_assert->condition);
+ kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_ptr_not_err_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_ptr_not_err_assert *ptr_assert = container_of(
+ assert, struct kunit_ptr_not_err_assert, assert);
+
+ kunit_base_assert_format(assert, stream);
+ if (!ptr_assert->value) {
+ kunit_stream_add(stream,
+ "\tExpected %s is not null, but is\n",
+ ptr_assert->text);
+ } else if (IS_ERR(ptr_assert->value)) {
+ kunit_stream_add(stream,
+ "\tExpected %s is not error, but is: %ld\n",
+ ptr_assert->text,
+ PTR_ERR(ptr_assert->value));
+ }
+ kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_binary_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_binary_assert *binary_assert = container_of(
+ assert, struct kunit_binary_assert, assert);
+
+ kunit_base_assert_format(assert, stream);
+ kunit_stream_add(stream,
+ "\tExpected %s %s %s, but\n",
+ binary_assert->left_text,
+ binary_assert->operation,
+ binary_assert->right_text);
+ kunit_stream_add(stream, "\t\t%s == %lld\n",
+ binary_assert->left_text,
+ binary_assert->left_value);
+ kunit_stream_add(stream, "\t\t%s == %lld",
+ binary_assert->right_text,
+ binary_assert->right_value);
+ kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_binary_ptr_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_binary_ptr_assert *binary_assert = container_of(
+ assert, struct kunit_binary_ptr_assert, assert);
+
+ kunit_base_assert_format(assert, stream);
+ kunit_stream_add(stream,
+ "\tExpected %s %s %s, but\n",
+ binary_assert->left_text,
+ binary_assert->operation,
+ binary_assert->right_text);
+ kunit_stream_add(stream, "\t\t%s == %pK\n",
+ binary_assert->left_text,
+ binary_assert->left_value);
+ kunit_stream_add(stream, "\t\t%s == %pK",
+ binary_assert->right_text,
+ binary_assert->right_value);
+ kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_binary_str_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_binary_str_assert *binary_assert = container_of(
+ assert, struct kunit_binary_str_assert, assert);
+
+ kunit_base_assert_format(assert, stream);
+ kunit_stream_add(stream,
+ "\tExpected %s %s %s, but\n",
+ binary_assert->left_text,
+ binary_assert->operation,
+ binary_assert->right_text);
+ kunit_stream_add(stream, "\t\t%s == %s\n",
+ binary_assert->left_text,
+ binary_assert->left_value);
+ kunit_stream_add(stream, "\t\t%s == %s",
+ binary_assert->right_text,
+ binary_assert->right_value);
+ kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_mock_declaration_format(struct kunit_mock_declaration *declaration,
+ struct kunit_stream *stream)
+{
+ int i;
+
+ kunit_stream_add(stream, "%s(", declaration->function_name);
+ for (i = 0; i < declaration->len; i++) {
+ mock_format_param(stream,
+ declaration->type_names[i],
+ declaration->params[i]);
+ if (i < declaration->len - 1)
+ kunit_stream_add(stream, ", ");
+ }
+ kunit_stream_add(stream, ")\n");
+}
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+ struct kunit_stream *stream)
+{
+ struct kunit_matcher_result *result;
+ size_t i;
+
+ kunit_stream_add(stream,
+ "Tried expectation: %s, but\n",
+ match->expectation_text);
+ for (i = 0; i < match->matcher_list_len; i++) {
+ result = &match->matcher_list[i];
+ kunit_stream_add(stream, "\t");
+ result->assert.format(&result->assert, stream);
+ kunit_stream_add(stream, "\n");
+ }
+}
+
+void kunit_mock_no_match_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_mock_assert *mock_assert = container_of(
+ assert, struct kunit_mock_assert, assert);
+ struct kunit_mock_no_match *no_match = container_of(
+ mock_assert, struct kunit_mock_no_match, assert);
+ struct kunit_mock_failed_match *expectation;
+
+ kunit_base_assert_format(assert, stream);
+ kunit_mock_declaration_format(&no_match->declaration, stream);
+
+ list_for_each_entry(expectation, &no_match->failed_match_list, node)
+ kunit_mock_failed_match_format(expectation, stream);
+}
diff --git a/kunit/mock.c b/kunit/mock.c
index ccb0abe111402..ab441a58a918c 100644
--- a/kunit/mock.c
+++ b/kunit/mock.c
@@ -269,9 +269,9 @@ struct mock_param_formatter *mock_find_formatter(const char *type_name)
return NULL;
}
-static void mock_format_param(struct kunit_stream *stream,
- const char *type_name,
- const void *param)
+void mock_format_param(struct kunit_stream *stream,
+ const char *type_name,
+ const void *param)
{
struct mock_param_formatter *formatter;
--
2.22.0.657.g960e92d24f-goog
^ permalink raw reply related
* [PATCH v3 2/2] f2fs: Support case-insensitive file name lookups
From: Daniel Rosenberg @ 2019-07-19 0:03 UTC (permalink / raw)
To: Jaegeuk Kim, Chao Yu, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team,
Daniel Rosenberg
In-Reply-To: <20190719000322.106163-1-drosen@google.com>
Modeled after commit b886ee3e778e ("ext4: Support case-insensitive file
name lookups")
"""
This patch implements the actual support for case-insensitive file name
lookups in f2fs, based on the feature bit and the encoding stored in the
superblock.
A filesystem that has the casefold feature set is able to configure
directories with the +F (F2FS_CASEFOLD_FL) attribute, enabling lookups
to succeed in that directory in a case-insensitive fashion, i.e: match
a directory entry even if the name used by userspace is not a byte per
byte match with the disk name, but is an equivalent case-insensitive
version of the Unicode string. This operation is called a
case-insensitive file name lookup.
The feature is configured as an inode attribute applied to directories
and inherited by its children. This attribute can only be enabled on
empty directories for filesystems that support the encoding feature,
thus preventing collision of file names that only differ by case.
* dcache handling:
For a +F directory, F2Fs only stores the first equivalent name dentry
used in the dcache. This is done to prevent unintentional duplication of
dentries in the dcache, while also allowing the VFS code to quickly find
the right entry in the cache despite which equivalent string was used in
a previous lookup, without having to resort to ->lookup().
d_hash() of casefolded directories is implemented as the hash of the
casefolded string, such that we always have a well-known bucket for all
the equivalencies of the same string. d_compare() uses the
utf8_strncasecmp() infrastructure, which handles the comparison of
equivalent, same case, names as well.
For now, negative lookups are not inserted in the dcache, since they
would need to be invalidated anyway, because we can't trust missing file
dentries. This is bad for performance but requires some leveraging of
the vfs layer to fix. We can live without that for now, and so does
everyone else.
* on-disk data:
Despite using a specific version of the name as the internal
representation within the dcache, the name stored and fetched from the
disk is a byte-per-byte match with what the user requested, making this
implementation 'name-preserving'. i.e. no actual information is lost
when writing to storage.
DX is supported by modifying the hashes used in +F directories to make
them case/encoding-aware. The new disk hashes are calculated as the
hash of the full casefolded string, instead of the string directly.
This allows us to efficiently search for file names in the htree without
requiring the user to provide an exact name.
* Dealing with invalid sequences:
By default, when a invalid UTF-8 sequence is identified, ext4 will treat
it as an opaque byte sequence, ignoring the encoding and reverting to
the old behavior for that unique file. This means that case-insensitive
file name lookup will not work only for that file. An optional bit can
be set in the superblock telling the filesystem code and userspace tools
to enforce the encoding. When that optional bit is set, any attempt to
create a file name using an invalid UTF-8 sequence will fail and return
an error to userspace.
* Normalization algorithm:
The UTF-8 algorithms used to compare strings in f2fs is implemented
in fs/unicode, and is based on a previous version developed by
SGI. It implements the Canonical decomposition (NFD) algorithm
described by the Unicode specification 12.1, or higher, combined with
the elimination of ignorable code points (NFDi) and full
case-folding (CF) as documented in fs/unicode/utf8_norm.c.
NFD seems to be the best normalization method for F2FS because:
- It has a lower cost than NFC/NFKC (which requires
decomposing to NFD as an intermediary step)
- It doesn't eliminate important semantic meaning like
compatibility decompositions.
Although:
- This implementation is not completely linguistic accurate, because
different languages have conflicting rules, which would require the
specialization of the filesystem to a given locale, which brings all
sorts of problems for removable media and for users who use more than
one language.
"""
Signed-off-by: Daniel Rosenberg <drosen@google.com>
---
fs/f2fs/dir.c | 126 +++++++++++++++++++++++++++++++++++++++++++----
fs/f2fs/f2fs.h | 15 ++++--
fs/f2fs/file.c | 9 ++++
fs/f2fs/hash.c | 35 ++++++++++++-
fs/f2fs/inline.c | 4 +-
fs/f2fs/inode.c | 4 +-
fs/f2fs/namei.c | 21 ++++++++
fs/f2fs/super.c | 6 +++
8 files changed, 203 insertions(+), 17 deletions(-)
diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c
index 85a1528f319f2..2913483473f30 100644
--- a/fs/f2fs/dir.c
+++ b/fs/f2fs/dir.c
@@ -8,6 +8,7 @@
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
#include <linux/sched/signal.h>
+#include <linux/unicode.h>
#include "f2fs.h"
#include "node.h"
#include "acl.h"
@@ -81,7 +82,8 @@ static unsigned long dir_block_index(unsigned int level,
return bidx;
}
-static struct f2fs_dir_entry *find_in_block(struct page *dentry_page,
+static struct f2fs_dir_entry *find_in_block(struct inode *dir,
+ struct page *dentry_page,
struct fscrypt_name *fname,
f2fs_hash_t namehash,
int *max_slots,
@@ -93,7 +95,7 @@ static struct f2fs_dir_entry *find_in_block(struct page *dentry_page,
dentry_blk = (struct f2fs_dentry_block *)page_address(dentry_page);
- make_dentry_ptr_block(NULL, &d, dentry_blk);
+ make_dentry_ptr_block(dir, &d, dentry_blk);
de = f2fs_find_target_dentry(fname, namehash, max_slots, &d);
if (de)
*res_page = dentry_page;
@@ -101,6 +103,39 @@ static struct f2fs_dir_entry *find_in_block(struct page *dentry_page,
return de;
}
+#ifdef CONFIG_UNICODE
+/*
+ * Test whether a case-insensitive directory entry matches the filename
+ * being searched for.
+ *
+ * Returns: 0 if the directory entry matches, more than 0 if it
+ * doesn't match or less than zero on error.
+ */
+int f2fs_ci_compare(const struct inode *parent, const struct qstr *name,
+ const struct qstr *entry)
+{
+ const struct f2fs_sb_info *sbi = F2FS_SB(parent->i_sb);
+ const struct unicode_map *um = sbi->s_encoding;
+ int ret;
+
+ ret = utf8_strncasecmp(um, name, entry);
+ if (ret < 0) {
+ /* Handle invalid character sequence as either an error
+ * or as an opaque byte sequence.
+ */
+ if (f2fs_has_strict_mode(sbi))
+ return -EINVAL;
+
+ if (name->len != entry->len)
+ return 1;
+
+ return !!memcmp(name->name, entry->name, name->len);
+ }
+
+ return ret;
+}
+#endif
+
struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
f2fs_hash_t namehash, int *max_slots,
struct f2fs_dentry_ptr *d)
@@ -108,6 +143,9 @@ struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
struct f2fs_dir_entry *de;
unsigned long bit_pos = 0;
int max_len = 0;
+#ifdef CONFIG_UNICODE
+ struct qstr entry;
+#endif
if (max_slots)
*max_slots = 0;
@@ -119,16 +157,28 @@ struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
}
de = &d->dentry[bit_pos];
+#ifdef CONFIG_UNICODE
+ entry.name = d->filename[bit_pos];
+ entry.len = de->name_len;
+#endif
if (unlikely(!de->name_len)) {
bit_pos++;
continue;
}
+ if (de->hash_code == namehash) {
+#ifdef CONFIG_UNICODE
+ if (F2FS_SB(d->inode->i_sb)->s_encoding &&
+ IS_CASEFOLDED(d->inode) &&
+ !f2fs_ci_compare(d->inode,
+ fname->usr_fname, &entry))
+ goto found;
- if (de->hash_code == namehash &&
- fscrypt_match_name(fname, d->filename[bit_pos],
- le16_to_cpu(de->name_len)))
- goto found;
+#endif
+ if (fscrypt_match_name(fname, d->filename[bit_pos],
+ le16_to_cpu(de->name_len)))
+ goto found;
+ }
if (max_slots && max_len > *max_slots)
*max_slots = max_len;
@@ -157,7 +207,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
struct f2fs_dir_entry *de = NULL;
bool room = false;
int max_slots;
- f2fs_hash_t namehash = f2fs_dentry_hash(&name, fname);
+ f2fs_hash_t namehash = f2fs_dentry_hash(dir, &name, fname);
nbucket = dir_buckets(level, F2FS_I(dir)->i_dir_level);
nblock = bucket_blocks(level);
@@ -179,8 +229,8 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
}
}
- de = find_in_block(dentry_page, fname, namehash, &max_slots,
- res_page);
+ de = find_in_block(dir, dentry_page, fname, namehash,
+ &max_slots, res_page);
if (de)
break;
@@ -250,6 +300,14 @@ struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir,
struct fscrypt_name fname;
int err;
+#ifdef CONFIG_UNICODE
+ if (f2fs_has_strict_mode(F2FS_I_SB(dir)) && IS_CASEFOLDED(dir) &&
+ utf8_validate(F2FS_I_SB(dir)->s_encoding, child)) {
+ *res_page = ERR_PTR(-EINVAL);
+ return NULL;
+ }
+#endif
+
err = fscrypt_setup_filename(dir, child, 1, &fname);
if (err) {
if (err == -ENOENT)
@@ -504,7 +562,7 @@ int f2fs_add_regular_entry(struct inode *dir, const struct qstr *new_name,
level = 0;
slots = GET_DENTRY_SLOTS(new_name->len);
- dentry_hash = f2fs_dentry_hash(new_name, NULL);
+ dentry_hash = f2fs_dentry_hash(dir, new_name, NULL);
current_depth = F2FS_I(dir)->i_current_depth;
if (F2FS_I(dir)->chash == dentry_hash) {
@@ -943,3 +1001,51 @@ const struct file_operations f2fs_dir_operations = {
.compat_ioctl = f2fs_compat_ioctl,
#endif
};
+
+#ifdef CONFIG_UNICODE
+static int f2fs_d_compare(const struct dentry *dentry, unsigned int len,
+ const char *str, const struct qstr *name)
+{
+ struct qstr qstr = {.name = str, .len = len };
+
+ if (!IS_CASEFOLDED(dentry->d_parent->d_inode)) {
+ if (len != name->len)
+ return -1;
+ return memcmp(str, name, len);
+ }
+
+ return f2fs_ci_compare(dentry->d_parent->d_inode, name, &qstr);
+}
+
+static int f2fs_d_hash(const struct dentry *dentry, struct qstr *str)
+{
+ struct f2fs_sb_info *sbi = F2FS_SB(dentry->d_sb);
+ const struct unicode_map *um = sbi->s_encoding;
+ unsigned char *norm;
+ int len, ret = 0;
+
+ if (!IS_CASEFOLDED(dentry->d_inode))
+ return 0;
+
+ norm = f2fs_kmalloc(sbi, PATH_MAX, GFP_ATOMIC);
+ if (!norm)
+ return -ENOMEM;
+
+ len = utf8_casefold(um, str, norm, PATH_MAX);
+ if (len < 0) {
+ if (f2fs_has_strict_mode(sbi))
+ ret = -EINVAL;
+ goto out;
+ }
+ str->hash = full_name_hash(dentry, norm, len);
+out:
+ kvfree(norm);
+ return ret;
+}
+
+const struct dentry_operations f2fs_dentry_ops = {
+ .d_hash = f2fs_d_hash,
+ .d_compare = f2fs_d_compare,
+};
+#endif
+
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index c6c7904572d0d..31fd2a268ba14 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -2364,10 +2364,12 @@ static inline void f2fs_change_bit(unsigned int nr, char *addr)
#define F2FS_INDEX_FL 0x00001000 /* hash-indexed directory */
#define F2FS_DIRSYNC_FL 0x00010000 /* dirsync behaviour (directories only) */
#define F2FS_PROJINHERIT_FL 0x20000000 /* Create with parents projid */
+#define F2FS_CASEFOLD_FL 0x40000000 /* Casefolded file */
/* Flags that should be inherited by new inodes from their parent. */
#define F2FS_FL_INHERITED (F2FS_SYNC_FL | F2FS_NODUMP_FL | F2FS_NOATIME_FL | \
- F2FS_DIRSYNC_FL | F2FS_PROJINHERIT_FL)
+ F2FS_DIRSYNC_FL | F2FS_PROJINHERIT_FL | \
+ F2FS_CASEFOLD_FL)
/* Flags that are appropriate for regular files (all but dir-specific ones). */
#define F2FS_REG_FLMASK (~(F2FS_DIRSYNC_FL | F2FS_PROJINHERIT_FL))
@@ -2930,6 +2932,10 @@ int f2fs_update_extension_list(struct f2fs_sb_info *sbi, const char *name,
bool hot, bool set);
struct dentry *f2fs_get_parent(struct dentry *child);
+extern int f2fs_ci_compare(const struct inode *parent,
+ const struct qstr *name,
+ const struct qstr *entry);
+
/*
* dir.c
*/
@@ -2993,8 +2999,8 @@ int f2fs_sanity_check_ckpt(struct f2fs_sb_info *sbi);
/*
* hash.c
*/
-f2fs_hash_t f2fs_dentry_hash(const struct qstr *name_info,
- struct fscrypt_name *fname);
+f2fs_hash_t f2fs_dentry_hash(const struct inode *dir,
+ const struct qstr *name_info, struct fscrypt_name *fname);
/*
* node.c
@@ -3437,6 +3443,9 @@ static inline void f2fs_destroy_root_stats(void) { }
#endif
extern const struct file_operations f2fs_dir_operations;
+#ifdef CONFIG_UNICODE
+extern const struct dentry_operations f2fs_dentry_ops;
+#endif
extern const struct file_operations f2fs_file_operations;
extern const struct inode_operations f2fs_file_inode_operations;
extern const struct address_space_operations f2fs_dblock_aops;
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index f8d46df8fa9ee..7adef2d8dbc47 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -1660,7 +1660,16 @@ static int f2fs_setflags_common(struct inode *inode, u32 iflags, u32 mask)
return -EPERM;
oldflags = fi->i_flags;
+ if ((iflags ^ oldflags) & F2FS_CASEFOLD_FL) {
+ if (!f2fs_sb_has_casefold(F2FS_I_SB(inode)))
+ return -EOPNOTSUPP;
+
+ if (!S_ISDIR(inode->i_mode))
+ return -ENOTDIR;
+ if (!f2fs_empty_dir(inode))
+ return -ENOTEMPTY;
+ }
if ((iflags ^ oldflags) & (F2FS_APPEND_FL | F2FS_IMMUTABLE_FL))
if (!capable(CAP_LINUX_IMMUTABLE))
return -EPERM;
diff --git a/fs/f2fs/hash.c b/fs/f2fs/hash.c
index cc82f142f811f..b7bd0ddbbdf01 100644
--- a/fs/f2fs/hash.c
+++ b/fs/f2fs/hash.c
@@ -14,6 +14,7 @@
#include <linux/f2fs_fs.h>
#include <linux/cryptohash.h>
#include <linux/pagemap.h>
+#include <linux/unicode.h>
#include "f2fs.h"
@@ -67,7 +68,7 @@ static void str2hashbuf(const unsigned char *msg, size_t len,
*buf++ = pad;
}
-f2fs_hash_t f2fs_dentry_hash(const struct qstr *name_info,
+static f2fs_hash_t __f2fs_dentry_hash(const struct qstr *name_info,
struct fscrypt_name *fname)
{
__u32 hash;
@@ -103,3 +104,35 @@ f2fs_hash_t f2fs_dentry_hash(const struct qstr *name_info,
f2fs_hash = cpu_to_le32(hash & ~F2FS_HASH_COL_BIT);
return f2fs_hash;
}
+
+f2fs_hash_t f2fs_dentry_hash(const struct inode *dir,
+ const struct qstr *name_info, struct fscrypt_name *fname)
+{
+#ifdef CONFIG_UNICODE
+ struct f2fs_sb_info *sbi = F2FS_SB(dir->i_sb);
+ const struct unicode_map *um = sbi->s_encoding;
+ int r, dlen;
+ unsigned char *buff;
+ struct qstr *folded;
+
+ if (name_info->len && IS_CASEFOLDED(dir)) {
+ buff = f2fs_kzalloc(sbi, sizeof(char) * PATH_MAX, GFP_KERNEL);
+ if (!buff)
+ return -ENOMEM;
+
+ dlen = utf8_casefold(um, name_info, buff, PATH_MAX);
+ if (dlen < 0) {
+ kfree(buff);
+ goto opaque_seq;
+ }
+ folded->name = buff;
+ folded->len = dlen;
+ r = __f2fs_dentry_hash(folded, fname);
+
+ kvfree(buff);
+ return r;
+ }
+opaque_seq:
+#endif
+ return __f2fs_dentry_hash(name_info, fname);
+}
diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c
index 3613efca8c00c..354f71cf9e6ba 100644
--- a/fs/f2fs/inline.c
+++ b/fs/f2fs/inline.c
@@ -320,7 +320,7 @@ struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir,
return NULL;
}
- namehash = f2fs_dentry_hash(&name, fname);
+ namehash = f2fs_dentry_hash(dir, &name, fname);
inline_dentry = inline_data_addr(dir, ipage);
@@ -580,7 +580,7 @@ int f2fs_add_inline_entry(struct inode *dir, const struct qstr *new_name,
f2fs_wait_on_page_writeback(ipage, NODE, true, true);
- name_hash = f2fs_dentry_hash(new_name, NULL);
+ name_hash = f2fs_dentry_hash(dir, new_name, NULL);
f2fs_update_dentry(ino, mode, &d, new_name, name_hash, bit_pos);
set_page_dirty(ipage);
diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c
index a33d7a849b2df..9a1f0d6616577 100644
--- a/fs/f2fs/inode.c
+++ b/fs/f2fs/inode.c
@@ -46,9 +46,11 @@ void f2fs_set_inode_flags(struct inode *inode)
new_fl |= S_DIRSYNC;
if (file_is_encrypt(inode))
new_fl |= S_ENCRYPTED;
+ if (flags & F2FS_CASEFOLD_FL)
+ new_fl |= S_CASEFOLD;
inode_set_flags(inode, new_fl,
S_SYNC|S_APPEND|S_IMMUTABLE|S_NOATIME|S_DIRSYNC|
- S_ENCRYPTED);
+ S_ENCRYPTED|S_CASEFOLD);
}
static void __get_inode_rdev(struct inode *inode, struct f2fs_inode *ri)
diff --git a/fs/f2fs/namei.c b/fs/f2fs/namei.c
index c5b99042e6f2b..727de2f8620f2 100644
--- a/fs/f2fs/namei.c
+++ b/fs/f2fs/namei.c
@@ -489,6 +489,17 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry,
goto out_iput;
}
out_splice:
+#ifdef CONFIG_UNICODE
+ if (!inode && IS_CASEFOLDED(dir)) {
+ /* Eventually we want to call d_add_ci(dentry, NULL)
+ * for negative dentries in the encoding case as
+ * well. For now, prevent the negative dentry
+ * from being cached.
+ */
+ trace_f2fs_lookup_end(dir, dentry, ino, err);
+ return NULL;
+ }
+#endif
new = d_splice_alias(inode, dentry);
err = PTR_ERR_OR_ZERO(new);
trace_f2fs_lookup_end(dir, dentry, ino, err);
@@ -537,6 +548,16 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry)
goto fail;
}
f2fs_delete_entry(de, page, dir, inode);
+#ifdef CONFIG_UNICODE
+ /* VFS negative dentries are incompatible with Encoding and
+ * Case-insensitiveness. Eventually we'll want avoid
+ * invalidating the dentries here, alongside with returning the
+ * negative dentries at f2fs_lookup(), when it is better
+ * supported by the VFS for the CI case.
+ */
+ if (IS_CASEFOLDED(dir))
+ d_invalidate(dentry);
+#endif
f2fs_unlock_op(sbi);
if (IS_DIRSYNC(dir))
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 82f7da93c3ed1..9c522d1abcb6d 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -3115,6 +3115,7 @@ static int f2fs_setup_casefold(struct f2fs_sb_info *sbi)
return -EINVAL;
}
#endif
+ return 0;
}
static void f2fs_tuning_parameters(struct f2fs_sb_info *sbi)
@@ -3410,6 +3411,11 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
goto free_node_inode;
}
+#ifdef CONFIG_UNICODE
+ if (sbi->s_encoding)
+ sb->s_d_op = &f2fs_dentry_ops;
+#endif
+
sb->s_root = d_make_root(root); /* allocate root dentry */
if (!sb->s_root) {
err = -ENOMEM;
--
2.22.0.657.g960e92d24f-goog
^ permalink raw reply related
* [PATCH v3 1/2] f2fs: include charset encoding information in the superblock
From: Daniel Rosenberg @ 2019-07-19 0:03 UTC (permalink / raw)
To: Jaegeuk Kim, Chao Yu, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team,
Daniel Rosenberg
In-Reply-To: <20190719000322.106163-1-drosen@google.com>
Add charset encoding to f2fs to support casefolding. It is modeled after
the same feature introduced in commit c83ad55eaa91 ("ext4: include charset
encoding information in the superblock")
Currently this is not compatible with encryption, similar to the current
ext4 imlpementation. This will change in the future.
From the ext4 patch:
"""
The s_encoding field stores a magic number indicating the encoding
format and version used globally by file and directory names in the
filesystem. The s_encoding_flags defines policies for using the charset
encoding, like how to handle invalid sequences. The magic number is
mapped to the exact charset table, but the mapping is specific to ext4.
Since we don't have any commitment to support old encodings, the only
encoding I am supporting right now is utf8-12.1.0.
The current implementation prevents the user from enabling encoding and
per-directory encryption on the same filesystem at the same time. The
incompatibility between these features lies in how we do efficient
directory searches when we cannot be sure the encryption of the user
provided fname will match the actual hash stored in the disk without
decrypting every directory entry, because of normalization cases. My
quickest solution is to simply block the concurrent use of these
features for now, and enable it later, once we have a better solution.
"""
Signed-off-by: Daniel Rosenberg <drosen@google.com>
---
fs/f2fs/f2fs.h | 6 +++
fs/f2fs/super.c | 94 +++++++++++++++++++++++++++++++++++++++++
fs/f2fs/sysfs.c | 23 ++++++++++
include/linux/f2fs_fs.h | 9 +++-
4 files changed, 131 insertions(+), 1 deletion(-)
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 17382da7f0bd9..c6c7904572d0d 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -153,6 +153,7 @@ struct f2fs_mount_info {
#define F2FS_FEATURE_LOST_FOUND 0x0200
#define F2FS_FEATURE_VERITY 0x0400 /* reserved */
#define F2FS_FEATURE_SB_CHKSUM 0x0800
+#define F2FS_FEATURE_CASEFOLD 0x1000
#define __F2FS_HAS_FEATURE(raw_super, mask) \
((raw_super->feature & cpu_to_le32(mask)) != 0)
@@ -1169,6 +1170,10 @@ struct f2fs_sb_info {
int valid_super_block; /* valid super block no */
unsigned long s_flag; /* flags for sbi */
struct mutex writepages; /* mutex for writepages() */
+#ifdef CONFIG_UNICODE
+ struct unicode_map *s_encoding;
+ __u16 s_encoding_flags;
+#endif
#ifdef CONFIG_BLK_DEV_ZONED
unsigned int blocks_per_blkz; /* F2FS blocks per zone */
@@ -3562,6 +3567,7 @@ F2FS_FEATURE_FUNCS(quota_ino, QUOTA_INO);
F2FS_FEATURE_FUNCS(inode_crtime, INODE_CRTIME);
F2FS_FEATURE_FUNCS(lost_found, LOST_FOUND);
F2FS_FEATURE_FUNCS(sb_chksum, SB_CHKSUM);
+F2FS_FEATURE_FUNCS(casefold, CASEFOLD);
#ifdef CONFIG_BLK_DEV_ZONED
static inline bool f2fs_blkz_is_seq(struct f2fs_sb_info *sbi, int devi,
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 6de6cda440315..82f7da93c3ed1 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -23,6 +23,7 @@
#include <linux/f2fs_fs.h>
#include <linux/sysfs.h>
#include <linux/quota.h>
+#include <linux/unicode.h>
#include "f2fs.h"
#include "node.h"
@@ -222,6 +223,36 @@ void f2fs_printk(struct f2fs_sb_info *sbi, const char *fmt, ...)
va_end(args);
}
+#ifdef CONFIG_UNICODE
+static const struct f2fs_sb_encodings {
+ __u16 magic;
+ char *name;
+ char *version;
+} f2fs_sb_encoding_map[] = {
+ {F2FS_ENC_UTF8_12_1, "utf8", "12.1.0"},
+};
+
+static int f2fs_sb_read_encoding(const struct f2fs_super_block *sb,
+ const struct f2fs_sb_encodings **encoding,
+ __u16 *flags)
+{
+ __u16 magic = le16_to_cpu(sb->s_encoding);
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(f2fs_sb_encoding_map); i++)
+ if (magic == f2fs_sb_encoding_map[i].magic)
+ break;
+
+ if (i >= ARRAY_SIZE(f2fs_sb_encoding_map))
+ return -EINVAL;
+
+ *encoding = &f2fs_sb_encoding_map[i];
+ *flags = le16_to_cpu(sb->s_encoding_flags);
+
+ return 0;
+}
+#endif
+
static inline void limit_reserve_root(struct f2fs_sb_info *sbi)
{
block_t limit = min((sbi->user_block_count << 1) / 1000,
@@ -798,6 +829,13 @@ static int parse_options(struct super_block *sb, char *options)
return -EINVAL;
}
#endif
+#ifndef CONFIG_UNICODE
+ if (f2fs_sb_has_casefold(sbi)) {
+ f2fs_err(sbi,
+ "Filesystem with casefold feature cannot be mounted without CONFIG_UNICODE");
+ return -EINVAL;
+ }
+#endif
if (F2FS_IO_SIZE_BITS(sbi) && !test_opt(sbi, LFS)) {
f2fs_err(sbi, "Should set mode=lfs with %uKB-sized IO",
@@ -1089,6 +1127,9 @@ static void f2fs_put_super(struct super_block *sb)
destroy_percpu_info(sbi);
for (i = 0; i < NR_PAGE_TYPE; i++)
kvfree(sbi->write_io[i]);
+#ifdef CONFIG_UNICODE
+ utf8_unload(sbi->s_encoding);
+#endif
kvfree(sbi);
}
@@ -3031,6 +3072,51 @@ static int f2fs_scan_devices(struct f2fs_sb_info *sbi)
return 0;
}
+static int f2fs_setup_casefold(struct f2fs_sb_info *sbi)
+{
+#ifdef CONFIG_UNICODE
+ if (f2fs_sb_has_casefold(sbi) && !sbi->s_encoding) {
+ const struct f2fs_sb_encodings *encoding_info;
+ struct unicode_map *encoding;
+ __u16 encoding_flags;
+
+ if (f2fs_sb_has_encrypt(sbi)) {
+ f2fs_err(sbi,
+ "Can't mount with encoding and encryption");
+ return -EINVAL;
+ }
+
+ if (f2fs_sb_read_encoding(sbi->raw_super, &encoding_info,
+ &encoding_flags)) {
+ f2fs_err(sbi,
+ "Encoding requested by superblock is unknown");
+ return -EINVAL;
+ }
+
+ encoding = utf8_load(encoding_info->version);
+ if (IS_ERR(encoding)) {
+ f2fs_err(sbi,
+ "can't mount with superblock charset: %s-%s "
+ "not supported by the kernel. flags: 0x%x.",
+ encoding_info->name, encoding_info->version,
+ encoding_flags);
+ return PTR_ERR(encoding);
+ }
+ f2fs_info(sbi, "Using encoding defined by superblock: "
+ "%s-%s with flags 0x%hx", encoding_info->name,
+ encoding_info->version?:"\b", encoding_flags);
+
+ sbi->s_encoding = encoding;
+ sbi->s_encoding_flags = encoding_flags;
+ }
+#else
+ if (f2fs_sb_has_casefold(sbi)) {
+ f2fs_err(sbi, "Filesystem with casefold feature cannot be mounted without CONFIG_UNICODE");
+ return -EINVAL;
+ }
+#endif
+}
+
static void f2fs_tuning_parameters(struct f2fs_sb_info *sbi)
{
struct f2fs_sm_info *sm_i = SM_I(sbi);
@@ -3127,6 +3213,10 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
le32_to_cpu(raw_super->log_blocksize);
sb->s_max_links = F2FS_LINK_MAX;
+ err = f2fs_setup_casefold(sbi);
+ if (err)
+ goto free_options;
+
#ifdef CONFIG_QUOTA
sb->dq_op = &f2fs_quota_operations;
sb->s_qcop = &f2fs_quotactl_ops;
@@ -3477,6 +3567,10 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
free_bio_info:
for (i = 0; i < NR_PAGE_TYPE; i++)
kvfree(sbi->write_io[i]);
+
+#ifdef CONFIG_UNICODE
+ utf8_unload(sbi->s_encoding);
+#endif
free_options:
#ifdef CONFIG_QUOTA
for (i = 0; i < MAXQUOTAS; i++)
diff --git a/fs/f2fs/sysfs.c b/fs/f2fs/sysfs.c
index 3aeacd0aacfd2..f9fcca695db9f 100644
--- a/fs/f2fs/sysfs.c
+++ b/fs/f2fs/sysfs.c
@@ -10,6 +10,7 @@
#include <linux/proc_fs.h>
#include <linux/f2fs_fs.h>
#include <linux/seq_file.h>
+#include <linux/unicode.h>
#include "f2fs.h"
#include "segment.h"
@@ -81,6 +82,19 @@ static ssize_t unusable_show(struct f2fs_attr *a,
(unsigned long long)unusable);
}
+static ssize_t encoding_show(struct f2fs_attr *a,
+ struct f2fs_sb_info *sbi, char *buf)
+{
+#ifdef CONFIG_UNICODE
+ if (f2fs_sb_has_casefold(sbi))
+ return snprintf(buf, PAGE_SIZE, "%s (%d.%d.%d)\n",
+ sbi->s_encoding->charset,
+ (sbi->s_encoding->version >> 16) & 0xff,
+ (sbi->s_encoding->version >> 8) & 0xff,
+ sbi->s_encoding->version & 0xff);
+#endif
+ return snprintf(buf, PAGE_SIZE, "(none)");
+}
static ssize_t lifetime_write_kbytes_show(struct f2fs_attr *a,
struct f2fs_sb_info *sbi, char *buf)
@@ -134,6 +148,9 @@ static ssize_t features_show(struct f2fs_attr *a,
if (f2fs_sb_has_sb_chksum(sbi))
len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
len ? ", " : "", "sb_checksum");
+ if (f2fs_sb_has_casefold(sbi))
+ len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
+ len ? ", " : "", "casefold");
len += snprintf(buf + len, PAGE_SIZE - len, "\n");
return len;
}
@@ -365,6 +382,7 @@ enum feat_id {
FEAT_INODE_CRTIME,
FEAT_LOST_FOUND,
FEAT_SB_CHECKSUM,
+ FEAT_CASEFOLD,
};
static ssize_t f2fs_feature_show(struct f2fs_attr *a,
@@ -382,6 +400,7 @@ static ssize_t f2fs_feature_show(struct f2fs_attr *a,
case FEAT_INODE_CRTIME:
case FEAT_LOST_FOUND:
case FEAT_SB_CHECKSUM:
+ case FEAT_CASEFOLD:
return snprintf(buf, PAGE_SIZE, "supported\n");
}
return 0;
@@ -455,6 +474,7 @@ F2FS_GENERAL_RO_ATTR(lifetime_write_kbytes);
F2FS_GENERAL_RO_ATTR(features);
F2FS_GENERAL_RO_ATTR(current_reserved_blocks);
F2FS_GENERAL_RO_ATTR(unusable);
+F2FS_GENERAL_RO_ATTR(encoding);
#ifdef CONFIG_FS_ENCRYPTION
F2FS_FEATURE_RO_ATTR(encryption, FEAT_CRYPTO);
@@ -471,6 +491,7 @@ F2FS_FEATURE_RO_ATTR(quota_ino, FEAT_QUOTA_INO);
F2FS_FEATURE_RO_ATTR(inode_crtime, FEAT_INODE_CRTIME);
F2FS_FEATURE_RO_ATTR(lost_found, FEAT_LOST_FOUND);
F2FS_FEATURE_RO_ATTR(sb_checksum, FEAT_SB_CHECKSUM);
+F2FS_FEATURE_RO_ATTR(casefold, FEAT_CASEFOLD);
#define ATTR_LIST(name) (&f2fs_attr_##name.attr)
static struct attribute *f2fs_attrs[] = {
@@ -515,6 +536,7 @@ static struct attribute *f2fs_attrs[] = {
ATTR_LIST(features),
ATTR_LIST(reserved_blocks),
ATTR_LIST(current_reserved_blocks),
+ ATTR_LIST(encoding),
NULL,
};
ATTRIBUTE_GROUPS(f2fs);
@@ -535,6 +557,7 @@ static struct attribute *f2fs_feat_attrs[] = {
ATTR_LIST(inode_crtime),
ATTR_LIST(lost_found),
ATTR_LIST(sb_checksum),
+ ATTR_LIST(casefold),
NULL,
};
ATTRIBUTE_GROUPS(f2fs_feat);
diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h
index 65559900d4d76..b7c9c7f721339 100644
--- a/include/linux/f2fs_fs.h
+++ b/include/linux/f2fs_fs.h
@@ -36,6 +36,11 @@
#define F2FS_MAX_QUOTAS 3
+#define F2FS_ENC_UTF8_12_1 1
+#define F2FS_ENC_STRICT_MODE_FL (1 << 0)
+#define f2fs_has_strict_mode(sbi) \
+ (sbi->s_encoding_flags & F2FS_ENC_STRICT_MODE_FL)
+
#define F2FS_IO_SIZE(sbi) (1 << F2FS_OPTION(sbi).write_io_size_bits) /* Blocks */
#define F2FS_IO_SIZE_KB(sbi) (1 << (F2FS_OPTION(sbi).write_io_size_bits + 2)) /* KB */
#define F2FS_IO_SIZE_BYTES(sbi) (1 << (F2FS_OPTION(sbi).write_io_size_bits + 12)) /* B */
@@ -109,7 +114,9 @@ struct f2fs_super_block {
struct f2fs_device devs[MAX_DEVICES]; /* device list */
__le32 qf_ino[F2FS_MAX_QUOTAS]; /* quota inode numbers */
__u8 hot_ext_count; /* # of hot file extension */
- __u8 reserved[310]; /* valid reserved region */
+ __le16 s_encoding; /* Filename charset encoding */
+ __le16 s_encoding_flags; /* Filename charset encoding flags */
+ __u8 reserved[306]; /* valid reserved region */
__le32 crc; /* checksum of superblock */
} __packed;
--
2.22.0.657.g960e92d24f-goog
^ permalink raw reply related
* [PATCH v3 0/2] Casefolding in F2FS
From: Daniel Rosenberg @ 2019-07-19 0:03 UTC (permalink / raw)
To: Jaegeuk Kim, Chao Yu, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team,
Daniel Rosenberg
These patches are largely based on the casefolding patches for ext4
v3: Addressed feedback, apart from F2FS_CASEFOLD_FL/FS_CASEFOLD_FL
Added sysfs file "encoding" to see the encoding set on a filesystem
v2: Rebased patches again master, changed f2fs_msg to f2fs_info/f2fs_err
Daniel Rosenberg (2):
f2fs: include charset encoding information in the superblock
f2fs: Support case-insensitive file name lookups
fs/f2fs/dir.c | 126 ++++++++++++++++++++++++++++++++++++----
fs/f2fs/f2fs.h | 21 ++++++-
fs/f2fs/file.c | 9 +++
fs/f2fs/hash.c | 35 ++++++++++-
fs/f2fs/inline.c | 4 +-
fs/f2fs/inode.c | 4 +-
fs/f2fs/namei.c | 21 +++++++
fs/f2fs/super.c | 100 +++++++++++++++++++++++++++++++
fs/f2fs/sysfs.c | 23 ++++++++
include/linux/f2fs_fs.h | 9 ++-
10 files changed, 334 insertions(+), 18 deletions(-)
--
2.22.0.657.g960e92d24f-goog
^ permalink raw reply
* Re: [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Kees Cook @ 2019-07-18 22:00 UTC (permalink / raw)
To: Sasha Levin
Cc: corbet, solar, will, peterz, gregkh, tyhicks, linux-doc,
linux-kernel
In-Reply-To: <20190717231103.13949-1-sashal@kernel.org>
On Wed, Jul 17, 2019 at 07:11:03PM -0400, Sasha Levin wrote:
> Provide more information about how to interact with the linux-distros
> mailing list for disclosing security bugs.
>
> Reference the linux-distros list policy and clarify that the reporter
> must read and understand those policies as they differ from
> security@kernel.org's policy.
>
> Suggested-by: Solar Designer <solar@openwall.com>
> Signed-off-by: Sasha Levin <sashal@kernel.org>
Sorry, but NACK, see below...
> ---
>
> Changes in v2:
> - Focus more on pointing to the linux-distros wiki and policies.
I think this is already happening in the text. What specifically do you
want described differently?
> - Remove explicit linux-distros email.
I don't like this because we had past trouble with notifications going
to the distros@ list and leaking Linux-only flaws to the BSDs. As there
isn't a separate linux-distros wiki, the clarification of WHICH list is
needed.
> - Remove various explanations of linux-distros policies.
I don't think there's value in removing the Tue-Thu comment, nor
providing context for why distros need time. This has been a regular
thing we've had to explain to researchers that aren't familiar with
update procedures and publication timing.
-Kees
>
> Documentation/admin-guide/security-bugs.rst | 19 +++++++++----------
> 1 file changed, 9 insertions(+), 10 deletions(-)
>
> diff --git a/Documentation/admin-guide/security-bugs.rst b/Documentation/admin-guide/security-bugs.rst
> index dcd6c93c7aac..380d44fd618d 100644
> --- a/Documentation/admin-guide/security-bugs.rst
> +++ b/Documentation/admin-guide/security-bugs.rst
> @@ -60,16 +60,15 @@ Coordination
> ------------
>
> Fixes for sensitive bugs, such as those that might lead to privilege
> -escalations, may need to be coordinated with the private
> -<linux-distros@vs.openwall.org> mailing list so that distribution vendors
> -are well prepared to issue a fixed kernel upon public disclosure of the
> -upstream fix. Distros will need some time to test the proposed patch and
> -will generally request at least a few days of embargo, and vendor update
> -publication prefers to happen Tuesday through Thursday. When appropriate,
> -the security team can assist with this coordination, or the reporter can
> -include linux-distros from the start. In this case, remember to prefix
> -the email Subject line with "[vs]" as described in the linux-distros wiki:
> -<http://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists>
> +escalations, may need to be coordinated with the private linux-distros mailing
> +list so that distribution vendors are well prepared to issue a fixed kernel
> +upon public disclosure of the upstream fix. Please read and follow the policies
> +of linux-distros as specified in the linux-distros wiki page before reporting:
> +<https://oss-security.openwall.org/wiki/mailing-lists/distros>. When
> +appropriate, the security team can assist with this coordination, or the
> +reporter can include linux-distros from the start. In this case, remember to
> +prefix the email Subject line with "[vs]" as described in the linux-distros
> +wiki.
>
> CVE assignment
> --------------
> --
> 2.20.1
>
--
Kees Cook
^ permalink raw reply
* Re: [PATCH v2 2/2] f2fs: Support case-insensitive file name lookups
From: Daniel Rosenberg @ 2019-07-18 21:31 UTC (permalink / raw)
To: Chao Yu, Jaegeuk Kim, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team
In-Reply-To: <cbaf59d4-0bd3-6980-4750-fbab14941bdb@huawei.com>
On 7/17/19 3:11 AM, Chao Yu wrote:
> We need to add one more entry f2fs_fsflags_map[] to map F2FS_CASEFOLD_FL to
> FS_CASEFOLD_FL correctly and adapt F2FS_GETTABLE_FS_FL/F2FS_SETTABLE_FS_FL as well.
I don't see FS_CASEFOLD_FL. It would make sense for it to exist, but unless it's in some recent patch I don't think that's currently in the kernel. Or are you suggesting adding it in this patch?
^ permalink raw reply
* Re: [PATCH v2 08/11] kbuild: create *.mod with full directory path and remove MODVERDIR
From: Joe Lawrence @ 2019-07-18 20:18 UTC (permalink / raw)
To: Masahiro Yamada
Cc: Linux Kbuild mailing list, Sam Ravnborg, Nicolas Pitre,
open list:DOCUMENTATION, Jonathan Corbet,
Linux Kernel Mailing List, Michal Marek
In-Reply-To: <CAK7LNAQ41NhPPO6xoVObgFctTO6WewSXPfZkE7_bZXsdAtKSpA@mail.gmail.com>
On 7/17/19 1:21 AM, Masahiro Yamada wrote:
>> Hi Masahiro,
>>
>> I'm following this patchset changes as they will affect the klp-convert
>> series [1] that the livepatching folks have been working on...
>
> Empty files .tmp_versions/*.livepatch are touched
> to keep track of 'LIVEPATCH_* := y', right?
>
Pretty much. From that patchset I think the rework would need to modify..
- [PATCH v4 02/10] kbuild: Support for Symbols.list creation: creates a
Symbols.list file of kernel and module symbols, but *not* including any
from LIVEPATCH_* modules.
- [PATCH v4 05/10] modpost: Integrate klp-convert: if a LIVEPATCH_*
module has changed, call a newly introduced klp-convert script on it.
- [PATCH v4 06/10] modpost: Add modinfo flag to livepatch modules: find
any LIVEPATCH_* mod, add it to a livepatchmods file, then pass that file
to modpost
> Perhaps, adding a new field
> to *.mod files might be cleaner.
I can look into that. By "field" you mean a new row in the file?
Regards,
-- Joe
^ permalink raw reply
* Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger
From: Brendan Higgins @ 2019-07-18 19:22 UTC (permalink / raw)
To: Stephen Boyd
Cc: Frank Rowand, Greg KH, Josh Poimboeuf, Kees Cook, Kieran Bingham,
Luis Chamberlain, Peter Zijlstra, Rob Herring, shuah,
Theodore Ts'o, Masahiro Yamada, devicetree, dri-devel,
kunit-dev, open list:DOCUMENTATION, linux-fsdevel, linux-kbuild,
Linux Kernel Mailing List, open list:KERNEL SELFTEST FRAMEWORK,
linux-nvdimm, linux-um, Sasha Levin, Bird, Timothy,
Amir Goldstein, Dan Carpenter, Daniel Vetter, Jeff Dike,
Joel Stanley, Julia Lawall, Kevin Hilman, Knut Omang,
Logan Gunthorpe, Michael Ellerman, Petr Mladek, Randy Dunlap,
Richard Weinberger, David Rientjes, Steven Rostedt, wfg
In-Reply-To: <20190718175024.C3EC421019@mail.kernel.org>
On Thu, Jul 18, 2019 at 10:50 AM Stephen Boyd <sboyd@kernel.org> wrote:
>
> Quoting Brendan Higgins (2019-07-16 11:52:01)
> > On Tue, Jul 16, 2019 at 10:50 AM Stephen Boyd <sboyd@kernel.org> wrote:
> > >
> >
> > > The only hypothetical case where this can't be done is a complicated
> > > assertion or expectation that does more than one check and can't be
> > > written as a function that dumps out what went wrong. Is this a real
> > > problem? Maybe such an assertion should just open code that logic so we
> > > don't have to build up a string for all the other simple cases.
> >
> > I have some expectations in follow up patchsets for which I created a
> > set of composable matchers for matching structures and function calls
> > that by their nature cannot be written as a single function. The
> > matcher thing is a bit speculative, I know, but for any kind of
> > function call matching, you need to store a record of functions you
> > are expecting to have called and then each one needs to have a set of
> > expectations defined by the user; I don't think there is a way to do
> > that that doesn't involve having multiple separate functions each
> > having some information useful to constructing the message.
> >
> > I know the code in question isn't in this patchset; the function
> > matching code was in one of the earlier versions of the RFC, but I
> > dropped it to make this patchset smaller and more manageable. So I get
> > it if you would like me to drop it and add it back in when I try to
> > get the function and structure matching stuff in, but I would really
> > prefer to keep it as is if you don't care too much.
>
> Do you have a link to those earlier patches?
This is the first patchset:
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788057.html
In particular you can see the code for matching functions here:
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788073.html
And parameter matching code here:
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788072.html
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788086.html
My apologies in advance, but the code at this early stage had not
adopted the kunit_* prefix and was still using the test_* and mock_*
prefix. (Hence, struct kunit_stream was known as struct test_stream).
> > > It seems far simpler to get rid of the string stream API and just have a
> > > struct for this.
> > >
> > > struct kunit_fail_msg {
> > > const char *line;
> > > const char *file;
> > > const char *func;
> > > const char *msg;
> > > };
> > >
> > > Then you can have the assertion macros create this on the stack (with
> > > another macro?).
> > >
> > > #define DEFINE_KUNIT_FAIL_MSG(name, _msg) \
> > > struct kunit_fail_msg name = { \
> > > .line = __LINE__, \
> > > .file = __FILE__, \
> > > .func = __func__, \
> > > .msg = _msg, \
> > > }
> > >
> > > I don't want to derail this whole series on this topic, but it seems
> > > like a bunch of code is there to construct this same set of information
> > > over and over again into a buffer a little bit at a time and then throw
> > > it away when nothing fails just because we may want to support the case
> > > where we have some unstructured data to inform the user about.
> >
> > Yeah, that's fair. I think there are a number of improvements to be
> > made with how the expectations are defined other than that, but I was
> > hoping I could do that after this patchset is merged. I just figured
> > with the kinds of things I would like to do, it would lead to a whole
> > new round of discussion.
> >
> > In either case, I think I would still like to use the `struct
> > kunit_stream` as part of the interface to share the failure message
> > with the test case runner code in test.c, at least eventually, so that
> > I only have to have one way to receive data from expectations, but I
> > think I can do that and still do what you suggest by just constructing
> > the kunit_stream at the end of expectations where it is feasible.
> >
> > All in all I agree with what you are saying, but I would rather do it
> > as a follow up possibly once we have some more code on the table. I
> > could just see this opening up a whole new can of worms where we
> > debate about exactly how expectations and assertions work for another
> > several months, only to rip it all out shortly there after. I know
> > that's how these things go, but that's my preference.
> >
> > I can do what you suggest if you feel strongly about it, but I would
> > prefer to hold off until later. It's your call.
> >
>
> The crux of my complaint is that the string stream API is too loosely
> defined to be usable. It allows tests to build up a string of
> unstructured information, but with certain calling constraints so we
> have to tread carefully. If there was more structure to the data that's
> being recorded then the test case runner could operate on the data
> without having to do string/stream operations, allocations, etc. This
> would make the assertion logic much more concrete and specific to kunit,
> instead of this small kunit wrapper that's been placed on top of string
> stream.
Yeah, I can see the point of wanting something that provides more
structure than the raw `struct kunit_stream` interface. In fact, it is
something I had already started working on, when I had determined it
would be a large effort to capture all the variations. I was further
put off from the idea when I had been asked to convert the KUnit
intermediate format from what I was using to TAP, because, as it is,
the current data printed out by KUnit doesn't contain all the data I
would like to put in it in a way that best takes advantage of the TAP
specification. One problematic area in particular: TAP already
provides a way to present a lot of the data I would like to export,
but it involves JSON serialization which was an idea that some of the
other reviewers understandably weren't too keen on. TAP also wants to
report data some time after it is available, which is generally not a
good idea for test debug information; you want to make it available as
soon as you can or you risk crashing with the data still inside.
Hence, I decided we could probably spend a good long while debating
how I present the information. So the idea of having a loose
definition seemed attractive to me in its own right since it would
likely conform to whatever we ended up deciding in the long run. Also,
all the better that it was what I already had and no one seemed to
mind too much.
The only constant I expect is that `struct kunit` will likely need to
take an abstract object with a `commit` method, or a `format` method
or whatever so it could control when data was going to be printed out
to the user. We will probably also use a string builder in there
somewhere.
> TL;DR: If we can get rid of the string stream API I'd view that as an
> improvement because building arbitrary strings in the kernel is complex,
> error prone and has calling context concerns.
True. No argument there.
> Is the intention that other code besides unit tests will use this string
> stream API to build up strings? Any targets in mind? This would be a
> good way to get the API merged upstream given that its 2019 and we
> haven't had such an API in the kernel so far.
Someone, (was it you?) asked about code sharing with a string builder
thingy that was used for creating structured human readable files, but
that seemed like a pretty massive undertaking.
Aside from that, no. I would kind of prefered that nobody used it for
anything else because I the issues you described.
Nevertheless, I think the debate over the usefulness of the
string_stream and kunit_stream are separate topics. Even if we made
kunit_stream more structured, I am pretty sure I would want to use
string_stream or some variation for constructing the message.
> An "object oriented" (strong quotes!) approach where kunit_fail_msg is
> the innermost struct in some assertion specific structure might work
> nicely and allow the test runner to call a generic 'format' function to
> print out the message based on the type of assertion/expectation it is.
> It probably would mean less code size too because the strings that are
> common will be in the common printing function instead of created twice,
> in the macros/code and then copied to the heap for the string stream.
>
> struct kunit_assert {
> const char *line;
> const char *file;
> const char *func;
> void (*format)(struct kunit_assert *assert);
> };
>
> struct kunit_comparison_assert {
> enum operator operator;
> const char *left;
> const char *right;
> struct kunit_assert assert;
> };
>
> struct kunit_bool_assert {
> const char *truth;
> const char *statement;
> struct kunit_assert assert;
> };
>
> void kunit_format_comparison(struct kunit_assert *assert)
> {
> struct kunit_comparison_assert *comp = container_of(assert, ...)
>
> kunit_printk(...)
> }
I started working on something similarish, but by the time I ended up
coming up with a parent object whose definition was loose enough to
satisfy all the properties required by the child classes it ended up
basically being the same as what I have now just with a more complex
hierarchy of message manipulation logic.
On the other hand, I didn't have the idea of doing the parent object
quite the way you did and that would clean up a lot of the duplicated
first line logic.
I would like to give it a try, but I am afraid I am going to get
sucked down a really deep rabbit hole.
> Maybe other people have opinions here on if you should do it now or
> later. Future coding is not a great argument because it's hard to
> predict the future. On the other hand, this patchset is in good shape to
Yeah, that's kind of why I am afraid to go down this road when I have
something that works now and I know works with the mocking stuff I
want to do.
I would like to try your suggestion, but I want to try to make it work
with my mocking patches before I commit to it because otherwise I am
just going to have to back it out in a follow up patchset.
> merge and I'd like to use it to write unit tests for code I maintain so
> I don't want to see this stall out. Sorry if I'm opening the can of
> worms you're talking about.
Don't be sorry. I agree with you that the kunit_stream stuff is not very pretty.
Shuah, have we missed the merge window for 5.3?
I saw you only sent one PR out so far for this release, and there
wasn't much in it; I imagine you are going to send at least one more?
I figure, if we still got time to try out your suggestion, Stephen, no
harm in trying.
Also if we missed it, then I have another couple months to play around with it.
What do you think?
^ permalink raw reply
* Re: Using rst2pdf for PDF output - Was: Re: [PATCH 0/5] PDF output fixes
From: Markus Heiser @ 2019-07-18 17:56 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Jonathan Corbet; +Cc: linux-doc
In-Reply-To: <20190718144655.5aa7deb4@coco.lan>
Am 18.07.19 um 19:46 schrieb Mauro Carvalho Chehab:
> Em Sat, 13 Jul 2019 00:41:25 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>
>> Em Fri, 12 Jul 2019 19:27:05 -0300
>> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>>
>>> Em Fri, 12 Jul 2019 14:19:21 -0600
>>> Jonathan Corbet <corbet@lwn.net> escreveu:
>>>
>>>> Can't you just make rst2pdf work instead? :)
>>>
>>> Well, we can try.
Thanks a lot for your investigation on this. I also checked the rst2pdf sources
a while ago, for me it was crap with crap requirements [1] .. my tip: don't
waste to much time on it.
[1] https://github.com/mchehab/rst2pdf/blob/master/requirements.txt#L31
-- Markus --
^ permalink raw reply
* Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger
From: Stephen Boyd @ 2019-07-18 17:50 UTC (permalink / raw)
To: Brendan Higgins
Cc: Frank Rowand, Greg KH, Josh Poimboeuf, Kees Cook, Kieran Bingham,
Luis Chamberlain, Peter Zijlstra, Rob Herring, shuah,
Theodore Ts'o, Masahiro Yamada, devicetree, dri-devel,
kunit-dev, open list:DOCUMENTATION, linux-fsdevel, linux-kbuild,
Linux Kernel Mailing List, open list:KERNEL SELFTEST FRAMEWORK,
linux-nvdimm, linux-um, Sasha Levin, Bird, Timothy,
Amir Goldstein, Dan Carpenter, Daniel Vetter, Jeff Dike,
Joel Stanley, Julia Lawall, Kevin Hilman, Knut Omang,
Logan Gunthorpe, Michael Ellerman, Petr Mladek, Randy Dunlap,
Richard Weinberger, David Rientjes, Steven Rostedt, wfg
In-Reply-To: <CAFd5g453vXeSUCZenCk_CzJ-8a1ym9RaPo0NVF=FujF9ac-5Ag@mail.gmail.com>
Quoting Brendan Higgins (2019-07-16 11:52:01)
> On Tue, Jul 16, 2019 at 10:50 AM Stephen Boyd <sboyd@kernel.org> wrote:
> >
>
> > The only hypothetical case where this can't be done is a complicated
> > assertion or expectation that does more than one check and can't be
> > written as a function that dumps out what went wrong. Is this a real
> > problem? Maybe such an assertion should just open code that logic so we
> > don't have to build up a string for all the other simple cases.
>
> I have some expectations in follow up patchsets for which I created a
> set of composable matchers for matching structures and function calls
> that by their nature cannot be written as a single function. The
> matcher thing is a bit speculative, I know, but for any kind of
> function call matching, you need to store a record of functions you
> are expecting to have called and then each one needs to have a set of
> expectations defined by the user; I don't think there is a way to do
> that that doesn't involve having multiple separate functions each
> having some information useful to constructing the message.
>
> I know the code in question isn't in this patchset; the function
> matching code was in one of the earlier versions of the RFC, but I
> dropped it to make this patchset smaller and more manageable. So I get
> it if you would like me to drop it and add it back in when I try to
> get the function and structure matching stuff in, but I would really
> prefer to keep it as is if you don't care too much.
Do you have a link to those earlier patches?
>
> > It seems far simpler to get rid of the string stream API and just have a
> > struct for this.
> >
> > struct kunit_fail_msg {
> > const char *line;
> > const char *file;
> > const char *func;
> > const char *msg;
> > };
> >
> > Then you can have the assertion macros create this on the stack (with
> > another macro?).
> >
> > #define DEFINE_KUNIT_FAIL_MSG(name, _msg) \
> > struct kunit_fail_msg name = { \
> > .line = __LINE__, \
> > .file = __FILE__, \
> > .func = __func__, \
> > .msg = _msg, \
> > }
> >
> > I don't want to derail this whole series on this topic, but it seems
> > like a bunch of code is there to construct this same set of information
> > over and over again into a buffer a little bit at a time and then throw
> > it away when nothing fails just because we may want to support the case
> > where we have some unstructured data to inform the user about.
>
> Yeah, that's fair. I think there are a number of improvements to be
> made with how the expectations are defined other than that, but I was
> hoping I could do that after this patchset is merged. I just figured
> with the kinds of things I would like to do, it would lead to a whole
> new round of discussion.
>
> In either case, I think I would still like to use the `struct
> kunit_stream` as part of the interface to share the failure message
> with the test case runner code in test.c, at least eventually, so that
> I only have to have one way to receive data from expectations, but I
> think I can do that and still do what you suggest by just constructing
> the kunit_stream at the end of expectations where it is feasible.
>
> All in all I agree with what you are saying, but I would rather do it
> as a follow up possibly once we have some more code on the table. I
> could just see this opening up a whole new can of worms where we
> debate about exactly how expectations and assertions work for another
> several months, only to rip it all out shortly there after. I know
> that's how these things go, but that's my preference.
>
> I can do what you suggest if you feel strongly about it, but I would
> prefer to hold off until later. It's your call.
>
The crux of my complaint is that the string stream API is too loosely
defined to be usable. It allows tests to build up a string of
unstructured information, but with certain calling constraints so we
have to tread carefully. If there was more structure to the data that's
being recorded then the test case runner could operate on the data
without having to do string/stream operations, allocations, etc. This
would make the assertion logic much more concrete and specific to kunit,
instead of this small kunit wrapper that's been placed on top of string
stream.
TL;DR: If we can get rid of the string stream API I'd view that as an
improvement because building arbitrary strings in the kernel is complex,
error prone and has calling context concerns.
Is the intention that other code besides unit tests will use this string
stream API to build up strings? Any targets in mind? This would be a
good way to get the API merged upstream given that its 2019 and we
haven't had such an API in the kernel so far.
An "object oriented" (strong quotes!) approach where kunit_fail_msg is
the innermost struct in some assertion specific structure might work
nicely and allow the test runner to call a generic 'format' function to
print out the message based on the type of assertion/expectation it is.
It probably would mean less code size too because the strings that are
common will be in the common printing function instead of created twice,
in the macros/code and then copied to the heap for the string stream.
struct kunit_assert {
const char *line;
const char *file;
const char *func;
void (*format)(struct kunit_assert *assert);
};
struct kunit_comparison_assert {
enum operator operator;
const char *left;
const char *right;
struct kunit_assert assert;
};
struct kunit_bool_assert {
const char *truth;
const char *statement;
struct kunit_assert assert;
};
void kunit_format_comparison(struct kunit_assert *assert)
{
struct kunit_comparison_assert *comp = container_of(assert, ...)
kunit_printk(...)
}
Maybe other people have opinions here on if you should do it now or
later. Future coding is not a great argument because it's hard to
predict the future. On the other hand, this patchset is in good shape to
merge and I'd like to use it to write unit tests for code I maintain so
I don't want to see this stall out. Sorry if I'm opening the can of
worms you're talking about.
^ permalink raw reply
* Using rst2pdf for PDF output - Was: Re: [PATCH 0/5] PDF output fixes
From: Mauro Carvalho Chehab @ 2019-07-18 17:46 UTC (permalink / raw)
To: Jonathan Corbet; +Cc: linux-doc
In-Reply-To: <20190713004125.6009b661@coco.lan>
Em Sat, 13 Jul 2019 00:41:25 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> Em Fri, 12 Jul 2019 19:27:05 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>
> > Em Fri, 12 Jul 2019 14:19:21 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:
> >
> > > Can't you just make rst2pdf work instead? :)
> >
> > Well, we can try.
Doing some tests right now with upstream version of it. I actually
created a fork:
https://github.com/mchehab/rst2pdf
basically upstream + some pending PRs + 2 fixes and a hack:
https://github.com/mchehab/rst2pdf/commits/master
> > Also, right now, rst2pdf only supports Python 2.7.
> >
> > I would wait for it to support Python 3.x before doing the actual
> > migration.
The upstream version runs on python 3.x. There are a few issues there
yet to be solved.
> There are a few issues with the quick hack to use rst2pdf. The
> enclosed one works better.
Did another improvement (see enclosed). This one should be applied
after my PDF fixes branch:
https://git.linuxtv.org/mchehab/experimental.git/log/?h=pdf_fixes_v1
I suspect that we'll also need a patch for kfigure.py, in order for it
to support kernel-figure output.
>
> Yet, from the existing 70 books (on my experimental tree), rst2pdf breaks
> with 14 ones.
I was able to find one of the reason some books failed: rst2pdf has a
"hidden" dependency:
svglib
Without that, it will silently ignore SVG images, causing the script
to break.
Even with that, it will still break (reporting success), even on not
so complex books. For example:
$ make SPHINXDIRS=hwmon pdfdocs
writing hwmon...
[ERROR] pdfbuilder.py:150 More than 10 pages generated without content - halting layout. Likely that a flowable is too large for any frame.
Traceback (most recent call last):
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/rst2pdf-0.95.dev0-py3.7.egg/rst2pdf/pdfbuilder.py", line 147, in write
docwriter.write(doctree, destination)
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/docutils/writers/__init__.py", line 80, in write
self.translate()
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/rst2pdf-0.95.dev0-py3.7.egg/rst2pdf/pdfbuilder.py", line 643, in translate
compressed=self.compressed)
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/rst2pdf-0.95.dev0-py3.7.egg/rst2pdf/createpdf.py", line 643, in createPdf
pdfdoc.multiBuild(elements)
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/reportlab/platypus/doctemplate.py", line 1144, in multiBuild
self.build(tempStory, **buildKwds)
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/reportlab/platypus/doctemplate.py", line 1057, in build
self.handle_flowable(flowables)
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/rst2pdf-0.95.dev0-py3.7.egg/rst2pdf/createpdf.py", line 791, in handle_flowable
self.handle_frameEnd()
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/reportlab/platypus/doctemplate.py", line 709, in handle_frameEnd
self.handle_pageEnd()
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/reportlab/platypus/doctemplate.py", line 651, in handle_pageEnd
raise LayoutError(ident)
reportlab.platypus.doctemplate.LayoutError: More than 10 pages generated without content - halting layout. Likely that a flowable is too large for any frame.
FAILED
build succeeded.
Patch enclosed.
Thanks,
Mauro
---
[PATCH] docs: experimental: build PDF with rst2pdf
Change the logic to use rst2pdf instead of LaTeX.
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 638c4c11d102..565acf6e6ee7 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -94,18 +94,22 @@ latexdocs:
ifeq ($(HAVE_PDFLATEX),0)
-pdfdocs:
+oldpdfdocs:
$(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
@echo " SKIP Sphinx $@ target."
else # HAVE_PDFLATEX
-pdfdocs: latexdocs
+oldpdfdocs: latexdocs
@$(srctree)/scripts/sphinx-pre-install --version-check
$(foreach var,$(SPHINXDIRS), $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" -C $(BUILDDIR)/$(var)/latex || exit;)
endif # HAVE_PDFLATEX
+pdfdocs:
+ @$(srctree)/scripts/sphinx-pre-install --version-check
+ @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,pdf,$(var),pdf,$(var)))
+
epubdocs:
@$(srctree)/scripts/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
@@ -131,6 +135,7 @@ dochelp:
@echo ' htmldocs - HTML'
@echo ' latexdocs - LaTeX'
@echo ' pdfdocs - PDF'
+ @echo ' rst2pdf - PDF, using experimental rst2pdf support'
@echo ' epubdocs - EPUB'
@echo ' xmldocs - XML'
@echo ' linkcheckdocs - check for broken external links (will connect to external hosts)'
diff --git a/Documentation/conf.py b/Documentation/conf.py
index c698f453bfab..4aa76e4b30ab 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -46,6 +46,8 @@ if (major == 1 and minor > 3) or (major > 1):
else:
extensions.append("sphinx.ext.pngmath")
+extensions.append("rst2pdf.pdfbuilder")
+
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
@@ -574,12 +576,15 @@ epub_exclude_files = ['search.html']
#
# See the Sphinx chapter of http://ralsina.me/static/manual.pdf
#
-# FIXME: Do not add the index file here; the result will be too big. Adding
-# multiple PDF files here actually tries to get the cross-referencing right
-# *between* PDF files.
-pdf_documents = [
- ('kernel-documentation', u'Kernel', u'Kernel', u'J. Random Bozo'),
-]
+
+# Add all LaTeX files to PDF documents as well
+pdf_documents = []
+for l in latex_documents:
+ doc = l[0]
+ fn = l[1].replace(".tex", "")
+ name = l[2]
+ authors = l[3]
+ pdf_documents.append((doc, fn, name, authors))
# kernel-doc extension configuration for running Sphinx directly (e.g. by Read
# the Docs). In a normal build, these are supplied from the Makefile via command
diff --git a/Documentation/media/Makefile b/Documentation/media/Makefile
index d75d70f191bc..3050136ed489 100644
--- a/Documentation/media/Makefile
+++ b/Documentation/media/Makefile
@@ -53,12 +53,13 @@ $(BUILDDIR)/lirc.h.rst: ${UAPI}/lirc.h ${PARSER} $(SRC_DIR)/lirc.h.rst.exception
# Media build rules
-.PHONY: all html epub xml latex
+.PHONY: all html epub xml latex pdf
all: $(IMGDOT) $(BUILDDIR) ${TARGETS}
html: all
epub: all
xml: all
+pdf: all
latex: $(IMGPDF) all
linkcheck:
diff --git a/Documentation/sphinx/load_config.py b/Documentation/sphinx/load_config.py
index 101e6f0b0fcf..a89d642d520e 100644
--- a/Documentation/sphinx/load_config.py
+++ b/Documentation/sphinx/load_config.py
@@ -43,6 +43,17 @@ def loadConfig(namespace):
namespace['latex_documents'] = new_latex_docs
+ # Add LaTeX files to PDF documents as well
+ pdf_documents = []
+ for l in new_latex_docs:
+ doc = l[0]
+ fn = l[1].replace(".tex", "")
+ name = l[2]
+ authors = l[3]
+ pdf_documents.append((doc, fn, name, authors))
+
+ namespace['pdf_documents'] = pdf_documents
+
# If there is an extra conf.py file, load it
if os.path.isfile(config_file):
sys.stdout.write("load additional sphinx-config: %s\n" % config_file)
diff --git a/Makefile b/Makefile
index ec21ebbd12c2..2a749335d0df 100644
--- a/Makefile
+++ b/Makefile
@@ -1564,7 +1564,7 @@ $(help-board-dirs): help-%:
# Documentation targets
# ---------------------------------------------------------------------------
DOC_TARGETS := xmldocs latexdocs pdfdocs htmldocs epubdocs cleandocs \
- linkcheckdocs dochelp refcheckdocs
+ linkcheckdocs dochelp refcheckdocs oldpdfdocs
PHONY += $(DOC_TARGETS)
$(DOC_TARGETS): scripts_basic FORCE
$(Q)$(MAKE) $(build)=Documentation $@
^ permalink raw reply related
* Re: [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Solar Designer @ 2019-07-18 14:14 UTC (permalink / raw)
To: Will Deacon
Cc: Sasha Levin, corbet, keescook, peterz, gregkh, tyhicks, linux-doc,
linux-kernel
In-Reply-To: <20190718094057.e4nclrw6qd2t4vw7@willie-the-truck>
On Thu, Jul 18, 2019 at 10:40:58AM +0100, Will Deacon wrote:
> On Wed, Jul 17, 2019 at 07:11:03PM -0400, Sasha Levin wrote:
> > Provide more information about how to interact with the linux-distros
> > mailing list for disclosing security bugs.
> >
> > Reference the linux-distros list policy and clarify that the reporter
> > must read and understand those policies as they differ from
> > security@kernel.org's policy.
> >
> > Suggested-by: Solar Designer <solar@openwall.com>
> > Signed-off-by: Sasha Levin <sashal@kernel.org>
> > ---
> >
> > Changes in v2:
> > - Focus more on pointing to the linux-distros wiki and policies.
> > - Remove explicit linux-distros email.
> > - Remove various explanations of linux-distros policies.
> >
> > Documentation/admin-guide/security-bugs.rst | 19 +++++++++----------
> > 1 file changed, 9 insertions(+), 10 deletions(-)
> >
> > diff --git a/Documentation/admin-guide/security-bugs.rst b/Documentation/admin-guide/security-bugs.rst
> > index dcd6c93c7aac..380d44fd618d 100644
> > --- a/Documentation/admin-guide/security-bugs.rst
> > +++ b/Documentation/admin-guide/security-bugs.rst
> > @@ -60,16 +60,15 @@ Coordination
> > ------------
> >
> > Fixes for sensitive bugs, such as those that might lead to privilege
> > -escalations, may need to be coordinated with the private
> > -<linux-distros@vs.openwall.org> mailing list so that distribution vendors
> > -are well prepared to issue a fixed kernel upon public disclosure of the
> > -upstream fix. Distros will need some time to test the proposed patch and
> > -will generally request at least a few days of embargo, and vendor update
> > -publication prefers to happen Tuesday through Thursday. When appropriate,
> > -the security team can assist with this coordination, or the reporter can
> > -include linux-distros from the start. In this case, remember to prefix
> > -the email Subject line with "[vs]" as described in the linux-distros wiki:
> > -<http://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists>
> > +escalations, may need to be coordinated with the private linux-distros mailing
> > +list so that distribution vendors are well prepared to issue a fixed kernel
> > +upon public disclosure of the upstream fix. Please read and follow the policies
> > +of linux-distros as specified in the linux-distros wiki page before reporting:
>
> can we add a "there" at the end of this sentence, so it can't be misread as
> implying that you must follow the linux-distros policies before reporting to
> security@kernel.org ?
Sasha's patch above and the addition suggested by Will look good to me.
Thanks!
Alexander
^ permalink raw reply
* Re: [PATCH] docs/vm: transhuge: fix typo in madvise reference
From: Mike Rapoport @ 2019-07-18 11:17 UTC (permalink / raw)
To: Jeremy Cline; +Cc: Jonathan Corbet, linux-doc, linux-kernel
In-Reply-To: <20190716144908.25843-1-jcline@redhat.com>
On Tue, Jul 16, 2019 at 10:49:08AM -0400, Jeremy Cline wrote:
> Fix an off-by-one typo in the transparent huge pages admin
> documentation.
>
> Signed-off-by: Jeremy Cline <jcline@redhat.com>
Acked-by: Mike Rapoport <rppt@linux.ibm.com>
> ---
> Documentation/admin-guide/mm/transhuge.rst | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst
> index 7ab93a8404b9..bd5714547cee 100644
> --- a/Documentation/admin-guide/mm/transhuge.rst
> +++ b/Documentation/admin-guide/mm/transhuge.rst
> @@ -53,7 +53,7 @@ disabled, there is ``khugepaged`` daemon that scans memory and
> collapses sequences of basic pages into huge pages.
>
> The THP behaviour is controlled via :ref:`sysfs <thp_sysfs>`
> -interface and using madivse(2) and prctl(2) system calls.
> +interface and using madvise(2) and prctl(2) system calls.
>
> Transparent Hugepage Support maximizes the usefulness of free memory
> if compared to the reservation approach of hugetlbfs by allowing all
> --
> 2.21.0
>
--
Sincerely yours,
Mike.
^ permalink raw reply
* Re: [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Will Deacon @ 2019-07-18 9:40 UTC (permalink / raw)
To: Sasha Levin
Cc: corbet, solar, keescook, peterz, gregkh, tyhicks, linux-doc,
linux-kernel
In-Reply-To: <20190717231103.13949-1-sashal@kernel.org>
On Wed, Jul 17, 2019 at 07:11:03PM -0400, Sasha Levin wrote:
> Provide more information about how to interact with the linux-distros
> mailing list for disclosing security bugs.
>
> Reference the linux-distros list policy and clarify that the reporter
> must read and understand those policies as they differ from
> security@kernel.org's policy.
>
> Suggested-by: Solar Designer <solar@openwall.com>
> Signed-off-by: Sasha Levin <sashal@kernel.org>
> ---
>
> Changes in v2:
> - Focus more on pointing to the linux-distros wiki and policies.
> - Remove explicit linux-distros email.
> - Remove various explanations of linux-distros policies.
>
> Documentation/admin-guide/security-bugs.rst | 19 +++++++++----------
> 1 file changed, 9 insertions(+), 10 deletions(-)
>
> diff --git a/Documentation/admin-guide/security-bugs.rst b/Documentation/admin-guide/security-bugs.rst
> index dcd6c93c7aac..380d44fd618d 100644
> --- a/Documentation/admin-guide/security-bugs.rst
> +++ b/Documentation/admin-guide/security-bugs.rst
> @@ -60,16 +60,15 @@ Coordination
> ------------
>
> Fixes for sensitive bugs, such as those that might lead to privilege
> -escalations, may need to be coordinated with the private
> -<linux-distros@vs.openwall.org> mailing list so that distribution vendors
> -are well prepared to issue a fixed kernel upon public disclosure of the
> -upstream fix. Distros will need some time to test the proposed patch and
> -will generally request at least a few days of embargo, and vendor update
> -publication prefers to happen Tuesday through Thursday. When appropriate,
> -the security team can assist with this coordination, or the reporter can
> -include linux-distros from the start. In this case, remember to prefix
> -the email Subject line with "[vs]" as described in the linux-distros wiki:
> -<http://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists>
> +escalations, may need to be coordinated with the private linux-distros mailing
> +list so that distribution vendors are well prepared to issue a fixed kernel
> +upon public disclosure of the upstream fix. Please read and follow the policies
> +of linux-distros as specified in the linux-distros wiki page before reporting:
can we add a "there" at the end of this sentence, so it can't be misread as
implying that you must follow the linux-distros policies before reporting to
security@kernel.org ?
Will
^ permalink raw reply
* Re: [PATCH v6 0/6] KASan for arm
From: Arnd Bergmann @ 2019-07-18 7:51 UTC (permalink / raw)
To: Florian Fainelli
Cc: Linus Walleij, Florian Fainelli, Mark Rutland, Alexandre Belloni,
Michal Hocko, Julien Thierry, Catalin Marinas,
linux-kernel@vger.kernel.org, David Howells, Masahiro Yamada,
Andrey Ryabinin, Alexander Potapenko, kvmarm, Jonathan Corbet,
Abbott Liu, Daniel Lezcano, Russell King, kasan-dev,
bcm-kernel-feedback-list, Dmitry Vyukov, Geert Uytterhoeven,
drjones, Vladimir Murzin, Kees Cook, Marc Zyngier, Andre Przywara,
philip, Jinbum Park, Thomas Gleixner, Linux ARM, Nicolas Pitre,
Greg KH, Ard Biesheuvel, Linux Doc Mailing List, Christoffer Dall,
Rob Landley, Philippe Ombredanne, Andrew Morton, Thomas Garnier,
Kirill A . Shutemov
In-Reply-To: <0ba50ae2-be09-f633-ab1f-860e8b053882@broadcom.com>
On Thu, Jul 11, 2019 at 7:00 PM Florian Fainelli
<florian.fainelli@broadcom.com> wrote:
> On 7/2/19 2:06 PM, Linus Walleij wrote:
>
> Great, thanks a lot for taking a look. FYI, I will be on holiday from
> July 19th till August 12th, if you think you have more feedback between
> now and then, I can try to pick it up and submit a v7 with that feedback
> addressed, or it will happen when I return, or you can pick it up if you
> refer, all options are possible!
>
> @Arnd, should we squash your patches in as well?
Yes, please do. I don't remember if I sent you all of them already,
here is the list of patches that I have applied locally on top of your
series to get a clean randconfig build:
123c3262f872 KASAN: push back KASAN_STACK to clang-10
d63dd9e2afd9 [HACK] ARM: disable KASAN+XIP_KERNEL
879eb3c22240 kasan: increase 32-bit stack frame warning limit
053555034bdf kasan: disable CONFIG_KASAN_STACK with clang on arm32
6c1a78a448c2 ARM: fix kasan link failures
Arnd
^ permalink raw reply
* [PATCH 2/2] doc:it_IT: rephrase statement
From: Federico Vaga @ 2019-07-18 7:47 UTC (permalink / raw)
To: Jonathan Corbet; +Cc: linux-doc, linux-kernel, Federico Vaga
The statement sounds more like a literal translation
Signed-off-by: Federico Vaga <federico.vaga@vaga.pv.it>
---
Documentation/translations/it_IT/doc-guide/sphinx.rst | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Documentation/translations/it_IT/doc-guide/sphinx.rst b/Documentation/translations/it_IT/doc-guide/sphinx.rst
index d9ee4b1f098f..f1ad4504b734 100644
--- a/Documentation/translations/it_IT/doc-guide/sphinx.rst
+++ b/Documentation/translations/it_IT/doc-guide/sphinx.rst
@@ -242,9 +242,9 @@ del kernel:
* Per inserire blocchi di testo con caratteri a dimensione fissa (codici di
esempio, casi d'uso, eccetera): utilizzate ``::`` quando non è necessario
evidenziare la sintassi, specialmente per piccoli frammenti; invece,
- utilizzate ``.. code-block:: <language>`` per blocchi di più lunghi che
- potranno beneficiare dell'avere la sintassi evidenziata. Per un breve pezzo
- di codice da inserire nel testo, usate \`\`.
+ utilizzate ``.. code-block:: <language>`` per blocchi più lunghi che
+ beneficeranno della sintassi evidenziata. Per un breve pezzo di codice da
+ inserire nel testo, usate \`\`.
Il dominio C
--
2.21.0
^ permalink raw reply related
* [PATCH 1/2] doc:it_IT: align translation to mainline
From: Federico Vaga @ 2019-07-18 7:47 UTC (permalink / raw)
To: Jonathan Corbet; +Cc: linux-doc, linux-kernel, Federico Vaga
The patch translates the following patches in Italian:
d9d7c0c497b8 docs: Note that :c:func: should no longer be used
83e8b971f81c sphinx.rst: Add note about code snippets embedded in the text
cca5e0b8a430 Documentation: PGP: update for newer HW devices
Signed-off-by: Federico Vaga <federico.vaga@vaga.pv.it>
---
.../translations/it_IT/doc-guide/sphinx.rst | 17 +++++++------
.../it_IT/process/maintainer-pgp-guide.rst | 25 ++++++++++++-------
2 files changed, 26 insertions(+), 16 deletions(-)
diff --git a/Documentation/translations/it_IT/doc-guide/sphinx.rst b/Documentation/translations/it_IT/doc-guide/sphinx.rst
index 1739cba8863e..d9ee4b1f098f 100644
--- a/Documentation/translations/it_IT/doc-guide/sphinx.rst
+++ b/Documentation/translations/it_IT/doc-guide/sphinx.rst
@@ -243,7 +243,8 @@ del kernel:
esempio, casi d'uso, eccetera): utilizzate ``::`` quando non è necessario
evidenziare la sintassi, specialmente per piccoli frammenti; invece,
utilizzate ``.. code-block:: <language>`` per blocchi di più lunghi che
- potranno beneficiare dell'avere la sintassi evidenziata.
+ potranno beneficiare dell'avere la sintassi evidenziata. Per un breve pezzo
+ di codice da inserire nel testo, usate \`\`.
Il dominio C
@@ -267,12 +268,14 @@ molto comune come ``open`` o ``ioctl``:
Il nome della funzione (per esempio ioctl) rimane nel testo ma il nome del suo
riferimento cambia da ``ioctl`` a ``VIDIOC_LOG_STATUS``. Anche la voce
-nell'indice cambia in ``VIDIOC_LOG_STATUS`` e si potrà quindi fare riferimento
-a questa funzione scrivendo:
-
-.. code-block:: rst
-
- :c:func:`VIDIOC_LOG_STATUS`
+nell'indice cambia in ``VIDIOC_LOG_STATUS``.
+
+Notate che per una funzione non c'è bisogno di usare ``c:func:`` per generarne
+i riferimenti nella documentazione. Grazie a qualche magica estensione a
+Sphinx, il sistema di generazione della documentazione trasformerà
+automaticamente un riferimento ad una ``funzione()`` in un riferimento
+incrociato quando questa ha una voce nell'indice. Se trovate degli usi di
+``c:func:`` nella documentazione del kernel, sentitevi liberi di rimuoverli.
Tabelle a liste
diff --git a/Documentation/translations/it_IT/process/maintainer-pgp-guide.rst b/Documentation/translations/it_IT/process/maintainer-pgp-guide.rst
index 276db0e37f43..118fb4153e8f 100644
--- a/Documentation/translations/it_IT/process/maintainer-pgp-guide.rst
+++ b/Documentation/translations/it_IT/process/maintainer-pgp-guide.rst
@@ -248,7 +248,10 @@ possano ricevere la vostra nuova sottochiave::
kernel.
Se per qualche ragione preferite rimanere con sottochiavi RSA, nel comando
- precedente, sostituite "ed25519" con "rsa2048".
+ precedente, sostituite "ed25519" con "rsa2048". In aggiunta, se avete
+ intenzione di usare un dispositivo hardware che non supporta le chiavi
+ ED25519 ECC, come la Nitrokey Pro o la Yubikey, allora dovreste usare
+ "nistp256" al posto di "ed25519".
Copia di riserva della chiave primaria per gestire il recupero da disastro
--------------------------------------------------------------------------
@@ -449,23 +452,27 @@ implementi le funzionalità delle smartcard. Sul mercato ci sono diverse
soluzioni disponibili:
- `Nitrokey Start`_: è Open hardware e Free Software, è basata sul progetto
- `GnuK`_ della FSIJ. Ha il supporto per chiavi ECC, ma meno funzionalità di
- sicurezza (come la resistenza alla manomissione o alcuni attacchi ad un
- canale laterale).
+ `GnuK`_ della FSIJ. Questo è uno dei pochi dispositivi a supportare le chiavi
+ ECC ED25519, ma offre meno funzionalità di sicurezza (come la resistenza
+ alla manomissione o alcuni attacchi ad un canale laterale).
- `Nitrokey Pro`_: è simile alla Nitrokey Start, ma è più resistente alla
- manomissione e offre più funzionalità di sicurezza, ma l'ECC.
-- `Yubikey 4`_: l'hardware e il software sono proprietari, ma è più economica
+ manomissione e offre più funzionalità di sicurezza. La Pro 2 supporta la
+ crittografia ECC (NISTP).
+- `Yubikey 5`_: l'hardware e il software sono proprietari, ma è più economica
della Nitrokey Pro ed è venduta anche con porta USB-C il che è utile con i
computer portatili più recenti. In aggiunta, offre altre funzionalità di
- sicurezza come FIDO, U2F, ma non l'ECC
+ sicurezza come FIDO, U2F, e ora supporta anche le chiavi ECC (NISTP)
`Su LWN c'è una buona recensione`_ dei modelli elencati qui sopra e altri.
+La scelta dipenderà dal costo, dalla disponibilità nella vostra area
+geografica e vostre considerazioni sull'hardware aperto/proprietario.
+
Se volete usare chiavi ECC, la vostra migliore scelta sul mercato è la
Nitrokey Start.
.. _`Nitrokey Start`: https://shop.nitrokey.com/shop/product/nitrokey-start-6
-.. _`Nitrokey Pro`: https://shop.nitrokey.com/shop/product/nitrokey-pro-3
-.. _`Yubikey 4`: https://www.yubico.com/product/yubikey-4-series/
+.. _`Nitrokey Pro 2`: https://shop.nitrokey.com/shop/product/nitrokey-pro-2-3
+.. _`Yubikey 5`: https://www.yubico.com/product/yubikey-5-overview/
.. _Gnuk: http://www.fsij.org/doc-gnuk/
.. _`Su LWN c'è una buona recensione`: https://lwn.net/Articles/736231/
--
2.21.0
^ permalink raw reply related
* [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Sasha Levin @ 2019-07-17 23:11 UTC (permalink / raw)
To: corbet, solar
Cc: will, keescook, peterz, gregkh, tyhicks, linux-doc, linux-kernel,
Sasha Levin
Provide more information about how to interact with the linux-distros
mailing list for disclosing security bugs.
Reference the linux-distros list policy and clarify that the reporter
must read and understand those policies as they differ from
security@kernel.org's policy.
Suggested-by: Solar Designer <solar@openwall.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
Changes in v2:
- Focus more on pointing to the linux-distros wiki and policies.
- Remove explicit linux-distros email.
- Remove various explanations of linux-distros policies.
Documentation/admin-guide/security-bugs.rst | 19 +++++++++----------
1 file changed, 9 insertions(+), 10 deletions(-)
diff --git a/Documentation/admin-guide/security-bugs.rst b/Documentation/admin-guide/security-bugs.rst
index dcd6c93c7aac..380d44fd618d 100644
--- a/Documentation/admin-guide/security-bugs.rst
+++ b/Documentation/admin-guide/security-bugs.rst
@@ -60,16 +60,15 @@ Coordination
------------
Fixes for sensitive bugs, such as those that might lead to privilege
-escalations, may need to be coordinated with the private
-<linux-distros@vs.openwall.org> mailing list so that distribution vendors
-are well prepared to issue a fixed kernel upon public disclosure of the
-upstream fix. Distros will need some time to test the proposed patch and
-will generally request at least a few days of embargo, and vendor update
-publication prefers to happen Tuesday through Thursday. When appropriate,
-the security team can assist with this coordination, or the reporter can
-include linux-distros from the start. In this case, remember to prefix
-the email Subject line with "[vs]" as described in the linux-distros wiki:
-<http://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists>
+escalations, may need to be coordinated with the private linux-distros mailing
+list so that distribution vendors are well prepared to issue a fixed kernel
+upon public disclosure of the upstream fix. Please read and follow the policies
+of linux-distros as specified in the linux-distros wiki page before reporting:
+<https://oss-security.openwall.org/wiki/mailing-lists/distros>. When
+appropriate, the security team can assist with this coordination, or the
+reporter can include linux-distros from the start. In this case, remember to
+prefix the email Subject line with "[vs]" as described in the linux-distros
+wiki.
CVE assignment
--------------
--
2.20.1
^ permalink raw reply related
* Re: [PATCH v5 02/11] of/platform: Add functional dependency link from DT bindings
From: Saravana Kannan @ 2019-07-17 20:38 UTC (permalink / raw)
To: Rob Herring
Cc: Mark Rutland, Greg Kroah-Hartman, Rafael J. Wysocki, Frank Rowand,
Jonathan Corbet,
open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
linux-kernel@vger.kernel.org, David Collins, Android Kernel Team,
Linux Doc Mailing List
In-Reply-To: <CAL_Jsq+Dfm8ng1OVcB+1N2ack05v8+u1VydfxM4Ukefqd9XK2w@mail.gmail.com>
On Wed, Jul 17, 2019 at 7:35 AM Rob Herring <robh+dt@kernel.org> wrote:
>
> On Tue, Jul 16, 2019 at 5:54 PM Saravana Kannan <saravanak@google.com> wrote:
> >
> > On Tue, Jul 16, 2019 at 4:43 PM Rob Herring <robh+dt@kernel.org> wrote:
> > >
> > > On Fri, Jul 12, 2019 at 5:52 PM Saravana Kannan <saravanak@google.com> wrote:
> > > >
> > > > Add device-links after the devices are created (but before they are
> > > > probed) by looking at common DT bindings like clocks and
> > > > interconnects.
> > > >
> > > > Automatically adding device-links for functional dependencies at the
> > > > framework level provides the following benefits:
> > > >
> > > > - Optimizes device probe order and avoids the useless work of
> > > > attempting probes of devices that will not probe successfully
> > > > (because their suppliers aren't present or haven't probed yet).
> > > >
> > > > For example, in a commonly available mobile SoC, registering just
> > > > one consumer device's driver at an initcall level earlier than the
> > > > supplier device's driver causes 11 failed probe attempts before the
> > > > consumer device probes successfully. This was with a kernel with all
> > > > the drivers statically compiled in. This problem gets a lot worse if
> > > > all the drivers are loaded as modules without direct symbol
> > > > dependencies.
> > > >
> > > > - Supplier devices like clock providers, interconnect providers, etc
> > > > need to keep the resources they provide active and at a particular
> > > > state(s) during boot up even if their current set of consumers don't
> > > > request the resource to be active. This is because the rest of the
> > > > consumers might not have probed yet and turning off the resource
> > > > before all the consumers have probed could lead to a hang or
> > > > undesired user experience.
> > > >
> > > > Some frameworks (Eg: regulator) handle this today by turning off
> > > > "unused" resources at late_initcall_sync and hoping all the devices
> > > > have probed by then. This is not a valid assumption for systems with
> > > > loadable modules. Other frameworks (Eg: clock) just don't handle
> > > > this due to the lack of a clear signal for when they can turn off
> > > > resources. This leads to downstream hacks to handle cases like this
> > > > that can easily be solved in the upstream kernel.
> > > >
> > > > By linking devices before they are probed, we give suppliers a clear
> > > > count of the number of dependent consumers. Once all of the
> > > > consumers are active, the suppliers can turn off the unused
> > > > resources without making assumptions about the number of consumers.
> > > >
> > > > By default we just add device-links to track "driver presence" (probe
> > > > succeeded) of the supplier device. If any other functionality provided
> > > > by device-links are needed, it is left to the consumer/supplier
> > > > devices to change the link when they probe.
> > > >
> > > > Signed-off-by: Saravana Kannan <saravanak@google.com>
> > > > ---
> > > > .../admin-guide/kernel-parameters.txt | 5 ++
> > > > drivers/of/platform.c | 57 +++++++++++++++++++
> > > > 2 files changed, 62 insertions(+)
> > > >
> > > > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > > > index 138f6664b2e2..109b4310844f 100644
> > > > --- a/Documentation/admin-guide/kernel-parameters.txt
> > > > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > > > @@ -3141,6 +3141,11 @@
> > > > This can be set from sysctl after boot.
> > > > See Documentation/sysctl/vm.txt for details.
> > > >
> > > > + of_devlink [KNL] Make device links from common DT bindings. Useful
> > > > + for optimizing probe order and making sure resources
> > > > + aren't turned off before the consumer devices have
> > > > + probed.
> > > > +
> > > > ohci1394_dma=early [HW] enable debugging via the ohci1394 driver.
> > > > See Documentation/debugging-via-ohci1394.txt for more
> > > > info.
> > > > diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> > > > index 04ad312fd85b..0930f9f89571 100644
> > > > --- a/drivers/of/platform.c
> > > > +++ b/drivers/of/platform.c
> > > > @@ -509,6 +509,62 @@ int of_platform_default_populate(struct device_node *root,
> > > > }
> > > > EXPORT_SYMBOL_GPL(of_platform_default_populate);
> > > >
> > > > +static int of_link_binding(struct device *dev,
> > > > + const char *binding, const char *cell)
> > > > +{
> > > > + struct of_phandle_args sup_args;
> > > > + struct platform_device *sup_dev;
> > > > + unsigned int i = 0, links = 0;
> > > > + u32 dl_flags = DL_FLAG_AUTOPROBE_CONSUMER;
> > > > +
> > > > + while (!of_parse_phandle_with_args(dev->of_node, binding, cell, i,
> > > > + &sup_args)) {
> > > > + i++;
> > > > + sup_dev = of_find_device_by_node(sup_args.np);
> > > > + of_node_put(sup_args.np);
> > > > + if (!sup_dev)
> > > > + continue;
> > > > + if (device_link_add(dev, &sup_dev->dev, dl_flags))
> > > > + links++;
> > > > + put_device(&sup_dev->dev);
> > > > + }
> > > > + if (links < i)
> > > > + return -ENODEV;
> > > > + return 0;
> > > > +}
> > > > +
> > > > +static bool of_devlink;
> > > > +core_param(of_devlink, of_devlink, bool, 0);
> > > > +
> > > > +/*
> > > > + * List of bindings and their cell names (use NULL if no cell names) from which
> > > > + * device links need to be created.
> > > > + */
> > > > +static const char * const link_bindings[] = {
> > > > + "clocks", "#clock-cells",
> > > > + "interconnects", "#interconnect-cells",
> > > > +};
> > > > +
> > > > +static int of_link_to_suppliers(struct device *dev)
> > > > +{
> > > > + unsigned int i = 0;
> > > > + bool done = true;
> > > > +
> > > > + if (!of_devlink)
> > > > + return 0;
> > > > + if (unlikely(!dev->of_node))
> > > > + return 0;
> > > > +
> > > > + for (i = 0; i < ARRAY_SIZE(link_bindings) / 2; i++)
> > > > + if (of_link_binding(dev, link_bindings[i * 2],
> > > > + link_bindings[i * 2 + 1]))
> > > > + done = false;
> > >
> > > Given the pending addition of regulators I think this should be
> > > structured a bit differently so that we abstract out the matching and
> > > phandle look-up so there's a clean separation of binding specifics.
> > > It's kind of messy with 2 patterns to parse already and if we added a
> > > 3rd? I would iterate over the properties as you do for regulators in
> > > both cases and for each property call a binding specific match
> > > function. The common pattern can of course be a common function. Let
> > > me know if that makes sense. If not I can try to flesh it out some
> > > more.
> >
> > I've added regulator support in this series and I've refactored this
> > code as I went along. I fully expect to squash some of the refactors
> > once the final result of the series is acceptable.
>
> It would be easier to review the final result than incremental changes
> which change the prior patches especially if the latter patches are
> ultimately required.
>
> > It's not clear to me if you got to the end of the series and still
> > think the final result needs to be refactored. Let me know what you
> > think about this towards the end of the series and I can clean it up
> > if you still think it needs some clean up.
>
> I probably should have replied on the regulator addition, but yes,
> looking at that is what prompted my concerns here.
Sounds good. Let me refactor the series and send it out again. Btw, if
you have other issues you noticed, I'd be happy to fix those too
before sending out the v6 of the series again.
-Saravana
^ permalink raw reply
* Re: [RFC v1 0/4] arm64: MMU enabled kexec kernel relocation
From: Pavel Tatashin @ 2019-07-17 19:13 UTC (permalink / raw)
To: James Morse
Cc: James Morris, Sasha Levin, Eric W. Biederman, kexec mailing list,
LKML, Jonathan Corbet, Catalin Marinas, will,
Linux Doc Mailing List, Linux ARM
In-Reply-To: <4c8a3a11-adc2-efa4-f765-6be338546ae4@arm.com>
Hi James,
Thank you for taking a look at this work.
> After a quick skim:
>
> This will map 'nomap' regions of memory with cacheable attributes. This is a non-starter.
> These regions were described by firmware as having content that was/is written with
> different attributes. The attributes must match whenever it is mapped, otherwise we have a
> loss of coherency. Mapping this stuff as cacheable means the CPU can prefetch it into the
> cache whenever it likes.
> It may be important that we do not ever map some of these regions, even though its
> described as memory. On AMD-Seattle the bottom page of memory is reserved by firmware for
> its own use; it is made secure-only, and any access causes an
> external-abort/machine-check. UEFI describes this as 'Reserved', and we preserve this in
> the kernel as 'nomap'. The equivalent DT support uses memreserve, possibly with the
> 'nomap' attribute.
>
> Mapping a 'new'/unknown region with cacheable attributes can never be safe, even if we
> trusted kexec-tool to only write the kernel to memory. The host may be using a bigger page
> size causing more memory to become cacheable than was intended.
> Linux's EFI support rounds the UEFI memory map to the largest support page size, (and
> winges about firmware bugs).
> If we're allowing kexec to load images in a region not described as IORESOURCE_SYSTEM_RAM,
> that is a bug we should fix.
We are allowing this. If you consider this to be a bug, I will fix it,
and this will actually simplify the idmap page table. User will
receive an error during kexec load if a request is made to load into
!IORESOURCE_SYSTEM_RAM region.
>
> The only way to do this properly is to copy the linear mapping. The arch code has lots of
> complex code to generate it correctly at boot, we do not want to duplicate it.
> (this is why hibernate copies the linear mapping)
As I understand, you would like to take a copy of idmap page table,
and add entries only for segment
sources and destinations into the new page table?
If so, there is a slight problem: arch hook machine_kexec_prepare() is
called prior to loading segments from userland. We can solve this by
adding another hook that is called after kimage_terminate().
> These patches do not remove the running page tables from TTBR1. As you overwrite the live
> page tables you will corrupt the state of the CPU. The page-table walker may access things
> that aren't memory, cache memory that shouldn't be cached (see above), and allocate
> conflicting entries in the TLB.
Indeed. However, I was following what is done in create_safe_exec_page():
https://soleen.com/source/xref/linux/arch/arm64/kernel/hibernate.c?r=af873fce#263
ttbr1 is not removed there. Am I missing something, or is not yet
configured there?
I will set ttbr1 to zero page.
> You cannot use the mm page table helpers to build an idmap on arm64. The mm page table
> helpers have a compile-time VA_BITS, and we support systems where there is no memory below
> 1<<VA_BITS. (crazy huh!). Picking on AMD-Seattle again: if you boot a 4K 39bit VA kernel,
> the idmap will have more page table levels than the page table helpers can build. This is
> why there are special helpers to load the idmap, and twiddle TCR_EL1.T0SZ.
> You already need to copy the linear-map, so using an idmap is extra work. You want to work
> with linear-map addresses, you probably need to add the field to the appropriate structure.
OK. Makes sense. I will do the way hibernate setup this table. I was
indeed following x86, hoping that eventually it would be possible to
unite: kasan, hibernate, and kexec implementation of this page table.
>
> The kexec relocation code still runs at EL2. You can't use a copy of the linear map here
> as there is only one TTBR on v8.0, and you'd need to setup EL2 as its been torn back to
> the hyp-stub.
As I understand normally on baremetal kexec runs at EL1 not EL2. On my
machine is_kernel_in_hyp_mode() == false in cpu_soft_restart.
This is the reason hibernate posts EL2 in a holding pen while it rewrites
> all of memory, then calls back to fixup EL2. Keeping the rewrite phase at EL1 means it
> doesn't need independently tweaking/testing. You need to do something similar, either
> calling EL2 to start the new image, or disabling the MMU at EL1 to start the new image there.
OK, I will study how hibernate does this. I was thinking that if we
are running in EL2 we can simply configure TTBR0_EL2 instead of
TTBR0_EL1. But, I need to understand this better.
>
> You will need to alter the relocation code to do nothing for kdump, as no relocation is
> required and building page-tables is extra work where the kernel may croak, preventing us
> from reaching kdump.
Yes, I was planning to do nothing for kdump, which involves not
allocating page table. It is not part of the current patchest, as the
current series is not ready.
>
> Finally, having this independent idmap machinery isn't desirable from a maintenance
> perspective. Please start with the hibernate code that already solves a very similar
> problem, as it already has most of these problems covered.
OK.
> > This patch series works in terms, that I can kexec-reboot both in QEMU
>
> I wouldn't expect Qemu's emulation of the MMU and caches to be performance accurate.
I am not measuring performance in QEMU, I use it for
development/verification only. The performance is measured on real
hardware only.
>
> > and on a physical machine. However, I do not see performance improvement
> > during relocation. The performance is just as slow as before with disabled
> > caches.
>
> > Am I missing something? Perhaps, there is some flag that I should also
> > enable in page table? Please provide me with any suggestions.
>
> Some information about the physical machine you tested this on would help.
> I'm guessing its v8.0, and booted at EL2....
I am using Broadcom's Stingray SoC. Because is_kernel_in_hyp_mode()
returns false, I believe it is EL1. How can I boot it at EL2?
So, I am still confused why I do not see performance improvements
during relocation on this machine. Any theories?
Thank you,
Pasha
^ permalink raw reply
* Re: [RFC v1 0/4] arm64: MMU enabled kexec kernel relocation
From: James Morse @ 2019-07-17 17:51 UTC (permalink / raw)
To: Pavel Tatashin
Cc: jmorris, sashal, ebiederm, kexec, linux-kernel, corbet,
catalin.marinas, will, linux-doc, linux-arm-kernel
In-Reply-To: <20190716165641.6990-1-pasha.tatashin@soleen.com>
Hi Pavel,
On 16/07/2019 17:56, Pavel Tatashin wrote:
> Added identity mapped page table, and keep MMU enabled while
> kernel is being relocated from sparse pages to the final
> destination during kexec.
The 'tl;dr' version of this: I strongly urge you to start with the hibernate code that
already covers all these known corner cases. x86 was not a good starting point.
After a quick skim:
This will map 'nomap' regions of memory with cacheable attributes. This is a non-starter.
These regions were described by firmware as having content that was/is written with
different attributes. The attributes must match whenever it is mapped, otherwise we have a
loss of coherency. Mapping this stuff as cacheable means the CPU can prefetch it into the
cache whenever it likes.
It may be important that we do not ever map some of these regions, even though its
described as memory. On AMD-Seattle the bottom page of memory is reserved by firmware for
its own use; it is made secure-only, and any access causes an
external-abort/machine-check. UEFI describes this as 'Reserved', and we preserve this in
the kernel as 'nomap'. The equivalent DT support uses memreserve, possibly with the
'nomap' attribute.
Mapping a 'new'/unknown region with cacheable attributes can never be safe, even if we
trusted kexec-tool to only write the kernel to memory. The host may be using a bigger page
size causing more memory to become cacheable than was intended.
Linux's EFI support rounds the UEFI memory map to the largest support page size, (and
winges about firmware bugs).
If we're allowing kexec to load images in a region not described as IORESOURCE_SYSTEM_RAM,
that is a bug we should fix.
The only way to do this properly is to copy the linear mapping. The arch code has lots of
complex code to generate it correctly at boot, we do not want to duplicate it.
(this is why hibernate copies the linear mapping)
These patches do not remove the running page tables from TTBR1. As you overwrite the live
page tables you will corrupt the state of the CPU. The page-table walker may access things
that aren't memory, cache memory that shouldn't be cached (see above), and allocate
conflicting entries in the TLB.
You cannot use the mm page table helpers to build an idmap on arm64. The mm page table
helpers have a compile-time VA_BITS, and we support systems where there is no memory below
1<<VA_BITS. (crazy huh!). Picking on AMD-Seattle again: if you boot a 4K 39bit VA kernel,
the idmap will have more page table levels than the page table helpers can build. This is
why there are special helpers to load the idmap, and twiddle TCR_EL1.T0SZ.
You already need to copy the linear-map, so using an idmap is extra work. You want to work
with linear-map addresses, you probably need to add the field to the appropriate structure.
The kexec relocation code still runs at EL2. You can't use a copy of the linear map here
as there is only one TTBR on v8.0, and you'd need to setup EL2 as its been torn back to
the hyp-stub. This is the reason hibernate posts EL2 in a holding pen while it rewrites
all of memory, then calls back to fixup EL2. Keeping the rewrite phase at EL1 means it
doesn't need independently tweaking/testing. You need to do something similar, either
calling EL2 to start the new image, or disabling the MMU at EL1 to start the new image there.
You will need to alter the relocation code to do nothing for kdump, as no relocation is
required and building page-tables is extra work where the kernel may croak, preventing us
from reaching kdump.
Finally, having this independent idmap machinery isn't desirable from a maintenance
perspective. Please start with the hibernate code that already solves a very similar
problem, as it already has most of these problems covered.
> This patch series works in terms, that I can kexec-reboot both in QEMU
I wouldn't expect Qemu's emulation of the MMU and caches to be performance accurate.
> and on a physical machine. However, I do not see performance improvement
> during relocation. The performance is just as slow as before with disabled
> caches.
> Am I missing something? Perhaps, there is some flag that I should also
> enable in page table? Please provide me with any suggestions.
Some information about the physical machine you tested this on would help.
I'm guessing its v8.0, and booted at EL2....
Thanks,
James
^ permalink raw reply
* Re: [PATCH v4 13/15] docs: ABI: testing: make the files compatible with ReST output
From: Jonathan Cameron @ 2019-07-17 16:13 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: gregkh, Rafael J. Wysocki, Len Brown, Jonathan Cameron,
Hartmut Knaack, Lars-Peter Clausen, Peter Meerwald-Stadler,
Peter Rosin, Benson Leung, Enric Balletbo i Serra, Guenter Roeck,
Maxime Coquelin, Alexandre Torgue, Fabrice Gasnier,
Frederic Barrat, Andrew Donnellan, Sebastian Reichel,
Heikki Krogerus, Boris Ostrovsky, Juergen Gross,
Stefano Stabellini, Mike Kravetz, Nicolas Ferre,
Alexandre Belloni, Ludovic Desroches, Richard Cochran,
Jonathan Corbet, linux-acpi, linux-iio, linux-stm32,
linux-arm-kernel, linuxppc-dev, linux-pm, linux-usb, xen-devel,
linux-mm, netdev, linux-doc
In-Reply-To: <88d15fa38167e3f2e73e65e1c1a1f39bca0267b4.1563365880.git.mchehab+samsung@kernel.org>
On Wed, 17 Jul 2019 09:28:17 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> Some files over there won't parse well by Sphinx.
>
> Fix them.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
Hi Mauro,
Does feel like this one should perhaps have been broken up a touch!
For the IIO ones I've eyeballed it rather than testing the results
Acked-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
^ permalink raw reply
* Re: [PATCH v5 02/11] of/platform: Add functional dependency link from DT bindings
From: Rob Herring @ 2019-07-17 14:35 UTC (permalink / raw)
To: Saravana Kannan
Cc: Mark Rutland, Greg Kroah-Hartman, Rafael J. Wysocki, Frank Rowand,
Jonathan Corbet,
open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
linux-kernel@vger.kernel.org, David Collins, Android Kernel Team,
Linux Doc Mailing List
In-Reply-To: <CAGETcx-5ykD=9X1Lo2-G+T5uokFncbY2FmiJM8eZrgQ9JaBgxw@mail.gmail.com>
On Tue, Jul 16, 2019 at 5:54 PM Saravana Kannan <saravanak@google.com> wrote:
>
> On Tue, Jul 16, 2019 at 4:43 PM Rob Herring <robh+dt@kernel.org> wrote:
> >
> > On Fri, Jul 12, 2019 at 5:52 PM Saravana Kannan <saravanak@google.com> wrote:
> > >
> > > Add device-links after the devices are created (but before they are
> > > probed) by looking at common DT bindings like clocks and
> > > interconnects.
> > >
> > > Automatically adding device-links for functional dependencies at the
> > > framework level provides the following benefits:
> > >
> > > - Optimizes device probe order and avoids the useless work of
> > > attempting probes of devices that will not probe successfully
> > > (because their suppliers aren't present or haven't probed yet).
> > >
> > > For example, in a commonly available mobile SoC, registering just
> > > one consumer device's driver at an initcall level earlier than the
> > > supplier device's driver causes 11 failed probe attempts before the
> > > consumer device probes successfully. This was with a kernel with all
> > > the drivers statically compiled in. This problem gets a lot worse if
> > > all the drivers are loaded as modules without direct symbol
> > > dependencies.
> > >
> > > - Supplier devices like clock providers, interconnect providers, etc
> > > need to keep the resources they provide active and at a particular
> > > state(s) during boot up even if their current set of consumers don't
> > > request the resource to be active. This is because the rest of the
> > > consumers might not have probed yet and turning off the resource
> > > before all the consumers have probed could lead to a hang or
> > > undesired user experience.
> > >
> > > Some frameworks (Eg: regulator) handle this today by turning off
> > > "unused" resources at late_initcall_sync and hoping all the devices
> > > have probed by then. This is not a valid assumption for systems with
> > > loadable modules. Other frameworks (Eg: clock) just don't handle
> > > this due to the lack of a clear signal for when they can turn off
> > > resources. This leads to downstream hacks to handle cases like this
> > > that can easily be solved in the upstream kernel.
> > >
> > > By linking devices before they are probed, we give suppliers a clear
> > > count of the number of dependent consumers. Once all of the
> > > consumers are active, the suppliers can turn off the unused
> > > resources without making assumptions about the number of consumers.
> > >
> > > By default we just add device-links to track "driver presence" (probe
> > > succeeded) of the supplier device. If any other functionality provided
> > > by device-links are needed, it is left to the consumer/supplier
> > > devices to change the link when they probe.
> > >
> > > Signed-off-by: Saravana Kannan <saravanak@google.com>
> > > ---
> > > .../admin-guide/kernel-parameters.txt | 5 ++
> > > drivers/of/platform.c | 57 +++++++++++++++++++
> > > 2 files changed, 62 insertions(+)
> > >
> > > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > > index 138f6664b2e2..109b4310844f 100644
> > > --- a/Documentation/admin-guide/kernel-parameters.txt
> > > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > > @@ -3141,6 +3141,11 @@
> > > This can be set from sysctl after boot.
> > > See Documentation/sysctl/vm.txt for details.
> > >
> > > + of_devlink [KNL] Make device links from common DT bindings. Useful
> > > + for optimizing probe order and making sure resources
> > > + aren't turned off before the consumer devices have
> > > + probed.
> > > +
> > > ohci1394_dma=early [HW] enable debugging via the ohci1394 driver.
> > > See Documentation/debugging-via-ohci1394.txt for more
> > > info.
> > > diff --git a/drivers/of/platform.c b/drivers/of/platform.c
> > > index 04ad312fd85b..0930f9f89571 100644
> > > --- a/drivers/of/platform.c
> > > +++ b/drivers/of/platform.c
> > > @@ -509,6 +509,62 @@ int of_platform_default_populate(struct device_node *root,
> > > }
> > > EXPORT_SYMBOL_GPL(of_platform_default_populate);
> > >
> > > +static int of_link_binding(struct device *dev,
> > > + const char *binding, const char *cell)
> > > +{
> > > + struct of_phandle_args sup_args;
> > > + struct platform_device *sup_dev;
> > > + unsigned int i = 0, links = 0;
> > > + u32 dl_flags = DL_FLAG_AUTOPROBE_CONSUMER;
> > > +
> > > + while (!of_parse_phandle_with_args(dev->of_node, binding, cell, i,
> > > + &sup_args)) {
> > > + i++;
> > > + sup_dev = of_find_device_by_node(sup_args.np);
> > > + of_node_put(sup_args.np);
> > > + if (!sup_dev)
> > > + continue;
> > > + if (device_link_add(dev, &sup_dev->dev, dl_flags))
> > > + links++;
> > > + put_device(&sup_dev->dev);
> > > + }
> > > + if (links < i)
> > > + return -ENODEV;
> > > + return 0;
> > > +}
> > > +
> > > +static bool of_devlink;
> > > +core_param(of_devlink, of_devlink, bool, 0);
> > > +
> > > +/*
> > > + * List of bindings and their cell names (use NULL if no cell names) from which
> > > + * device links need to be created.
> > > + */
> > > +static const char * const link_bindings[] = {
> > > + "clocks", "#clock-cells",
> > > + "interconnects", "#interconnect-cells",
> > > +};
> > > +
> > > +static int of_link_to_suppliers(struct device *dev)
> > > +{
> > > + unsigned int i = 0;
> > > + bool done = true;
> > > +
> > > + if (!of_devlink)
> > > + return 0;
> > > + if (unlikely(!dev->of_node))
> > > + return 0;
> > > +
> > > + for (i = 0; i < ARRAY_SIZE(link_bindings) / 2; i++)
> > > + if (of_link_binding(dev, link_bindings[i * 2],
> > > + link_bindings[i * 2 + 1]))
> > > + done = false;
> >
> > Given the pending addition of regulators I think this should be
> > structured a bit differently so that we abstract out the matching and
> > phandle look-up so there's a clean separation of binding specifics.
> > It's kind of messy with 2 patterns to parse already and if we added a
> > 3rd? I would iterate over the properties as you do for regulators in
> > both cases and for each property call a binding specific match
> > function. The common pattern can of course be a common function. Let
> > me know if that makes sense. If not I can try to flesh it out some
> > more.
>
> I've added regulator support in this series and I've refactored this
> code as I went along. I fully expect to squash some of the refactors
> once the final result of the series is acceptable.
It would be easier to review the final result than incremental changes
which change the prior patches especially if the latter patches are
ultimately required.
> It's not clear to me if you got to the end of the series and still
> think the final result needs to be refactored. Let me know what you
> think about this towards the end of the series and I can clean it up
> if you still think it needs some clean up.
I probably should have replied on the regulator addition, but yes,
looking at that is what prompted my concerns here.
Rob
^ permalink raw reply
* If interested
From: Tk Allen @ 2019-07-17 13:32 UTC (permalink / raw)
To: all.en.m61.2.0
My name is Mr. Allen, I have a Business Proposal of Four million five hundred thousand united states dollars for you to handle with me from my bank. I will need you to assist me in executing this Business .
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox