* Re: [PATCH v2 08/11] kbuild: create *.mod with full directory path and remove MODVERDIR
From: Masahiro Yamada @ 2019-07-20 5:09 UTC (permalink / raw)
To: Joe Lawrence
Cc: Linux Kbuild mailing list, Sam Ravnborg, Nicolas Pitre,
open list:DOCUMENTATION, Jonathan Corbet,
Linux Kernel Mailing List, Michal Marek
In-Reply-To: <f1222c8a-9301-1e76-981d-a36e8687a29f@redhat.com>
Hi Joe,
On Fri, Jul 19, 2019 at 5:18 AM Joe Lawrence <joe.lawrence@redhat.com> wrote:
> > Perhaps, adding a new field
> > to *.mod files might be cleaner.
>
> I can look into that. By "field" you mean a new row in the file?
Yes.
--
Best Regards
Masahiro Yamada
^ permalink raw reply
* Re: [PATCH 4/4] Documentation:kernel-per-CPU-kthreads.txt: Remove reference to elevator=
From: Hannes Reinecke @ 2019-07-19 13:37 UTC (permalink / raw)
To: Marcos Paulo de Souza, linux-kernel
Cc: linux-block, linux-doc, Jonathan Corbet
In-Reply-To: <20190714053453.1655-5-marcos.souza.org@gmail.com>
On 7/14/19 7:34 AM, Marcos Paulo de Souza wrote:
> This argument was not being considered since blk-mq was set by default,
> so removed this documentation to avoid confusion.
>
> Signed-off-by: Marcos Paulo de Souza <marcos.souza.org@gmail.com>
> ---
> Documentation/kernel-per-CPU-kthreads.txt | 8 +++-----
> 1 file changed, 3 insertions(+), 5 deletions(-)
>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
^ permalink raw reply
* Re: [PATCH 3/4] Documenation: switching-sched: Remove notes about elevator argument
From: Hannes Reinecke @ 2019-07-19 13:37 UTC (permalink / raw)
To: Marcos Paulo de Souza, linux-kernel
Cc: linux-block, linux-doc, Jonathan Corbet, Jens Axboe,
Andreas Herrmann
In-Reply-To: <20190714053453.1655-4-marcos.souza.org@gmail.com>
On 7/14/19 7:34 AM, Marcos Paulo de Souza wrote:
> This argument was ignored since blk-mq was set as default, so remove it
> from documentation.
>
> Signed-off-by: Marcos Paulo de Souza <marcos.souza.org@gmail.com>
> ---
> Documentation/block/switching-sched.txt | 4 ----
> 1 file changed, 4 deletions(-)
>
> diff --git a/Documentation/block/switching-sched.txt b/Documentation/block/switching-sched.txt
> index 7977f6fb8b20..431d56471227 100644
> --- a/Documentation/block/switching-sched.txt
> +++ b/Documentation/block/switching-sched.txt
> @@ -1,7 +1,3 @@
> -To choose IO schedulers at boot time, use the argument 'elevator=deadline'.
> -'noop' and 'cfq' (the default) are also available. IO schedulers are assigned
> -globally at boot time only presently.
> -
> Each io queue has a set of io scheduler tunables associated with it. These
> tunables control how the io scheduler works. You can find these entries
> in:
>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
^ permalink raw reply
* Re: [PATCH 2/4] kernel-parameters.txt: Remove elevator argument
From: Hannes Reinecke @ 2019-07-19 13:36 UTC (permalink / raw)
To: Marcos Paulo de Souza, linux-kernel
Cc: linux-block, linux-doc, Jonathan Corbet, Thomas Gleixner,
Kees Cook, Andrew Morton, Ingo Molnar, Mauro Carvalho Chehab,
Paul E. McKenney, Josh Poimboeuf, Logan Gunthorpe, Lu Baolu,
Andy Lutomirski
In-Reply-To: <20190714053453.1655-3-marcos.souza.org@gmail.com>
On 7/14/19 7:34 AM, Marcos Paulo de Souza wrote:
> This argument was not being used since the legacy IO path was removed,
> when blk-mq was enabled by default. So removed it from the kernel
> parameters documentation.
>
> Signed-off-by: Marcos Paulo de Souza <marcos.souza.org@gmail.com>
> ---
> Documentation/admin-guide/kernel-parameters.txt | 6 ------
> 1 file changed, 6 deletions(-)
>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
^ permalink raw reply
* Re: [PATCH 1/4] block: elevator.c: Remove now unused elevator= argument
From: Hannes Reinecke @ 2019-07-19 13:36 UTC (permalink / raw)
To: Marcos Paulo de Souza, linux-kernel; +Cc: linux-block, linux-doc, Jens Axboe
In-Reply-To: <20190714053453.1655-2-marcos.souza.org@gmail.com>
On 7/14/19 7:34 AM, Marcos Paulo de Souza wrote:
> Since the inclusion of blk-mq, elevator argument was not being
> considered anymore, and it's utility died long with the legacy IO path,
> now removed too.
>
> Signed-off-by: Marcos Paulo de Souza <marcos.souza.org@gmail.com>
> ---
> block/elevator.c | 14 --------------
> 1 file changed, 14 deletions(-)
>
> diff --git a/block/elevator.c b/block/elevator.c
> index 2f17d66d0e61..f56d9c7d5cbc 100644
> --- a/block/elevator.c
> +++ b/block/elevator.c
> @@ -135,20 +135,6 @@ static struct elevator_type *elevator_get(struct request_queue *q,
> return e;
> }
>
> -static char chosen_elevator[ELV_NAME_MAX];
> -
> -static int __init elevator_setup(char *str)
> -{
> - /*
> - * Be backwards-compatible with previous kernels, so users
> - * won't get the wrong elevator.
> - */
> - strncpy(chosen_elevator, str, sizeof(chosen_elevator) - 1);
> - return 1;
> -}
> -
> -__setup("elevator=", elevator_setup);
> -
> static struct kobj_type elv_ktype;
>
> struct elevator_queue *elevator_alloc(struct request_queue *q,
>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
^ permalink raw reply
* Re: [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Solar Designer @ 2019-07-19 8:42 UTC (permalink / raw)
To: Kees Cook
Cc: Sasha Levin, corbet, will, peterz, gregkh, tyhicks, linux-doc,
linux-kernel
In-Reply-To: <201907181833.EF0D93C@keescook>
On Thu, Jul 18, 2019 at 06:51:07PM -0700, Kees Cook wrote:
> On Thu, Jul 18, 2019 at 08:39:19PM -0400, Sasha Levin wrote:
> > On Thu, Jul 18, 2019 at 03:00:55PM -0700, Kees Cook wrote:
> > > On Wed, Jul 17, 2019 at 07:11:03PM -0400, Sasha Levin wrote:
> > > > Provide more information about how to interact with the linux-distros
> > > > mailing list for disclosing security bugs.
> > > >
> > > > Reference the linux-distros list policy and clarify that the reporter
> > > > must read and understand those policies as they differ from
> > > > security@kernel.org's policy.
> > > >
> > > > Suggested-by: Solar Designer <solar@openwall.com>
> > > > Signed-off-by: Sasha Levin <sashal@kernel.org>
> > >
> > > Sorry, but NACK, see below...
I like Sasha's PATCH v2 better, but if Kees insists on NACK'ing it then
I suggest that we apply Sasha's first revision of the patch instead.
I think either revision is an improvement on the status quo.
> I think reinforcing information to avoid past mistakes is appropriate
> here.
Maybe, but from my perspective common past issues with Linux kernel bugs
reported to linux-distros were:
- The reporter having been directed to post from elsewhere (and I
suspect this documentation file) without being aware of list policy.
- The reporter not mentioning (and sometimes not replying even when
asked) whether they're also coordinating with security@k.o or whether
they want someone on linux-distros to help coordinate with security@k.o.
(Maybe this is something we want to write about here.)
- The Linux kernel bug having been introduced too recently to be of much
interest to distros.
> Reports have regularly missed the "[vs]" detail or suggested
> embargoes that ended on Fridays, etc.
This happens too. Regarding missing the "[vs]" detail, technically
there are also a number of other conditions that also let the message
through, but those are changing and are deliberately not advertised.
> Sending to the distros@ list risks exposing Linux-only flaws to non-Linux
> distros.
Right.
> This has caused leaks in the past
Do you mean leaks to *BSD security teams or to the public? I'm not
aware of past leaks to the public via the non-Linux distros present on
the distros@ list. Are you?
Alexander
^ permalink raw reply
* Re: [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Sasha Levin @ 2019-07-19 3:41 UTC (permalink / raw)
To: Kees Cook
Cc: corbet, solar, will, peterz, gregkh, tyhicks, linux-doc,
linux-kernel
In-Reply-To: <201907181833.EF0D93C@keescook>
On Thu, Jul 18, 2019 at 06:51:07PM -0700, Kees Cook wrote:
>On Thu, Jul 18, 2019 at 08:39:19PM -0400, Sasha Levin wrote:
>> On Thu, Jul 18, 2019 at 03:00:55PM -0700, Kees Cook wrote:
>> > On Wed, Jul 17, 2019 at 07:11:03PM -0400, Sasha Levin wrote:
>> > > Provide more information about how to interact with the linux-distros
>> > > mailing list for disclosing security bugs.
>> > >
>> > > Reference the linux-distros list policy and clarify that the reporter
>> > > must read and understand those policies as they differ from
>> > > security@kernel.org's policy.
>> > >
>> > > Suggested-by: Solar Designer <solar@openwall.com>
>> > > Signed-off-by: Sasha Levin <sashal@kernel.org>
>> >
>> > Sorry, but NACK, see below...
>> >
>> > > ---
>> > >
>> > > Changes in v2:
>> > > - Focus more on pointing to the linux-distros wiki and policies.
>> >
>> > I think this is already happening in the text. What specifically do you
>> > want described differently?
>>
>> The main issue was that there isn't anything pointing to the
>> linux-distros policies. The current text outlines a few of them ("add
>> [vs]", and "there should be an embargo period"), but it effectively just
>> gives out the linux-distros mailing address and tells the reporter to
>> contact it.
>
>The current text includes the wiki link, but yes, the anchor tag is not
>present at the wiki anymore. I would agree that's due for updating.
>
>I think reinforcing information to avoid past mistakes is appropriate
>here. Reports have regularly missed the "[vs]" detail or suggested
>embargoes that ended on Fridays, etc.
Right, but this is a sign that the reporter didn't read the wiki.
Explaining things like this encourages reporters to skip reading the
wiki and just send their report out.
>> > > - Remove explicit linux-distros email.
>> >
>> > I don't like this because we had past trouble with notifications going
>> > to the distros@ list and leaking Linux-only flaws to the BSDs. As there
>> > isn't a separate linux-distros wiki, the clarification of WHICH list is
>> > needed.
>>
>> Why would removing the explicit linux-distros email encourage people to
>> send reports to it?
>
>What? No, I'm saying we should _keep_ linux-distros@... in our text so
>that people don't send to the wrong list.
But doesn't this just encourage mails being sent to linux-distros@
without the policies being followed? That was Alexander's concern at
least.
>> I also don't understand what you mean by "there isn't a separate
>> linux-distros wiki"? There is one, and I want to point the reporter
>> there.
>
>That URL is a combined page for two lists. The very fact that it's
>not obvious that there are two lists described there is exactly why I
>think we need to keep an explicit mention of which to use. There are
>two mailing lists described at the wiki URL:
>
> distros@lists.openwall.com
> linux-distros@lists.openwall.com
>
>Sending to the distros@ list risks exposing Linux-only flaws to non-Linux
>distros. This has caused leaks in the past, and we do not want people
>guessing at which list they should use.
>
>Also note that nowhere on the openwall wiki is the email address
>actually spelled out; this is another reason to spell it out in our
>documentation: no misunderstanding.
>
>(And historically there WAS a specific linux-distros wiki:
>https://oss-security.openwall.org/wiki/mailing-lists/linux-distros
>but it redirects to the combined one now...)
>
>> > > - Remove various explanations of linux-distros policies.
>> >
>> > I don't think there's value in removing the Tue-Thu comment, nor
>> > providing context for why distros need time. This has been a regular
>> > thing we've had to explain to researchers that aren't familiar with
>> > update procedures and publication timing.
>>
>> To be fair, the Tue-Thu comment is listed in the section describing how
>> to do coordination with linux-distros, and linux-distros don't have a
>> Tue-Thu policy. If it's a security@kernel.org policy then let's list it
>> elsewhere.
>
>It's a distro preference. Many researchers aren't thinking about the
>larger Linux ecosystem that has to consume fixes. It's not a _policy_,
>but it makes the researchers understand how to construct better embargoes.
If it's an accepted preference then we should just document it in a few
other places like the linux-distros@ wiki. My concern with this is that
it's not, and it's actually one of the only things Alexander pointed out
in this document as surprising.
--
Thanks,
Sasha
>> If you feel that there is a consensus around Tue-Thu let's just add it
>> to the linux-distros policy wiki, there's no point in listing random
>> policies from that wiki.
>
>I think it'd be a good idea to add that note also to the wiki, but I
>don't want it removed from our text because I have had to repeat that
>information regularly in the past.
>
>--
>Kees Cook
^ permalink raw reply
* Re: Using rst2pdf for PDF output - Was: Re: [PATCH 0/5] PDF output fixes
From: Mauro Carvalho Chehab @ 2019-07-19 2:44 UTC (permalink / raw)
To: Markus Heiser; +Cc: Jonathan Corbet, linux-doc
In-Reply-To: <04a1a65f-c96c-1f4a-d987-d8b9e605d7c1@darmarit.de>
Em Thu, 18 Jul 2019 19:56:57 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:
> Am 18.07.19 um 19:46 schrieb Mauro Carvalho Chehab:
> > Em Sat, 13 Jul 2019 00:41:25 -0300
> > Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> >
> >> Em Fri, 12 Jul 2019 19:27:05 -0300
> >> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> >>
> >>> Em Fri, 12 Jul 2019 14:19:21 -0600
> >>> Jonathan Corbet <corbet@lwn.net> escreveu:
> >>>
> >>>> Can't you just make rst2pdf work instead? :)
> >>>
> >>> Well, we can try.
>
> Thanks a lot for your investigation on this. I also checked the rst2pdf sources
> a while ago, for me it was crap with crap requirements [1] .. my tip: don't
> waste to much time on it.
>
> [1] https://github.com/mchehab/rst2pdf/blob/master/requirements.txt#L31
>
Yeah, a simple test shows that this upstream rst2pdf + latest reportlab has
some issues.
Running this:
$ rst2pdf Documentation/process/license-rules.rst
causes an error with reportlab-3.5.23. It has to be downgraded to version
3.4 in order to avoid this error:
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/reportlab/platypus/doctemplate.py", line 651, in handle_pageEnd
raise LayoutError(ident)
reportlab.platypus.doctemplate.LayoutError: More than 10 pages generated without content - halting layout. Likely that a flowable is too large for any frame.
Another solution would be to do this:
diff --git a/Documentation/process/license-rules.rst b/Documentation/process/license-rules.rst
index 2ef44ada3f11..19a480ebd69a 100644
--- a/Documentation/process/license-rules.rst
+++ b/Documentation/process/license-rules.rst
@@ -452,7 +452,10 @@ _`MODULE_LICENSE`
module source is dual licensed under a
GPL v2 variant and MIT license. Please do
not use in new code.
+ ============================= =============================================
+
+ ============================= =============================================
"Dual MIT/GPL" The correct way of expressing that the
module is dual licensed under a GPL v2
variant or MIT license choice.
But it sucks needing to break long tables because reportlab's handling
for big tables is broken.
Thanks,
Mauro
^ permalink raw reply related
* Re: [PATCH v3 1/2] f2fs: include charset encoding information in the superblock
From: Chao Yu @ 2019-07-19 2:03 UTC (permalink / raw)
To: Daniel Rosenberg, Jaegeuk Kim, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team
In-Reply-To: <20190719000322.106163-2-drosen@google.com>
On 2019/7/19 8:03, Daniel Rosenberg wrote:
> Add charset encoding to f2fs to support casefolding. It is modeled after
> the same feature introduced in commit c83ad55eaa91 ("ext4: include charset
> encoding information in the superblock")
>
> Currently this is not compatible with encryption, similar to the current
> ext4 imlpementation. This will change in the future.
>
>>From the ext4 patch:
> """
> The s_encoding field stores a magic number indicating the encoding
> format and version used globally by file and directory names in the
> filesystem. The s_encoding_flags defines policies for using the charset
> encoding, like how to handle invalid sequences. The magic number is
> mapped to the exact charset table, but the mapping is specific to ext4.
> Since we don't have any commitment to support old encodings, the only
> encoding I am supporting right now is utf8-12.1.0.
>
> The current implementation prevents the user from enabling encoding and
> per-directory encryption on the same filesystem at the same time. The
> incompatibility between these features lies in how we do efficient
> directory searches when we cannot be sure the encryption of the user
> provided fname will match the actual hash stored in the disk without
> decrypting every directory entry, because of normalization cases. My
> quickest solution is to simply block the concurrent use of these
> features for now, and enable it later, once we have a better solution.
> """
>
> Signed-off-by: Daniel Rosenberg <drosen@google.com>
> ---
> fs/f2fs/f2fs.h | 6 +++
> fs/f2fs/super.c | 94 +++++++++++++++++++++++++++++++++++++++++
> fs/f2fs/sysfs.c | 23 ++++++++++
> include/linux/f2fs_fs.h | 9 +++-
> 4 files changed, 131 insertions(+), 1 deletion(-)
>
> diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
> index 17382da7f0bd9..c6c7904572d0d 100644
> --- a/fs/f2fs/f2fs.h
> +++ b/fs/f2fs/f2fs.h
> @@ -153,6 +153,7 @@ struct f2fs_mount_info {
> #define F2FS_FEATURE_LOST_FOUND 0x0200
> #define F2FS_FEATURE_VERITY 0x0400 /* reserved */
> #define F2FS_FEATURE_SB_CHKSUM 0x0800
> +#define F2FS_FEATURE_CASEFOLD 0x1000
>
> #define __F2FS_HAS_FEATURE(raw_super, mask) \
> ((raw_super->feature & cpu_to_le32(mask)) != 0)
> @@ -1169,6 +1170,10 @@ struct f2fs_sb_info {
> int valid_super_block; /* valid super block no */
> unsigned long s_flag; /* flags for sbi */
> struct mutex writepages; /* mutex for writepages() */
> +#ifdef CONFIG_UNICODE
> + struct unicode_map *s_encoding;
> + __u16 s_encoding_flags;
> +#endif
>
> #ifdef CONFIG_BLK_DEV_ZONED
> unsigned int blocks_per_blkz; /* F2FS blocks per zone */
> @@ -3562,6 +3567,7 @@ F2FS_FEATURE_FUNCS(quota_ino, QUOTA_INO);
> F2FS_FEATURE_FUNCS(inode_crtime, INODE_CRTIME);
> F2FS_FEATURE_FUNCS(lost_found, LOST_FOUND);
> F2FS_FEATURE_FUNCS(sb_chksum, SB_CHKSUM);
> +F2FS_FEATURE_FUNCS(casefold, CASEFOLD);
>
> #ifdef CONFIG_BLK_DEV_ZONED
> static inline bool f2fs_blkz_is_seq(struct f2fs_sb_info *sbi, int devi,
> diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
> index 6de6cda440315..82f7da93c3ed1 100644
> --- a/fs/f2fs/super.c
> +++ b/fs/f2fs/super.c
> @@ -23,6 +23,7 @@
> #include <linux/f2fs_fs.h>
> #include <linux/sysfs.h>
> #include <linux/quota.h>
> +#include <linux/unicode.h>
>
> #include "f2fs.h"
> #include "node.h"
> @@ -222,6 +223,36 @@ void f2fs_printk(struct f2fs_sb_info *sbi, const char *fmt, ...)
> va_end(args);
> }
>
> +#ifdef CONFIG_UNICODE
> +static const struct f2fs_sb_encodings {
> + __u16 magic;
> + char *name;
> + char *version;
> +} f2fs_sb_encoding_map[] = {
> + {F2FS_ENC_UTF8_12_1, "utf8", "12.1.0"},
> +};
> +
> +static int f2fs_sb_read_encoding(const struct f2fs_super_block *sb,
> + const struct f2fs_sb_encodings **encoding,
> + __u16 *flags)
> +{
> + __u16 magic = le16_to_cpu(sb->s_encoding);
> + int i;
> +
> + for (i = 0; i < ARRAY_SIZE(f2fs_sb_encoding_map); i++)
> + if (magic == f2fs_sb_encoding_map[i].magic)
> + break;
> +
> + if (i >= ARRAY_SIZE(f2fs_sb_encoding_map))
> + return -EINVAL;
> +
> + *encoding = &f2fs_sb_encoding_map[i];
> + *flags = le16_to_cpu(sb->s_encoding_flags);
> +
> + return 0;
> +}
> +#endif
> +
> static inline void limit_reserve_root(struct f2fs_sb_info *sbi)
> {
> block_t limit = min((sbi->user_block_count << 1) / 1000,
> @@ -798,6 +829,13 @@ static int parse_options(struct super_block *sb, char *options)
> return -EINVAL;
> }
> #endif
> +#ifndef CONFIG_UNICODE
> + if (f2fs_sb_has_casefold(sbi)) {
> + f2fs_err(sbi,
> + "Filesystem with casefold feature cannot be mounted without CONFIG_UNICODE");
> + return -EINVAL;
> + }
> +#endif
>
> if (F2FS_IO_SIZE_BITS(sbi) && !test_opt(sbi, LFS)) {
> f2fs_err(sbi, "Should set mode=lfs with %uKB-sized IO",
> @@ -1089,6 +1127,9 @@ static void f2fs_put_super(struct super_block *sb)
> destroy_percpu_info(sbi);
> for (i = 0; i < NR_PAGE_TYPE; i++)
> kvfree(sbi->write_io[i]);
> +#ifdef CONFIG_UNICODE
> + utf8_unload(sbi->s_encoding);
> +#endif
> kvfree(sbi);
> }
>
> @@ -3031,6 +3072,51 @@ static int f2fs_scan_devices(struct f2fs_sb_info *sbi)
> return 0;
> }
>
> +static int f2fs_setup_casefold(struct f2fs_sb_info *sbi)
> +{
> +#ifdef CONFIG_UNICODE
> + if (f2fs_sb_has_casefold(sbi) && !sbi->s_encoding) {
> + const struct f2fs_sb_encodings *encoding_info;
> + struct unicode_map *encoding;
> + __u16 encoding_flags;
> +
> + if (f2fs_sb_has_encrypt(sbi)) {
> + f2fs_err(sbi,
> + "Can't mount with encoding and encryption");
> + return -EINVAL;
> + }
> +
> + if (f2fs_sb_read_encoding(sbi->raw_super, &encoding_info,
> + &encoding_flags)) {
> + f2fs_err(sbi,
> + "Encoding requested by superblock is unknown");
> + return -EINVAL;
> + }
> +
> + encoding = utf8_load(encoding_info->version);
> + if (IS_ERR(encoding)) {
> + f2fs_err(sbi,
> + "can't mount with superblock charset: %s-%s "
> + "not supported by the kernel. flags: 0x%x.",
> + encoding_info->name, encoding_info->version,
> + encoding_flags);
> + return PTR_ERR(encoding);
> + }
> + f2fs_info(sbi, "Using encoding defined by superblock: "
> + "%s-%s with flags 0x%hx", encoding_info->name,
> + encoding_info->version?:"\b", encoding_flags);
> +
> + sbi->s_encoding = encoding;
> + sbi->s_encoding_flags = encoding_flags;
> + }
> +#else
> + if (f2fs_sb_has_casefold(sbi)) {
> + f2fs_err(sbi, "Filesystem with casefold feature cannot be mounted without CONFIG_UNICODE");
> + return -EINVAL;
> + }
> +#endif
> +}
> +
> static void f2fs_tuning_parameters(struct f2fs_sb_info *sbi)
> {
> struct f2fs_sm_info *sm_i = SM_I(sbi);
> @@ -3127,6 +3213,10 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
> le32_to_cpu(raw_super->log_blocksize);
> sb->s_max_links = F2FS_LINK_MAX;
>
> + err = f2fs_setup_casefold(sbi);
> + if (err)
> + goto free_options;
> +
> #ifdef CONFIG_QUOTA
> sb->dq_op = &f2fs_quota_operations;
> sb->s_qcop = &f2fs_quotactl_ops;
> @@ -3477,6 +3567,10 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
> free_bio_info:
> for (i = 0; i < NR_PAGE_TYPE; i++)
> kvfree(sbi->write_io[i]);
> +
> +#ifdef CONFIG_UNICODE
> + utf8_unload(sbi->s_encoding);
> +#endif
> free_options:
> #ifdef CONFIG_QUOTA
> for (i = 0; i < MAXQUOTAS; i++)
> diff --git a/fs/f2fs/sysfs.c b/fs/f2fs/sysfs.c
> index 3aeacd0aacfd2..f9fcca695db9f 100644
> --- a/fs/f2fs/sysfs.c
> +++ b/fs/f2fs/sysfs.c
> @@ -10,6 +10,7 @@
> #include <linux/proc_fs.h>
> #include <linux/f2fs_fs.h>
> #include <linux/seq_file.h>
> +#include <linux/unicode.h>
>
> #include "f2fs.h"
> #include "segment.h"
> @@ -81,6 +82,19 @@ static ssize_t unusable_show(struct f2fs_attr *a,
> (unsigned long long)unusable);
> }
>
> +static ssize_t encoding_show(struct f2fs_attr *a,
> + struct f2fs_sb_info *sbi, char *buf)
> +{
> +#ifdef CONFIG_UNICODE
> + if (f2fs_sb_has_casefold(sbi))
> + return snprintf(buf, PAGE_SIZE, "%s (%d.%d.%d)\n",
> + sbi->s_encoding->charset,
> + (sbi->s_encoding->version >> 16) & 0xff,
> + (sbi->s_encoding->version >> 8) & 0xff,
> + sbi->s_encoding->version & 0xff);
> +#endif
> + return snprintf(buf, PAGE_SIZE, "(none)");
> +}
>
> static ssize_t lifetime_write_kbytes_show(struct f2fs_attr *a,
> struct f2fs_sb_info *sbi, char *buf)
> @@ -134,6 +148,9 @@ static ssize_t features_show(struct f2fs_attr *a,
> if (f2fs_sb_has_sb_chksum(sbi))
> len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
> len ? ", " : "", "sb_checksum");
> + if (f2fs_sb_has_casefold(sbi))
> + len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
> + len ? ", " : "", "casefold");
> len += snprintf(buf + len, PAGE_SIZE - len, "\n");
> return len;
> }
> @@ -365,6 +382,7 @@ enum feat_id {
> FEAT_INODE_CRTIME,
> FEAT_LOST_FOUND,
> FEAT_SB_CHECKSUM,
> + FEAT_CASEFOLD,
> };
>
> static ssize_t f2fs_feature_show(struct f2fs_attr *a,
> @@ -382,6 +400,7 @@ static ssize_t f2fs_feature_show(struct f2fs_attr *a,
> case FEAT_INODE_CRTIME:
> case FEAT_LOST_FOUND:
> case FEAT_SB_CHECKSUM:
> + case FEAT_CASEFOLD:
> return snprintf(buf, PAGE_SIZE, "supported\n");
> }
> return 0;
> @@ -455,6 +474,7 @@ F2FS_GENERAL_RO_ATTR(lifetime_write_kbytes);
> F2FS_GENERAL_RO_ATTR(features);
> F2FS_GENERAL_RO_ATTR(current_reserved_blocks);
> F2FS_GENERAL_RO_ATTR(unusable);
> +F2FS_GENERAL_RO_ATTR(encoding);
>
> #ifdef CONFIG_FS_ENCRYPTION
> F2FS_FEATURE_RO_ATTR(encryption, FEAT_CRYPTO);
> @@ -471,6 +491,7 @@ F2FS_FEATURE_RO_ATTR(quota_ino, FEAT_QUOTA_INO);
> F2FS_FEATURE_RO_ATTR(inode_crtime, FEAT_INODE_CRTIME);
> F2FS_FEATURE_RO_ATTR(lost_found, FEAT_LOST_FOUND);
> F2FS_FEATURE_RO_ATTR(sb_checksum, FEAT_SB_CHECKSUM);
> +F2FS_FEATURE_RO_ATTR(casefold, FEAT_CASEFOLD);
>
> #define ATTR_LIST(name) (&f2fs_attr_##name.attr)
> static struct attribute *f2fs_attrs[] = {
> @@ -515,6 +536,7 @@ static struct attribute *f2fs_attrs[] = {
> ATTR_LIST(features),
> ATTR_LIST(reserved_blocks),
> ATTR_LIST(current_reserved_blocks),
> + ATTR_LIST(encoding),
Let's document this entry in f2fs.txt and sysfs-fs-f2fs.
Thanks,
> NULL,
> };
> ATTRIBUTE_GROUPS(f2fs);
> @@ -535,6 +557,7 @@ static struct attribute *f2fs_feat_attrs[] = {
> ATTR_LIST(inode_crtime),
> ATTR_LIST(lost_found),
> ATTR_LIST(sb_checksum),
> + ATTR_LIST(casefold),
> NULL,
> };
> ATTRIBUTE_GROUPS(f2fs_feat);
> diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h
> index 65559900d4d76..b7c9c7f721339 100644
> --- a/include/linux/f2fs_fs.h
> +++ b/include/linux/f2fs_fs.h
> @@ -36,6 +36,11 @@
>
> #define F2FS_MAX_QUOTAS 3
>
> +#define F2FS_ENC_UTF8_12_1 1
> +#define F2FS_ENC_STRICT_MODE_FL (1 << 0)
> +#define f2fs_has_strict_mode(sbi) \
> + (sbi->s_encoding_flags & F2FS_ENC_STRICT_MODE_FL)
> +
> #define F2FS_IO_SIZE(sbi) (1 << F2FS_OPTION(sbi).write_io_size_bits) /* Blocks */
> #define F2FS_IO_SIZE_KB(sbi) (1 << (F2FS_OPTION(sbi).write_io_size_bits + 2)) /* KB */
> #define F2FS_IO_SIZE_BYTES(sbi) (1 << (F2FS_OPTION(sbi).write_io_size_bits + 12)) /* B */
> @@ -109,7 +114,9 @@ struct f2fs_super_block {
> struct f2fs_device devs[MAX_DEVICES]; /* device list */
> __le32 qf_ino[F2FS_MAX_QUOTAS]; /* quota inode numbers */
> __u8 hot_ext_count; /* # of hot file extension */
> - __u8 reserved[310]; /* valid reserved region */
> + __le16 s_encoding; /* Filename charset encoding */
> + __le16 s_encoding_flags; /* Filename charset encoding flags */
> + __u8 reserved[306]; /* valid reserved region */
> __le32 crc; /* checksum of superblock */
> } __packed;
>
>
^ permalink raw reply
* Re: [PATCH v3 2/2] f2fs: Support case-insensitive file name lookups
From: Chao Yu @ 2019-07-19 2:11 UTC (permalink / raw)
To: Daniel Rosenberg, Jaegeuk Kim, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team
In-Reply-To: <20190719000322.106163-3-drosen@google.com>
On 2019/7/19 8:03, Daniel Rosenberg wrote:
> Modeled after commit b886ee3e778e ("ext4: Support case-insensitive file
> name lookups")
>
> """
> This patch implements the actual support for case-insensitive file name
> lookups in f2fs, based on the feature bit and the encoding stored in the
> superblock.
>
> A filesystem that has the casefold feature set is able to configure
> directories with the +F (F2FS_CASEFOLD_FL) attribute, enabling lookups
> to succeed in that directory in a case-insensitive fashion, i.e: match
> a directory entry even if the name used by userspace is not a byte per
> byte match with the disk name, but is an equivalent case-insensitive
> version of the Unicode string. This operation is called a
> case-insensitive file name lookup.
>
> The feature is configured as an inode attribute applied to directories
> and inherited by its children. This attribute can only be enabled on
> empty directories for filesystems that support the encoding feature,
> thus preventing collision of file names that only differ by case.
>
> * dcache handling:
>
> For a +F directory, F2Fs only stores the first equivalent name dentry
> used in the dcache. This is done to prevent unintentional duplication of
> dentries in the dcache, while also allowing the VFS code to quickly find
> the right entry in the cache despite which equivalent string was used in
> a previous lookup, without having to resort to ->lookup().
>
> d_hash() of casefolded directories is implemented as the hash of the
> casefolded string, such that we always have a well-known bucket for all
> the equivalencies of the same string. d_compare() uses the
> utf8_strncasecmp() infrastructure, which handles the comparison of
> equivalent, same case, names as well.
>
> For now, negative lookups are not inserted in the dcache, since they
> would need to be invalidated anyway, because we can't trust missing file
> dentries. This is bad for performance but requires some leveraging of
> the vfs layer to fix. We can live without that for now, and so does
> everyone else.
>
> * on-disk data:
>
> Despite using a specific version of the name as the internal
> representation within the dcache, the name stored and fetched from the
> disk is a byte-per-byte match with what the user requested, making this
> implementation 'name-preserving'. i.e. no actual information is lost
> when writing to storage.
>
> DX is supported by modifying the hashes used in +F directories to make
> them case/encoding-aware. The new disk hashes are calculated as the
> hash of the full casefolded string, instead of the string directly.
> This allows us to efficiently search for file names in the htree without
> requiring the user to provide an exact name.
>
> * Dealing with invalid sequences:
>
> By default, when a invalid UTF-8 sequence is identified, ext4 will treat
> it as an opaque byte sequence, ignoring the encoding and reverting to
> the old behavior for that unique file. This means that case-insensitive
> file name lookup will not work only for that file. An optional bit can
> be set in the superblock telling the filesystem code and userspace tools
> to enforce the encoding. When that optional bit is set, any attempt to
> create a file name using an invalid UTF-8 sequence will fail and return
> an error to userspace.
>
> * Normalization algorithm:
>
> The UTF-8 algorithms used to compare strings in f2fs is implemented
> in fs/unicode, and is based on a previous version developed by
> SGI. It implements the Canonical decomposition (NFD) algorithm
> described by the Unicode specification 12.1, or higher, combined with
> the elimination of ignorable code points (NFDi) and full
> case-folding (CF) as documented in fs/unicode/utf8_norm.c.
>
> NFD seems to be the best normalization method for F2FS because:
>
> - It has a lower cost than NFC/NFKC (which requires
> decomposing to NFD as an intermediary step)
> - It doesn't eliminate important semantic meaning like
> compatibility decompositions.
>
> Although:
>
> - This implementation is not completely linguistic accurate, because
> different languages have conflicting rules, which would require the
> specialization of the filesystem to a given locale, which brings all
> sorts of problems for removable media and for users who use more than
> one language.
> """
>
> Signed-off-by: Daniel Rosenberg <drosen@google.com>
> ---
> fs/f2fs/dir.c | 126 +++++++++++++++++++++++++++++++++++++++++++----
> fs/f2fs/f2fs.h | 15 ++++--
> fs/f2fs/file.c | 9 ++++
> fs/f2fs/hash.c | 35 ++++++++++++-
> fs/f2fs/inline.c | 4 +-
> fs/f2fs/inode.c | 4 +-
> fs/f2fs/namei.c | 21 ++++++++
> fs/f2fs/super.c | 6 +++
> 8 files changed, 203 insertions(+), 17 deletions(-)
>
> diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c
> index 85a1528f319f2..2913483473f30 100644
> --- a/fs/f2fs/dir.c
> +++ b/fs/f2fs/dir.c
> @@ -8,6 +8,7 @@
> #include <linux/fs.h>
> #include <linux/f2fs_fs.h>
> #include <linux/sched/signal.h>
> +#include <linux/unicode.h>
> #include "f2fs.h"
> #include "node.h"
> #include "acl.h"
> @@ -81,7 +82,8 @@ static unsigned long dir_block_index(unsigned int level,
> return bidx;
> }
>
> -static struct f2fs_dir_entry *find_in_block(struct page *dentry_page,
> +static struct f2fs_dir_entry *find_in_block(struct inode *dir,
> + struct page *dentry_page,
> struct fscrypt_name *fname,
> f2fs_hash_t namehash,
> int *max_slots,
> @@ -93,7 +95,7 @@ static struct f2fs_dir_entry *find_in_block(struct page *dentry_page,
>
> dentry_blk = (struct f2fs_dentry_block *)page_address(dentry_page);
>
> - make_dentry_ptr_block(NULL, &d, dentry_blk);
> + make_dentry_ptr_block(dir, &d, dentry_blk);
> de = f2fs_find_target_dentry(fname, namehash, max_slots, &d);
> if (de)
> *res_page = dentry_page;
> @@ -101,6 +103,39 @@ static struct f2fs_dir_entry *find_in_block(struct page *dentry_page,
> return de;
> }
>
> +#ifdef CONFIG_UNICODE
> +/*
> + * Test whether a case-insensitive directory entry matches the filename
> + * being searched for.
> + *
> + * Returns: 0 if the directory entry matches, more than 0 if it
> + * doesn't match or less than zero on error.
> + */
> +int f2fs_ci_compare(const struct inode *parent, const struct qstr *name,
> + const struct qstr *entry)
> +{
> + const struct f2fs_sb_info *sbi = F2FS_SB(parent->i_sb);
> + const struct unicode_map *um = sbi->s_encoding;
> + int ret;
> +
> + ret = utf8_strncasecmp(um, name, entry);
> + if (ret < 0) {
> + /* Handle invalid character sequence as either an error
> + * or as an opaque byte sequence.
> + */
> + if (f2fs_has_strict_mode(sbi))
> + return -EINVAL;
> +
> + if (name->len != entry->len)
> + return 1;
> +
> + return !!memcmp(name->name, entry->name, name->len);
> + }
> +
> + return ret;
> +}
> +#endif
> +
> struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
> f2fs_hash_t namehash, int *max_slots,
> struct f2fs_dentry_ptr *d)
> @@ -108,6 +143,9 @@ struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
> struct f2fs_dir_entry *de;
> unsigned long bit_pos = 0;
> int max_len = 0;
> +#ifdef CONFIG_UNICODE
> + struct qstr entry;
> +#endif
>
> if (max_slots)
> *max_slots = 0;
> @@ -119,16 +157,28 @@ struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
> }
>
> de = &d->dentry[bit_pos];
> +#ifdef CONFIG_UNICODE
> + entry.name = d->filename[bit_pos];
> + entry.len = de->name_len;
> +#endif
>
> if (unlikely(!de->name_len)) {
> bit_pos++;
> continue;
> }
> + if (de->hash_code == namehash) {
> +#ifdef CONFIG_UNICODE
> + if (F2FS_SB(d->inode->i_sb)->s_encoding &&
> + IS_CASEFOLDED(d->inode) &&
> + !f2fs_ci_compare(d->inode,
> + fname->usr_fname, &entry))
> + goto found;
>
> - if (de->hash_code == namehash &&
> - fscrypt_match_name(fname, d->filename[bit_pos],
> - le16_to_cpu(de->name_len)))
> - goto found;
> +#endif
> + if (fscrypt_match_name(fname, d->filename[bit_pos],
> + le16_to_cpu(de->name_len)))
> + goto found;
> + }
>
> if (max_slots && max_len > *max_slots)
> *max_slots = max_len;
> @@ -157,7 +207,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
> struct f2fs_dir_entry *de = NULL;
> bool room = false;
> int max_slots;
> - f2fs_hash_t namehash = f2fs_dentry_hash(&name, fname);
> + f2fs_hash_t namehash = f2fs_dentry_hash(dir, &name, fname);
>
> nbucket = dir_buckets(level, F2FS_I(dir)->i_dir_level);
> nblock = bucket_blocks(level);
> @@ -179,8 +229,8 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
> }
> }
>
> - de = find_in_block(dentry_page, fname, namehash, &max_slots,
> - res_page);
> + de = find_in_block(dir, dentry_page, fname, namehash,
> + &max_slots, res_page);
> if (de)
> break;
>
> @@ -250,6 +300,14 @@ struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir,
> struct fscrypt_name fname;
> int err;
>
> +#ifdef CONFIG_UNICODE
> + if (f2fs_has_strict_mode(F2FS_I_SB(dir)) && IS_CASEFOLDED(dir) &&
> + utf8_validate(F2FS_I_SB(dir)->s_encoding, child)) {
> + *res_page = ERR_PTR(-EINVAL);
> + return NULL;
> + }
> +#endif
> +
> err = fscrypt_setup_filename(dir, child, 1, &fname);
> if (err) {
> if (err == -ENOENT)
> @@ -504,7 +562,7 @@ int f2fs_add_regular_entry(struct inode *dir, const struct qstr *new_name,
>
> level = 0;
> slots = GET_DENTRY_SLOTS(new_name->len);
> - dentry_hash = f2fs_dentry_hash(new_name, NULL);
> + dentry_hash = f2fs_dentry_hash(dir, new_name, NULL);
>
> current_depth = F2FS_I(dir)->i_current_depth;
> if (F2FS_I(dir)->chash == dentry_hash) {
> @@ -943,3 +1001,51 @@ const struct file_operations f2fs_dir_operations = {
> .compat_ioctl = f2fs_compat_ioctl,
> #endif
> };
> +
> +#ifdef CONFIG_UNICODE
> +static int f2fs_d_compare(const struct dentry *dentry, unsigned int len,
> + const char *str, const struct qstr *name)
> +{
> + struct qstr qstr = {.name = str, .len = len };
> +
> + if (!IS_CASEFOLDED(dentry->d_parent->d_inode)) {
> + if (len != name->len)
> + return -1;
> + return memcmp(str, name, len);
> + }
> +
> + return f2fs_ci_compare(dentry->d_parent->d_inode, name, &qstr);
> +}
> +
> +static int f2fs_d_hash(const struct dentry *dentry, struct qstr *str)
> +{
> + struct f2fs_sb_info *sbi = F2FS_SB(dentry->d_sb);
> + const struct unicode_map *um = sbi->s_encoding;
> + unsigned char *norm;
> + int len, ret = 0;
> +
> + if (!IS_CASEFOLDED(dentry->d_inode))
> + return 0;
> +
> + norm = f2fs_kmalloc(sbi, PATH_MAX, GFP_ATOMIC);
> + if (!norm)
> + return -ENOMEM;
> +
> + len = utf8_casefold(um, str, norm, PATH_MAX);
> + if (len < 0) {
> + if (f2fs_has_strict_mode(sbi))
> + ret = -EINVAL;
> + goto out;
> + }
> + str->hash = full_name_hash(dentry, norm, len);
> +out:
> + kvfree(norm);
> + return ret;
> +}
> +
> +const struct dentry_operations f2fs_dentry_ops = {
> + .d_hash = f2fs_d_hash,
> + .d_compare = f2fs_d_compare,
> +};
> +#endif
> +
> diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
> index c6c7904572d0d..31fd2a268ba14 100644
> --- a/fs/f2fs/f2fs.h
> +++ b/fs/f2fs/f2fs.h
> @@ -2364,10 +2364,12 @@ static inline void f2fs_change_bit(unsigned int nr, char *addr)
> #define F2FS_INDEX_FL 0x00001000 /* hash-indexed directory */
> #define F2FS_DIRSYNC_FL 0x00010000 /* dirsync behaviour (directories only) */
> #define F2FS_PROJINHERIT_FL 0x20000000 /* Create with parents projid */
> +#define F2FS_CASEFOLD_FL 0x40000000 /* Casefolded file */
>
> /* Flags that should be inherited by new inodes from their parent. */
> #define F2FS_FL_INHERITED (F2FS_SYNC_FL | F2FS_NODUMP_FL | F2FS_NOATIME_FL | \
> - F2FS_DIRSYNC_FL | F2FS_PROJINHERIT_FL)
> + F2FS_DIRSYNC_FL | F2FS_PROJINHERIT_FL | \
> + F2FS_CASEFOLD_FL)
>
> /* Flags that are appropriate for regular files (all but dir-specific ones). */
> #define F2FS_REG_FLMASK (~(F2FS_DIRSYNC_FL | F2FS_PROJINHERIT_FL))
> @@ -2930,6 +2932,10 @@ int f2fs_update_extension_list(struct f2fs_sb_info *sbi, const char *name,
> bool hot, bool set);
> struct dentry *f2fs_get_parent(struct dentry *child);
>
> +extern int f2fs_ci_compare(const struct inode *parent,
> + const struct qstr *name,
> + const struct qstr *entry);
> +
> /*
> * dir.c
> */
> @@ -2993,8 +2999,8 @@ int f2fs_sanity_check_ckpt(struct f2fs_sb_info *sbi);
> /*
> * hash.c
> */
> -f2fs_hash_t f2fs_dentry_hash(const struct qstr *name_info,
> - struct fscrypt_name *fname);
> +f2fs_hash_t f2fs_dentry_hash(const struct inode *dir,
> + const struct qstr *name_info, struct fscrypt_name *fname);
>
> /*
> * node.c
> @@ -3437,6 +3443,9 @@ static inline void f2fs_destroy_root_stats(void) { }
> #endif
>
> extern const struct file_operations f2fs_dir_operations;
> +#ifdef CONFIG_UNICODE
> +extern const struct dentry_operations f2fs_dentry_ops;
> +#endif
> extern const struct file_operations f2fs_file_operations;
> extern const struct inode_operations f2fs_file_inode_operations;
> extern const struct address_space_operations f2fs_dblock_aops;
> diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
> index f8d46df8fa9ee..7adef2d8dbc47 100644
> --- a/fs/f2fs/file.c
> +++ b/fs/f2fs/file.c
> @@ -1660,7 +1660,16 @@ static int f2fs_setflags_common(struct inode *inode, u32 iflags, u32 mask)
> return -EPERM;
>
> oldflags = fi->i_flags;
> + if ((iflags ^ oldflags) & F2FS_CASEFOLD_FL) {
> + if (!f2fs_sb_has_casefold(F2FS_I_SB(inode)))
> + return -EOPNOTSUPP;
> +
> + if (!S_ISDIR(inode->i_mode))
> + return -ENOTDIR;
>
> + if (!f2fs_empty_dir(inode))
> + return -ENOTEMPTY;
> + }
> if ((iflags ^ oldflags) & (F2FS_APPEND_FL | F2FS_IMMUTABLE_FL))
> if (!capable(CAP_LINUX_IMMUTABLE))
> return -EPERM;
> diff --git a/fs/f2fs/hash.c b/fs/f2fs/hash.c
> index cc82f142f811f..b7bd0ddbbdf01 100644
> --- a/fs/f2fs/hash.c
> +++ b/fs/f2fs/hash.c
> @@ -14,6 +14,7 @@
> #include <linux/f2fs_fs.h>
> #include <linux/cryptohash.h>
> #include <linux/pagemap.h>
> +#include <linux/unicode.h>
>
> #include "f2fs.h"
>
> @@ -67,7 +68,7 @@ static void str2hashbuf(const unsigned char *msg, size_t len,
> *buf++ = pad;
> }
>
> -f2fs_hash_t f2fs_dentry_hash(const struct qstr *name_info,
> +static f2fs_hash_t __f2fs_dentry_hash(const struct qstr *name_info,
> struct fscrypt_name *fname)
> {
> __u32 hash;
> @@ -103,3 +104,35 @@ f2fs_hash_t f2fs_dentry_hash(const struct qstr *name_info,
> f2fs_hash = cpu_to_le32(hash & ~F2FS_HASH_COL_BIT);
> return f2fs_hash;
> }
> +
> +f2fs_hash_t f2fs_dentry_hash(const struct inode *dir,
> + const struct qstr *name_info, struct fscrypt_name *fname)
> +{
> +#ifdef CONFIG_UNICODE
> + struct f2fs_sb_info *sbi = F2FS_SB(dir->i_sb);
> + const struct unicode_map *um = sbi->s_encoding;
> + int r, dlen;
> + unsigned char *buff;
> + struct qstr *folded;
> +
> + if (name_info->len && IS_CASEFOLDED(dir)) {
> + buff = f2fs_kzalloc(sbi, sizeof(char) * PATH_MAX, GFP_KERNEL);
> + if (!buff)
> + return -ENOMEM;
> +
> + dlen = utf8_casefold(um, name_info, buff, PATH_MAX);
> + if (dlen < 0) {
> + kfree(buff);
kvfree()
> + goto opaque_seq;
> + }
> + folded->name = buff;
> + folded->len = dlen;
> + r = __f2fs_dentry_hash(folded, fname);
> +
> + kvfree(buff);
> + return r;
> + }
> +opaque_seq:
> +#endif
> + return __f2fs_dentry_hash(name_info, fname);
> +}
> diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c
> index 3613efca8c00c..354f71cf9e6ba 100644
> --- a/fs/f2fs/inline.c
> +++ b/fs/f2fs/inline.c
> @@ -320,7 +320,7 @@ struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir,
> return NULL;
> }
>
> - namehash = f2fs_dentry_hash(&name, fname);
> + namehash = f2fs_dentry_hash(dir, &name, fname);
>
> inline_dentry = inline_data_addr(dir, ipage);
>
> @@ -580,7 +580,7 @@ int f2fs_add_inline_entry(struct inode *dir, const struct qstr *new_name,
>
> f2fs_wait_on_page_writeback(ipage, NODE, true, true);
>
> - name_hash = f2fs_dentry_hash(new_name, NULL);
> + name_hash = f2fs_dentry_hash(dir, new_name, NULL);
> f2fs_update_dentry(ino, mode, &d, new_name, name_hash, bit_pos);
>
> set_page_dirty(ipage);
> diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c
> index a33d7a849b2df..9a1f0d6616577 100644
> --- a/fs/f2fs/inode.c
> +++ b/fs/f2fs/inode.c
> @@ -46,9 +46,11 @@ void f2fs_set_inode_flags(struct inode *inode)
> new_fl |= S_DIRSYNC;
> if (file_is_encrypt(inode))
> new_fl |= S_ENCRYPTED;
> + if (flags & F2FS_CASEFOLD_FL)
> + new_fl |= S_CASEFOLD;
> inode_set_flags(inode, new_fl,
> S_SYNC|S_APPEND|S_IMMUTABLE|S_NOATIME|S_DIRSYNC|
> - S_ENCRYPTED);
> + S_ENCRYPTED|S_CASEFOLD);
> }
>
> static void __get_inode_rdev(struct inode *inode, struct f2fs_inode *ri)
> diff --git a/fs/f2fs/namei.c b/fs/f2fs/namei.c
> index c5b99042e6f2b..727de2f8620f2 100644
> --- a/fs/f2fs/namei.c
> +++ b/fs/f2fs/namei.c
> @@ -489,6 +489,17 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry,
> goto out_iput;
> }
> out_splice:
> +#ifdef CONFIG_UNICODE
> + if (!inode && IS_CASEFOLDED(dir)) {
> + /* Eventually we want to call d_add_ci(dentry, NULL)
> + * for negative dentries in the encoding case as
> + * well. For now, prevent the negative dentry
> + * from being cached.
> + */
> + trace_f2fs_lookup_end(dir, dentry, ino, err);
> + return NULL;
> + }
> +#endif
> new = d_splice_alias(inode, dentry);
> err = PTR_ERR_OR_ZERO(new);
> trace_f2fs_lookup_end(dir, dentry, ino, err);
> @@ -537,6 +548,16 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry)
> goto fail;
> }
> f2fs_delete_entry(de, page, dir, inode);
> +#ifdef CONFIG_UNICODE
> + /* VFS negative dentries are incompatible with Encoding and
> + * Case-insensitiveness. Eventually we'll want avoid
> + * invalidating the dentries here, alongside with returning the
> + * negative dentries at f2fs_lookup(), when it is better
> + * supported by the VFS for the CI case.
> + */
> + if (IS_CASEFOLDED(dir))
> + d_invalidate(dentry);
> +#endif
> f2fs_unlock_op(sbi);
>
> if (IS_DIRSYNC(dir))
> diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
> index 82f7da93c3ed1..9c522d1abcb6d 100644
> --- a/fs/f2fs/super.c
> +++ b/fs/f2fs/super.c
> @@ -3115,6 +3115,7 @@ static int f2fs_setup_casefold(struct f2fs_sb_info *sbi)
> return -EINVAL;
> }
> #endif
> + return 0;
It needs to relocate this line to PATCH 1/2
> }
>
> static void f2fs_tuning_parameters(struct f2fs_sb_info *sbi)
> @@ -3410,6 +3411,11 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
> goto free_node_inode;
> }
>
> +#ifdef CONFIG_UNICODE
> + if (sbi->s_encoding)
> + sb->s_d_op = &f2fs_dentry_ops;
> +#endif
How about moving this to f2fs_setup_casefold()?
Thanks,
> +
> sb->s_root = d_make_root(root); /* allocate root dentry */
> if (!sb->s_root) {
> err = -ENOMEM;
>
^ permalink raw reply
* Re: [PATCH v2 2/2] f2fs: Support case-insensitive file name lookups
From: Chao Yu @ 2019-07-19 2:01 UTC (permalink / raw)
To: Daniel Rosenberg, Jaegeuk Kim, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team
In-Reply-To: <4ef17922-d1e9-1b83-9e89-d332ea6fb7ae@google.com>
On 2019/7/19 5:31, Daniel Rosenberg wrote:
>
> On 7/17/19 3:11 AM, Chao Yu wrote:
>> We need to add one more entry f2fs_fsflags_map[] to map F2FS_CASEFOLD_FL to
>> FS_CASEFOLD_FL correctly and adapt F2FS_GETTABLE_FS_FL/F2FS_SETTABLE_FS_FL as well.
>
> I don't see FS_CASEFOLD_FL. It would make sense for it to exist, but unless it's in some recent patch I don't think that's currently in the kernel. Or are you suggesting adding it in this patch?
Yeah, I think we can use a separated patch to propose uplifting the flag to a
common one in fs.h, and then adjust
f2fs_fsflags_map/F2FS_GETTABLE_FS_FL/F2FS_SETTABLE_FS_FL mapping. Otherwise we
will fail to set CASEFOLD flag to inode.
Thanks,
>
> .
>
^ permalink raw reply
* Re: [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Kees Cook @ 2019-07-19 1:51 UTC (permalink / raw)
To: Sasha Levin
Cc: corbet, solar, will, peterz, gregkh, tyhicks, linux-doc,
linux-kernel
In-Reply-To: <20190719003919.GC4240@sasha-vm>
On Thu, Jul 18, 2019 at 08:39:19PM -0400, Sasha Levin wrote:
> On Thu, Jul 18, 2019 at 03:00:55PM -0700, Kees Cook wrote:
> > On Wed, Jul 17, 2019 at 07:11:03PM -0400, Sasha Levin wrote:
> > > Provide more information about how to interact with the linux-distros
> > > mailing list for disclosing security bugs.
> > >
> > > Reference the linux-distros list policy and clarify that the reporter
> > > must read and understand those policies as they differ from
> > > security@kernel.org's policy.
> > >
> > > Suggested-by: Solar Designer <solar@openwall.com>
> > > Signed-off-by: Sasha Levin <sashal@kernel.org>
> >
> > Sorry, but NACK, see below...
> >
> > > ---
> > >
> > > Changes in v2:
> > > - Focus more on pointing to the linux-distros wiki and policies.
> >
> > I think this is already happening in the text. What specifically do you
> > want described differently?
>
> The main issue was that there isn't anything pointing to the
> linux-distros policies. The current text outlines a few of them ("add
> [vs]", and "there should be an embargo period"), but it effectively just
> gives out the linux-distros mailing address and tells the reporter to
> contact it.
The current text includes the wiki link, but yes, the anchor tag is not
present at the wiki anymore. I would agree that's due for updating.
I think reinforcing information to avoid past mistakes is appropriate
here. Reports have regularly missed the "[vs]" detail or suggested
embargoes that ended on Fridays, etc.
> > > - Remove explicit linux-distros email.
> >
> > I don't like this because we had past trouble with notifications going
> > to the distros@ list and leaking Linux-only flaws to the BSDs. As there
> > isn't a separate linux-distros wiki, the clarification of WHICH list is
> > needed.
>
> Why would removing the explicit linux-distros email encourage people to
> send reports to it?
What? No, I'm saying we should _keep_ linux-distros@... in our text so
that people don't send to the wrong list.
> I also don't understand what you mean by "there isn't a separate
> linux-distros wiki"? There is one, and I want to point the reporter
> there.
That URL is a combined page for two lists. The very fact that it's
not obvious that there are two lists described there is exactly why I
think we need to keep an explicit mention of which to use. There are
two mailing lists described at the wiki URL:
distros@lists.openwall.com
linux-distros@lists.openwall.com
Sending to the distros@ list risks exposing Linux-only flaws to non-Linux
distros. This has caused leaks in the past, and we do not want people
guessing at which list they should use.
Also note that nowhere on the openwall wiki is the email address
actually spelled out; this is another reason to spell it out in our
documentation: no misunderstanding.
(And historically there WAS a specific linux-distros wiki:
https://oss-security.openwall.org/wiki/mailing-lists/linux-distros
but it redirects to the combined one now...)
> > > - Remove various explanations of linux-distros policies.
> >
> > I don't think there's value in removing the Tue-Thu comment, nor
> > providing context for why distros need time. This has been a regular
> > thing we've had to explain to researchers that aren't familiar with
> > update procedures and publication timing.
>
> To be fair, the Tue-Thu comment is listed in the section describing how
> to do coordination with linux-distros, and linux-distros don't have a
> Tue-Thu policy. If it's a security@kernel.org policy then let's list it
> elsewhere.
It's a distro preference. Many researchers aren't thinking about the
larger Linux ecosystem that has to consume fixes. It's not a _policy_,
but it makes the researchers understand how to construct better embargoes.
> If you feel that there is a consensus around Tue-Thu let's just add it
> to the linux-distros policy wiki, there's no point in listing random
> policies from that wiki.
I think it'd be a good idea to add that note also to the wiki, but I
don't want it removed from our text because I have had to repeat that
information regularly in the past.
--
Kees Cook
^ permalink raw reply
* Re: [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Sasha Levin @ 2019-07-19 0:39 UTC (permalink / raw)
To: Kees Cook
Cc: corbet, solar, will, peterz, gregkh, tyhicks, linux-doc,
linux-kernel
In-Reply-To: <201907181457.D61AC061C@keescook>
On Thu, Jul 18, 2019 at 03:00:55PM -0700, Kees Cook wrote:
>On Wed, Jul 17, 2019 at 07:11:03PM -0400, Sasha Levin wrote:
>> Provide more information about how to interact with the linux-distros
>> mailing list for disclosing security bugs.
>>
>> Reference the linux-distros list policy and clarify that the reporter
>> must read and understand those policies as they differ from
>> security@kernel.org's policy.
>>
>> Suggested-by: Solar Designer <solar@openwall.com>
>> Signed-off-by: Sasha Levin <sashal@kernel.org>
>
>Sorry, but NACK, see below...
>
>> ---
>>
>> Changes in v2:
>> - Focus more on pointing to the linux-distros wiki and policies.
>
>I think this is already happening in the text. What specifically do you
>want described differently?
The main issue was that there isn't anything pointing to the
linux-distros policies. The current text outlines a few of them ("add
[vs]", and "there should be an embargo period"), but it effectively just
gives out the linux-distros mailing address and tells the reporter to
contact it.
>> - Remove explicit linux-distros email.
>
>I don't like this because we had past trouble with notifications going
>to the distros@ list and leaking Linux-only flaws to the BSDs. As there
>isn't a separate linux-distros wiki, the clarification of WHICH list is
>needed.
Why would removing the explicit linux-distros email encourage people to
send reports to it?
I also don't understand what you mean by "there isn't a separate
linux-distros wiki"? There is one, and I want to point the reporter
there.
>> - Remove various explanations of linux-distros policies.
>
>I don't think there's value in removing the Tue-Thu comment, nor
>providing context for why distros need time. This has been a regular
>thing we've had to explain to researchers that aren't familiar with
>update procedures and publication timing.
To be fair, the Tue-Thu comment is listed in the section describing how
to do coordination with linux-distros, and linux-distros don't have a
Tue-Thu policy. If it's a security@kernel.org policy then let's list it
elsewhere.
If you feel that there is a consensus around Tue-Thu let's just add it
to the linux-distros policy wiki, there's no point in listing random
policies from that wiki.
--
Thanks,
Sasha
^ permalink raw reply
* Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger
From: Brendan Higgins @ 2019-07-19 0:08 UTC (permalink / raw)
To: Stephen Boyd
Cc: Frank Rowand, Greg KH, Josh Poimboeuf, Kees Cook, Kieran Bingham,
Luis Chamberlain, Peter Zijlstra, Rob Herring, shuah,
Theodore Ts'o, Masahiro Yamada, devicetree, dri-devel,
kunit-dev, open list:DOCUMENTATION, linux-fsdevel, linux-kbuild,
Linux Kernel Mailing List, open list:KERNEL SELFTEST FRAMEWORK,
linux-nvdimm, linux-um, Sasha Levin, Bird, Timothy,
Amir Goldstein, Dan Carpenter, Daniel Vetter, Jeff Dike,
Joel Stanley, Julia Lawall, Kevin Hilman, Knut Omang,
Logan Gunthorpe, Michael Ellerman, Petr Mladek, Randy Dunlap,
Richard Weinberger, David Rientjes, Steven Rostedt, wfg
In-Reply-To: <CAFd5g46a7C1+R6ZcE_SkqaYqgrH5Rx3M=X7orFyaMgFLDbeYYA@mail.gmail.com>
On Thu, Jul 18, 2019 at 12:22:33PM -0700, Brendan Higgins wrote:
> On Thu, Jul 18, 2019 at 10:50 AM Stephen Boyd <sboyd@kernel.org> wrote:
> >
> > Quoting Brendan Higgins (2019-07-16 11:52:01)
> > > On Tue, Jul 16, 2019 at 10:50 AM Stephen Boyd <sboyd@kernel.org> wrote:
[...]
> > Do you have a link to those earlier patches?
>
> This is the first patchset:
>
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788057.html
>
> In particular you can see the code for matching functions here:
>
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788073.html
>
> And parameter matching code here:
>
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788072.html
>
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788086.html
>
> My apologies in advance, but the code at this early stage had not
> adopted the kunit_* prefix and was still using the test_* and mock_*
> prefix. (Hence, struct kunit_stream was known as struct test_stream).
[...]
> > The crux of my complaint is that the string stream API is too loosely
> > defined to be usable. It allows tests to build up a string of
> > unstructured information, but with certain calling constraints so we
> > have to tread carefully. If there was more structure to the data that's
> > being recorded then the test case runner could operate on the data
> > without having to do string/stream operations, allocations, etc. This
> > would make the assertion logic much more concrete and specific to kunit,
> > instead of this small kunit wrapper that's been placed on top of string
> > stream.
>
> Yeah, I can see the point of wanting something that provides more
> structure than the raw `struct kunit_stream` interface. In fact, it is
> something I had already started working on, when I had determined it
> would be a large effort to capture all the variations. I was further
> put off from the idea when I had been asked to convert the KUnit
> intermediate format from what I was using to TAP, because, as it is,
> the current data printed out by KUnit doesn't contain all the data I
> would like to put in it in a way that best takes advantage of the TAP
> specification. One problematic area in particular: TAP already
> provides a way to present a lot of the data I would like to export,
> but it involves JSON serialization which was an idea that some of the
> other reviewers understandably weren't too keen on. TAP also wants to
> report data some time after it is available, which is generally not a
> good idea for test debug information; you want to make it available as
> soon as you can or you risk crashing with the data still inside.
>
> Hence, I decided we could probably spend a good long while debating
> how I present the information. So the idea of having a loose
> definition seemed attractive to me in its own right since it would
> likely conform to whatever we ended up deciding in the long run. Also,
> all the better that it was what I already had and no one seemed to
> mind too much.
>
> The only constant I expect is that `struct kunit` will likely need to
> take an abstract object with a `commit` method, or a `format` method
> or whatever so it could control when data was going to be printed out
> to the user. We will probably also use a string builder in there
> somewhere.
>
> > TL;DR: If we can get rid of the string stream API I'd view that as an
> > improvement because building arbitrary strings in the kernel is complex,
> > error prone and has calling context concerns.
>
> True. No argument there.
>
> > Is the intention that other code besides unit tests will use this string
> > stream API to build up strings? Any targets in mind? This would be a
> > good way to get the API merged upstream given that its 2019 and we
> > haven't had such an API in the kernel so far.
>
> Someone, (was it you?) asked about code sharing with a string builder
> thingy that was used for creating structured human readable files, but
> that seemed like a pretty massive undertaking.
>
> Aside from that, no. I would kind of prefered that nobody used it for
> anything else because I the issues you described.
>
> Nevertheless, I think the debate over the usefulness of the
> string_stream and kunit_stream are separate topics. Even if we made
> kunit_stream more structured, I am pretty sure I would want to use
> string_stream or some variation for constructing the message.
>
> > An "object oriented" (strong quotes!) approach where kunit_fail_msg is
> > the innermost struct in some assertion specific structure might work
> > nicely and allow the test runner to call a generic 'format' function to
> > print out the message based on the type of assertion/expectation it is.
> > It probably would mean less code size too because the strings that are
> > common will be in the common printing function instead of created twice,
> > in the macros/code and then copied to the heap for the string stream.
> >
> > struct kunit_assert {
> > const char *line;
> > const char *file;
> > const char *func;
> > void (*format)(struct kunit_assert *assert);
> > };
> >
> > struct kunit_comparison_assert {
> > enum operator operator;
> > const char *left;
> > const char *right;
> > struct kunit_assert assert;
> > };
> >
> > struct kunit_bool_assert {
> > const char *truth;
> > const char *statement;
> > struct kunit_assert assert;
> > };
> >
> > void kunit_format_comparison(struct kunit_assert *assert)
> > {
> > struct kunit_comparison_assert *comp = container_of(assert, ...)
> >
> > kunit_printk(...)
> > }
I started poking around with your suggestion while we are waiting. A
couple early observations:
1) It is actually easier to do than I previously thought and will probably
help with getting more of the planned TAP output stuff working.
That being said, this is still a pretty substantial undertaking and
will likely take *at least* a week to implement and properly review.
Assuming everything goes extremely well (no unexpected issues on my
end, very responsive reviewers, etc).
2) It *will* eliminate the need for kunit_stream.
3) ...but, it *will not* eliminate the need for string_stream.
Based on my early observations, I do think it is worth doing, but I
don't think it is worth trying to make it in this patchset (unless I
have already missed the window, or it is going to be open for a while):
I do think it will make things much cleaner, but I don't think it will
achieve your desired goal of getting rid of an unstructured
{kunit|string}_stream style interface; it just adds a layer on top of it
that makes it harder to misuse.
I attached a patch of what I have so far at the end of this email so you
can see what I am talking about. And of course, if you agree with my
assessment, so we can start working on it as a future patch.
A couple things in regard to the patch I attached:
1) I wrote it pretty quickly so there are almost definitely mistakes.
You should consider it RFC. I did verify it compiles though.
2) Also, I did use kunit_stream in writing it: all occurences should be
pretty easy to replace with string_stream; nevertheless, the reason
for this is just to make it easier to play with the current APIs. I
wanted to have something working before I went through a big tedious
refactoring. So sorry if it causes any confusion.
3) I also based the patch on all the KUnit patches I have queued up
(includes things like mocking and such) since I want to see how this
serialization thing will work with mocks and matchers and things like
that.
> I started working on something similarish, but by the time I ended up
> coming up with a parent object whose definition was loose enough to
> satisfy all the properties required by the child classes it ended up
> basically being the same as what I have now just with a more complex
> hierarchy of message manipulation logic.
>
> On the other hand, I didn't have the idea of doing the parent object
> quite the way you did and that would clean up a lot of the duplicated
> first line logic.
>
> I would like to give it a try, but I am afraid I am going to get
> sucked down a really deep rabbit hole.
>
> > Maybe other people have opinions here on if you should do it now or
> > later. Future coding is not a great argument because it's hard to
> > predict the future. On the other hand, this patchset is in good shape to
>
> Yeah, that's kind of why I am afraid to go down this road when I have
> something that works now and I know works with the mocking stuff I
> want to do.
>
> I would like to try your suggestion, but I want to try to make it work
> with my mocking patches before I commit to it because otherwise I am
> just going to have to back it out in a follow up patchset.
>
> > merge and I'd like to use it to write unit tests for code I maintain so
> > I don't want to see this stall out. Sorry if I'm opening the can of
> > worms you're talking about.
>
> Don't be sorry. I agree with you that the kunit_stream stuff is not very pretty.
>
> Shuah, have we missed the merge window for 5.3?
>
> I saw you only sent one PR out so far for this release, and there
> wasn't much in it; I imagine you are going to send at least one more?
>
> I figure, if we still got time to try out your suggestion, Stephen, no
> harm in trying.
>
> Also if we missed it, then I have another couple months to play around with it.
>
> What do you think?
I attached the patch mentioned above below. Let me know what you think!
Cheers!
From 53d475d3d56afcf92b452c6d347dbedfa1a17d34 Mon Sep 17 00:00:00 2001
From: Brendan Higgins <brendanhiggins@google.com>
Date: Thu, 18 Jul 2019 16:08:52 -0700
Subject: [PATCH v1] DO NOT MERGE: started playing around with the
serialization api
---
include/kunit/assert.h | 130 ++++++++++++++++++++++++++++++
include/kunit/mock.h | 4 +
kunit/Makefile | 3 +-
kunit/assert.c | 179 +++++++++++++++++++++++++++++++++++++++++
kunit/mock.c | 6 +-
5 files changed, 318 insertions(+), 4 deletions(-)
create mode 100644 include/kunit/assert.h
create mode 100644 kunit/assert.c
diff --git a/include/kunit/assert.h b/include/kunit/assert.h
new file mode 100644
index 0000000000000..e054fdff4642f
--- /dev/null
+++ b/include/kunit/assert.h
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <brendanhiggins@google.com>
+ */
+
+#ifndef _KUNIT_ASSERT_H
+#define _KUNIT_ASSERT_H
+
+#include <kunit/test.h>
+#include <kunit/mock.h>
+
+enum kunit_assert_type {
+ KUNIT_ASSERTION,
+ KUNIT_EXPECTATION,
+};
+
+struct kunit_assert {
+ enum kunit_assert_type type;
+ const char *line;
+ const char *file;
+ struct va_format message;
+ void (*format)(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+};
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_unary_assert {
+ struct kunit_assert assert;
+ const char *condition;
+ bool expected_true;
+};
+
+void kunit_unary_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_ptr_not_err_assert {
+ struct kunit_assert assert;
+ const char *text;
+ void *value;
+};
+
+void kunit_ptr_not_err_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_binary_assert {
+ struct kunit_assert assert;
+ const char *operation;
+ const char *left_text;
+ long long left_value;
+ const char *right_text;
+ long long right_value;
+};
+
+void kunit_binary_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_binary_ptr_assert {
+ struct kunit_assert assert;
+ const char *operation;
+ const char *left_text;
+ void *left_value;
+ const char *right_text;
+ void *right_value;
+};
+
+void kunit_binary_ptr_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_binary_str_assert {
+ struct kunit_assert assert;
+ const char *operation;
+ const char *left_text;
+ const char *left_value;
+ const char *right_text;
+ const char *right_value;
+};
+
+void kunit_binary_str_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+struct kunit_mock_assert {
+ struct kunit_assert assert;
+};
+
+struct kunit_mock_no_expectations {
+ struct kunit_mock_assert assert;
+};
+
+struct kunit_mock_declaration {
+ const char *function_name;
+ const char **type_names;
+ const void **params;
+ int len;
+};
+
+void kunit_mock_declaration_format(struct kunit_mock_declaration *declaration,
+ struct kunit_stream *stream);
+
+struct kunit_matcher_result {
+ struct kunit_assert assert;
+};
+
+struct kunit_mock_failed_match {
+ struct list_head node;
+ const char *expectation_text;
+ struct kunit_matcher_result *matcher_list;
+ size_t matcher_list_len;
+};
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+ struct kunit_stream *stream);
+
+struct kunit_mock_no_match {
+ struct kunit_mock_assert assert;
+ struct kunit_mock_declaration declaration;
+ struct list_head failed_match_list;
+};
+
+void kunit_mock_no_match_format(struct kunit_assert *assert,
+ struct kunit_stream *stream);
+
+#endif /* _KUNIT_ASSERT_H */
diff --git a/include/kunit/mock.h b/include/kunit/mock.h
index 001b96af62f1e..52c9e427c831b 100644
--- a/include/kunit/mock.h
+++ b/include/kunit/mock.h
@@ -144,6 +144,10 @@ void mock_register_formatter(struct mock_param_formatter *formatter);
void mock_unregister_formatter(struct mock_param_formatter *formatter);
+void mock_format_param(struct kunit_stream *stream,
+ const char *type_name,
+ const void *param);
+
struct mock *mock_get_global_mock(void);
#define MOCK(name) name##_mock
diff --git a/kunit/Makefile b/kunit/Makefile
index bbf43fcfb93a9..149d856a30f04 100644
--- a/kunit/Makefile
+++ b/kunit/Makefile
@@ -3,7 +3,8 @@ obj-$(CONFIG_KUNIT) += test.o \
common-mocks.o \
string-stream.o \
kunit-stream.o \
- try-catch.o
+ try-catch.o \
+ assert.o
obj-$(CONFIG_KUNIT_TEST) += test-test.o \
test-mock.o \
diff --git a/kunit/assert.c b/kunit/assert.c
new file mode 100644
index 0000000000000..75bb6922a994e
--- /dev/null
+++ b/kunit/assert.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <brendanhiggins@google.com>
+ */
+#include <kunit/assert.h>
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ const char *expect_or_assert;
+
+ if (assert->type == KUNIT_EXPECTATION)
+ expect_or_assert = "EXPECTATION";
+ else
+ expect_or_assert = "ASSERTION";
+
+ kunit_stream_add(stream, "%s FAILED at %s:%s\n",
+ expect_or_assert, assert->file, assert->line);
+}
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ if (assert->message.fmt)
+ kunit_stream_add(stream, "\n%pV", &assert->message);
+}
+
+void kunit_unary_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_unary_assert *unary_assert = container_of(
+ assert, struct kunit_unary_assert, assert);
+
+ kunit_base_assert_format(assert, stream);
+ if (unary_assert->expected_true)
+ kunit_stream_add(stream,
+ "\tExpected %s to be true, but is false\n",
+ unary_assert->condition);
+ else
+ kunit_stream_add(stream,
+ "\tExpected %s to be false, but is true\n",
+ unary_assert->condition);
+ kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_ptr_not_err_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_ptr_not_err_assert *ptr_assert = container_of(
+ assert, struct kunit_ptr_not_err_assert, assert);
+
+ kunit_base_assert_format(assert, stream);
+ if (!ptr_assert->value) {
+ kunit_stream_add(stream,
+ "\tExpected %s is not null, but is\n",
+ ptr_assert->text);
+ } else if (IS_ERR(ptr_assert->value)) {
+ kunit_stream_add(stream,
+ "\tExpected %s is not error, but is: %ld\n",
+ ptr_assert->text,
+ PTR_ERR(ptr_assert->value));
+ }
+ kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_binary_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_binary_assert *binary_assert = container_of(
+ assert, struct kunit_binary_assert, assert);
+
+ kunit_base_assert_format(assert, stream);
+ kunit_stream_add(stream,
+ "\tExpected %s %s %s, but\n",
+ binary_assert->left_text,
+ binary_assert->operation,
+ binary_assert->right_text);
+ kunit_stream_add(stream, "\t\t%s == %lld\n",
+ binary_assert->left_text,
+ binary_assert->left_value);
+ kunit_stream_add(stream, "\t\t%s == %lld",
+ binary_assert->right_text,
+ binary_assert->right_value);
+ kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_binary_ptr_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_binary_ptr_assert *binary_assert = container_of(
+ assert, struct kunit_binary_ptr_assert, assert);
+
+ kunit_base_assert_format(assert, stream);
+ kunit_stream_add(stream,
+ "\tExpected %s %s %s, but\n",
+ binary_assert->left_text,
+ binary_assert->operation,
+ binary_assert->right_text);
+ kunit_stream_add(stream, "\t\t%s == %pK\n",
+ binary_assert->left_text,
+ binary_assert->left_value);
+ kunit_stream_add(stream, "\t\t%s == %pK",
+ binary_assert->right_text,
+ binary_assert->right_value);
+ kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_binary_str_assert_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_binary_str_assert *binary_assert = container_of(
+ assert, struct kunit_binary_str_assert, assert);
+
+ kunit_base_assert_format(assert, stream);
+ kunit_stream_add(stream,
+ "\tExpected %s %s %s, but\n",
+ binary_assert->left_text,
+ binary_assert->operation,
+ binary_assert->right_text);
+ kunit_stream_add(stream, "\t\t%s == %s\n",
+ binary_assert->left_text,
+ binary_assert->left_value);
+ kunit_stream_add(stream, "\t\t%s == %s",
+ binary_assert->right_text,
+ binary_assert->right_value);
+ kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_mock_declaration_format(struct kunit_mock_declaration *declaration,
+ struct kunit_stream *stream)
+{
+ int i;
+
+ kunit_stream_add(stream, "%s(", declaration->function_name);
+ for (i = 0; i < declaration->len; i++) {
+ mock_format_param(stream,
+ declaration->type_names[i],
+ declaration->params[i]);
+ if (i < declaration->len - 1)
+ kunit_stream_add(stream, ", ");
+ }
+ kunit_stream_add(stream, ")\n");
+}
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+ struct kunit_stream *stream)
+{
+ struct kunit_matcher_result *result;
+ size_t i;
+
+ kunit_stream_add(stream,
+ "Tried expectation: %s, but\n",
+ match->expectation_text);
+ for (i = 0; i < match->matcher_list_len; i++) {
+ result = &match->matcher_list[i];
+ kunit_stream_add(stream, "\t");
+ result->assert.format(&result->assert, stream);
+ kunit_stream_add(stream, "\n");
+ }
+}
+
+void kunit_mock_no_match_format(struct kunit_assert *assert,
+ struct kunit_stream *stream)
+{
+ struct kunit_mock_assert *mock_assert = container_of(
+ assert, struct kunit_mock_assert, assert);
+ struct kunit_mock_no_match *no_match = container_of(
+ mock_assert, struct kunit_mock_no_match, assert);
+ struct kunit_mock_failed_match *expectation;
+
+ kunit_base_assert_format(assert, stream);
+ kunit_mock_declaration_format(&no_match->declaration, stream);
+
+ list_for_each_entry(expectation, &no_match->failed_match_list, node)
+ kunit_mock_failed_match_format(expectation, stream);
+}
diff --git a/kunit/mock.c b/kunit/mock.c
index ccb0abe111402..ab441a58a918c 100644
--- a/kunit/mock.c
+++ b/kunit/mock.c
@@ -269,9 +269,9 @@ struct mock_param_formatter *mock_find_formatter(const char *type_name)
return NULL;
}
-static void mock_format_param(struct kunit_stream *stream,
- const char *type_name,
- const void *param)
+void mock_format_param(struct kunit_stream *stream,
+ const char *type_name,
+ const void *param)
{
struct mock_param_formatter *formatter;
--
2.22.0.657.g960e92d24f-goog
^ permalink raw reply related
* [PATCH v3 2/2] f2fs: Support case-insensitive file name lookups
From: Daniel Rosenberg @ 2019-07-19 0:03 UTC (permalink / raw)
To: Jaegeuk Kim, Chao Yu, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team,
Daniel Rosenberg
In-Reply-To: <20190719000322.106163-1-drosen@google.com>
Modeled after commit b886ee3e778e ("ext4: Support case-insensitive file
name lookups")
"""
This patch implements the actual support for case-insensitive file name
lookups in f2fs, based on the feature bit and the encoding stored in the
superblock.
A filesystem that has the casefold feature set is able to configure
directories with the +F (F2FS_CASEFOLD_FL) attribute, enabling lookups
to succeed in that directory in a case-insensitive fashion, i.e: match
a directory entry even if the name used by userspace is not a byte per
byte match with the disk name, but is an equivalent case-insensitive
version of the Unicode string. This operation is called a
case-insensitive file name lookup.
The feature is configured as an inode attribute applied to directories
and inherited by its children. This attribute can only be enabled on
empty directories for filesystems that support the encoding feature,
thus preventing collision of file names that only differ by case.
* dcache handling:
For a +F directory, F2Fs only stores the first equivalent name dentry
used in the dcache. This is done to prevent unintentional duplication of
dentries in the dcache, while also allowing the VFS code to quickly find
the right entry in the cache despite which equivalent string was used in
a previous lookup, without having to resort to ->lookup().
d_hash() of casefolded directories is implemented as the hash of the
casefolded string, such that we always have a well-known bucket for all
the equivalencies of the same string. d_compare() uses the
utf8_strncasecmp() infrastructure, which handles the comparison of
equivalent, same case, names as well.
For now, negative lookups are not inserted in the dcache, since they
would need to be invalidated anyway, because we can't trust missing file
dentries. This is bad for performance but requires some leveraging of
the vfs layer to fix. We can live without that for now, and so does
everyone else.
* on-disk data:
Despite using a specific version of the name as the internal
representation within the dcache, the name stored and fetched from the
disk is a byte-per-byte match with what the user requested, making this
implementation 'name-preserving'. i.e. no actual information is lost
when writing to storage.
DX is supported by modifying the hashes used in +F directories to make
them case/encoding-aware. The new disk hashes are calculated as the
hash of the full casefolded string, instead of the string directly.
This allows us to efficiently search for file names in the htree without
requiring the user to provide an exact name.
* Dealing with invalid sequences:
By default, when a invalid UTF-8 sequence is identified, ext4 will treat
it as an opaque byte sequence, ignoring the encoding and reverting to
the old behavior for that unique file. This means that case-insensitive
file name lookup will not work only for that file. An optional bit can
be set in the superblock telling the filesystem code and userspace tools
to enforce the encoding. When that optional bit is set, any attempt to
create a file name using an invalid UTF-8 sequence will fail and return
an error to userspace.
* Normalization algorithm:
The UTF-8 algorithms used to compare strings in f2fs is implemented
in fs/unicode, and is based on a previous version developed by
SGI. It implements the Canonical decomposition (NFD) algorithm
described by the Unicode specification 12.1, or higher, combined with
the elimination of ignorable code points (NFDi) and full
case-folding (CF) as documented in fs/unicode/utf8_norm.c.
NFD seems to be the best normalization method for F2FS because:
- It has a lower cost than NFC/NFKC (which requires
decomposing to NFD as an intermediary step)
- It doesn't eliminate important semantic meaning like
compatibility decompositions.
Although:
- This implementation is not completely linguistic accurate, because
different languages have conflicting rules, which would require the
specialization of the filesystem to a given locale, which brings all
sorts of problems for removable media and for users who use more than
one language.
"""
Signed-off-by: Daniel Rosenberg <drosen@google.com>
---
fs/f2fs/dir.c | 126 +++++++++++++++++++++++++++++++++++++++++++----
fs/f2fs/f2fs.h | 15 ++++--
fs/f2fs/file.c | 9 ++++
fs/f2fs/hash.c | 35 ++++++++++++-
fs/f2fs/inline.c | 4 +-
fs/f2fs/inode.c | 4 +-
fs/f2fs/namei.c | 21 ++++++++
fs/f2fs/super.c | 6 +++
8 files changed, 203 insertions(+), 17 deletions(-)
diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c
index 85a1528f319f2..2913483473f30 100644
--- a/fs/f2fs/dir.c
+++ b/fs/f2fs/dir.c
@@ -8,6 +8,7 @@
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
#include <linux/sched/signal.h>
+#include <linux/unicode.h>
#include "f2fs.h"
#include "node.h"
#include "acl.h"
@@ -81,7 +82,8 @@ static unsigned long dir_block_index(unsigned int level,
return bidx;
}
-static struct f2fs_dir_entry *find_in_block(struct page *dentry_page,
+static struct f2fs_dir_entry *find_in_block(struct inode *dir,
+ struct page *dentry_page,
struct fscrypt_name *fname,
f2fs_hash_t namehash,
int *max_slots,
@@ -93,7 +95,7 @@ static struct f2fs_dir_entry *find_in_block(struct page *dentry_page,
dentry_blk = (struct f2fs_dentry_block *)page_address(dentry_page);
- make_dentry_ptr_block(NULL, &d, dentry_blk);
+ make_dentry_ptr_block(dir, &d, dentry_blk);
de = f2fs_find_target_dentry(fname, namehash, max_slots, &d);
if (de)
*res_page = dentry_page;
@@ -101,6 +103,39 @@ static struct f2fs_dir_entry *find_in_block(struct page *dentry_page,
return de;
}
+#ifdef CONFIG_UNICODE
+/*
+ * Test whether a case-insensitive directory entry matches the filename
+ * being searched for.
+ *
+ * Returns: 0 if the directory entry matches, more than 0 if it
+ * doesn't match or less than zero on error.
+ */
+int f2fs_ci_compare(const struct inode *parent, const struct qstr *name,
+ const struct qstr *entry)
+{
+ const struct f2fs_sb_info *sbi = F2FS_SB(parent->i_sb);
+ const struct unicode_map *um = sbi->s_encoding;
+ int ret;
+
+ ret = utf8_strncasecmp(um, name, entry);
+ if (ret < 0) {
+ /* Handle invalid character sequence as either an error
+ * or as an opaque byte sequence.
+ */
+ if (f2fs_has_strict_mode(sbi))
+ return -EINVAL;
+
+ if (name->len != entry->len)
+ return 1;
+
+ return !!memcmp(name->name, entry->name, name->len);
+ }
+
+ return ret;
+}
+#endif
+
struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
f2fs_hash_t namehash, int *max_slots,
struct f2fs_dentry_ptr *d)
@@ -108,6 +143,9 @@ struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
struct f2fs_dir_entry *de;
unsigned long bit_pos = 0;
int max_len = 0;
+#ifdef CONFIG_UNICODE
+ struct qstr entry;
+#endif
if (max_slots)
*max_slots = 0;
@@ -119,16 +157,28 @@ struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
}
de = &d->dentry[bit_pos];
+#ifdef CONFIG_UNICODE
+ entry.name = d->filename[bit_pos];
+ entry.len = de->name_len;
+#endif
if (unlikely(!de->name_len)) {
bit_pos++;
continue;
}
+ if (de->hash_code == namehash) {
+#ifdef CONFIG_UNICODE
+ if (F2FS_SB(d->inode->i_sb)->s_encoding &&
+ IS_CASEFOLDED(d->inode) &&
+ !f2fs_ci_compare(d->inode,
+ fname->usr_fname, &entry))
+ goto found;
- if (de->hash_code == namehash &&
- fscrypt_match_name(fname, d->filename[bit_pos],
- le16_to_cpu(de->name_len)))
- goto found;
+#endif
+ if (fscrypt_match_name(fname, d->filename[bit_pos],
+ le16_to_cpu(de->name_len)))
+ goto found;
+ }
if (max_slots && max_len > *max_slots)
*max_slots = max_len;
@@ -157,7 +207,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
struct f2fs_dir_entry *de = NULL;
bool room = false;
int max_slots;
- f2fs_hash_t namehash = f2fs_dentry_hash(&name, fname);
+ f2fs_hash_t namehash = f2fs_dentry_hash(dir, &name, fname);
nbucket = dir_buckets(level, F2FS_I(dir)->i_dir_level);
nblock = bucket_blocks(level);
@@ -179,8 +229,8 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
}
}
- de = find_in_block(dentry_page, fname, namehash, &max_slots,
- res_page);
+ de = find_in_block(dir, dentry_page, fname, namehash,
+ &max_slots, res_page);
if (de)
break;
@@ -250,6 +300,14 @@ struct f2fs_dir_entry *f2fs_find_entry(struct inode *dir,
struct fscrypt_name fname;
int err;
+#ifdef CONFIG_UNICODE
+ if (f2fs_has_strict_mode(F2FS_I_SB(dir)) && IS_CASEFOLDED(dir) &&
+ utf8_validate(F2FS_I_SB(dir)->s_encoding, child)) {
+ *res_page = ERR_PTR(-EINVAL);
+ return NULL;
+ }
+#endif
+
err = fscrypt_setup_filename(dir, child, 1, &fname);
if (err) {
if (err == -ENOENT)
@@ -504,7 +562,7 @@ int f2fs_add_regular_entry(struct inode *dir, const struct qstr *new_name,
level = 0;
slots = GET_DENTRY_SLOTS(new_name->len);
- dentry_hash = f2fs_dentry_hash(new_name, NULL);
+ dentry_hash = f2fs_dentry_hash(dir, new_name, NULL);
current_depth = F2FS_I(dir)->i_current_depth;
if (F2FS_I(dir)->chash == dentry_hash) {
@@ -943,3 +1001,51 @@ const struct file_operations f2fs_dir_operations = {
.compat_ioctl = f2fs_compat_ioctl,
#endif
};
+
+#ifdef CONFIG_UNICODE
+static int f2fs_d_compare(const struct dentry *dentry, unsigned int len,
+ const char *str, const struct qstr *name)
+{
+ struct qstr qstr = {.name = str, .len = len };
+
+ if (!IS_CASEFOLDED(dentry->d_parent->d_inode)) {
+ if (len != name->len)
+ return -1;
+ return memcmp(str, name, len);
+ }
+
+ return f2fs_ci_compare(dentry->d_parent->d_inode, name, &qstr);
+}
+
+static int f2fs_d_hash(const struct dentry *dentry, struct qstr *str)
+{
+ struct f2fs_sb_info *sbi = F2FS_SB(dentry->d_sb);
+ const struct unicode_map *um = sbi->s_encoding;
+ unsigned char *norm;
+ int len, ret = 0;
+
+ if (!IS_CASEFOLDED(dentry->d_inode))
+ return 0;
+
+ norm = f2fs_kmalloc(sbi, PATH_MAX, GFP_ATOMIC);
+ if (!norm)
+ return -ENOMEM;
+
+ len = utf8_casefold(um, str, norm, PATH_MAX);
+ if (len < 0) {
+ if (f2fs_has_strict_mode(sbi))
+ ret = -EINVAL;
+ goto out;
+ }
+ str->hash = full_name_hash(dentry, norm, len);
+out:
+ kvfree(norm);
+ return ret;
+}
+
+const struct dentry_operations f2fs_dentry_ops = {
+ .d_hash = f2fs_d_hash,
+ .d_compare = f2fs_d_compare,
+};
+#endif
+
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index c6c7904572d0d..31fd2a268ba14 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -2364,10 +2364,12 @@ static inline void f2fs_change_bit(unsigned int nr, char *addr)
#define F2FS_INDEX_FL 0x00001000 /* hash-indexed directory */
#define F2FS_DIRSYNC_FL 0x00010000 /* dirsync behaviour (directories only) */
#define F2FS_PROJINHERIT_FL 0x20000000 /* Create with parents projid */
+#define F2FS_CASEFOLD_FL 0x40000000 /* Casefolded file */
/* Flags that should be inherited by new inodes from their parent. */
#define F2FS_FL_INHERITED (F2FS_SYNC_FL | F2FS_NODUMP_FL | F2FS_NOATIME_FL | \
- F2FS_DIRSYNC_FL | F2FS_PROJINHERIT_FL)
+ F2FS_DIRSYNC_FL | F2FS_PROJINHERIT_FL | \
+ F2FS_CASEFOLD_FL)
/* Flags that are appropriate for regular files (all but dir-specific ones). */
#define F2FS_REG_FLMASK (~(F2FS_DIRSYNC_FL | F2FS_PROJINHERIT_FL))
@@ -2930,6 +2932,10 @@ int f2fs_update_extension_list(struct f2fs_sb_info *sbi, const char *name,
bool hot, bool set);
struct dentry *f2fs_get_parent(struct dentry *child);
+extern int f2fs_ci_compare(const struct inode *parent,
+ const struct qstr *name,
+ const struct qstr *entry);
+
/*
* dir.c
*/
@@ -2993,8 +2999,8 @@ int f2fs_sanity_check_ckpt(struct f2fs_sb_info *sbi);
/*
* hash.c
*/
-f2fs_hash_t f2fs_dentry_hash(const struct qstr *name_info,
- struct fscrypt_name *fname);
+f2fs_hash_t f2fs_dentry_hash(const struct inode *dir,
+ const struct qstr *name_info, struct fscrypt_name *fname);
/*
* node.c
@@ -3437,6 +3443,9 @@ static inline void f2fs_destroy_root_stats(void) { }
#endif
extern const struct file_operations f2fs_dir_operations;
+#ifdef CONFIG_UNICODE
+extern const struct dentry_operations f2fs_dentry_ops;
+#endif
extern const struct file_operations f2fs_file_operations;
extern const struct inode_operations f2fs_file_inode_operations;
extern const struct address_space_operations f2fs_dblock_aops;
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index f8d46df8fa9ee..7adef2d8dbc47 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -1660,7 +1660,16 @@ static int f2fs_setflags_common(struct inode *inode, u32 iflags, u32 mask)
return -EPERM;
oldflags = fi->i_flags;
+ if ((iflags ^ oldflags) & F2FS_CASEFOLD_FL) {
+ if (!f2fs_sb_has_casefold(F2FS_I_SB(inode)))
+ return -EOPNOTSUPP;
+
+ if (!S_ISDIR(inode->i_mode))
+ return -ENOTDIR;
+ if (!f2fs_empty_dir(inode))
+ return -ENOTEMPTY;
+ }
if ((iflags ^ oldflags) & (F2FS_APPEND_FL | F2FS_IMMUTABLE_FL))
if (!capable(CAP_LINUX_IMMUTABLE))
return -EPERM;
diff --git a/fs/f2fs/hash.c b/fs/f2fs/hash.c
index cc82f142f811f..b7bd0ddbbdf01 100644
--- a/fs/f2fs/hash.c
+++ b/fs/f2fs/hash.c
@@ -14,6 +14,7 @@
#include <linux/f2fs_fs.h>
#include <linux/cryptohash.h>
#include <linux/pagemap.h>
+#include <linux/unicode.h>
#include "f2fs.h"
@@ -67,7 +68,7 @@ static void str2hashbuf(const unsigned char *msg, size_t len,
*buf++ = pad;
}
-f2fs_hash_t f2fs_dentry_hash(const struct qstr *name_info,
+static f2fs_hash_t __f2fs_dentry_hash(const struct qstr *name_info,
struct fscrypt_name *fname)
{
__u32 hash;
@@ -103,3 +104,35 @@ f2fs_hash_t f2fs_dentry_hash(const struct qstr *name_info,
f2fs_hash = cpu_to_le32(hash & ~F2FS_HASH_COL_BIT);
return f2fs_hash;
}
+
+f2fs_hash_t f2fs_dentry_hash(const struct inode *dir,
+ const struct qstr *name_info, struct fscrypt_name *fname)
+{
+#ifdef CONFIG_UNICODE
+ struct f2fs_sb_info *sbi = F2FS_SB(dir->i_sb);
+ const struct unicode_map *um = sbi->s_encoding;
+ int r, dlen;
+ unsigned char *buff;
+ struct qstr *folded;
+
+ if (name_info->len && IS_CASEFOLDED(dir)) {
+ buff = f2fs_kzalloc(sbi, sizeof(char) * PATH_MAX, GFP_KERNEL);
+ if (!buff)
+ return -ENOMEM;
+
+ dlen = utf8_casefold(um, name_info, buff, PATH_MAX);
+ if (dlen < 0) {
+ kfree(buff);
+ goto opaque_seq;
+ }
+ folded->name = buff;
+ folded->len = dlen;
+ r = __f2fs_dentry_hash(folded, fname);
+
+ kvfree(buff);
+ return r;
+ }
+opaque_seq:
+#endif
+ return __f2fs_dentry_hash(name_info, fname);
+}
diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c
index 3613efca8c00c..354f71cf9e6ba 100644
--- a/fs/f2fs/inline.c
+++ b/fs/f2fs/inline.c
@@ -320,7 +320,7 @@ struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir,
return NULL;
}
- namehash = f2fs_dentry_hash(&name, fname);
+ namehash = f2fs_dentry_hash(dir, &name, fname);
inline_dentry = inline_data_addr(dir, ipage);
@@ -580,7 +580,7 @@ int f2fs_add_inline_entry(struct inode *dir, const struct qstr *new_name,
f2fs_wait_on_page_writeback(ipage, NODE, true, true);
- name_hash = f2fs_dentry_hash(new_name, NULL);
+ name_hash = f2fs_dentry_hash(dir, new_name, NULL);
f2fs_update_dentry(ino, mode, &d, new_name, name_hash, bit_pos);
set_page_dirty(ipage);
diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c
index a33d7a849b2df..9a1f0d6616577 100644
--- a/fs/f2fs/inode.c
+++ b/fs/f2fs/inode.c
@@ -46,9 +46,11 @@ void f2fs_set_inode_flags(struct inode *inode)
new_fl |= S_DIRSYNC;
if (file_is_encrypt(inode))
new_fl |= S_ENCRYPTED;
+ if (flags & F2FS_CASEFOLD_FL)
+ new_fl |= S_CASEFOLD;
inode_set_flags(inode, new_fl,
S_SYNC|S_APPEND|S_IMMUTABLE|S_NOATIME|S_DIRSYNC|
- S_ENCRYPTED);
+ S_ENCRYPTED|S_CASEFOLD);
}
static void __get_inode_rdev(struct inode *inode, struct f2fs_inode *ri)
diff --git a/fs/f2fs/namei.c b/fs/f2fs/namei.c
index c5b99042e6f2b..727de2f8620f2 100644
--- a/fs/f2fs/namei.c
+++ b/fs/f2fs/namei.c
@@ -489,6 +489,17 @@ static struct dentry *f2fs_lookup(struct inode *dir, struct dentry *dentry,
goto out_iput;
}
out_splice:
+#ifdef CONFIG_UNICODE
+ if (!inode && IS_CASEFOLDED(dir)) {
+ /* Eventually we want to call d_add_ci(dentry, NULL)
+ * for negative dentries in the encoding case as
+ * well. For now, prevent the negative dentry
+ * from being cached.
+ */
+ trace_f2fs_lookup_end(dir, dentry, ino, err);
+ return NULL;
+ }
+#endif
new = d_splice_alias(inode, dentry);
err = PTR_ERR_OR_ZERO(new);
trace_f2fs_lookup_end(dir, dentry, ino, err);
@@ -537,6 +548,16 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry)
goto fail;
}
f2fs_delete_entry(de, page, dir, inode);
+#ifdef CONFIG_UNICODE
+ /* VFS negative dentries are incompatible with Encoding and
+ * Case-insensitiveness. Eventually we'll want avoid
+ * invalidating the dentries here, alongside with returning the
+ * negative dentries at f2fs_lookup(), when it is better
+ * supported by the VFS for the CI case.
+ */
+ if (IS_CASEFOLDED(dir))
+ d_invalidate(dentry);
+#endif
f2fs_unlock_op(sbi);
if (IS_DIRSYNC(dir))
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 82f7da93c3ed1..9c522d1abcb6d 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -3115,6 +3115,7 @@ static int f2fs_setup_casefold(struct f2fs_sb_info *sbi)
return -EINVAL;
}
#endif
+ return 0;
}
static void f2fs_tuning_parameters(struct f2fs_sb_info *sbi)
@@ -3410,6 +3411,11 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
goto free_node_inode;
}
+#ifdef CONFIG_UNICODE
+ if (sbi->s_encoding)
+ sb->s_d_op = &f2fs_dentry_ops;
+#endif
+
sb->s_root = d_make_root(root); /* allocate root dentry */
if (!sb->s_root) {
err = -ENOMEM;
--
2.22.0.657.g960e92d24f-goog
^ permalink raw reply related
* [PATCH v3 1/2] f2fs: include charset encoding information in the superblock
From: Daniel Rosenberg @ 2019-07-19 0:03 UTC (permalink / raw)
To: Jaegeuk Kim, Chao Yu, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team,
Daniel Rosenberg
In-Reply-To: <20190719000322.106163-1-drosen@google.com>
Add charset encoding to f2fs to support casefolding. It is modeled after
the same feature introduced in commit c83ad55eaa91 ("ext4: include charset
encoding information in the superblock")
Currently this is not compatible with encryption, similar to the current
ext4 imlpementation. This will change in the future.
From the ext4 patch:
"""
The s_encoding field stores a magic number indicating the encoding
format and version used globally by file and directory names in the
filesystem. The s_encoding_flags defines policies for using the charset
encoding, like how to handle invalid sequences. The magic number is
mapped to the exact charset table, but the mapping is specific to ext4.
Since we don't have any commitment to support old encodings, the only
encoding I am supporting right now is utf8-12.1.0.
The current implementation prevents the user from enabling encoding and
per-directory encryption on the same filesystem at the same time. The
incompatibility between these features lies in how we do efficient
directory searches when we cannot be sure the encryption of the user
provided fname will match the actual hash stored in the disk without
decrypting every directory entry, because of normalization cases. My
quickest solution is to simply block the concurrent use of these
features for now, and enable it later, once we have a better solution.
"""
Signed-off-by: Daniel Rosenberg <drosen@google.com>
---
fs/f2fs/f2fs.h | 6 +++
fs/f2fs/super.c | 94 +++++++++++++++++++++++++++++++++++++++++
fs/f2fs/sysfs.c | 23 ++++++++++
include/linux/f2fs_fs.h | 9 +++-
4 files changed, 131 insertions(+), 1 deletion(-)
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 17382da7f0bd9..c6c7904572d0d 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -153,6 +153,7 @@ struct f2fs_mount_info {
#define F2FS_FEATURE_LOST_FOUND 0x0200
#define F2FS_FEATURE_VERITY 0x0400 /* reserved */
#define F2FS_FEATURE_SB_CHKSUM 0x0800
+#define F2FS_FEATURE_CASEFOLD 0x1000
#define __F2FS_HAS_FEATURE(raw_super, mask) \
((raw_super->feature & cpu_to_le32(mask)) != 0)
@@ -1169,6 +1170,10 @@ struct f2fs_sb_info {
int valid_super_block; /* valid super block no */
unsigned long s_flag; /* flags for sbi */
struct mutex writepages; /* mutex for writepages() */
+#ifdef CONFIG_UNICODE
+ struct unicode_map *s_encoding;
+ __u16 s_encoding_flags;
+#endif
#ifdef CONFIG_BLK_DEV_ZONED
unsigned int blocks_per_blkz; /* F2FS blocks per zone */
@@ -3562,6 +3567,7 @@ F2FS_FEATURE_FUNCS(quota_ino, QUOTA_INO);
F2FS_FEATURE_FUNCS(inode_crtime, INODE_CRTIME);
F2FS_FEATURE_FUNCS(lost_found, LOST_FOUND);
F2FS_FEATURE_FUNCS(sb_chksum, SB_CHKSUM);
+F2FS_FEATURE_FUNCS(casefold, CASEFOLD);
#ifdef CONFIG_BLK_DEV_ZONED
static inline bool f2fs_blkz_is_seq(struct f2fs_sb_info *sbi, int devi,
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 6de6cda440315..82f7da93c3ed1 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -23,6 +23,7 @@
#include <linux/f2fs_fs.h>
#include <linux/sysfs.h>
#include <linux/quota.h>
+#include <linux/unicode.h>
#include "f2fs.h"
#include "node.h"
@@ -222,6 +223,36 @@ void f2fs_printk(struct f2fs_sb_info *sbi, const char *fmt, ...)
va_end(args);
}
+#ifdef CONFIG_UNICODE
+static const struct f2fs_sb_encodings {
+ __u16 magic;
+ char *name;
+ char *version;
+} f2fs_sb_encoding_map[] = {
+ {F2FS_ENC_UTF8_12_1, "utf8", "12.1.0"},
+};
+
+static int f2fs_sb_read_encoding(const struct f2fs_super_block *sb,
+ const struct f2fs_sb_encodings **encoding,
+ __u16 *flags)
+{
+ __u16 magic = le16_to_cpu(sb->s_encoding);
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(f2fs_sb_encoding_map); i++)
+ if (magic == f2fs_sb_encoding_map[i].magic)
+ break;
+
+ if (i >= ARRAY_SIZE(f2fs_sb_encoding_map))
+ return -EINVAL;
+
+ *encoding = &f2fs_sb_encoding_map[i];
+ *flags = le16_to_cpu(sb->s_encoding_flags);
+
+ return 0;
+}
+#endif
+
static inline void limit_reserve_root(struct f2fs_sb_info *sbi)
{
block_t limit = min((sbi->user_block_count << 1) / 1000,
@@ -798,6 +829,13 @@ static int parse_options(struct super_block *sb, char *options)
return -EINVAL;
}
#endif
+#ifndef CONFIG_UNICODE
+ if (f2fs_sb_has_casefold(sbi)) {
+ f2fs_err(sbi,
+ "Filesystem with casefold feature cannot be mounted without CONFIG_UNICODE");
+ return -EINVAL;
+ }
+#endif
if (F2FS_IO_SIZE_BITS(sbi) && !test_opt(sbi, LFS)) {
f2fs_err(sbi, "Should set mode=lfs with %uKB-sized IO",
@@ -1089,6 +1127,9 @@ static void f2fs_put_super(struct super_block *sb)
destroy_percpu_info(sbi);
for (i = 0; i < NR_PAGE_TYPE; i++)
kvfree(sbi->write_io[i]);
+#ifdef CONFIG_UNICODE
+ utf8_unload(sbi->s_encoding);
+#endif
kvfree(sbi);
}
@@ -3031,6 +3072,51 @@ static int f2fs_scan_devices(struct f2fs_sb_info *sbi)
return 0;
}
+static int f2fs_setup_casefold(struct f2fs_sb_info *sbi)
+{
+#ifdef CONFIG_UNICODE
+ if (f2fs_sb_has_casefold(sbi) && !sbi->s_encoding) {
+ const struct f2fs_sb_encodings *encoding_info;
+ struct unicode_map *encoding;
+ __u16 encoding_flags;
+
+ if (f2fs_sb_has_encrypt(sbi)) {
+ f2fs_err(sbi,
+ "Can't mount with encoding and encryption");
+ return -EINVAL;
+ }
+
+ if (f2fs_sb_read_encoding(sbi->raw_super, &encoding_info,
+ &encoding_flags)) {
+ f2fs_err(sbi,
+ "Encoding requested by superblock is unknown");
+ return -EINVAL;
+ }
+
+ encoding = utf8_load(encoding_info->version);
+ if (IS_ERR(encoding)) {
+ f2fs_err(sbi,
+ "can't mount with superblock charset: %s-%s "
+ "not supported by the kernel. flags: 0x%x.",
+ encoding_info->name, encoding_info->version,
+ encoding_flags);
+ return PTR_ERR(encoding);
+ }
+ f2fs_info(sbi, "Using encoding defined by superblock: "
+ "%s-%s with flags 0x%hx", encoding_info->name,
+ encoding_info->version?:"\b", encoding_flags);
+
+ sbi->s_encoding = encoding;
+ sbi->s_encoding_flags = encoding_flags;
+ }
+#else
+ if (f2fs_sb_has_casefold(sbi)) {
+ f2fs_err(sbi, "Filesystem with casefold feature cannot be mounted without CONFIG_UNICODE");
+ return -EINVAL;
+ }
+#endif
+}
+
static void f2fs_tuning_parameters(struct f2fs_sb_info *sbi)
{
struct f2fs_sm_info *sm_i = SM_I(sbi);
@@ -3127,6 +3213,10 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
le32_to_cpu(raw_super->log_blocksize);
sb->s_max_links = F2FS_LINK_MAX;
+ err = f2fs_setup_casefold(sbi);
+ if (err)
+ goto free_options;
+
#ifdef CONFIG_QUOTA
sb->dq_op = &f2fs_quota_operations;
sb->s_qcop = &f2fs_quotactl_ops;
@@ -3477,6 +3567,10 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
free_bio_info:
for (i = 0; i < NR_PAGE_TYPE; i++)
kvfree(sbi->write_io[i]);
+
+#ifdef CONFIG_UNICODE
+ utf8_unload(sbi->s_encoding);
+#endif
free_options:
#ifdef CONFIG_QUOTA
for (i = 0; i < MAXQUOTAS; i++)
diff --git a/fs/f2fs/sysfs.c b/fs/f2fs/sysfs.c
index 3aeacd0aacfd2..f9fcca695db9f 100644
--- a/fs/f2fs/sysfs.c
+++ b/fs/f2fs/sysfs.c
@@ -10,6 +10,7 @@
#include <linux/proc_fs.h>
#include <linux/f2fs_fs.h>
#include <linux/seq_file.h>
+#include <linux/unicode.h>
#include "f2fs.h"
#include "segment.h"
@@ -81,6 +82,19 @@ static ssize_t unusable_show(struct f2fs_attr *a,
(unsigned long long)unusable);
}
+static ssize_t encoding_show(struct f2fs_attr *a,
+ struct f2fs_sb_info *sbi, char *buf)
+{
+#ifdef CONFIG_UNICODE
+ if (f2fs_sb_has_casefold(sbi))
+ return snprintf(buf, PAGE_SIZE, "%s (%d.%d.%d)\n",
+ sbi->s_encoding->charset,
+ (sbi->s_encoding->version >> 16) & 0xff,
+ (sbi->s_encoding->version >> 8) & 0xff,
+ sbi->s_encoding->version & 0xff);
+#endif
+ return snprintf(buf, PAGE_SIZE, "(none)");
+}
static ssize_t lifetime_write_kbytes_show(struct f2fs_attr *a,
struct f2fs_sb_info *sbi, char *buf)
@@ -134,6 +148,9 @@ static ssize_t features_show(struct f2fs_attr *a,
if (f2fs_sb_has_sb_chksum(sbi))
len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
len ? ", " : "", "sb_checksum");
+ if (f2fs_sb_has_casefold(sbi))
+ len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
+ len ? ", " : "", "casefold");
len += snprintf(buf + len, PAGE_SIZE - len, "\n");
return len;
}
@@ -365,6 +382,7 @@ enum feat_id {
FEAT_INODE_CRTIME,
FEAT_LOST_FOUND,
FEAT_SB_CHECKSUM,
+ FEAT_CASEFOLD,
};
static ssize_t f2fs_feature_show(struct f2fs_attr *a,
@@ -382,6 +400,7 @@ static ssize_t f2fs_feature_show(struct f2fs_attr *a,
case FEAT_INODE_CRTIME:
case FEAT_LOST_FOUND:
case FEAT_SB_CHECKSUM:
+ case FEAT_CASEFOLD:
return snprintf(buf, PAGE_SIZE, "supported\n");
}
return 0;
@@ -455,6 +474,7 @@ F2FS_GENERAL_RO_ATTR(lifetime_write_kbytes);
F2FS_GENERAL_RO_ATTR(features);
F2FS_GENERAL_RO_ATTR(current_reserved_blocks);
F2FS_GENERAL_RO_ATTR(unusable);
+F2FS_GENERAL_RO_ATTR(encoding);
#ifdef CONFIG_FS_ENCRYPTION
F2FS_FEATURE_RO_ATTR(encryption, FEAT_CRYPTO);
@@ -471,6 +491,7 @@ F2FS_FEATURE_RO_ATTR(quota_ino, FEAT_QUOTA_INO);
F2FS_FEATURE_RO_ATTR(inode_crtime, FEAT_INODE_CRTIME);
F2FS_FEATURE_RO_ATTR(lost_found, FEAT_LOST_FOUND);
F2FS_FEATURE_RO_ATTR(sb_checksum, FEAT_SB_CHECKSUM);
+F2FS_FEATURE_RO_ATTR(casefold, FEAT_CASEFOLD);
#define ATTR_LIST(name) (&f2fs_attr_##name.attr)
static struct attribute *f2fs_attrs[] = {
@@ -515,6 +536,7 @@ static struct attribute *f2fs_attrs[] = {
ATTR_LIST(features),
ATTR_LIST(reserved_blocks),
ATTR_LIST(current_reserved_blocks),
+ ATTR_LIST(encoding),
NULL,
};
ATTRIBUTE_GROUPS(f2fs);
@@ -535,6 +557,7 @@ static struct attribute *f2fs_feat_attrs[] = {
ATTR_LIST(inode_crtime),
ATTR_LIST(lost_found),
ATTR_LIST(sb_checksum),
+ ATTR_LIST(casefold),
NULL,
};
ATTRIBUTE_GROUPS(f2fs_feat);
diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h
index 65559900d4d76..b7c9c7f721339 100644
--- a/include/linux/f2fs_fs.h
+++ b/include/linux/f2fs_fs.h
@@ -36,6 +36,11 @@
#define F2FS_MAX_QUOTAS 3
+#define F2FS_ENC_UTF8_12_1 1
+#define F2FS_ENC_STRICT_MODE_FL (1 << 0)
+#define f2fs_has_strict_mode(sbi) \
+ (sbi->s_encoding_flags & F2FS_ENC_STRICT_MODE_FL)
+
#define F2FS_IO_SIZE(sbi) (1 << F2FS_OPTION(sbi).write_io_size_bits) /* Blocks */
#define F2FS_IO_SIZE_KB(sbi) (1 << (F2FS_OPTION(sbi).write_io_size_bits + 2)) /* KB */
#define F2FS_IO_SIZE_BYTES(sbi) (1 << (F2FS_OPTION(sbi).write_io_size_bits + 12)) /* B */
@@ -109,7 +114,9 @@ struct f2fs_super_block {
struct f2fs_device devs[MAX_DEVICES]; /* device list */
__le32 qf_ino[F2FS_MAX_QUOTAS]; /* quota inode numbers */
__u8 hot_ext_count; /* # of hot file extension */
- __u8 reserved[310]; /* valid reserved region */
+ __le16 s_encoding; /* Filename charset encoding */
+ __le16 s_encoding_flags; /* Filename charset encoding flags */
+ __u8 reserved[306]; /* valid reserved region */
__le32 crc; /* checksum of superblock */
} __packed;
--
2.22.0.657.g960e92d24f-goog
^ permalink raw reply related
* [PATCH v3 0/2] Casefolding in F2FS
From: Daniel Rosenberg @ 2019-07-19 0:03 UTC (permalink / raw)
To: Jaegeuk Kim, Chao Yu, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team,
Daniel Rosenberg
These patches are largely based on the casefolding patches for ext4
v3: Addressed feedback, apart from F2FS_CASEFOLD_FL/FS_CASEFOLD_FL
Added sysfs file "encoding" to see the encoding set on a filesystem
v2: Rebased patches again master, changed f2fs_msg to f2fs_info/f2fs_err
Daniel Rosenberg (2):
f2fs: include charset encoding information in the superblock
f2fs: Support case-insensitive file name lookups
fs/f2fs/dir.c | 126 ++++++++++++++++++++++++++++++++++++----
fs/f2fs/f2fs.h | 21 ++++++-
fs/f2fs/file.c | 9 +++
fs/f2fs/hash.c | 35 ++++++++++-
fs/f2fs/inline.c | 4 +-
fs/f2fs/inode.c | 4 +-
fs/f2fs/namei.c | 21 +++++++
fs/f2fs/super.c | 100 +++++++++++++++++++++++++++++++
fs/f2fs/sysfs.c | 23 ++++++++
include/linux/f2fs_fs.h | 9 ++-
10 files changed, 334 insertions(+), 18 deletions(-)
--
2.22.0.657.g960e92d24f-goog
^ permalink raw reply
* Re: [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Kees Cook @ 2019-07-18 22:00 UTC (permalink / raw)
To: Sasha Levin
Cc: corbet, solar, will, peterz, gregkh, tyhicks, linux-doc,
linux-kernel
In-Reply-To: <20190717231103.13949-1-sashal@kernel.org>
On Wed, Jul 17, 2019 at 07:11:03PM -0400, Sasha Levin wrote:
> Provide more information about how to interact with the linux-distros
> mailing list for disclosing security bugs.
>
> Reference the linux-distros list policy and clarify that the reporter
> must read and understand those policies as they differ from
> security@kernel.org's policy.
>
> Suggested-by: Solar Designer <solar@openwall.com>
> Signed-off-by: Sasha Levin <sashal@kernel.org>
Sorry, but NACK, see below...
> ---
>
> Changes in v2:
> - Focus more on pointing to the linux-distros wiki and policies.
I think this is already happening in the text. What specifically do you
want described differently?
> - Remove explicit linux-distros email.
I don't like this because we had past trouble with notifications going
to the distros@ list and leaking Linux-only flaws to the BSDs. As there
isn't a separate linux-distros wiki, the clarification of WHICH list is
needed.
> - Remove various explanations of linux-distros policies.
I don't think there's value in removing the Tue-Thu comment, nor
providing context for why distros need time. This has been a regular
thing we've had to explain to researchers that aren't familiar with
update procedures and publication timing.
-Kees
>
> Documentation/admin-guide/security-bugs.rst | 19 +++++++++----------
> 1 file changed, 9 insertions(+), 10 deletions(-)
>
> diff --git a/Documentation/admin-guide/security-bugs.rst b/Documentation/admin-guide/security-bugs.rst
> index dcd6c93c7aac..380d44fd618d 100644
> --- a/Documentation/admin-guide/security-bugs.rst
> +++ b/Documentation/admin-guide/security-bugs.rst
> @@ -60,16 +60,15 @@ Coordination
> ------------
>
> Fixes for sensitive bugs, such as those that might lead to privilege
> -escalations, may need to be coordinated with the private
> -<linux-distros@vs.openwall.org> mailing list so that distribution vendors
> -are well prepared to issue a fixed kernel upon public disclosure of the
> -upstream fix. Distros will need some time to test the proposed patch and
> -will generally request at least a few days of embargo, and vendor update
> -publication prefers to happen Tuesday through Thursday. When appropriate,
> -the security team can assist with this coordination, or the reporter can
> -include linux-distros from the start. In this case, remember to prefix
> -the email Subject line with "[vs]" as described in the linux-distros wiki:
> -<http://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists>
> +escalations, may need to be coordinated with the private linux-distros mailing
> +list so that distribution vendors are well prepared to issue a fixed kernel
> +upon public disclosure of the upstream fix. Please read and follow the policies
> +of linux-distros as specified in the linux-distros wiki page before reporting:
> +<https://oss-security.openwall.org/wiki/mailing-lists/distros>. When
> +appropriate, the security team can assist with this coordination, or the
> +reporter can include linux-distros from the start. In this case, remember to
> +prefix the email Subject line with "[vs]" as described in the linux-distros
> +wiki.
>
> CVE assignment
> --------------
> --
> 2.20.1
>
--
Kees Cook
^ permalink raw reply
* Re: [PATCH v2 2/2] f2fs: Support case-insensitive file name lookups
From: Daniel Rosenberg @ 2019-07-18 21:31 UTC (permalink / raw)
To: Chao Yu, Jaegeuk Kim, Jonathan Corbet, linux-f2fs-devel
Cc: linux-kernel, linux-doc, linux-fsdevel, kernel-team
In-Reply-To: <cbaf59d4-0bd3-6980-4750-fbab14941bdb@huawei.com>
On 7/17/19 3:11 AM, Chao Yu wrote:
> We need to add one more entry f2fs_fsflags_map[] to map F2FS_CASEFOLD_FL to
> FS_CASEFOLD_FL correctly and adapt F2FS_GETTABLE_FS_FL/F2FS_SETTABLE_FS_FL as well.
I don't see FS_CASEFOLD_FL. It would make sense for it to exist, but unless it's in some recent patch I don't think that's currently in the kernel. Or are you suggesting adding it in this patch?
^ permalink raw reply
* Re: [PATCH v2 08/11] kbuild: create *.mod with full directory path and remove MODVERDIR
From: Joe Lawrence @ 2019-07-18 20:18 UTC (permalink / raw)
To: Masahiro Yamada
Cc: Linux Kbuild mailing list, Sam Ravnborg, Nicolas Pitre,
open list:DOCUMENTATION, Jonathan Corbet,
Linux Kernel Mailing List, Michal Marek
In-Reply-To: <CAK7LNAQ41NhPPO6xoVObgFctTO6WewSXPfZkE7_bZXsdAtKSpA@mail.gmail.com>
On 7/17/19 1:21 AM, Masahiro Yamada wrote:
>> Hi Masahiro,
>>
>> I'm following this patchset changes as they will affect the klp-convert
>> series [1] that the livepatching folks have been working on...
>
> Empty files .tmp_versions/*.livepatch are touched
> to keep track of 'LIVEPATCH_* := y', right?
>
Pretty much. From that patchset I think the rework would need to modify..
- [PATCH v4 02/10] kbuild: Support for Symbols.list creation: creates a
Symbols.list file of kernel and module symbols, but *not* including any
from LIVEPATCH_* modules.
- [PATCH v4 05/10] modpost: Integrate klp-convert: if a LIVEPATCH_*
module has changed, call a newly introduced klp-convert script on it.
- [PATCH v4 06/10] modpost: Add modinfo flag to livepatch modules: find
any LIVEPATCH_* mod, add it to a livepatchmods file, then pass that file
to modpost
> Perhaps, adding a new field
> to *.mod files might be cleaner.
I can look into that. By "field" you mean a new row in the file?
Regards,
-- Joe
^ permalink raw reply
* Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger
From: Brendan Higgins @ 2019-07-18 19:22 UTC (permalink / raw)
To: Stephen Boyd
Cc: Frank Rowand, Greg KH, Josh Poimboeuf, Kees Cook, Kieran Bingham,
Luis Chamberlain, Peter Zijlstra, Rob Herring, shuah,
Theodore Ts'o, Masahiro Yamada, devicetree, dri-devel,
kunit-dev, open list:DOCUMENTATION, linux-fsdevel, linux-kbuild,
Linux Kernel Mailing List, open list:KERNEL SELFTEST FRAMEWORK,
linux-nvdimm, linux-um, Sasha Levin, Bird, Timothy,
Amir Goldstein, Dan Carpenter, Daniel Vetter, Jeff Dike,
Joel Stanley, Julia Lawall, Kevin Hilman, Knut Omang,
Logan Gunthorpe, Michael Ellerman, Petr Mladek, Randy Dunlap,
Richard Weinberger, David Rientjes, Steven Rostedt, wfg
In-Reply-To: <20190718175024.C3EC421019@mail.kernel.org>
On Thu, Jul 18, 2019 at 10:50 AM Stephen Boyd <sboyd@kernel.org> wrote:
>
> Quoting Brendan Higgins (2019-07-16 11:52:01)
> > On Tue, Jul 16, 2019 at 10:50 AM Stephen Boyd <sboyd@kernel.org> wrote:
> > >
> >
> > > The only hypothetical case where this can't be done is a complicated
> > > assertion or expectation that does more than one check and can't be
> > > written as a function that dumps out what went wrong. Is this a real
> > > problem? Maybe such an assertion should just open code that logic so we
> > > don't have to build up a string for all the other simple cases.
> >
> > I have some expectations in follow up patchsets for which I created a
> > set of composable matchers for matching structures and function calls
> > that by their nature cannot be written as a single function. The
> > matcher thing is a bit speculative, I know, but for any kind of
> > function call matching, you need to store a record of functions you
> > are expecting to have called and then each one needs to have a set of
> > expectations defined by the user; I don't think there is a way to do
> > that that doesn't involve having multiple separate functions each
> > having some information useful to constructing the message.
> >
> > I know the code in question isn't in this patchset; the function
> > matching code was in one of the earlier versions of the RFC, but I
> > dropped it to make this patchset smaller and more manageable. So I get
> > it if you would like me to drop it and add it back in when I try to
> > get the function and structure matching stuff in, but I would really
> > prefer to keep it as is if you don't care too much.
>
> Do you have a link to those earlier patches?
This is the first patchset:
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788057.html
In particular you can see the code for matching functions here:
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788073.html
And parameter matching code here:
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788072.html
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1788086.html
My apologies in advance, but the code at this early stage had not
adopted the kunit_* prefix and was still using the test_* and mock_*
prefix. (Hence, struct kunit_stream was known as struct test_stream).
> > > It seems far simpler to get rid of the string stream API and just have a
> > > struct for this.
> > >
> > > struct kunit_fail_msg {
> > > const char *line;
> > > const char *file;
> > > const char *func;
> > > const char *msg;
> > > };
> > >
> > > Then you can have the assertion macros create this on the stack (with
> > > another macro?).
> > >
> > > #define DEFINE_KUNIT_FAIL_MSG(name, _msg) \
> > > struct kunit_fail_msg name = { \
> > > .line = __LINE__, \
> > > .file = __FILE__, \
> > > .func = __func__, \
> > > .msg = _msg, \
> > > }
> > >
> > > I don't want to derail this whole series on this topic, but it seems
> > > like a bunch of code is there to construct this same set of information
> > > over and over again into a buffer a little bit at a time and then throw
> > > it away when nothing fails just because we may want to support the case
> > > where we have some unstructured data to inform the user about.
> >
> > Yeah, that's fair. I think there are a number of improvements to be
> > made with how the expectations are defined other than that, but I was
> > hoping I could do that after this patchset is merged. I just figured
> > with the kinds of things I would like to do, it would lead to a whole
> > new round of discussion.
> >
> > In either case, I think I would still like to use the `struct
> > kunit_stream` as part of the interface to share the failure message
> > with the test case runner code in test.c, at least eventually, so that
> > I only have to have one way to receive data from expectations, but I
> > think I can do that and still do what you suggest by just constructing
> > the kunit_stream at the end of expectations where it is feasible.
> >
> > All in all I agree with what you are saying, but I would rather do it
> > as a follow up possibly once we have some more code on the table. I
> > could just see this opening up a whole new can of worms where we
> > debate about exactly how expectations and assertions work for another
> > several months, only to rip it all out shortly there after. I know
> > that's how these things go, but that's my preference.
> >
> > I can do what you suggest if you feel strongly about it, but I would
> > prefer to hold off until later. It's your call.
> >
>
> The crux of my complaint is that the string stream API is too loosely
> defined to be usable. It allows tests to build up a string of
> unstructured information, but with certain calling constraints so we
> have to tread carefully. If there was more structure to the data that's
> being recorded then the test case runner could operate on the data
> without having to do string/stream operations, allocations, etc. This
> would make the assertion logic much more concrete and specific to kunit,
> instead of this small kunit wrapper that's been placed on top of string
> stream.
Yeah, I can see the point of wanting something that provides more
structure than the raw `struct kunit_stream` interface. In fact, it is
something I had already started working on, when I had determined it
would be a large effort to capture all the variations. I was further
put off from the idea when I had been asked to convert the KUnit
intermediate format from what I was using to TAP, because, as it is,
the current data printed out by KUnit doesn't contain all the data I
would like to put in it in a way that best takes advantage of the TAP
specification. One problematic area in particular: TAP already
provides a way to present a lot of the data I would like to export,
but it involves JSON serialization which was an idea that some of the
other reviewers understandably weren't too keen on. TAP also wants to
report data some time after it is available, which is generally not a
good idea for test debug information; you want to make it available as
soon as you can or you risk crashing with the data still inside.
Hence, I decided we could probably spend a good long while debating
how I present the information. So the idea of having a loose
definition seemed attractive to me in its own right since it would
likely conform to whatever we ended up deciding in the long run. Also,
all the better that it was what I already had and no one seemed to
mind too much.
The only constant I expect is that `struct kunit` will likely need to
take an abstract object with a `commit` method, or a `format` method
or whatever so it could control when data was going to be printed out
to the user. We will probably also use a string builder in there
somewhere.
> TL;DR: If we can get rid of the string stream API I'd view that as an
> improvement because building arbitrary strings in the kernel is complex,
> error prone and has calling context concerns.
True. No argument there.
> Is the intention that other code besides unit tests will use this string
> stream API to build up strings? Any targets in mind? This would be a
> good way to get the API merged upstream given that its 2019 and we
> haven't had such an API in the kernel so far.
Someone, (was it you?) asked about code sharing with a string builder
thingy that was used for creating structured human readable files, but
that seemed like a pretty massive undertaking.
Aside from that, no. I would kind of prefered that nobody used it for
anything else because I the issues you described.
Nevertheless, I think the debate over the usefulness of the
string_stream and kunit_stream are separate topics. Even if we made
kunit_stream more structured, I am pretty sure I would want to use
string_stream or some variation for constructing the message.
> An "object oriented" (strong quotes!) approach where kunit_fail_msg is
> the innermost struct in some assertion specific structure might work
> nicely and allow the test runner to call a generic 'format' function to
> print out the message based on the type of assertion/expectation it is.
> It probably would mean less code size too because the strings that are
> common will be in the common printing function instead of created twice,
> in the macros/code and then copied to the heap for the string stream.
>
> struct kunit_assert {
> const char *line;
> const char *file;
> const char *func;
> void (*format)(struct kunit_assert *assert);
> };
>
> struct kunit_comparison_assert {
> enum operator operator;
> const char *left;
> const char *right;
> struct kunit_assert assert;
> };
>
> struct kunit_bool_assert {
> const char *truth;
> const char *statement;
> struct kunit_assert assert;
> };
>
> void kunit_format_comparison(struct kunit_assert *assert)
> {
> struct kunit_comparison_assert *comp = container_of(assert, ...)
>
> kunit_printk(...)
> }
I started working on something similarish, but by the time I ended up
coming up with a parent object whose definition was loose enough to
satisfy all the properties required by the child classes it ended up
basically being the same as what I have now just with a more complex
hierarchy of message manipulation logic.
On the other hand, I didn't have the idea of doing the parent object
quite the way you did and that would clean up a lot of the duplicated
first line logic.
I would like to give it a try, but I am afraid I am going to get
sucked down a really deep rabbit hole.
> Maybe other people have opinions here on if you should do it now or
> later. Future coding is not a great argument because it's hard to
> predict the future. On the other hand, this patchset is in good shape to
Yeah, that's kind of why I am afraid to go down this road when I have
something that works now and I know works with the mocking stuff I
want to do.
I would like to try your suggestion, but I want to try to make it work
with my mocking patches before I commit to it because otherwise I am
just going to have to back it out in a follow up patchset.
> merge and I'd like to use it to write unit tests for code I maintain so
> I don't want to see this stall out. Sorry if I'm opening the can of
> worms you're talking about.
Don't be sorry. I agree with you that the kunit_stream stuff is not very pretty.
Shuah, have we missed the merge window for 5.3?
I saw you only sent one PR out so far for this release, and there
wasn't much in it; I imagine you are going to send at least one more?
I figure, if we still got time to try out your suggestion, Stephen, no
harm in trying.
Also if we missed it, then I have another couple months to play around with it.
What do you think?
^ permalink raw reply
* Re: Using rst2pdf for PDF output - Was: Re: [PATCH 0/5] PDF output fixes
From: Markus Heiser @ 2019-07-18 17:56 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Jonathan Corbet; +Cc: linux-doc
In-Reply-To: <20190718144655.5aa7deb4@coco.lan>
Am 18.07.19 um 19:46 schrieb Mauro Carvalho Chehab:
> Em Sat, 13 Jul 2019 00:41:25 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>
>> Em Fri, 12 Jul 2019 19:27:05 -0300
>> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>>
>>> Em Fri, 12 Jul 2019 14:19:21 -0600
>>> Jonathan Corbet <corbet@lwn.net> escreveu:
>>>
>>>> Can't you just make rst2pdf work instead? :)
>>>
>>> Well, we can try.
Thanks a lot for your investigation on this. I also checked the rst2pdf sources
a while ago, for me it was crap with crap requirements [1] .. my tip: don't
waste to much time on it.
[1] https://github.com/mchehab/rst2pdf/blob/master/requirements.txt#L31
-- Markus --
^ permalink raw reply
* Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger
From: Stephen Boyd @ 2019-07-18 17:50 UTC (permalink / raw)
To: Brendan Higgins
Cc: Frank Rowand, Greg KH, Josh Poimboeuf, Kees Cook, Kieran Bingham,
Luis Chamberlain, Peter Zijlstra, Rob Herring, shuah,
Theodore Ts'o, Masahiro Yamada, devicetree, dri-devel,
kunit-dev, open list:DOCUMENTATION, linux-fsdevel, linux-kbuild,
Linux Kernel Mailing List, open list:KERNEL SELFTEST FRAMEWORK,
linux-nvdimm, linux-um, Sasha Levin, Bird, Timothy,
Amir Goldstein, Dan Carpenter, Daniel Vetter, Jeff Dike,
Joel Stanley, Julia Lawall, Kevin Hilman, Knut Omang,
Logan Gunthorpe, Michael Ellerman, Petr Mladek, Randy Dunlap,
Richard Weinberger, David Rientjes, Steven Rostedt, wfg
In-Reply-To: <CAFd5g453vXeSUCZenCk_CzJ-8a1ym9RaPo0NVF=FujF9ac-5Ag@mail.gmail.com>
Quoting Brendan Higgins (2019-07-16 11:52:01)
> On Tue, Jul 16, 2019 at 10:50 AM Stephen Boyd <sboyd@kernel.org> wrote:
> >
>
> > The only hypothetical case where this can't be done is a complicated
> > assertion or expectation that does more than one check and can't be
> > written as a function that dumps out what went wrong. Is this a real
> > problem? Maybe such an assertion should just open code that logic so we
> > don't have to build up a string for all the other simple cases.
>
> I have some expectations in follow up patchsets for which I created a
> set of composable matchers for matching structures and function calls
> that by their nature cannot be written as a single function. The
> matcher thing is a bit speculative, I know, but for any kind of
> function call matching, you need to store a record of functions you
> are expecting to have called and then each one needs to have a set of
> expectations defined by the user; I don't think there is a way to do
> that that doesn't involve having multiple separate functions each
> having some information useful to constructing the message.
>
> I know the code in question isn't in this patchset; the function
> matching code was in one of the earlier versions of the RFC, but I
> dropped it to make this patchset smaller and more manageable. So I get
> it if you would like me to drop it and add it back in when I try to
> get the function and structure matching stuff in, but I would really
> prefer to keep it as is if you don't care too much.
Do you have a link to those earlier patches?
>
> > It seems far simpler to get rid of the string stream API and just have a
> > struct for this.
> >
> > struct kunit_fail_msg {
> > const char *line;
> > const char *file;
> > const char *func;
> > const char *msg;
> > };
> >
> > Then you can have the assertion macros create this on the stack (with
> > another macro?).
> >
> > #define DEFINE_KUNIT_FAIL_MSG(name, _msg) \
> > struct kunit_fail_msg name = { \
> > .line = __LINE__, \
> > .file = __FILE__, \
> > .func = __func__, \
> > .msg = _msg, \
> > }
> >
> > I don't want to derail this whole series on this topic, but it seems
> > like a bunch of code is there to construct this same set of information
> > over and over again into a buffer a little bit at a time and then throw
> > it away when nothing fails just because we may want to support the case
> > where we have some unstructured data to inform the user about.
>
> Yeah, that's fair. I think there are a number of improvements to be
> made with how the expectations are defined other than that, but I was
> hoping I could do that after this patchset is merged. I just figured
> with the kinds of things I would like to do, it would lead to a whole
> new round of discussion.
>
> In either case, I think I would still like to use the `struct
> kunit_stream` as part of the interface to share the failure message
> with the test case runner code in test.c, at least eventually, so that
> I only have to have one way to receive data from expectations, but I
> think I can do that and still do what you suggest by just constructing
> the kunit_stream at the end of expectations where it is feasible.
>
> All in all I agree with what you are saying, but I would rather do it
> as a follow up possibly once we have some more code on the table. I
> could just see this opening up a whole new can of worms where we
> debate about exactly how expectations and assertions work for another
> several months, only to rip it all out shortly there after. I know
> that's how these things go, but that's my preference.
>
> I can do what you suggest if you feel strongly about it, but I would
> prefer to hold off until later. It's your call.
>
The crux of my complaint is that the string stream API is too loosely
defined to be usable. It allows tests to build up a string of
unstructured information, but with certain calling constraints so we
have to tread carefully. If there was more structure to the data that's
being recorded then the test case runner could operate on the data
without having to do string/stream operations, allocations, etc. This
would make the assertion logic much more concrete and specific to kunit,
instead of this small kunit wrapper that's been placed on top of string
stream.
TL;DR: If we can get rid of the string stream API I'd view that as an
improvement because building arbitrary strings in the kernel is complex,
error prone and has calling context concerns.
Is the intention that other code besides unit tests will use this string
stream API to build up strings? Any targets in mind? This would be a
good way to get the API merged upstream given that its 2019 and we
haven't had such an API in the kernel so far.
An "object oriented" (strong quotes!) approach where kunit_fail_msg is
the innermost struct in some assertion specific structure might work
nicely and allow the test runner to call a generic 'format' function to
print out the message based on the type of assertion/expectation it is.
It probably would mean less code size too because the strings that are
common will be in the common printing function instead of created twice,
in the macros/code and then copied to the heap for the string stream.
struct kunit_assert {
const char *line;
const char *file;
const char *func;
void (*format)(struct kunit_assert *assert);
};
struct kunit_comparison_assert {
enum operator operator;
const char *left;
const char *right;
struct kunit_assert assert;
};
struct kunit_bool_assert {
const char *truth;
const char *statement;
struct kunit_assert assert;
};
void kunit_format_comparison(struct kunit_assert *assert)
{
struct kunit_comparison_assert *comp = container_of(assert, ...)
kunit_printk(...)
}
Maybe other people have opinions here on if you should do it now or
later. Future coding is not a great argument because it's hard to
predict the future. On the other hand, this patchset is in good shape to
merge and I'd like to use it to write unit tests for code I maintain so
I don't want to see this stall out. Sorry if I'm opening the can of
worms you're talking about.
^ permalink raw reply
* Using rst2pdf for PDF output - Was: Re: [PATCH 0/5] PDF output fixes
From: Mauro Carvalho Chehab @ 2019-07-18 17:46 UTC (permalink / raw)
To: Jonathan Corbet; +Cc: linux-doc
In-Reply-To: <20190713004125.6009b661@coco.lan>
Em Sat, 13 Jul 2019 00:41:25 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> Em Fri, 12 Jul 2019 19:27:05 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>
> > Em Fri, 12 Jul 2019 14:19:21 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:
> >
> > > Can't you just make rst2pdf work instead? :)
> >
> > Well, we can try.
Doing some tests right now with upstream version of it. I actually
created a fork:
https://github.com/mchehab/rst2pdf
basically upstream + some pending PRs + 2 fixes and a hack:
https://github.com/mchehab/rst2pdf/commits/master
> > Also, right now, rst2pdf only supports Python 2.7.
> >
> > I would wait for it to support Python 3.x before doing the actual
> > migration.
The upstream version runs on python 3.x. There are a few issues there
yet to be solved.
> There are a few issues with the quick hack to use rst2pdf. The
> enclosed one works better.
Did another improvement (see enclosed). This one should be applied
after my PDF fixes branch:
https://git.linuxtv.org/mchehab/experimental.git/log/?h=pdf_fixes_v1
I suspect that we'll also need a patch for kfigure.py, in order for it
to support kernel-figure output.
>
> Yet, from the existing 70 books (on my experimental tree), rst2pdf breaks
> with 14 ones.
I was able to find one of the reason some books failed: rst2pdf has a
"hidden" dependency:
svglib
Without that, it will silently ignore SVG images, causing the script
to break.
Even with that, it will still break (reporting success), even on not
so complex books. For example:
$ make SPHINXDIRS=hwmon pdfdocs
writing hwmon...
[ERROR] pdfbuilder.py:150 More than 10 pages generated without content - halting layout. Likely that a flowable is too large for any frame.
Traceback (most recent call last):
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/rst2pdf-0.95.dev0-py3.7.egg/rst2pdf/pdfbuilder.py", line 147, in write
docwriter.write(doctree, destination)
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/docutils/writers/__init__.py", line 80, in write
self.translate()
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/rst2pdf-0.95.dev0-py3.7.egg/rst2pdf/pdfbuilder.py", line 643, in translate
compressed=self.compressed)
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/rst2pdf-0.95.dev0-py3.7.egg/rst2pdf/createpdf.py", line 643, in createPdf
pdfdoc.multiBuild(elements)
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/reportlab/platypus/doctemplate.py", line 1144, in multiBuild
self.build(tempStory, **buildKwds)
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/reportlab/platypus/doctemplate.py", line 1057, in build
self.handle_flowable(flowables)
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/rst2pdf-0.95.dev0-py3.7.egg/rst2pdf/createpdf.py", line 791, in handle_flowable
self.handle_frameEnd()
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/reportlab/platypus/doctemplate.py", line 709, in handle_frameEnd
self.handle_pageEnd()
File "/devel/v4l/docs_temp/sphinx_2.0.1/lib/python3.7/site-packages/reportlab/platypus/doctemplate.py", line 651, in handle_pageEnd
raise LayoutError(ident)
reportlab.platypus.doctemplate.LayoutError: More than 10 pages generated without content - halting layout. Likely that a flowable is too large for any frame.
FAILED
build succeeded.
Patch enclosed.
Thanks,
Mauro
---
[PATCH] docs: experimental: build PDF with rst2pdf
Change the logic to use rst2pdf instead of LaTeX.
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 638c4c11d102..565acf6e6ee7 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -94,18 +94,22 @@ latexdocs:
ifeq ($(HAVE_PDFLATEX),0)
-pdfdocs:
+oldpdfdocs:
$(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
@echo " SKIP Sphinx $@ target."
else # HAVE_PDFLATEX
-pdfdocs: latexdocs
+oldpdfdocs: latexdocs
@$(srctree)/scripts/sphinx-pre-install --version-check
$(foreach var,$(SPHINXDIRS), $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" -C $(BUILDDIR)/$(var)/latex || exit;)
endif # HAVE_PDFLATEX
+pdfdocs:
+ @$(srctree)/scripts/sphinx-pre-install --version-check
+ @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,pdf,$(var),pdf,$(var)))
+
epubdocs:
@$(srctree)/scripts/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
@@ -131,6 +135,7 @@ dochelp:
@echo ' htmldocs - HTML'
@echo ' latexdocs - LaTeX'
@echo ' pdfdocs - PDF'
+ @echo ' rst2pdf - PDF, using experimental rst2pdf support'
@echo ' epubdocs - EPUB'
@echo ' xmldocs - XML'
@echo ' linkcheckdocs - check for broken external links (will connect to external hosts)'
diff --git a/Documentation/conf.py b/Documentation/conf.py
index c698f453bfab..4aa76e4b30ab 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -46,6 +46,8 @@ if (major == 1 and minor > 3) or (major > 1):
else:
extensions.append("sphinx.ext.pngmath")
+extensions.append("rst2pdf.pdfbuilder")
+
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
@@ -574,12 +576,15 @@ epub_exclude_files = ['search.html']
#
# See the Sphinx chapter of http://ralsina.me/static/manual.pdf
#
-# FIXME: Do not add the index file here; the result will be too big. Adding
-# multiple PDF files here actually tries to get the cross-referencing right
-# *between* PDF files.
-pdf_documents = [
- ('kernel-documentation', u'Kernel', u'Kernel', u'J. Random Bozo'),
-]
+
+# Add all LaTeX files to PDF documents as well
+pdf_documents = []
+for l in latex_documents:
+ doc = l[0]
+ fn = l[1].replace(".tex", "")
+ name = l[2]
+ authors = l[3]
+ pdf_documents.append((doc, fn, name, authors))
# kernel-doc extension configuration for running Sphinx directly (e.g. by Read
# the Docs). In a normal build, these are supplied from the Makefile via command
diff --git a/Documentation/media/Makefile b/Documentation/media/Makefile
index d75d70f191bc..3050136ed489 100644
--- a/Documentation/media/Makefile
+++ b/Documentation/media/Makefile
@@ -53,12 +53,13 @@ $(BUILDDIR)/lirc.h.rst: ${UAPI}/lirc.h ${PARSER} $(SRC_DIR)/lirc.h.rst.exception
# Media build rules
-.PHONY: all html epub xml latex
+.PHONY: all html epub xml latex pdf
all: $(IMGDOT) $(BUILDDIR) ${TARGETS}
html: all
epub: all
xml: all
+pdf: all
latex: $(IMGPDF) all
linkcheck:
diff --git a/Documentation/sphinx/load_config.py b/Documentation/sphinx/load_config.py
index 101e6f0b0fcf..a89d642d520e 100644
--- a/Documentation/sphinx/load_config.py
+++ b/Documentation/sphinx/load_config.py
@@ -43,6 +43,17 @@ def loadConfig(namespace):
namespace['latex_documents'] = new_latex_docs
+ # Add LaTeX files to PDF documents as well
+ pdf_documents = []
+ for l in new_latex_docs:
+ doc = l[0]
+ fn = l[1].replace(".tex", "")
+ name = l[2]
+ authors = l[3]
+ pdf_documents.append((doc, fn, name, authors))
+
+ namespace['pdf_documents'] = pdf_documents
+
# If there is an extra conf.py file, load it
if os.path.isfile(config_file):
sys.stdout.write("load additional sphinx-config: %s\n" % config_file)
diff --git a/Makefile b/Makefile
index ec21ebbd12c2..2a749335d0df 100644
--- a/Makefile
+++ b/Makefile
@@ -1564,7 +1564,7 @@ $(help-board-dirs): help-%:
# Documentation targets
# ---------------------------------------------------------------------------
DOC_TARGETS := xmldocs latexdocs pdfdocs htmldocs epubdocs cleandocs \
- linkcheckdocs dochelp refcheckdocs
+ linkcheckdocs dochelp refcheckdocs oldpdfdocs
PHONY += $(DOC_TARGETS)
$(DOC_TARGETS): scripts_basic FORCE
$(Q)$(MAKE) $(build)=Documentation $@
^ permalink raw reply related
* Re: [PATCH v2] Documentation/security-bugs: provide more information about linux-distros
From: Solar Designer @ 2019-07-18 14:14 UTC (permalink / raw)
To: Will Deacon
Cc: Sasha Levin, corbet, keescook, peterz, gregkh, tyhicks, linux-doc,
linux-kernel
In-Reply-To: <20190718094057.e4nclrw6qd2t4vw7@willie-the-truck>
On Thu, Jul 18, 2019 at 10:40:58AM +0100, Will Deacon wrote:
> On Wed, Jul 17, 2019 at 07:11:03PM -0400, Sasha Levin wrote:
> > Provide more information about how to interact with the linux-distros
> > mailing list for disclosing security bugs.
> >
> > Reference the linux-distros list policy and clarify that the reporter
> > must read and understand those policies as they differ from
> > security@kernel.org's policy.
> >
> > Suggested-by: Solar Designer <solar@openwall.com>
> > Signed-off-by: Sasha Levin <sashal@kernel.org>
> > ---
> >
> > Changes in v2:
> > - Focus more on pointing to the linux-distros wiki and policies.
> > - Remove explicit linux-distros email.
> > - Remove various explanations of linux-distros policies.
> >
> > Documentation/admin-guide/security-bugs.rst | 19 +++++++++----------
> > 1 file changed, 9 insertions(+), 10 deletions(-)
> >
> > diff --git a/Documentation/admin-guide/security-bugs.rst b/Documentation/admin-guide/security-bugs.rst
> > index dcd6c93c7aac..380d44fd618d 100644
> > --- a/Documentation/admin-guide/security-bugs.rst
> > +++ b/Documentation/admin-guide/security-bugs.rst
> > @@ -60,16 +60,15 @@ Coordination
> > ------------
> >
> > Fixes for sensitive bugs, such as those that might lead to privilege
> > -escalations, may need to be coordinated with the private
> > -<linux-distros@vs.openwall.org> mailing list so that distribution vendors
> > -are well prepared to issue a fixed kernel upon public disclosure of the
> > -upstream fix. Distros will need some time to test the proposed patch and
> > -will generally request at least a few days of embargo, and vendor update
> > -publication prefers to happen Tuesday through Thursday. When appropriate,
> > -the security team can assist with this coordination, or the reporter can
> > -include linux-distros from the start. In this case, remember to prefix
> > -the email Subject line with "[vs]" as described in the linux-distros wiki:
> > -<http://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists>
> > +escalations, may need to be coordinated with the private linux-distros mailing
> > +list so that distribution vendors are well prepared to issue a fixed kernel
> > +upon public disclosure of the upstream fix. Please read and follow the policies
> > +of linux-distros as specified in the linux-distros wiki page before reporting:
>
> can we add a "there" at the end of this sentence, so it can't be misread as
> implying that you must follow the linux-distros policies before reporting to
> security@kernel.org ?
Sasha's patch above and the addition suggested by Will look good to me.
Thanks!
Alexander
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox