* Re: [RFC v2 1/4] fs: Add generic file system event notifications
From: Beata Michalska @ 2015-04-28 14:46 UTC (permalink / raw)
To: Greg KH
Cc: Jan Kara, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA,
linux-api-u79uwXL29TY76Z2rM5mHXA, tytso-3s7WtUTddSA,
adilger.kernel-m1MBpc4rdrD3fQ9qLvQP4Q,
hughd-hpIqsD4AKlfQT0dZR+AlfA, lczerner-H+wXaHxf7aLQT0dZR+AlfA,
hch-wEGCiKHe2LqWVfeAwA7xHQ, linux-ext4-u79uwXL29TY76Z2rM5mHXA,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
kyungmin.park-Sze3O3UU22JBDgjK7y7TUQ,
kmpark-wEGCiKHe2LqWVfeAwA7xHQ
In-Reply-To: <20150428140936.GA13406-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org>
On 04/28/2015 04:09 PM, Greg KH wrote:
> On Tue, Apr 28, 2015 at 03:56:53PM +0200, Jan Kara wrote:
>> On Mon 27-04-15 17:37:11, Greg KH wrote:
>>> On Mon, Apr 27, 2015 at 05:08:27PM +0200, Beata Michalska wrote:
>>>> On 04/27/2015 04:24 PM, Greg KH wrote:
>>>>> On Mon, Apr 27, 2015 at 01:51:41PM +0200, Beata Michalska wrote:
>>>>>> Introduce configurable generic interface for file
>>>>>> system-wide event notifications, to provide file
>>>>>> systems with a common way of reporting any potential
>>>>>> issues as they emerge.
>>>>>>
>>>>>> The notifications are to be issued through generic
>>>>>> netlink interface by newly introduced multicast group.
>>>>>>
>>>>>> Threshold notifications have been included, allowing
>>>>>> triggering an event whenever the amount of free space drops
>>>>>> below a certain level - or levels to be more precise as two
>>>>>> of them are being supported: the lower and the upper range.
>>>>>> The notifications work both ways: once the threshold level
>>>>>> has been reached, an event shall be generated whenever
>>>>>> the number of available blocks goes up again re-activating
>>>>>> the threshold.
>>>>>>
>>>>>> The interface has been exposed through a vfs. Once mounted,
>>>>>> it serves as an entry point for the set-up where one can
>>>>>> register for particular file system events.
>>>>>>
>>>>>> Signed-off-by: Beata Michalska <b.michalska-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>>>>>> ---
>>>>>> Documentation/filesystems/events.txt | 231 ++++++++++
>>>>>> fs/Makefile | 1 +
>>>>>> fs/events/Makefile | 6 +
>>>>>> fs/events/fs_event.c | 770 ++++++++++++++++++++++++++++++++++
>>>>>> fs/events/fs_event.h | 25 ++
>>>>>> fs/events/fs_event_netlink.c | 99 +++++
>>>>>> fs/namespace.c | 1 +
>>>>>> include/linux/fs.h | 6 +-
>>>>>> include/linux/fs_event.h | 58 +++
>>>>>> include/uapi/linux/fs_event.h | 54 +++
>>>>>> include/uapi/linux/genetlink.h | 1 +
>>>>>> net/netlink/genetlink.c | 7 +-
>>>>>> 12 files changed, 1257 insertions(+), 2 deletions(-)
>>>>>> create mode 100644 Documentation/filesystems/events.txt
>>>>>> create mode 100644 fs/events/Makefile
>>>>>> create mode 100644 fs/events/fs_event.c
>>>>>> create mode 100644 fs/events/fs_event.h
>>>>>> create mode 100644 fs/events/fs_event_netlink.c
>>>>>> create mode 100644 include/linux/fs_event.h
>>>>>> create mode 100644 include/uapi/linux/fs_event.h
>>>>>
>>>>> Any reason why you just don't do uevents for the block devices today,
>>>>> and not create a new type of netlink message and userspace tool required
>>>>> to read these?
>>>>
>>>> The idea here is to have support for filesystems with no backing device as well.
>>>> Parsing the message with libnl is really simple and requires few lines of code
>>>> (sample application has been presented in the initial version of this RFC)
>>>
>>> I'm not saying it's not "simple" to parse, just that now you are doing
>>> something that requires a different tool. If you have a block device,
>>> you should be able to emit uevents for it, you don't need a backing
>>> device, we handle virtual filesystems in /sys/block/ just fine :)
>>>
>>> People already have tools that listen to libudev for system monitoring
>>> and management, why require them to hook up to yet-another-library? And
>>> what is going to provide the ability for multiple userspace tools to
>>> listen to these netlink messages in case you have more than one program
>>> that wants to watch for these things (i.e. multiple desktop filesystem
>>> monitoring tools, system-health checkers, etc.)?
>> As much as I understand your concerns I'm not convinced uevent interface
>> is a good fit. There are filesystems that don't have underlying block
>> device - think of e.g. tmpfs or filesystems working directly on top of
>> flash devices. These still want to send notification to userspace (one of
>> primary motivation for this interfaces was so that tmpfs can notify about
>> something). And creating some fake nodes in /sys/block for tmpfs and
>> similar filesystems seems like doing more harm than good to me...
>
> If these are "fake" block devices, what's going to be present in the
> block major/minor fields of the netlink message? For some reason I
> thought it was a required field, and because of that, I thought we had a
> "real" filesystem somewhere to refer to, otherwise how would userspace
> know what filesystem was creating these events?
>
> What am I missing here?
>
> confused,
>
> greg k-h
>
For those 'fake' block devs, upon mount, get_anon_bdev will assign
the major:minor numbers. Userspace might get those through stat.
BR
Beata
^ permalink raw reply
* Re: Should mmap MAP_LOCKED fail if mm_poppulate fails?
From: Linus Torvalds @ 2015-04-28 16:01 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-mm, Cyril Hrubis, Andrew Morton, Hugh Dickins,
Michel Lespinasse, Rik van Riel, Michael Kerrisk, LKML, Linux API
In-Reply-To: <1430223111-14817-1-git-send-email-mhocko@suse.cz>
On Tue, Apr 28, 2015 at 5:11 AM, Michal Hocko <mhocko@suse.cz> wrote:
>
> The first patch is dumb and straightforward. It should be safe as is and
> also good without the follow up 2 patches which try to handle potential
> allocation failures in the do_munmap path more gracefully. As we still
> do not fail small allocations even the first patch could be simplified
> a bit and the retry loop replaced by a BUG_ON right away.
I think the BUG_ON() is a bad idea in the first place, and is in fact
a good reason to ignore the patch series entirely.
What is the point of that BUG_ON()?
Hell, people add too many of those things. There is *no* excuse for
killing the kernel for things like this (and in certain setups,
BUG_ON() *will* cause the machine to be rebooted). None. It's
completely inexcusable.
Thinking like this must go. BUG_ON() is for things where our internal
data structures are so corrupted that we don't know what to do, and
there's no way to continue. Not for "I want to sprinkle these things
around and this should not happen".
I also think that the whole complex "do_munmap_nofail()" is broken to
begin with, along with the crazy "!fatal_signal_pending()" thing.
There is absolutely no excuse for any of this.
Your code is also fundamentally buggy in that it tries to do unmap()
after it has dropped all locks, and things went wrong. So you may nto
be unmapping some other threads data.
There is no way in hell any of these patches can ever be applied.
There's a reason we don't handle populate failures - it's exactly
because we've dropped the locks etc. After dropping the locks, we
*cannot* clean up any more, because there's no way to know whather
we're cleaning up the right thing. You'd have to hold the write lock
over the whole populate, which has serious problems of its own.
So NAK on this series. I think just documenting the man-page might be
better. I don't think MAP_LOCKED is sanely fixable.
We might improve on MAP_LOCKED by having a heuristic up-front
(*before* actually doing any mmap) to verify that it's *likely* that
it will work. So we could return ENOMEM early if it looks like the
user would hit the resource limits, for example. That wouldn't be any
guarantee (another process might eat up the resource limit anyway),
and in fact it might be overly eager to fail (maybe the
mmap(MAP_LOCKED ends up unmapping an older locked mapping and we deny
it too eagerly), but it would probably work well enough in practice.
That, together with a warning in the man-page about mmap(MAP_LOCKED)
not being able to return "I only locked part of the mapping", if you
want full error handling you need to do mmap()+mlock() and check the
two errors separately.
Hmm? But I really dislike your patch-series as-is.
Linus
Linus
Linus
^ permalink raw reply
* Re: Should mmap MAP_LOCKED fail if mm_poppulate fails?
From: Michal Hocko @ 2015-04-28 16:43 UTC (permalink / raw)
To: Linus Torvalds
Cc: linux-mm, Cyril Hrubis, Andrew Morton, Hugh Dickins,
Michel Lespinasse, Rik van Riel, Michael Kerrisk, LKML, Linux API
In-Reply-To: <CA+55aFxzLXx=cC309h_tEc-Gkn_zH4ipR7PsefVcE-97Uj066g@mail.gmail.com>
On Tue 28-04-15 09:01:59, Linus Torvalds wrote:
[...]
> Your code is also fundamentally buggy in that it tries to do unmap()
> after it has dropped all locks, and things went wrong. So you may nto
> be unmapping some other threads data.
Hmm, no other thread has the address from the current mmap call except
for MAP_FIXED (more on that below).
Well I can imagine userspace doing nasty things like watching
/proc/self/maps and using the address from there or using an address as
an mmap hint and then using it before mmap returns by other threads. But
would those be valid usecases? They sound crazy and buggy to me.
Another nasty case would be MAP_FIXED from a different thread destroying
the mmap I am trying to poppulate but that is not so interesting because
nothing protects from that even now.
Or this being MAP_FIXED|MAP_LOCKED which has already destroyed a part
of somebody's else mapping and the cleanup would lead to an unexpected
SIGSEGV for the other thread. Is this the case you are worried about?
Or am I missing other cases?
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] usb: core: add usb3 lpm sysfs
From: Zhuang Jin Can @ 2015-04-28 16:51 UTC (permalink / raw)
To: Greg KH
Cc: rafael.j.wysocki, stern, dan.j.williams, pmladek, peter.chen,
jwerner, linux-api, linux-kernel, linux-usb
In-Reply-To: <20150428104224.GB540@kroah.com>
Hi Greg KH,
On Tue, Apr 28, 2015 at 12:42:24PM +0200, Greg KH wrote:
> On Sun, Apr 19, 2015 at 11:46:12AM +0800, Zhuang Jin Can wrote:
> > Some usb3 devices may not support usb3 lpm well.
> > The patch adds a sysfs to enable/disable u1 or u2 of the port.The
> > settings apply to both before and after device enumeration.
> > Supported values are "0" - u1 and u2 are disabled, "u1" - only u1 is
> > enabled, "u2" - only u2 is enabled, "u1_u2" - u1 and u2 are enabled.
> >
> > The interface is useful for testing some USB3 devices during
> > development, and provides a way to disable usb3 lpm if the issues can
> > not be fixed in final products.
>
> How is a user supposed to "know" to make this setting for a device? Why
> can't the kernel automatically set this value properly? Why does it
> need to be a kernel issue at all?
>
By default kernel enables u1 u2 of all USB3 devices. This interface
provides the user to change this policy. User may set the policy
according to PID/VID of uevent or according to the platform information
known by userspace.
It's not a kernel issue, as u1 u2 is mandatory by USB3 compliance. But
for some internal hardwired USB3 connection, e.g. SSIC, passing USB3
compliance is not mandatory. So the interface provides a way for vendor
to ship with u1 or u2 broken products. Of course, this is not encouraged :).
> And when you are doing development of broken devices, the kernel doesn't
> have to support you, you can run with debugging patches of your own
> until you fix your firmware :)
>
Understood. But I think other vendor or developer may face the same
issue in final product shipment or during development. Moreover, the
interface provide the flexibility for developer to separately
disable/enable u1 or u2, e.g. If they're debugging an u2 issue, they
can disable u1 to simplify the situtation.
Thanks
Jincan
^ permalink raw reply
* Re: [PATCH v3 1/4] fs: configfs: Add unlocked version of configfs_depend_item()
From: Felipe Balbi @ 2015-04-28 16:52 UTC (permalink / raw)
To: Krzysztof Opasiak
Cc: balbi-l0cyMroinI0, gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r,
jlbec-aKy9MeLSZ9dg9hUCZPvPmw, andrzej.p-Sze3O3UU22JBDgjK7y7TUQ,
m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-usb-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <1428596290-23637-2-git-send-email-k.opasiak-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 3016 bytes --]
Hi,
On Thu, Apr 09, 2015 at 06:18:07PM +0200, Krzysztof Opasiak wrote:
> Sometimes it might be desirable to prohibit removing a directory
> in configfs. One example is USB gadget (mass_storage function):
> when gadget is already bound, if lun directory is removed,
> the gadget must be thrown away, too. A better solution would be
> to fail with e.g. -EBUSY.
>
> Currently configfs has configfs_depend/undepend_item() methods
> but they cannot be used in configfs callbacks. This commit
> adds unlocked version of this methods which can be used
> only in configfs callbacks.
>
> Signed-off-by: Krzysztof Opasiak <k.opasiak-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
Joel, any comments to this patch ?
> ---
> fs/configfs/dir.c | 29 +++++++++++++++++++++++++++++
> include/linux/configfs.h | 9 +++++++++
> 2 files changed, 38 insertions(+)
>
> diff --git a/fs/configfs/dir.c b/fs/configfs/dir.c
> index cf0db00..7875a5e 100644
> --- a/fs/configfs/dir.c
> +++ b/fs/configfs/dir.c
> @@ -1152,6 +1152,35 @@ void configfs_undepend_item(struct configfs_subsystem *subsys,
> }
> EXPORT_SYMBOL(configfs_undepend_item);
>
> +int configfs_depend_item_unlocked(struct config_item *target)
> +{
> + struct configfs_dirent *sd;
> + int ret = -ENOENT;
> +
> + spin_lock(&configfs_dirent_lock);
> + BUG_ON(!target->ci_dentry);
> +
> + sd = target->ci_dentry->d_fsdata;
> + if ((sd->s_type & CONFIGFS_DIR) &&
> + ((sd->s_type & CONFIGFS_USET_DROPPING) ||
> + (sd->s_type & CONFIGFS_USET_CREATING)))
> + goto out_unlock_dirent_lock;
> +
> + sd->s_dependent_count += 1;
> + ret = 0;
> +
> +out_unlock_dirent_lock:
> + spin_unlock(&configfs_dirent_lock);
> + return ret;
> +}
> +EXPORT_SYMBOL(configfs_depend_item_unlocked);
> +
> +void configfs_undepend_item_unlocked(struct config_item *target)
> +{
> + configfs_undepend_item(NULL, target);
> +}
> +EXPORT_SYMBOL(configfs_undepend_item_unlocked);
> +
> static int configfs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode)
> {
> int ret = 0;
> diff --git a/include/linux/configfs.h b/include/linux/configfs.h
> index 34025df..e9dbf01 100644
> --- a/include/linux/configfs.h
> +++ b/include/linux/configfs.h
> @@ -257,4 +257,13 @@ void configfs_unregister_subsystem(struct configfs_subsystem *subsys);
> int configfs_depend_item(struct configfs_subsystem *subsys, struct config_item *target);
> void configfs_undepend_item(struct configfs_subsystem *subsys, struct config_item *target);
>
> +/*
> + * These functions can sleep and can alloc with GFP_KERNEL
> + * NOTE: These should be called only underneath configfs callbacks.
> + * WARNING: These cannot be called on newly created item
> + * (in make_group()/make_item callback)
> + */
> +int configfs_depend_item_unlocked(struct config_item *target);
> +void configfs_undepend_item_unlocked(struct config_item *target);
> +
> #endif /* _CONFIGFS_H_ */
> --
> 1.7.9.5
>
--
balbi
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply
* Re: Should mmap MAP_LOCKED fail if mm_poppulate fails?
From: Linus Torvalds @ 2015-04-28 16:57 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-mm, Cyril Hrubis, Andrew Morton, Hugh Dickins,
Michel Lespinasse, Rik van Riel, Michael Kerrisk, LKML, Linux API
In-Reply-To: <20150428164302.GI2659@dhcp22.suse.cz>
On Tue, Apr 28, 2015 at 9:43 AM, Michal Hocko <mhocko@suse.cz> wrote:
>
> Hmm, no other thread has the address from the current mmap call except
> for MAP_FIXED (more on that below).
With things like opportunistic SIGSEGV handlers that map/unmap things
as the user takes faults, that's actually not at all guaranteed.
Yeah, it's unusual, but I've seen it, with threaded applications where
people play games with user-space memory management, and do "demand
allocation" with mmap() in response to signals.
Admittedly we already do bad things in mmap(MAP_FIXED) for that case,
since we dropped the vm lock. But at least it shouldn't be any worse
than a thread speculatively touching the pages..
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC v2 1/4] fs: Add generic file system event notifications
From: Greg KH @ 2015-04-28 17:39 UTC (permalink / raw)
To: Beata Michalska
Cc: Jan Kara, linux-kernel, linux-fsdevel, linux-api, tytso,
adilger.kernel, hughd, lczerner, hch, linux-ext4, linux-mm,
kyungmin.park, kmpark
In-Reply-To: <553F9D56.6030301@samsung.com>
On Tue, Apr 28, 2015 at 04:46:46PM +0200, Beata Michalska wrote:
> On 04/28/2015 04:09 PM, Greg KH wrote:
> > On Tue, Apr 28, 2015 at 03:56:53PM +0200, Jan Kara wrote:
> >> On Mon 27-04-15 17:37:11, Greg KH wrote:
> >>> On Mon, Apr 27, 2015 at 05:08:27PM +0200, Beata Michalska wrote:
> >>>> On 04/27/2015 04:24 PM, Greg KH wrote:
> >>>>> On Mon, Apr 27, 2015 at 01:51:41PM +0200, Beata Michalska wrote:
> >>>>>> Introduce configurable generic interface for file
> >>>>>> system-wide event notifications, to provide file
> >>>>>> systems with a common way of reporting any potential
> >>>>>> issues as they emerge.
> >>>>>>
> >>>>>> The notifications are to be issued through generic
> >>>>>> netlink interface by newly introduced multicast group.
> >>>>>>
> >>>>>> Threshold notifications have been included, allowing
> >>>>>> triggering an event whenever the amount of free space drops
> >>>>>> below a certain level - or levels to be more precise as two
> >>>>>> of them are being supported: the lower and the upper range.
> >>>>>> The notifications work both ways: once the threshold level
> >>>>>> has been reached, an event shall be generated whenever
> >>>>>> the number of available blocks goes up again re-activating
> >>>>>> the threshold.
> >>>>>>
> >>>>>> The interface has been exposed through a vfs. Once mounted,
> >>>>>> it serves as an entry point for the set-up where one can
> >>>>>> register for particular file system events.
> >>>>>>
> >>>>>> Signed-off-by: Beata Michalska <b.michalska@samsung.com>
> >>>>>> ---
> >>>>>> Documentation/filesystems/events.txt | 231 ++++++++++
> >>>>>> fs/Makefile | 1 +
> >>>>>> fs/events/Makefile | 6 +
> >>>>>> fs/events/fs_event.c | 770 ++++++++++++++++++++++++++++++++++
> >>>>>> fs/events/fs_event.h | 25 ++
> >>>>>> fs/events/fs_event_netlink.c | 99 +++++
> >>>>>> fs/namespace.c | 1 +
> >>>>>> include/linux/fs.h | 6 +-
> >>>>>> include/linux/fs_event.h | 58 +++
> >>>>>> include/uapi/linux/fs_event.h | 54 +++
> >>>>>> include/uapi/linux/genetlink.h | 1 +
> >>>>>> net/netlink/genetlink.c | 7 +-
> >>>>>> 12 files changed, 1257 insertions(+), 2 deletions(-)
> >>>>>> create mode 100644 Documentation/filesystems/events.txt
> >>>>>> create mode 100644 fs/events/Makefile
> >>>>>> create mode 100644 fs/events/fs_event.c
> >>>>>> create mode 100644 fs/events/fs_event.h
> >>>>>> create mode 100644 fs/events/fs_event_netlink.c
> >>>>>> create mode 100644 include/linux/fs_event.h
> >>>>>> create mode 100644 include/uapi/linux/fs_event.h
> >>>>>
> >>>>> Any reason why you just don't do uevents for the block devices today,
> >>>>> and not create a new type of netlink message and userspace tool required
> >>>>> to read these?
> >>>>
> >>>> The idea here is to have support for filesystems with no backing device as well.
> >>>> Parsing the message with libnl is really simple and requires few lines of code
> >>>> (sample application has been presented in the initial version of this RFC)
> >>>
> >>> I'm not saying it's not "simple" to parse, just that now you are doing
> >>> something that requires a different tool. If you have a block device,
> >>> you should be able to emit uevents for it, you don't need a backing
> >>> device, we handle virtual filesystems in /sys/block/ just fine :)
> >>>
> >>> People already have tools that listen to libudev for system monitoring
> >>> and management, why require them to hook up to yet-another-library? And
> >>> what is going to provide the ability for multiple userspace tools to
> >>> listen to these netlink messages in case you have more than one program
> >>> that wants to watch for these things (i.e. multiple desktop filesystem
> >>> monitoring tools, system-health checkers, etc.)?
> >> As much as I understand your concerns I'm not convinced uevent interface
> >> is a good fit. There are filesystems that don't have underlying block
> >> device - think of e.g. tmpfs or filesystems working directly on top of
> >> flash devices. These still want to send notification to userspace (one of
> >> primary motivation for this interfaces was so that tmpfs can notify about
> >> something). And creating some fake nodes in /sys/block for tmpfs and
> >> similar filesystems seems like doing more harm than good to me...
> >
> > If these are "fake" block devices, what's going to be present in the
> > block major/minor fields of the netlink message? For some reason I
> > thought it was a required field, and because of that, I thought we had a
> > "real" filesystem somewhere to refer to, otherwise how would userspace
> > know what filesystem was creating these events?
> >
> > What am I missing here?
> >
> > confused,
> >
> > greg k-h
> >
>
> For those 'fake' block devs, upon mount, get_anon_bdev will assign
> the major:minor numbers. Userspace might get those through stat.
How can userspace do the mapping backwards from this "anonymous"
major:minor number for these types of filesystems in such a way that
they can "know" how to report the block device that is causing the
event?
thanks,
greg k-h
^ permalink raw reply
* Re: Should mmap MAP_LOCKED fail if mm_poppulate fails?
From: Michal Hocko @ 2015-04-28 18:35 UTC (permalink / raw)
To: Linus Torvalds
Cc: linux-mm, Cyril Hrubis, Andrew Morton, Hugh Dickins,
Michel Lespinasse, Rik van Riel, Michael Kerrisk, LKML, Linux API
In-Reply-To: <CA+55aFydkG-BgZzry5DrTzueVh9VvEcVJdLV8iOyUphQk=0vpw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Tue 28-04-15 09:57:11, Linus Torvalds wrote:
> On Tue, Apr 28, 2015 at 9:43 AM, Michal Hocko <mhocko-AlSwsSmVLrQ@public.gmane.org> wrote:
> >
> > Hmm, no other thread has the address from the current mmap call except
> > for MAP_FIXED (more on that below).
>
> With things like opportunistic SIGSEGV handlers that map/unmap things
> as the user takes faults, that's actually not at all guaranteed.
>
> Yeah, it's unusual, but I've seen it, with threaded applications where
> people play games with user-space memory management, and do "demand
> allocation" with mmap() in response to signals.
I am still not sure I see the problem here. Let's say we have a
userspace page fault handler which would do mmap(fault_addr, MAP_FIXED),
right?
If we had a racy mmap(NULL, MAP_LOCKED) that could have mapped
fault_addr by the time handler does its work then this is buggy wrt. to
MAP_LOCKED semantic because the fault handler would discard the locked
part. This wouldn't lead to a data loss but still makes MAP_LOCKED usage
buggy IMO.
If the racing thread did mmap(around_fault_addr, MAP_FIXED|MAP_LOCKED)
then it would be broken as well, and even worse I would say, because the
original fault could have been discarded and data lost.
I would expect that user fault handlers would be synchronized with
other mmap activity otherwise I have hard time to see how this can all
have a well defined behavior. Especially when MAP_FIXED is involved.
> Admittedly we already do bad things in mmap(MAP_FIXED) for that case,
> since we dropped the vm lock. But at least it shouldn't be any worse
> than a thread speculatively touching the pages..
Actually we already allow to mmap(MAP_FIXED) to fail after
discarding an existing mmaped area (see mmap_region and e.g.
security_vm_enough_memory_mm or other failure cases).
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* Re: Should mmap MAP_LOCKED fail if mm_poppulate fails?
From: Linus Torvalds @ 2015-04-28 18:38 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-mm, Cyril Hrubis, Andrew Morton, Hugh Dickins,
Michel Lespinasse, Rik van Riel, Michael Kerrisk, LKML, Linux API
In-Reply-To: <20150428183535.GB30918-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org>
On Tue, Apr 28, 2015 at 11:35 AM, Michal Hocko <mhocko-AlSwsSmVLrQ@public.gmane.org> wrote:
>
> I am still not sure I see the problem here.
Basically, I absolutely hate the notion of us doing something
unsynchronized, when I can see us undoing a mmap that another thread
is doing. It's wrong.
You also didn't react to all the *other* things that were wrong in
that patch-set. The games you play with !fatal_signal_pending() etc
are just crazy.
End result: I absolutely detest the whole thing. I told you what I
consider an acceptable solution instead, that is much simpler and
doesn't have any of the problems of your patchset.
Linus
^ permalink raw reply
* Re: Should mmap MAP_LOCKED fail if mm_poppulate fails?
From: Michal Hocko @ 2015-04-28 20:21 UTC (permalink / raw)
To: Linus Torvalds
Cc: linux-mm, Cyril Hrubis, Andrew Morton, Hugh Dickins,
Michel Lespinasse, Rik van Riel, Michael Kerrisk, LKML, Linux API
In-Reply-To: <CA+55aFxzLXx=cC309h_tEc-Gkn_zH4ipR7PsefVcE-97Uj066g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On Tue 28-04-15 09:01:59, Linus Torvalds wrote:
> On Tue, Apr 28, 2015 at 5:11 AM, Michal Hocko <mhocko-AlSwsSmVLrQ@public.gmane.org> wrote:
> >
> > The first patch is dumb and straightforward. It should be safe as is and
> > also good without the follow up 2 patches which try to handle potential
> > allocation failures in the do_munmap path more gracefully. As we still
> > do not fail small allocations even the first patch could be simplified
> > a bit and the retry loop replaced by a BUG_ON right away.
>
> I think the BUG_ON() is a bad idea in the first place, and is in fact
> a good reason to ignore the patch series entirely.
> What is the point of that BUG_ON()?
>
> Hell, people add too many of those things. There is *no* excuse for
> killing the kernel for things like this (and in certain setups,
> BUG_ON() *will* cause the machine to be rebooted). None. It's
> completely inexcusable.
>
> Thinking like this must go. BUG_ON() is for things where our internal
> data structures are so corrupted that we don't know what to do, and
> there's no way to continue. Not for "I want to sprinkle these things
> around and this should not happen".
The BUG_ON in do_munmap_nofail was to catch an unexpected failure
mode which would be caused by later changes. So it was a way to express
an invariant.
Anyway I understand your point above.
> I also think that the whole complex "do_munmap_nofail()" is broken to
> begin with,
Could you be more specific please?
> along with the crazy "!fatal_signal_pending()" thing.
The primary motivation was to back out when we know that the whole
thread group will go and we will cleanup the whole state anyway. As
the only real reason to fail do_munmap is an allocation failure (the
sysctl_max_map_count one is pro-actively avoided) then this basically
means that we have been OOM killed.
On the other hand the allocating thread will get TIF_MEMDIE and access
to memory reserves sooner or later if we are really OOM so the explicit
check is not really needed and it can be dropped.
> There is absolutely no excuse for any of this.
>
> Your code is also fundamentally buggy in that it tries to do unmap()
> after it has dropped all locks, and things went wrong. So you may nto
> be unmapping some other threads data.
>
> There is no way in hell any of these patches can ever be applied.
>
> There's a reason we don't handle populate failures - it's exactly
> because we've dropped the locks etc. After dropping the locks, we
> *cannot* clean up any more, because there's no way to know whather
> we're cleaning up the right thing. You'd have to hold the write lock
> over the whole populate, which has serious problems of its own.
>
> So NAK on this series. I think just documenting the man-page might be
> better. I don't think MAP_LOCKED is sanely fixable.
I am OK with this answer as well. Users who really need no-later faults
behavior should use mlock(). I will cook up a patch for man pages and
post it tomorrow.
> We might improve on MAP_LOCKED by having a heuristic up-front
> (*before* actually doing any mmap) to verify that it's *likely* that
> it will work. So we could return ENOMEM early if it looks like the
> user would hit the resource limits, for example. That wouldn't be any
> guarantee (another process might eat up the resource limit anyway),
> and in fact it might be overly eager to fail (maybe the
> mmap(MAP_LOCKED ends up unmapping an older locked mapping and we deny
> it too eagerly), but it would probably work well enough in practice.
As you've said. This would be inherently racy. Some of those checks are
already done before any destructive actions but this doesn't cover all
of them and certainly cannot cover the area fault in by definition.
> That, together with a warning in the man-page about mmap(MAP_LOCKED)
> not being able to return "I only locked part of the mapping", if you
> want full error handling you need to do mmap()+mlock() and check the
> two errors separately.
>
> Hmm? But I really dislike your patch-series as-is.
>
> Linus
>
> Linus
>
> Linus
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* Re: [PATCH 01/11] coresight-etm4x: Adding CoreSight ETM4x driver
From: Mathieu Poirier @ 2015-04-28 20:32 UTC (permalink / raw)
To: Ivan T. Ivanov
Cc: Greg KH,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Kaixu Xia,
Chunyan Zhang
In-Reply-To: <1430209858.1904.1.camel-NEYub+7Iv8PQT0dZR+AlfA@public.gmane.org>
On 28 April 2015 at 02:30, Ivan T. Ivanov <iivanov-NEYub+7Iv8PQT0dZR+AlfA@public.gmane.org> wrote:
>
> On Mon, 2015-04-27 at 09:48 -0600, Mathieu Poirier wrote:
>> On 24 April 2015 at 09:41, Ivan T. Ivanov <iivanov-NEYub+7Iv8PQT0dZR+AlfA@public.gmane.org> wrote:
>> > On Wed, 2015-04-22 at 16:40 -0600, Mathieu Poirier wrote:
>> >
>> > > +static struct amba_id etm4_ids[] = {
>> > > + { /* ETM 4.0 - Hi6220 board */
>> > > + .id = 0x0003b95d,
>> > > + .mask = 0x0003ffff,
>> > > + .data = "ETM 4.0",
>> > > + },
>> > > + { /* ETM 4.0 - Juno board */
>> > > + .id = 0x000bb95e,
>> > > + .mask = 0x000bffff,
>> >
>> > Mask looks suspicious.
>>
>> Can you please expand the "suspicious" part ?
>
> Well, 'b' part of the mask. I have to admit that
> I don't know how this is mapped on this platform.
Yes, I see it now - thanks for pointing that out.
>
> Regards,
> Ivan
^ permalink raw reply
* Re: Should mmap MAP_LOCKED fail if mm_poppulate fails?
From: Michal Hocko @ 2015-04-28 20:36 UTC (permalink / raw)
To: Linus Torvalds
Cc: linux-mm, Cyril Hrubis, Andrew Morton, Hugh Dickins,
Michel Lespinasse, Rik van Riel, Michael Kerrisk, LKML, Linux API
In-Reply-To: <CA+55aFyajquhGhw59qNWKGK4dBV0TPmDD7-1XqPo7DZWvO_hPg@mail.gmail.com>
On Tue 28-04-15 11:38:35, Linus Torvalds wrote:
> On Tue, Apr 28, 2015 at 11:35 AM, Michal Hocko <mhocko@suse.cz> wrote:
> >
> > I am still not sure I see the problem here.
>
> Basically, I absolutely hate the notion of us doing something
> unsynchronized, when I can see us undoing a mmap that another thread
> is doing. It's wrong.
>
> You also didn't react to all the *other* things that were wrong in
> that patch-set. The games you play with !fatal_signal_pending() etc
> are just crazy.
I planed to get to those later, because I felt the locks vs. racing
mmaps argument was the most important objection.
> End result: I absolutely detest the whole thing. I told you what I
> consider an acceptable solution instead, that is much simpler and
> doesn't have any of the problems of your patchset.
I will surely think about those. As I've written in the cover email
already, I am fine with patching the man page and be clear about a long
term behavior. The primary motivation for this RFC was to start the
discussion.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH] usb: core: add usb3 lpm sysfs
From: Greg KH @ 2015-04-28 21:11 UTC (permalink / raw)
To: Zhuang Jin Can
Cc: rafael.j.wysocki, stern, dan.j.williams, pmladek, peter.chen,
jwerner, linux-api, linux-kernel, linux-usb
In-Reply-To: <20150428165127.GA26638@intel.com>
On Wed, Apr 29, 2015 at 12:51:27AM +0800, Zhuang Jin Can wrote:
> Hi Greg KH,
>
> On Tue, Apr 28, 2015 at 12:42:24PM +0200, Greg KH wrote:
> > On Sun, Apr 19, 2015 at 11:46:12AM +0800, Zhuang Jin Can wrote:
> > > Some usb3 devices may not support usb3 lpm well.
> > > The patch adds a sysfs to enable/disable u1 or u2 of the port.The
> > > settings apply to both before and after device enumeration.
> > > Supported values are "0" - u1 and u2 are disabled, "u1" - only u1 is
> > > enabled, "u2" - only u2 is enabled, "u1_u2" - u1 and u2 are enabled.
> > >
> > > The interface is useful for testing some USB3 devices during
> > > development, and provides a way to disable usb3 lpm if the issues can
> > > not be fixed in final products.
> >
> > How is a user supposed to "know" to make this setting for a device? Why
> > can't the kernel automatically set this value properly? Why does it
> > need to be a kernel issue at all?
> >
> By default kernel enables u1 u2 of all USB3 devices. This interface
> provides the user to change this policy. User may set the policy
> according to PID/VID of uevent or according to the platform information
> known by userspace.
And why would they ever want to do that?
> It's not a kernel issue, as u1 u2 is mandatory by USB3 compliance. But
> for some internal hardwired USB3 connection, e.g. SSIC, passing USB3
> compliance is not mandatory. So the interface provides a way for vendor
> to ship with u1 or u2 broken products. Of course, this is not encouraged :).
If the state is broken for those devices, we can't require the user to
fix it for us, the kernel should do it automatically.
> > And when you are doing development of broken devices, the kernel doesn't
> > have to support you, you can run with debugging patches of your own
> > until you fix your firmware :)
> >
> Understood. But I think other vendor or developer may face the same
> issue in final product shipment or during development. Moreover, the
> interface provide the flexibility for developer to separately
> disable/enable u1 or u2, e.g. If they're debugging an u2 issue, they
> can disable u1 to simplify the situtation.
For debugging only, perhaps, but for a "normal" user, please let's
handle this automatically and don't create a switch that never gets used
by anyone or anything.
thanks,
greg k-h
^ permalink raw reply
* Re: [RFC 1/3] mm: mmap make MAP_LOCKED really mlock semantic
From: Andrew Morton @ 2015-04-28 23:10 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-mm-Bw31MaZKKs3YtjvyW6yDsg, Cyril Hrubis, Hugh Dickins,
Michel Lespinasse, Linus Torvalds, Rik van Riel, Michael Kerrisk,
LKML, Linux API
In-Reply-To: <1430223111-14817-2-git-send-email-mhocko-AlSwsSmVLrQ@public.gmane.org>
On Tue, 28 Apr 2015 14:11:49 +0200 Michal Hocko <mhocko-AlSwsSmVLrQ@public.gmane.org> wrote:
> The man page however says
> "
> MAP_LOCKED (since Linux 2.5.37)
> Lock the pages of the mapped region into memory in the manner of
> mlock(2). This flag is ignored in older kernels.
> "
I'm trying to remember why we implemented MAP_LOCKED in the first
place. Was it better than mmap+mlock in some fashion?
afaict we had a #define MAP_LOCKED in the header file but it wasn't
implemented, so we went and wired it up. 13 years ago:
https://lkml.org/lkml/2002/9/18/108
Anyway... the third way of doing this is to use plain old mmap() while
mlockall(MCL_FUTURE) is in force. Has anyone looked at that, checked
that the behaviour is sane and compared it with the mmap+mlock
behaviour, the MAP_LOCKED behaviour and the manpages?
^ permalink raw reply
* Re: [PATCH v3 0/3] idle memory tracking
From: Minchan Kim @ 2015-04-29 3:57 UTC (permalink / raw)
To: Vladimir Davydov
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Greg Thelen,
Michel Lespinasse, David Rientjes, Pavel Emelyanov,
Cyrill Gorcunov, Jonathan Corbet, linux-api, linux-doc, linux-mm,
cgroups, linux-kernel
In-Reply-To: <cover.1430217477.git.vdavydov@parallels.com>
Hello Vladimir,
On Tue, Apr 28, 2015 at 03:24:39PM +0300, Vladimir Davydov wrote:
> Hi,
>
> This patch set introduces a new user API for tracking user memory pages
> that have not been used for a given period of time. The purpose of this
> is to provide the userspace with the means of tracking a workload's
> working set, i.e. the set of pages that are actively used by the
> workload. Knowing the working set size can be useful for partitioning
> the system more efficiently, e.g. by tuning memory cgroup limits
> appropriately, or for job placement within a compute cluster.
>
> ---- USE CASES ----
>
> The unified cgroup hierarchy has memory.low and memory.high knobs, which
> are defined as the low and high boundaries for the workload working set
> size. However, the working set size of a workload may be unknown or
> change in time. With this patch set, one can periodically estimate the
> amount of memory unused by each cgroup and tune their memory.low and
> memory.high parameters accordingly, therefore optimizing the overall
> memory utilization.
>
> Another use case is balancing workloads within a compute cluster.
> Knowing how much memory is not really used by a workload unit may help
> take a more optimal decision when considering migrating the unit to
> another node within the cluster.
>
> ---- USER API ----
>
> The user API consists of two new proc files:
>
> * /proc/kpageidle. For each page this file contains a 64-bit number, which
> equals 1 if the page is idle or 0 otherwise, indexed by PFN. A page is
Why do we need 64bit per page to indicate just idle or not?
What do you imagine we're happy with other 63bit in future?
--
Kind regards,
Minchan Kim
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v3 3/3] proc: add kpageidle file
From: Minchan Kim @ 2015-04-29 4:35 UTC (permalink / raw)
To: Vladimir Davydov
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Greg Thelen,
Michel Lespinasse, David Rientjes, Pavel Emelyanov,
Cyrill Gorcunov, Jonathan Corbet, linux-api, linux-doc, linux-mm,
cgroups, linux-kernel
In-Reply-To: <4c24a6bf2c9711dd4dbb72a43a16eba6867527b7.1430217477.git.vdavydov@parallels.com>
On Tue, Apr 28, 2015 at 03:24:42PM +0300, Vladimir Davydov wrote:
> Knowing the portion of memory that is not used by a certain application
> or memory cgroup (idle memory) can be useful for partitioning the system
> efficiently, e.g. by setting memory cgroup limits appropriately.
> Currently, the only means to estimate the amount of idle memory provided
> by the kernel is /proc/PID/{clear_refs,smaps}: the user can clear the
> access bit for all pages mapped to a particular process by writing 1 to
> clear_refs, wait for some time, and then count smaps:Referenced.
> However, this method has two serious shortcomings:
>
> - it does not count unmapped file pages
> - it affects the reclaimer logic
>
> To overcome these drawbacks, this patch introduces two new page flags,
> Idle and Young, and a new proc file, /proc/kpageidle. A page's Idle flag
> can only be set from userspace by writing 1 to /proc/kpageidle at the
> offset corresponding to the page, and it is cleared whenever the page is
> accessed either through page tables (it is cleared in page_referenced()
> in this case) or using the read(2) system call (mark_page_accessed()).
> Thus by setting the Idle flag for pages of a particular workload, which
> can be found e.g. by reading /proc/PID/pagemap, waiting for some time to
> let the workload access its working set, and then reading the kpageidle
> file, one can estimate the amount of pages that are not used by the
> workload.
>
> The Young page flag is used to avoid interference with the memory
> reclaimer. A page's Young flag is set whenever the Access bit of a page
> table entry pointing to the page is cleared by writing to kpageidle. If
> page_referenced() is called on a Young page, it will add 1 to its return
> value, therefore concealing the fact that the Access bit was cleared.
>
> Note, since there is no room for extra page flags on 32 bit, this
> feature uses extended page flags when compiled on 32 bit.
Thanks for considering 32bit.
Anyway, I believe it's good feature but not sure it's worth to consume
2bit of page flag but have no idea with saving page flags.
>
> Signed-off-by: Vladimir Davydov <vdavydov@parallels.com>
> ---
> Documentation/vm/pagemap.txt | 10 ++-
> fs/proc/page.c | 154 ++++++++++++++++++++++++++++++++++++++++++
> fs/proc/task_mmu.c | 4 +-
> include/linux/mm.h | 88 ++++++++++++++++++++++++
> include/linux/page-flags.h | 9 +++
> include/linux/page_ext.h | 4 ++
> mm/Kconfig | 12 ++++
> mm/debug.c | 4 ++
> mm/page_ext.c | 3 +
> mm/rmap.c | 7 ++
> mm/swap.c | 2 +
> 11 files changed, 295 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/vm/pagemap.txt b/Documentation/vm/pagemap.txt
> index a9b7afc8fbc6..ac6fd32a9296 100644
> --- a/Documentation/vm/pagemap.txt
> +++ b/Documentation/vm/pagemap.txt
> @@ -5,7 +5,7 @@ pagemap is a new (as of 2.6.25) set of interfaces in the kernel that allow
> userspace programs to examine the page tables and related information by
> reading files in /proc.
>
> -There are four components to pagemap:
> +There are five components to pagemap:
>
> * /proc/pid/pagemap. This file lets a userspace process find out which
> physical frame each virtual page is mapped to. It contains one 64-bit
> @@ -69,6 +69,14 @@ There are four components to pagemap:
> memory cgroup each page is charged to, indexed by PFN. Only available when
> CONFIG_MEMCG is set.
>
> + * /proc/kpageidle. For each page this file contains a 64-bit number, which
> + equals 1 if the page is idle or 0 otherwise, indexed by PFN. A page is
As I replied to the cover letter, we need justification consume 64bit per page.
> + considered idle if it has not been accessed since it was marked idle. To
> + mark a page idle one should write 1 to this file at the offset corresponding
> + to the page. Only user memory pages can be marked idle, for other page types
> + input is silently ignored. Writing to this file beyond max PFN results in
> + the ENXIO error. Only available when CONFIG_IDLE_PAGE_TRACKING is set.
> +
> Short descriptions to the page flags:
>
> 0. LOCKED
> diff --git a/fs/proc/page.c b/fs/proc/page.c
> index 70d23245dd43..cfc55ba7fee6 100644
> --- a/fs/proc/page.c
> +++ b/fs/proc/page.c
> @@ -275,6 +275,156 @@ static const struct file_operations proc_kpagecgroup_operations = {
> };
> #endif /* CONFIG_MEMCG */
>
> +#ifdef CONFIG_IDLE_PAGE_TRACKING
> +static struct page *kpageidle_get_page(unsigned long pfn)
> +{
> + struct page *page;
> +
> + if (!pfn_valid(pfn))
> + return NULL;
> + page = pfn_to_page(pfn);
> + /*
> + * We are only interested in user memory pages, i.e. pages that are
> + * allocated and on an LRU list.
> + */
> + if (!page || page_count(page) == 0 || !PageLRU(page))
Why do you check (page_count == 0) even if we check it with get_page_unless_zero
below?
> + return NULL;
> + if (!get_page_unless_zero(page))
> + return NULL;
> + if (unlikely(!PageLRU(page))) {
What lock protect the check PageLRU?
If it is racing ClearPageLRU, what happens?
> + put_page(page);
> + return NULL;
> + }
> + return page;
> +}
> +
> +static void kpageidle_clear_refs(struct page *page)
> +{
> + unsigned long dummy;
> +
> + if (page_referenced(page, 0, NULL, &dummy))
> + /*
> + * This page was referenced. To avoid interference with the
> + * reclaimer, mark it young so that the next call will also
next what call?
It just works with mapped page so kpageidle_clear_pte_refs as function name
is more clear.
One more, kpageidle_clear_refs removes PG_idle via page_referenced which
is important feature for the function. Please document it so we could
understand why we need double check for PG_idle after calling
kpageidle_clear_refs for pte access bit.
> + * return > 0 (see page_referenced_one)
> + */
> + set_page_young(page);
> +}
> +
> +static ssize_t kpageidle_read(struct file *file, char __user *buf,
> + size_t count, loff_t *ppos)
> +{
> + u64 __user *out = (u64 __user *)buf;
> + struct page *page;
> + unsigned long src = *ppos;
> + unsigned long pfn;
> + ssize_t ret = 0;
> + u64 val;
> +
> + pfn = src / KPMSIZE;
> + count = min_t(unsigned long, count, (max_pfn * KPMSIZE) - src);
> + if (src & KPMMASK || count & KPMMASK)
> + return -EINVAL;
> +
> + while (count > 0) {
> + val = 0;
> + page = kpageidle_get_page(pfn);
> + if (page) {
> + if (page_is_idle(page)) {
> + /*
> + * The page might have been referenced via a
> + * pte, in which case it is not idle. Clear
> + * refs and recheck.
> + */
> + kpageidle_clear_refs(page);
> + if (page_is_idle(page))
> + val = 1;
> + }
> + put_page(page);
> + }
> +
> + if (put_user(val, out)) {
> + ret = -EFAULT;
> + break;
> + }
> +
> + pfn++;
> + out++;
> + count -= KPMSIZE;
> + }
> +
> + *ppos += (char __user *)out - buf;
> + if (!ret)
> + ret = (char __user *)out - buf;
> + return ret;
> +}
> +
> +static ssize_t kpageidle_write(struct file *file, const char __user *buf,
> + size_t count, loff_t *ppos)
> +{
> + const u64 __user *in = (u64 __user *)buf;
> + struct page *page;
> + unsigned long src = *ppos;
> + unsigned long pfn;
> + ssize_t ret = 0;
> + u64 val;
> +
> + pfn = src / KPMSIZE;
> + if (src & KPMMASK || count & KPMMASK)
> + return -EINVAL;
> +
> + while (count > 0) {
> + if (pfn >= max_pfn) {
> + if ((char __user *)in == buf)
> + ret = -ENXIO;
> + break;
> + }
> +
> + if (get_user(val, in)) {
> + ret = -EFAULT;
> + break;
> + }
> +
> + if (val == 1) {
> + page = kpageidle_get_page(pfn);
> + if (page) {
> + kpageidle_clear_refs(page);
> + set_page_idle(page);
> + put_page(page);
> + }
> + } else if (val) {
> + ret = -EINVAL;
> + break;
> + }
> +
> + pfn++;
> + in++;
> + count -= KPMSIZE;
> + }
> +
> + *ppos += (char __user *)in - buf;
> + if (!ret)
> + ret = (char __user *)in - buf;
> + return ret;
> +}
> +
> +static const struct file_operations proc_kpageidle_operations = {
> + .llseek = mem_lseek,
> + .read = kpageidle_read,
> + .write = kpageidle_write,
> +};
> +
> +#ifndef CONFIG_64BIT
> +static bool need_page_idle(void)
> +{
> + return true;
> +}
> +struct page_ext_operations page_idle_ops = {
> + .need = need_page_idle,
> +};
> +#endif
> +#endif /* CONFIG_IDLE_PAGE_TRACKING */
> +
> static int __init proc_page_init(void)
> {
> proc_create("kpagecount", S_IRUSR, NULL, &proc_kpagecount_operations);
> @@ -282,6 +432,10 @@ static int __init proc_page_init(void)
> #ifdef CONFIG_MEMCG
> proc_create("kpagecgroup", S_IRUSR, NULL, &proc_kpagecgroup_operations);
> #endif
> +#ifdef CONFIG_IDLE_PAGE_TRACKING
> + proc_create("kpageidle", S_IRUSR | S_IWUSR, NULL,
> + &proc_kpageidle_operations);
> +#endif
> return 0;
> }
> fs_initcall(proc_page_init);
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 6dee68d013ff..ab04846f7dd5 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
> @@ -458,7 +458,7 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
>
> mss->resident += size;
> /* Accumulate the size in pages that have been accessed. */
> - if (young || PageReferenced(page))
> + if (young || page_is_young(page) || PageReferenced(page))
> mss->referenced += size;
> mapcount = page_mapcount(page);
> if (mapcount >= 2) {
> @@ -808,6 +808,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
>
> /* Clear accessed and referenced bits. */
> pmdp_test_and_clear_young(vma, addr, pmd);
> + clear_page_young(page);
> ClearPageReferenced(page);
> out:
> spin_unlock(ptl);
> @@ -835,6 +836,7 @@ out:
>
> /* Clear accessed and referenced bits. */
> ptep_test_and_clear_young(vma, addr, pte);
> + clear_page_young(page);
> ClearPageReferenced(page);
> }
> pte_unmap_unlock(pte - 1, ptl);
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 0755b9fd03a7..794d29aa2317 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2200,5 +2200,93 @@ void __init setup_nr_node_ids(void);
> static inline void setup_nr_node_ids(void) {}
> #endif
>
> +#ifdef CONFIG_IDLE_PAGE_TRACKING
> +#ifdef CONFIG_64BIT
> +static inline bool page_is_young(struct page *page)
> +{
> + return PageYoung(page);
> +}
> +
> +static inline void set_page_young(struct page *page)
> +{
> + SetPageYoung(page);
> +}
> +
> +static inline void clear_page_young(struct page *page)
> +{
> + ClearPageYoung(page);
> +}
> +
> +static inline bool page_is_idle(struct page *page)
> +{
> + return PageIdle(page);
> +}
> +
> +static inline void set_page_idle(struct page *page)
> +{
> + SetPageIdle(page);
> +}
> +
> +static inline void clear_page_idle(struct page *page)
> +{
> + ClearPageIdle(page);
> +}
> +#else /* !CONFIG_64BIT */
> +/*
> + * If there is not enough space to store Idle and Young bits in page flags, use
> + * page ext flags instead.
> + */
> +extern struct page_ext_operations page_idle_ops;
> +
> +static inline bool page_is_young(struct page *page)
> +{
> + return test_bit(PAGE_EXT_YOUNG, &lookup_page_ext(page)->flags);
> +}
> +
> +static inline void set_page_young(struct page *page)
> +{
> + set_bit(PAGE_EXT_YOUNG, &lookup_page_ext(page)->flags);
> +}
> +
> +static inline void clear_page_young(struct page *page)
> +{
> + clear_bit(PAGE_EXT_YOUNG, &lookup_page_ext(page)->flags);
> +}
> +
> +static inline bool page_is_idle(struct page *page)
> +{
> + return test_bit(PAGE_EXT_IDLE, &lookup_page_ext(page)->flags);
> +}
> +
> +static inline void set_page_idle(struct page *page)
> +{
> + set_bit(PAGE_EXT_IDLE, &lookup_page_ext(page)->flags);
> +}
> +
> +static inline void clear_page_idle(struct page *page)
> +{
> + clear_bit(PAGE_EXT_IDLE, &lookup_page_ext(page)->flags);
> +}
> +#endif /* CONFIG_64BIT */
> +#else /* !CONFIG_IDLE_PAGE_TRACKING */
> +static inline bool page_is_young(struct page *page)
> +{
> + return false;
> +}
> +
> +static inline void clear_page_young(struct page *page)
> +{
> +}
> +
> +static inline bool page_is_idle(struct page *page)
> +{
> + return false;
> +}
> +
> +static inline void clear_page_idle(struct page *page)
> +{
> +}
> +#endif /* CONFIG_IDLE_PAGE_TRACKING */
> +
> #endif /* __KERNEL__ */
> #endif /* _LINUX_MM_H */
> diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
> index f34e040b34e9..5e7c4f50a644 100644
> --- a/include/linux/page-flags.h
> +++ b/include/linux/page-flags.h
> @@ -109,6 +109,10 @@ enum pageflags {
> #ifdef CONFIG_TRANSPARENT_HUGEPAGE
> PG_compound_lock,
> #endif
> +#if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT)
> + PG_young,
> + PG_idle,
> +#endif
> __NR_PAGEFLAGS,
>
> /* Filesystems */
> @@ -289,6 +293,11 @@ PAGEFLAG_FALSE(HWPoison)
> #define __PG_HWPOISON 0
> #endif
>
> +#if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT)
> +PAGEFLAG(Young, young)
> +PAGEFLAG(Idle, idle)
> +#endif
> +
> /*
> * On an anonymous page mapped into a user virtual memory area,
> * page->mapping points to its anon_vma, not to a struct address_space;
> diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
> index c42981cd99aa..17f118a82854 100644
> --- a/include/linux/page_ext.h
> +++ b/include/linux/page_ext.h
> @@ -26,6 +26,10 @@ enum page_ext_flags {
> PAGE_EXT_DEBUG_POISON, /* Page is poisoned */
> PAGE_EXT_DEBUG_GUARD,
> PAGE_EXT_OWNER,
> +#if defined(CONFIG_IDLE_PAGE_TRACKING) && !defined(CONFIG_64BIT)
> + PAGE_EXT_YOUNG,
> + PAGE_EXT_IDLE,
> +#endif
> };
>
> /*
> diff --git a/mm/Kconfig b/mm/Kconfig
> index 390214da4546..3600eace4774 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -635,3 +635,15 @@ config MAX_STACK_SIZE_MB
> changed to a smaller value in which case that is used.
>
> A sane initial value is 80 MB.
> +
> +config IDLE_PAGE_TRACKING
> + bool "Enable idle page tracking"
> + select PROC_PAGE_MONITOR
> + select PAGE_EXTENSION if !64BIT
> + help
> + This feature allows to estimate the amount of user pages that have
> + not been touched during a given period of time. This information can
> + be useful to tune memory cgroup limits and/or for job placement
> + within a compute cluster.
> +
> + See Documentation/vm/pagemap.txt for more details.
> diff --git a/mm/debug.c b/mm/debug.c
> index 3eb3ac2fcee7..bb66f9ccec03 100644
> --- a/mm/debug.c
> +++ b/mm/debug.c
> @@ -48,6 +48,10 @@ static const struct trace_print_flags pageflag_names[] = {
> #ifdef CONFIG_TRANSPARENT_HUGEPAGE
> {1UL << PG_compound_lock, "compound_lock" },
> #endif
> +#if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT)
> + {1UL << PG_young, "young" },
> + {1UL << PG_idle, "idle" },
> +#endif
> };
>
> static void dump_flags(unsigned long flags,
> diff --git a/mm/page_ext.c b/mm/page_ext.c
> index d86fd2f5353f..e4b3af054bf2 100644
> --- a/mm/page_ext.c
> +++ b/mm/page_ext.c
> @@ -59,6 +59,9 @@ static struct page_ext_operations *page_ext_ops[] = {
> #ifdef CONFIG_PAGE_OWNER
> &page_owner_ops,
> #endif
> +#if defined(CONFIG_IDLE_PAGE_TRACKING) && !defined(CONFIG_64BIT)
> + &page_idle_ops,
> +#endif
> };
>
> static unsigned long total_usage;
> diff --git a/mm/rmap.c b/mm/rmap.c
> index 24dd3f9fee27..12e73b758d9e 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
> @@ -784,6 +784,13 @@ static int page_referenced_one(struct page *page, struct vm_area_struct *vma,
> if (referenced) {
> pra->referenced++;
> pra->vm_flags |= vma->vm_flags;
> + if (page_is_idle(page))
> + clear_page_idle(page);
> + }
> +
> + if (page_is_young(page)) {
> + clear_page_young(page);
> + pra->referenced++;
If a page was page_is_young and referenced recenlty,
pra->referenced is increased doubly and it changes current
behavior for file-backed page promotion. Look at page_check_references.
> }
>
> pra->mapcount--;
> diff --git a/mm/swap.c b/mm/swap.c
> index a7251a8ed532..6bf6f293a9ea 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -623,6 +623,8 @@ void mark_page_accessed(struct page *page)
> } else if (!PageReferenced(page)) {
> SetPageReferenced(page);
> }
> + if (page_is_idle(page))
> + clear_page_idle(page);
> }
> EXPORT_SYMBOL(mark_page_accessed);
>
> --
> 1.7.10.4
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Kind regards,
Minchan Kim
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v3 3/3] proc: add kpageidle file
From: Minchan Kim @ 2015-04-29 4:57 UTC (permalink / raw)
To: Vladimir Davydov
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Greg Thelen,
Michel Lespinasse, David Rientjes, Pavel Emelyanov,
Cyrill Gorcunov, Jonathan Corbet,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-doc-u79uwXL29TY76Z2rM5mHXA, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
cgroups-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <4c24a6bf2c9711dd4dbb72a43a16eba6867527b7.1430217477.git.vdavydov-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org>
On Tue, Apr 28, 2015 at 03:24:42PM +0300, Vladimir Davydov wrote:
> Knowing the portion of memory that is not used by a certain application
> or memory cgroup (idle memory) can be useful for partitioning the system
> efficiently, e.g. by setting memory cgroup limits appropriately.
> Currently, the only means to estimate the amount of idle memory provided
> by the kernel is /proc/PID/{clear_refs,smaps}: the user can clear the
> access bit for all pages mapped to a particular process by writing 1 to
> clear_refs, wait for some time, and then count smaps:Referenced.
> However, this method has two serious shortcomings:
>
> - it does not count unmapped file pages
> - it affects the reclaimer logic
>
> To overcome these drawbacks, this patch introduces two new page flags,
> Idle and Young, and a new proc file, /proc/kpageidle. A page's Idle flag
> can only be set from userspace by writing 1 to /proc/kpageidle at the
> offset corresponding to the page, and it is cleared whenever the page is
> accessed either through page tables (it is cleared in page_referenced()
> in this case) or using the read(2) system call (mark_page_accessed()).
> Thus by setting the Idle flag for pages of a particular workload, which
> can be found e.g. by reading /proc/PID/pagemap, waiting for some time to
> let the workload access its working set, and then reading the kpageidle
> file, one can estimate the amount of pages that are not used by the
> workload.
>
> The Young page flag is used to avoid interference with the memory
> reclaimer. A page's Young flag is set whenever the Access bit of a page
> table entry pointing to the page is cleared by writing to kpageidle. If
> page_referenced() is called on a Young page, it will add 1 to its return
> value, therefore concealing the fact that the Access bit was cleared.
>
> Note, since there is no room for extra page flags on 32 bit, this
> feature uses extended page flags when compiled on 32 bit.
>
> Signed-off-by: Vladimir Davydov <vdavydov-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org>
> ---
> Documentation/vm/pagemap.txt | 10 ++-
> fs/proc/page.c | 154 ++++++++++++++++++++++++++++++++++++++++++
> fs/proc/task_mmu.c | 4 +-
> include/linux/mm.h | 88 ++++++++++++++++++++++++
> include/linux/page-flags.h | 9 +++
> include/linux/page_ext.h | 4 ++
> mm/Kconfig | 12 ++++
> mm/debug.c | 4 ++
> mm/page_ext.c | 3 +
> mm/rmap.c | 7 ++
> mm/swap.c | 2 +
> 11 files changed, 295 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/vm/pagemap.txt b/Documentation/vm/pagemap.txt
> index a9b7afc8fbc6..ac6fd32a9296 100644
> --- a/Documentation/vm/pagemap.txt
> +++ b/Documentation/vm/pagemap.txt
> @@ -5,7 +5,7 @@ pagemap is a new (as of 2.6.25) set of interfaces in the kernel that allow
> userspace programs to examine the page tables and related information by
> reading files in /proc.
>
> -There are four components to pagemap:
> +There are five components to pagemap:
>
> * /proc/pid/pagemap. This file lets a userspace process find out which
> physical frame each virtual page is mapped to. It contains one 64-bit
> @@ -69,6 +69,14 @@ There are four components to pagemap:
> memory cgroup each page is charged to, indexed by PFN. Only available when
> CONFIG_MEMCG is set.
>
> + * /proc/kpageidle. For each page this file contains a 64-bit number, which
> + equals 1 if the page is idle or 0 otherwise, indexed by PFN. A page is
> + considered idle if it has not been accessed since it was marked idle. To
> + mark a page idle one should write 1 to this file at the offset corresponding
> + to the page. Only user memory pages can be marked idle, for other page types
> + input is silently ignored. Writing to this file beyond max PFN results in
> + the ENXIO error. Only available when CONFIG_IDLE_PAGE_TRACKING is set.
> +
How about using kpageflags for reading part?
I mean PG_idle is one of the page flags and we already have a feature to
parse of each PFN flag so we could reuse existing feature for reading
idleness.
--
Kind regards,
Minchan Kim
^ permalink raw reply
* Re: [PATCH v3 0/3] idle memory tracking
From: Minchan Kim @ 2015-04-29 5:02 UTC (permalink / raw)
To: Vladimir Davydov
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Greg Thelen,
Michel Lespinasse, David Rientjes, Pavel Emelyanov,
Cyrill Gorcunov, Jonathan Corbet,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-doc-u79uwXL29TY76Z2rM5mHXA, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
cgroups-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <cover.1430217477.git.vdavydov-bzQdu9zFT3WakBO8gow8eQ@public.gmane.org>
On Tue, Apr 28, 2015 at 03:24:39PM +0300, Vladimir Davydov wrote:
> Hi,
>
> This patch set introduces a new user API for tracking user memory pages
> that have not been used for a given period of time. The purpose of this
> is to provide the userspace with the means of tracking a workload's
> working set, i.e. the set of pages that are actively used by the
> workload. Knowing the working set size can be useful for partitioning
> the system more efficiently, e.g. by tuning memory cgroup limits
> appropriately, or for job placement within a compute cluster.
>
> ---- USE CASES ----
>
> The unified cgroup hierarchy has memory.low and memory.high knobs, which
> are defined as the low and high boundaries for the workload working set
> size. However, the working set size of a workload may be unknown or
> change in time. With this patch set, one can periodically estimate the
> amount of memory unused by each cgroup and tune their memory.low and
> memory.high parameters accordingly, therefore optimizing the overall
> memory utilization.
>
> Another use case is balancing workloads within a compute cluster.
> Knowing how much memory is not really used by a workload unit may help
> take a more optimal decision when considering migrating the unit to
> another node within the cluster.
Another usecase I have a interest is working with per-process reclaim.
https://lwn.net/Articles/545668/
With idle tracking, we could reclaim idle pages only by smart user
memory-manager.
Thanks.
--
Kind regards,
Minchan Kim
^ permalink raw reply
* Re: [v14 3/4] ext4: adds FS_IOC_FSSETXATTR/FS_IOC_FSGETXATTR interface support
From: Li Xi @ 2015-04-29 5:49 UTC (permalink / raw)
To: Dave Chinner
Cc: linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Ext4 Developers List,
linux-api-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Theodore Ts'o, Andreas Dilger, Jan Kara,
viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org,
hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org, Dmitry Monakhov
In-Reply-To: <20150428044331.GV21261@dastard>
Hi Dave,
Thanks for the advices. I tried to run latest xfstests again. However,
the kernel crashed when runing generic/051 generic/054 and
generic/055. And please note that the kernel also crashed on original
linux-4.0 without any of my patches. Following is one of the dump
stack:
run fstests generic/055 at 2015-04-29 13:43:39
------------[ cut here ]------------
WARNING: CPU: 0 PID: 31915 at lib/list_debug.c:33 __list_add+0xbe/0xd0()
list_add corruption. prev->next should be next (ffffffff81e05018), but
was (null). (prev=ffff8800d8ff3ca0).
Modules linked in: dm_flakey xfs exportfs libcrc32c nfsv3 nfs_acl
rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs fscache lockd grace
sunrpc ipv6 dm_mirror dm_region_hash dm_log dm_mod ppdev floppy
parport_pc parport microcode pcspkr virtio_balloon sg 8139too 8139cp
mii i2c_piix4 i2c_core ext4 jbd2 mbcache sr_mod cdrom sd_mod pata_acpi
ata_generic ata_piix virtio_pci virtio_ring virtio [last unloaded:
speedstep_lib]
CPU: 0 PID: 31915 Comm: kworker/0:0 Not tainted 4.0.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
Workqueue: events vmstat_shepherd
0000000000000021 ffff8800db177bf8 ffffffff815ccaf6 0000000000000021
ffff8800db177c48 ffff8800db177c38 ffffffff81059fc5 ffff8800db177c38
ffffffff81a8d480 ffffffff81e05018 ffff8800d8ff3ca0 0000000000000000
Call Trace:
[<ffffffff815ccaf6>] dump_stack+0x48/0x5a
[<ffffffff81059fc5>] warn_slowpath_common+0x95/0xe0
[<ffffffff8105a0c6>] warn_slowpath_fmt+0x46/0x70
[<ffffffff812c04fe>] __list_add+0xbe/0xd0
[<ffffffff810bb84b>] __internal_add_timer+0x9b/0x110
[<ffffffff810bb8f9>] internal_add_timer+0x39/0x90
[<ffffffff810bd8c9>] mod_timer+0xf9/0x1d0
[<ffffffff810bd9b8>] add_timer+0x18/0x30
[<ffffffff81071a22>] __queue_delayed_work+0x92/0x1a0
[<ffffffff81071bcd>] queue_delayed_work_on+0x1d/0x40
[<ffffffff81160d5c>] vmstat_shepherd+0x10c/0x120
[<ffffffff810722ed>] process_one_work+0x14d/0x440
[<ffffffff810726ff>] worker_thread+0x11f/0x3d0
[<ffffffff815ccfaf>] ? __schedule+0x36f/0x800
[<ffffffff810725e0>] ? process_one_work+0x440/0x440
[<ffffffff810725e0>] ? process_one_work+0x440/0x440
[<ffffffff810774ce>] kthread+0xce/0xf0
[<ffffffff8104d96e>] ? __do_page_fault+0x17e/0x430
[<ffffffff81077400>] ? kthread_freezable_should_stop+0x70/0x70
[<ffffffff815d1052>] ret_from_fork+0x42/0x70
[<ffffffff81077400>] ? kthread_freezable_should_stop+0x70/0x70
---[ end trace 97c6b752be15ac57 ]---
XFS (sdb2): Mounting V4 Filesystem
XFS (sdb2): Ending clean mount
XFS (sdb2): Quotacheck needed: Please wait.
XFS (sdb2): Quotacheck: Done.
XFS (sdb2): xfs_log_force: error -5 returned.
XFS (sdb2): xfs_log_force: error -5 returned.
XFS (sdb2): xfs_log_force: error -5 returned.
BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
IP: [<ffffffff810bbc88>] get_next_timer_interrupt+0x158/0x230
PGD d8e7a067 PUD db654067 PMD 0
Oops: 0000 [#1] SMP
Modules linked in: dm_flakey xfs exportfs libcrc32c nfsv3 nfs_acl
rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs fscache lockd grace
sunrpc ipv6 dm_mirror dm_region_hash dm_log dm_mod ppdev floppy
parport_pc parport microcode pcspkr virtio_balloon sg 8139too 8139cp
mii i2c_piix4 i2c_core ext4 jbd2 mbcache sr_mod cdrom sd_mod pata_acpi
ata_generic ata_piix virtio_pci virtio_ring virtio [last unloaded:
speedstep_lib]
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 4.0.0+ #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
task: ffffffff81a134a0 ti: ffffffff81a00000 task.ti: ffffffff81a00000
RIP: 0010:[<ffffffff810bbc88>] [<ffffffff810bbc88>]
get_next_timer_interrupt+0x158/0x230
RSP: 0018:ffff88011fc03e48 EFLAGS: 00010013
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff81e05008
RDX: 0000000000000001 RSI: 0000000000000011 RDI: ffffffff81e04ef8
RBP: ffff88011fc03ea8 R08: 0000000000000011 R09: 0000000001000551
R10: ffff88011fc03e60 R11: ffff88011fc03e78 R12: 0000000140055030
R13: 0000000100055031 R14: ffffffff81e03ec0 R15: 0000000000000040
FS: 0000000000000000(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000018 CR3: 00000000d8d8f000 CR4: 00000000000006f0
Stack:
ffff88011fc03e88 ffffffff810bbdf7 ffffffff81e04ef8 ffffffff81e052f8
ffffffff81e056f8 ffffffff81e05af8 0000000000000000 ffff88011fc0f8a0
0000000100055031 0000000000000000 ffff88011fc0bfc0 ffffffff81a00000
Call Trace:
<IRQ>
[<ffffffff810bbdf7>] ? call_timer_fn+0x47/0x110
[<ffffffff810cdf1d>] tick_nohz_stop_sched_tick+0x1cd/0x310
[<ffffffff810ce108>] __tick_nohz_idle_enter+0xa8/0x150
[<ffffffff810ce1dd>] tick_nohz_irq_exit+0x2d/0x40
[<ffffffff8105deaf>] irq_exit+0x9f/0xc0
[<ffffffff815d34aa>] smp_apic_timer_interrupt+0x4a/0x59
[<ffffffff815d1a3b>] apic_timer_interrupt+0x6b/0x70
<EOI>
[<ffffffff8100f100>] ? default_idle+0x20/0xb0
[<ffffffff8100e74f>] arch_cpu_idle+0xf/0x20
[<ffffffff810986a9>] cpuidle_idle_call+0x89/0x220
[<ffffffff81078232>] ? __atomic_notifier_call_chain+0x12/0x20
[<ffffffff81098975>] cpu_idle_loop+0x135/0x1f0
[<ffffffff81098a43>] cpu_startup_entry+0x13/0x20
[<ffffffff815c5e1c>] rest_init+0x7c/0x80
[<ffffffff81b4e372>] start_kernel+0x3d8/0x3df
[<ffffffff81b4ddb8>] ? set_init_arg+0x5d/0x5d
[<ffffffff815cb906>] ? memblock_reserve+0x4c/0x51
[<ffffffff81b4d5ad>] x86_64_start_reservations+0x2a/0x2c
[<ffffffff81b4d6e4>] x86_64_start_kernel+0x135/0x13c
Code: 00 48 89 45 c8 45 89 c8 41 83 e0 3f 44 89 c6 0f 1f 40 00 48 63
ce 48 c1 e1 04 48 8b 04 39 48 8d 0c 0f 48 39 c8 74 22 0f 1f 40 00 <f6>
40 18 01 75 10 48 8b 50 10 48 39 da 48 0f 48 da ba 01 00 00
RIP [<ffffffff810bbc88>] get_next_timer_interrupt+0x158/0x230
RSP <ffff88011fc03e48>
CR2: 0000000000000018
On Tue, Apr 28, 2015 at 12:43 PM, Dave Chinner <david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org> wrote:
> On Tue, Apr 28, 2015 at 10:01:07AM +0800, Li Xi wrote:
>> Hi Dave,
>>
>> I ran xfstests on the kernel with this series of patches.
>> Unfortunately, 5 test suits failed. But I don't think they are caused
>> by this patch. Following is the result. Please let me know if there is
>> any problem about it.
>>
>> Output of xfstests:
>>
>> FSTYP -- xfs (non-debug)
>> PLATFORM -- Linux/x86_64 vm15 4.0.0+
>> MKFS_OPTIONS -- -f -bsize=4096 /dev/sdb2
>> MOUNT_OPTIONS -- /dev/sdb2 /mnt/scratch
>>
>> generic/001 3s ... 2s
>> generic/002 0s ... 0s
>> generic/003 10s ... 10s
>> generic/004 [not run] xfs_io flink support is missing
>> generic/005 0s ... 0s
>> generic/006 1s ... 0s
>> generic/007 0s ... 0s
>> generic/008 [not run] xfs_io fzero support is missing
>> generic/009 [not run] xfs_io fzero support is missing
>> generic/010 1s ... 0s
>> generic/011 1s ... 0s
>> generic/012 [not run] xfs_io fpunch support is missing
>> generic/013 92s ... 90s
>> generic/014 3s ... 3s
>> generic/015 1s ... 1s
>> generic/016 [not run] xfs_io fpunch support is missing
>> generic/017 [not run] xfs_io fiemap support is missing
>> generic/018 [not run] xfs_io fiemap support is missing
>
> You really need to update your xfsprogs install. You aren't testing
> half of what you need to be testing if you are missing basic
> functionality like fiemap support (which has been in xfs_io since
> 2011).
>
>> generic/020 38s ... 31s
>> generic/021 [not run] xfs_io fpunch support is missing
>> generic/022 [not run] xfs_io fpunch support is missing
>> generic/023 1s ... 0s
>> generic/024 1s ... 0s
>> generic/025 0s ... 0s
>> generic/026 0s ... 0s
>> generic/027 57s ... 57s
>> generic/028 5s ... 5s
>> generic/053 1s ... 2s
>> generic/062 1s ... 2s
>> generic/068 60s ... 61s
>> generic/069 4s ... 3s
>> generic/070 13s ... 14s
>> generic/074 164s ... 162s
>> generic/075 87s ... 86s
>> generic/076 1s ... 1s
>> generic/077 [not run] fsgqa user not defined.
>
> ANd if you don't have this user defined, then several quota tests
> don't get run.
>
>> generic/079 1s ... 1s
>> generic/083 36s ... 39s
>> generic/088 1s ... 0s
>> generic/089 4s ... 4s
>> generic/091 62s ... 62s
>> generic/093 [not run] not suitable for this OS: Linux
>> generic/097 [not run] not suitable for this OS: Linux
>> generic/099 [not run] not suitable for this OS: Linux
>> generic/100 12s ... 12s
>> generic/105 0s ... 0s
>> generic/112 [not run] fsx not built with AIO for this platform
>> generic/113 [not run] aio-stress not built for this platform
>
> Ouch. There's another whole class of functionality you aren't
> testing.
>
>> generic/299 [not run] utility required, skipped this test
>> generic/300 [not run] xfs_io fpunch support is missing
>> generic/306 - output mismatch (see
>> /root/work/quota/ext4_inode_field/xfstests.git/results//generic/306.out.bad)
>> --- tests/generic/306.out 2014-07-16 10:19:26.196995657 +0800
>> +++ /root/work/quota/ext4_inode_field/xfstests.git/results//generic/306.out.bad
>> 2015-04-27 22:40:13.365445316 +0800
>> @@ -2,11 +2,9 @@
>> == try to create new file
>> touch: cannot touch 'SCRATCH_MNT/this_should_fail': Read-only file system
>> == pwrite to null device
>> -wrote 512/512 bytes at offset 0
>> -XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +xfs_io: specified file ["/mnt/scratch/devnull"] is not on an XFS filesystem
>> == pread from zero device
>> ...
>> (Run 'diff -u tests/generic/306.out
>> /root/work/quota/ext4_inode_field/xfstests.git/results//generic/306.out.bad'
>> to see the entire diff)
>
> That's caused by having a very old xfs_io.
>
>> xfs/229 134s ... [failed, exit status 23] - output mismatch (see
>> /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/229.out.bad)
>> --- tests/xfs/229.out 2014-07-16 10:19:26.215995657 +0800
>> +++ /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/229.out.bad
>> 2015-04-27 23:25:48.709093428 +0800
>> @@ -1,4 +1,31 @@
>> QA output created by 229
>> generating 10 files
>> +Write did not return correct amount
>> +Write did not return correct amount
>> +Write did not return correct amount
>> +Write did not return correct amount
>> comparing files
>
> Can't say that I've seen that one fail for a long time. I can't say
> anything useful about it, however, given how old your xfsprogs
> installation is.
>
>> ...
>> (Run 'diff -u tests/xfs/229.out
>> /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/229.out.bad'
>> to see the entire diff)
>> xfs/238 1s ... 1s
>> xfs/242 [not run] zero command not supported
>> xfs/244 2s ... 2s
>> xfs/250 [failed, exit status 1] - output mismatch (see
>> /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/250.out.bad)
>> --- tests/xfs/250.out 2014-07-16 10:19:26.215995657 +0800
>> +++ /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/250.out.bad
>> 2015-04-27 23:26:15.137452337 +0800
>> @@ -11,4 +11,4 @@
>> *** preallocate large file
>> *** unmount loop filesystem
>> *** check loop filesystem
>> -*** done
>> +_check_xfs_filesystem: filesystem on /mnt/test/250.fs is
>> inconsistent (r) (see
>> /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/250.full)
>> ...
>> (Run 'diff -u tests/xfs/250.out
>> /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/250.out.bad'
>> to see the entire diff)
>
> Your xfstests is not up to date. This is fixed by commit ee6ad7f
> ("xfs/049: umount -d fails when kernel wins teardown race").
>
>> xfs/301 - output mismatch (see
>> /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/301.out.bad)
>> --- tests/xfs/301.out 2014-07-16 10:19:26.217995657 +0800
>> +++ /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/301.out.bad
>> 2015-04-27 23:33:33.629182381 +0800
>> @@ -29,18 +29,21 @@
>> Attribute "attr4" had a 10 byte value for DUMP_DIR/sub/biggg:
>> some_text4
>> EAs on restore
>> +getfattr: /mnt/scratch/restoredir/dumpdir: No such file or directory
>> +getfattr: /mnt/scratch/restoredir/dumpdir: No such file or directory
>> User names
>> -Attribute "attr5" had a 8 byte value for DUMP_DIR/dir:
>> ...
>> (Run 'diff -u tests/xfs/301.out
>
> $ ./lsqa.pl tests/xfs/301
> FS QA Test No. 301
>
> Verify multi-stream xfsdump/restore preserves extended attributes
>
> $
>
> Your xfsdump package is out of date and needs upgrading.
>
>> /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/301.out.bad'
>> to see the entire diff)
>> xfs/302 [failed, exit status 1] - output mismatch (see
>> /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/302.out.bad)
>> --- tests/xfs/302.out 2014-07-16 10:19:26.217995657 +0800
>> +++ /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/302.out.bad
>> 2015-04-27 23:33:46.102767709 +0800
>> @@ -1,2 +1,4 @@
>> QA output created by 302
>> Silence is golden.
>> +dump failed
>> +(see /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/302.full
>> for details)
>> ...
>> (Run 'diff -u tests/xfs/302.out
>> /root/work/quota/ext4_inode_field/xfstests.git/results//xfs/302.out.bad'
>> to see the entire diff)
>
> Same again.
>
> You need to upgrade everything to current xfstests/xfsprogs/xfsdump
> and retest *everything*. That means rerunning all your ext4 testing,
> too, because you're not exercising all the cases where the
> interesting accounting bugs lie (i.e. in fallocate operations).
>
> I'd also suggest that you run the tests using MOUNT_OPTIONS="-o
> pquota" after setting up default configurations for TEST_MNT and
> SCRATCH_MNT so that you actually give the project quota code a
> significant amount of work to do, and do the same for ext4,
> otherwise you're not really testing it at all when you run xfstests
> on ext4....
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org
^ permalink raw reply
* Re: [RFC v2 1/4] fs: Add generic file system event notifications
From: Beata Michalska @ 2015-04-29 7:03 UTC (permalink / raw)
To: Greg KH
Cc: Jan Kara, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA,
linux-api-u79uwXL29TY76Z2rM5mHXA, tytso-3s7WtUTddSA,
adilger.kernel-m1MBpc4rdrD3fQ9qLvQP4Q,
hughd-hpIqsD4AKlfQT0dZR+AlfA, lczerner-H+wXaHxf7aLQT0dZR+AlfA,
hch-wEGCiKHe2LqWVfeAwA7xHQ, linux-ext4-u79uwXL29TY76Z2rM5mHXA,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
kyungmin.park-Sze3O3UU22JBDgjK7y7TUQ,
kmpark-wEGCiKHe2LqWVfeAwA7xHQ
In-Reply-To: <20150428173900.GA16708-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org>
On 04/28/2015 07:39 PM, Greg KH wrote:
> On Tue, Apr 28, 2015 at 04:46:46PM +0200, Beata Michalska wrote:
>> On 04/28/2015 04:09 PM, Greg KH wrote:
>>> On Tue, Apr 28, 2015 at 03:56:53PM +0200, Jan Kara wrote:
>>>> On Mon 27-04-15 17:37:11, Greg KH wrote:
>>>>> On Mon, Apr 27, 2015 at 05:08:27PM +0200, Beata Michalska wrote:
>>>>>> On 04/27/2015 04:24 PM, Greg KH wrote:
>>>>>>> On Mon, Apr 27, 2015 at 01:51:41PM +0200, Beata Michalska wrote:
>>>>>>>> Introduce configurable generic interface for file
>>>>>>>> system-wide event notifications, to provide file
>>>>>>>> systems with a common way of reporting any potential
>>>>>>>> issues as they emerge.
>>>>>>>>
>>>>>>>> The notifications are to be issued through generic
>>>>>>>> netlink interface by newly introduced multicast group.
>>>>>>>>
>>>>>>>> Threshold notifications have been included, allowing
>>>>>>>> triggering an event whenever the amount of free space drops
>>>>>>>> below a certain level - or levels to be more precise as two
>>>>>>>> of them are being supported: the lower and the upper range.
>>>>>>>> The notifications work both ways: once the threshold level
>>>>>>>> has been reached, an event shall be generated whenever
>>>>>>>> the number of available blocks goes up again re-activating
>>>>>>>> the threshold.
>>>>>>>>
>>>>>>>> The interface has been exposed through a vfs. Once mounted,
>>>>>>>> it serves as an entry point for the set-up where one can
>>>>>>>> register for particular file system events.
>>>>>>>>
>>>>>>>> Signed-off-by: Beata Michalska <b.michalska-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>
>>>>>>>> ---
>>>>>>>> Documentation/filesystems/events.txt | 231 ++++++++++
>>>>>>>> fs/Makefile | 1 +
>>>>>>>> fs/events/Makefile | 6 +
>>>>>>>> fs/events/fs_event.c | 770 ++++++++++++++++++++++++++++++++++
>>>>>>>> fs/events/fs_event.h | 25 ++
>>>>>>>> fs/events/fs_event_netlink.c | 99 +++++
>>>>>>>> fs/namespace.c | 1 +
>>>>>>>> include/linux/fs.h | 6 +-
>>>>>>>> include/linux/fs_event.h | 58 +++
>>>>>>>> include/uapi/linux/fs_event.h | 54 +++
>>>>>>>> include/uapi/linux/genetlink.h | 1 +
>>>>>>>> net/netlink/genetlink.c | 7 +-
>>>>>>>> 12 files changed, 1257 insertions(+), 2 deletions(-)
>>>>>>>> create mode 100644 Documentation/filesystems/events.txt
>>>>>>>> create mode 100644 fs/events/Makefile
>>>>>>>> create mode 100644 fs/events/fs_event.c
>>>>>>>> create mode 100644 fs/events/fs_event.h
>>>>>>>> create mode 100644 fs/events/fs_event_netlink.c
>>>>>>>> create mode 100644 include/linux/fs_event.h
>>>>>>>> create mode 100644 include/uapi/linux/fs_event.h
>>>>>>>
>>>>>>> Any reason why you just don't do uevents for the block devices today,
>>>>>>> and not create a new type of netlink message and userspace tool required
>>>>>>> to read these?
>>>>>>
>>>>>> The idea here is to have support for filesystems with no backing device as well.
>>>>>> Parsing the message with libnl is really simple and requires few lines of code
>>>>>> (sample application has been presented in the initial version of this RFC)
>>>>>
>>>>> I'm not saying it's not "simple" to parse, just that now you are doing
>>>>> something that requires a different tool. If you have a block device,
>>>>> you should be able to emit uevents for it, you don't need a backing
>>>>> device, we handle virtual filesystems in /sys/block/ just fine :)
>>>>>
>>>>> People already have tools that listen to libudev for system monitoring
>>>>> and management, why require them to hook up to yet-another-library? And
>>>>> what is going to provide the ability for multiple userspace tools to
>>>>> listen to these netlink messages in case you have more than one program
>>>>> that wants to watch for these things (i.e. multiple desktop filesystem
>>>>> monitoring tools, system-health checkers, etc.)?
>>>> As much as I understand your concerns I'm not convinced uevent interface
>>>> is a good fit. There are filesystems that don't have underlying block
>>>> device - think of e.g. tmpfs or filesystems working directly on top of
>>>> flash devices. These still want to send notification to userspace (one of
>>>> primary motivation for this interfaces was so that tmpfs can notify about
>>>> something). And creating some fake nodes in /sys/block for tmpfs and
>>>> similar filesystems seems like doing more harm than good to me...
>>>
>>> If these are "fake" block devices, what's going to be present in the
>>> block major/minor fields of the netlink message? For some reason I
>>> thought it was a required field, and because of that, I thought we had a
>>> "real" filesystem somewhere to refer to, otherwise how would userspace
>>> know what filesystem was creating these events?
>>>
>>> What am I missing here?
>>>
>>> confused,
>>>
>>> greg k-h
>>>
>>
>> For those 'fake' block devs, upon mount, get_anon_bdev will assign
>> the major:minor numbers. Userspace might get those through stat.
>
> How can userspace do the mapping backwards from this "anonymous"
> major:minor number for these types of filesystems in such a way that
> they can "know" how to report the block device that is causing the
> event?
>
> thanks,
>
> greg k-h
>
It needs to be done internally by the app but is doable.
The app knows what it is watching, so it can maintain the mappings.
So prior to activating the notifications it can call 'stat' on the mount point.
Stat struct gives the 'st_dev' which is the device id. Same will be reported
within the message payload (through major:minor numbers). So having this,
the app is able to get any other information it needs.
Note that the events refer to the file system as a whole and they may not
necessarily have anything to do with the actual block device.
BR
Beata
^ permalink raw reply
* Re: [PATCH] usb: core: add usb3 lpm sysfs
From: Zhuang Jin Can @ 2015-04-29 7:20 UTC (permalink / raw)
To: Greg KH
Cc: rafael.j.wysocki-ral2JQCrhuEAvxtiuMwx3w,
stern-nwvwT67g6+6dFdvTe/nMLpVzexx5G7lz,
dan.j.williams-ral2JQCrhuEAvxtiuMwx3w, pmladek-AlSwsSmVLrQ,
peter.chen-KZfg59tc24xl57MIdRCFDg, jwerner-F7+t8E8rja9g9hUCZPvPmw,
linux-api-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-usb-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20150428211110.GA32383-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org>
On Tue, Apr 28, 2015 at 11:11:10PM +0200, Greg KH wrote:
> On Wed, Apr 29, 2015 at 12:51:27AM +0800, Zhuang Jin Can wrote:
> > Hi Greg KH,
> >
> > On Tue, Apr 28, 2015 at 12:42:24PM +0200, Greg KH wrote:
> > > On Sun, Apr 19, 2015 at 11:46:12AM +0800, Zhuang Jin Can wrote:
> > > > Some usb3 devices may not support usb3 lpm well.
> > > > The patch adds a sysfs to enable/disable u1 or u2 of the port.The
> > > > settings apply to both before and after device enumeration.
> > > > Supported values are "0" - u1 and u2 are disabled, "u1" - only u1 is
> > > > enabled, "u2" - only u2 is enabled, "u1_u2" - u1 and u2 are enabled.
> > > >
> > > > The interface is useful for testing some USB3 devices during
> > > > development, and provides a way to disable usb3 lpm if the issues can
> > > > not be fixed in final products.
> > >
> > > How is a user supposed to "know" to make this setting for a device? Why
> > > can't the kernel automatically set this value properly? Why does it
> > > need to be a kernel issue at all?
> > >
> > By default kernel enables u1 u2 of all USB3 devices. This interface
> > provides the user to change this policy. User may set the policy
> > according to PID/VID of uevent or according to the platform information
> > known by userspace.
>
> And why would they ever want to do that?
>
> > It's not a kernel issue, as u1 u2 is mandatory by USB3 compliance. But
> > for some internal hardwired USB3 connection, e.g. SSIC, passing USB3
> > compliance is not mandatory. So the interface provides a way for vendor
> > to ship with u1 or u2 broken products. Of course, this is not encouraged :).
>
> If the state is broken for those devices, we can't require the user to
> fix it for us, the kernel should do it automatically.
>
> > > And when you are doing development of broken devices, the kernel doesn't
> > > have to support you, you can run with debugging patches of your own
> > > until you fix your firmware :)
> > >
> > Understood. But I think other vendor or developer may face the same
> > issue in final product shipment or during development. Moreover, the
> > interface provide the flexibility for developer to separately
> > disable/enable u1 or u2, e.g. If they're debugging an u2 issue, they
> > can disable u1 to simplify the situtation.
>
> For debugging only, perhaps, but for a "normal" user, please let's
> handle this automatically and don't create a switch that never gets used
> by anyone or anything.
>
Thanks Greg. Since so far the patch has no interesting value to the
community, I'll drop the patch.
Regards
Jincan
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [RFC v2 1/4] fs: Add generic file system event notifications
From: Jan Kara @ 2015-04-29 7:42 UTC (permalink / raw)
To: Beata Michalska
Cc: Greg KH, Jan Kara, linux-kernel, linux-fsdevel, linux-api, tytso,
adilger.kernel, hughd, lczerner, hch, linux-ext4, linux-mm,
kyungmin.park, kmpark
In-Reply-To: <5540822C.10000@samsung.com>
On Wed 29-04-15 09:03:08, Beata Michalska wrote:
> On 04/28/2015 07:39 PM, Greg KH wrote:
> > On Tue, Apr 28, 2015 at 04:46:46PM +0200, Beata Michalska wrote:
> >> On 04/28/2015 04:09 PM, Greg KH wrote:
> >>> On Tue, Apr 28, 2015 at 03:56:53PM +0200, Jan Kara wrote:
> >>>> On Mon 27-04-15 17:37:11, Greg KH wrote:
> >>>>> On Mon, Apr 27, 2015 at 05:08:27PM +0200, Beata Michalska wrote:
> >>>>>> On 04/27/2015 04:24 PM, Greg KH wrote:
> >>>>>>> On Mon, Apr 27, 2015 at 01:51:41PM +0200, Beata Michalska wrote:
> >>>>>>>> Introduce configurable generic interface for file
> >>>>>>>> system-wide event notifications, to provide file
> >>>>>>>> systems with a common way of reporting any potential
> >>>>>>>> issues as they emerge.
> >>>>>>>>
> >>>>>>>> The notifications are to be issued through generic
> >>>>>>>> netlink interface by newly introduced multicast group.
> >>>>>>>>
> >>>>>>>> Threshold notifications have been included, allowing
> >>>>>>>> triggering an event whenever the amount of free space drops
> >>>>>>>> below a certain level - or levels to be more precise as two
> >>>>>>>> of them are being supported: the lower and the upper range.
> >>>>>>>> The notifications work both ways: once the threshold level
> >>>>>>>> has been reached, an event shall be generated whenever
> >>>>>>>> the number of available blocks goes up again re-activating
> >>>>>>>> the threshold.
> >>>>>>>>
> >>>>>>>> The interface has been exposed through a vfs. Once mounted,
> >>>>>>>> it serves as an entry point for the set-up where one can
> >>>>>>>> register for particular file system events.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Beata Michalska <b.michalska@samsung.com>
> >>>>>>>> ---
> >>>>>>>> Documentation/filesystems/events.txt | 231 ++++++++++
> >>>>>>>> fs/Makefile | 1 +
> >>>>>>>> fs/events/Makefile | 6 +
> >>>>>>>> fs/events/fs_event.c | 770 ++++++++++++++++++++++++++++++++++
> >>>>>>>> fs/events/fs_event.h | 25 ++
> >>>>>>>> fs/events/fs_event_netlink.c | 99 +++++
> >>>>>>>> fs/namespace.c | 1 +
> >>>>>>>> include/linux/fs.h | 6 +-
> >>>>>>>> include/linux/fs_event.h | 58 +++
> >>>>>>>> include/uapi/linux/fs_event.h | 54 +++
> >>>>>>>> include/uapi/linux/genetlink.h | 1 +
> >>>>>>>> net/netlink/genetlink.c | 7 +-
> >>>>>>>> 12 files changed, 1257 insertions(+), 2 deletions(-)
> >>>>>>>> create mode 100644 Documentation/filesystems/events.txt
> >>>>>>>> create mode 100644 fs/events/Makefile
> >>>>>>>> create mode 100644 fs/events/fs_event.c
> >>>>>>>> create mode 100644 fs/events/fs_event.h
> >>>>>>>> create mode 100644 fs/events/fs_event_netlink.c
> >>>>>>>> create mode 100644 include/linux/fs_event.h
> >>>>>>>> create mode 100644 include/uapi/linux/fs_event.h
> >>>>>>>
> >>>>>>> Any reason why you just don't do uevents for the block devices today,
> >>>>>>> and not create a new type of netlink message and userspace tool required
> >>>>>>> to read these?
> >>>>>>
> >>>>>> The idea here is to have support for filesystems with no backing device as well.
> >>>>>> Parsing the message with libnl is really simple and requires few lines of code
> >>>>>> (sample application has been presented in the initial version of this RFC)
> >>>>>
> >>>>> I'm not saying it's not "simple" to parse, just that now you are doing
> >>>>> something that requires a different tool. If you have a block device,
> >>>>> you should be able to emit uevents for it, you don't need a backing
> >>>>> device, we handle virtual filesystems in /sys/block/ just fine :)
> >>>>>
> >>>>> People already have tools that listen to libudev for system monitoring
> >>>>> and management, why require them to hook up to yet-another-library? And
> >>>>> what is going to provide the ability for multiple userspace tools to
> >>>>> listen to these netlink messages in case you have more than one program
> >>>>> that wants to watch for these things (i.e. multiple desktop filesystem
> >>>>> monitoring tools, system-health checkers, etc.)?
> >>>> As much as I understand your concerns I'm not convinced uevent interface
> >>>> is a good fit. There are filesystems that don't have underlying block
> >>>> device - think of e.g. tmpfs or filesystems working directly on top of
> >>>> flash devices. These still want to send notification to userspace (one of
> >>>> primary motivation for this interfaces was so that tmpfs can notify about
> >>>> something). And creating some fake nodes in /sys/block for tmpfs and
> >>>> similar filesystems seems like doing more harm than good to me...
> >>>
> >>> If these are "fake" block devices, what's going to be present in the
> >>> block major/minor fields of the netlink message? For some reason I
> >>> thought it was a required field, and because of that, I thought we had a
> >>> "real" filesystem somewhere to refer to, otherwise how would userspace
> >>> know what filesystem was creating these events?
> >>>
> >>> What am I missing here?
> >>>
> >>> confused,
> >>>
> >>> greg k-h
> >>>
> >>
> >> For those 'fake' block devs, upon mount, get_anon_bdev will assign
> >> the major:minor numbers. Userspace might get those through stat.
> >
> > How can userspace do the mapping backwards from this "anonymous"
> > major:minor number for these types of filesystems in such a way that
> > they can "know" how to report the block device that is causing the
> > event?
> >
> > thanks,
> >
> > greg k-h
> >
>
> It needs to be done internally by the app but is doable.
> The app knows what it is watching, so it can maintain the mappings.
> So prior to activating the notifications it can call 'stat' on the mount point.
> Stat struct gives the 'st_dev' which is the device id. Same will be reported
> within the message payload (through major:minor numbers). So having this,
> the app is able to get any other information it needs.
> Note that the events refer to the file system as a whole and they may not
> necessarily have anything to do with the actual block device.
Or you can use /proc/self/mountinfo for the mapping. There you can see
device numbers, real device names if applicable and mountpoints. This has
the advantage that it works even if filesystem mountpoints change.
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [RFC 1/3] mm: mmap make MAP_LOCKED really mlock semantic
From: Michal Hocko @ 2015-04-29 7:52 UTC (permalink / raw)
To: Andrew Morton
Cc: linux-mm, Cyril Hrubis, Hugh Dickins, Michel Lespinasse,
Linus Torvalds, Rik van Riel, Michael Kerrisk, LKML, Linux API
In-Reply-To: <20150428161001.e854fb3eaf82f738865130af@linux-foundation.org>
On Tue 28-04-15 16:10:01, Andrew Morton wrote:
> On Tue, 28 Apr 2015 14:11:49 +0200 Michal Hocko <mhocko@suse.cz> wrote:
>
> > The man page however says
> > "
> > MAP_LOCKED (since Linux 2.5.37)
> > Lock the pages of the mapped region into memory in the manner of
> > mlock(2). This flag is ignored in older kernels.
> > "
>
> I'm trying to remember why we implemented MAP_LOCKED in the first
> place. Was it better than mmap+mlock in some fashion?
>
> afaict we had a #define MAP_LOCKED in the header file but it wasn't
> implemented, so we went and wired it up. 13 years ago:
> https://lkml.org/lkml/2002/9/18/108
Yeah I have encountered this one while digging though the history as
well but there was no real usecase described - except "it doesn't work
currently".
The only sensible usecase I was able to come up with was a userspace
fault handling when we need to mmap and lock the faulting address in an
atomic way so that other threads cannot possibly leak data to the swap.
These guys can live with the current implementation, though.
I do not really believe that 2 instead of 1 syscall really justifies the
complexity.
> Anyway... the third way of doing this is to use plain old mmap() while
> mlockall(MCL_FUTURE) is in force. Has anyone looked at that, checked
> that the behaviour is sane and compared it with the mmap+mlock
> behaviour, the MAP_LOCKED behaviour and the manpages?
AFAICS this will behave the same way as mmap(MAP_LOCKED). VMA will be
marked VM_LOCKED but the popullation might fail for the very same
reason.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v3 0/3] idle memory tracking
From: Vladimir Davydov @ 2015-04-29 7:58 UTC (permalink / raw)
To: Minchan Kim
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Greg Thelen,
Michel Lespinasse, David Rientjes, Pavel Emelyanov,
Cyrill Gorcunov, Jonathan Corbet, linux-api, linux-doc, linux-mm,
cgroups, linux-kernel
In-Reply-To: <20150429035722.GA11486@blaptop>
Hi Minchan,
Thank you for taking a look at this patch set.
On Wed, Apr 29, 2015 at 12:57:22PM +0900, Minchan Kim wrote:
> On Tue, Apr 28, 2015 at 03:24:39PM +0300, Vladimir Davydov wrote:
> > * /proc/kpageidle. For each page this file contains a 64-bit number, which
> > equals 1 if the page is idle or 0 otherwise, indexed by PFN. A page is
>
> Why do we need 64bit per page to indicate just idle or not?
I don't think we need this, actually. I made this file 64 bit per page
only to conform to other /proc/kpage* files. Currently, I can't think of
any potential use for residual 63 bits, so personally I'm fine with 1
bit per page. If nobody comes with an idea about how > 1 bits could be
used, I'll switch /proc/kpageidle to a bitmask in the next iteration.
Thanks,
Vladimir
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH v3 3/3] proc: add kpageidle file
From: Vladimir Davydov @ 2015-04-29 8:31 UTC (permalink / raw)
To: Minchan Kim
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Greg Thelen,
Michel Lespinasse, David Rientjes, Pavel Emelyanov,
Cyrill Gorcunov, Jonathan Corbet, linux-api, linux-doc, linux-mm,
cgroups, linux-kernel
In-Reply-To: <20150429045759.GA27051@blaptop>
On Wed, Apr 29, 2015 at 01:57:59PM +0900, Minchan Kim wrote:
> On Tue, Apr 28, 2015 at 03:24:42PM +0300, Vladimir Davydov wrote:
> > @@ -69,6 +69,14 @@ There are four components to pagemap:
> > memory cgroup each page is charged to, indexed by PFN. Only available when
> > CONFIG_MEMCG is set.
> >
> > + * /proc/kpageidle. For each page this file contains a 64-bit number, which
> > + equals 1 if the page is idle or 0 otherwise, indexed by PFN. A page is
> > + considered idle if it has not been accessed since it was marked idle. To
> > + mark a page idle one should write 1 to this file at the offset corresponding
> > + to the page. Only user memory pages can be marked idle, for other page types
> > + input is silently ignored. Writing to this file beyond max PFN results in
> > + the ENXIO error. Only available when CONFIG_IDLE_PAGE_TRACKING is set.
> > +
>
> How about using kpageflags for reading part?
>
> I mean PG_idle is one of the page flags and we already have a feature to
> parse of each PFN flag so we could reuse existing feature for reading
> idleness.
Reading PG_idle implies clearing all pte references to make sure the
page was not referenced via a pte. This means that exporting it via
/proc/kpageflags would increase the cost of reading this file, even for
users that don't care about PG_idle. I'm not sure all users of
/proc/kpageflags will be fine with it.
Thanks,
Vladimir
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox