Re: [RFC 0/4] Generic file system events interface

Re: [RFC 0/4] Generic file system events interface

[Beata Michalska]

Hi,

On 04/17/2015 10:17 AM, Jan Kara wrote:
>   Hello,
> 
> On Wed 15-04-15 09:15:43, Beata Michalska wrote:
>> The following patchset is a result of previous discussions regarding
>> file system threshold notifiactions. It introduces support for file
>> system event notifications, sent through generic netlinik interface
>> whenever an fs-related event occurs. Included are also some shmem
>> and ext4 changes showing how the new interface might actually be used.
>>
>> The vary idea of using the generic netlink interface has been previoulsy
>> suggested here:  https://lkml.org/lkml/2011/8/18/169
>>
>> The basic description of the new functionality can be found in
>> the first patch from this set - both in the commit message and
>> in the doc file.
>>
>> Some very basic tests have been performed though still this is
>> a PoC version. Below though is a sample user space application
>> which subscribes to the new multicast group and listens for
>> potential fs-related events. The code has been based on libnl 3.4
>> and its test application for the generic netlink.
>   Thanks for the patches! As a general note for the next posting, please CC
> also linux-fsdevel@vger.kernel.org (since this has implications for other
> filesystems as well, specifically I know about XFS guys thinking about some
> notification system as well) and linux-api@vger.kernel.org (since this is a
> new kernel interface to userspace).
> 
> 								Honza

My bad  - with the CC list. Apologies for that.

BR
Beata

>>
>> ---
>>
>> Beata Michalska (4):
>>   fs: Add generic file system event notifications
>>   ext4: Add helper function to mark group as corrupted
>>   ext4: Add support for generic FS events
>>   shmem: Add support for generic FS events
>>
>>  Documentation/filesystems/events.txt |  254 +++++++++++
>>  fs/Makefile                          |    1 +
>>  fs/events/Makefile                   |    6 +
>>  fs/events/fs_event.c                 |  775 ++++++++++++++++++++++++++++++++++
>>  fs/events/fs_event.h                 |   27 ++
>>  fs/events/fs_event_netlink.c         |   94 +++++
>>  fs/ext4/balloc.c                     |   26 +-
>>  fs/ext4/ext4.h                       |   10 +
>>  fs/ext4/ialloc.c                     |    5 +-
>>  fs/ext4/inode.c                      |    2 +-
>>  fs/ext4/mballoc.c                    |   17 +-
>>  fs/ext4/resize.c                     |    1 +
>>  fs/ext4/super.c                      |   43 ++
>>  fs/namespace.c                       |    1 +
>>  include/linux/fs.h                   |    6 +-
>>  include/linux/fs_event.h             |   69 +++
>>  include/uapi/linux/fs_event.h        |   62 +++
>>  include/uapi/linux/genetlink.h       |    1 +
>>  mm/shmem.c                           |   39 +-
>>  net/netlink/genetlink.c              |    7 +-
>>  20 files changed, 1412 insertions(+), 34 deletions(-)
>>  create mode 100644 Documentation/filesystems/events.txt
>>  create mode 100644 fs/events/Makefile
>>  create mode 100644 fs/events/fs_event.c
>>  create mode 100644 fs/events/fs_event.h
>>  create mode 100644 fs/events/fs_event_netlink.c
>>  create mode 100644 include/linux/fs_event.h
>>  create mode 100644 include/uapi/linux/fs_event.h
>>
>> ---
>> Sample application:
>>
>> #include <netlink/cli/utils.h>
>> #include <fs_event.h>
>>
>> #define ARRAY_SIZE(x) (sizeof(x)/sizeof((x)[0]))
>> #define LOG(args...) fprintf(stderr, args)
>>
>> static int parse_info(struct nl_cache_ops *unused, struct genl_cmd *cmd,
>> 		struct genl_info *info, void *arg)
>> {
>> 	LOG("New trace %d:\n",
>> 		info->attrs[FS_EVENT_ATR_FS_ID]
>> 		?  nla_get_u32(info->attrs[FS_EVENT_ATR_FS_ID])
>> 		: -1);
>> 	LOG("Mout point: %s\n", info->attrs[FS_EVENT_ATR_MOUNT]
>> 		? nla_get_string(info->attrs[FS_EVENT_ATR_MOUNT])
>> 		: "unknown");
>> 	return 0;
>> }
>>
>> static int parse_thres(struct nl_cache_ops *unused, struct genl_cmd *cmd,
>> 		struct genl_info *info, void *arg)
>> {
>>
>> 	LOG("Threshold notification received for trace %d:\n",
>> 		info->attrs[FS_EVENT_ATR_FS_ID]
>> 		?  nla_get_u32(info->attrs[FS_EVENT_ATR_FS_ID])
>> 		: -1);
>>
>> 	if (info->attrs[FS_EVENT_ATR_DEV_MAJOR])
>> 		LOG("Backing dev major: %u\n",
>> 			nla_get_u32(info->attrs[FS_EVENT_ATR_DEV_MAJOR]));
>> 	if (info->attrs[FS_EVENT_ATR_DEV_MINOR])
>> 		LOG("Backing dev minor: %u\n",
>> 			nla_get_u32(info->attrs[FS_EVENT_ATR_DEV_MINOR]));
>> 	LOG("Proc:              %u\n", info->attrs[FS_EVENT_ATR_CAUSED_ID] ?
>> 			nla_get_u32(info->attrs[FS_EVENT_ATR_CAUSED_ID]) : -1);
>> 	LOG("Threshold data:    %llu\n", info->attrs[FS_EVENT_ATR_DATA]
>> 		? nla_get_u64(info->attrs[FS_EVENT_ATR_DATA])
>> 		: 0);
>>
>> 	return 0;
>> }
>>
>> static int parse_warning(struct nl_cache_ops *unused, struct genl_cmd *cmd,
>> 			 struct genl_info *info, void *arg)
>> {
>>
>> 	LOG("Warning recieved for trace %d\n", info->attrs[FS_EVENT_ATR_FS_ID] ?
>> 		nla_get_u32(info->attrs[FS_EVENT_ATR_FS_ID]) : -1);
>> 	if (info->attrs[FS_EVENT_ATR_DEV_MAJOR])
>> 		LOG("Backing dev major: %u\n",
>> 			nla_get_u32(info->attrs[FS_EVENT_ATR_DEV_MAJOR]));
>> 	if (info->attrs[FS_EVENT_ATR_DEV_MINOR])
>> 		LOG("Backing dev minor: %u\n",
>> 			nla_get_u32(info->attrs[FS_EVENT_ATR_DEV_MINOR]));
>> 	LOG("Proc:              %u\n", info->attrs[FS_EVENT_ATR_CAUSED_ID] ?
>> 		nla_get_u32(info->attrs[FS_EVENT_ATR_CAUSED_ID]) : -1);
>> 	LOG("Warning:           %u\n", info->attrs[FS_EVENT_ATR_ID] ?
>> 		nla_get_u32(info->attrs[FS_EVENT_ATR_ID]) : -1);
>>
>> 	return 0;
>> }
>>
>> static struct genl_cmd cmd[] = {
>> 	{
>> 		.c_id = FS_EVENT_TYPE_NEW_TRACE,
>> 		.c_name = "info",
>> 		.c_maxattr = 2,
>> 		.c_msg_parser = parse_info,
>> 	}, {
>> 		.c_id = FS_EVENT_TYPE_THRESH,
>> 		.c_name = "thres",
>> 		.c_maxattr = 6,
>> 		.c_msg_parser = parse_thres,
>> 	}, {
>> 		.c_id = FS_EVENT_TYPE_WARN,
>> 		.c_name = "warn",
>> 		.c_maxattr = 5,
>> 		.c_msg_parser = parse_warning,
>> 	},
>> };
>>
>> static struct genl_ops ops = {
>> 	.o_id = GENL_ID_FS_EVENT,
>> 	.o_name = "FS_EVENT",
>> 	.o_hdrsize = 0,
>> 	.o_cmds = cmd,
>> 	.o_ncmds = ARRAY_SIZE(cmd),
>> };
>>
>>
>> int events_cb(struct nl_msg *msg, void *arg)
>> {
>> 	 return  genl_handle_msg(msg, arg);
>> }
>>
>> int main(int argc, char **argv)
>> {
>> 	struct nl_sock *sock;
>> 	int ret;
>>
>> 	sock = nl_cli_alloc_socket();
>> 	nl_socket_set_local_port(sock, 0);
>> 	nl_socket_disable_seq_check(sock);
>>
>> 	nl_socket_modify_cb(sock, NL_CB_VALID, NL_CB_CUSTOM, events_cb, NULL);
>>
>> 	nl_cli_connect(sock, NETLINK_GENERIC);
>>
>> 	if ((ret = nl_socket_add_membership(sock, GENL_ID_FS_EVENT))) {
>> 		LOG("Failed to add membership\n");
>> 		goto leave;
>> 	}
>>
>> 	if((ret = genl_register_family(&ops))) {
>> 		LOG("Failed to register protocol family\n");
>> 		goto leave;
>> 	}
>>
>> 	if ((ret = genl_ops_resolve(sock, &ops) < 0)) {
>> 		LOG("Unable to resolve the family name\n");
>> 		goto leave;
>> 	}
>>
>> 	if (genl_ctrl_resolve(sock, "FS_EVENT") < 0) {
>> 		LOG("Failed to resolve  the family name\n");
>> 		goto leave;
>> 	}
>>
>> 	while (1) {
>> 		if ((ret = nl_recvmsgs_default(sock)) < 0)
>> 			LOG("Unable to receive message: %s\n", nl_geterror(ret));
>> 	}
>>
>> leave:
>> 	nl_close(sock);
>> 	nl_socket_free(sock);
>> 	return 0;
>> }
>>
>> -- 
>> 1.7.9.5
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC 1/4] fs: Add generic file system event notifications

[Beata Michalska]

Hi,

On 04/16/2015 10:10 PM, Hugh Dickins wrote:
> On Thu, 16 Apr 2015, Beata Michalska wrote:
>> On 04/16/2015 05:46 AM, Eric Sandeen wrote:
>>> On 4/15/15 2:15 AM, Beata Michalska wrote:
>>>> Introduce configurable generic interface for file
>>>> system-wide event notifications to provide file
>>>> systems with a common way of reporting any potential
>>>> issues as they emerge.
>>>>
>>>> The notifications are to be issued through generic
>>>> netlink interface, by a dedicated, for file system
>>>> events, multicast group. The file systems might as
>>>> well use this group to send their own custom messages.
>>>
>>> ...
>>>
>>>> + 4.3 Threshold notifications:
>>>> +
>>>> + #include <linux/fs_event.h>
>>>> + void fs_event_alloc_space(struct super_block *sb, u64 ncount);
>>>> + void fs_event_free_space(struct super_block *sb, u64 ncount);
>>>> +
>>>> + Each filesystme supporting the treshold notifiactions should call
>>>> + fs_event_alloc_space/fs_event_free_space repsectively whenever the
>>>> + ammount of availbale blocks changes.
>>>> + - sb:     the filesystem's super block
>>>> + - ncount: number of blocks being acquired/released
>>>
>>> so:
>>>
>>>> +void fs_event_alloc_space(struct super_block *sb, u64 ncount)
>>>> +{
>>>> +	struct fs_trace_entry *en;
>>>> +	s64 count;
>>>> +
>>>> +	spin_lock(&fs_trace_lock);
>>>
>>> Every allocation/free for every supported filesystem system-wide will be
>>> serialized on this global spinlock?  That sounds like a non-starter...
>>>
>>> -Eric
>>>
>> I guess there is a plenty room for improvements as this is an early version.
>> I do agree that this might be a performance bottleneck event though I've tried
>> to keep this to minimum - it's being taken only for hashtable look-up. But still...
>> I was considering placing the trace object within the super_block to skip
>> this look-up part but I'd like to gather more comments, especially on the concept
>> itself.
> 
> Sorry, I have no opinion on the netlink fs notifications concept
> itself, not my area of expertise at all.
> 
> No doubt you Cc'ed me for tmpfs: I am very glad you're now trying the
> generic filesystem route, and yes, I'd be happy to have the support
> in tmpfs, thank you - if it is generally agreed to be suitable for
> filesystems; but wouldn't want this as a special for tmpfs.
> 
> However, I must echo Eric's point: please take a look at 7e496299d4d2
> "tmpfs: make tmpfs scalable with percpu_counter for used blocks":
> Tim would be unhappy if you added overhead back into that path.
> 
> (And please Cc linux-fsdevel@vger.kernel.org next time you post these.)
> 
> Hugh
> 

Well, the concept of using netlink interface here is just a part of the overall
idea - so any comments are really welcomed here. The more of them the better solution
can be worked out, as I believe.

As for the possible overhead: this is the last thing I would want, so I'll
definitely do may best to not to introduce any. I will definitely rework this.

Thanks for Your comments,

BR
Beata


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC 1/4] fs: Add generic file system event notifications

[Beata Michalska]

Hi,

On 04/16/2015 11:56 PM, Heinrich Schuchardt wrote:
> On 15.04.2015 09:15, Beata Michalska wrote:
>> Introduce configurable generic interface for file
>> system-wide event notifications to provide file
>> systems with a common way of reporting any potential
>> issues as they emerge.
>>
>> The notifications are to be issued through generic
>> netlink interface, by a dedicated, for file system
>> events, multicast group. The file systems might as
>> well use this group to send their own custom messages.
>>
>> The events have been split into four base categories:
>> information, warnings, errors and threshold notifications,
>> with some very basic event types like running out of space
>> or file system being remounted as read-only.
>>
>> Threshold notifications have been included to allow
>> triggering an event whenever the amount of free space
>> drops below a certain level - or levels to be more precise
>> as two of them are being supported: the lower and the upper
>> range. The notifications work both ways: once the threshold
>> level has been reached, an event shall be generated whenever
>> the number of available blocks goes up again re-activating
>> the threshold.
>>
>> The interface has been exposed through a vfs. Once mounted,
>> it serves as an entry point for the set-up where one can
>> register for particular file system events.
> 
> Having a framework for notification for file systems is a great idea.
> Your solution covers an important part of the possible application scope.
> 
> Before moving forward I suggest we should analyze if this scope should
> be enlarged.
> 
> Many filesystems are remote (e.g. CIFS/Samba) or distributed over many
> network nodes (e.g. Lustre). How should file system notification work here?
> 
> How will fuse file systems be served?
> 
> The current point of reference is a single mount point.
> Every time I insert an USB stick several file system may be automounted.
> I would like to receive events for these automounted file systems.
> 
> A similar case arises when starting new virtual machines. How will I
> receive events on the host system for the file systems of the virtual
> machines?

> In your implementation events are received via Netlink.
> Using Netlink for marking mounts for notification would create a much
> more homogenous interface. So why should we use a virtual file system here?
> 
> Best regards
> 
> Heinrich Schuchardt
> 
> 

I'd be more than happy to extend the scope of suggested changes.
I hope I'll be able to collect more comments - in this way there 
is a chance we might get here smth that is really useful, for everyone.

I've tried to make the interface rather flexible, so that new cases
can be easily added - so the notification whenever a file system
is being mounted is definitely doable.

The vfs here merely serves the purpose to configure which type of events
and for which filesystems are to be issued. Having this done through
netlink is also an option, though it needs some more thoughts. The way
notifications are being sent might be extended: so there could be more
than one option for this. We might also want to consider if we want to
have this widely available - everything for everyone. (?)

As for the rest, I must admit I'm not really an fs person, so I assume
there will be more comments and questions like yours. This is also why
any comments/hints/remarks/doubts/issues etc would me more than just
welcomed. I'll try to answer them all, though this will require some
time on my side, thus apologies if I have some delays.


I'll get beck to this asap.

BR
Beata

Re: [RFC 1/4] fs: Add generic file system event notifications

[Beata Michalska]

On 04/17/2015 01:31 PM, Jan Kara wrote:
> On Wed 15-04-15 09:15:44, Beata Michalska wrote:
>> Introduce configurable generic interface for file
>> system-wide event notifications to provide file
>> systems with a common way of reporting any potential
>> issues as they emerge.
>>
>> The notifications are to be issued through generic
>> netlink interface, by a dedicated, for file system
>> events, multicast group. The file systems might as
>> well use this group to send their own custom messages.
>>
>> The events have been split into four base categories:
>> information, warnings, errors and threshold notifications,
>> with some very basic event types like running out of space
>> or file system being remounted as read-only.
>>
>> Threshold notifications have been included to allow
>> triggering an event whenever the amount of free space
>> drops below a certain level - or levels to be more precise
>> as two of them are being supported: the lower and the upper
>> range. The notifications work both ways: once the threshold
>> level has been reached, an event shall be generated whenever
>> the number of available blocks goes up again re-activating
>> the threshold.
>>
>> The interface has been exposed through a vfs. Once mounted,
>> it serves as an entry point for the set-up where one can
>> register for particular file system events.
>>
>> Signed-off-by: Beata Michalska <b.michalska@samsung.com>
>   Thanks for the patches! Some comments are below.
> 
>> ---
>>  Documentation/filesystems/events.txt |  254 +++++++++++
>>  fs/Makefile                          |    1 +
>>  fs/events/Makefile                   |    6 +
>>  fs/events/fs_event.c                 |  775 ++++++++++++++++++++++++++++++++++
>>  fs/events/fs_event.h                 |   27 ++
>>  fs/events/fs_event_netlink.c         |   94 +++++
>>  fs/namespace.c                       |    1 +
>>  include/linux/fs.h                   |    6 +-
>>  include/linux/fs_event.h             |   69 +++
>>  include/uapi/linux/fs_event.h        |   62 +++
>>  include/uapi/linux/genetlink.h       |    1 +
>>  net/netlink/genetlink.c              |    7 +-
>>  12 files changed, 1301 insertions(+), 2 deletions(-)
>>  create mode 100644 Documentation/filesystems/events.txt
>>  create mode 100644 fs/events/Makefile
>>  create mode 100644 fs/events/fs_event.c
>>  create mode 100644 fs/events/fs_event.h
>>  create mode 100644 fs/events/fs_event_netlink.c
>>  create mode 100644 include/linux/fs_event.h
>>  create mode 100644 include/uapi/linux/fs_event.h
>>
>> diff --git a/Documentation/filesystems/events.txt b/Documentation/filesystems/events.txt
>> new file mode 100644
>> index 0000000..c85dd88
>> --- /dev/null
>> +++ b/Documentation/filesystems/events.txt
>> @@ -0,0 +1,254 @@
>> +
>> +	Generic file system event notification interface
>> +
>> +Document created 09 April 2015 by Beata Michalska <b.michalska@samsung.com>
>> +
>> +1. The reason behind:
>> +=====================
>> +
>> +There are many corner cases when things might get messy with the filesystems.
>> +And it is not always obvious what and when went wrong. Sometimes you might
>> +get some subtle hints that there is something going on - but by the time
>> +you realise it, it might be too late as you are already out-of-space
>> +or the filesystem has been remounted as read-only (i.e.). The generic
>> +interface for the filesystem events fills the gap by providing a rather
>> +easy way of real-time notifications triggered whenever something intreseting
>> +happens, allowing filesystems to report events in a common way, as they occur.
>> +
>> +2. How does it work:
>> +====================
>> +
>> +The interface itself has been exposed as fstrace-type Virtual File System,
>> +primarily to ease the process of setting up the configuration for the file
>> +system notifications. So for starters it needs to get mounted (obviously):
>> +
>> +	mount -t fstrace none /sys/fs/events
>> +
>> +This will unveil the single fstrace filesystem entry - the 'config' file,
>> +through which the notification are being set-up.
>> +
>> +Activating notifications for particular filesystem is as straightforward
>> +as writing into the 'config' file. Note that by default all events despite
>> +the actual filesystem type are being disregarded.
>   Is there a reason to have a special filesystem for this? Do you expect
> extending it by (many) more files? Why not just creating a file in sysfs or
> something like that?

No particular reason here - just for possible future extension if needed.
I'm totally fine with having a single sysfs entry.

> 
>> +Synopsis of config:
>> +------------------
>> +
>> +	MOUNT EVENT_TYPE [L1] [L2]
>> +
>> + MOUNT      : the filesystem's mount point
>   I'm not quite decided but is mountpoint really the right thing to pass
> via the interface? They aren't unique (filesystem can be mounted in
> multiple places) and more importantly can change over time. So won't it be
> better to pass major:minor over the interface? These are stable, unique to
> the filesystem, and userspace can easily get them by calling stat(2) on the
> desired path (or directly from /proc/self/mountinfo). That could be also
> used as an fs identifier instead of assigned ID (and thus we won't need
> those events about creation of new trace which look somewhat strange to
> me).
> 
Even if a given filesystem is being mounted in many places this will come
down to single super_block - the interface will add trace for the first mount
point. This is just to ease the usage: internally a particular trace is 
associated with a super_block. 

> OTOH using major:minor may have issues in container world where processes
> could watch events from filesystems inaccessible to the container if they
> guess the device number. So maybe we could use 'path' when creating new
> trace but I'd still like to use the device number internally and for all
> outgoing communication because of above mentioned problems with
> mountpoints.

Alright then, so dropping the idea of announcing new trace (with assigned id)
and switching to using the major:minor numbers. Sounds OK to me.

> 
>> + EVENT_TYPE : type of events to be enabled: info,warn,err,thr;
>> +              at least one type needs to be specified;
>> +              note the comma delimiter and lack of spaces between
>> +	      those options
>> + L1         : the threshold limit - lower range
>> + L2         : the threshold limit - upper range
>> + 	      case enabling threshold notifications the lower level is
>> +	      mandatory, whereas the upper one remains optional;
>> +	      note though, that as those refer to the number of available
>> +	      blocks, the lower level needs to be higher than the upper one
>> +
>> +Sample request could look like the follwoing:
>> +
>> + echo /sample/mount/point warn,err,thr 710000 500000 > /sys/fs/events/config
>> +
>> +Multiple request might be specified provided they are separated with semicolon.
>   Is this necessary? It somewhat complicates syntax and parsing in kernel
> and I don't see a need for that. I'd prefer to keep the interface as simple
> as possible.
> 

This is not necessary but could ease the usage - i.e. through scripting: to specify
multiple traces and register them in one go. 

> Also I think that we should make it clear that each event type has
> different set of arguments. For threshold events they'll be L1 & L2, for
> other events there may be no arguments, for other events maybe something
> else...
> 

Currently only the threshold events use arguments -  not sure what arguments
could be used for the remaining notifications. But any suggestions are welcomed.

> ...
>> +static const match_table_t fs_etypes = {
>> +	{ FS_EVENT_INFO,    "info"  },
>> +	{ FS_EVENT_WARN,    "warn"  },
>> +	{ FS_EVENT_THRESH,  "thr"   },
>> +	{ FS_EVENT_ERR,     "err"   },
>> +	{ 0, NULL },
>> +};
>   Why are there these generic message types? Threshold messages make good
> sense to me. But not so much the rest. If they don't have a clear meaning,
> it will be a mess. So I also agree with a message like - "filesystem has
> trouble, you should probably unmount and run fsck" - that's fine. But
> generic "info" or "warning" doesn't really carry any meaning on its own and
> thus seems pretty useless to me. To explain a bit more, AFAIU this
> shouldn't be a generic logging interface where something like severity
> makes sense but rather a relatively specific interface notifying about
> events in filesystem userspace should know about so I expect relatively low
> number of types of events, not tens or even hundreds...
> 
> 								Honza

Getting rid of those would simplify the configuration part, indeed.
So we would be left with 'generic' and threshold events.
I guess I've overdone this part.

Thanks for Your comments so far.

BR
Beata





--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC 1/4] fs: Add generic file system event notifications

[Beata Michalska]

On 04/17/2015 03:04 PM, Beata Michalska wrote:
> On 04/17/2015 01:31 PM, Jan Kara wrote:
>> On Wed 15-04-15 09:15:44, Beata Michalska wrote:
>>> Introduce configurable generic interface for file
>>> system-wide event notifications to provide file
>>> systems with a common way of reporting any potential
>>> issues as they emerge.
>>>
>>> The notifications are to be issued through generic
>>> netlink interface, by a dedicated, for file system
>>> events, multicast group. The file systems might as
>>> well use this group to send their own custom messages.
>>>
>>> The events have been split into four base categories:
>>> information, warnings, errors and threshold notifications,
>>> with some very basic event types like running out of space
>>> or file system being remounted as read-only.
>>>
>>> Threshold notifications have been included to allow
>>> triggering an event whenever the amount of free space
>>> drops below a certain level - or levels to be more precise
>>> as two of them are being supported: the lower and the upper
>>> range. The notifications work both ways: once the threshold
>>> level has been reached, an event shall be generated whenever
>>> the number of available blocks goes up again re-activating
>>> the threshold.
>>>
>>> The interface has been exposed through a vfs. Once mounted,
>>> it serves as an entry point for the set-up where one can
>>> register for particular file system events.
>>>
>>> Signed-off-by: Beata Michalska <b.michalska@samsung.com>
>>   Thanks for the patches! Some comments are below.
>>
>>> ---
>>>  Documentation/filesystems/events.txt |  254 +++++++++++
>>>  fs/Makefile                          |    1 +
>>>  fs/events/Makefile                   |    6 +
>>>  fs/events/fs_event.c                 |  775 ++++++++++++++++++++++++++++++++++
>>>  fs/events/fs_event.h                 |   27 ++
>>>  fs/events/fs_event_netlink.c         |   94 +++++
>>>  fs/namespace.c                       |    1 +
>>>  include/linux/fs.h                   |    6 +-
>>>  include/linux/fs_event.h             |   69 +++
>>>  include/uapi/linux/fs_event.h        |   62 +++
>>>  include/uapi/linux/genetlink.h       |    1 +
>>>  net/netlink/genetlink.c              |    7 +-
>>>  12 files changed, 1301 insertions(+), 2 deletions(-)
>>>  create mode 100644 Documentation/filesystems/events.txt
>>>  create mode 100644 fs/events/Makefile
>>>  create mode 100644 fs/events/fs_event.c
>>>  create mode 100644 fs/events/fs_event.h
>>>  create mode 100644 fs/events/fs_event_netlink.c
>>>  create mode 100644 include/linux/fs_event.h
>>>  create mode 100644 include/uapi/linux/fs_event.h
>>>
>>> diff --git a/Documentation/filesystems/events.txt b/Documentation/filesystems/events.txt
>>> new file mode 100644
>>> index 0000000..c85dd88
>>> --- /dev/null
>>> +++ b/Documentation/filesystems/events.txt
>>> @@ -0,0 +1,254 @@
>>> +
>>> +	Generic file system event notification interface
>>> +
>>> +Document created 09 April 2015 by Beata Michalska <b.michalska@samsung.com>
>>> +
>>> +1. The reason behind:
>>> +=====================
>>> +
>>> +There are many corner cases when things might get messy with the filesystems.
>>> +And it is not always obvious what and when went wrong. Sometimes you might
>>> +get some subtle hints that there is something going on - but by the time
>>> +you realise it, it might be too late as you are already out-of-space
>>> +or the filesystem has been remounted as read-only (i.e.). The generic
>>> +interface for the filesystem events fills the gap by providing a rather
>>> +easy way of real-time notifications triggered whenever something intreseting
>>> +happens, allowing filesystems to report events in a common way, as they occur.
>>> +
>>> +2. How does it work:
>>> +====================
>>> +
>>> +The interface itself has been exposed as fstrace-type Virtual File System,
>>> +primarily to ease the process of setting up the configuration for the file
>>> +system notifications. So for starters it needs to get mounted (obviously):
>>> +
>>> +	mount -t fstrace none /sys/fs/events
>>> +
>>> +This will unveil the single fstrace filesystem entry - the 'config' file,
>>> +through which the notification are being set-up.
>>> +
>>> +Activating notifications for particular filesystem is as straightforward
>>> +as writing into the 'config' file. Note that by default all events despite
>>> +the actual filesystem type are being disregarded.
>>   Is there a reason to have a special filesystem for this? Do you expect
>> extending it by (many) more files? Why not just creating a file in sysfs or
>> something like that?
> 
> No particular reason here - just for possible future extension if needed.
> I'm totally fine with having a single sysfs entry.
> 

On the other hand .... sysfs entries are mostly single-valued or are sets
of values of a single type, so not sure if we would fit in here -
with the current configuration for the interface.

>>
>>> +Synopsis of config:
>>> +------------------
>>> +
>>> +	MOUNT EVENT_TYPE [L1] [L2]
>>> +
>>> + MOUNT      : the filesystem's mount point
>>   I'm not quite decided but is mountpoint really the right thing to pass
>> via the interface? They aren't unique (filesystem can be mounted in
>> multiple places) and more importantly can change over time. So won't it be
>> better to pass major:minor over the interface? These are stable, unique to
>> the filesystem, and userspace can easily get them by calling stat(2) on the
>> desired path (or directly from /proc/self/mountinfo). That could be also
>> used as an fs identifier instead of assigned ID (and thus we won't need
>> those events about creation of new trace which look somewhat strange to
>> me).
>>
> Even if a given filesystem is being mounted in many places this will come
> down to single super_block - the interface will add trace for the first mount
> point. This is just to ease the usage: internally a particular trace is 
> associated with a super_block. 
> 
>> OTOH using major:minor may have issues in container world where processes
>> could watch events from filesystems inaccessible to the container if they
>> guess the device number. So maybe we could use 'path' when creating new
>> trace but I'd still like to use the device number internally and for all
>> outgoing communication because of above mentioned problems with
>> mountpoints.
> 
> Alright then, so dropping the idea of announcing new trace (with assigned id)
> and switching to using the major:minor numbers. Sounds OK to me.
> 
>>
>>> + EVENT_TYPE : type of events to be enabled: info,warn,err,thr;
>>> +              at least one type needs to be specified;
>>> +              note the comma delimiter and lack of spaces between
>>> +	      those options
>>> + L1         : the threshold limit - lower range
>>> + L2         : the threshold limit - upper range
>>> + 	      case enabling threshold notifications the lower level is
>>> +	      mandatory, whereas the upper one remains optional;
>>> +	      note though, that as those refer to the number of available
>>> +	      blocks, the lower level needs to be higher than the upper one
>>> +
>>> +Sample request could look like the follwoing:
>>> +
>>> + echo /sample/mount/point warn,err,thr 710000 500000 > /sys/fs/events/config
>>> +
>>> +Multiple request might be specified provided they are separated with semicolon.
>>   Is this necessary? It somewhat complicates syntax and parsing in kernel
>> and I don't see a need for that. I'd prefer to keep the interface as simple
>> as possible.
>>
> 
> This is not necessary but could ease the usage - i.e. through scripting: to specify
> multiple traces and register them in one go. 
> 
>> Also I think that we should make it clear that each event type has
>> different set of arguments. For threshold events they'll be L1 & L2, for
>> other events there may be no arguments, for other events maybe something
>> else...
>>
> 
> Currently only the threshold events use arguments -  not sure what arguments
> could be used for the remaining notifications. But any suggestions are welcomed.
> 
>> ...
>>> +static const match_table_t fs_etypes = {
>>> +	{ FS_EVENT_INFO,    "info"  },
>>> +	{ FS_EVENT_WARN,    "warn"  },
>>> +	{ FS_EVENT_THRESH,  "thr"   },
>>> +	{ FS_EVENT_ERR,     "err"   },
>>> +	{ 0, NULL },
>>> +};
>>   Why are there these generic message types? Threshold messages make good
>> sense to me. But not so much the rest. If they don't have a clear meaning,
>> it will be a mess. So I also agree with a message like - "filesystem has
>> trouble, you should probably unmount and run fsck" - that's fine. But
>> generic "info" or "warning" doesn't really carry any meaning on its own and
>> thus seems pretty useless to me. To explain a bit more, AFAIU this
>> shouldn't be a generic logging interface where something like severity
>> makes sense but rather a relatively specific interface notifying about
>> events in filesystem userspace should know about so I expect relatively low
>> number of types of events, not tens or even hundreds...
>>
>> 								Honza
> 
> Getting rid of those would simplify the configuration part, indeed.
> So we would be left with 'generic' and threshold events.
> I guess I've overdone this part.
> 
> Thanks for Your comments so far.
> 
> BR
> Beata
> 
> 
> 
> 
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC 1/4] fs: Add generic file system event notifications

[Jan Kara]

On Fri 17-04-15 15:04:37, Beata Michalska wrote:
> On 04/17/2015 01:31 PM, Jan Kara wrote:
> > On Wed 15-04-15 09:15:44, Beata Michalska wrote:
> > Also I think that we should make it clear that each event type has
> > different set of arguments. For threshold events they'll be L1 & L2, for
> > other events there may be no arguments, for other events maybe something
> > else...
> > 
> 
> Currently only the threshold events use arguments -  not sure what arguments
> could be used for the remaining notifications. But any suggestions are welcomed.
  Me neither be someone will surely find something in future ;)

> > ...
> >> +static const match_table_t fs_etypes = {
> >> +	{ FS_EVENT_INFO,    "info"  },
> >> +	{ FS_EVENT_WARN,    "warn"  },
> >> +	{ FS_EVENT_THRESH,  "thr"   },
> >> +	{ FS_EVENT_ERR,     "err"   },
> >> +	{ 0, NULL },
> >> +};
> >   Why are there these generic message types? Threshold messages make good
> > sense to me. But not so much the rest. If they don't have a clear meaning,
> > it will be a mess. So I also agree with a message like - "filesystem has
> > trouble, you should probably unmount and run fsck" - that's fine. But
> > generic "info" or "warning" doesn't really carry any meaning on its own and
> > thus seems pretty useless to me. To explain a bit more, AFAIU this
> > shouldn't be a generic logging interface where something like severity
> > makes sense but rather a relatively specific interface notifying about
> > events in filesystem userspace should know about so I expect relatively low
> > number of types of events, not tens or even hundreds...
> > 
> 
> Getting rid of those would simplify the configuration part, indeed.
> So we would be left with 'generic' and threshold events.
> I guess I've overdone this part.
  Well, I would avoid defining anything that's not really used. So
currently you can define threshold events and we start with just those.
When someone hooks up filesystem error paths to send notification, we can
create event type for telling "filesystem corrupted". And so on... We just
have to be careful to document that new event types can be added and
userspace has to ignore events it does not understand.

								Honza
-- 
Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org>
SUSE Labs, CR

Re: [RFC 1/4] fs: Add generic file system event notifications

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1677 bytes --]

On 2015-04-17 09:04, Beata Michalska wrote:
> On 04/17/2015 01:31 PM, Jan Kara wrote:
>> On Wed 15-04-15 09:15:44, Beata Michalska wrote:
>> ...
>>> +static const match_table_t fs_etypes = {
>>> +	{ FS_EVENT_INFO,    "info"  },
>>> +	{ FS_EVENT_WARN,    "warn"  },
>>> +	{ FS_EVENT_THRESH,  "thr"   },
>>> +	{ FS_EVENT_ERR,     "err"   },
>>> +	{ 0, NULL },
>>> +};
>>    Why are there these generic message types? Threshold messages make good
>> sense to me. But not so much the rest. If they don't have a clear meaning,
>> it will be a mess. So I also agree with a message like - "filesystem has
>> trouble, you should probably unmount and run fsck" - that's fine. But
>> generic "info" or "warning" doesn't really carry any meaning on its own and
>> thus seems pretty useless to me. To explain a bit more, AFAIU this
>> shouldn't be a generic logging interface where something like severity
>> makes sense but rather a relatively specific interface notifying about
>> events in filesystem userspace should know about so I expect relatively low
>> number of types of events, not tens or even hundreds...
>>
>> 								Honza
>
> Getting rid of those would simplify the configuration part, indeed.
> So we would be left with 'generic' and threshold events.
> I guess I've overdone this part.

For some filesystems, it may make sense to differentiate between a 
generic warning and an error.  For BTRFS and ZFS for example, if there 
is a csum error on a block, this will get automatically corrected in 
many configurations, and won't require anything like fsck to be run, but 
monitoring applications will still probably want to be notified.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]

Re: [RFC 1/4] fs: Add generic file system event notifications

[Jan Kara]

On Fri 17-04-15 09:23:35, Austin S Hemmelgarn wrote:
> On 2015-04-17 09:04, Beata Michalska wrote:
> >On 04/17/2015 01:31 PM, Jan Kara wrote:
> >>On Wed 15-04-15 09:15:44, Beata Michalska wrote:
> >>...
> >>>+static const match_table_t fs_etypes = {
> >>>+	{ FS_EVENT_INFO,    "info"  },
> >>>+	{ FS_EVENT_WARN,    "warn"  },
> >>>+	{ FS_EVENT_THRESH,  "thr"   },
> >>>+	{ FS_EVENT_ERR,     "err"   },
> >>>+	{ 0, NULL },
> >>>+};
> >>   Why are there these generic message types? Threshold messages make good
> >>sense to me. But not so much the rest. If they don't have a clear meaning,
> >>it will be a mess. So I also agree with a message like - "filesystem has
> >>trouble, you should probably unmount and run fsck" - that's fine. But
> >>generic "info" or "warning" doesn't really carry any meaning on its own and
> >>thus seems pretty useless to me. To explain a bit more, AFAIU this
> >>shouldn't be a generic logging interface where something like severity
> >>makes sense but rather a relatively specific interface notifying about
> >>events in filesystem userspace should know about so I expect relatively low
> >>number of types of events, not tens or even hundreds...
> >>
> >>								Honza
> >
> >Getting rid of those would simplify the configuration part, indeed.
> >So we would be left with 'generic' and threshold events.
> >I guess I've overdone this part.
> 
> For some filesystems, it may make sense to differentiate between a
> generic warning and an error.  For BTRFS and ZFS for example, if
> there is a csum error on a block, this will get automatically
> corrected in many configurations, and won't require anything like
> fsck to be run, but monitoring applications will still probably want
> to be notified.
   Sure, but in that case just create an event CORRECTED_CHECKSUM_ERROR and
use that. Then userspace knows what it should do with the event. No need to
hide it behind warning / error category.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [RFC 1/4] fs: Add generic file system event notifications

[John Spray]

On 17/04/2015 14:23, Austin S Hemmelgarn wrote:
> On 2015-04-17 09:04, Beata Michalska wrote:
>> On 04/17/2015 01:31 PM, Jan Kara wrote:
>>> On Wed 15-04-15 09:15:44, Beata Michalska wrote:
>>> ...
>>>> +static const match_table_t fs_etypes = {
>>>> +    { FS_EVENT_INFO,    "info"  },
>>>> +    { FS_EVENT_WARN,    "warn"  },
>>>> +    { FS_EVENT_THRESH,  "thr"   },
>>>> +    { FS_EVENT_ERR,     "err"   },
>>>> +    { 0, NULL },
>>>> +};
>>>    Why are there these generic message types? Threshold messages 
>>> make good
>>> sense to me. But not so much the rest. If they don't have a clear 
>>> meaning,
>>> it will be a mess. So I also agree with a message like - "filesystem 
>>> has
>>> trouble, you should probably unmount and run fsck" - that's fine. But
>>> generic "info" or "warning" doesn't really carry any meaning on its 
>>> own and
>>> thus seems pretty useless to me. To explain a bit more, AFAIU this
>>> shouldn't be a generic logging interface where something like severity
>>> makes sense but rather a relatively specific interface notifying about
>>> events in filesystem userspace should know about so I expect 
>>> relatively low
>>> number of types of events, not tens or even hundreds...
>>>
>>>                                 Honza
>>
>> Getting rid of those would simplify the configuration part, indeed.
>> So we would be left with 'generic' and threshold events.
>> I guess I've overdone this part.
>
> For some filesystems, it may make sense to differentiate between a 
> generic warning and an error.  For BTRFS and ZFS for example, if there 
> is a csum error on a block, this will get automatically corrected in 
> many configurations, and won't require anything like fsck to be run, 
> but monitoring applications will still probably want to be notified.

Another key differentiation IMHO is between transient errors (like 
server is unavailable in a distributed filesystem) that will block the 
filesystem but might clear on their own, vs. permanent errors like 
unreadable drives that definitely will not clear until the administrator 
takes some action.  It's usually a reasonable approximation to call 
transient issues warnings, and permanent issues errors.

John




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC 1/4] fs: Add generic file system event notifications

[Jan Kara]

On Fri 17-04-15 15:51:14, John Spray wrote:
> On 17/04/2015 14:23, Austin S Hemmelgarn wrote:
> >On 2015-04-17 09:04, Beata Michalska wrote:
> >>On 04/17/2015 01:31 PM, Jan Kara wrote:
> >>>On Wed 15-04-15 09:15:44, Beata Michalska wrote:
> >>>...
> >>>>+static const match_table_t fs_etypes = {
> >>>>+    { FS_EVENT_INFO,    "info"  },
> >>>>+    { FS_EVENT_WARN,    "warn"  },
> >>>>+    { FS_EVENT_THRESH,  "thr"   },
> >>>>+    { FS_EVENT_ERR,     "err"   },
> >>>>+    { 0, NULL },
> >>>>+};
> >>>   Why are there these generic message types? Threshold
> >>>messages make good
> >>>sense to me. But not so much the rest. If they don't have a
> >>>clear meaning,
> >>>it will be a mess. So I also agree with a message like -
> >>>"filesystem has
> >>>trouble, you should probably unmount and run fsck" - that's fine. But
> >>>generic "info" or "warning" doesn't really carry any meaning
> >>>on its own and
> >>>thus seems pretty useless to me. To explain a bit more, AFAIU this
> >>>shouldn't be a generic logging interface where something like severity
> >>>makes sense but rather a relatively specific interface notifying about
> >>>events in filesystem userspace should know about so I expect
> >>>relatively low
> >>>number of types of events, not tens or even hundreds...
> >>>
> >>>                                Honza
> >>
> >>Getting rid of those would simplify the configuration part, indeed.
> >>So we would be left with 'generic' and threshold events.
> >>I guess I've overdone this part.
> >
> >For some filesystems, it may make sense to differentiate between a
> >generic warning and an error.  For BTRFS and ZFS for example, if
> >there is a csum error on a block, this will get automatically
> >corrected in many configurations, and won't require anything like
> >fsck to be run, but monitoring applications will still probably
> >want to be notified.
> 
> Another key differentiation IMHO is between transient errors (like
> server is unavailable in a distributed filesystem) that will block
> the filesystem but might clear on their own, vs. permanent errors
> like unreadable drives that definitely will not clear until the
> administrator takes some action.  It's usually a reasonable
> approximation to call transient issues warnings, and permanent
> issues errors.
  So you can have events like FS_UNAVAILABLE and FS_AVAILABLE but what use
would this have? I wouldn't like the interface to be dumping ground for
random crap - we have dmesg for that :).

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC 1/4] fs: Add generic file system event notifications

[John Spray]

On 17/04/2015 16:43, Jan Kara wrote:
> On Fri 17-04-15 15:51:14, John Spray wrote:
>> On 17/04/2015 14:23, Austin S Hemmelgarn wrote:
>>
>>> For some filesystems, it may make sense to differentiate between a
>>> generic warning and an error.  For BTRFS and ZFS for example, if
>>> there is a csum error on a block, this will get automatically
>>> corrected in many configurations, and won't require anything like
>>> fsck to be run, but monitoring applications will still probably
>>> want to be notified.
>> Another key differentiation IMHO is between transient errors (like
>> server is unavailable in a distributed filesystem) that will block
>> the filesystem but might clear on their own, vs. permanent errors
>> like unreadable drives that definitely will not clear until the
>> administrator takes some action.  It's usually a reasonable
>> approximation to call transient issues warnings, and permanent
>> issues errors.
>    So you can have events like FS_UNAVAILABLE and FS_AVAILABLE but what use
> would this have? I wouldn't like the interface to be dumping ground for
> random crap - we have dmesg for that :).
In that case I'm confused -- why would ENOSPC be an appropriate use of 
this interface if the mount being entirely blocked would be 
inappropriate?  Isn't being unable to service any I/O a more fundamental 
and severe thing than being up and healthy but full?

Were you intending the interface to be exclusively for data integrity 
issues like checksum failures, rather than more general events about a 
mount that userspace would probably like to know about?

John

Re: [RFC 1/4] fs: Add generic file system event notifications

[Jan Kara]

On Fri 17-04-15 17:08:10, John Spray wrote:
> 
> On 17/04/2015 16:43, Jan Kara wrote:
> >On Fri 17-04-15 15:51:14, John Spray wrote:
> >>On 17/04/2015 14:23, Austin S Hemmelgarn wrote:
> >>
> >>>For some filesystems, it may make sense to differentiate between a
> >>>generic warning and an error.  For BTRFS and ZFS for example, if
> >>>there is a csum error on a block, this will get automatically
> >>>corrected in many configurations, and won't require anything like
> >>>fsck to be run, but monitoring applications will still probably
> >>>want to be notified.
> >>Another key differentiation IMHO is between transient errors (like
> >>server is unavailable in a distributed filesystem) that will block
> >>the filesystem but might clear on their own, vs. permanent errors
> >>like unreadable drives that definitely will not clear until the
> >>administrator takes some action.  It's usually a reasonable
> >>approximation to call transient issues warnings, and permanent
> >>issues errors.
> >   So you can have events like FS_UNAVAILABLE and FS_AVAILABLE but what use
> >would this have? I wouldn't like the interface to be dumping ground for
> >random crap - we have dmesg for that :).
> In that case I'm confused -- why would ENOSPC be an appropriate use
> of this interface if the mount being entirely blocked would be
> inappropriate?  Isn't being unable to service any I/O a more
> fundamental and severe thing than being up and healthy but full?
> 
> Were you intending the interface to be exclusively for data
> integrity issues like checksum failures, rather than more general
> events about a mount that userspace would probably like to know
> about?
  Well, I'm not saying we cannot have those events for fs availability /
inavailability. I'm just saying I'd like to see some use for that first.
I don't want events to be added just because it's possible...

For ENOSPC we have thin provisioned storage and the userspace deamon
shuffling real storage underneath. So there I know the usecase.

								Honza
-- 
Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org>
SUSE Labs, CR

Re: [RFC 1/4] fs: Add generic file system event notifications

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2311 bytes --]

On 2015-04-17 12:22, Jan Kara wrote:
> On Fri 17-04-15 17:08:10, John Spray wrote:
>>
>> On 17/04/2015 16:43, Jan Kara wrote:
>>> On Fri 17-04-15 15:51:14, John Spray wrote:
>>>> On 17/04/2015 14:23, Austin S Hemmelgarn wrote:
>>>>
>>>>> For some filesystems, it may make sense to differentiate between a
>>>>> generic warning and an error.  For BTRFS and ZFS for example, if
>>>>> there is a csum error on a block, this will get automatically
>>>>> corrected in many configurations, and won't require anything like
>>>>> fsck to be run, but monitoring applications will still probably
>>>>> want to be notified.
>>>> Another key differentiation IMHO is between transient errors (like
>>>> server is unavailable in a distributed filesystem) that will block
>>>> the filesystem but might clear on their own, vs. permanent errors
>>>> like unreadable drives that definitely will not clear until the
>>>> administrator takes some action.  It's usually a reasonable
>>>> approximation to call transient issues warnings, and permanent
>>>> issues errors.
>>>    So you can have events like FS_UNAVAILABLE and FS_AVAILABLE but what use
>>> would this have? I wouldn't like the interface to be dumping ground for
>>> random crap - we have dmesg for that :).
>> In that case I'm confused -- why would ENOSPC be an appropriate use
>> of this interface if the mount being entirely blocked would be
>> inappropriate?  Isn't being unable to service any I/O a more
>> fundamental and severe thing than being up and healthy but full?
>>
>> Were you intending the interface to be exclusively for data
>> integrity issues like checksum failures, rather than more general
>> events about a mount that userspace would probably like to know
>> about?
>    Well, I'm not saying we cannot have those events for fs availability /
> inavailability. I'm just saying I'd like to see some use for that first.
> I don't want events to be added just because it's possible...
>
> For ENOSPC we have thin provisioned storage and the userspace deamon
> shuffling real storage underneath. So there I know the usecase.
>
> 								Honza
>
The use-case that immediately comes to mind for me would be diskless 
nodes with root-on-nfs needing to know if they can actually access the 
root filesystem.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]

Re: [RFC 1/4] fs: Add generic file system event notifications

[Jan Kara]

On Fri 17-04-15 12:29:07, Austin S Hemmelgarn wrote:
> On 2015-04-17 12:22, Jan Kara wrote:
> >On Fri 17-04-15 17:08:10, John Spray wrote:
> >>
> >>On 17/04/2015 16:43, Jan Kara wrote:
> >>>On Fri 17-04-15 15:51:14, John Spray wrote:
> >>>>On 17/04/2015 14:23, Austin S Hemmelgarn wrote:
> >>>>
> >>>>>For some filesystems, it may make sense to differentiate between a
> >>>>>generic warning and an error.  For BTRFS and ZFS for example, if
> >>>>>there is a csum error on a block, this will get automatically
> >>>>>corrected in many configurations, and won't require anything like
> >>>>>fsck to be run, but monitoring applications will still probably
> >>>>>want to be notified.
> >>>>Another key differentiation IMHO is between transient errors (like
> >>>>server is unavailable in a distributed filesystem) that will block
> >>>>the filesystem but might clear on their own, vs. permanent errors
> >>>>like unreadable drives that definitely will not clear until the
> >>>>administrator takes some action.  It's usually a reasonable
> >>>>approximation to call transient issues warnings, and permanent
> >>>>issues errors.
> >>>   So you can have events like FS_UNAVAILABLE and FS_AVAILABLE but what use
> >>>would this have? I wouldn't like the interface to be dumping ground for
> >>>random crap - we have dmesg for that :).
> >>In that case I'm confused -- why would ENOSPC be an appropriate use
> >>of this interface if the mount being entirely blocked would be
> >>inappropriate?  Isn't being unable to service any I/O a more
> >>fundamental and severe thing than being up and healthy but full?
> >>
> >>Were you intending the interface to be exclusively for data
> >>integrity issues like checksum failures, rather than more general
> >>events about a mount that userspace would probably like to know
> >>about?
> >   Well, I'm not saying we cannot have those events for fs availability /
> >inavailability. I'm just saying I'd like to see some use for that first.
> >I don't want events to be added just because it's possible...
> >
> >For ENOSPC we have thin provisioned storage and the userspace deamon
> >shuffling real storage underneath. So there I know the usecase.
> >
> The use-case that immediately comes to mind for me would be diskless
> nodes with root-on-nfs needing to know if they can actually access
> the root filesystem.
  Well, most apps will access the root file system regardless of what we
send over netlink... So I don't see netlink events improving the situation
there too much. You could try to use it for something like failover but
even there I'm not too convinced - just doing some IO, waiting for timeout,
and failing over if IO doesn't complete works just fine for that these
days. That's why I was asking because I didn't see convincing usecase
myself...

								Honza
-- 
Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org>
SUSE Labs, CR

Re: [RFC 1/4] fs: Add generic file system event notifications

[John Spray]

On 17/04/2015 17:22, Jan Kara wrote:
> On Fri 17-04-15 17:08:10, John Spray wrote:
>> On 17/04/2015 16:43, Jan Kara wrote:
>> In that case I'm confused -- why would ENOSPC be an appropriate use
>> of this interface if the mount being entirely blocked would be
>> inappropriate?  Isn't being unable to service any I/O a more
>> fundamental and severe thing than being up and healthy but full?
>>
>> Were you intending the interface to be exclusively for data
>> integrity issues like checksum failures, rather than more general
>> events about a mount that userspace would probably like to know
>> about?
>    Well, I'm not saying we cannot have those events for fs availability /
> inavailability. I'm just saying I'd like to see some use for that first.
> I don't want events to be added just because it's possible...
>
> For ENOSPC we have thin provisioned storage and the userspace deamon
> shuffling real storage underneath. So there I know the usecase.
>

Ah, OK.  So I can think of a couple of use cases:
  * a cluster scheduling service (think MPI jobs or docker containers) 
might check for events like this.  If it can see the cluster filesystem 
is unavailable, then it can avoid scheduling the job, so that the 
(multi-node) application does not get hung on one node with a bad 
mount.  If it sees a mount go bad (unavailable, or client evicted) 
partway through a job, then it can kill -9 the process that was relying 
on the bad mount, and go run it somewhere else.
  * Boring but practical case: a nagios health check for checking if 
mounts are OK.

We don't have to invent these event types now of course, but something 
to bear in mind.  Hopefully if/when any of the distributed filesystems 
(Lustre/Ceph/etc) choose to implement this, we can look at making the 
event types common at that time though.

BTW in any case an interface for filesystem events to userspace will be 
a useful addition, thank you!

Cheers,
John

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC 1/4] fs: Add generic file system event notifications

[Andreas Dilger]

On Apr 17, 2015, at 11:37 AM, John Spray <john.spray@redhat.com> wrote:
> On 17/04/2015 17:22, Jan Kara wrote:
>> On Fri 17-04-15 17:08:10, John Spray wrote:
>>> On 17/04/2015 16:43, Jan Kara wrote:
>>> In that case I'm confused -- why would ENOSPC be an appropriate use
>>> of this interface if the mount being entirely blocked would be
>>> inappropriate?  Isn't being unable to service any I/O a more
>>> fundamental and severe thing than being up and healthy but full?
>>> 
>>> Were you intending the interface to be exclusively for data
>>> integrity issues like checksum failures, rather than more general
>>> events about a mount that userspace would probably like to know
>>> about?
>>   Well, I'm not saying we cannot have those events for fs availability /
>> inavailability. I'm just saying I'd like to see some use for that first.
>> I don't want events to be added just because it's possible...
>> 
>> For ENOSPC we have thin provisioned storage and the userspace deamon
>> shuffling real storage underneath. So there I know the usecase.
>> 
> 
> Ah, OK.  So I can think of a couple of use cases:
> * a cluster scheduling service (think MPI jobs or docker containers) might check for events like this.  If it can see the cluster filesystem is unavailable, then it can avoid scheduling the job, so that the (multi-node) application does not get hung on one node with a bad mount.  If it sees a mount go bad (unavailable, or client evicted) partway through a job, then it can kill -9 the process that was relying on the bad mount, and go run it somewhere else.
> * Boring but practical case: a nagios health check for checking if mounts are OK.

John,
thanks for chiming in, as I was just about to write the same.  Some users
were just asking yesterday at the Lustre User Group meeting about adding
an interface to notify job schedulers for your #1 point, and I'd much
rather use a generic interface than inventing our own for Lustre.

Cheers, Andreas

> We don't have to invent these event types now of course, but something to bear in mind.  Hopefully if/when any of the distributed filesystems (Lustre/Ceph/etc) choose to implement this, we can look at making the event types common at that time though.
> 
> BTW in any case an interface for filesystem events to userspace will be a useful addition, thank you!
> 
> Cheers,
> John


Cheers, Andreas





--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC 1/4] fs: Add generic file system event notifications

[Beata Michalska]

On 04/17/2015 06:08 PM, John Spray wrote:
> 
> On 17/04/2015 16:43, Jan Kara wrote:
>> On Fri 17-04-15 15:51:14, John Spray wrote:
>>> On 17/04/2015 14:23, Austin S Hemmelgarn wrote:
>>>
>>>> For some filesystems, it may make sense to differentiate between a
>>>> generic warning and an error.  For BTRFS and ZFS for example, if
>>>> there is a csum error on a block, this will get automatically
>>>> corrected in many configurations, and won't require anything like
>>>> fsck to be run, but monitoring applications will still probably
>>>> want to be notified.
>>> Another key differentiation IMHO is between transient errors (like
>>> server is unavailable in a distributed filesystem) that will block
>>> the filesystem but might clear on their own, vs. permanent errors
>>> like unreadable drives that definitely will not clear until the
>>> administrator takes some action.  It's usually a reasonable
>>> approximation to call transient issues warnings, and permanent
>>> issues errors.
>>    So you can have events like FS_UNAVAILABLE and FS_AVAILABLE but what use
>> would this have? I wouldn't like the interface to be dumping ground for
>> random crap - we have dmesg for that :).
> In that case I'm confused -- why would ENOSPC be an appropriate use of this interface if the mount being entirely blocked would be inappropriate?  Isn't being unable to service any I/O a more fundamental and severe thing than being up and healthy but full?
> 
> Were you intending the interface to be exclusively for data integrity issues like checksum failures, rather than more general events about a mount that userspace would probably like to know about?
> 
> John
> 

I think we should support both and leave the decision on what
is to be reported or not to particular file systems keeping it
to a reasonable extent, of course. The interface should hand it over
to user space - acting as a go-between. I would though avoid
any filesystem specific events (when it comes to specifying those),
keeping it as generic as possible.


BR
Beata