Fwd: Query regarding inotify and mmap-ed files

Fwd: Query regarding inotify and mmap-ed files

[Chinmay V S]

Hi All,

Recently i have implemented fairly simple IPC between two processes by
mmap-ing the same file in both.

To send an "event" from one process to the other i thought of using
msync() and inotify. I found that there is no inotify event that can
be triggered upon msync() on a mmap-ed file.

Luckily i was able to patch the kernel and add a new watch to inotify
that is triggered upon msync.

https://gist.github.com/TheCodeArtist/5874669

This is derived from a previous work by Simo Sorce(wayyy back in 2009)
http://osdir.com/ml/linux-kernel/2009-03/msg13455.html

This is NOT part of the mainline kernel yet.
Does this approach have any technical "gotchas"?
_OR_
is there a far simpler approach i am missing?

thanks & regards
ChinmayVS

Re: Query regarding inotify and mmap-ed files

[Peter Zijlstra]

On Thu, Jun 27, 2013 at 05:44:52PM +0800, Chinmay V S wrote:
> Hi All,
> 
> Recently i have implemented fairly simple IPC between two processes by
> mmap-ing the same file in both.
> 
> To send an "event" from one process to the other i thought of using msync()
> and inotify. I found that there is no inotify event that can be triggered
> upon msync() on a mmap-ed file.
> 
> Luckily i was able to patch the kernel and add a new watch to inotify that
> is triggered upon msync.
> 
> https://gist.github.com/TheCodeArtist/5874669
> 
> This is derived from a previous work by Simo Sorce(wayyy back in 2009)
> http://osdir.com/ml/linux-kernel/2009-03/msg13455.html
> 
> This is NOT part of the mainline kernel yet.
> Does this approach have any technical "gotchas"?
> _OR_
> is there a far simpler approach i am missing?

What's wrong with the myriad of socket based options? Use a named pipe or unix
socket?

Re: Query regarding inotify and mmap-ed files

[Chinmay V S]

Hi All,

On Thu, Jun 27, 2013 at 6:47 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Thu, Jun 27, 2013 at 05:44:52PM +0800, Chinmay V S wrote:
>> Hi All,
>>
>> Recently i have implemented fairly simple IPC between two processes by
>> mmap-ing the same file in both.
>>
>> To send an "event" from one process to the other i thought of using msync()
>> and inotify. I found that there is no inotify event that can be triggered
>> upon msync() on a mmap-ed file.
>>
>> Luckily i was able to patch the kernel and add a new watch to inotify that
>> is triggered upon msync.
>>
>> https://gist.github.com/TheCodeArtist/5874669
>>
>> This is derived from a previous work by Simo Sorce(wayyy back in 2009)
>> http://osdir.com/ml/linux-kernel/2009-03/msg13455.html
>>
>> This is NOT part of the mainline kernel yet.
>> Does this approach have any technical "gotchas"?
>> _OR_
>> is there a far simpler approach i am missing?
>
> What's wrong with the myriad of socket based options? Use a named pipe or unix
> socket?
>

The system design requirements are as follows:
a. Single data "writer" process that acquires the data from peripheral hardware.
b. Many "reader" processes. Each needs to be notified of every single
event. i.e. each time the writer obtains new set of values, the set
needs to be broadcast  and each "reader" needs to obtain a copy.
c. Writer and reader process are not parent/child and no
pre-determined order of execution.
d. File-backed for system-stability. Events may NOT be dropped. Even
if any process crashes, the executable will be launched again and it
should be able to continue processing from the last event.

stackoverflow.com/questions/17123108/notify-signal-when-memory-mapped-file-modified

Considering all these limitations (trust me it IS NOT a contrived
example, but an actual requirement on this particular embedded device
design i am working on currently), implementing IPC using an mmap-ed
file across all the processes is working pretty well so far.

I was surprised that its would be so difficult to implement such a
design. Then again i thought i might be missing some very obvious
simpler way to achieve the same.

[1] In either case, please let me know any prior
experiences/suggestions and design flaws that you can see.

[2] Also considering that inotify is *THE* "goto framework" for
monitoring the filesystem, would the patch to add support to inotify
for tracking msync() mmap-ed files be mainlined?
https://gist.github.com/TheCodeArtist/5874669
(if so, would be more than happy to rework it for the latest Linux
Kernel version)

regards
ChinmayVS

Re: Query regarding inotify and mmap-ed files

[Al Viro]

On Thu, Jun 27, 2013 at 10:22:35PM +0800, Chinmay V S wrote:

> [2] Also considering that inotify is *THE* "goto framework" for
> monitoring the filesystem,

inotify is a mistake, with rather serious design restrictions on
possible uses.  Don't step into it...

Re: Query regarding inotify and mmap-ed files

[Chinmay V S]

On Fri, Jun 28, 2013 at 4:15 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Thu, Jun 27, 2013 at 10:22:35PM +0800, Chinmay V S wrote:
>
>> [2] Also considering that inotify is *THE* "goto framework" for
>> monitoring the filesystem,
>
> inotify is a mistake, with rather serious design restrictions on
> possible uses.  Don't step into it...

*sigh* ok Al, i wish i had i had posted this query earlier, i really
do. Inotify seemed a perfect solution for the system i had to design
a. Single data "writer".
b. Notification to all/any "reader" process on a write.
c. Writer and reader process are not parent/child and no
pre-determined order of execution.
d. Events may NOT be dropped.

And so far it works well for me too. Just that the kernel needs to be
patched with support for IN_SYNC event in inotify that is triggered
upon msync() upon a mmap-ed file.

The only problem i have seen is that during stress testing the system
(ARM Cortex A8 1Ghz, rootfs on NAND, kernel v2.6.37+inotify_patch) to
~100% cpu usage, and firing msync() events as fast as i can in the
writer(>1000/s), consecutive events will get coalesced quite
frequently before a reader is notified, i.e. one of the prior events
is effectively dropped.

The solution i have worked out for that as follows:
1. Mmap same file in both writer and reader maintain a circular buffer
of data as a shared region.
2. Writer writes a record into the circular buffer and updates the
"head" (a field in the shared region).
3. Writer performs an msync()
4. Readers receive an inotify event (running patched kernel with
IN_SYNC support).
5. Readers check "head" in shared region and read all data from the
last head value(tracked locally in the reader process) to the current
head.
6. Update local head value to the value of the latest "head" in the
shared region.

While all this may sound a tad complicated, its all wrapped-up into
10KB of C code (and one kernel patch).
Patch for reference: https://gist.github.com/TheCodeArtist/5874669

But i just could not sit in peace knowing that this inotify patch has
NOT been accepted into the mainline, indicating some serious issues
with this approach. I would really appreciate it if you(or anyone else
for that matter) could please mention/elaborate *ANY* potential
issues.

thanks and regards
ChinmayVS

Re: Query regarding inotify and mmap-ed files

[Jan Kara]

On Thu 27-06-13 22:22:35, Chinmay V S wrote:
> Hi All,
> 
> On Thu, Jun 27, 2013 at 6:47 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> > On Thu, Jun 27, 2013 at 05:44:52PM +0800, Chinmay V S wrote:
> >> Hi All,
> >>
> >> Recently i have implemented fairly simple IPC between two processes by
> >> mmap-ing the same file in both.
> >>
> >> To send an "event" from one process to the other i thought of using msync()
> >> and inotify. I found that there is no inotify event that can be triggered
> >> upon msync() on a mmap-ed file.
> >>
> >> Luckily i was able to patch the kernel and add a new watch to inotify that
> >> is triggered upon msync.
> >>
> >> https://gist.github.com/TheCodeArtist/5874669
> >>
> >> This is derived from a previous work by Simo Sorce(wayyy back in 2009)
> >> http://osdir.com/ml/linux-kernel/2009-03/msg13455.html
> >>
> >> This is NOT part of the mainline kernel yet.
> >> Does this approach have any technical "gotchas"?
> >> _OR_
> >> is there a far simpler approach i am missing?
> >
> > What's wrong with the myriad of socket based options? Use a named pipe or unix
> > socket?
> >
> 
> The system design requirements are as follows:
> a. Single data "writer" process that acquires the data from peripheral hardware.
> b. Many "reader" processes. Each needs to be notified of every single
> event. i.e. each time the writer obtains new set of values, the set
> needs to be broadcast  and each "reader" needs to obtain a copy.
> c. Writer and reader process are not parent/child and no
> pre-determined order of execution.
> d. File-backed for system-stability. Events may NOT be dropped. Even
> if any process crashes, the executable will be launched again and it
> should be able to continue processing from the last event.
> 
> stackoverflow.com/questions/17123108/notify-signal-when-memory-mapped-file-modified
> 
> Considering all these limitations (trust me it IS NOT a contrived
> example, but an actual requirement on this particular embedded device
> design i am working on currently), implementing IPC using an mmap-ed
> file across all the processes is working pretty well so far.
> 
> I was surprised that its would be so difficult to implement such a
> design. Then again i thought i might be missing some very obvious
> simpler way to achieve the same.
> 
> [1] In either case, please let me know any prior
> experiences/suggestions and design flaws that you can see.
> 
> [2] Also considering that inotify is *THE* "goto framework" for
> monitoring the filesystem, would the patch to add support to inotify
> for tracking msync() mmap-ed files be mainlined?
> https://gist.github.com/TheCodeArtist/5874669
> (if so, would be more than happy to rework it for the latest Linux
> Kernel version)
  As Al wrote, inotify has its problems (e.g. if you want to watch lots of
files, when you want some reliability across operations such as rename
etc.) so he is reluctant to extend it. In your case it can work OK though.

But why can't you use current IN_MODIFY event?

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: Query regarding inotify and mmap-ed files

[Chinmay V S]

> But why can't you use current IN_MODIFY event?
>

Initially i registered a "catch all" inotify watch on the mmap-ed file
by calling inotify_add_watch() with IN_ALL_EVENTS in the "reader"
process. But modifications to the file by mmap-ing it in the "writer"
process did NOT trigger the inotify watch. Even calling msync() did
NOT trigger any inotify events.

(BTW, Is this expected behaviour?? v2.6.37 on ARMCortexA8, rootfs on NAND)

The file was mapped using MAP_SHARED to ensure that no copies were
made in memory and the changes made by the "writer" process are being
flushed to the file.

Hence added support for IN_SYNC event in the inotify framework.
https://gist.github.com/TheCodeArtist/5874669

IMHO, this is a well justified addition to the inotify mechanism.
Since it simplifies the creation of an event notification framework
for IPC, is any reason why IN_SYNC should *NOT* be mainlined?

PS: Once again, i would really really welcome any suggestions to
implement a simple high performance notification mechanism for IPC
using any existing alternatives. Running a "patched" kernel is
something i would like to avoid.

System requirements:
a. Single data "writer" process.
b. Multiple "reader" processes, each to be notified of every update by
the writer.
c. Processes "writer" and "readers" NOT related. No per-determined
order of execution.
d. Events may NOT be dropped.

regards
ChinmayVS

Re: Query regarding inotify and mmap-ed files

[Jan Kara]

On Mon 01-07-13 22:08:28, Chinmay V S wrote:
> > But why can't you use current IN_MODIFY event?
> >
> 
> Initially i registered a "catch all" inotify watch on the mmap-ed file
> by calling inotify_add_watch() with IN_ALL_EVENTS in the "reader"
> process. But modifications to the file by mmap-ing it in the "writer"
> process did NOT trigger the inotify watch. Even calling msync() did
> NOT trigger any inotify events.
  Ah right. If you wrote the data to the file using write(2), you'd see
IN_MODIFY events.

> (BTW, Is this expected behaviour?? v2.6.37 on ARMCortexA8, rootfs on NAND)
> 
> The file was mapped using MAP_SHARED to ensure that no copies were
> made in memory and the changes made by the "writer" process are being
> flushed to the file.
> 
> Hence added support for IN_SYNC event in the inotify framework.
> https://gist.github.com/TheCodeArtist/5874669
> 
> IMHO, this is a well justified addition to the inotify mechanism.
> Since it simplifies the creation of an event notification framework
> for IPC, is any reason why IN_SYNC should *NOT* be mainlined?
  There are more places that can sync a file (e.g. sync_file_range() or it
is disputable whether sync(2) should generate the event or not). Generally
it seems to me that you just need to broadcast an event notification to a
group of processes and the fact that you use inotify is a bit arbitrary.

> PS: Once again, i would really really welcome any suggestions to
> implement a simple high performance notification mechanism for IPC
> using any existing alternatives. Running a "patched" kernel is
> something i would like to avoid.
> 
> System requirements:
> a. Single data "writer" process.
> b. Multiple "reader" processes, each to be notified of every update by
> the writer.
> c. Processes "writer" and "readers" NOT related. No per-determined
> order of execution.
> d. Events may NOT be dropped.
  Well, you could create eventfd descriptor for each reader in the writer,
pass it to each reader via Unix domain sockets and use that for
notification. Passing of actual data can still be done via mmap. But I
agree that it's likely more code than just using inotify.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR