how to best map/buffer a file to memory by being able to lock sequences thread-safe

how to best map/buffer a file to memory by being able to lock sequences thread-safe

[Dennis Heuer]

hello

i tried to write a memory cache for a file and keep it in sync with the
file while threads or foreign processes are writing safely to it. then
i found mmap and lockf but the both don't work together. i studied all
the other alternatives i found, like fcntl or flockfile, but they
don't support the one or the other action. for example, fcntl locks are
not based on a per-thread base. flockfile only works on the full FILE
object, etc. also, if i don't use mmap, as far as i can see, i have to
map the file to a buffer myself, which causes sync-problems especially
with foreign processes.

how can i reach the same level of features and comfort like with lockf
and mmap without one of the both and by staying posix-compliant (at
least)?

by the way: can i lock sections of a plain memory buffer (mmap'ed or
allocated with malloc or whatever) like i can do with fcntl or lockf
to have threads not running over eachother?

regards,
dennis heuer

Re: how to best map/buffer a file to memory by being able to lock sequences thread-safe

[Glynn Clements]

Dennis Heuer wrote:

> i tried to write a memory cache for a file and keep it in sync with the
> file while threads or foreign processes are writing safely to it. then
> i found mmap and lockf but the both don't work together. i studied all
> the other alternatives i found, like fcntl or flockfile, but they
> don't support the one or the other action. for example, fcntl locks are
> not based on a per-thread base. flockfile only works on the full FILE
> object, etc. also, if i don't use mmap, as far as i can see, i have to
> map the file to a buffer myself, which causes sync-problems especially
> with foreign processes.
> 
> how can i reach the same level of features and comfort like with lockf
> and mmap without one of the both and by staying posix-compliant (at
> least)?

One option is to create another file of the appropriate size, and
obtain locks on that file instead of the mapped file. If the file is
sparse (i.e. is enlarged using ftruncate rather than by having data
written to it), it won't use significant disk space.

> by the way: can i lock sections of a plain memory buffer (mmap'ed or
> allocated with malloc or whatever) like i can do with fcntl or lockf
> to have threads not running over eachother?

You can implement locks for anything using semaphore sets, condition
variables, mutexes etc.

The only way that file locks are really any different is that they let
you effectively have a separate lock for each byte of the file without
the overhead of actually having a separate lock for each byte.

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: how to best map/buffer a file to memory by being able to lock sequences thread-safe

[Dennis Heuer]

i did not try obstacks yet, will have a look at them. what bothers me
at the moment is that mremap() is not a standard and that it operates
strangely to me:

ENOMEM
    The region is private writable, and insufficient virtual memory is
available to extend it. Also, this error will occur if MREMAP_MAYMOVE
is not given and the extension would collide with another mapped region.

i thought that mmap'ed space is only virtually aligned. possibly it's
better to write buffers oneself? the posix interfaces look strange to
me. they seem halfhearted and weirdly incompatible. consider a file
that is growing. how to best remap it?

regards,
dennis heuer

Re: how to best map/buffer a file to memory by being able to lock sequences thread-safe

[Glynn Clements]

Dennis Heuer wrote:

> i did not try obstacks yet, will have a look at them. what bothers me
> at the moment is that mremap() is not a standard and that it operates
> strangely to me:
> 
> ENOMEM
>     The region is private writable, and insufficient virtual memory is
> available to extend it. Also, this error will occur if MREMAP_MAYMOVE
> is not given and the extension would collide with another mapped region.
> 
> i thought that mmap'ed space is only virtually aligned. possibly it's
> better to write buffers oneself? the posix interfaces look strange to
> me. they seem halfhearted and weirdly incompatible.

I don't understand what you're saying in any of the preceding
paragraph.

> consider a file that is growing. how to best remap it?

If you don't need portability, use mremap() with MREMAP_MAYMOVE. If
you need portability, use munmap() and mmap().

The only advantage of mremap() (other than efficiency) is that it can
be used to remap private or anonymous mappings, while mmap/munmap will
only work if changes are written back to the file.

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: how to best map/buffer a file to memory by being able to lock sequences thread-safe

[Dennis Heuer]

On Tue, 16 Oct 2007 11:54:01 +0100
Glynn Clements <glynn@gclements.plus.com> wrote:

> 
> Dennis Heuer wrote:
> 
> > i did not try obstacks yet, will have a look at them. what bothers me
> > at the moment is that mremap() is not a standard and that it operates
> > strangely to me:
> > 
> > ENOMEM
> >     The region is private writable, and insufficient virtual memory is
> > available to extend it. Also, this error will occur if MREMAP_MAYMOVE
> > is not given and the extension would collide with another mapped region.
> > 
> > i thought that mmap'ed space is only virtually aligned. possibly it's
> > better to write buffers oneself? the posix interfaces look strange to
> > me. they seem halfhearted and weirdly incompatible.
> 
> I don't understand what you're saying in any of the preceding
> paragraph.

i actually don't understand why you don't understand :( possibly i
should've written 'caches' instead of 'buffers'. however, what shall i
use a posix interface for if it only works in this but not in that case
and then only inefficient, as you yourself mentioned. efficiency is
quite important. portability too. the documentation is also bad. one
never knows what's meant with 'process', if things are thread-safe
or how they behave in combination (if that's not strictly forbidden.)
posix is an assortment but not a structure or even a base. however, i
found out that some interfaces aren't supported by cygwin and will
write the file cache and my memory buffers myself based on malloc, even
if that means that i now have to implement an efficient tree. it also
means that one always again invents the wheel though the system already
has it implemented but doesn't like to offer it a sane way. sorry but
i'm quite pissed about posix at the moment.

regards,
dennis heuer

Re: how to best map/buffer a file to memory by being able to lock sequences thread-safe

[Glynn Clements]

Dennis Heuer wrote:

> > > i did not try obstacks yet, will have a look at them. what bothers me
> > > at the moment is that mremap() is not a standard and that it operates
> > > strangely to me:
> > > 
> > > ENOMEM
> > >     The region is private writable, and insufficient virtual memory is
> > > available to extend it. Also, this error will occur if MREMAP_MAYMOVE
> > > is not given and the extension would collide with another mapped region.
> > > 
> > > i thought that mmap'ed space is only virtually aligned. possibly it's
> > > better to write buffers oneself? the posix interfaces look strange to
> > > me. they seem halfhearted and weirdly incompatible.
> > 
> > I don't understand what you're saying in any of the preceding
> > paragraph.
> 
> i actually don't understand why you don't understand :(

Well, let's go over it a sentence at a time.

> i thought that mmap'ed space is only virtually aligned

What does "virtually aligned" mean? I've never heard the term before.

> possibly it's better to write buffers oneself?

Why? I mean, there are all sorts of reasons why you might want to use
user-space buffers, but it's not possible to comment on why they may
or may not be better without understanding which problem(s) you are
facing with mmap.

> the posix interfaces look strange to me. they seem halfhearted and
> weirdly incompatible.

They (mmap and munmap) seem fairly rational to me. What do you find
odd about them?

> possibly i should've written 'caches' instead of 'buffers'.

That wouldn't make it any more clear.

> however, what shall i
> use a posix interface for if it only works in this but not in that case

It works for what it's supposed to work for: mapping a portion of a
file to the process' address space, as if it were memory.

What case doesn't it work for?

> and then only inefficient, as you yourself mentioned. efficiency is
> quite important.

The potential inefficiency of munmap/mmap (vs mremap) is trivial
compared to the inefficiency of explicit read/write calls.

> portability too.

mmap/mmunmap are available on all POSIX systems.

> the documentation is also bad.

What is wrong with it?

> one
> never knows what's meant with 'process', if things are thread-safe
> or how they behave in combination (if that's not strictly forbidden.)

By "process", it means "process". All threads within a process share
the same address space. mmap maps a file into the process' address
space.

> posix is an assortment but not a structure or even a base. however, i
> found out that some interfaces aren't supported by cygwin

Cygwin supports mmap() in terms of the functionality specified by
POSIX, and provides some extensions (but not all of the ones which
Linux provides).

> and will
> write the file cache and my memory buffers myself based on malloc, even
> if that means that i now have to implement an efficient tree. it also
> means that one always again invents the wheel though the system already
> has it implemented but doesn't like to offer it a sane way.

What do you think that read/write will offer that mmap() doesn't?

> sorry but i'm quite pissed about posix at the moment.

From what you've written, I suspect that the problem has more to do
with you not understanding the interface or not knowing how to use it
than with what is being provided.

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: how to best map/buffer a file to memory by being able to lock sequences thread-safe

[Bryan Christ]

I haven't tried it yet, but glib has a facility called GMappedFile which 
may suit your need.  That should be portable.

Glynn Clements wrote:
> Dennis Heuer wrote:
> 
>>>> i did not try obstacks yet, will have a look at them. what bothers me
>>>> at the moment is that mremap() is not a standard and that it operates
>>>> strangely to me:
>>>>
>>>> ENOMEM
>>>>     The region is private writable, and insufficient virtual memory is
>>>> available to extend it. Also, this error will occur if MREMAP_MAYMOVE
>>>> is not given and the extension would collide with another mapped region.
>>>>
>>>> i thought that mmap'ed space is only virtually aligned. possibly it's
>>>> better to write buffers oneself? the posix interfaces look strange to
>>>> me. they seem halfhearted and weirdly incompatible.
>>> I don't understand what you're saying in any of the preceding
>>> paragraph.
>> i actually don't understand why you don't understand :(
> 
> Well, let's go over it a sentence at a time.
> 
>> i thought that mmap'ed space is only virtually aligned
> 
> What does "virtually aligned" mean? I've never heard the term before.
> 
>> possibly it's better to write buffers oneself?
> 
> Why? I mean, there are all sorts of reasons why you might want to use
> user-space buffers, but it's not possible to comment on why they may
> or may not be better without understanding which problem(s) you are
> facing with mmap.
> 
>> the posix interfaces look strange to me. they seem halfhearted and
>> weirdly incompatible.
> 
> They (mmap and munmap) seem fairly rational to me. What do you find
> odd about them?
> 
>> possibly i should've written 'caches' instead of 'buffers'.
> 
> That wouldn't make it any more clear.
> 
>> however, what shall i
>> use a posix interface for if it only works in this but not in that case
> 
> It works for what it's supposed to work for: mapping a portion of a
> file to the process' address space, as if it were memory.
> 
> What case doesn't it work for?
> 
>> and then only inefficient, as you yourself mentioned. efficiency is
>> quite important.
> 
> The potential inefficiency of munmap/mmap (vs mremap) is trivial
> compared to the inefficiency of explicit read/write calls.
> 
>> portability too.
> 
> mmap/mmunmap are available on all POSIX systems.
> 
>> the documentation is also bad.
> 
> What is wrong with it?
> 
>> one
>> never knows what's meant with 'process', if things are thread-safe
>> or how they behave in combination (if that's not strictly forbidden.)
> 
> By "process", it means "process". All threads within a process share
> the same address space. mmap maps a file into the process' address
> space.
> 
>> posix is an assortment but not a structure or even a base. however, i
>> found out that some interfaces aren't supported by cygwin
> 
> Cygwin supports mmap() in terms of the functionality specified by
> POSIX, and provides some extensions (but not all of the ones which
> Linux provides).
> 
>> and will
>> write the file cache and my memory buffers myself based on malloc, even
>> if that means that i now have to implement an efficient tree. it also
>> means that one always again invents the wheel though the system already
>> has it implemented but doesn't like to offer it a sane way.
> 
> What do you think that read/write will offer that mmap() doesn't?
> 
>> sorry but i'm quite pissed about posix at the moment.
> 
> From what you've written, I suspect that the problem has more to do
> with you not understanding the interface or not knowing how to use it
> than with what is being provided.
>