Concerning a post that you made about expandable anonymous shared mappings

Concerning a post that you made about expandable anonymous shared mappings

[William Tambe]

I read a post that you made about not being able to expand anonymous 
shared mapping with mremap(). And I am actually having that issue now.

You made the post in 2004 and we are now in 2007. I would like to know 
if that feature was added because the code below always fail with bus 
error on my machine. I use glibc 2.5

Thank you for helping.

#define _GNU_SOURCE
#include <sys/mman.h>
#include <unistd.h>

#include <stdio.h>

main() {
         void *ptr;
         if ((ptr=mmap(0, 4096, PROT_READ|PROT_WRITE,
                 MAP_ANONYMOUS|MAP_SHARED|MAP_GROWSDOWN, 0, 0)) == -1) {
                 printf("failed to mmap\n");
                 return;
         }

         if ((ptr=mremap(ptr, 4096, 8192, MREMAP_MAYMOVE)) == -1) {
                 printf("failed to mremap\n");
                 return;
         }

         //why does this failed. I am well in the interval [4096, 8192]
         *(unsigned int *)(ptr + 4096 + 8)= 10;
}

Re: Concerning a post that you made about expandable anonymous shared mappings

[Hugh Dickins]

On Fri, 29 Jun 2007, William Tambe wrote:

> I read a post that you made about not being able to expand anonymous shared
> mapping with mremap(). And I am actually having that issue now.

I guess you're referring to the thread at
http://lkml.org/lkml/2004/6/16/155
and you're asking either Stas or me.

> 
> You made the post in 2004 and we are now in 2007. I would like to know if that
> feature was added because the code below always fail with bus error on my
> machine. I use glibc 2.5

You've answered your own question: we did not make the change Stas
suggested, IIRC because I remained a little uneasy with that change
in behaviour, and nobody else spoke up for it.

I haven't given it any thought since then:
do you have a good case for us to reconsider it?

Hugh

> 
> Thank you for helping.
> 
> #define _GNU_SOURCE
> #include <sys/mman.h>
> #include <unistd.h>
> 
> #include <stdio.h>
> 
> main() {
>         void *ptr;
>         if ((ptr=mmap(0, 4096, PROT_READ|PROT_WRITE,
>                 MAP_ANONYMOUS|MAP_SHARED|MAP_GROWSDOWN, 0, 0)) == -1) {
>                 printf("failed to mmap\n");
>                 return;
>         }
> 
>         if ((ptr=mremap(ptr, 4096, 8192, MREMAP_MAYMOVE)) == -1) {
>                 printf("failed to mremap\n");
>                 return;
>         }
> 
>         //why does this failed. I am well in the interval [4096, 8192]
>         *(unsigned int *)(ptr + 4096 + 8)= 10;
> }

Re: Concerning a post that you made about expandable anonymous shared mappings

[William Tambe]

Yes, I have a good case, but my case may not sound interesting until you 
see it working.

Ok, I am developing a dynamic memory allocation routine which takes 
direct advantage of the ability of a machine to use Virtual Memory to 
make everything look contiguous and fast.

And it just doesn't make sens to have mmap() map ANONYMOUS shared memory 
and mremap() not to expand it and make the expanded area available.

I am not a 100% sure, but I don't think correcting its behavior would 
cause a problem.

Would you happen to know how I can work around that issue for now, and 
make writing in an expended area not to generate a Bus error?

Sincerely,
William Tambe


Hugh Dickins wrote:
> On Fri, 29 Jun 2007, William Tambe wrote:
> 
>> I read a post that you made about not being able to expand anonymous shared
>> mapping with mremap(). And I am actually having that issue now.
> 
> I guess you're referring to the thread at
> http://lkml.org/lkml/2004/6/16/155
> and you're asking either Stas or me.
> 
>> You made the post in 2004 and we are now in 2007. I would like to know if that
>> feature was added because the code below always fail with bus error on my
>> machine. I use glibc 2.5
> 
> You've answered your own question: we did not make the change Stas
> suggested, IIRC because I remained a little uneasy with that change
> in behaviour, and nobody else spoke up for it.
> 
> I haven't given it any thought since then:
> do you have a good case for us to reconsider it?
> 
> Hugh
> 
>> Thank you for helping.
>>
>> #define _GNU_SOURCE
>> #include <sys/mman.h>
>> #include <unistd.h>
>>
>> #include <stdio.h>
>>
>> main() {
>>         void *ptr;
>>         if ((ptr=mmap(0, 4096, PROT_READ|PROT_WRITE,
>>                 MAP_ANONYMOUS|MAP_SHARED|MAP_GROWSDOWN, 0, 0)) == -1) {
>>                 printf("failed to mmap\n");
>>                 return;
>>         }
>>
>>         if ((ptr=mremap(ptr, 4096, 8192, MREMAP_MAYMOVE)) == -1) {
>>                 printf("failed to mremap\n");
>>                 return;
>>         }
>>
>>         //why does this failed. I am well in the interval [4096, 8192]
>>         *(unsigned int *)(ptr + 4096 + 8)= 10;
>> }
> 

Re: Concerning a post that you made about expandable anonymous shared mappings

[Stas Sergeev]

Hello.

William Tambe wrote:
> And it just doesn't make sens to have mmap() map ANONYMOUS shared memory 
> and mremap() not to expand it and make the expanded area available.
I agree with this, but the argument against
that approach was that then you can only
enlarge the backing-store, but never shrink.
I personally think it is a valid argument,
even though the problem is probably not very
important.
Also, you can't expand the SysV SHM with mremap
just as well - it will give you a SIGBUS too IIRC.
So for that discussion of 2004, I lost the
battle and was convinced that the proposed
approach is not very good...

> Would you happen to know how I can work around that issue for now, and 
> make writing in an expended area not to generate a Bus error?
Have you tried the Posix SHM instead? It works
very well for me. Back in 2004 the glibc had
bugs, so I couldn't easily use posix shm and
was thinking about the different approaches.
But now it should suffice.

Re: Concerning a post that you made about expandable anonymous shared mappings

[Stas Sergeev]

Hi.

Hugh Dickins wrote:
> You've answered your own question: we did not make the change Stas
> suggested, IIRC because I remained a little uneasy with that change
> in behaviour, and nobody else spoke up for it.
IIRC your argument, that made sense to me,
was that with such an approach, you can only
expand the backing-store, but never shrink.
I agree this is a problem from some point of
view. I have no idea whether it is important
or not, but it definitely _looks_ not very good.

>>         //why does this failed. I am well in the interval [4096, 8192]
>>         *(unsigned int *)(ptr + 4096 + 8)= 10;
>> }
Well, this generates a bus error, while, for
example, doing the same trick with having a
/dev/mem as a backing-store, simply maps the
"enlarged" space with the anonymous memory,
and so does not generate a SIGBUS (not producing
a desired effect either, of course).
Why do we have it both ways? Shouldn't they
(i.e. /dev/zero and /dev/mem mapping) behave
the same after expanding with mremap?

> I haven't given it any thought since then:
I was thinking about it a bit, and I imagined
we need something like
    int mopen(void *addr, size_t len, unsigned int flags);
which will give you an fd for the memory area,
which you can then ftruncate() and mmap() (and
mremap).
It looks like vmsplice() is a very close one,
but unfortunately it involves pipe, which will
not give you an ability to ftruncate() I suppose.
I even asked Jens Axboe about the possibility
to have mopen(). He said it might be a good
optimization (having only one syscall whereas
now 2 are needed (pipe/vmsplice)), but not
worth an efforts.
Now, if maybe someone have a time and patience,
he can explain me what was an advantage of using
pipe with vmsplice() at all. Why it was not
possible to have something like the mopen() above,
that will give you an fd right away, without a
pipe, so that ftruncate/mmap/lseek etc can be
used on it? Well, I guess this was discussed
many times, but I failed to google anything
relevant...
Such a thing could solve that mremap() problem
that me and William Tambe have.

Re: Concerning a post that you made about expandable anonymous shared mappings

[Hugh Dickins]

On Mon, 2 Jul 2007, Stas Sergeev wrote:
> Hugh Dickins wrote:
> > You've answered your own question: we did not make the change Stas
> > suggested, IIRC because I remained a little uneasy with that change
> > in behaviour, and nobody else spoke up for it.
> IIRC your argument, that made sense to me,
> was that with such an approach, you can only
> expand the backing-store, but never shrink.
> I agree this is a problem from some point of
> view. I have no idea whether it is important
> or not, but it definitely _looks_ not very good.

You were gracious enough to accept my arguments back then, but after
mulling this over overnight, I've come to think I was just too timid
back then, and gave too much weight to the issue of there being no
shrink, and to the issue that child might expand the object without
parent knowing (that surplus remaining allocated until both exited).

I've come right around to your original view, Stas, and William's:
if that mmap creates such an object, then the expanding mremap really
ought to be useful, and allow the underlying object to be expanded.
The shared anonymous object is already anomalous: expanding it on
fault makes it more consistent with its own nature, not less.

> 
> > >         //why does this failed. I am well in the interval [4096, 8192]
> > >         *(unsigned int *)(ptr + 4096 + 8)= 10;
> >> }
> Well, this generates a bus error, while, for
> example, doing the same trick with having a
> /dev/mem as a backing-store, simply maps the
> "enlarged" space with the anonymous memory,
> and so does not generate a SIGBUS (not producing
> a desired effect either, of course).
> Why do we have it both ways? Shouldn't they
> (i.e. /dev/zero and /dev/mem mapping) behave
> the same after expanding with mremap?

They've very different: mapping /dev/mem maps an existing object;
whereas mapping /dev/zero is a convention by which a new object
is created - and we can't afford the memory to make it of infinite
extent, so have limited it to the mapped size.

> 
> > I haven't given it any thought since then:
> I was thinking about it a bit, and I imagined
> we need something like
>    int mopen(void *addr, size_t len, unsigned int flags);
> which will give you an fd for the memory area,
> which you can then ftruncate() and mmap() (and
> mremap).

Ah, if we added an mopen(), there's no end to the discussions
we could have about what it can do.  It may be a great idea:
but it's really not needed to solve this particular little
problem.  As last time around, you were suggesting .mremap
callouts; but I much prefer your original shmem_zero_nopage,
which is a solution of the scale appropriate to the problem.

> Such a thing could solve that mremap() problem
> that me and William Tambe have.

If you're thinking of going that way, for this problem it
would be better just to stick with POSIX shm_open, as Christoph
suggested before, and you suggest to William in other mail.

But I now accept your view, that the shared anonymous object
is not behaving consistently in response to mremap, and would
like to put in a patch for that.  This isn't a particularly good
time for such a patch - it's unclear right now whether 2.6.23 will
have shmem_nopage like 2.6.22 or shmem_fault like 2.6.22-rc-mm;
but we can easily adjust to whichever.

Here's a patch against 2.6.22-rc7: would you, Stas, put your
Signed-off-by into this, and accept authorship - although I'm
sending this back to you, it's very much your idea, and only
trivially modified from your three-year-old patch by me.  If
you're agreeable, I can then forward it or its shmem_zero_fault
equivalent to Andrew when we see which way 2.6.23 is going.

[I'll fill in the patch comment later]

Signed-off-by: Hugh Dickins <hugh@veritas.com>
----

 mm/shmem.c |   41 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 40 insertions(+), 1 deletion(-)

--- 2.6.22-rc7/mm/shmem.c	2007-06-17 10:51:02.000000000 +0100
+++ linux/mm/shmem.c	2007-07-03 15:35:32.000000000 +0100
@@ -1320,6 +1320,36 @@ static struct page *shmem_nopage(struct 
 	return page;
 }
 
+struct page *shmem_zero_nopage(struct vm_area_struct *vma,
+				unsigned long address, int *type)
+{
+	struct page *page;
+
+	page = shmem_nopage(vma, address, type);
+	if (unlikely(page == NOPAGE_SIGBUS)) {
+		struct inode *inode = vma->vm_file->f_dentry->d_inode;
+		struct shmem_inode_info *info = SHMEM_I(inode);
+		loff_t vm_size, i_size;
+
+		/*
+		 * If mremap has been used to extend the vma,
+		 * now extend the underlying object to include this page.
+		 */
+		vm_size = (PAGE_ALIGN(address) - vma->vm_start) +
+				((loff_t)vma->vm_pgoff << PAGE_SHIFT);
+		spin_lock(&info->lock);
+		i_size = i_size_read(inode);
+		if (i_size < vm_size && vm_size <= SHMEM_MAX_BYTES &&
+		    !shmem_acct_size(info->flags, vm_size - i_size)) {
+			i_size_write(inode, vm_size);
+			inode->i_ctime = inode->i_mtime = CURRENT_TIME;
+		}
+		spin_unlock(&info->lock);
+		page = shmem_nopage(vma, address, type);
+	}
+	return page;
+}
+
 static int shmem_populate(struct vm_area_struct *vma,
 	unsigned long addr, unsigned long len,
 	pgprot_t prot, unsigned long pgoff, int nonblock)
@@ -2471,6 +2501,14 @@ static struct vm_operations_struct shmem
 #endif
 };
 
+static struct vm_operations_struct shmem_zero_vm_ops = {
+	.nopage		= shmem_zero_nopage,
+	.populate	= shmem_populate,
+#ifdef CONFIG_NUMA
+	.set_policy     = shmem_set_policy,
+	.get_policy     = shmem_get_policy,
+#endif
+};
 
 static int shmem_get_sb(struct file_system_type *fs_type,
 	int flags, const char *dev_name, void *data, struct vfsmount *mnt)
@@ -2599,6 +2637,7 @@ int shmem_zero_setup(struct vm_area_stru
 	if (vma->vm_file)
 		fput(vma->vm_file);
 	vma->vm_file = file;
-	vma->vm_ops = &shmem_vm_ops;
+	vma->vm_ops = &shmem_zero_vm_ops;
+	vma->vm_pgoff = 0;
 	return 0;
 }

Re: Concerning a post that you made about expandable anonymous shared mappings

[Stas Sergeev]

Hi.

Hugh Dickins wrote:
> You were gracious enough to accept my arguments back then, but after
> mulling this over overnight, I've come to think I was just too timid
> back then, and gave too much weight to the issue of there being no
> shrink, and to the issue that child might expand the object without
> parent knowing (that surplus remaining allocated until both exited).
I simply think that the anonymous shared mapping
is a bit of an ad-hoc interface. The result is that
whenever you extend it in some way, you get a problem
elsewhere. So I just stopped using it.

> The shared anonymous object is already anomalous:
Exactly.

> expanding it on
> fault makes it more consistent with its own nature, not less.
That's certainly true, you were agree with this even in
the past. It is more consistent to have it that way. The
only question is whether the side-effects you outlined,
do outweight the benefit, or not. But since it was you
who outlined the problems, I guess you have the right to
say that they, while being valid, do not outweight the benefit. :)
I am not going to oppose my own patch, of course.

> They've very different: mapping /dev/mem maps an existing object;
> whereas mapping /dev/zero is a convention by which a new object
> is created - and we can't afford the memory to make it of infinite
> extent, so have limited it to the mapped size.
OK. And I guess comparing /dev/mem mapping with
/dev/shm/xxx mapping is not valid too, since the
/dev/mem perhaps doesn't permit ftruncate().
So do you say, silently mapping the private anonymous
pages for /dev/mem is a correct behaveour, even though
the /proc/pid/maps shows that the mapping is shared?

> Ah, if we added an mopen(), there's no end to the discussions
> we could have about what it can do.  It may be a great idea:
> but it's really not needed to solve this particular little
> problem.
Well, I simply thought that something like this is
needed anyway, not only for that problem. But then
vmsplice() appeared, so of course such an mopen() is
no longer much relevant. I am just puzzled as to why
the pipe was used for vmsplice() and can't find an
answer, but I am probably missing something obvious...

>> Such a thing could solve that mremap() problem
>> that me and William Tambe have.
> If you're thinking of going that way, for this problem it
No, of course not, at least, not any more.
Back in 2004 it was a good idea, but now there is no
longer a vacant place for this wonderfull syscall. :)

> would be better just to stick with POSIX shm_open, as Christoph
> suggested before, and you suggest to William in other mail.
The only scenario I can image where you would prefer
the anonymous mmaps over the Posix SHM, is when you want
to manage many small memory areas separately, and do not
want to track them back to their fd's. But even then this
probably doesn't save more than a few dozens of code lines. :)

> But I now accept your view, that the shared anonymous object
> is not behaving consistently in response to mremap,
IIRC this view was never under any doubts. The discussion
was only whether or not to count a few problems you pointed.

> Here's a patch against 2.6.22-rc7: would you, Stas, put your
> Signed-off-by into this, and accept authorship - although I'm
> sending this back to you, it's very much your idea, and only
> trivially modified from your three-year-old patch by me.  If
> you're agreeable, I can then forward it or its shmem_zero_fault
> equivalent to Andrew when we see which way 2.6.23 is going.
OK, no problems. Except that I've already forgotten the
details, and pretty much lost the need for that patch.
So let's assume the authorship is yours. :)

> Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Stas Sergeev <stsp@aknet.ru>


----

 mm/shmem.c |   41 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 40 insertions(+), 1 deletion(-)

--- 2.6.22-rc7/mm/shmem.c	2007-06-17 10:51:02.000000000 +0100
+++ linux/mm/shmem.c	2007-07-03 15:35:32.000000000 +0100
@@ -1320,6 +1320,36 @@ static struct page *shmem_nopage(struct 
 	return page;
 }
 
+struct page *shmem_zero_nopage(struct vm_area_struct *vma,
+				unsigned long address, int *type)
+{
+	struct page *page;
+
+	page = shmem_nopage(vma, address, type);
+	if (unlikely(page == NOPAGE_SIGBUS)) {
+		struct inode *inode = vma->vm_file->f_dentry->d_inode;
+		struct shmem_inode_info *info = SHMEM_I(inode);
+		loff_t vm_size, i_size;
+
+		/*
+		 * If mremap has been used to extend the vma,
+		 * now extend the underlying object to include this page.
+		 */
+		vm_size = (PAGE_ALIGN(address) - vma->vm_start) +
+				((loff_t)vma->vm_pgoff << PAGE_SHIFT);
+		spin_lock(&info->lock);
+		i_size = i_size_read(inode);
+		if (i_size < vm_size && vm_size <= SHMEM_MAX_BYTES &&
+		    !shmem_acct_size(info->flags, vm_size - i_size)) {
+			i_size_write(inode, vm_size);
+			inode->i_ctime = inode->i_mtime = CURRENT_TIME;
+		}
+		spin_unlock(&info->lock);
+		page = shmem_nopage(vma, address, type);
+	}
+	return page;
+}
+
 static int shmem_populate(struct vm_area_struct *vma,
 	unsigned long addr, unsigned long len,
 	pgprot_t prot, unsigned long pgoff, int nonblock)
@@ -2471,6 +2501,14 @@ static struct vm_operations_struct shmem
 #endif
 };
 
+static struct vm_operations_struct shmem_zero_vm_ops = {
+	.nopage		= shmem_zero_nopage,
+	.populate	= shmem_populate,
+#ifdef CONFIG_NUMA
+	.set_policy     = shmem_set_policy,
+	.get_policy     = shmem_get_policy,
+#endif
+};
 
 static int shmem_get_sb(struct file_system_type *fs_type,
 	int flags, const char *dev_name, void *data, struct vfsmount *mnt)
@@ -2599,6 +2637,7 @@ int shmem_zero_setup(struct vm_area_stru
 	if (vma->vm_file)
 		fput(vma->vm_file);
 	vma->vm_file = file;
-	vma->vm_ops = &shmem_vm_ops;
+	vma->vm_ops = &shmem_zero_vm_ops;
+	vma->vm_pgoff = 0;
 	return 0;
 }

Re: Concerning a post that you made about expandable anonymous shared mappings

[William Tambe]

Hugh Dickins wrote:
> On Mon, 2 Jul 2007, Stas Sergeev wrote:
>> Hugh Dickins wrote:
>>> You've answered your own question: we did not make the change Stas
>>> suggested, IIRC because I remained a little uneasy with that change
>>> in behaviour, and nobody else spoke up for it.
>> IIRC your argument, that made sense to me,
>> was that with such an approach, you can only
>> expand the backing-store, but never shrink.
>> I agree this is a problem from some point of
>> view. I have no idea whether it is important
>> or not, but it definitely _looks_ not very good.
> 
> You were gracious enough to accept my arguments back then, but after
> mulling this over overnight, I've come to think I was just too timid
> back then, and gave too much weight to the issue of there being no
> shrink, and to the issue that child might expand the object without
> parent knowing (that surplus remaining allocated until both exited).
> 
> I've come right around to your original view, Stas, and William's:
> if that mmap creates such an object, then the expanding mremap really
> ought to be useful, and allow the underlying object to be expanded.
> The shared anonymous object is already anomalous: expanding it on
> fault makes it more consistent with its own nature, not less.
> 
>>>>         //why does this failed. I am well in the interval [4096, 8192]
>>>>         *(unsigned int *)(ptr + 4096 + 8)= 10;
>>>> }
>> Well, this generates a bus error, while, for
>> example, doing the same trick with having a
>> /dev/mem as a backing-store, simply maps the
>> "enlarged" space with the anonymous memory,
>> and so does not generate a SIGBUS (not producing
>> a desired effect either, of course).
>> Why do we have it both ways? Shouldn't they
>> (i.e. /dev/zero and /dev/mem mapping) behave
>> the same after expanding with mremap?
> 
> They've very different: mapping /dev/mem maps an existing object;
> whereas mapping /dev/zero is a convention by which a new object
> is created - and we can't afford the memory to make it of infinite
> extent, so have limited it to the mapped size.
> 
>>> I haven't given it any thought since then:
>> I was thinking about it a bit, and I imagined
>> we need something like
>>    int mopen(void *addr, size_t len, unsigned int flags);
>> which will give you an fd for the memory area,
>> which you can then ftruncate() and mmap() (and
>> mremap).
> 
> Ah, if we added an mopen(), there's no end to the discussions
> we could have about what it can do.  It may be a great idea:
> but it's really not needed to solve this particular little
> problem.  As last time around, you were suggesting .mremap
> callouts; but I much prefer your original shmem_zero_nopage,
> which is a solution of the scale appropriate to the problem.
> 
>> Such a thing could solve that mremap() problem
>> that me and William Tambe have.
> 
> If you're thinking of going that way, for this problem it
> would be better just to stick with POSIX shm_open, as Christoph
> suggested before, and you suggest to William in other mail.
> 
> But I now accept your view, that the shared anonymous object
> is not behaving consistently in response to mremap, and would
> like to put in a patch for that.  This isn't a particularly good
> time for such a patch - it's unclear right now whether 2.6.23 will
> have shmem_nopage like 2.6.22 or shmem_fault like 2.6.22-rc-mm;
> but we can easily adjust to whichever.
> 
> Here's a patch against 2.6.22-rc7: would you, Stas, put your
> Signed-off-by into this, and accept authorship - although I'm
> sending this back to you, it's very much your idea, and only
> trivially modified from your three-year-old patch by me.  If
> you're agreeable, I can then forward it or its shmem_zero_fault
> equivalent to Andrew when we see which way 2.6.23 is going.
> 
> [I'll fill in the patch comment later]
> 
> Signed-off-by: Hugh Dickins <hugh@veritas.com>
> ----
> 
>  mm/shmem.c |   41 ++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 40 insertions(+), 1 deletion(-)
> 
> --- 2.6.22-rc7/mm/shmem.c	2007-06-17 10:51:02.000000000 +0100
> +++ linux/mm/shmem.c	2007-07-03 15:35:32.000000000 +0100
> @@ -1320,6 +1320,36 @@ static struct page *shmem_nopage(struct 
>  	return page;
>  }
>  
> +struct page *shmem_zero_nopage(struct vm_area_struct *vma,
> +				unsigned long address, int *type)
> +{
> +	struct page *page;
> +
> +	page = shmem_nopage(vma, address, type);
> +	if (unlikely(page == NOPAGE_SIGBUS)) {
> +		struct inode *inode = vma->vm_file->f_dentry->d_inode;
> +		struct shmem_inode_info *info = SHMEM_I(inode);
> +		loff_t vm_size, i_size;
> +
> +		/*
> +		 * If mremap has been used to extend the vma,
> +		 * now extend the underlying object to include this page.
> +		 */
> +		vm_size = (PAGE_ALIGN(address) - vma->vm_start) +
> +				((loff_t)vma->vm_pgoff << PAGE_SHIFT);
> +		spin_lock(&info->lock);
> +		i_size = i_size_read(inode);
> +		if (i_size < vm_size && vm_size <= SHMEM_MAX_BYTES &&
> +		    !shmem_acct_size(info->flags, vm_size - i_size)) {
> +			i_size_write(inode, vm_size);
> +			inode->i_ctime = inode->i_mtime = CURRENT_TIME;
> +		}
> +		spin_unlock(&info->lock);
> +		page = shmem_nopage(vma, address, type);
> +	}
> +	return page;
> +}
> +
>  static int shmem_populate(struct vm_area_struct *vma,
>  	unsigned long addr, unsigned long len,
>  	pgprot_t prot, unsigned long pgoff, int nonblock)
> @@ -2471,6 +2501,14 @@ static struct vm_operations_struct shmem
>  #endif
>  };
>  
> +static struct vm_operations_struct shmem_zero_vm_ops = {
> +	.nopage		= shmem_zero_nopage,
> +	.populate	= shmem_populate,
> +#ifdef CONFIG_NUMA
> +	.set_policy     = shmem_set_policy,
> +	.get_policy     = shmem_get_policy,
> +#endif
> +};
>  
>  static int shmem_get_sb(struct file_system_type *fs_type,
>  	int flags, const char *dev_name, void *data, struct vfsmount *mnt)
> @@ -2599,6 +2637,7 @@ int shmem_zero_setup(struct vm_area_stru
>  	if (vma->vm_file)
>  		fput(vma->vm_file);
>  	vma->vm_file = file;
> -	vma->vm_ops = &shmem_vm_ops;
> +	vma->vm_ops = &shmem_zero_vm_ops;
> +	vma->vm_pgoff = 0;
>  	return 0;
>  }
> 

Will this patch be added to stable versions of the linux kernel?
Please let me know.

Sincerely,
William Tambe

Re: Concerning a post that you made about expandable anonymous shared mappings

[Hugh Dickins]

On Mon, 9 Jul 2007, William Tambe wrote:
> Hugh Dickins wrote:
> > 
> > I've come right around to your original view, Stas, and William's:
> > if that mmap creates such an object, then the expanding mremap really
> > ought to be useful, and allow the underlying object to be expanded.
> > The shared anonymous object is already anomalous: expanding it on
> > fault makes it more consistent with its own nature, not less.
> > ...
> > Here's a patch against 2.6.22-rc7: would you, Stas, put your
> > Signed-off-by into this, and accept authorship - although I'm
> > sending this back to you, it's very much your idea, and only
> > trivially modified from your three-year-old patch by me.  If
> > you're agreeable, I can then forward it or its shmem_zero_fault
> > equivalent to Andrew when we see which way 2.6.23 is going.
>... 
> Will this patch be added to stable versions of the linux kernel?
> Please let me know.

I confess that the lukewarm response from Stas cooled my enthusiasm,
and left me feeling that perhaps I'm an idiot to be adding such a
feature so many years too late; and my old caution about the way
a child could use up memory not freed on child's exit, unknown to
parent, returned to haunt me.  That could be documented for new
usages, but I just don't know what usages are already out there,
and fear I'd be introducing an exploit.

It most certainly will not be added to a stable version of the
linux kernel, if by that you mean 2.6.22.N or 2.6.21.N etc.
Though it can be viewed as a bugfix, the patch as it stands
seems in danger of introducing its own bug, and it's just too
much of a feature to be suitable for a -stable release.

But more probably you meant, will it be in 2.6.23 or 2.6.24?
Sorry to be such a vacillatiing wimp, but I don't know.
How well are you managing with the shm_open approach?

Hugh

Re: Concerning a post that you made about expandable anonymous shared mappings

[William Tambe]

Hugh Dickins wrote:
> On Mon, 9 Jul 2007, William Tambe wrote:
>> Hugh Dickins wrote:
>>> I've come right around to your original view, Stas, and William's:
>>> if that mmap creates such an object, then the expanding mremap really
>>> ought to be useful, and allow the underlying object to be expanded.
>>> The shared anonymous object is already anomalous: expanding it on
>>> fault makes it more consistent with its own nature, not less.
>>> ...
>>> Here's a patch against 2.6.22-rc7: would you, Stas, put your
>>> Signed-off-by into this, and accept authorship - although I'm
>>> sending this back to you, it's very much your idea, and only
>>> trivially modified from your three-year-old patch by me.  If
>>> you're agreeable, I can then forward it or its shmem_zero_fault
>>> equivalent to Andrew when we see which way 2.6.23 is going.
>> ... 
>> Will this patch be added to stable versions of the linux kernel?
>> Please let me know.
> 
> I confess that the lukewarm response from Stas cooled my enthusiasm,
> and left me feeling that perhaps I'm an idiot to be adding such a
> feature so many years too late; and my old caution about the way
> a child could use up memory not freed on child's exit, unknown to
> parent, returned to haunt me.  That could be documented for new
> usages, but I just don't know what usages are already out there,
> and fear I'd be introducing an exploit.
> 
> It most certainly will not be added to a stable version of the
> linux kernel, if by that you mean 2.6.22.N or 2.6.21.N etc.
> Though it can be viewed as a bugfix, the patch as it stands
> seems in danger of introducing its own bug, and it's just too
> much of a feature to be suitable for a -stable release.
> 
> But more probably you meant, will it be in 2.6.23 or 2.6.24?
> Sorry to be such a vacillatiing wimp, but I don't know.
> How well are you managing with the shm_open approach?
> 
> Hugh
> 

I understand your concern. But since I am working on a dynamic memory 
management code that I wish to use with other projects that I have, I 
didn't find appropriate to use shm_open.

In fact there is a name associated with the shared memory requested with 
shm_open, so that it can be mmap(ed) in another process. And I do not 
wish to have it accessible by any other process, unless I choose to do so.

shm_open is great, but it doesn't quite fit my needs.
And I think remap(ing) ANONYMOUS memory kind of make a lot of things easier.

Can that feature be added at least to release candidates series so that 
everybody can try it and see how well it does?

Sincerely,
William Tambe

Re: Concerning a post that you made about expandable anonymous shared mappings

[Stas Sergeev]

Hi.

William Tambe wrote:
> I understand your concern. But since I am working on a dynamic memory 
> management code that I wish to use with other projects that I have, I 
> didn't find appropriate to use shm_open.
Could you please provide a detailed list of the
problems you have with shm_open? If they are
valid, then I can bet the patch will be applied,
no matter what. :)

> In fact there is a name associated with the shared memory requested with 
> shm_open, so that it can be mmap(ed) in another process. And I do not 
> wish to have it accessible by any other process, unless I choose to do so.
In this case you need to use shm_unlink() right
after shm_open(). Then this shm will be accessable
only to your process and its children, via an fd,
and not to anyone else. And you still can do anything
with it (ftruncate/mmap/mremap whatever).

> And I think remap(ing) ANONYMOUS memory kind of make a lot of things easier.
In what way, exactly?

Re: Concerning a post that you made about expandable anonymous shared mappings

[William Tambe]

Stas Sergeev wrote:
> Hi.
> 
> William Tambe wrote:
>> I understand your concern. But since I am working on a dynamic memory 
>> management code that I wish to use with other projects that I have, I 
>> didn't find appropriate to use shm_open.
> Could you please provide a detailed list of the
> problems you have with shm_open? If they are
> valid, then I can bet the patch will be applied,
> no matter what. :)
> 
>> In fact there is a name associated with the shared memory requested 
>> with shm_open, so that it can be mmap(ed) in another process. And I do 
>> not wish to have it accessible by any other process, unless I choose 
>> to do so.
> In this case you need to use shm_unlink() right
> after shm_open(). Then this shm will be accessable
> only to your process and its children, via an fd,
> and not to anyone else. And you still can do anything
> with it (ftruncate/mmap/mremap whatever).
> 

Ok, now I find myself without any other arguments :-) shm_unlink() right 
after shm_open() is a solution.


>> And I think remap(ing) ANONYMOUS memory kind of make a lot of things 
>> easier.
> In what way, exactly?
> 
> 

I wrote the above not knowing that I could use shm_unlink() right after 
shm_open(). But still, I have lost a considerable amount of time trying 
to figure that out.
It appeared all natural to me that I could just remap ANONYMOUS and get 
what I wanted. And the worst thing here is that the man pages do not let 
you know about that.

Sincerely,
William Tambe