Re: [RFC] [patch 0/39] remap_file_pages protection support, try 2

Re: [RFC] [patch 0/39] remap_file_pages protection support, try 2

[David S. Miller]

Please do not BOMB linux-kernel with 39 patches in one
go, that will kill the list server.

Try to consolidate your patch groups into smaller pieces,
like so about 10 or 15 at a time.  And send any that remain
on some later date.

[RFC] [patch 0/39] remap_file_pages protection support, try 2

[Blaisorblade]

[-- Attachment #1: Type: text/plain, Size: 4645 bytes --]

Ok, I've been working for the past two weeks learning well the Linux VM, 
understanding the Ingo's remap_file_pages protection support and its various 
weakness (due to lack of time on his part), and splitting and finishing it.

Here follow a series of 39 _little_ patches against the git-commit-id 
889371f61fd5bb914d0331268f12432590cf7e85, which means between 2.6.13-rc4 and 
-rc5.

Actually, the first 7 ones are unrelated trivial cleanups which somehow get in 
the way on this work and that can probably be merged even now (many are just 
comment fixes).

Since I was a VM newbie until two weeks ago, I've separated my changes into 
many little patches.

To avoid the noise, I'm CC:ing many people only on this message, while I'm 
sending the full patch series only to akpm, mingo and LKML. Or actually, I'm 
trying - my provider seem not to like me sending so many patches. I attached 
an exported tarball to this mail, since it's very little.

I hope these changes can be included inside -mm, but I guess that they'll 
probably conflict with pagefault scalability patches, and that some of them 
are not completely polished. Still, the patch is IMHO in better shape, in 
many ways, than when it was in -mm last time.

I'll appreciate any comments.

==============
Changes from 2.6.5-mm1/dropped version of the patches:
==============
*) Actually implemented _real_ and _anal_ protection support, safe against 
swapout; programs get SIGSEGV *always* when they should. I've used the 
attached test program (an improved version of Ingo's one) to check that.
I tested just until patch 25, onto UML. The subsequent ones are either patches 
for foreign archs or proposed

*) Fixed many changes present in the patches.
*) Fixed UML bits
*) Added several headaches for arches ports. I've also included some patches 
which reduce this 

*) No more usage of a new syscall slot: to use the new interface, application 
will use the new MAP_NOINHERIT flag I've added. I've still the patches to use 
the old -mm ABI, if there's any reason they're needed.

*) Fixed a regression wrt using mprotect() against remapped area (see patch 
15)

======
Still to do:
======
*) fix mprotect VS remap_file_pages(MAP_NOINHERIT) interaction - see long 
discussion in patch 15 changelog
*) ->populate flushes each TLB individually, instead of using mmu_gathers as 
it should; this was suggested even by Ingo when sending the patch, but it 
seems he didn't get the time to finish this. Seems rewriting the kernel 
locking is a quite time-consuming task!

======
Patch summaries
======
Each patch has an attached changelog, but I'm giving here a summary (sorry for 
using the patch numbers, but I found no other way).

The first 7 are just generic cleanups (mostly for comments) which bugged me 
along the way, however some of them are needed for the subsequent patches to 
apply.

08-11 ones are arch bits for some arches (the ones I have access too).

12 is the core change for generic code, 13-17 are various changes to the 
syscall code, as 20, 21 and 23, 35 and 36, to review individually. Most of 
those changes (except #23, which is a fix for try_to_unmap_one I missed 
initially) are just speedups, and it should be possible to individually drop 
them.

18, 19, 22, 32, 33, 34 move partially the handling of protection checks from 
the arches' page faults handler to the generic code, by introducing 
VM_FAULT_SIGSEGV. In fact, the VMA protection are not reliable for 
VM_NONUNIFORM areas. This aspect was just begun in Ingo's code, and was the 
weakest area of his patch. I must now pass the *full* kind of fault to the 
generic code, and test it against the PTE or possibly the VMA protections. 
However, in these patches it's done in a kludgy way, because we check the VMA 
protections against VM_READ/WRITE/EXEC with no consideration of the 
architecture-specific dependencies between them (like READ_IMPLIES_EXEC and 
so on), so arches have to workaround this. This is fixed in patch 33, which 
is untested however.

24 and 25 are some fixes for UML code, needed to make it work even with this 
change.

26-31 are other arch's compile fix for the introduction of pte_to_pgoff.

The last three ones (37-39) are not to apply - they are some possible changes 
I'm either really uncertain about, or which I'm sure are wrong in that form 
but express possibly correct ideas. 36 should be a fixed version of the #37 
one, but I wrote it in the past few minutes.
-- 
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

[-- Attachment #2: fremap-prot-complete-broken-out.tar.bz2 --]
[-- Type: application/x-tbz, Size: 29351 bytes --]

[-- Attachment #3: fremap-test-complete.c.bz2 --]
[-- Type: application/x-bzip2, Size: 4075 bytes --]

Re: [RFC] [patch 0/39] remap_file_pages protection support, try 2

[Andi Kleen]

Blaisorblade <blaisorblade@yahoo.it> writes:

> Ok, I've been working for the past two weeks learning well the Linux VM, 
> understanding the Ingo's remap_file_pages protection support and its various 
> weakness (due to lack of time on his part), and splitting and finishing it.

I'm not sure remap_file_pages was ever intended to be more integrated.
It pretty much always was a Oracle specific performance hack. The problem
with making it more powerful is that it will become more invasive then
(like your patchbomb shows) and that will make it a bigger maintenance
issue longer term and complicate all of VM. And it's probably not
worth doing all that.

So in short I think it's better to keep it into its corner with minimum
functionality and let it not expand to other parts.

-Andi

Re: [RFC] [patch 0/39] remap_file_pages protection support, try 2

[Ingo Molnar]

* Blaisorblade <blaisorblade@yahoo.it> wrote:

> Ok, I've been working for the past two weeks learning well the Linux 
> VM, understanding the Ingo's remap_file_pages protection support and 
> its various weakness (due to lack of time on his part), and splitting 
> and finishing it.
> 
> Here follow a series of 39 _little_ patches against the git-commit-id 
> 889371f61fd5bb914d0331268f12432590cf7e85, which means between 
> 2.6.13-rc4 and -rc5.
> 
> Actually, the first 7 ones are unrelated trivial cleanups which 
> somehow get in the way on this work and that can probably be merged 
> even now (many are just comment fixes).
> 
> Since I was a VM newbie until two weeks ago, I've separated my changes 
> into many little patches.

hi. Great work! I'm wondering about this comment in 
rfp-fix-unmap-linear.patch:

> Additionally, add a missing TLB flush in both locations. However, 
> there'is some excess of flushes in these functions.

excess TLB flushes one of the reasons of bad UML performance, so you 
should really review them and not do spurious TLB flushes.

	Ingo

Re: [RFC] [patch 0/39] remap_file_pages protection support, try 2

[Blaisorblade]

On Friday 12 August 2005 22:52, Andi Kleen wrote:
> Blaisorblade <blaisorblade@yahoo.it> writes:
> > Ok, I've been working for the past two weeks learning well the Linux VM,
> > understanding the Ingo's remap_file_pages protection support and its
> > various weakness (due to lack of time on his part), and splitting and
> > finishing it.

> I'm not sure remap_file_pages was ever intended to be more integrated.
> It pretty much always was a Oracle specific performance hack. The problem
> with making it more powerful is that it will become more invasive then
> (like your patchbomb shows)

(Note that it's probably even *too* subtly splitout).

> and that will make it a bigger maintenance 
> issue longer term and complicate all of VM. And it's probably not
> worth doing all that.

The reason for this work, this time, is for UML (it's not casual I'm working 
on it). However, it's true that what I implemented is actually more powerful 
than what's needed by UML.

It only needs (IIRC) to do void (i.e. PROT_NONE) mmaps, and then to remap from 
them changing both protections and file offset. And such mappings must 
support swapout.

This will replace the current use of mmap()/mprotect() done one page at a time 
(when the guest has more than 256 M we easily hit 
the /proc/sys/vm/max_map_count and must increase it).

I've gone implementing and fixing the general thing, including mappings where 
only the protections are changed but which are still linear.

Still, even without all this the patch will still be bigger than when it was 
in -mm last time (i.e. until 2.6.5-mm1), since that didn't handle swapout.

However, in terms of invasiveness, the changes needed to support the nonlinear 
mappings are way bigger than what's needed for protections, especially for 
nonlinear truncation.

> So in short I think it's better to keep it into its corner with minimum
> functionality and let it not expand to other parts.
Ok, I think this may be reasonable. However, for now I'm still finishing the 
full implementation, I'll start dropping pieces when we've decided what's 
needed.
-- 
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

	

	
		
___________________________________ 
Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB 
http://mail.yahoo.it