Re: [RFC] de-asmify the x86-64 system call slowpath

["Kirill A. Shutemov" <kirill@shutemov.name>]

On Wed, Feb 05, 2014 at 04:32:55PM -0800, Linus Torvalds wrote:
> On Sun, Jan 26, 2014 at 2:28 PM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > Comments? This was obviously brought on by my frustration with the
> > currently nasty do_notify_resume() always returning to iret for the
> > task_work case, and PeterZ's patch that fixed that, but made the asm
> > mess even *worse*.
> 
> Actually, I should have taken a closer look.
> 
> Yes, do_notify_resume() is a real issue, and my stupid open/close
> test-case showed that part of the profile.
> 
> But the "iretq" that dominates on the kernel build is actually the
> page fault one.
> 
> I noticed this when I compared "-e cycles:pp" with "-e cycles:p". The
> single-p version shows largely the same profile for the kernel, except
> that instead of showing "iretq" as the big cost, it shows the first
> instruction in "page_fault".
> 
> In fact, even when *not* zoomed into the kernel DSO, "page_fault"
> actually takes 5% of CPU time according to pref report. That's really
> quite impressive.
> 
> I suspect the Haswell architecture has made everything else cheaper,
> and the exception overhead hasn't kept up. I'm wondering if there is
> anything we could do to speed this up - like doing gang lookup in the
> page cache and pre-populating the page tables opportunistically.

One thing that could help is THP for file-backed pages. And there's
prototype with basic infrasturure and support for ramfs and
shmem/tmpfs (by Ning Qu). Work in progress.

-- 
 Kirill A. Shutemov