Question on mmap / expecting more calls to vfs_read

Question on mmap / expecting more calls to vfs_read

[Sebastian Pipping]

Hello!


Intro / context
===============
I'm trying to get a precise list of file names, offsets and byte counts
of all read and write operations.  I need that information to feed it
into a simulator I am working on.  I am running qemu-kvm with a slightly
patched Linux kernel (additions of calls to printk only) sucking the
kernel log from the serial console (kernel option
earlyprintk=serial,keep) to a file.  The setup itself works okay for me.


Question / problem
==================
When analyzing the kernel logs I produced I noticed stumbled over the
case of a few libraries where only the beginning of the file (maybe its
ELF header) was read repeated times, but nothing after:

  file                   /lib/ld-2.11.2.so
  max offset+count       456
  total bytes read ever  502840

Especially as libc is among them, I guess some reads pass by me somehow.
Any ideas what is shoveling the file bytes around?

I found calls to do_mmap_pgoff for these file, though.
That's when I thought I may need help understanding how mmap works.

Is data read into the mapped region instantly all at once?  Or is it
feed into it on demand when the kernel gets read requests to a mapped
area?  In either case, where does the data come from if neither vfs_read
nor do_readv_writev are ever called on the rest of the file?


Thanks in advance for any help.




Sebastian

Question on mmap / expecting more calls to vfs_read

[Mulyadi Santosa]

Hi...

On Thu, Jan 6, 2011 at 03:16, Sebastian Pipping <sebastian@pipping.org> wrote:
> When analyzing the kernel logs I produced I noticed stumbled over the
> case of a few libraries where only the beginning of the file (maybe its
> ELF header) was read repeated times, but nothing after:
>
> ?file ? ? ? ? ? ? ? ? ? /lib/ld-2.11.2.so
> ?max offset+count ? ? ? 456
> ?total bytes read ever ?502840
>
> Especially as libc is among them, I guess some reads pass by me somehow.
> Any ideas what is shoveling the file bytes around?
>
> I found calls to do_mmap_pgoff for these file, though.
> That's when I thought I may need help understanding how mmap works.

AFAIK, mmap will eventually calls VFS read in the case of major fault.
However, in other cases the content could already be in page cache,
that's why you don't see any filesystem or block device related
operation.

As we know, ld.so and libc.so are one of the most linked library....
therefore it is very likely they are already in the page cache. "But I
saw initial read, why is that?". IMHO that's because filesystem layer
need to make sure, it is the same file which is already cached in the
page cache.

To be precise, inode number, in which major and minor block device it
resides...and possibly other determinant factor. Just after it is
decided it's the same, page cache could be 100%
utilized..again..assuming all the content of ld.so and libc.so stays
in cache and never get flushed back.

Take the above hint with lots of salt, ok? :D

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

Question on mmap / expecting more calls to vfs_read

[Rajat Sharma]

Hi Sebastian,

you guess for ELF header seems to be valid to me. When executables or
binaries are loaded memory, it is done through mmap call to the file,
and to understand what file is and what binary handler in kernel can
handle its section, kernel needs to know its header first, which is
within the first page of the header with fixed location for a magic
number (identifier for binary handler e.g. ELF handler which further
loads its other sections by reading section table). Note that there
are multiple binary format handles within the kernel e.g. ELF, A.OUT
which are tried sequentially to identify the file format.

>From the file system perspective, mmap does not use vfs_read or
vfs_write calls at all, thats why you don't see them. It directly
works on address space operations of an inode (file) to populate data
in page-cache. For a mmapped region, if you don't see a page in
memory, kenel page faults and tries to read-in the page using readpage
method of address_space_operations. Similarly when you modify a page,
writepage method is called, but since executables are accessed
read-only, you won't see writepage method getting called either.

Hope this makes it clearer.

Rajat

On Thu, Jan 6, 2011 at 3:45 AM, Mulyadi Santosa
<mulyadi.santosa@gmail.com> wrote:
> Hi...
>
> On Thu, Jan 6, 2011 at 03:16, Sebastian Pipping <sebastian@pipping.org> wrote:
>> When analyzing the kernel logs I produced I noticed stumbled over the
>> case of a few libraries where only the beginning of the file (maybe its
>> ELF header) was read repeated times, but nothing after:
>>
>> ?file ? ? ? ? ? ? ? ? ? /lib/ld-2.11.2.so
>> ?max offset+count ? ? ? 456
>> ?total bytes read ever ?502840
>>
>> Especially as libc is among them, I guess some reads pass by me somehow.
>> Any ideas what is shoveling the file bytes around?
>>
>> I found calls to do_mmap_pgoff for these file, though.
>> That's when I thought I may need help understanding how mmap works.
>
> AFAIK, mmap will eventually calls VFS read in the case of major fault.
> However, in other cases the content could already be in page cache,
> that's why you don't see any filesystem or block device related
> operation.
>
> As we know, ld.so and libc.so are one of the most linked library....
> therefore it is very likely they are already in the page cache. "But I
> saw initial read, why is that?". IMHO that's because filesystem layer
> need to make sure, it is the same file which is already cached in the
> page cache.
>
> To be precise, inode number, in which major and minor block device it
> resides...and possibly other determinant factor. Just after it is
> decided it's the same, page cache could be 100%
> utilized..again..assuming all the content of ld.so and libc.so stays
> in cache and never get flushed back.
>
> Take the above hint with lots of salt, ok? :D
>
> --
> regards,
>
> Mulyadi Santosa
> Freelance Linux trainer and consultant
>
> blog: the-hydra.blogspot.com
> training: mulyaditraining.blogspot.com
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

Question on mmap / expecting more calls to vfs_read

[Sebastian Pipping]

On 01/06/11 07:53, Rajat Sharma wrote:
> Hi Sebastian,
> 
> you guess for ELF header seems to be valid to me. When executables or
> binaries are loaded memory, it is done through mmap call to the file,
> and to understand what file is and what binary handler in kernel can
> handle its section, kernel needs to know its header first, which is
> within the first page of the header with fixed location for a magic
> number (identifier for binary handler e.g. ELF handler which further
> loads its other sections by reading section table). Note that there
> are multiple binary format handles within the kernel e.g. ELF, A.OUT
> which are tried sequentially to identify the file format.
> 
> From the file system perspective, mmap does not use vfs_read or
> vfs_write calls at all, thats why you don't see them. It directly
> works on address space operations of an inode (file) to populate data
> in page-cache. For a mmapped region, if you don't see a page in
> memory, kenel page faults and tries to read-in the page using readpage
> method of address_space_operations. Similarly when you modify a page,
> writepage method is called, but since executables are accessed
> read-only, you won't see writepage method getting called either.
> 
> Hope this makes it clearer.

Excellent, thank you!  I find calls to readpages on file
/lib/libc-2.11.2.so now.  That may be the missing reads.

Any ideas how get offset and length (like with vfs_read) for a certain
page passed to readpage(file, page) ?

Best,



Sebastian

Question on mmap / expecting more calls to vfs_read

[Manish Katiyar]

On Thu, Jan 6, 2011 at 3:49 PM, Sebastian Pipping <sebastian@pipping.org> wrote:
> On 01/06/11 07:53, Rajat Sharma wrote:
>> Hi Sebastian,
>>
>> you guess for ELF header seems to be valid to me. When executables or
>> binaries are loaded memory, it is done through mmap call to the file,
>> and to understand what file is and what binary handler in kernel can
>> handle its section, kernel needs to know its header first, which is
>> within the first page of the header with fixed location for a magic
>> number (identifier for binary handler e.g. ELF handler which further
>> loads its other sections by reading section table). Note that there
>> are multiple binary format handles within the kernel e.g. ELF, A.OUT
>> which are tried sequentially to identify the file format.
>>
>> From the file system perspective, mmap does not use vfs_read or
>> vfs_write calls at all, thats why you don't see them. It directly
>> works on address space operations of an inode (file) to populate data
>> in page-cache. For a mmapped region, if you don't see a page in
>> memory, kenel page faults and tries to read-in the page using readpage
>> method of address_space_operations. Similarly when you modify a page,
>> writepage method is called, but since executables are accessed
>> read-only, you won't see writepage method getting called either.
>>
>> Hope this makes it clearer.
>
> Excellent, thank you! ?I find calls to readpages on file
> /lib/libc-2.11.2.so now. ?That may be the missing reads.
>
> Any ideas how get offset and length (like with vfs_read) for a certain
> page passed to readpage(file, page) ?

Add your stats hooks to do_mmap or mmap_region() ???

-- 
Thanks -
Manish
==================================
[$\*.^ -- I miss being one of them
==================================

Question on mmap / expecting more calls to vfs_read

[Rajat Sharma]

> Add your stats hooks to do_mmap or mmap_region() ???

mmap is one time call to initialize virtual address space of current process
with corresponding region on file's disk image. Once this setting is done,
this the actual calls to read data from file is accomplish through memory
area's (vma) page fault handlers which in turn call readpage address space
operation of an inode:

inode->i_mapping->a_ops->readpage(file, page).

so, suitable position is to add hooks on readpage a_op. And of-course for
doing that, you may have to capture various path thorugh which inode can
come in memory, e.g. lookup and create directory inode operation (for
regular files). For your worst nightmare, NFS implements its readdir with an
additional feature with v3 protocol called READDIR PLUS, which not only
gives you name of children of a directory, but also initializes their inodes
in memory, so you may have to hook readdir as well and trap aops of all
regular file child after nfs_readdir is finished.

As far as offset and length of I/O are concerned, page->index gives you its
index in the page cache which in turn is equivalent to file offset
(page->index << PAGE_SHIFT). readpage is invoked to bring in complete page
in memory. It may so happen that page is a partial page (e.g. last page of
file), in that case your I/O lenght will be inode->i_size & ~PAGE_MASK,
otherwise it can be PAGE_SIZE. don't worry about file wholes, that is taken
care by filesystem's original readpage method.

Having said above, it will still be better if you can state what you want to
achieve in little layman language.

Rajat

On Fri, Jan 7, 2011 at 5:39 AM, Manish Katiyar <mkatiyar@gmail.com> wrote:

> On Thu, Jan 6, 2011 at 3:49 PM, Sebastian Pipping <sebastian@pipping.org>
> wrote:
> > On 01/06/11 07:53, Rajat Sharma wrote:
> >> Hi Sebastian,
> >>
> >> you guess for ELF header seems to be valid to me. When executables or
> >> binaries are loaded memory, it is done through mmap call to the file,
> >> and to understand what file is and what binary handler in kernel can
> >> handle its section, kernel needs to know its header first, which is
> >> within the first page of the header with fixed location for a magic
> >> number (identifier for binary handler e.g. ELF handler which further
> >> loads its other sections by reading section table). Note that there
> >> are multiple binary format handles within the kernel e.g. ELF, A.OUT
> >> which are tried sequentially to identify the file format.
> >>
> >> From the file system perspective, mmap does not use vfs_read or
> >> vfs_write calls at all, thats why you don't see them. It directly
> >> works on address space operations of an inode (file) to populate data
> >> in page-cache. For a mmapped region, if you don't see a page in
> >> memory, kenel page faults and tries to read-in the page using readpage
> >> method of address_space_operations. Similarly when you modify a page,
> >> writepage method is called, but since executables are accessed
> >> read-only, you won't see writepage method getting called either.
> >>
> >> Hope this makes it clearer.
> >
> > Excellent, thank you!  I find calls to readpages on file
> > /lib/libc-2.11.2.so now.  That may be the missing reads.
> >
> > Any ideas how get offset and length (like with vfs_read) for a certain
> > page passed to readpage(file, page) ?
>
> Add your stats hooks to do_mmap or mmap_region() ???
>
> --
> Thanks -
> Manish
> ==================================
> [$\*.^ -- I miss being one of them
> ==================================
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110107/3f190eca/attachment.html 

Question on mmap / expecting more calls to vfs_read

[Rajat Sharma]

I meant 'file holes' not 'wholes' in last mail... my bad in shooting mails
:(

Rajat

On Fri, Jan 7, 2011 at 11:56 AM, Rajat Sharma <fs.rajat@gmail.com> wrote:

> > Add your stats hooks to do_mmap or mmap_region() ???
>
> mmap is one time call to initialize virtual address space of current
> process with corresponding region on file's disk image. Once this setting is
> done, this the actual calls to read data from file is accomplish through
> memory area's (vma) page fault handlers which in turn call readpage address
> space operation of an inode:
>
> inode->i_mapping->a_ops->readpage(file, page).
>
> so, suitable position is to add hooks on readpage a_op. And of-course for
> doing that, you may have to capture various path thorugh which inode can
> come in memory, e.g. lookup and create directory inode operation (for
> regular files). For your worst nightmare, NFS implements its readdir with an
> additional feature with v3 protocol called READDIR PLUS, which not only
> gives you name of children of a directory, but also initializes their inodes
> in memory, so you may have to hook readdir as well and trap aops of all
> regular file child after nfs_readdir is finished.
>
> As far as offset and length of I/O are concerned, page->index gives you its
> index in the page cache which in turn is equivalent to file offset
> (page->index << PAGE_SHIFT). readpage is invoked to bring in complete page
> in memory. It may so happen that page is a partial page (e.g. last page of
> file), in that case your I/O lenght will be inode->i_size & ~PAGE_MASK,
> otherwise it can be PAGE_SIZE. don't worry about file wholes, that is taken
> care by filesystem's original readpage method.
>
> Having said above, it will still be better if you can state what you want
> to achieve in little layman language.
>
> Rajat
>
>
> On Fri, Jan 7, 2011 at 5:39 AM, Manish Katiyar <mkatiyar@gmail.com> wrote:
>
>> On Thu, Jan 6, 2011 at 3:49 PM, Sebastian Pipping <sebastian@pipping.org>
>> wrote:
>> > On 01/06/11 07:53, Rajat Sharma wrote:
>> >> Hi Sebastian,
>> >>
>> >> you guess for ELF header seems to be valid to me. When executables or
>> >> binaries are loaded memory, it is done through mmap call to the file,
>> >> and to understand what file is and what binary handler in kernel can
>> >> handle its section, kernel needs to know its header first, which is
>> >> within the first page of the header with fixed location for a magic
>> >> number (identifier for binary handler e.g. ELF handler which further
>> >> loads its other sections by reading section table). Note that there
>> >> are multiple binary format handles within the kernel e.g. ELF, A.OUT
>> >> which are tried sequentially to identify the file format.
>> >>
>> >> From the file system perspective, mmap does not use vfs_read or
>> >> vfs_write calls at all, thats why you don't see them. It directly
>> >> works on address space operations of an inode (file) to populate data
>> >> in page-cache. For a mmapped region, if you don't see a page in
>> >> memory, kenel page faults and tries to read-in the page using readpage
>> >> method of address_space_operations. Similarly when you modify a page,
>> >> writepage method is called, but since executables are accessed
>> >> read-only, you won't see writepage method getting called either.
>> >>
>> >> Hope this makes it clearer.
>> >
>> > Excellent, thank you!  I find calls to readpages on file
>> > /lib/libc-2.11.2.so now.  That may be the missing reads.
>> >
>> > Any ideas how get offset and length (like with vfs_read) for a certain
>> > page passed to readpage(file, page) ?
>>
>> Add your stats hooks to do_mmap or mmap_region() ???
>>
>> --
>> Thanks -
>> Manish
>> ==================================
>> [$\*.^ -- I miss being one of them
>> ==================================
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110107/2216691d/attachment-0001.html 

Question on mmap / expecting more calls to vfs_read

[Sebastian Pipping]

On 01/07/11 07:26, Rajat Sharma wrote:
> so, suitable position is to add hooks on readpage a_op. And of-course
> for doing that, you may have to capture various path thorugh which inode
> can come in memory, e.g. lookup and create directory inode operation
> (for regular files). For your worst nightmare, NFS implements its
> readdir with an additional feature with v3 protocol called READDIR PLUS,
> which not only gives you name of children of a directory, but also
> initializes their inodes in memory, so you may have to hook readdir as
> well and trap aops of all regular file child after nfs_readdir is
finished.

I see.


> As far as offset and length of I/O are concerned, page->index gives you
> its index in the page cache which in turn is equivalent to file offset
> (page->index << PAGE_SHIFT). readpage is invoked to bring in complete
> page in memory. It may so happen that page is a partial page (e.g. last
> page of file), in that case your I/O lenght will be inode->i_size &
> ~PAGE_MASK, otherwise it can be PAGE_SIZE. don't worry about file
> wholes, that is taken care by filesystem's original readpage method.

That sounds great.  I have added these lines of code to read_pages() of
mm/readahead.c:

================================================
{
  struct page *page = list_to_page(pages);
  for (page_idx = 0; page_idx < nr_pages;
      page_idx++,
      page = list_to_page(page->lru.next)) {
    pgoff_t offset = 0;
    pgoff_t len = PAGE_SIZE;

    if (page) {
      offset = page->index << PAGE_SHIFT;

      if (page->mapping && page->mapping->host) {
        struct inode *inode = page->mapping->host;
        len = inode->i_size & ~PAGE_MASK;
      }
    }

    printk(KERN_DEBUG "VFS: BODY at 1 read_pages(...)["
        "page_idx=%u, offset=%lu, len=%lu, page_size=%lu]",
        page_idx, offset, len, PAGE_SIZE);
  }
}
================================================

Suprisingly I never get len != PAGE_SIZE, i.e. partial pages.
Also, in a list of pages given, the offsets are all the same, all 0 in
this case:

================================================
[ 2811.993564] VFS: BODY at 1 read_pages(...)[page_idx=0, offset=0,
len=4096, page_size=4096]
[ 2811.996456] VFS: BODY at 1 read_pages(...)[page_idx=1, offset=0,
len=4096, page_size=4096]
[ 2811.999305] VFS: BODY at 1 read_pages(...)[page_idx=2, offset=0,
len=4096, page_size=4096]
[ 2812.002152] VFS: BODY at 1 read_pages(...)[page_idx=3, offset=0,
len=4096, page_size=4096]
================================================

Is  page = list_to_page(page->lru.next)  the correct way of walking
pages?  I am wondering how the main loop of read_pages()

	for (page_idx = 0; page_idx < nr_pages; page_idx++) {
		struct page *page = list_to_page(pages);
		[..]
	}

really iterates, if neither the address of pages changes in the loop nor
page_idx is passed to anything.  I would think the loop is actually
accessing the same page again and again, but somehow my own

  page = list_to_page(page->lru.next)

above doesn't seem to do better.  Any insights?

Thanks!


> Having said above, it will still be better if you can state what you
> want to achieve in little layman language.

I want to trace down all reads and writes to the file system including
filename, offset and byte count into a log that I can reply inside a
simulator.

Best,




Sebastian

Question on mmap / expecting more calls to vfs_read

[Rajat Sharma]

Hi Sebastian,

read_pages is a small piece of code, please paste the complete
function with your changes over here, I am sure there must be some
minor bug otherwise page->lru.next looks fine to me.

> I am wondering how the main loop of read_pages()
>
>        for (page_idx = 0; page_idx < nr_pages; page_idx++) {
>                struct page *page = list_to_page(pages);
>                [..]
>        }
>
> really iterates, if neither the address of pages changes in the loop nor
> page_idx is passed to anything.  I would think the loop is actually
> accessing the same page again and again, but somehow my own

No its not iterating same page, you missed the call to
list_del(&page->lru) in the main loop. Remember it removes the first
element from the head of the list (as this page is now responsibility
of page cache) and next element automatically becomes head of the list
ie. list_head *pages.

> I want to trace down all reads and writes to the file system including
> filename, offset and byte count into a log that I can reply inside a
> simulator.
>

for filename you have to track open calls to file system but remember
for resolving the complete path name, you have to get into depths of
namespace and play with vfsmount data structure.

Rajat

On Wed, Jan 12, 2011 at 12:46 AM, Sebastian Pipping
<sebastian@pipping.org> wrote:
> On 01/07/11 07:26, Rajat Sharma wrote:
>> so, suitable position is to add hooks on readpage a_op. And of-course
>> for doing that, you may have to capture various path thorugh which inode
>> can come in memory, e.g. lookup and create directory inode operation
>> (for regular files). For your worst nightmare, NFS implements its
>> readdir with an additional feature with v3 protocol called READDIR PLUS,
>> which not only gives you name of children of a directory, but also
>> initializes their inodes in memory, so you may have to hook readdir as
>> well and trap aops of all regular file child after nfs_readdir is
> finished.
>
> I see.
>
>
>> As far as offset and length of I/O are concerned, page->index gives you
>> its index in the page cache which in turn is equivalent to file offset
>> (page->index << PAGE_SHIFT). readpage is invoked to bring in complete
>> page in memory. It may so happen that page is a partial page (e.g. last
>> page of file), in that case your I/O lenght will be inode->i_size &
>> ~PAGE_MASK, otherwise it can be PAGE_SIZE. don't worry about file
>> wholes, that is taken care by filesystem's original readpage method.
>
> That sounds great. ?I have added these lines of code to read_pages() of
> mm/readahead.c:
>
> ================================================
> {
> ?struct page *page = list_to_page(pages);
> ?for (page_idx = 0; page_idx < nr_pages;
> ? ? ?page_idx++,
> ? ? ?page = list_to_page(page->lru.next)) {
> ? ?pgoff_t offset = 0;
> ? ?pgoff_t len = PAGE_SIZE;
>
> ? ?if (page) {
> ? ? ?offset = page->index << PAGE_SHIFT;
>
> ? ? ?if (page->mapping && page->mapping->host) {
> ? ? ? ?struct inode *inode = page->mapping->host;
> ? ? ? ?len = inode->i_size & ~PAGE_MASK;
> ? ? ?}
> ? ?}
>
> ? ?printk(KERN_DEBUG "VFS: BODY at 1 read_pages(...)["
> ? ? ? ?"page_idx=%u, offset=%lu, len=%lu, page_size=%lu]",
> ? ? ? ?page_idx, offset, len, PAGE_SIZE);
> ?}
> }
> ================================================
>
> Suprisingly I never get len != PAGE_SIZE, i.e. partial pages.
> Also, in a list of pages given, the offsets are all the same, all 0 in
> this case:
>
> ================================================
> [ 2811.993564] VFS: BODY at 1 read_pages(...)[page_idx=0, offset=0,
> len=4096, page_size=4096]
> [ 2811.996456] VFS: BODY at 1 read_pages(...)[page_idx=1, offset=0,
> len=4096, page_size=4096]
> [ 2811.999305] VFS: BODY at 1 read_pages(...)[page_idx=2, offset=0,
> len=4096, page_size=4096]
> [ 2812.002152] VFS: BODY at 1 read_pages(...)[page_idx=3, offset=0,
> len=4096, page_size=4096]
> ================================================
>
> Is ?page = list_to_page(page->lru.next) ?the correct way of walking
> pages? ?I am wondering how the main loop of read_pages()
>
> ? ? ? ?for (page_idx = 0; page_idx < nr_pages; page_idx++) {
> ? ? ? ? ? ? ? ?struct page *page = list_to_page(pages);
> ? ? ? ? ? ? ? ?[..]
> ? ? ? ?}
>
> really iterates, if neither the address of pages changes in the loop nor
> page_idx is passed to anything. ?I would think the loop is actually
> accessing the same page again and again, but somehow my own
>
> ?page = list_to_page(page->lru.next)
>
> above doesn't seem to do better. ?Any insights?
>
> Thanks!
>
>
>> Having said above, it will still be better if you can state what you
>> want to achieve in little layman language.
>
> I want to trace down all reads and writes to the file system including
> filename, offset and byte count into a log that I can reply inside a
> simulator.
>
> Best,
>
>
>
>
> Sebastian
>