[RFC] perf tools: Add hugetlbfs memory recognition

[RFC] perf tools: Add hugetlbfs memory recognition

[Hou Pengyang]

Maps for JIT is helpful for symbols-parsing for anon-executable-memory.
What we need to do is to add (START, SIZE, symbolname) to /tmp/perf-%d.map
(%d = pid of process), and perf would parse symbol located in this area
according to /tmp/perf-%d.map. It works well for normal mmap.

However, when we alloc such memory from hugetlbfs by the following code:

......

    fd = open("/mnt/huge/hugepagefile", O_CREAT | O_RDWR, 0755);
    if (fd < 0) {
        perror("Open failed");
        exit(1);
    }   
    addr = mmap(ADDR, LENGTH, PROT_READ | PROT_WRITE |     \   
        PROT_EXEC, MAP_SHARED, fd, 0); 

......
where hugetlbfs is mounted in /mnt/huge. Symbols could not be parsed correctly:

#perf report
   86.96%     0.00%  hugepage-mmap  hugepagefile      [.] 0xffffffe00000005c      
   86.96%     0.00%  hugepage-mmap  hugepagefile      [.] 0xffffffe0000000ac

This is because such mmap area's file backed is "/mnt/huge/hugepagefile", a node 
in pseudo filesystem, it is useless for symbol-parsing. so wo had better recognize 
such hugetlbfs area, and rename it's filename to /tmp/perf-%d.map, just as anon-memory
and no_dso_memory do. 

This patch imports a new function named is_hugetlb_memory to check if this memory
is from hugetlbfs. If true, change its name.  

After this patch:
#perf report 
   86.96%     0.00%  hugepage-mmap  perf-182.map        [.] 0x000000200000005c 
   86.96%     0.00%  hugepage-mmap  perf-182.map        [.] 0x00000020000000ac

We can add maps info to perf-182.map for further symbols-parsing.

Signed-off-by: Hou Pengyang <houpengyang@huawei.com>
---
 tools/perf/util/map.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index b5a5e9c..796db08 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -15,6 +15,7 @@
 #include "debug.h"
 #include "machine.h"
 #include <linux/string.h>
+#include <sys/mman.h>
 
 static void __maps__insert(struct maps *maps, struct map *map);
 
@@ -43,6 +44,11 @@ static inline int is_android_lib(const char *filename)
 	       !strncmp(filename, "/system/lib", 11);
 }
 
+static inline int is_hugetlb_memory(u32 flags)
+{
+	return flags & MAP_HUGETLB;
+}
+
 static inline bool replace_android_lib(const char *filename, char *newfilename)
 {
 	const char *libname;
@@ -151,12 +157,13 @@ struct map *map__new(struct machine *machine, u64 start, u64 len,
 	if (map != NULL) {
 		char newfilename[PATH_MAX];
 		struct dso *dso;
-		int anon, no_dso, vdso, android;
+		int anon, no_dso, vdso, android, hugetlb;
 
 		android = is_android_lib(filename);
 		anon = is_anon_memory(filename);
 		vdso = is_vdso_map(filename);
 		no_dso = is_no_dso_memory(filename);
+		hugetlb = is_hugetlb_memory(flags);
 
 		map->maj = d_maj;
 		map->min = d_min;
@@ -165,7 +172,7 @@ struct map *map__new(struct machine *machine, u64 start, u64 len,
 		map->prot = prot;
 		map->flags = flags;
 
-		if ((anon || no_dso) && type == MAP__FUNCTION) {
+		if ((anon || no_dso || hugetlb) && type == MAP__FUNCTION) {
 			snprintf(newfilename, sizeof(newfilename), "/tmp/perf-%d.map", pid);
 			filename = newfilename;
 		}
-- 
1.8.3.4

Re: [RFC] perf tools: Add hugetlbfs memory recognition

[Hou Pengyang]

On 2015/6/27 16:49, Hou Pengyang wrote:
> Maps for JIT is helpful for symbols-parsing for anon-executable-memory.
> What we need to do is to add (START, SIZE, symbolname) to /tmp/perf-%d.map
> (%d = pid of process), and perf would parse symbol located in this area
> according to /tmp/perf-%d.map. It works well for normal mmap.
>
> However, when we alloc such memory from hugetlbfs by the following code:
>
> ......
>
>      fd = open("/mnt/huge/hugepagefile", O_CREAT | O_RDWR, 0755);
>      if (fd < 0) {
>          perror("Open failed");
>          exit(1);
>      }
>      addr = mmap(ADDR, LENGTH, PROT_READ | PROT_WRITE |     \
>          PROT_EXEC, MAP_SHARED, fd, 0);
>
> ......
> where hugetlbfs is mounted in /mnt/huge. Symbols could not be parsed correctly:
>
> #perf report
>     86.96%     0.00%  hugepage-mmap  hugepagefile      [.] 0xffffffe00000005c
>     86.96%     0.00%  hugepage-mmap  hugepagefile      [.] 0xffffffe0000000ac
>
> This is because such mmap area's file backed is "/mnt/huge/hugepagefile", a node
> in pseudo filesystem, it is useless for symbol-parsing. so wo had better recognize
> such hugetlbfs area, and rename it's filename to /tmp/perf-%d.map, just as anon-memory
> and no_dso_memory do.
>
> This patch imports a new function named is_hugetlb_memory to check if this memory
> is from hugetlbfs. If true, change its name.
>
> After this patch:
> #perf report
>     86.96%     0.00%  hugepage-mmap  perf-182.map        [.] 0x000000200000005c
>     86.96%     0.00%  hugepage-mmap  perf-182.map        [.] 0x00000020000000ac
>
> We can add maps info to perf-182.map for further symbols-parsing.
>
> Signed-off-by: Hou Pengyang <houpengyang@huawei.com>
> ---
>   tools/perf/util/map.c | 11 +++++++++--
>   1 file changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
> index b5a5e9c..796db08 100644
> --- a/tools/perf/util/map.c
> +++ b/tools/perf/util/map.c
> @@ -15,6 +15,7 @@
>   #include "debug.h"
>   #include "machine.h"
>   #include <linux/string.h>
> +#include <sys/mman.h>
>
>   static void __maps__insert(struct maps *maps, struct map *map);
>
> @@ -43,6 +44,11 @@ static inline int is_android_lib(const char *filename)
>   	       !strncmp(filename, "/system/lib", 11);
>   }
>
> +static inline int is_hugetlb_memory(u32 flags)
> +{
> +	return flags & MAP_HUGETLB;
> +}

There is something about MAP_HUGETLB.

In this patch, we check if a mmap area is hugetlbfs area by MAP_HUGETLB,
a bit in MMAP2 event.

However, if mmap area is hugetlb related, MAP_HUGETLB does not always 
appear. Because, there are two ways generating MMAP2 event.

1) when a new vm_area_struct is created, its info would be exported
    as a MMAP2 event.
2) perf reads /proc/pid/maps for generating MMAP2 event.

MAP_HUGETLB appears if MMAP2 event is generated on situation 1),
while not on situation 2).

This is because on situation 2), perf reads /proc/pid/maps, which
contains only PROT_READ/WRITE/EXEC, MAP_SHARED/MAP_PRIVATE, while more 
details appear in /proc/pid/smaps, such as MAP_HUGETLB.

So I wonder if there is a need to read /proc/pid/smaps instead of 
/proc/pid/maps to generate MMAP2 event. Or we should solve the problem 
by another way?

Thanks!

> +
>   static inline bool replace_android_lib(const char *filename, char *newfilename)
>   {
>   	const char *libname;
> @@ -151,12 +157,13 @@ struct map *map__new(struct machine *machine, u64 start, u64 len,
>   	if (map != NULL) {
>   		char newfilename[PATH_MAX];
>   		struct dso *dso;
> -		int anon, no_dso, vdso, android;
> +		int anon, no_dso, vdso, android, hugetlb;
>
>   		android = is_android_lib(filename);
>   		anon = is_anon_memory(filename);
>   		vdso = is_vdso_map(filename);
>   		no_dso = is_no_dso_memory(filename);
> +		hugetlb = is_hugetlb_memory(flags);
>
>   		map->maj = d_maj;
>   		map->min = d_min;
> @@ -165,7 +172,7 @@ struct map *map__new(struct machine *machine, u64 start, u64 len,
>   		map->prot = prot;
>   		map->flags = flags;
>
> -		if ((anon || no_dso) && type == MAP__FUNCTION) {
> +		if ((anon || no_dso || hugetlb) && type == MAP__FUNCTION) {
>   			snprintf(newfilename, sizeof(newfilename), "/tmp/perf-%d.map", pid);
>   			filename = newfilename;
>   		}
>

Re: [RFC] perf tools: Add hugetlbfs memory recognition

[Arnaldo Carvalho de Melo]

Em Sat, Jun 27, 2015 at 05:08:20PM +0800, Hou Pengyang escreveu:
> There is something about MAP_HUGETLB.
 
> In this patch, we check if a mmap area is hugetlbfs area by MAP_HUGETLB,
> a bit in MMAP2 event.
 
> However, if mmap area is hugetlb related, MAP_HUGETLB does not always
> appear. Because, there are two ways generating MMAP2 event.
 
> 1) when a new vm_area_struct is created, its info would be exported
>    as a MMAP2 event.
> 2) perf reads /proc/pid/maps for generating MMAP2 event.
 
> MAP_HUGETLB appears if MMAP2 event is generated on situation 1),
> while not on situation 2).
 
> This is because on situation 2), perf reads /proc/pid/maps, which
> contains only PROT_READ/WRITE/EXEC, MAP_SHARED/MAP_PRIVATE, while more
> details appear in /proc/pid/smaps, such as MAP_HUGETLB.

Humm:

[root@zoo ~]# wc -l /proc/`pidof firefox`/maps
934 /proc/4551/maps
[root@zoo ~]# wc -l /proc/`pidof firefox`/smaps
14944 /proc/4551/smaps
[root@zoo ~]#
 
> So I wonder if there is a need to read /proc/pid/smaps instead of
> /proc/pid/maps to generate MMAP2 event. Or we should solve the problem by
> another way?

Doing some research now...

- Arnaldo

Re: [RFC] perf tools: Add hugetlbfs memory recognition

[Arnaldo Carvalho de Melo]

Em Mon, Jun 29, 2015 at 10:23:29AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Sat, Jun 27, 2015 at 05:08:20PM +0800, Hou Pengyang escreveu:
> > There is something about MAP_HUGETLB.
>  
> > In this patch, we check if a mmap area is hugetlbfs area by MAP_HUGETLB,
> > a bit in MMAP2 event.
>  
> > However, if mmap area is hugetlb related, MAP_HUGETLB does not always
> > appear. Because, there are two ways generating MMAP2 event.
>  
> > 1) when a new vm_area_struct is created, its info would be exported
> >    as a MMAP2 event.
> > 2) perf reads /proc/pid/maps for generating MMAP2 event.
>  
> > MAP_HUGETLB appears if MMAP2 event is generated on situation 1),
> > while not on situation 2).
>  
> > This is because on situation 2), perf reads /proc/pid/maps, which
> > contains only PROT_READ/WRITE/EXEC, MAP_SHARED/MAP_PRIVATE, while more
> > details appear in /proc/pid/smaps, such as MAP_HUGETLB.
> 
> Humm:
> 
> [root@zoo ~]# wc -l /proc/`pidof firefox`/maps
> 934 /proc/4551/maps
> [root@zoo ~]# wc -l /proc/`pidof firefox`/smaps
> 14944 /proc/4551/smaps
> [root@zoo ~]#
>  
> > So I wonder if there is a need to read /proc/pid/smaps instead of
> > /proc/pid/maps to generate MMAP2 event. Or we should solve the problem by
> > another way?
> 
> Doing some research now...

Bummer, seems that only smaps -> ... -> show_smap_vma_flags() will
expose that to userspace...

Perhaps we can look at some global stat for HugeTLB fs to figure out if
we really, really need to parse smaps instead of just maps? I.e. in my
system, a desktop one, F21, I have:

[root@zoo ~]# find /proc -name smaps | xargs grep -w ht
grep: /proc/31971/task/31971/smaps: No such file or directory
grep: /proc/31971/smaps: No such file or directory
[root@zoo ~]#

No "ht" HugeTLB areas, so no need to parse 16 times more bytes than by
just using /proc/PID/maps.

- Arnaldo

Re: [RFC] perf tools: Add hugetlbfs memory recognition

[Hou Pengyang]

On 2015/6/29 21:42, Arnaldo Carvalho de Melo wrote:
> Em Mon, Jun 29, 2015 at 10:23:29AM -0300, Arnaldo Carvalho de Melo escreveu:
>> Em Sat, Jun 27, 2015 at 05:08:20PM +0800, Hou Pengyang escreveu:
>>> There is something about MAP_HUGETLB.
>>
>>> In this patch, we check if a mmap area is hugetlbfs area by MAP_HUGETLB,
>>> a bit in MMAP2 event.
>>
>>> However, if mmap area is hugetlb related, MAP_HUGETLB does not always
>>> appear. Because, there are two ways generating MMAP2 event.
>>
>>> 1) when a new vm_area_struct is created, its info would be exported
>>>     as a MMAP2 event.
>>> 2) perf reads /proc/pid/maps for generating MMAP2 event.
>>
>>> MAP_HUGETLB appears if MMAP2 event is generated on situation 1),
>>> while not on situation 2).
>>
>>> This is because on situation 2), perf reads /proc/pid/maps, which
>>> contains only PROT_READ/WRITE/EXEC, MAP_SHARED/MAP_PRIVATE, while more
>>> details appear in /proc/pid/smaps, such as MAP_HUGETLB.
>>
>> Humm:
>>
>> [root@zoo ~]# wc -l /proc/`pidof firefox`/maps
>> 934 /proc/4551/maps
>> [root@zoo ~]# wc -l /proc/`pidof firefox`/smaps
>> 14944 /proc/4551/smaps
>> [root@zoo ~]#
>>
>>> So I wonder if there is a need to read /proc/pid/smaps instead of
>>> /proc/pid/maps to generate MMAP2 event. Or we should solve the problem by
>>> another way?
>>
>> Doing some research now...
>
> Bummer, seems that only smaps -> ... -> show_smap_vma_flags() will
> expose that to userspace...
>
> Perhaps we can look at some global stat for HugeTLB fs to figure out if
> we really, really need to parse smaps instead of just maps? I.e. in my
> system, a desktop one, F21, I have:
>

It seems no other info tell us if one process is using hugetlbfs.
So how about an option to tell perf explicitly which file is from
hugetlbfs, like:

./perf report --hugetlb-file=/mnt/huge/hugepagefile

So that, we can check if a mmap area is from hugetlbfs by its name
instead of MAP_HUGETLB.

> [root@zoo ~]# find /proc -name smaps | xargs grep -w ht
> grep: /proc/31971/task/31971/smaps: No such file or directory
> grep: /proc/31971/smaps: No such file or directory
> [root@zoo ~]#
>
> No "ht" HugeTLB areas, so no need to parse 16 times more bytes than by
> just using /proc/PID/maps.



>
> - Arnaldo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
> .
>

Re: [RFC] perf tools: Add hugetlbfs memory recognition

[Arnaldo Carvalho de Melo]

Em Tue, Jun 30, 2015 at 05:33:05PM +0800, Hou Pengyang escreveu:
> On 2015/6/29 21:42, Arnaldo Carvalho de Melo wrote:
> >Em Mon, Jun 29, 2015 at 10:23:29AM -0300, Arnaldo Carvalho de Melo escreveu:
> >>Em Sat, Jun 27, 2015 at 05:08:20PM +0800, Hou Pengyang escreveu:
> >>>So I wonder if there is a need to read /proc/pid/smaps instead of
> >>>/proc/pid/maps to generate MMAP2 event. Or we should solve the problem by
> >>>another way?
> >>
> >>Doing some research now...
> >
> >Bummer, seems that only smaps -> ... -> show_smap_vma_flags() will
> >expose that to userspace...
> >
> >Perhaps we can look at some global stat for HugeTLB fs to figure out if
> >we really, really need to parse smaps instead of just maps? I.e. in my
> >system, a desktop one, F21, I have:
> >
> 
> It seems no other info tell us if one process is using hugetlbfs.
> So how about an option to tell perf explicitly which file is from
> hugetlbfs, like:
> 
> ./perf report --hugetlb-file=/mnt/huge/hugepagefile
> 
> So that, we can check if a mmap area is from hugetlbfs by its name
> instead of MAP_HUGETLB.

Well, we _can_ detect this automatically, its just a matter of parsing
/proc/PID/smaps, right?

What I was discussing was a way only parse smaps when we know we need
to, i.e. when we, for instance, parsing /proc/PID/maps, find a map that
we think may be a hugetlb one, maybe some other way to tell us that
hugetlb maps are in place, looking at the hugetlbfs stats somehow?

- Arnaldo

Re: [RFC] perf tools: Add hugetlbfs memory recognition

[Hou Pengyang]

On 2015/6/30 22:50, Arnaldo Carvalho de Melo wrote:
> Em Tue, Jun 30, 2015 at 05:33:05PM +0800, Hou Pengyang escreveu:
>> On 2015/6/29 21:42, Arnaldo Carvalho de Melo wrote:
>>> Em Mon, Jun 29, 2015 at 10:23:29AM -0300, Arnaldo Carvalho de Melo escreveu:
>>>> Em Sat, Jun 27, 2015 at 05:08:20PM +0800, Hou Pengyang escreveu:
>>>>> So I wonder if there is a need to read /proc/pid/smaps instead of
>>>>> /proc/pid/maps to generate MMAP2 event. Or we should solve the problem by
>>>>> another way?
>>>>
>>>> Doing some research now...
>>>
>>> Bummer, seems that only smaps -> ... -> show_smap_vma_flags() will
>>> expose that to userspace...
>>>
>>> Perhaps we can look at some global stat for HugeTLB fs to figure out if
>>> we really, really need to parse smaps instead of just maps? I.e. in my
>>> system, a desktop one, F21, I have:
>>>
>>
>> It seems no other info tell us if one process is using hugetlbfs.
>> So how about an option to tell perf explicitly which file is from
>> hugetlbfs, like:
>>
>> ./perf report --hugetlb-file=/mnt/huge/hugepagefile
>>
>> So that, we can check if a mmap area is from hugetlbfs by its name
>> instead of MAP_HUGETLB.
>
> Well, we _can_ detect this automatically, its just a matter of parsing
> /proc/PID/smaps, right?
>
> What I was discussing was a way only parse smaps when we know we need
> to, i.e. when we, for instance, parsing /proc/PID/maps, find a map that
> we think may be a hugetlb one, maybe some other way to tell us that
> hugetlb maps are in place, looking at the hugetlbfs stats somehow?
>

Another solution, we can parse /proc/self/mounts. Here is the
/proc/self/mounts in my system:

.....
/dev/root / ext2 rw,relatime,errors=remount-ro 0 0
tracefs /sys/kernel/debug/tracing tracefs rw,relatime 0 0
nodev /mnt/huge hugetlbfs rw,relatime 0 0
....

from /proc/self/mounts, we can get mount point of hugetlbfs.
in my system, it is "/mnt/huge". So that, when mmap2 event
comes from userspace, we can compare prefix of filename with
hugetlbfs mount point (e.g "/mnt/huge") to check if it is a
file in hugetlbfs. if it is, set MAP_HUGELTB, otherwise, not set.

There is no need to parse /proc/PID/smaps now, what's more,
it is not difficult to parse "/proc/self/mounts".

> - Arnaldo
>
> .
>

Re: [RFC] perf tools: Add hugetlbfs memory recognition

[Arnaldo Carvalho de Melo]

Em Fri, Jul 03, 2015 at 06:21:48PM +0800, Hou Pengyang escreveu:
> On 2015/6/30 22:50, Arnaldo Carvalho de Melo wrote:
> >Well, we _can_ detect this automatically, its just a matter of parsing
> >/proc/PID/smaps, right?

<SNIP>

> >What I was discussing was a way only parse smaps when we know we need
> >to, i.e. when we, for instance, parsing /proc/PID/maps, find a map that
> >we think may be a hugetlb one, maybe some other way to tell us that
> >hugetlb maps are in place, looking at the hugetlbfs stats somehow?

<SNIP>

> from /proc/self/mounts, we can get mount point of hugetlbfs.
> in my system, it is "/mnt/huge". So that, when mmap2 event
> comes from userspace, we can compare prefix of filename with
> hugetlbfs mount point (e.g "/mnt/huge") to check if it is a
> file in hugetlbfs. if it is, set MAP_HUGELTB, otherwise, not set.
 
> There is no need to parse /proc/PID/smaps now, what's more,
> it is not difficult to parse "/proc/self/mounts".

A-ha! please take a look at using the find_mountpoint() function
in tools/lib/api/fs/, I guess that should be enough for you?

- Arnaldo