-fno-omit-frame-pointer

-fno-omit-frame-pointer

[Sage Weil]

Hey Mark,

I just watched Brendan Gregg's talk at SCALE (which was extremely good) 
and one of the things he mentioned is the importance of 
-fno-omit-frame-pointer.  I notice we are setting it via do_autogen.sh 
(i.e., for developers) but not in the production builds.  Should we be?  
I think it's important to be able to get meaningful callgraph output from 
production deployments and I suspect there is minimal performance impact.

http://www.slideshare.net/brendangregg/scale2015-linux-perfprofiling

sage

Re: -fno-omit-frame-pointer

[Alex Elsayed]

Sage Weil wrote:

> Hey Mark,
> 
> I just watched Brendan Gregg's talk at SCALE (which was extremely good)
> and one of the things he mentioned is the importance of
> -fno-omit-frame-pointer.  I notice we are setting it via do_autogen.sh
> (i.e., for developers) but not in the production builds.  Should we be?
> I think it's important to be able to get meaningful callgraph output from
> production deployments and I suspect there is minimal performance impact.
> 
> http://www.slideshare.net/brendangregg/scale2015-linux-perfprofiling

I personally favor DWARF-based unwinding - for one, on register-starved 
architectures such as 32-bit x86, the frame pointer itself can skew 
performance.

With regard to it inflating instance size, that's something that can be 
mitigated - via split debuginfo (only pull in the ones you actually need), 
compressed debug info (zlib -9 shrinks it a good bit), and semantic 
deduplication/compression with dwz.

Re: -fno-omit-frame-pointer

[Mark Nelson]

On 02/21/2015 03:43 PM, Sage Weil wrote:
> Hey Mark,
>
> I just watched Brendan Gregg's talk at SCALE (which was extremely good)
> and one of the things he mentioned is the importance of
> -fno-omit-frame-pointer.  I notice we are setting it via do_autogen.sh
> (i.e., for developers) but not in the production builds.  Should we be?
> I think it's important to be able to get meaningful callgraph output from
> production deployments and I suspect there is minimal performance impact.
>
> http://www.slideshare.net/brendangregg/scale2015-linux-perfprofiling
>
> sage
>

Ah, Brendan Gregg is quite good.  He moved over to the Linux world when 
he went to work for Netflix after being a Solaris guy forever at Joyent. 
  He's probably one of the most knowledgeable people in the world 
regarding dtrace and in a short time has become very good with perf as 
well.  He wrote a very nice perf cheat sheet here:

http://www.brendangregg.com/perf.html

I didn't realize our production builds skip -fno-omit-frame-pointer.  I 
swear we looked at that like 2 years ago and enabled it to try and make 
perf show symbols properly.  Maybe I'm misremembering.  The performance 
impact discussion is interesting.  I've seen some people say up to 20% 
with GCC but others say it really doesn't matter.  I'd be curious to 
hear what people's opinions these days are.

I've still had the best luck using DWARF though.  I was able to get much 
better stack traces with it.  A semi-modern kernel with libunwind 
support in perf is needed though.  I just checked one of our RHEL7 and 
it sadly doesn't look like they have it linked.  We really should hassle 
them to get this in.  Perf on our Ubuntu 14.04 does use libunwind, 
primarily due to the kernel bug we submitted:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1248289

Mark

Re: -fno-omit-frame-pointer

[Sage Weil]

I got Brendan's flamegraphs to work without any changes on the rex fedora 
dev nodes.  See

	http://ceph.com/sage/foo.svg


This is about 10 seconds of a rados bench on a vstart cluster, so nothing 
fancy.  It's a pretty easy way to visualize where the time is spent, 
though.

A few things I noticed:

 - lots of time in DBObjectMap::set_keys (2%).  a surprising amount (>1%) 
in setxattr (kernel), too.  Hopefully NewStore can do better here.

 - the ReplicatedPG::do_request view is interesting, too.. not 
surprisingly a fair bit of time spent with the PG logs.

 - 1% spend on writing pg info and log (PG::write_if_dirty).  most of that 
in encode

sage

Re: -fno-omit-frame-pointer

[Sage Weil]

On Mon, 23 Feb 2015, Sage Weil wrote:
> I got Brendan's flamegraphs to work without any changes on the rex fedora 
> dev nodes.  See
> 
> 	http://ceph.com/sage/foo.svg

 cd ~/src
 git clone https://github.com/brendangregg/FlameGraph.git
 cd ~/src/ceph/src
 ./vstart -d -n -x -l
 ./rados -p rbd bench 30 write -b 123 &
 sudo perf record -g
 ^C
 sudo perf script | ~/src/FlameGraph/stackcollapse-perf.pl | ~/src/FlameGraph/flamegraph.pl > /tmp/foo.svg

...and view in firefox.

sage


> 
> 
> This is about 10 seconds of a rados bench on a vstart cluster, so nothing 
> fancy.  It's a pretty easy way to visualize where the time is spent, 
> though.
> 
> A few things I noticed:
> 
>  - lots of time in DBObjectMap::set_keys (2%).  a surprising amount (>1%) 
> in setxattr (kernel), too.  Hopefully NewStore can do better here.
> 
>  - the ReplicatedPG::do_request view is interesting, too.. not 
> surprisingly a fair bit of time spent with the PG logs.
> 
>  - 1% spend on writing pg info and log (PG::write_if_dirty).  most of that 
> in encode
> 
> sage
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: -fno-omit-frame-pointer

[Mark Nelson]

On 02/21/2015 05:27 PM, Mark Nelson wrote:
> I've still had the best luck using DWARF though.  I was able to get much
> better stack traces with it.  A semi-modern kernel with libunwind
> support in perf is needed though.  I just checked one of our RHEL7 and
> it sadly doesn't look like they have it linked.  We really should hassle
> them to get this in.  Perf on our Ubuntu 14.04 does use libunwind,
> primarily due to the kernel bug we submitted:
>
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1248289

FWIW, I followed up with the RHEL7 kernel guys who pointed out that 
while RHEL 7.0 doesn't have dwarf support in perf, 7.1 does, and it's 
using what appears to be a better libdw unwind implementation:

http://lwn.net/Articles/579508/

>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html