Re: Horrible L2 cache effects from kernel compile

[William Lee Irwin III <wli@holomorphy.com>]

On Mon, Feb 24, 2003 at 04:41:37PM -0800, Dave Hansen wrote:
> c01cc948 334779   8.82177     __copy_to_user_ll
> c015797c 314165   8.27857     d_lookup
> c012c964 157716   4.15598     find_get_page
> c013cae0 156345   4.11985     page_add_rmap
> c0176557 148116   3.90301     .text.lock.inode
> c013cc38 132228   3.48434     page_remove_rmap
> c0176338 90851    2.39402     ext3_dirty_inode
> c0117320 80918    2.13227     scheduler_tick
> c01ccae0 79475    2.09425     atomic_dec_and_lock
> c0138cd4 75367    1.986       do_no_page
> c0139b60 75317    1.98468     vm_enough_memory
> c017d78c 72929    1.92175     start_this_handle

Your profile was upside down. I've re-sorted it.
It probably confused people who were wondering why the numbers
at the top of the profile were lower than the ones below them.


On Mon, Feb 24, 2003 at 04:41:37PM -0800, Dave Hansen wrote:
> a snippet from d_lookup(), annotated.  I've seen oprofile be off by a
> line here, but we can be pretty sure it is in this area.
> 		smp_read_barrier_depends();
> 		/* 106 0.002793% */
> 		dentry = list_entry(tmp, struct dentry, d_hash);
> 		/* if lookup ends up in a different bucket
> 		 * due to concurrent rename, fail it
> 		 */
> 		/* 154991 4.084% */
> 		if (unlikely(dentry->d_bucket != head))
> 			break;
> 		/* to avoid race if dentry keep coming back to original
> 		 * bucket due to double moves
> 		 */
> 		/* 634 0.01671% */
> 		if (unlikely(++lookup_count > max_dentries))
> 			break;
> 		...

Well, you're walking hash chains, you're going to get a lot of cache
misses, about the whole length of the chain's worth. You could try
changing the dcache to use something like a B+ tree or some such
nonsense to reduce its cache footprint.

I'd collect statistics to see if the dcache hash function is badly
distributing things. That's relatively easily fixable if so.


-- wli