Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

Brian Gerst
2003-03-18T18:20:08Z

Linus Torvalds wrote:
> On Tue, 18 Mar 2003, Kevin Pedretti wrote:
>
>> I wasn't aware of what you state below but it makes sense. What I
>>haven't been able to figure out, and nobody seems to know, is why the
>>rodata section of an executable is placed in the text section and is not
>>page aligned. This seems to be a mixing of code and data on the same
>>page. Maybe it doesn't matter since it is read only?
>
>
> It's a bad idea to share even read-only data, but the impact of read-only
> data is much less that read-write. In particular, you should avoid sharing
> _any_ code and data in the same physical L1 cache-line, since that will be
> a big problem for any CPU with exclusion between the I$ and D$.
>
> HOWEVER, modern x86 CPU's tend to have the I$ be part of the cache
> coherency protocol, so instead of having exclusion they allow sharing as
> long as the D$ isn't actually dirty. In that case it's fine to share
> read-only data and code, although the cache utilization goes down if you
> do a lot of it.
>
> Anyway, as long as they are in separate cache-lines, you should be ok even
> on something with cache exclusion.
>
> When it comes to actually _writing_ to the data, at least on the P4 you
> don't want to have read-write data anywhere _near_ the I$ (somebody
> reported half-page granularity). This is true on crusoe too, btw (at a
> 128-byte granularity).
>
> Anyway, I think gcc should make sure that even the ro-data section is at
> least cacheline-aligned so that it stays away from cachelines used for I$.
> That makes sense even on CPU's that don't have exclusion, since it
> actually gives slightly better L1 cache utilization.
>
> You can run this (stupid) test-program to try. On my P4 I get
>
> empty overhead=320 cycles
> load overhead=0 cycles
> I$ load overhead=0 cycles
> I$ load overhead=0 cycles
> I$ store overhead=264 cycles
>
> and on my PIII I get
>
> empty overhead=74 cycles
> load overhead=8 cycles
> I$ load overhead=8 cycles
> I$ load overhead=8 cycles
> I$ store overhead=103 cycles
>
> and (just for fun) on an old crusoe I get
>
> empty overhead=67 cycles
> load overhead=-9 cycles
> I$ load overhead=-14 cycles
> I$ load overhead=-14 cycles
> I$ store overhead=12 cycles
>
> where that "negative overhead" just shows that we do some strnge things to
> scheduling, and the loop actually ends up faster if it has a load in it
> than without the load..
>
> But you can see that storing to code is a really bad idea. Especially on a
> P4, where the overhead for a store was 264 cycles! (You can also see the
> cost of doing just the empty synchronization and rdtsc - 320 cycles for a
> rdtsc and two locked memory accesses on a P4).
>
> I don't have access to an old Pentium - I think that was the one that had
> the strict exclusion between the L1 I$ and D$, and then you should see the
> I$ load overhead go up.
>
> Linus

Here's a few more data points:

vendor_id : AuthenticAMD
cpu family : 5
model : 8
model name : AMD-K6(tm) 3D processor
stepping : 12
cpu MHz : 451.037
empty overhead=105 cycles
load overhead=-2 cycles
I$ load overhead=30 cycles
I$ load overhead=90 cycles
I$ store overhead=95 cycles

vendor_id : GenuineIntel
cpu family : 6
model : 3
model name : Pentium II (Klamath)
stepping : 3
cpu MHz : 265.913
empty overhead=73 cycles
load overhead=10 cycles
I$ load overhead=10 cycles
I$ load overhead=10 cycles
I$ store overhead=2 cycles

vendor_id : AuthenticAMD
cpu family : 6
model : 6
model name : AMD Athlon(tm) Processor
stepping : 2
cpu MHz : 1409.946
empty overhead=11 cycles
load overhead=5 cycles
I$ load overhead=5 cycles
I$ load overhead=5 cycles
I$ store overhead=826 cycles

The Athlon XP shows really bad behavior when you store to the text area.

--
Brian Gerst