Re: Poor HVM performance with 8 vcpus

[Juergen Gross <juergen.gross@ts.fujitsu.com>]

Tim Deegan wrote:
> At 10:40 +0100 on 07 Oct (1254912041), Juergen Gross wrote:
>> calls to shadow_alloc              438     427     424     480     436     422
>> number of shadow pages in use     2765    2151    2386    2509    4885    1391
>> calls to shadow_free               168     132     185     144     181     105
>> calls to shadow_fault            65271   69132   60495   53756   73363   52449
>> shadow_fault fast path n/p        7347    8081    6713    6134    8521    6112
>> shadow_fault fast path error        14      12      15       3      13      11
>> shadow_fault really guest fault  24004   25723   22815   19709   27049   19190
>> shadow_fault emulates a write     1045     949    1018     995    1015     901
>> shadow_fault fast emulate          424     361     449     348     387     314
>> shadow_fault fixed fault         32503   34264   29624   26689   36641   26096
>> calls to shadow_validate_gl2e      875     748     917     731     795     667
>> calls to shadow_validate_gl3e      481     456     443     491     489     446
>> calls to shadow_validate_gl4e      104      97      95     112     105      95
>> calls to shadow_hash_lookup    2109654 2203254 2228896 2245849 2164727 2309059
>> shadow hash hit in bucket head 2012828 2111164 2161113 2177591 2104099 2242458
>> shadow hash misses                 851     840     841     910     852     838
>> calls to get_shadow_status     2110031 2202828 2228769 2246689 2164213 2309241
>> calls to shadow_hash_insert        438     436     428     481     437     430
>> calls to shadow_hash_delete        168     150     185     154     202     128
>> shadow removes write access        335     324     329     385     330     336
>> shadow writeable: linux high       130     139     152     155     138     149
>> shadow writeable: sl1p           14508   15402   12961   11823   16474   11472
>> shadow writeable brute-force       205     185     177     230     192     187
>> shadow unshadows for fork/exit       9      12      12      12      18      12
>> shadow unshadows a page             10      13      13      13      19      13
>> shadow walks guest tables       647527  727336  649397  646601  659655  621289
>> shadow checks gwalk                526     544     535     550     614     554
>> shadow flush tlb by rem wr perm    235     233     229     268     238     237
>> shadow emulates invlpg           14688   15499   14604   12630   16627   11370
>> shadow OOS fixup adds            14467   15335   13059   11840   16624   11339
>> shadow OOS unsyncs               14467   15335   13058   11840   16624   11339
>> shadow OOS evictions               566     449     565     369     589     336
>> shadow OOS resyncs               14510   15407   12964   11828   16478   11481
>>
>> I don't think the "shadow writable brute-force" is the problem.
>> get_shadow_status seems to be a more critical candidate.
> 
> get_shadow_status is a simple hash lookup to find the shadow of a frame;
> it's expected to happen multiple times per pagefault.  Even so those
> numbers look high.  ~10k guest PT walks per CPU per second, each causing
> 3-4 shadow hash lookups.  That's much higher than the number of
> pagefaults.  
> 
> I take it you reset the performance counters at the start of that run?

Yes.

> 
> Are there any other numbers (outside the shadow stats) that are up
> around 600k/cpu?

No. All other counts are below 100k.

> 
> I wonder whether this is caused by pagetables changing under out feet in
> the shadow fault handler -- in order to avoid taking the shadow lock too
> often, the fault handler walks the pagetables first, then takes the lock
> and double-checks its work.   If the other cpus are aggressively writing
> to the pagetables that this CPU is running on that could cause the
> pagefault handler to spin, locking and unlocking the shadow lock as it
> goes.   I would expect to see the OOS unsyncs number much higher if that
> was the case though.  
> 
> Can you try the attached patch and see if it makes any difference?

Not really. Still 4 vcpus waiting at the shadow lock, performance counters as
follows (test ran with 8 vcpus this time, performance was 2% higher than
without patch, but uncertainty is in the same order).
I copied only the counters for 1 cpu, as the others are similar. Counters are
smaller as before, because performance was less as with 6 vcpus and lock
profiling costs about 20% performance with this lock load (the previous
measurement had no lock profiling support).

calls to shadow_alloc               700
number of shadow pages in use      1986
calls to shadow_free                 48
calls to shadow_fault             33508
shadow_fault fast path n/p         3785
shadow_fault fast path error         10
shadow_fault really guest fault   11290
shadow_fault emulates a write      1313
shadow_fault fast emulate           257
shadow_fault fixed fault          16853
calls to shadow_validate_gl2e       604
calls to shadow_validate_gl3e       786
calls to shadow_validate_gl4e       179
calls to shadow_hash_lookup     1079773
shadow hash hit in bucket head  1030265
shadow hash misses                 1258
calls to get_shadow_status      1079233
calls to shadow_hash_insert         700
calls to shadow_hash_delete          48
shadow removes write access         650
shadow writeable: linux high        624
shadow writeable: sl1p             7459
shadow writeable brute-force         26
shadow unshadows for fork/exit        1
shadow unshadows a page               1
shadow walks guest tables        393663
shadow checks gwalk                1251
shadow flush tlb by rem wr perm     439
shadow emulates invlpg             6212
shadow OOS fixup adds              7330
shadow OOS unsyncs                 7328
shadow OOS evictions                175
shadow OOS resyncs                 7459


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 636 47950
Fujitsu Technolgy Solutions               e-mail: juergen.gross@ts.fujitsu.com
Otto-Hahn-Ring 6                        Internet: ts.fujitsu.com
D-81739 Muenchen                 Company details: ts.fujitsu.com/imprint.html