Re: [Qemu-devel] Benchmarking linux-user performance

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Emilio G. Cota (cota@braap.org) wrote:
> On Fri, Mar 10, 2017 at 11:45:33 +0000, Dr. David Alan Gilbert wrote:
> > * Emilio G. Cota (cota@braap.org) wrote:
> > >   https://github.com/cota/dbt-bench
> > > I'm using NBench because (1) it's just a few files and they take
> > > very little time to run (~5min per QEMU version, if performance
> > > on the host machine is stable), (2) AFAICT its sources are in the
> > > public domain (whereas SPEC's sources cannot be redistributed),
> > > and (3) with NBench I get results similar to SPEC's.
> > 
> > Does NBench include anything with lots of small processes, or a large
> > chunk of code.  Using benchmarks with small code tends to skew DBT optimisations
> > towards very heavy block optimisation that dont work in real applications where
> > the cost of translation can hurt if it's too high.
> 
> Yes this is a valid point.
> 
> I haven't looked at the NBench code in detail, but I'd expect all programs
> in the suite to be small and have hotspots (this is consistent with
> the fact that performance doesn't change even if the TB hash table
> isn't used, i.e. the loops are small enough to remain in tb_jmp_cache.)
> IOW, we'd be mostly measuring the quality of the translated code,
> not the translation overhead.
> 
> It seems that a good benchmark to take translation overhead into account
> would be gcc/perlbench from SPEC (see [1]; ~20% of exec time is spent
> on translation). Unfortunately, none of them can be redistributed.
> 
> I'll consider other options. For instance, I looked today at using golang's
> compilation tests, but they crash under qemu-user. I'll keep looking
> at other options -- the requirement is to have something that is easy
> to build (i.e. gcc is not an option) and that it runs fast.

Yes, needs to be self contained but large enough to be interesting.
Isn't SPECs perlbench just a variant of a standard free benchmark
that can be used?
(Select alternative preferred language).

> A hack that one can do to measure code translation as opposed to execution
> is to disable caching with a 2-liner to avoid insertions to the TB hash
> table and tb_jmp_cache. The problem is that then we basically just
> measure code translation performance, which isn't really realistic
> either.
> 
> In any case, note that most efforts I've seen to compile very good code
> (with QEMU or other cross-ISA DBT), do some sort of profiling so that
> only hot blocks are optimized -- see for example [1] and [2].

Right, and often there's a trade off between an interpret step, and one or
more translate/optimisation steps and have to pick thresholds etc.

Dave

> [1] "Characterization of Dynamic Binary Translation Overhead".
>     Edson Borin and Youfeng Wu. IISWC 2009.
>     http://amas-bt.cs.virginia.edu/2008proceedings/AmasBT2008.pdf#page=4
> 
> [2] "HQEMU: a multi-threaded and retargetable dynamic binary translator
>     on multicores".
>     Ding-Yong Hong, Chun-Chen Hsu, Pen-Chung Yew, Jan-Jan Wu, Wei-Chung Hsu
>     Pangfeng Liu, Chien-Min Wang and Yeh-Ching Chung. CGO 2012.
>     http://www.iis.sinica.edu.tw/papers/dyhong/18239-F.pdf
> 
> 
> > > Here are linux-user performance numbers from v1.0 to v2.8 (higher
> > > is better):
> > > 
> > >                         x86_64 NBench Integer Performance
> > >                  Host: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz                
> > >                                                                                
> > >   36 +-+-+---+---+---+--+---+---+---+---+---+---+---+---+--+---+---+---+-+-+   
> > >      |   +   +   +   +  +   +   +   +   +   +   +   +   +  +   +   +  ***  |   
> > >   34 +-+                                                             #*A*+-+   
> > >      |                                                            *A*      |   
> > >   32 +-+                                                          #      +-+   
> > >   30 +-+                                                          #      +-+   
> > >      |                                                           #         |   
> > >   28 +-+                                                        #        +-+   
> > >      |                                 *A*#*A*#*A*#*A*#*A*#     #          |   
> > >   26 +-+                   *A*#*A*#***#    ***         ******#*A*        +-+   
> > >      |                     #       *A*                    *A* ***          |   
> > >   24 +-+                  #                                              +-+   
> > >   22 +-+                 #                                               +-+   
> > >      |             #*A**A*                                                 |   
> > >   20 +-+       #*A*                                                      +-+   
> > >      |  *A*#*A*  +   +  +   +   +   +   +   +   +   +   +  +   +   +   +   |   
> > >   18 +-+-+---+---+---+--+---+---+---+---+---+---+---+---+--+---+---+---+-+-+   
> > >        v1.v1.1v1.2v1.v1.4v1.5v1.6v1.7v2.0v2.1v2.2v2.3v2.v2.5v2.6v2.7v2.8.0     
> > >                                   QEMU version                                 
> > 
> > Nice, there was someone on list complaining about 2.6 being slower for them.
> > 
> > >                      x86_64 NBench Floating Point Performance                  
> > >                   Host: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz               
> > >                                                                                
> > >   1.88 +-+-+---+--+---+---+---+--+---+---+---+---+--+---+---+---+--+---+-+-+   
> > >        |   +   +  +  *A*#*A*  +  +   +   +   +   +  +   +   +   +  +   +   |   
> > >   1.86 +-+           *** ***                                             +-+   
> > >        |            #       #   *A*#***                                    |   
> > >        |      *A*# #         # ##   *A*                                    |   
> > >   1.84 +-+    #  *A*         *A*      #                                  +-+   
> > >        |      #                        #                              *A*  |   
> > >   1.82 +-+   #                          #                            ##  +-+   
> > >        |     #                          *A*#                        #      |   
> > >    1.8 +-+  #                               #  #*A*               *A*    +-+   
> > >        |    #                               *A*   #                #       |   
> > >   1.78 +-+*A*                                      #       *A*    #      +-+   
> > >        |                                           #   ***#  #    #        |   
> > >        |                                           *A*#*A*    #  #         |   
> > >   1.76 +-+                                         ***         # #       +-+   
> > >        |   +   +  +   +   +   +  +   +   +   +   +  +   +   +  *A* +   +   |   
> > >   1.74 +-+-+---+--+---+---+---+--+---+---+---+---+--+---+---+---+--+---+-+-+   
> > >          v1.v1.v1.2v1.3v1.4v1.v1.6v1.7v2.0v2.1v2.v2.3v2.4v2.5v2.v2.7v2.8.0     
> > >                                    QEMU version                                
> > 
> > I'm assuming the dips are where QEMU fixed something and cared about corner
> > cases/accuracy?
> 
> It'd be hard to say why the numbers vary across versions without running
> a profiler and git bisect. I only know the reason for v2.7, where most if not all
> of the improvement is due to the removal of tb_lock() when executing
> code in qemu-user thanks to the QHT work.
> 
> 		E.
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK