Re: [Qemu-devel] Benchmarking linux-user performance

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Emilio G. Cota (cota@braap.org) wrote:
> Hi all,
> 
> Inspired by SimBench[1], I have written a set of scripts ("DBT-bench")
> to easily obtain and plot performance numbers for linux-user.
> 
> The (Perl) scripts are available here:
>   https://github.com/cota/dbt-bench
> [ It's better to clone with --recursive because the benchmarks
> (NBench) are pulled as a submodule. ]
> 
> I'm using NBench because (1) it's just a few files and they take
> very little time to run (~5min per QEMU version, if performance
> on the host machine is stable), (2) AFAICT its sources are in the
> public domain (whereas SPEC's sources cannot be redistributed),
> and (3) with NBench I get results similar to SPEC's.

Does NBench include anything with lots of small processes, or a large
chunk of code.  Using benchmarks with small code tends to skew DBT optimisations
towards very heavy block optimisation that dont work in real applications where
the cost of translation can hurt if it's too high.

> Here are linux-user performance numbers from v1.0 to v2.8 (higher
> is better):
> 
>                         x86_64 NBench Integer Performance
>                  Host: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz                
>                                                                                
>   36 +-+-+---+---+---+--+---+---+---+---+---+---+---+---+--+---+---+---+-+-+   
>      |   +   +   +   +  +   +   +   +   +   +   +   +   +  +   +   +  ***  |   
>   34 +-+                                                             #*A*+-+   
>      |                                                            *A*      |   
>   32 +-+                                                          #      +-+   
>   30 +-+                                                          #      +-+   
>      |                                                           #         |   
>   28 +-+                                                        #        +-+   
>      |                                 *A*#*A*#*A*#*A*#*A*#     #          |   
>   26 +-+                   *A*#*A*#***#    ***         ******#*A*        +-+   
>      |                     #       *A*                    *A* ***          |   
>   24 +-+                  #                                              +-+   
>   22 +-+                 #                                               +-+   
>      |             #*A**A*                                                 |   
>   20 +-+       #*A*                                                      +-+   
>      |  *A*#*A*  +   +  +   +   +   +   +   +   +   +   +  +   +   +   +   |   
>   18 +-+-+---+---+---+--+---+---+---+---+---+---+---+---+--+---+---+---+-+-+   
>        v1.v1.1v1.2v1.v1.4v1.5v1.6v1.7v2.0v2.1v2.2v2.3v2.v2.5v2.6v2.7v2.8.0     
>                                   QEMU version                                 

Nice, there was someone on list complaining about 2.6 being slower for them.

>                      x86_64 NBench Floating Point Performance                  
>                   Host: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz               
>                                                                                
>   1.88 +-+-+---+--+---+---+---+--+---+---+---+---+--+---+---+---+--+---+-+-+   
>        |   +   +  +  *A*#*A*  +  +   +   +   +   +  +   +   +   +  +   +   |   
>   1.86 +-+           *** ***                                             +-+   
>        |            #       #   *A*#***                                    |   
>        |      *A*# #         # ##   *A*                                    |   
>   1.84 +-+    #  *A*         *A*      #                                  +-+   
>        |      #                        #                              *A*  |   
>   1.82 +-+   #                          #                            ##  +-+   
>        |     #                          *A*#                        #      |   
>    1.8 +-+  #                               #  #*A*               *A*    +-+   
>        |    #                               *A*   #                #       |   
>   1.78 +-+*A*                                      #       *A*    #      +-+   
>        |                                           #   ***#  #    #        |   
>        |                                           *A*#*A*    #  #         |   
>   1.76 +-+                                         ***         # #       +-+   
>        |   +   +  +   +   +   +  +   +   +   +   +  +   +   +  *A* +   +   |   
>   1.74 +-+-+---+--+---+---+---+--+---+---+---+---+--+---+---+---+--+---+-+-+   
>          v1.v1.v1.2v1.3v1.4v1.v1.6v1.7v2.0v2.1v2.v2.3v2.4v2.5v2.v2.7v2.8.0     
>                                    QEMU version                                

I'm assuming the dips are where QEMU fixed something and cared about corner
cases/accuracy?

Dave

> Same plots, in PNG: http://imgur.com/a/nF7Ls
> 
> These plots are obtained simply by running
> 	$ QEMU_PATH=path/to/qemu QEMU_ARCH=x86_64 make -j
> from dbt-bench, although note that some user intervention was needed
> to compile old QEMU versions.
> 
> I think having some well-defined, easy-to-run benchmarks (even
> if far from perfect, like these) to aid development is better
> than not having any. My hope is that having these will encourage
> future performance improvements to the emulation loop and TCG -- or
> at least serve as a warning when performance regresses excessively :-)
> 
> Let me know if you find this work useful.
> 
> Thanks,
> 
> 		Emilio
> 
> [1] https://bitbucket.org/simbench/simbench
> Simbench's authors have a paper on it, although it is not publicly
> available yet (will be presented at the ISPASS'17 conference in April).
> The abstract can be accessed here though: http://tinyurl.com/hahb4yj
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK