Re: Xen optimization

From: Andrii Anisov <andrii.anisov@gmail.com>
To: Stefano Stabellini <sstabellini@kernel.org>
Cc: nd@arm.com, andrii_anisov@epam.com,
	Milan Boberic <milanboberic94@gmail.com>,
	Dario Faggioli <dfaggioli@suse.com>,
	Julien Grall <julien.grall@arm.com>,
	Meng Xu <xumengpanda@gmail.com>,
	xen-devel@lists.xenproject.org,
	Stefano Stabellini <stefano.stabellini@xilinx.com>
Subject: Re: Xen optimization
Date: Mon, 10 Dec 2018 12:58:20 +0200	[thread overview]
Message-ID: <bb88ca15-381c-014c-2f6c-e5de33b4b15a@gmail.com> (raw)
In-Reply-To: <alpine.DEB.2.10.1811271320170.4577@sstabellini-ThinkPad-X260>

Hello All,

On 27.11.18 23:27, Stefano Stabellini wrote:
> See the following:
> 
> https://marc.info/?l=xen-devel&m=148668817704668
So I did port that stuff to the current staging [1].
Also, the correspondent tbm, itself is here [2].
Having 4 big cores on my SoC I run XEN with the following command line:

     dom0_mem=3G console=dtuart dtuart=serial0 dom0_max_vcpus=2 bootscrub=0 loglvl=all cpufreq=none tbuf_size=8192 loglvl=all/none guest_loglvl=all/none

The TBM's domain configuration file is as following:

     seclabel='system_u:system_r:domU_t'
     name = "DomP"
     kernel = "/home/root/ctest-bare.bin"
     extra = "console=hvc0 rw"
     memory = 128
     vcpus = 1
     cpus = "3"

This gives me setup where Domain-0 runs on cores 0 and 1 solely and TBM runs exclusively on core 3. So that we can rely that it shows us a pure IRQ latency of hypervisor.
My board is Renesas Salvator-X with H3 ES3.0 SoC and 8GB RAM. Generic timer runs at 8.333 MHz freq, what gives my 120ns resolution for measurements.
XEN hypervisor is build without debug and TBM does wfi in the idle loop for all experiments.
With that setup IRQ latency numbers are (in ns):

Old vgic:
                         AVG     MIN     MAX     WARM MAX
credit, vwfi=trap       7706    7560    9480    8400
credit, vwfi=native     2908    2880    3120    4800
credit2, vwfi=trap      7221    7200    9240    7440
credit2, vwfi=native    2906    2880    3120    5040

New vgic:
                         AVG     MIN     MAX     WARM MAX
credit, vwfi=trap       8481    8040    10200   8880
credit, vwfi=native     4115    3960    4800    4200
credit2, vwfi=trap      8425    8400    9600    9000
credit2, vwfi=native    4227    3960    5040    4680

Here we can see that the new vgic underperforms the old one in a trivial use-case modeled with TBM.

Old vgic with optimizations [3] (without [4], because it breaks the setup):
                         AVG     MIN     MAX     WARM MAX
credit, vwfi=trap       7309    7080    8760    7680
credit, vwfi=native     3007    3000    4320    3120
credit2, vwfi=trap      6877    6720    8880    7200
credit2, vwfi=native    2680    2640    4440    2880

[1] https://github.com/aanisov/xen/tree/4tbm
[2] https://github.com/aanisov/tbm/commits/4xen
[3] https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg03328.html
[4] https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg03288.html

-- 
Sincerely,
Andrii Anisov.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel