From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42788) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bkvd6-0007zP-Jd for qemu-devel@nongnu.org; Fri, 16 Sep 2016 12:04:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bkvd4-0007Bj-TL for qemu-devel@nongnu.org; Fri, 16 Sep 2016 12:04:44 -0400 References: <1473915856-23801-1-git-send-email-wei@redhat.com> <1474028956.3600.39.camel@redhat.com> From: Wei Huang Message-ID: Date: Fri, 16 Sep 2016 11:04:31 -0500 MIME-Version: 1.0 In-Reply-To: <1474028956.3600.39.camel@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH V3 0/2] Add option to configure guest vPMU List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Andrea Bolognani , qemu-arm@nongnu.org Cc: qemu-devel@nongnu.org, peter.maydell@linaro.org, drjones@redhat.com, shannon.zhao@linaro.org On 09/16/2016 07:29 AM, Andrea Bolognani wrote: > On Thu, 2016-09-15 at 01:04 -0400, Wei Huang wrote: >> This patchset adds a pmu=[on/off] option to enable/disable vPMU support >> for guest VM. There are several reasons to justify this option. First, >> vPMU can be problematic for cross-migration between different SoC as perf >> counters are architecture-dependent. It is more flexible to have an option >> to turn it on/off. Secondly Secondly this option matches the "pmu" option >> as supported in libvirt. To make sure backward compatible, a PMU property >> is added to mach-virt machine types. > > [...] > >> Wei Huang (2): >> arm64: Add an option to turn on/off vPMU support >> arm: virt: add PMU property to mach-virt machine type >> >> hw/arm/virt-acpi-build.c | 2 +- >> hw/arm/virt.c | 15 ++++++++++++++- >> target-arm/cpu.c | 22 ++++++++++++++++++++++ >> target-arm/cpu.h | 1 + >> target-arm/cpu64.c | 2 ++ >> target-arm/kvm64.c | 19 +++++++++++++++---- >> 6 files changed, 55 insertions(+), 6 deletions(-) > > This doesn't seem to be working :( > > My guest configuration looks like > > > abologna-rhel73 > 78f74104-65f2-4faf-9faa-26107c1071ef > 2097152 > 2097152 > 4 > > hvm > /usr/share/AAVMF/AAVMF_CODE.fd > /var/lib/libvirt/qemu/nvram/abologna-rhel73_VARS.fd > > > > > > > > > destroy > restart > restart > > /usr/libexec/abologna-qemu-kvm > > > > >
> > > >
> > > > > >
> > > > > > > > > >
> > > > > and based on that configuration, libvirt creates a QEMU > command line that looks like > > LC_ALL=C \ > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \ > QEMU_AUDIO_DRV=none \ > /usr/libexec/abologna-qemu-kvm \ > -name guest=abologna-rhel73,debug-threads=on \ > -S \ > -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-abologna-rhel73/master-key.aes \ > -machine virt-2.7,accel=kvm,usb=off \ > -cpu host,pmu=off \ > -drive file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on \ > -drive file=/var/lib/libvirt/qemu/nvram/abologna-rhel73_VARS.fd,if=pflash,format=raw,unit=1 \ > -m 2048 \ > -realtime mlock=off \ > -smp 4,sockets=4,cores=1,threads=1 \ > -uuid 78f74104-65f2-4faf-9faa-26107c1071ef \ > -nographic \ > -no-user-config \ > -nodefaults \ > -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-abologna-rhel73/monitor.sock,server,nowait \ > -mon chardev=charmonitor,id=monitor,mode=control \ > -rtc base=utc \ > -no-shutdown \ > -boot strict=on \ > -device virtio-serial-device,id=virtio-serial0 \ > -drive file=/var/lib/libvirt/images/abologna-rhel73.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \ > -device virtio-blk-device,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ > -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 \ > -device virtio-net-device,netdev=hostnet0,id=net0,mac=52:54:00:57:76:ad \ > -serial pty \ > -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-5-abologna-rhel73/org.qemu.guest_agent.0,server,nowait \ > -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ > -msg timestamp=on > > Inside the guest, however, even with pmu=off, I can still see > the PMU being initialized during boot > > # dmesg | grep -i pmu > [ 0.000000] ACPI-PMU: Assign CPU 0 girq 23 level 0 > [ 0.000000] ACPI-PMU: Assign CPU 1 girq 23 level 0 > [ 0.000000] ACPI-PMU: Assign CPU 2 girq 23 level 0 > [ 0.000000] ACPI-PMU: Assign CPU 3 girq 23 level 0 > [ 0.087277] ACPI-PMU: Setting up 4 PMUs for CPU type 0 > [ 0.576255] hw perfevents: enabled with armv8_pmuv3 PMU driver, > 5 counters available > > and perf can list the counters > > # perf list | grep Hardware > branch-misses [Hardware event] > cache-misses [Hardware event] > cache-references [Hardware event] > cpu-cycles OR cycles [Hardware event] > instructions [Hardware event] > L1-dcache-load-misses [Hardware cache event] > L1-dcache-loads [Hardware cache event] > L1-dcache-store-misses [Hardware cache event] > L1-dcache-stores [Hardware cache event] > branch-load-misses [Hardware cache event] > branch-loads [Hardware cache event] > mem:[/len][:access] [Hardware breakpoint] > > and use them > > # perf stat true > > Performance counter stats for 'true': > > 0.610960 task-clock (msec) # 0.102 CPUs utilized > 0 context-switches # 0.000 K/sec > 0 cpu-migrations # 0.000 K/sec > 24 page-faults # 0.039 M/sec > 1,383,676 cycles # 2.265 GHz > stalled-cycles-frontend > stalled-cycles-backend > 396,335 instructions # 0.29 insns per cycle > branches > 5,536 branch-misses # 0.00% of all branches > > 0.006013060 seconds time elapsed > > same way it can when using pmu=on or not passing any pmu= > value to QEMU. > > I've tested this both on Moonshot and on ThunderX, the > behavior is the same. > > Am I doing something wrong? I can replicate the problem. To answer your question: No, I think it is a bug caused in this patch. Following code path, it turns out that machvirt_init() is called after ARMCPU->has_pmu property has been configured. So in your example, machvirt->pmu_default_on will overwrite the ARMCPU->has_pmu value to be TRUE (always). Even worse, our desired PMU setting can't be fully covered by a Boolean ARMCPU->has_pmu field because machvirt_init() can't tell ARMCPU->has_pmu is default-OFF or forced-OFF. Here is the matrix of one example: ARMCPU->has_pmu VMC->pmu_default_on vPMU -M virt-2.7 -cpu host default-OFF TRUE ON -M virt-2.7 -cpu host,pmu=off forced-OFF TRUE OFF* -M virt-2.7 -cpu host,pmu=on forced-ON TRUE ON -M virt-2.6 -cpu host default-OFF FALSE OFF -M virt-2.6 -cpu host,pmu=off forced-OFF FALSE OFF -M virt-2.6 -cpu host,pmu=on forced-ON FALSE ON -M virt-2.8+ -cpu host default-OFF FALSE OFF -M virt-2.8+ -cpu host,pmu=off forced-OFF FALSE OFF -M virt-2.8+ -cpu host,pmu=on forced-ON FALSE ON The problematic one is the second line (*). To solve the problem, we might want to change ARMCPU->has_pmu to an int (or enum). > > -- > Andrea Bolognani / Red Hat / Virtualization >