* [PATCH v4 0/3] Add support for the RAPL MSRs series
@ 2024-03-18 15:12 Anthony Harivel
2024-03-18 15:12 ` [PATCH v4 1/3] qio: add support for SO_PEERCRED for socket channel Anthony Harivel
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Anthony Harivel @ 2024-03-18 15:12 UTC (permalink / raw)
To: pbonzini, mtosatti, berrange
Cc: qemu-devel, vchundur, rjarry, Anthony Harivel
Dear maintainers,
First of all, thank you very much for your review of my patch
[1].
In this version (v4), I have attempted to address all the problems
addressed during the last review. I hope I did not forget anything.
I added more than 400 lines of code, I guess it's time we review that.
However, one question remains unanswered from Friday 1 March pointing
the issue with the location of "/var/local/run/qemu-vmsr-helper.sock",
created by compute_default_paths(). QEMU is not allowed to reach the
socket here.
Thank you again for your continued guidance.
v3 -> v4
--------
- Correct memory leaks with AddressSanitizer
- Add sanity check for QEMU and qemu-vmsr-helper for checking if host is
INTEL and if RAPL is activated.
- Rename poor variables naming for easier comprehension
- Move code that checks Host before creating the VMSR thread
- Get rid of libnuma: create function that read sysfs for reading the
Host topology instead
v2 -> v3
--------
- Move all memory allocations from Clib to Glib
- Compile on *BSD (working on Linux only)
- No more limitation on the virtual package: each vCPU that belongs to
the same virtual package is giving the same results like expected on
a real CPU.
This has been tested topology like:
-smp 4,sockets=2
-smp 16,sockets=4,cores=2,threads=2
v1 -> v2
--------
- To overcome the CVE-2020-8694 a socket communication is created
to a priviliged helper
- Add the priviliged helper (qemu-vmsr-helper)
- Add SO_PEERCRED in qio channel socket
RFC -> v1
---------
- Add vmsr_* in front of all vmsr specific function
- Change malloc()/calloc()... with all glib equivalent
- Pre-allocate all dynamic memories when possible
- Add a Documentation of implementation, limitation and usage
Best regards,
Anthony
[1]: https://lore.kernel.org/all/20240125072214.318382-1-aharivel@redhat.com/#t
Anthony Harivel (3):
qio: add support for SO_PEERCRED for socket channel
tools: build qemu-vmsr-helper
Add support for RAPL MSRs in KVM/Qemu
accel/kvm/kvm-all.c | 27 ++
contrib/systemd/qemu-vmsr-helper.service | 15 +
contrib/systemd/qemu-vmsr-helper.socket | 9 +
docs/specs/index.rst | 1 +
docs/specs/rapl-msr.rst | 155 +++++++
docs/tools/index.rst | 1 +
docs/tools/qemu-vmsr-helper.rst | 89 ++++
include/io/channel.h | 21 +
include/sysemu/kvm.h | 2 +
include/sysemu/kvm_int.h | 30 ++
io/channel-socket.c | 24 +
io/channel.c | 12 +
meson.build | 5 +
target/i386/cpu.h | 8 +
target/i386/kvm/kvm-cpu.c | 7 +
target/i386/kvm/kvm.c | 420 +++++++++++++++++
target/i386/kvm/meson.build | 1 +
target/i386/kvm/vmsr_energy.c | 381 +++++++++++++++
target/i386/kvm/vmsr_energy.h | 97 ++++
tools/i386/qemu-vmsr-helper.c | 564 +++++++++++++++++++++++
tools/i386/rapl-msr-index.h | 28 ++
21 files changed, 1897 insertions(+)
create mode 100644 contrib/systemd/qemu-vmsr-helper.service
create mode 100644 contrib/systemd/qemu-vmsr-helper.socket
create mode 100644 docs/specs/rapl-msr.rst
create mode 100644 docs/tools/qemu-vmsr-helper.rst
create mode 100644 target/i386/kvm/vmsr_energy.c
create mode 100644 target/i386/kvm/vmsr_energy.h
create mode 100644 tools/i386/qemu-vmsr-helper.c
create mode 100644 tools/i386/rapl-msr-index.h
--
2.44.0
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH v4 1/3] qio: add support for SO_PEERCRED for socket channel 2024-03-18 15:12 [PATCH v4 0/3] Add support for the RAPL MSRs series Anthony Harivel @ 2024-03-18 15:12 ` Anthony Harivel 2024-03-20 17:18 ` Daniel P. Berrangé 2024-03-18 15:12 ` [PATCH v4 2/3] tools: build qemu-vmsr-helper Anthony Harivel 2024-03-18 15:12 ` [PATCH v4 3/3] Add support for RAPL MSRs in KVM/Qemu Anthony Harivel 2 siblings, 1 reply; 10+ messages in thread From: Anthony Harivel @ 2024-03-18 15:12 UTC (permalink / raw) To: pbonzini, mtosatti, berrange Cc: qemu-devel, vchundur, rjarry, Anthony Harivel The function qio_channel_get_peercred() returns a pointer to the credentials of the peer process connected to this socket. This credentials structure is defined in <sys/socket.h> as follows: struct ucred { pid_t pid; /* Process ID of the sending process */ uid_t uid; /* User ID of the sending process */ gid_t gid; /* Group ID of the sending process */ }; The use of this function is possible only for connected AF_UNIX stream sockets and for AF_UNIX stream and datagram socket pairs. On platform other than Linux, the function return 0. Signed-off-by: Anthony Harivel <aharivel@redhat.com> --- include/io/channel.h | 21 +++++++++++++++++++++ io/channel-socket.c | 24 ++++++++++++++++++++++++ io/channel.c | 12 ++++++++++++ 3 files changed, 57 insertions(+) diff --git a/include/io/channel.h b/include/io/channel.h index 7986c49c713a..01ad7bd7e430 100644 --- a/include/io/channel.h +++ b/include/io/channel.h @@ -160,6 +160,9 @@ struct QIOChannelClass { void *opaque); int (*io_flush)(QIOChannel *ioc, Error **errp); + void (*io_peerpid)(QIOChannel *ioc, + unsigned int *pid, + Error **errp); }; /* General I/O handling functions */ @@ -981,4 +984,22 @@ int coroutine_mixed_fn qio_channel_writev_full_all(QIOChannel *ioc, int qio_channel_flush(QIOChannel *ioc, Error **errp); +/** + * qio_channel_get_peercred: + * @ioc: the channel object + * @pid: pointer to pid + * @errp: pointer to a NULL-initialized error object + * + * Returns the pid of the peer process connected to this socket. + * + * The use of this function is possible only for connected + * AF_UNIX stream sockets and for AF_UNIX stream and datagram + * socket pairs on Linux. + * Return an error with pid -1 for the non-Linux OS. + * + */ +void qio_channel_get_peerpid(QIOChannel *ioc, + unsigned int *pid, + Error **errp); + #endif /* QIO_CHANNEL_H */ diff --git a/io/channel-socket.c b/io/channel-socket.c index 3a899b060858..fcff92ecc151 100644 --- a/io/channel-socket.c +++ b/io/channel-socket.c @@ -841,6 +841,29 @@ qio_channel_socket_set_cork(QIOChannel *ioc, socket_set_cork(sioc->fd, v); } +static void +qio_channel_socket_get_peerpid(QIOChannel *ioc, + unsigned int *pid, + Error **errp) +{ +#ifdef CONFIG_LINUX + QIOChannelSocket *sioc = QIO_CHANNEL_SOCKET(ioc); + Error *err = NULL; + socklen_t len = sizeof(struct ucred); + + struct ucred cred; + if (getsockopt(sioc->fd, + SOL_SOCKET, SO_PEERCRED, + &cred, &len) == -1) { + error_setg_errno(&err, errno, "Unable to get peer credentials"); + error_propagate(errp, err); + } + *pid = (unsigned int)cred.pid; +#else + error_setg(errp, "Unsupported feature"); + *pid = -1; +#endif +} static int qio_channel_socket_close(QIOChannel *ioc, @@ -938,6 +961,7 @@ static void qio_channel_socket_class_init(ObjectClass *klass, #ifdef QEMU_MSG_ZEROCOPY ioc_klass->io_flush = qio_channel_socket_flush; #endif + ioc_klass->io_peerpid = qio_channel_socket_get_peerpid; } static const TypeInfo qio_channel_socket_info = { diff --git a/io/channel.c b/io/channel.c index a1f12f8e9096..777989bc9a81 100644 --- a/io/channel.c +++ b/io/channel.c @@ -548,6 +548,18 @@ void qio_channel_set_cork(QIOChannel *ioc, } } +void qio_channel_get_peerpid(QIOChannel *ioc, + unsigned int *pid, + Error **errp) +{ + QIOChannelClass *klass = QIO_CHANNEL_GET_CLASS(ioc); + + if (!klass->io_peerpid) { + error_setg(errp, "Channel does not support peer pid"); + return; + } + klass->io_peerpid(ioc, pid, errp); +} off_t qio_channel_io_seek(QIOChannel *ioc, off_t offset, -- 2.44.0 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v4 1/3] qio: add support for SO_PEERCRED for socket channel 2024-03-18 15:12 ` [PATCH v4 1/3] qio: add support for SO_PEERCRED for socket channel Anthony Harivel @ 2024-03-20 17:18 ` Daniel P. Berrangé 0 siblings, 0 replies; 10+ messages in thread From: Daniel P. Berrangé @ 2024-03-20 17:18 UTC (permalink / raw) To: Anthony Harivel; +Cc: pbonzini, mtosatti, qemu-devel, vchundur, rjarry On Mon, Mar 18, 2024 at 04:12:14PM +0100, Anthony Harivel wrote: > The function qio_channel_get_peercred() returns a pointer to the > credentials of the peer process connected to this socket. > > This credentials structure is defined in <sys/socket.h> as follows: > > struct ucred { > pid_t pid; /* Process ID of the sending process */ > uid_t uid; /* User ID of the sending process */ > gid_t gid; /* Group ID of the sending process */ > }; > > The use of this function is possible only for connected AF_UNIX stream > sockets and for AF_UNIX stream and datagram socket pairs. > > On platform other than Linux, the function return 0. > > Signed-off-by: Anthony Harivel <aharivel@redhat.com> > --- > include/io/channel.h | 21 +++++++++++++++++++++ > io/channel-socket.c | 24 ++++++++++++++++++++++++ > io/channel.c | 12 ++++++++++++ > 3 files changed, 57 insertions(+) > > diff --git a/include/io/channel.h b/include/io/channel.h > index 7986c49c713a..01ad7bd7e430 100644 > --- a/include/io/channel.h > +++ b/include/io/channel.h > @@ -160,6 +160,9 @@ struct QIOChannelClass { > void *opaque); > int (*io_flush)(QIOChannel *ioc, > Error **errp); > + void (*io_peerpid)(QIOChannel *ioc, > + unsigned int *pid, > + Error **errp); > }; > > /* General I/O handling functions */ > @@ -981,4 +984,22 @@ int coroutine_mixed_fn qio_channel_writev_full_all(QIOChannel *ioc, > int qio_channel_flush(QIOChannel *ioc, > Error **errp); > > +/** > + * qio_channel_get_peercred: > + * @ioc: the channel object > + * @pid: pointer to pid > + * @errp: pointer to a NULL-initialized error object > + * > + * Returns the pid of the peer process connected to this socket. > + * > + * The use of this function is possible only for connected > + * AF_UNIX stream sockets and for AF_UNIX stream and datagram > + * socket pairs on Linux. > + * Return an error with pid -1 for the non-Linux OS. > + * > + */ > +void qio_channel_get_peerpid(QIOChannel *ioc, > + unsigned int *pid, > + Error **errp); > + > #endif /* QIO_CHANNEL_H */ > diff --git a/io/channel-socket.c b/io/channel-socket.c > index 3a899b060858..fcff92ecc151 100644 > --- a/io/channel-socket.c > +++ b/io/channel-socket.c > @@ -841,6 +841,29 @@ qio_channel_socket_set_cork(QIOChannel *ioc, > socket_set_cork(sioc->fd, v); > } > > +static void > +qio_channel_socket_get_peerpid(QIOChannel *ioc, > + unsigned int *pid, > + Error **errp) > +{ > +#ifdef CONFIG_LINUX > + QIOChannelSocket *sioc = QIO_CHANNEL_SOCKET(ioc); > + Error *err = NULL; > + socklen_t len = sizeof(struct ucred); > + > + struct ucred cred; > + if (getsockopt(sioc->fd, > + SOL_SOCKET, SO_PEERCRED, > + &cred, &len) == -1) { Set '*pid = -1' > + error_setg_errno(&err, errno, "Unable to get peer credentials"); > + error_propagate(errp, err); and 'return;' here, since accessing 'cred.pid' below is undefined behaviour if getsockopt failed. > + } > + *pid = (unsigned int)cred.pid; > +#else > + error_setg(errp, "Unsupported feature"); > + *pid = -1; > +#endif > +} > > static int > qio_channel_socket_close(QIOChannel *ioc, > @@ -938,6 +961,7 @@ static void qio_channel_socket_class_init(ObjectClass *klass, > #ifdef QEMU_MSG_ZEROCOPY > ioc_klass->io_flush = qio_channel_socket_flush; > #endif > + ioc_klass->io_peerpid = qio_channel_socket_get_peerpid; > } > > static const TypeInfo qio_channel_socket_info = { > diff --git a/io/channel.c b/io/channel.c > index a1f12f8e9096..777989bc9a81 100644 > --- a/io/channel.c > +++ b/io/channel.c > @@ -548,6 +548,18 @@ void qio_channel_set_cork(QIOChannel *ioc, > } > } > > +void qio_channel_get_peerpid(QIOChannel *ioc, > + unsigned int *pid, > + Error **errp) > +{ > + QIOChannelClass *klass = QIO_CHANNEL_GET_CLASS(ioc); > + > + if (!klass->io_peerpid) { > + error_setg(errp, "Channel does not support peer pid"); > + return; > + } > + klass->io_peerpid(ioc, pid, errp); > +} > > off_t qio_channel_io_seek(QIOChannel *ioc, > off_t offset, > -- > 2.44.0 > With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v4 2/3] tools: build qemu-vmsr-helper 2024-03-18 15:12 [PATCH v4 0/3] Add support for the RAPL MSRs series Anthony Harivel 2024-03-18 15:12 ` [PATCH v4 1/3] qio: add support for SO_PEERCRED for socket channel Anthony Harivel @ 2024-03-18 15:12 ` Anthony Harivel 2024-03-21 11:33 ` Daniel P. Berrangé 2024-03-18 15:12 ` [PATCH v4 3/3] Add support for RAPL MSRs in KVM/Qemu Anthony Harivel 2 siblings, 1 reply; 10+ messages in thread From: Anthony Harivel @ 2024-03-18 15:12 UTC (permalink / raw) To: pbonzini, mtosatti, berrange Cc: qemu-devel, vchundur, rjarry, Anthony Harivel Introduce a privileged helper to access RAPL MSR. The privileged helper tool, qemu-vmsr-helper, is designed to provide virtual machines with the ability to read specific RAPL (Running Average Power Limit) MSRs without requiring CAP_SYS_RAWIO privileges or relying on external, out-of-tree patches. The helper tool leverages Unix permissions and SO_PEERCRED socket options to enforce access control, ensuring that only processes explicitly requesting read access via readmsr() from a valid Thread ID can access these MSRs. The list of RAPL MSRs that are allowed to be read by the helper tool is defined in rapl-msr-index.h. This list corresponds to the RAPL MSRs that will be supported in the next commit titled "Add support for RAPL MSRs in KVM/QEMU." The tool is intentionally designed to run on the Linux x86 platform. This initial implementation is tailored for Intel CPUs but can be extended to support AMD CPUs in the future. Signed-off-by: Anthony Harivel <aharivel@redhat.com> --- contrib/systemd/qemu-vmsr-helper.service | 15 + contrib/systemd/qemu-vmsr-helper.socket | 9 + docs/tools/index.rst | 1 + docs/tools/qemu-vmsr-helper.rst | 89 ++++ meson.build | 5 + tools/i386/qemu-vmsr-helper.c | 564 +++++++++++++++++++++++ tools/i386/rapl-msr-index.h | 28 ++ 7 files changed, 711 insertions(+) create mode 100644 contrib/systemd/qemu-vmsr-helper.service create mode 100644 contrib/systemd/qemu-vmsr-helper.socket create mode 100644 docs/tools/qemu-vmsr-helper.rst create mode 100644 tools/i386/qemu-vmsr-helper.c create mode 100644 tools/i386/rapl-msr-index.h diff --git a/contrib/systemd/qemu-vmsr-helper.service b/contrib/systemd/qemu-vmsr-helper.service new file mode 100644 index 000000000000..8fd397bf79a9 --- /dev/null +++ b/contrib/systemd/qemu-vmsr-helper.service @@ -0,0 +1,15 @@ +[Unit] +Description=Virtual RAPL MSR Daemon for QEMU + +[Service] +WorkingDirectory=/tmp +Type=simple +ExecStart=/usr/bin/qemu-vmsr-helper +PrivateTmp=yes +ProtectSystem=strict +ReadWritePaths=/var/run +RestrictAddressFamilies=AF_UNIX +Restart=always +RestartSec=0 + +[Install] diff --git a/contrib/systemd/qemu-vmsr-helper.socket b/contrib/systemd/qemu-vmsr-helper.socket new file mode 100644 index 000000000000..183e8304d6e2 --- /dev/null +++ b/contrib/systemd/qemu-vmsr-helper.socket @@ -0,0 +1,9 @@ +[Unit] +Description=Virtual RAPL MSR helper for QEMU + +[Socket] +ListenStream=/run/qemu-vmsr-helper.sock +SocketMode=0600 + +[Install] +WantedBy=multi-user.target diff --git a/docs/tools/index.rst b/docs/tools/index.rst index 8e65ce0dfc7b..33ad438e86f6 100644 --- a/docs/tools/index.rst +++ b/docs/tools/index.rst @@ -16,3 +16,4 @@ command line utilities and other standalone programs. qemu-pr-helper qemu-trace-stap virtfs-proxy-helper + qemu-vmsr-helper diff --git a/docs/tools/qemu-vmsr-helper.rst b/docs/tools/qemu-vmsr-helper.rst new file mode 100644 index 000000000000..6ec87b49d962 --- /dev/null +++ b/docs/tools/qemu-vmsr-helper.rst @@ -0,0 +1,89 @@ +================================== +QEMU virtual RAPL MSR helper +================================== + +Synopsis +-------- + +**qemu-vmsr-helper** [*OPTION*] + +Description +----------- + +Implements the virtual RAPL MSR helper for QEMU. + +Accessing the RAPL (Running Average Power Limit) MSR enables the RAPL powercap +driver to advertise and monitor the power consumption or accumulated energy +consumption of different power domains, such as CPU packages, DRAM, and other +components when available. + +However those register are accesible under priviliged access (CAP_SYS_RAWIO). +QEMU can use an external helper to access those priviliged register. + +:program:`qemu-vmsr-helper` is that external helper; it creates a listener +socket which will accept incoming connections for communication with QEMU. + +If you want to run VMs in a setup like this, this helper should be started as a +system service, and you should read the QEMU manual section on "RAPL MSR +support" to find out how to configure QEMU to connect to the socket created by +:program:`qemu-vmsr-helper`. + +After connecting to the socket, :program:`qemu-vmsr-helper` can +optionally drop root privileges, except for those capabilities that +are needed for its operation. + +:program:`qemu-vmsr-helper` can also use the systemd socket activation +protocol. In this case, the systemd socket unit should specify a +Unix stream socket, like this:: + + [Socket] + ListenStream=/var/run/qemu-vmsr-helper.sock + +Options +------- + +.. program:: qemu-vmsr-helper + +.. option:: -d, --daemon + + run in the background (and create a PID file) + +.. option:: -q, --quiet + + decrease verbosity + +.. option:: -v, --verbose + + increase verbosity + +.. option:: -f, --pidfile=PATH + + PID file when running as a daemon. By default the PID file + is created in the system runtime state directory, for example + :file:`/var/run/qemu-vmsr-helper.pid`. + +.. option:: -k, --socket=PATH + + path to the socket. By default the socket is created in + the system runtime state directory, for example + :file:`/var/run/qemu-vmsr-helper.sock`. + +.. option:: -T, --trace [[enable=]PATTERN][,events=FILE][,file=FILE] + + .. include:: ../qemu-option-trace.rst.inc + +.. option:: -u, --user=USER + + user to drop privileges to + +.. option:: -g, --group=GROUP + + group to drop privileges to + +.. option:: -h, --help + + Display a help message and exit. + +.. option:: -V, --version + + Display version information and exit. diff --git a/meson.build b/meson.build index b375248a7614..376da49b60ab 100644 --- a/meson.build +++ b/meson.build @@ -4052,6 +4052,11 @@ if have_tools dependencies: [authz, crypto, io, qom, qemuutil, libcap_ng, mpathpersist], install: true) + + executable('qemu-vmsr-helper', files('tools/i386/qemu-vmsr-helper.c'), + dependencies: [authz, crypto, io, qom, qemuutil, + libcap_ng, mpathpersist], + install: true) endif if have_ivshmem diff --git a/tools/i386/qemu-vmsr-helper.c b/tools/i386/qemu-vmsr-helper.c new file mode 100644 index 000000000000..d8439dc173af --- /dev/null +++ b/tools/i386/qemu-vmsr-helper.c @@ -0,0 +1,564 @@ +/* + * Privileged RAPL MSR helper commands for QEMU + * + * Copyright (C) 2024 Red Hat, Inc. <aharivel@redhat.com> + * + * Author: Anthony Harivel <aharivel@redhat.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; under version 2 of the License. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see <http://www.gnu.org/licenses/>. + */ + +#include "qemu/osdep.h" +#include <getopt.h> +#include <stdbool.h> +#include <sys/ioctl.h> +#ifdef CONFIG_LIBCAP_NG +#include <cap-ng.h> +#endif +#include <pwd.h> +#include <grp.h> + +#include "qemu/help-texts.h" +#include "qapi/error.h" +#include "qemu/cutils.h" +#include "qemu/main-loop.h" +#include "qemu/module.h" +#include "qemu/error-report.h" +#include "qemu/config-file.h" +#include "qemu-version.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "qemu/log.h" +#include "qemu/systemd.h" +#include "io/channel.h" +#include "io/channel-socket.h" +#include "trace/control.h" +#include "qemu-version.h" +#include "rapl-msr-index.h" + +#define MSR_PATH_TEMPLATE "/dev/cpu/%u/msr" + +static char *socket_path; +static char *pidfile; +static enum { RUNNING, TERMINATE, TERMINATING } state; +static QIOChannelSocket *server_ioc; +static int server_watch; +static int num_active_sockets = 1; + +#ifdef CONFIG_LIBCAP_NG +static int uid = -1; +static int gid = -1; +#endif + +static void compute_default_paths(void) +{ + g_autofree char *state = qemu_get_local_state_dir(); + + socket_path = g_build_filename(state, "run", "qemu-vmsr-helper.sock", NULL); + pidfile = g_build_filename(state, "run", "qemu-vmsr-helper.pid", NULL); +} + +static int is_intel_processor(void) +{ + int result; + int ebx, ecx, edx; + + /* Execute CPUID instruction with eax=0 (basic identification) */ + asm volatile ( + "cpuid" + : "=b" (ebx), "=c" (ecx), "=d" (edx) + : "a" (0) + ); + + /* + * Check if processor is "GenuineIntel" + * 0x756e6547 = "Genu" + * 0x49656e69 = "ineI" + * 0x6c65746e = "ntel" + */ + result = (ebx == 0x756e6547) && (edx == 0x49656e69) && (ecx == 0x6c65746e); + + return result; +} + +static int is_rapl_enabled(void) +{ + const char *path = "/sys/class/powercap/intel-rapl/enabled"; + FILE *file = fopen(path, "r"); + int value = 0; + + if (file != NULL) { + if (fscanf(file, "%d", &value) != 1) { + error_report("INTEL RAPL not enabled"); + } + fclose(file); + } else { + error_report("Error opening %s", path); + } + + return value; +} + +/* + * Check if the TID that request the MSR read + * belongs to the peer. It be should a TID of a vCPU. + */ +static bool is_tid_present(pid_t pid, pid_t tid) +{ + g_autofree char *pidStr; + g_autofree char *tidStr; + + pidStr = g_strdup_printf("%d", pid); + tidStr = g_strdup_printf("%d", tid); + + char *tidPath; + + tidPath = g_strdup_printf("/proc/%s/task/%s", pidStr, tidStr); + + /* Check if the TID directory exists within the PID directory */ + if (access(tidPath, F_OK) == 0) { + return true; + } + + error_report("Failed to open /proc at %s", tidPath); + return false; +} + +/* + * Only the RAPL MSR in target/i386/cpu.h are allowed + */ +static bool is_msr_allowed(uint32_t reg) +{ + switch (reg) { + case MSR_RAPL_POWER_UNIT: + case MSR_PKG_POWER_LIMIT: + case MSR_PKG_ENERGY_STATUS: + case MSR_PKG_POWER_INFO: + return true; + default: + return false; + } +} + +static uint64_t vmsr_read_msr(uint32_t msr_register, unsigned int cpu_id) +{ + int fd; + uint64_t result = 0; + + g_autofree char *path; + path = g_strdup_printf(MSR_PATH_TEMPLATE, cpu_id); + + /* + * Check if the specified CPU is included in the thread's affinity + */ + cpu_set_t cpu_set; + CPU_ZERO(&cpu_set); + sched_getaffinity(0, sizeof(cpu_set_t), &cpu_set); + + if (!CPU_ISSET(cpu_id, &cpu_set)) { + fprintf(stderr, "CPU %u is not in the thread's affinity.\n", cpu_id); + return result; + } + + fd = open(path, O_RDONLY); + if (fd < 0) { + error_report("Failed to open MSR file at %s", path); + return result; + } + + if (pread(fd, &result, sizeof(result), msr_register) != sizeof(result)) { + error_report("Failed to read MSR"); + result = 0; + } + + close(fd); + return result; +} + +static void usage(const char *name) +{ + (printf) ( +"Usage: %s [OPTIONS] FILE\n" +"Virtual RAPL MSR helper program for QEMU\n" +"\n" +" -h, --help display this help and exit\n" +" -V, --version output version information and exit\n" +"\n" +" -d, --daemon run in the background\n" +" -f, --pidfile=PATH PID file when running as a daemon\n" +" (default '%s')\n" +" -k, --socket=PATH path to the unix socket\n" +" (default '%s')\n" +" -T, --trace [[enable=]<pattern>][,events=<file>][,file=<file>]\n" +" specify tracing options\n" +#ifdef CONFIG_LIBCAP_NG +" -u, --user=USER user to drop privileges to\n" +" -g, --group=GROUP group to drop privileges to\n" +#endif +"\n" +QEMU_HELP_BOTTOM "\n" + , name, pidfile, socket_path); +} + +static void version(const char *name) +{ + printf( +"%s " QEMU_FULL_VERSION "\n" +"Written by Anthony Harivel.\n" +"\n" +QEMU_COPYRIGHT "\n" +"This is free software; see the source for copying conditions. There is NO\n" +"warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n" + , name); +} + +typedef struct VMSRHelperClient { + QIOChannelSocket *ioc; + Coroutine *co; +} VMSRHelperClient; + +static void coroutine_fn vh_co_entry(void *opaque) +{ + VMSRHelperClient *client = opaque; + Error *local_err = NULL; + unsigned int peer_pid; + uint32_t request[3]; + uint64_t vmsr; + int r; + + qio_channel_set_blocking(QIO_CHANNEL(client->ioc), + false, NULL); + + qio_channel_set_follow_coroutine_ctx(QIO_CHANNEL(client->ioc), true); + + /* + * Check peer credentials + */ + qio_channel_get_peerpid(QIO_CHANNEL(client->ioc), &peer_pid, &local_err); + + if (peer_pid == 0) { + if (local_err != NULL) { + error_report_err(local_err); + } + error_report("Failed to get peer credentials"); + goto out; + } + + /* + * Read the requested MSR + * Only RAPL MSR in rapl-msr-index.h is allowed + */ + r = qio_channel_read_all(QIO_CHANNEL(client->ioc), + (char *) &request, sizeof(request), &local_err); + if ((local_err != NULL) || r < 0) { + error_report("Read request fail"); + error_report_err(local_err); + goto out; + } + if (!is_msr_allowed(request[0])) { + error_report("Requested unallowed msr: %d", request[0]); + goto out; + } + + vmsr = vmsr_read_msr(request[0], request[1]); + + if (!is_tid_present(peer_pid, request[2])) { + error_report("Requested TID not in peer PID: %d %d", + peer_pid, request[2]); + vmsr = 0; + } + + r = qio_channel_write_all(QIO_CHANNEL(client->ioc), + (char *) &vmsr, sizeof(vmsr), &local_err); + if ((local_err != NULL) || r < 0) { + error_report("Write request fail"); + error_report_err(local_err); + goto out; + } + +out: + object_unref(OBJECT(client->ioc)); +} + +static gboolean accept_client(QIOChannel *ioc, + GIOCondition cond, + gpointer opaque) +{ + QIOChannelSocket *cioc; + VMSRHelperClient *vmsrh; + + cioc = qio_channel_socket_accept(QIO_CHANNEL_SOCKET(ioc), + NULL); + if (!cioc) { + return FALSE; + } + + vmsrh = g_new(VMSRHelperClient, 1); + vmsrh->ioc = cioc; + vmsrh->co = qemu_coroutine_create(vh_co_entry, vmsrh); + qemu_coroutine_enter(vmsrh->co); + + return TRUE; +} + +static void termsig_handler(int signum) +{ + qatomic_cmpxchg(&state, RUNNING, TERMINATE); + qemu_notify_event(); +} + +static void close_server_socket(void) +{ + assert(server_ioc); + + g_source_remove(server_watch); + server_watch = -1; + object_unref(OBJECT(server_ioc)); + num_active_sockets--; +} + +#ifdef CONFIG_LIBCAP_NG +static int drop_privileges(void) +{ + /* clear all capabilities */ + capng_clear(CAPNG_SELECT_BOTH); + + if (capng_update(CAPNG_ADD, CAPNG_EFFECTIVE | CAPNG_PERMITTED, + CAP_SYS_RAWIO) < 0) { + return -1; + } + + /* + * Change user/group id, retaining the capabilities. + * Because file descriptors are passed via SCM_RIGHTS, + * we don't need supplementary groups (and in fact the helper + * can run as "nobody"). + */ + if (capng_change_id(uid != -1 ? uid : getuid(), + gid != -1 ? gid : getgid(), + CAPNG_DROP_SUPP_GRP | CAPNG_CLEAR_BOUNDING)) { + return -1; + } + + return 0; +} +#endif + +int main(int argc, char **argv) +{ + const char *sopt = "hVk:f:dT:u:g:vq"; + struct option lopt[] = { + { "help", no_argument, NULL, 'h' }, + { "version", no_argument, NULL, 'V' }, + { "socket", required_argument, NULL, 'k' }, + { "pidfile", required_argument, NULL, 'f' }, + { "daemon", no_argument, NULL, 'd' }, + { "trace", required_argument, NULL, 'T' }, + { "user", required_argument, NULL, 'u' }, + { "group", required_argument, NULL, 'g' }, + { "verbose", no_argument, NULL, 'v' }, + { NULL, 0, NULL, 0 } + }; + int opt_ind = 0; + int ch; + Error *local_err = NULL; + bool daemonize = false; + bool pidfile_specified = false; + bool socket_path_specified = false; + unsigned socket_activation; + + struct sigaction sa_sigterm; + memset(&sa_sigterm, 0, sizeof(sa_sigterm)); + sa_sigterm.sa_handler = termsig_handler; + sigaction(SIGTERM, &sa_sigterm, NULL); + sigaction(SIGINT, &sa_sigterm, NULL); + sigaction(SIGHUP, &sa_sigterm, NULL); + + signal(SIGPIPE, SIG_IGN); + + error_init(argv[0]); + module_call_init(MODULE_INIT_TRACE); + module_call_init(MODULE_INIT_QOM); + qemu_add_opts(&qemu_trace_opts); + qemu_init_exec_dir(argv[0]); + + compute_default_paths(); + + /* + * Sanity check + * 1. cpu must be Intel cpu + * 2. RAPL must be enabled + */ + if (!is_intel_processor()) { + error_report("error: CPU is not INTEL cpu"); + exit(EXIT_FAILURE); + } + + if (!is_rapl_enabled()) { + error_report("error: RAPL driver not enable"); + exit(EXIT_FAILURE); + } + + while ((ch = getopt_long(argc, argv, sopt, lopt, &opt_ind)) != -1) { + switch (ch) { + case 'k': + g_free(socket_path); + socket_path = g_strdup(optarg); + socket_path_specified = true; + if (socket_path[0] != '/') { + error_report("socket path must be absolute"); + exit(EXIT_FAILURE); + } + break; + case 'f': + g_free(pidfile); + pidfile = g_strdup(optarg); + pidfile_specified = true; + break; +#ifdef CONFIG_LIBCAP_NG + case 'u': { + unsigned long res; + struct passwd *userinfo = getpwnam(optarg); + if (userinfo) { + uid = userinfo->pw_uid; + } else if (qemu_strtoul(optarg, NULL, 10, &res) == 0 && + (uid_t)res == res) { + uid = res; + } else { + error_report("invalid user '%s'", optarg); + exit(EXIT_FAILURE); + } + break; + } + case 'g': { + unsigned long res; + struct group *groupinfo = getgrnam(optarg); + if (groupinfo) { + gid = groupinfo->gr_gid; + } else if (qemu_strtoul(optarg, NULL, 10, &res) == 0 && + (gid_t)res == res) { + gid = res; + } else { + error_report("invalid group '%s'", optarg); + exit(EXIT_FAILURE); + } + break; + } +#else + case 'u': + case 'g': + error_report("-%c not supported by this %s", ch, argv[0]); + exit(1); +#endif + case 'd': + daemonize = true; + break; + case 'T': + trace_opt_parse(optarg); + break; + case 'V': + version(argv[0]); + exit(EXIT_SUCCESS); + break; + case 'h': + usage(argv[0]); + exit(EXIT_SUCCESS); + break; + case '?': + error_report("Try `%s --help' for more information.", argv[0]); + exit(EXIT_FAILURE); + } + } + + if (!trace_init_backends()) { + exit(EXIT_FAILURE); + } + trace_init_file(); + qemu_set_log(LOG_TRACE, &error_fatal); + + socket_activation = check_socket_activation(); + if (socket_activation == 0) { + SocketAddress saddr; + saddr = (SocketAddress){ + .type = SOCKET_ADDRESS_TYPE_UNIX, + .u.q_unix.path = socket_path, + }; + server_ioc = qio_channel_socket_new(); + if (qio_channel_socket_listen_sync(server_ioc, &saddr, + 1, &local_err) < 0) { + object_unref(OBJECT(server_ioc)); + error_report_err(local_err); + return 1; + } + } else { + /* Using socket activation - check user didn't use -p etc. */ + if (socket_path_specified) { + error_report("Unix socket can't be set when" + "using socket activation"); + exit(EXIT_FAILURE); + } + + /* Can only listen on a single socket. */ + if (socket_activation > 1) { + error_report("%s does not support socket activation" + "with LISTEN_FDS > 1", + argv[0]); + exit(EXIT_FAILURE); + } + server_ioc = qio_channel_socket_new_fd(FIRST_SOCKET_ACTIVATION_FD, + &local_err); + if (server_ioc == NULL) { + error_reportf_err(local_err, + "Failed to use socket activation: "); + exit(EXIT_FAILURE); + } + } + + qemu_init_main_loop(&error_fatal); + + server_watch = qio_channel_add_watch(QIO_CHANNEL(server_ioc), + G_IO_IN, + accept_client, + NULL, NULL); + + if (daemonize) { + if (daemon(0, 0) < 0) { + error_report("Failed to daemonize: %s", strerror(errno)); + exit(EXIT_FAILURE); + } + } + + if (daemonize || pidfile_specified) { + qemu_write_pidfile(pidfile, &error_fatal); + } + +#ifdef CONFIG_LIBCAP_NG + if (drop_privileges() < 0) { + error_report("Failed to drop privileges: %s", strerror(errno)); + exit(EXIT_FAILURE); + } +#endif + + info_report("Listening on %s", socket_path); + + state = RUNNING; + do { + main_loop_wait(false); + if (state == TERMINATE) { + state = TERMINATING; + close_server_socket(); + } + } while (num_active_sockets > 0); + + exit(EXIT_SUCCESS); +} diff --git a/tools/i386/rapl-msr-index.h b/tools/i386/rapl-msr-index.h new file mode 100644 index 000000000000..9a7118639ae3 --- /dev/null +++ b/tools/i386/rapl-msr-index.h @@ -0,0 +1,28 @@ +/* + * Allowed list of MSR for Privileged RAPL MSR helper commands for QEMU + * + * Copyright (C) 2023 Red Hat, Inc. <aharivel@redhat.com> + * + * Author: Anthony Harivel <aharivel@redhat.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; under version 2 of the License. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see <http://www.gnu.org/licenses/>. + */ + +/* + * Should stay in sync with the RAPL MSR + * in target/i386/cpu.h + */ +#define MSR_RAPL_POWER_UNIT 0x00000606 +#define MSR_PKG_POWER_LIMIT 0x00000610 +#define MSR_PKG_ENERGY_STATUS 0x00000611 +#define MSR_PKG_POWER_INFO 0x00000614 -- 2.44.0 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v4 2/3] tools: build qemu-vmsr-helper 2024-03-18 15:12 ` [PATCH v4 2/3] tools: build qemu-vmsr-helper Anthony Harivel @ 2024-03-21 11:33 ` Daniel P. Berrangé 2024-03-28 10:27 ` Anthony Harivel 0 siblings, 1 reply; 10+ messages in thread From: Daniel P. Berrangé @ 2024-03-21 11:33 UTC (permalink / raw) To: Anthony Harivel; +Cc: pbonzini, mtosatti, qemu-devel, vchundur, rjarry On Mon, Mar 18, 2024 at 04:12:15PM +0100, Anthony Harivel wrote: > Introduce a privileged helper to access RAPL MSR. > > The privileged helper tool, qemu-vmsr-helper, is designed to provide > virtual machines with the ability to read specific RAPL (Running Average > Power Limit) MSRs without requiring CAP_SYS_RAWIO privileges or relying > on external, out-of-tree patches. > > The helper tool leverages Unix permissions and SO_PEERCRED socket > options to enforce access control, ensuring that only processes > explicitly requesting read access via readmsr() from a valid Thread ID > can access these MSRs. > > The list of RAPL MSRs that are allowed to be read by the helper tool is > defined in rapl-msr-index.h. This list corresponds to the RAPL MSRs that > will be supported in the next commit titled "Add support for RAPL MSRs > in KVM/QEMU." > > The tool is intentionally designed to run on the Linux x86 platform. > This initial implementation is tailored for Intel CPUs but can be > extended to support AMD CPUs in the future. > > Signed-off-by: Anthony Harivel <aharivel@redhat.com> > --- > contrib/systemd/qemu-vmsr-helper.service | 15 + > contrib/systemd/qemu-vmsr-helper.socket | 9 + > docs/tools/index.rst | 1 + > docs/tools/qemu-vmsr-helper.rst | 89 ++++ > meson.build | 5 + > tools/i386/qemu-vmsr-helper.c | 564 +++++++++++++++++++++++ > tools/i386/rapl-msr-index.h | 28 ++ > 7 files changed, 711 insertions(+) > create mode 100644 contrib/systemd/qemu-vmsr-helper.service > create mode 100644 contrib/systemd/qemu-vmsr-helper.socket > create mode 100644 docs/tools/qemu-vmsr-helper.rst > create mode 100644 tools/i386/qemu-vmsr-helper.c > create mode 100644 tools/i386/rapl-msr-index.h > > diff --git a/meson.build b/meson.build > index b375248a7614..376da49b60ab 100644 > --- a/meson.build > +++ b/meson.build > @@ -4052,6 +4052,11 @@ if have_tools > dependencies: [authz, crypto, io, qom, qemuutil, > libcap_ng, mpathpersist], > install: true) > + > + executable('qemu-vmsr-helper', files('tools/i386/qemu-vmsr-helper.c'), > + dependencies: [authz, crypto, io, qom, qemuutil, > + libcap_ng, mpathpersist], > + install: true) > endif Missed feedback from v2 saying this must /only/ be built on x86 architectures. It fails to build on others due to the ASM usage eg https://gitlab.com/berrange/qemu/-/jobs/6445384073 > > if have_ivshmem > diff --git a/tools/i386/qemu-vmsr-helper.c b/tools/i386/qemu-vmsr-helper.c > new file mode 100644 > index 000000000000..d8439dc173af > --- /dev/null > +++ b/tools/i386/qemu-vmsr-helper.c > @@ -0,0 +1,564 @@ > +/* > + * Privileged RAPL MSR helper commands for QEMU > + * > + * Copyright (C) 2024 Red Hat, Inc. <aharivel@redhat.com> > + * > + * Author: Anthony Harivel <aharivel@redhat.com> > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; under version 2 of the License. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program; if not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include "qemu/osdep.h" > +#include <getopt.h> > +#include <stdbool.h> > +#include <sys/ioctl.h> > +#ifdef CONFIG_LIBCAP_NG > +#include <cap-ng.h> > +#endif > +#include <pwd.h> > +#include <grp.h> > + > +#include "qemu/help-texts.h" > +#include "qapi/error.h" > +#include "qemu/cutils.h" > +#include "qemu/main-loop.h" > +#include "qemu/module.h" > +#include "qemu/error-report.h" > +#include "qemu/config-file.h" > +#include "qemu-version.h" > +#include "qapi/error.h" > +#include "qemu/error-report.h" > +#include "qemu/log.h" > +#include "qemu/systemd.h" > +#include "io/channel.h" > +#include "io/channel-socket.h" > +#include "trace/control.h" > +#include "qemu-version.h" > +#include "rapl-msr-index.h" > + > +#define MSR_PATH_TEMPLATE "/dev/cpu/%u/msr" > + > +static char *socket_path; > +static char *pidfile; > +static enum { RUNNING, TERMINATE, TERMINATING } state; > +static QIOChannelSocket *server_ioc; > +static int server_watch; > +static int num_active_sockets = 1; > + > +#ifdef CONFIG_LIBCAP_NG > +static int uid = -1; > +static int gid = -1; > +#endif > + > +static void compute_default_paths(void) > +{ > + g_autofree char *state = qemu_get_local_state_dir(); > + > + socket_path = g_build_filename(state, "run", "qemu-vmsr-helper.sock", NULL); > + pidfile = g_build_filename(state, "run", "qemu-vmsr-helper.pid", NULL); > +} > + > +static int is_intel_processor(void) > +{ > + int result; > + int ebx, ecx, edx; > + > + /* Execute CPUID instruction with eax=0 (basic identification) */ > + asm volatile ( > + "cpuid" > + : "=b" (ebx), "=c" (ecx), "=d" (edx) > + : "a" (0) > + ); > + > + /* > + * Check if processor is "GenuineIntel" > + * 0x756e6547 = "Genu" > + * 0x49656e69 = "ineI" > + * 0x6c65746e = "ntel" > + */ > + result = (ebx == 0x756e6547) && (edx == 0x49656e69) && (ecx == 0x6c65746e); > + > + return result; > +} > + > +static int is_rapl_enabled(void) > +{ > + const char *path = "/sys/class/powercap/intel-rapl/enabled"; > + FILE *file = fopen(path, "r"); > + int value = 0; > + > + if (file != NULL) { > + if (fscanf(file, "%d", &value) != 1) { > + error_report("INTEL RAPL not enabled"); > + } > + fclose(file); > + } else { > + error_report("Error opening %s", path); > + } > + > + return value; > +} > + > +/* > + * Check if the TID that request the MSR read > + * belongs to the peer. It be should a TID of a vCPU. > + */ > +static bool is_tid_present(pid_t pid, pid_t tid) > +{ > + g_autofree char *pidStr; > + g_autofree char *tidStr; > + > + pidStr = g_strdup_printf("%d", pid); > + tidStr = g_strdup_printf("%d", tid); > + > + char *tidPath; > + > + tidPath = g_strdup_printf("/proc/%s/task/%s", pidStr, tidStr); The initial printfs are pointless, simplify to g_autofree *tidPath = g_strdup_printf("/proc/%d/task/%d", pid, tid); > + > + /* Check if the TID directory exists within the PID directory */ > + if (access(tidPath, F_OK) == 0) { > + return true; > + } > + > + error_report("Failed to open /proc at %s", tidPath); > + return false; > +} > + > +/* > + * Only the RAPL MSR in target/i386/cpu.h are allowed > + */ > +static bool is_msr_allowed(uint32_t reg) > +{ > + switch (reg) { > + case MSR_RAPL_POWER_UNIT: > + case MSR_PKG_POWER_LIMIT: > + case MSR_PKG_ENERGY_STATUS: > + case MSR_PKG_POWER_INFO: > + return true; > + default: > + return false; > + } > +} > + > +static uint64_t vmsr_read_msr(uint32_t msr_register, unsigned int cpu_id) > +{ > + int fd; > + uint64_t result = 0; > + > + g_autofree char *path; > + path = g_strdup_printf(MSR_PATH_TEMPLATE, cpu_id); Please initialize at same time as declaring for anything marked 'g_autofree' > + > + /* > + * Check if the specified CPU is included in the thread's affinity > + */ > + cpu_set_t cpu_set; > + CPU_ZERO(&cpu_set); > + sched_getaffinity(0, sizeof(cpu_set_t), &cpu_set); As mentioned on previous review, this breaks on large machines. > + > + if (!CPU_ISSET(cpu_id, &cpu_set)) { > + fprintf(stderr, "CPU %u is not in the thread's affinity.\n", cpu_id); > + return result; > + } As mentioned on previous review ,this is inconsistent and should use error_report. Having said that, what's the point in the CPU affinity checking ? Whether or not this process runs on the requested CPU appears irrelevant - it can query the MSR from any CPU, using the file it is able to open. The kernel doesn't seem to limit this AFAICS. So i thin this check can go. > + > + fd = open(path, O_RDONLY); > + if (fd < 0) { > + error_report("Failed to open MSR file at %s", path); > + return result; > + } > + > + if (pread(fd, &result, sizeof(result), msr_register) != sizeof(result)) { > + error_report("Failed to read MSR"); > + result = 0; > + } > + > + close(fd); > + return result; > +} > + > +typedef struct VMSRHelperClient { > + QIOChannelSocket *ioc; > + Coroutine *co; > +} VMSRHelperClient; > + > +static void coroutine_fn vh_co_entry(void *opaque) > +{ > + VMSRHelperClient *client = opaque; > + Error *local_err = NULL; > + unsigned int peer_pid; > + uint32_t request[3]; > + uint64_t vmsr; > + int r; > + > + qio_channel_set_blocking(QIO_CHANNEL(client->ioc), > + false, NULL); > + > + qio_channel_set_follow_coroutine_ctx(QIO_CHANNEL(client->ioc), true); > + > + /* > + * Check peer credentials > + */ > + qio_channel_get_peerpid(QIO_CHANNEL(client->ioc), &peer_pid, &local_err); > + > + if (peer_pid == 0) { > + if (local_err != NULL) { > + error_report_err(local_err); > + } > + error_report("Failed to get peer credentials"); > + goto out; > + } get_peerpid should be made to retun -1 on error such that you can do: r = qio_channel_get_peerpid(...); if (r < 0) { error_report_err(locla_err); goto out; } > + > + /* > + * Read the requested MSR > + * Only RAPL MSR in rapl-msr-index.h is allowed > + */ > + r = qio_channel_read_all(QIO_CHANNEL(client->ioc), > + (char *) &request, sizeof(request), &local_err); > + if ((local_err != NULL) || r < 0) { This should only be checking 'r < 0', when that's true, 'local_err' will be assumed to be non-NULL > + error_report("Read request fail"); Not required, sinec you have the next line giving the read error meessage: > + error_report_err(local_err); > + goto out; > + } > + if (!is_msr_allowed(request[0])) { > + error_report("Requested unallowed msr: %d", request[0]); > + goto out; > + } > + > + vmsr = vmsr_read_msr(request[0], request[1]); > + > + if (!is_tid_present(peer_pid, request[2])) { > + error_report("Requested TID not in peer PID: %d %d", > + peer_pid, request[2]); Indent is off - align with the " > + vmsr = 0; > + } > + > + r = qio_channel_write_all(QIO_CHANNEL(client->ioc), > + (char *) &vmsr, sizeof(vmsr), &local_err); Mis aligned indent - match with the 'Q' of the line above > + if ((local_err != NULL) || r < 0) { Again, only check 'r < 0' > + error_report("Write request fail"); and drop this line > + error_report_err(local_err); > + goto out; > + } > + > +out: > + object_unref(OBJECT(client->ioc)); g_free(client) > +} > + > +static gboolean accept_client(QIOChannel *ioc, > + GIOCondition cond, > + gpointer opaque) > +{ > + QIOChannelSocket *cioc; > + VMSRHelperClient *vmsrh; > + > + cioc = qio_channel_socket_accept(QIO_CHANNEL_SOCKET(ioc), > + NULL); > + if (!cioc) { > + return FALSE; > + } This needs to be 'TRUE', otherwise the callback gets unregistered and no further connections accepted. > + > + vmsrh = g_new(VMSRHelperClient, 1); > + vmsrh->ioc = cioc; > + vmsrh->co = qemu_coroutine_create(vh_co_entry, vmsrh); > + qemu_coroutine_enter(vmsrh->co); > + > + return TRUE; > +} > + > +static void termsig_handler(int signum) > +{ > + qatomic_cmpxchg(&state, RUNNING, TERMINATE); > + qemu_notify_event(); > +} > + > +static void close_server_socket(void) > +{ > + assert(server_ioc); > + > + g_source_remove(server_watch); > + server_watch = -1; > + object_unref(OBJECT(server_ioc)); > + num_active_sockets--; > +} > + > +#ifdef CONFIG_LIBCAP_NG > +static int drop_privileges(void) > +{ > + /* clear all capabilities */ > + capng_clear(CAPNG_SELECT_BOTH); > + > + if (capng_update(CAPNG_ADD, CAPNG_EFFECTIVE | CAPNG_PERMITTED, > + CAP_SYS_RAWIO) < 0) { > + return -1; > + } > + > + /* > + * Change user/group id, retaining the capabilities. > + * Because file descriptors are passed via SCM_RIGHTS, > + * we don't need supplementary groups (and in fact the helper > + * can run as "nobody"). > + */ > + if (capng_change_id(uid != -1 ? uid : getuid(), > + gid != -1 ? gid : getgid(), > + CAPNG_DROP_SUPP_GRP | CAPNG_CLEAR_BOUNDING)) { > + return -1; > + } As mentioned in previous v2, if this drosp user/group, then it will be unable to open the MSR device file. Must always pass '0' for the uid & gid. > + > + return 0; > +} > +#endif > + > +int main(int argc, char **argv) > +{ > + const char *sopt = "hVk:f:dT:u:g:vq"; > + struct option lopt[] = { > + { "help", no_argument, NULL, 'h' }, > + { "version", no_argument, NULL, 'V' }, > + { "socket", required_argument, NULL, 'k' }, > + { "pidfile", required_argument, NULL, 'f' }, > + { "daemon", no_argument, NULL, 'd' }, > + { "trace", required_argument, NULL, 'T' }, > + { "user", required_argument, NULL, 'u' }, > + { "group", required_argument, NULL, 'g' }, 'user' and 'group' args should be removed entirely, and from the man page, since this must run as root. > + { "verbose", no_argument, NULL, 'v' }, > + { NULL, 0, NULL, 0 } > + }; > + int opt_ind = 0; > + int ch; > + Error *local_err = NULL; > + bool daemonize = false; > + bool pidfile_specified = false; > + bool socket_path_specified = false; > + unsigned socket_activation; > + > + struct sigaction sa_sigterm; > + memset(&sa_sigterm, 0, sizeof(sa_sigterm)); > + sa_sigterm.sa_handler = termsig_handler; > + sigaction(SIGTERM, &sa_sigterm, NULL); > + sigaction(SIGINT, &sa_sigterm, NULL); > + sigaction(SIGHUP, &sa_sigterm, NULL); > + > + signal(SIGPIPE, SIG_IGN); > + > + error_init(argv[0]); > + module_call_init(MODULE_INIT_TRACE); > + module_call_init(MODULE_INIT_QOM); > + qemu_add_opts(&qemu_trace_opts); > + qemu_init_exec_dir(argv[0]); > + > + compute_default_paths(); > + > + /* > + * Sanity check > + * 1. cpu must be Intel cpu > + * 2. RAPL must be enabled > + */ > + if (!is_intel_processor()) { > + error_report("error: CPU is not INTEL cpu"); > + exit(EXIT_FAILURE); > + } > + > + if (!is_rapl_enabled()) { > + error_report("error: RAPL driver not enable"); > + exit(EXIT_FAILURE); > + } > + > + while ((ch = getopt_long(argc, argv, sopt, lopt, &opt_ind)) != -1) { > + switch (ch) { > + case 'k': > + g_free(socket_path); > + socket_path = g_strdup(optarg); > + socket_path_specified = true; > + if (socket_path[0] != '/') { > + error_report("socket path must be absolute"); > + exit(EXIT_FAILURE); > + } > + break; > + case 'f': > + g_free(pidfile); > + pidfile = g_strdup(optarg); > + pidfile_specified = true; > + break; > +#ifdef CONFIG_LIBCAP_NG > + case 'u': { > + unsigned long res; > + struct passwd *userinfo = getpwnam(optarg); > + if (userinfo) { > + uid = userinfo->pw_uid; > + } else if (qemu_strtoul(optarg, NULL, 10, &res) == 0 && > + (uid_t)res == res) { > + uid = res; > + } else { > + error_report("invalid user '%s'", optarg); > + exit(EXIT_FAILURE); > + } > + break; > + } > + case 'g': { > + unsigned long res; > + struct group *groupinfo = getgrnam(optarg); > + if (groupinfo) { > + gid = groupinfo->gr_gid; > + } else if (qemu_strtoul(optarg, NULL, 10, &res) == 0 && > + (gid_t)res == res) { > + gid = res; > + } else { > + error_report("invalid group '%s'", optarg); > + exit(EXIT_FAILURE); > + } > + break; > + } > +#else > + case 'u': > + case 'g': > + error_report("-%c not supported by this %s", ch, argv[0]); > + exit(1); > +#endif > + case 'd': > + daemonize = true; > + break; > + case 'T': > + trace_opt_parse(optarg); > + break; > + case 'V': > + version(argv[0]); > + exit(EXIT_SUCCESS); > + break; > + case 'h': > + usage(argv[0]); > + exit(EXIT_SUCCESS); > + break; > + case '?': > + error_report("Try `%s --help' for more information.", argv[0]); > + exit(EXIT_FAILURE); > + } > + } > + > + if (!trace_init_backends()) { > + exit(EXIT_FAILURE); > + } > + trace_init_file(); > + qemu_set_log(LOG_TRACE, &error_fatal); > + > + socket_activation = check_socket_activation(); > + if (socket_activation == 0) { > + SocketAddress saddr; > + saddr = (SocketAddress){ > + .type = SOCKET_ADDRESS_TYPE_UNIX, > + .u.q_unix.path = socket_path, > + }; > + server_ioc = qio_channel_socket_new(); > + if (qio_channel_socket_listen_sync(server_ioc, &saddr, > + 1, &local_err) < 0) { > + object_unref(OBJECT(server_ioc)); > + error_report_err(local_err); > + return 1; > + } > + } else { > + /* Using socket activation - check user didn't use -p etc. */ > + if (socket_path_specified) { > + error_report("Unix socket can't be set when" > + "using socket activation"); > + exit(EXIT_FAILURE); > + } > + > + /* Can only listen on a single socket. */ > + if (socket_activation > 1) { > + error_report("%s does not support socket activation" > + "with LISTEN_FDS > 1", > + argv[0]); > + exit(EXIT_FAILURE); > + } > + server_ioc = qio_channel_socket_new_fd(FIRST_SOCKET_ACTIVATION_FD, > + &local_err); > + if (server_ioc == NULL) { > + error_reportf_err(local_err, > + "Failed to use socket activation: "); > + exit(EXIT_FAILURE); > + } > + } > + > + qemu_init_main_loop(&error_fatal); > + > + server_watch = qio_channel_add_watch(QIO_CHANNEL(server_ioc), > + G_IO_IN, > + accept_client, > + NULL, NULL); > + > + if (daemonize) { > + if (daemon(0, 0) < 0) { > + error_report("Failed to daemonize: %s", strerror(errno)); > + exit(EXIT_FAILURE); > + } > + } > + > + if (daemonize || pidfile_specified) { > + qemu_write_pidfile(pidfile, &error_fatal); > + } > + > +#ifdef CONFIG_LIBCAP_NG > + if (drop_privileges() < 0) { > + error_report("Failed to drop privileges: %s", strerror(errno)); > + exit(EXIT_FAILURE); > + } > +#endif > + > + info_report("Listening on %s", socket_path); > + > + state = RUNNING; > + do { > + main_loop_wait(false); > + if (state == TERMINATE) { > + state = TERMINATING; > + close_server_socket(); > + } > + } while (num_active_sockets > 0); > + > + exit(EXIT_SUCCESS); > +} > diff --git a/tools/i386/rapl-msr-index.h b/tools/i386/rapl-msr-index.h > new file mode 100644 > index 000000000000..9a7118639ae3 > --- /dev/null > +++ b/tools/i386/rapl-msr-index.h > @@ -0,0 +1,28 @@ > +/* > + * Allowed list of MSR for Privileged RAPL MSR helper commands for QEMU > + * > + * Copyright (C) 2023 Red Hat, Inc. <aharivel@redhat.com> > + * > + * Author: Anthony Harivel <aharivel@redhat.com> > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License as published by > + * the Free Software Foundation; under version 2 of the License. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program; if not, see <http://www.gnu.org/licenses/>. > + */ > + > +/* > + * Should stay in sync with the RAPL MSR > + * in target/i386/cpu.h > + */ > +#define MSR_RAPL_POWER_UNIT 0x00000606 > +#define MSR_PKG_POWER_LIMIT 0x00000610 > +#define MSR_PKG_ENERGY_STATUS 0x00000611 > +#define MSR_PKG_POWER_INFO 0x00000614 > -- > 2.44.0 > With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v4 2/3] tools: build qemu-vmsr-helper 2024-03-21 11:33 ` Daniel P. Berrangé @ 2024-03-28 10:27 ` Anthony Harivel 2024-03-28 10:46 ` Daniel P. Berrangé 0 siblings, 1 reply; 10+ messages in thread From: Anthony Harivel @ 2024-03-28 10:27 UTC (permalink / raw) To: Daniel P. Berrangé; +Cc: pbonzini, mtosatti, qemu-devel, vchundur, rjarry Hi Daniel, My apologies for all the missed feedback in v2. I'll be more organized for my next iteration. For this specific comment below, I would like to make sure I'm testing the right way. > > diff --git a/meson.build b/meson.build > > index b375248a7614..376da49b60ab 100644 > > --- a/meson.build > > +++ b/meson.build > > @@ -4052,6 +4052,11 @@ if have_tools > > dependencies: [authz, crypto, io, qom, qemuutil, > > libcap_ng, mpathpersist], > > install: true) > > + > > + executable('qemu-vmsr-helper', files('tools/i386/qemu-vmsr-helper.c'), > > + dependencies: [authz, crypto, io, qom, qemuutil, > > + libcap_ng, mpathpersist], > > + install: true) > > endif > > Missed feedback from v2 saying this must /only/ be built > on x86 architectures. It fails to build on others due > to the ASM usage eg > > https://gitlab.com/berrange/qemu/-/jobs/6445384073 > To recreate your build system, I need to, for example, compile with the following configuration for arm64 (aarch64): ../configure --enable-werror --disable-docs --enable-fdt=system --disable-user --cross-prefix=aarch64-linux-gnu- --target-list-exclude="arm-softmmu cris-softmmu i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu sparc-softmmu xtensa-softmmu" This is cross-compiling on x86 right? Because on my laptop I've got the following error: WARNING: unrecognized host CPU, proceeding with 'uname -m' output 'x86_64' python determined to be '/usr/bin/python3' python version: Python 3.12.2 mkvenv: Creating non-isolated virtual environment at 'pyvenv' mkvenv: checking for meson>=0.63.0 ERROR: Unrecognized host OS (uname -s reports 'Linux') It looks like it wants to build natively on aarch64. Maybe I need to create a VM with aarch64 Debian and compile natively? Might take a long time but I'm not sure this is the best way. Regards, Anthony ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v4 2/3] tools: build qemu-vmsr-helper 2024-03-28 10:27 ` Anthony Harivel @ 2024-03-28 10:46 ` Daniel P. Berrangé 0 siblings, 0 replies; 10+ messages in thread From: Daniel P. Berrangé @ 2024-03-28 10:46 UTC (permalink / raw) To: Anthony Harivel; +Cc: pbonzini, mtosatti, qemu-devel, vchundur, rjarry On Thu, Mar 28, 2024 at 11:27:50AM +0100, Anthony Harivel wrote: > Hi Daniel, > > My apologies for all the missed feedback in v2. > I'll be more organized for my next iteration. > > For this specific comment below, I would like to make sure I'm testing > the right way. > > > > diff --git a/meson.build b/meson.build > > > index b375248a7614..376da49b60ab 100644 > > > --- a/meson.build > > > +++ b/meson.build > > > @@ -4052,6 +4052,11 @@ if have_tools > > > dependencies: [authz, crypto, io, qom, qemuutil, > > > libcap_ng, mpathpersist], > > > install: true) > > > + > > > + executable('qemu-vmsr-helper', files('tools/i386/qemu-vmsr-helper.c'), > > > + dependencies: [authz, crypto, io, qom, qemuutil, > > > + libcap_ng, mpathpersist], > > > + install: true) > > > endif > > > > Missed feedback from v2 saying this must /only/ be built > > on x86 architectures. It fails to build on others due > > to the ASM usage eg > > > > https://gitlab.com/berrange/qemu/-/jobs/6445384073 > > > > To recreate your build system, I need to, for example, compile with the > following configuration for arm64 (aarch64): > > ../configure --enable-werror --disable-docs --enable-fdt=system > --disable-user --cross-prefix=aarch64-linux-gnu- > --target-list-exclude="arm-softmmu cris-softmmu i386-softmmu > microblaze-softmmu mips-softmmu mipsel-softmmu mips64-softmmu > ppc-softmmu riscv32-softmmu sh4-softmmu sparc-softmmu xtensa-softmmu" > > This is cross-compiling on x86 right? > Because on my laptop I've got the following error: > > WARNING: unrecognized host CPU, proceeding with 'uname -m' output 'x86_64' > python determined to be '/usr/bin/python3' > python version: Python 3.12.2 > mkvenv: Creating non-isolated virtual environment at 'pyvenv' > mkvenv: checking for meson>=0.63.0 > > ERROR: Unrecognized host OS (uname -s reports 'Linux') > > It looks like it wants to build natively on aarch64. > Maybe I need to create a VM with aarch64 Debian and compile natively? > Might take a long time but I'm not sure this is the best way. You can do this all with our Debian cross build containers locally Either build the container fresh podman build --tag qemuarm64 -f tests/docker/dockerfiles/debian-arm64-cross.docker Or pull down the one from CI podman pull registry.gitlab.com/qemu-project/qemu/qemu/debian-arm64-cross Inside the container you just need to clone your QEMU repo and then run ./configure $QEMU_CONFIGURE_OPTS --target-list=x86_64-softmmu $QEMU_CONFIGURE_OPTS expands to the args needed for a cross-compile, as it set by the container itself. Alternatively fork the project on gitlab, and then push your branch to your fork, while requesting CI git push -o ci.variable=QEMU_CI=1 <gitlab remote> <branch name> then in the gitlab UI, you can press play on the individual jobs you want to test. See docs/devel/testing.rst for info on that. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v4 3/3] Add support for RAPL MSRs in KVM/Qemu 2024-03-18 15:12 [PATCH v4 0/3] Add support for the RAPL MSRs series Anthony Harivel 2024-03-18 15:12 ` [PATCH v4 1/3] qio: add support for SO_PEERCRED for socket channel Anthony Harivel 2024-03-18 15:12 ` [PATCH v4 2/3] tools: build qemu-vmsr-helper Anthony Harivel @ 2024-03-18 15:12 ` Anthony Harivel 2024-03-21 13:35 ` Daniel P. Berrangé 2 siblings, 1 reply; 10+ messages in thread From: Anthony Harivel @ 2024-03-18 15:12 UTC (permalink / raw) To: pbonzini, mtosatti, berrange Cc: qemu-devel, vchundur, rjarry, Anthony Harivel Starting with the "Sandy Bridge" generation, Intel CPUs provide a RAPL interface (Running Average Power Limit) for advertising the accumulated energy consumption of various power domains (e.g. CPU packages, DRAM, etc.). The consumption is reported via MSRs (model specific registers) like MSR_PKG_ENERGY_STATUS for the CPU package power domain. These MSRs are 64 bits registers that represent the accumulated energy consumption in micro Joules. They are updated by microcode every ~1ms. For now, KVM always returns 0 when the guest requests the value of these MSRs. Use the KVM MSR filtering mechanism to allow QEMU handle these MSRs dynamically in userspace. To limit the amount of system calls for every MSR call, create a new thread in QEMU that updates the "virtual" MSR values asynchronously. Each vCPU has its own vMSR to reflect the independence of vCPUs. The thread updates the vMSR values with the ratio of energy consumed of the whole physical CPU package the vCPU thread runs on and the thread's utime and stime values. All other non-vCPU threads are also taken into account. Their energy consumption is evenly distributed among all vCPUs threads running on the same physical CPU package. To overcome the problem that reading the RAPL MSR requires priviliged access, a socket communication between QEMU and the qemu-vmsr-helper is mandatory. You can specified the socket path in the parameter. This feature is activated with -accel kvm,rapl=true,path=/path/sock.sock Actual limitation: - Works only on Intel host CPU because AMD CPUs are using different MSR adresses. - Only the Package Power-Plane (MSR_PKG_ENERGY_STATUS) is reported at the moment. Signed-off-by: Anthony Harivel <aharivel@redhat.com> --- accel/kvm/kvm-all.c | 27 +++ docs/specs/index.rst | 1 + docs/specs/rapl-msr.rst | 155 +++++++++++++ include/sysemu/kvm.h | 2 + include/sysemu/kvm_int.h | 30 +++ target/i386/cpu.h | 8 + target/i386/kvm/kvm-cpu.c | 7 + target/i386/kvm/kvm.c | 420 ++++++++++++++++++++++++++++++++++ target/i386/kvm/meson.build | 1 + target/i386/kvm/vmsr_energy.c | 381 ++++++++++++++++++++++++++++++ target/i386/kvm/vmsr_energy.h | 97 ++++++++ 11 files changed, 1129 insertions(+) create mode 100644 docs/specs/rapl-msr.rst create mode 100644 target/i386/kvm/vmsr_energy.c create mode 100644 target/i386/kvm/vmsr_energy.h diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index a8cecd040ebc..7649f226767a 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -3613,6 +3613,21 @@ static void kvm_set_device(Object *obj, s->device = g_strdup(value); } +static void kvm_set_kvm_rapl(Object *obj, bool value, Error **errp) +{ + KVMState *s = KVM_STATE(obj); + s->msr_energy.enable = value; +} + +static void kvm_set_kvm_rapl_socket_path(Object *obj, + const char *str, + Error **errp) +{ + KVMState *s = KVM_STATE(obj); + g_free(s->msr_energy.socket_path); + s->msr_energy.socket_path = g_strdup(str); +} + static void kvm_accel_instance_init(Object *obj) { KVMState *s = KVM_STATE(obj); @@ -3632,6 +3647,7 @@ static void kvm_accel_instance_init(Object *obj) s->xen_gnttab_max_frames = 64; s->xen_evtchn_max_pirq = 256; s->device = NULL; + s->msr_energy.enable = false; } /** @@ -3676,6 +3692,17 @@ static void kvm_accel_class_init(ObjectClass *oc, void *data) object_class_property_set_description(oc, "device", "Path to the device node to use (default: /dev/kvm)"); + object_class_property_add_bool(oc, "rapl", + NULL, + kvm_set_kvm_rapl); + object_class_property_set_description(oc, "rapl", + "Allow energy related MSRs for RAPL interface in Guest"); + + object_class_property_add_str(oc, "rapl-helper-socket", NULL, + kvm_set_kvm_rapl_socket_path); + object_class_property_set_description(oc, "rapl-helper-socket", + "Socket Path for comminucating with the Virtual MSR helper daemon"); + kvm_arch_accel_class_init(oc); } diff --git a/docs/specs/index.rst b/docs/specs/index.rst index 1484e3e76077..e738ea7d102f 100644 --- a/docs/specs/index.rst +++ b/docs/specs/index.rst @@ -33,3 +33,4 @@ guest hardware that is specific to QEMU. virt-ctlr vmcoreinfo vmgenid + rapl-msr diff --git a/docs/specs/rapl-msr.rst b/docs/specs/rapl-msr.rst new file mode 100644 index 000000000000..1202ee89bee0 --- /dev/null +++ b/docs/specs/rapl-msr.rst @@ -0,0 +1,155 @@ +================ +RAPL MSR support +================ + +The RAPL interface (Running Average Power Limit) is advertising the accumulated +energy consumption of various power domains (e.g. CPU packages, DRAM, etc.). + +The consumption is reported via MSRs (model specific registers) like +MSR_PKG_ENERGY_STATUS for the CPU package power domain. These MSRs are 64 bits +registers that represent the accumulated energy consumption in micro Joules. + +Thanks to the MSR Filtering patch [#a]_ not all MSRs are handled by KVM. Some +of them can now be handled by the userspace (QEMU). It uses a mechanism called +"MSR filtering" where a list of MSRs is given at init time of a VM to KVM so +that a callback is put in place. The design of this patch uses only this +mechanism for handling the MSRs between guest/host. + +At the moment the following MSRs are involved: + +.. code:: C + + #define MSR_RAPL_POWER_UNIT 0x00000606 + #define MSR_PKG_POWER_LIMIT 0x00000610 + #define MSR_PKG_ENERGY_STATUS 0x00000611 + #define MSR_PKG_POWER_INFO 0x00000614 + +The ``*_POWER_UNIT``, ``*_POWER_LIMIT``, ``*_POWER INFO`` are part of the RAPL +spec and specify the power limit of the package, provide range of parameter(min +power, max power,..) and also the information of the multiplier for the energy +counter to calculate the power. Those MSRs are populated once at the beginning +by reading the host CPU MSRs and are given back to the guest 1:1 when +requested. + +The MSR_PKG_ENERGY_STATUS is a counter; it represents the total amount of +energy consumed since the last time the register was cleared. If you multiply +it with the UNIT provided above you'll get the power in micro-joules. This +counter is always increasing and it increases more or less faster depending on +the consumption of the package. This counter is supposed to overflow at some +point. + +Each core belonging to the same Package reading the MSR_PKG_ENERGY_STATUS (i.e +"rdmsr 0x611") will retrieve the same value. The value represents the energy +for the whole package. Whatever Core reading it will get the same value and a +core that belongs to PKG-0 will not be able to get the value of PKG-1 and +vice-versa. + +High level implementation +------------------------- + +In order to update the value of the virtual MSR, a QEMU thread is created. +The thread is basically just an infinity loop that does: + +1. Snapshot of the time metrics of all QEMU threads (Time spent scheduled in + Userspace and System) + +2. Snapshot of the actual MSR_PKG_ENERGY_STATUS counter of all packages where + the QEMU threads are running on. + +3. Sleep for 1 second - During this pause the vcpu and other non-vcpu threads + will do what they have to do and so the energy counter will increase. + +4. Repeat 2. and 3. and calculate the delta of every metrics representing the + time spent scheduled for each QEMU thread *and* the energy spent by the + packages during the pause. + +5. Filter the vcpu threads and the non-vcpu threads. + +6. Retrieve the topology of the Virtual Machine. This helps identify which + vCPU is running on which virtual package. + +7. The total energy spent by the non-vcpu threads is divided by the number + of vcpu threads so that each vcpu thread will get an equal part of the + energy spent by the QEMU workers. + +8. Calculate the ratio of energy spent per vcpu threads. + +9. Calculate the energy for each virtual package. + +10. The virtual MSRs are updated for each virtual package. Each vCPU that + belongs to the same package will return the same value when accessing the + the MSR. + +11. Loop back to 1. + +Ratio calculation +----------------- + +In Linux, a process has an execution time associated with it. The scheduler is +dividing the time in clock ticks. The number of clock ticks per second can be +found by the sysconf system call. A typical value of clock ticks per second is +100. So a core can run a process at the maximum of 100 ticks per second. If a +package has 4 cores, 400 ticks maximum can be scheduled on all the cores +of the package for a period of 1 second. + +The /proc/[pid]/stat [#b]_ is a sysfs file that can give the executed time of a +process with the [pid] as the process ID. It gives the amount of ticks the +process has been scheduled in userspace (utime) and kernel space (stime). + +By reading those metrics for a thread, one can calculate the ratio of time the +package has spent executing the thread. + +Example: + +A 4 cores package can schedule a maximum of 400 ticks per second with 100 ticks +per second per core. If a thread was scheduled for 100 ticks between a second +on this package, that means my thread has been scheduled for 1/4 of the whole +package. With that, the calculation of the energy spent by the thread on this +package during this whole second is 1/4 of the total energy spent by the +package. + +Usage +----- + +Currently this feature is only working on an Intel CPU that has the RAPL driver +mounted and available in the sysfs. if not, QEMU fails at start-up. + +This feature is activated with -accel +kvm,rapl=true,rapl-helper-socket=/path/sock.sock + +It is important that the socket path is the same as the one +:program:`qemu-vmsr-helper` is listening to. + +qemu-vmsr-helper +---------------- + +The qemu-vmsr-helper is working very much like the qemu-pr-helper. Instead of +making persistent reservation, qemu-vmsr-helper is here to overcome the +CVE-2020-8694 which remove user access to the rapl msr attributes. + +A socket communication is established between QEMU processes that has the RAPL +MSR support activated and the qemu-vmsr-helper. A systemd service and socket +activation is provided in contrib/systemd/qemu-vmsr-helper.(service/socket). + +The systemd socket uses 600, like contrib/systemd/qemu-pr-helper.socket. The +socket can be passed via SCM_RIGHTS by libvirt, or its permissions can be +changed (e.g. 660 and root:kvm for a Debian system for example). Libvirt could +also start a separate helper if needed. All in all, the policy is left to the +user. + +See the qemu-pr-helper documentation or manpage for further details. + +Current Limitations +------------------- + +- Works only on Intel host CPUs because AMD CPUs are using different MSR + addresses. + +- Only the Package Power-Plane (MSR_PKG_ENERGY_STATUS) is reported at the + moment. + +References +---------- + +.. [#a] https://patchwork.kernel.org/project/kvm/patch/20200916202951.23760-7-graf@amazon.com/ +.. [#b] https://man7.org/linux/man-pages/man5/proc.5.html diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index fad9a7e8ff30..37f68c496807 100644 --- a/include/sysemu/kvm.h +++ b/include/sysemu/kvm.h @@ -544,4 +544,6 @@ uint32_t kvm_dirty_ring_size(void); * reported for the VM. */ bool kvm_hwpoisoned_mem(void); + +bool kvm_is_rapl_feat_enable(CPUState *cs); #endif diff --git a/include/sysemu/kvm_int.h b/include/sysemu/kvm_int.h index 882e37e12c5b..ea4587b53eb1 100644 --- a/include/sysemu/kvm_int.h +++ b/include/sysemu/kvm_int.h @@ -14,6 +14,8 @@ #include "qemu/accel.h" #include "qemu/queue.h" #include "sysemu/kvm.h" +#include "hw/boards.h" +#include "hw/i386/topology.h" typedef struct KVMSlot { @@ -48,6 +50,33 @@ typedef struct KVMMemoryListener { #define KVM_MSI_HASHTAB_SIZE 256 +typedef struct KVMHostTopoInfo { + /* Number of package on the Host */ + unsigned int maxpkgs; + /* Number of cpus on the Host */ + unsigned int maxcpus; + /* Number of cpus on each different package */ + unsigned int *pkg_cpu_count; + /* Each package can have different maxticks */ + unsigned int *maxticks; +} KVMHostTopoInfo; + +struct KVMMsrEnergy { + pid_t pid; + bool enable; + char *socket_path; + QemuThread msr_thr; + unsigned int vcpus; + unsigned int vsockets; + X86CPUTopoInfo topo_info; + KVMHostTopoInfo host_topo; + const CPUArchIdList *cpu_list; + uint64_t *msr_value; + uint64_t msr_unit; + uint64_t msr_limit; + uint64_t msr_info; +}; + enum KVMDirtyRingReaperState { KVM_DIRTY_RING_REAPER_NONE = 0, /* The reaper is sleeping */ @@ -114,6 +143,7 @@ struct KVMState bool kvm_dirty_ring_with_bitmap; uint64_t kvm_eager_split_size; /* Eager Page Splitting chunk size */ struct KVMDirtyRingReaper reaper; + struct KVMMsrEnergy msr_energy; NotifyVmexitOption notify_vmexit; uint32_t notify_window; uint32_t xen_version; diff --git a/target/i386/cpu.h b/target/i386/cpu.h index 952174bb6f52..135bda8a5db9 100644 --- a/target/i386/cpu.h +++ b/target/i386/cpu.h @@ -400,6 +400,10 @@ typedef enum X86Seg { #define MSR_IA32_TSX_CTRL 0x122 #define MSR_IA32_TSCDEADLINE 0x6e0 #define MSR_IA32_PKRS 0x6e1 +#define MSR_RAPL_POWER_UNIT 0x00000606 +#define MSR_PKG_POWER_LIMIT 0x00000610 +#define MSR_PKG_ENERGY_STATUS 0x00000611 +#define MSR_PKG_POWER_INFO 0x00000614 #define MSR_ARCH_LBR_CTL 0x000014ce #define MSR_ARCH_LBR_DEPTH 0x000014cf #define MSR_ARCH_LBR_FROM_0 0x00001500 @@ -1795,6 +1799,10 @@ typedef struct CPUArchState { uintptr_t retaddr; + /* RAPL MSR */ + uint64_t msr_rapl_power_unit; + uint64_t msr_pkg_energy_status; + /* Fields up to this point are cleared by a CPU reset */ struct {} end_reset_fields; diff --git a/target/i386/kvm/kvm-cpu.c b/target/i386/kvm/kvm-cpu.c index 9c791b7b0520..246de2bac2f1 100644 --- a/target/i386/kvm/kvm-cpu.c +++ b/target/i386/kvm/kvm-cpu.c @@ -50,6 +50,13 @@ static bool kvm_cpu_realizefn(CPUState *cs, Error **errp) MSR_IA32_UCODE_REV); } } + if (kvm_is_rapl_feat_enable(cs)) { + if (IS_INTEL_CPU(env)) { + error_setg(errp, "RAPL feature is enable and CPU is not INTEL CPU"); + return false; + }; + }; + return host_cpu_realizefn(cs, errp); } diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c index e68cbe929302..27dacdf1aa54 100644 --- a/target/i386/kvm/kvm.c +++ b/target/i386/kvm/kvm.c @@ -16,9 +16,12 @@ #include "qapi/qapi-events-run-state.h" #include "qapi/error.h" #include "qapi/visitor.h" +#include <math.h> #include <sys/ioctl.h> #include <sys/utsname.h> #include <sys/syscall.h> +#include <sys/resource.h> +#include <sys/time.h> #include <linux/kvm.h> #include "standard-headers/asm-x86/kvm_para.h" @@ -26,6 +29,7 @@ #include "cpu.h" #include "host-cpu.h" +#include "vmsr_energy.h" #include "sysemu/sysemu.h" #include "sysemu/hw_accel.h" #include "sysemu/kvm_int.h" @@ -2479,6 +2483,54 @@ static bool kvm_rdmsr_core_thread_count(X86CPU *cpu, uint32_t msr, return true; } +static bool kvm_rdmsr_rapl_power_unit(X86CPU *cpu, uint32_t msr, + uint64_t *val) +{ + + CPUState *cs = CPU(cpu); + + *val = cs->kvm_state->msr_energy.msr_unit; + + return true; +} + +static bool kvm_rdmsr_pkg_power_limit(X86CPU *cpu, uint32_t msr, + uint64_t *val) +{ + + CPUState *cs = CPU(cpu); + + *val = cs->kvm_state->msr_energy.msr_limit; + + return true; +} + +static bool kvm_rdmsr_pkg_power_info(X86CPU *cpu, uint32_t msr, + uint64_t *val) +{ + + CPUState *cs = CPU(cpu); + + *val = cs->kvm_state->msr_energy.msr_info; + + return true; +} + +static bool kvm_rdmsr_pkg_energy_status(X86CPU *cpu, uint32_t msr, + uint64_t *val) +{ + + CPUState *cs = CPU(cpu); + *val = cs->kvm_state->msr_energy.msr_value[cs->cpu_index]; + + return true; +} + +bool kvm_is_rapl_feat_enable(CPUState *cs) +{ + return cs->kvm_state->msr_energy.enable; +} + static Notifier smram_machine_done; static KVMMemoryListener smram_listener; static AddressSpace smram_address_space; @@ -2513,6 +2565,331 @@ static void register_smram_listener(Notifier *n, void *unused) &smram_address_space, 1, "kvm-smram"); } +static void *kvm_msr_energy_thread(void *data) +{ + KVMState *s = data; + struct KVMMsrEnergy *vmsr = &s->msr_energy; + + g_autofree package_energy_stat *pkg_stat = NULL; + g_autofree thread_stat *thd_stat = NULL; + g_autofree pid_t *thread_ids = NULL; + g_autofree CPUState *cpu = NULL; + g_autofree unsigned int *vpkgs_energy_stat = NULL; + unsigned int num_threads = 0; + unsigned int tmp_num_threads = 0; + + X86CPUTopoIDs topo_ids; + + rcu_register_thread(); + + /* Allocate memory for each package energy status */ + pkg_stat = g_new0(package_energy_stat, vmsr->host_topo.maxpkgs); + + /* Allocate memory for thread stats */ + thd_stat = g_new0(thread_stat, 1); + + /* Allocate memory for holding virtual package energy counter */ + vpkgs_energy_stat = g_new0(unsigned int, vmsr->vsockets); + + /* Populate the max tick of each packages */ + for (int i = 0; i <= vmsr->host_topo.maxpkgs; i++) { + /* + * Max numbers of ticks per package + * Time in second * Number of ticks/second * Number of cores/package + * ex: 100 ticks/second/CPU, 12 CPUs per Package gives 1200 ticks max + */ + vmsr->host_topo.maxticks[i] = (MSR_ENERGY_THREAD_SLEEP_US / 1000000) + * sysconf(_SC_CLK_TCK) + * vmsr->host_topo.pkg_cpu_count[i]; + } + + while (true) { + /* Get all qemu threads id */ + thread_ids = vmsr_get_thread_ids(vmsr->pid, &num_threads); + + if (thread_ids == NULL) { + goto clean; + } + + if (tmp_num_threads < num_threads) { + thd_stat = g_renew(thread_stat, thd_stat, num_threads); + } + + tmp_num_threads = num_threads; + + /* Populate all the thread stats */ + for (int i = 0; i < num_threads; i++) { + thd_stat[i].utime = g_new0(unsigned long long, 2); + thd_stat[i].stime = g_new0(unsigned long long, 2); + thd_stat[i].thread_id = thread_ids[i]; + vmsr_read_thread_stat(vmsr->pid, + thd_stat[i].thread_id, + thd_stat[i].utime, + thd_stat[i].stime, + &thd_stat[i].cpu_id); + thd_stat[i].pkg_id = + vmsr_get_physical_package_id(thd_stat[i].cpu_id); + } + + /* Retrieve all packages power plane energy counter */ + for (int i = 0; i < vmsr->host_topo.maxpkgs; i++) { + for (int j = 0; j < num_threads; j++) { + /* + * Use the first thread we found that ran on the CPU + * of the package to read the packages energy counter + */ + if (thd_stat[j].pkg_id == i) { + pkg_stat[i].e_start = + vmsr_read_msr(MSR_PKG_ENERGY_STATUS, + thd_stat[j].cpu_id, + thd_stat[j].thread_id, + s->msr_energy.socket_path); + break; + } + } + } + + /* Sleep a short period while the other threads are working */ + usleep(MSR_ENERGY_THREAD_SLEEP_US); + + /* + * Retrieve all packages power plane energy counter + * Calculate the delta of all packages + */ + for (int i = 0; i < vmsr->host_topo.maxpkgs; i++) { + for (int j = 0; j < num_threads; j++) { + /* + * Use the first thread we found that ran on the CPU + * of the package to read the packages energy counter + */ + if (thd_stat[j].pkg_id == i) { + pkg_stat[i].e_end = + vmsr_read_msr(MSR_PKG_ENERGY_STATUS, + thd_stat[j].cpu_id, + thd_stat[j].thread_id, + s->msr_energy.socket_path); + /* + * Prevent the case we have migrate the VM + * during the sleep period or any other cases + * were energy counter might be lower after + * the sleep period. + */ + if (pkg_stat[i].e_end > pkg_stat[i].e_start) { + pkg_stat[i].e_delta = + pkg_stat[i].e_end - pkg_stat[i].e_start; + } else { + pkg_stat[i].e_delta = 0; + } + break; + } + } + } + + /* Delta of ticks spend by each thread between the sample */ + for (int i = 0; i < num_threads; i++) { + vmsr_read_thread_stat(vmsr->pid, + thd_stat[i].thread_id, + thd_stat[i].utime, + thd_stat[i].stime, + &thd_stat[i].cpu_id); + + if (vmsr->pid < 0) { + /* + * We don't count the dead thread + * i.e threads that existed before the sleep + * and not anymore + */ + thd_stat[i].delta_ticks = 0; + } else { + vmsr_delta_ticks(thd_stat, i); + } + } + + /* + * Identify the vcpu threads + * Calculate the number of vcpu per package + */ + CPU_FOREACH(cpu) { + for (int i = 0; i < num_threads; i++) { + if (cpu->thread_id == thd_stat[i].thread_id) { + thd_stat[i].is_vcpu = true; + thd_stat[i].vcpu_id = cpu->cpu_index; + pkg_stat[thd_stat[i].pkg_id].nb_vcpu++; + thd_stat[i].acpi_id = kvm_arch_vcpu_id(cpu); + break; + } + } + } + + /* Retrieve the virtual package number of each vCPU */ + for (int i = 0; i < vmsr->x86_cpu_list->len; i++) { + for (int j = 0; j < num_threads; j++) { + if ((thd_stat[j].acpi_id == vmsr->x86_cpu_list->cpus[i].arch_id) + && (thd_stat[j].is_vcpu == true)) { + x86_topo_ids_from_apicid(thd_stat[j].acpi_id, + &vmsr->topo_info, &topo_ids); + thd_stat[j].vpkg_id = topo_ids.pkg_id; + } + } + } + + /* Calculate the total energy of all non-vCPU thread */ + for (int i = 0; i < num_threads; i++) { + double temp; + if ((thd_stat[i].is_vcpu != true) && + (thd_stat[i].delta_ticks > 0)) { + temp = vmsr_get_ratio(pkg_stat[thd_stat[i].pkg_id].e_delta, + thd_stat[i].delta_ticks, + vmsr->host_topo.maxticks[thd_stat[i].pkg_id]); + pkg_stat[thd_stat[i].pkg_id].e_ratio + += (uint64_t)lround(temp); + } + } + + /* Calculate the ratio per non-vCPU thread of each package */ + for (int i = 0; i < vmsr->host_topo.maxpkgs; i++) { + if (pkg_stat[i].nb_vcpu > 0) { + pkg_stat[i].e_ratio = pkg_stat[i].e_ratio / pkg_stat[i].nb_vcpu; + } + } + + /* + * Calculate the energy for each Package: + * Energy Package = sum of each vCPU energy that belongs to the package + */ + for (int i = 0; i < num_threads; i++) { + double temp; + + if ((thd_stat[i].is_vcpu == true) && \ + (thd_stat[i].delta_ticks > 0)) { + temp = vmsr_get_ratio(pkg_stat[thd_stat[i].pkg_id].e_delta, + thd_stat[i].delta_ticks, + vmsr->host_topo.maxticks[thd_stat[i].pkg_id]); + vpkgs_energy_stat[thd_stat[i].vpkg_id] += + (uint64_t)lround(temp); + vpkgs_energy_stat[thd_stat[i].vpkg_id] += + pkg_stat[thd_stat[i].pkg_id].e_ratio; + } + } + + /* + * Finally populate the vmsr register of each vCPU with the total + * package value to emulate the real hardware where each CPU return the + * value of the package it belongs. + */ + for (int i = 0; i < num_threads; i++) { + if ((thd_stat[i].is_vcpu == true) && \ + (thd_stat[i].delta_ticks > 0)) { + vmsr->msr_value[thd_stat[i].vcpu_id] = \ + vpkgs_energy_stat[thd_stat[i].vpkg_id]; + } + } + + /* Freeing memory before zeroing the pointer */ + for (int i = 0; i < num_threads; i++) { + g_free(thd_stat[i].utime); + g_free(thd_stat[i].stime); + } + /* Zeroing the reused memory with g_renew */ + memset(thd_stat, 0, num_threads * sizeof(thread_stat)); + memset(thread_ids, 0, sizeof(pid_t)); + } + +clean: + rcu_unregister_thread(); + return NULL; +} + +static int kvm_msr_energy_thread_init(KVMState *s, MachineState *ms) +{ + struct KVMMsrEnergy *r = &s->msr_energy; + int ret = 0; + + /* + * Sanity check + * 1. Host cpu must be Intel cpu + * 2. RAPL must be enabled on the Host + */ + if (is_host_cpu_intel()) { + error_report("error: Host CPU is not INTEL cpu"); + ret = 1; + goto out; + } + + if (!is_rapl_enabled()) { + error_report("error: RAPL driver not enable on Host"); + ret = 1; + goto out; + } + + /* Retrieve the virtual topology */ + vmsr_init_topo_info(&r->topo_info, ms); + + /* Retrieve the number of vcpu */ + r->vcpus = ms->smp.cpus; + + /* Retrieve the number of virtual sockets */ + r->vsockets = ms->smp.sockets; + + /* Allocate register memory (MSR_PKG_STATUS) for each vcpu */ + r->msr_value = g_new0(uint64_t, r->vcpus); + + /* Retrieve the CPUArchIDlist */ + r->x86_cpu_list = x86_possible_cpu_arch_ids(ms); + + /* Max number of cpus on the Host */ + r->host_topo.maxcpus = vmsr_get_maxcpus(); + if (r->host_topo.maxcpus <= 0) { + error_report("error: host max cpus <= 0"); + ret = 1; + goto out; + } + + /* Max number of packages on the host */ + r->host_topo.maxpkgs = vmsr_get_max_physical_package(r->host_topo.maxcpus); + if (r->host_topo.maxpkgs <= 0) { + error_report("error: host max pkgs <= 0"); + ret = 1; + goto out; + } + + /* Allocate memory for each package on the host */ + r->host_topo.pkg_cpu_count = g_new0(unsigned int, r->host_topo.maxpkgs); + r->host_topo.maxticks = g_new0(unsigned int, r->host_topo.maxpkgs); + + vmsr_count_cpus_per_package(r->host_topo.pkg_cpu_count, + r->host_topo.maxpkgs); + for (int i = 0; i < r->host_topo.maxpkgs; i++) { + if (r->host_topo.pkg_cpu_count[i] <= 0) { + error_report("error: cpu per packages <= 0 on package_%d", i); + ret = 1; + goto out; + } + } + + /* Get QEMU PID*/ + r->pid = getpid(); + + /* Those MSR values should not change */ + r->msr_unit = vmsr_read_msr(MSR_RAPL_POWER_UNIT, 0, r->pid, + s->msr_energy.socket_path); + r->msr_limit = vmsr_read_msr(MSR_PKG_POWER_LIMIT, 0, r->pid, + s->msr_energy.socket_path); + r->msr_info = vmsr_read_msr(MSR_PKG_POWER_INFO, 0, r->pid, + s->msr_energy.socket_path); + if (r->msr_unit == 0 || r->msr_limit == 0 || r->msr_info == 0) { + error_report("error: can't read any virtual msr"); + ret = 1; + goto out; + } + + qemu_thread_create(&r->msr_thr, "kvm-msr", + kvm_msr_energy_thread, + s, QEMU_THREAD_JOINABLE); +out: + return ret; +} + int kvm_arch_get_default_type(MachineState *ms) { return 0; @@ -2715,6 +3092,49 @@ int kvm_arch_init(MachineState *ms, KVMState *s) strerror(-ret)); exit(1); } + + if (s->msr_energy.enable == true) { + r = kvm_filter_msr(s, MSR_RAPL_POWER_UNIT, + kvm_rdmsr_rapl_power_unit, NULL); + if (!r) { + error_report("Could not install MSR_RAPL_POWER_UNIT \ + handler: %s", + strerror(-ret)); + exit(1); + } + + r = kvm_filter_msr(s, MSR_PKG_POWER_LIMIT, + kvm_rdmsr_pkg_power_limit, NULL); + if (!r) { + error_report("Could not install MSR_PKG_POWER_LIMIT \ + handler: %s", + strerror(-ret)); + exit(1); + } + + r = kvm_filter_msr(s, MSR_PKG_POWER_INFO, + kvm_rdmsr_pkg_power_info, NULL); + if (!r) { + error_report("Could not install MSR_PKG_POWER_INFO \ + handler: %s", + strerror(-ret)); + exit(1); + } + r = kvm_filter_msr(s, MSR_PKG_ENERGY_STATUS, + kvm_rdmsr_pkg_energy_status, NULL); + if (!r) { + error_report("Could not install MSR_PKG_ENERGY_STATUS \ + handler: %s", + strerror(-ret)); + exit(1); + } + r = kvm_msr_energy_thread_init(s, ms); + if (r) { + error_report("kvm : error RAPL feature requirement not meet"); + exit(1); + } + + } } return 0; diff --git a/target/i386/kvm/meson.build b/target/i386/kvm/meson.build index 84d9143e6029..16010638df69 100644 --- a/target/i386/kvm/meson.build +++ b/target/i386/kvm/meson.build @@ -3,6 +3,7 @@ i386_kvm_ss = ss.source_set() i386_kvm_ss.add(files( 'kvm.c', 'kvm-cpu.c', + 'vmsr_energy.c', )) i386_kvm_ss.add(when: 'CONFIG_XEN_EMU', if_true: files('xen-emu.c')) diff --git a/target/i386/kvm/vmsr_energy.c b/target/i386/kvm/vmsr_energy.c new file mode 100644 index 000000000000..20ac50c3f0b6 --- /dev/null +++ b/target/i386/kvm/vmsr_energy.c @@ -0,0 +1,381 @@ +/* + * QEMU KVM support -- x86 virtual RAPL msr + * + * Copyright 2024 Red Hat, Inc. 2024 + * + * Author: + * Anthony Harivel <aharivel@redhat.com> + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include "qemu/error-report.h" +#include "vmsr_energy.h" +#include "io/channel.h" +#include "io/channel-socket.h" +#include "hw/boards.h" +#include "cpu.h" +#include "host-cpu.h" + +static char *compute_default_paths(void) +{ + g_autofree char *state = qemu_get_local_state_dir(); + + return g_build_filename(state, "run", "qemu-vmsr-helper.sock", NULL); +} + +bool is_host_cpu_intel(void) +{ + int family, model, stepping; + char vendor[CPUID_VENDOR_SZ + 1]; + + host_cpu_vendor_fms(vendor, &family, &model, &stepping); + + return strcmp(vendor, CPUID_VENDOR_INTEL); +} + +int is_rapl_enabled(void) +{ + const char *path = "/sys/class/powercap/intel-rapl/enabled"; + FILE *file = fopen(path, "r"); + int value = 0; + + if (file != NULL) { + if (fscanf(file, "%d", &value) != 1) { + error_report("INTEL RAPL not enabled"); + } + fclose(file); + } else { + error_report("Error opening %s", path); + } + + return value; +} + +static int vmsr_helper_socket_write(QIOChannel *ioc, + int fd, + const void *buf, int sz, Error **errp) +{ + size_t nfds = (fd != -1); + while (sz > 0) { + struct iovec iov; + ssize_t n_written; + + iov.iov_base = (void *)buf; + iov.iov_len = sz; + n_written = qio_channel_writev_full(QIO_CHANNEL(ioc), &iov, 1, + nfds ? &fd : NULL, nfds, 0, errp); + + if (n_written <= 0) { + assert(n_written != QIO_CHANNEL_ERR_BLOCK); + object_unref(OBJECT(ioc)); + ioc = NULL; + return n_written < 0 ? -EINVAL : 0; + } + + nfds = 0; + buf += n_written; + sz -= n_written; + } + + return 0; +} + +static int vmsr_helper_socket_read(QIOChannel *ioc, + void *buf, int sz, Error **errp) +{ + ssize_t r = qio_channel_read_all(ioc, buf, sz, errp); + + if (r < 0) { + object_unref(OBJECT(ioc)); + ioc = NULL; + return -EINVAL; + } + + return 0; +} + +uint64_t vmsr_read_msr(uint32_t reg, unsigned int cpu_id, uint32_t tid, + const char *path) +{ + uint64_t data = 0; + char *socket_path = NULL; + char buffer[3]; + + if (path == NULL) { + socket_path = compute_default_paths(); + } else { + socket_path = g_strdup(path); + } + + SocketAddress saddr = { + .type = SOCKET_ADDRESS_TYPE_UNIX, + .u.q_unix.path = socket_path + }; + QIOChannelSocket *sioc = qio_channel_socket_new(); + Error *local_err = NULL; + + int r; + + qio_channel_set_name(QIO_CHANNEL(sioc), "vmsr-helper"); + qio_channel_socket_connect_sync(sioc, + &saddr, + &local_err); + g_free(socket_path); + if (local_err) { + goto out_close; + } + + /* + * Send the required arguments: + * 1. RAPL MSR register to read + * 2. On which CPU ID + * 3. From which vCPU (Thread ID) + */ + buffer[0] = reg; + buffer[1] = cpu_id; + buffer[2] = tid; + + r = vmsr_helper_socket_write(QIO_CHANNEL(sioc), + -1, + &buffer, sizeof(buffer), + &local_err); + if (r < 0) { + goto out_close; + } + + r = vmsr_helper_socket_read(QIO_CHANNEL(sioc), + &data, sizeof(data), + &local_err); + if (r < 0) { + data = 0; + goto out_close; + } + +out_close: + /* Close socket. */ + qio_channel_close(QIO_CHANNEL(sioc), NULL); + object_unref(OBJECT(sioc)); + return data; +} + +/* Retrieve the max number of physical package */ +unsigned int vmsr_get_max_physical_package(unsigned int max_cpus) +{ + const char *dir = "/sys/devices/system/cpu/"; + const char *topo_path = "topology/physical_package_id"; + g_autofree int *uniquePackages = 0; + unsigned int packageCount = 0; + FILE *file = NULL; + + uniquePackages = g_new0(int, max_cpus); + + for (int i = 0; i < max_cpus; i++) { + g_autofree char *filePath = NULL; + g_autofree char *cpuid = g_strdup_printf("cpu%d", i); + + filePath = g_build_filename(dir, cpuid, topo_path, NULL); + + file = fopen(filePath, "r"); + + if (file == NULL) { + error_report("Error opening physical_package_id file"); + return 0; + } + + char packageId[10]; + if (fgets(packageId, sizeof(packageId), file) == NULL) { + packageCount = 0; + } + + fclose(file); + + int currentPackageId = atoi(packageId); + + bool isUnique = true; + for (int j = 0; j < packageCount; j++) { + if (uniquePackages[j] == currentPackageId) { + isUnique = false; + break; + } + } + + if (isUnique) { + uniquePackages[packageCount] = currentPackageId; + packageCount++; + + if (packageCount >= max_cpus) { + break; + } + } + } + + return (packageCount == 0) ? 1 : packageCount; +} + +/* Retrieve the max number of physical cpu on the host */ +unsigned int vmsr_get_maxcpus(void) +{ + GDir *dir; + const gchar *entry_name; + unsigned int cpu_count = 0; + const char *path = "/sys/devices/system/cpu/"; + + dir = g_dir_open(path, 0, NULL); + if (dir == NULL) { + error_report("Unable to open cpu directory"); + return -1; + } + + while ((entry_name = g_dir_read_name(dir)) != NULL) { + if (g_ascii_strncasecmp(entry_name, "cpu", 3) == 0 && + isdigit(entry_name[3])) { + cpu_count++; + } + } + + g_dir_close(dir); + + return cpu_count; +} + +/* Count the number of physical cpu on each packages */ +unsigned int vmsr_count_cpus_per_package(unsigned int *package_count, + unsigned int max_pkgs) +{ + g_autofree char *file_contents = NULL; + g_autofree char *path = NULL; + gsize length; + + /* Iterate over cpus and count cpus in each package */ + for (int cpu_id = 0; ; cpu_id++) { + path = g_build_filename( + g_strdup_printf("/sys/devices/system/cpu/cpu%d/" + "topology/physical_package_id", cpu_id), NULL); + + if (!g_file_get_contents(path, &file_contents, &length, NULL)) { + break; /* No more cpus */ + } + + /* Get the physical package ID for this CPU */ + int package_id = atoi(file_contents); + + /* Check if the package ID is within the known number of packages */ + if (package_id >= 0 && package_id < max_pkgs) { + /* If yes, count the cpu for this package*/ + package_count[package_id]++; + } + } + + return 0; +} + +/* Get the physical package id from a given cpu id */ +int vmsr_get_physical_package_id(int cpu_id) +{ + g_autofree char *file_contents = NULL; + g_autofree char *file_path = NULL; + int package_id = -1; + gsize length; + + file_path = g_strdup_printf("/sys/devices/system/cpu/cpu%d\ + /topology/physical_package_id", cpu_id); + + if (!g_file_get_contents(file_path, &file_contents, &length, NULL)) { + goto out; + } + + package_id = atoi(file_contents); + +out: + return package_id; +} + +/* Read the scheduled time for a given thread of a give pid */ +void vmsr_read_thread_stat(pid_t pid, + unsigned int thread_id, + unsigned long long *utime, + unsigned long long *stime, + unsigned int *cpu_id) +{ + g_autofree char *path = NULL; + + path = g_build_filename(g_strdup_printf("/proc/%u/task/%d/stat", pid, \ + thread_id), NULL); + + FILE *file = fopen(path, "r"); + if (file == NULL) { + pid = -1; + return; + } + + if (fscanf(file, "%*d (%*[^)]) %*c %*d %*d %*d %*d %*d %*u %*u %*u %*u %*u" + " %llu %llu %*d %*d %*d %*d %*d %*d %*u %*u %*d %*u %*u" + " %*u %*u %*u %*u %*u %*u %*u %*u %*u %*d %*u %*u %u", + utime, stime, cpu_id) != 3) + { + pid = -1; + return; + } + + fclose(file); + return; +} + +/* Read QEMU stat task folder to retrieve all QEMU threads ID */ +pid_t *vmsr_get_thread_ids(pid_t pid, unsigned int *num_threads) +{ + g_autofree char *path = g_build_filename("/proc", + g_strdup_printf("%d/task", pid), NULL); + + DIR *dir = opendir(path); + if (dir == NULL) { + error_report("Error opening /proc/qemu/task"); + return NULL; + } + + pid_t *thread_ids = NULL; + unsigned int thread_count = 0; + + g_autofree struct dirent *ent = NULL; + while ((ent = readdir(dir)) != NULL) { + if (ent->d_name[0] == '.') { + continue; + } + pid_t tid = atoi(ent->d_name); + if (pid != tid) { + thread_ids = g_renew(pid_t, thread_ids, (thread_count + 1)); + thread_ids[thread_count] = tid; + thread_count++; + } + } + + closedir(dir); + + *num_threads = thread_count; + return thread_ids; +} + +void vmsr_delta_ticks(thread_stat *thd_stat, int i) +{ + thd_stat[i].delta_ticks = (thd_stat[i].utime[1] + thd_stat[i].stime[1]) + - (thd_stat[i].utime[0] + thd_stat[i].stime[0]); +} + +double vmsr_get_ratio(uint64_t e_delta, + unsigned long long delta_ticks, + unsigned int maxticks) +{ + return (e_delta / 100.0) * ((100.0 / maxticks) * delta_ticks); +} + +void vmsr_init_topo_info(X86CPUTopoInfo *topo_info, + const MachineState *ms) +{ + topo_info->dies_per_pkg = ms->smp.dies; + topo_info->cores_per_die = ms->smp.cores; + topo_info->threads_per_core = ms->smp.threads; +} diff --git a/target/i386/kvm/vmsr_energy.h b/target/i386/kvm/vmsr_energy.h new file mode 100644 index 000000000000..e8aff1fabdfa --- /dev/null +++ b/target/i386/kvm/vmsr_energy.h @@ -0,0 +1,97 @@ +/* + * QEMU KVM support -- x86 virtual energy-related MSR. + * + * Copyright 2024 Red Hat, Inc. 2024 + * + * Author: + * Anthony Harivel <aharivel@redhat.com> + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#ifndef VMSR_ENERGY_H +#define VMSR_ENERGY_H + +#include "qemu/osdep.h" +#include <numa.h> +#include <stdint.h> +#include "hw/i386/topology.h" + +/* + * Define the interval time in micro seconds between 2 samples of + * energy related MSRs + */ +#define MSR_ENERGY_THREAD_SLEEP_US 1000000.0 + +/* + * Thread statistic + * @ thread_id: TID (thread ID) + * @ is_vcpu: true if TID is vCPU thread + * @ cpu_id: CPU number last executed on + * @ pkg_id: package number of the CPU + * @ vcpu_id: vCPU ID + * @ vpkg: virtual package number + * @ acpi_id: APIC id of the vCPU + * @ utime: amount of clock ticks the thread + * has been scheduled in User mode + * @ stime: amount of clock ticks the thread + * has been scheduled in System mode + * @ delta_ticks: delta of utime+stime between + * the two samples (before/after sleep) + */ +struct thread_stat { + unsigned int thread_id; + bool is_vcpu; + unsigned int cpu_id; + unsigned int pkg_id; + unsigned int vpkg_id; + unsigned int vcpu_id; + unsigned long acpi_id; + unsigned long long *utime; + unsigned long long *stime; + unsigned long long delta_ticks; +}; + +/* + * Package statistic + * @ e_start: package energy counter before the sleep + * @ e_end: package energy counter after the sleep + * @ e_delta: delta of package energy counter + * @ e_ratio: store the energy ratio of non-vCPU thread + * @ nb_vcpu: number of vCPU running on this package + */ +struct package_energy_stat { + uint64_t e_start; + uint64_t e_end; + uint64_t e_delta; + uint64_t e_ratio; + unsigned int nb_vcpu; +}; + +typedef struct thread_stat thread_stat; +typedef struct package_energy_stat package_energy_stat; + +void vmsr_read_thread_stat(pid_t pid, + unsigned int thread_id, + unsigned long long *utime, + unsigned long long *stime, + unsigned int *cpu_id); + +uint64_t vmsr_read_msr(uint32_t reg, unsigned int cpu_id, + unsigned int tid, const char *path); +void vmsr_delta_ticks(thread_stat *thd_stat, int i); +unsigned int vmsr_get_maxcpus(void); +unsigned int vmsr_get_max_physical_package(unsigned int max_cpus); +unsigned int vmsr_count_cpus_per_package(unsigned int *package_count, + unsigned int max_pkgs); +int vmsr_get_physical_package_id(int cpu_id); +pid_t *vmsr_get_thread_ids(pid_t pid, unsigned int *num_threads); +double vmsr_get_ratio(uint64_t e_delta, + unsigned long long delta_ticks, + unsigned int maxticks); +void vmsr_init_topo_info(X86CPUTopoInfo *topo_info, const MachineState *ms); +bool is_host_cpu_intel(void); +int is_rapl_enabled(void); +#endif /* VMSR_ENERGY_H */ -- 2.44.0 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v4 3/3] Add support for RAPL MSRs in KVM/Qemu 2024-03-18 15:12 ` [PATCH v4 3/3] Add support for RAPL MSRs in KVM/Qemu Anthony Harivel @ 2024-03-21 13:35 ` Daniel P. Berrangé 2024-04-05 8:33 ` Anthony Harivel 0 siblings, 1 reply; 10+ messages in thread From: Daniel P. Berrangé @ 2024-03-21 13:35 UTC (permalink / raw) To: Anthony Harivel; +Cc: pbonzini, mtosatti, qemu-devel, vchundur, rjarry On Mon, Mar 18, 2024 at 04:12:16PM +0100, Anthony Harivel wrote: > Starting with the "Sandy Bridge" generation, Intel CPUs provide a RAPL > interface (Running Average Power Limit) for advertising the accumulated > energy consumption of various power domains (e.g. CPU packages, DRAM, > etc.). > > The consumption is reported via MSRs (model specific registers) like > MSR_PKG_ENERGY_STATUS for the CPU package power domain. These MSRs are > 64 bits registers that represent the accumulated energy consumption in > micro Joules. They are updated by microcode every ~1ms. > > For now, KVM always returns 0 when the guest requests the value of > these MSRs. Use the KVM MSR filtering mechanism to allow QEMU handle > these MSRs dynamically in userspace. > > To limit the amount of system calls for every MSR call, create a new > thread in QEMU that updates the "virtual" MSR values asynchronously. > > Each vCPU has its own vMSR to reflect the independence of vCPUs. The > thread updates the vMSR values with the ratio of energy consumed of > the whole physical CPU package the vCPU thread runs on and the > thread's utime and stime values. > > All other non-vCPU threads are also taken into account. Their energy > consumption is evenly distributed among all vCPUs threads running on > the same physical CPU package. > > To overcome the problem that reading the RAPL MSR requires priviliged > access, a socket communication between QEMU and the qemu-vmsr-helper is > mandatory. You can specified the socket path in the parameter. > > This feature is activated with -accel kvm,rapl=true,path=/path/sock.sock > > Actual limitation: > - Works only on Intel host CPU because AMD CPUs are using different MSR > adresses. > > - Only the Package Power-Plane (MSR_PKG_ENERGY_STATUS) is reported at > the moment. > > Signed-off-by: Anthony Harivel <aharivel@redhat.com> > --- > accel/kvm/kvm-all.c | 27 +++ > docs/specs/index.rst | 1 + > docs/specs/rapl-msr.rst | 155 +++++++++++++ > include/sysemu/kvm.h | 2 + > include/sysemu/kvm_int.h | 30 +++ > target/i386/cpu.h | 8 + > target/i386/kvm/kvm-cpu.c | 7 + > target/i386/kvm/kvm.c | 420 ++++++++++++++++++++++++++++++++++ > target/i386/kvm/meson.build | 1 + > target/i386/kvm/vmsr_energy.c | 381 ++++++++++++++++++++++++++++++ > target/i386/kvm/vmsr_energy.h | 97 ++++++++ > 11 files changed, 1129 insertions(+) > create mode 100644 docs/specs/rapl-msr.rst > create mode 100644 target/i386/kvm/vmsr_energy.c > create mode 100644 target/i386/kvm/vmsr_energy.h > > diff --git a/include/sysemu/kvm_int.h b/include/sysemu/kvm_int.h > index 882e37e12c5b..ea4587b53eb1 100644 > --- a/include/sysemu/kvm_int.h > +++ b/include/sysemu/kvm_int.h > @@ -14,6 +14,8 @@ > #include "qemu/accel.h" > #include "qemu/queue.h" > #include "sysemu/kvm.h" > +#include "hw/boards.h" > +#include "hw/i386/topology.h" > > typedef struct KVMSlot > { > @@ -48,6 +50,33 @@ typedef struct KVMMemoryListener { > > #define KVM_MSI_HASHTAB_SIZE 256 > > +typedef struct KVMHostTopoInfo { > + /* Number of package on the Host */ > + unsigned int maxpkgs; > + /* Number of cpus on the Host */ > + unsigned int maxcpus; > + /* Number of cpus on each different package */ > + unsigned int *pkg_cpu_count; > + /* Each package can have different maxticks */ > + unsigned int *maxticks; > +} KVMHostTopoInfo; > + > +struct KVMMsrEnergy { > + pid_t pid; > + bool enable; > + char *socket_path; > + QemuThread msr_thr; > + unsigned int vcpus; > + unsigned int vsockets; > + X86CPUTopoInfo topo_info; > + KVMHostTopoInfo host_topo; > + const CPUArchIdList *cpu_list; > + uint64_t *msr_value; > + uint64_t msr_unit; > + uint64_t msr_limit; > + uint64_t msr_info; > +}; > + > enum KVMDirtyRingReaperState { > KVM_DIRTY_RING_REAPER_NONE = 0, > /* The reaper is sleeping */ > @@ -114,6 +143,7 @@ struct KVMState > bool kvm_dirty_ring_with_bitmap; > uint64_t kvm_eager_split_size; /* Eager Page Splitting chunk size */ > struct KVMDirtyRingReaper reaper; > + struct KVMMsrEnergy msr_energy; > NotifyVmexitOption notify_vmexit; > uint32_t notify_window; > uint32_t xen_version; > diff --git a/target/i386/kvm/kvm-cpu.c b/target/i386/kvm/kvm-cpu.c > index 9c791b7b0520..246de2bac2f1 100644 > --- a/target/i386/kvm/kvm-cpu.c > +++ b/target/i386/kvm/kvm-cpu.c > @@ -50,6 +50,13 @@ static bool kvm_cpu_realizefn(CPUState *cs, Error **errp) > MSR_IA32_UCODE_REV); > } > } > + if (kvm_is_rapl_feat_enable(cs)) { > + if (IS_INTEL_CPU(env)) { You need to invert this check if (!IS_INTEL_CPU(...)) { > + error_setg(errp, "RAPL feature is enable and CPU is not INTEL CPU"); Tweak the message "The RAPL feature can only be enabled with Intel CPU models" > + return false; > + }; > + }; No semi-colon after } > + > return host_cpu_realizefn(cs, errp); > } > > + > +static int kvm_msr_energy_thread_init(KVMState *s, MachineState *ms) > +{ > + struct KVMMsrEnergy *r = &s->msr_energy; > + int ret = 0; > + > + /* > + * Sanity check > + * 1. Host cpu must be Intel cpu > + * 2. RAPL must be enabled on the Host > + */ > + if (is_host_cpu_intel()) { > + error_report("error: Host CPU is not INTEL cpu"); > + ret = 1; > + goto out; > + } Don't need to include 'error:' in the message, as error_report will already print that. Suggest "The RAPL feature can only be enabled on hosts with Intel CPU models" > + > + if (!is_rapl_enabled()) { > + error_report("error: RAPL driver not enable on Host"); This method is already calling error_report, so reporting it again is redundant. > + ret = 1; > + goto out; > + } > + > + /* Retrieve the virtual topology */ > + vmsr_init_topo_info(&r->topo_info, ms); > + > + /* Retrieve the number of vcpu */ > + r->vcpus = ms->smp.cpus; > + > + /* Retrieve the number of virtual sockets */ > + r->vsockets = ms->smp.sockets; > + > + /* Allocate register memory (MSR_PKG_STATUS) for each vcpu */ > + r->msr_value = g_new0(uint64_t, r->vcpus); > + > + /* Retrieve the CPUArchIDlist */ > + r->x86_cpu_list = x86_possible_cpu_arch_ids(ms); > + > + /* Max number of cpus on the Host */ > + r->host_topo.maxcpus = vmsr_get_maxcpus(); > + if (r->host_topo.maxcpus <= 0) { > + error_report("error: host max cpus <= 0"); Another redundant 'error: ' in strings, and more below > + ret = 1; > + goto out; > + } > + > + /* Max number of packages on the host */ > + r->host_topo.maxpkgs = vmsr_get_max_physical_package(r->host_topo.maxcpus); > + if (r->host_topo.maxpkgs <= 0) { > + error_report("error: host max pkgs <= 0"); > + ret = 1; > + goto out; > + } > + > + /* Allocate memory for each package on the host */ > + r->host_topo.pkg_cpu_count = g_new0(unsigned int, r->host_topo.maxpkgs); > + r->host_topo.maxticks = g_new0(unsigned int, r->host_topo.maxpkgs); > + > + vmsr_count_cpus_per_package(r->host_topo.pkg_cpu_count, > + r->host_topo.maxpkgs); > + for (int i = 0; i < r->host_topo.maxpkgs; i++) { > + if (r->host_topo.pkg_cpu_count[i] <= 0) { > + error_report("error: cpu per packages <= 0 on package_%d", i); Checking '<=' is inappropriate this 'pkg_cpu_count' is an unsigned field. IOW, use ' == 0' not ' <= 0'. > + ret = 1; > + goto out; > + } > + } > + > + /* Get QEMU PID*/ > + r->pid = getpid(); > + > + /* Those MSR values should not change */ > + r->msr_unit = vmsr_read_msr(MSR_RAPL_POWER_UNIT, 0, r->pid, > + s->msr_energy.socket_path); > + r->msr_limit = vmsr_read_msr(MSR_PKG_POWER_LIMIT, 0, r->pid, > + s->msr_energy.socket_path); > + r->msr_info = vmsr_read_msr(MSR_PKG_POWER_INFO, 0, r->pid, > + s->msr_energy.socket_path); > + if (r->msr_unit == 0 || r->msr_limit == 0 || r->msr_info == 0) { > + error_report("error: can't read any virtual msr"); > + ret = 1; > + goto out; > + } > + > + qemu_thread_create(&r->msr_thr, "kvm-msr", > + kvm_msr_energy_thread, > + s, QEMU_THREAD_JOINABLE); > +out: > + return ret; > +} > + > int kvm_arch_get_default_type(MachineState *ms) > { > return 0; > @@ -2715,6 +3092,49 @@ int kvm_arch_init(MachineState *ms, KVMState *s) > strerror(-ret)); > exit(1); > } > + > + if (s->msr_energy.enable == true) { > + r = kvm_filter_msr(s, MSR_RAPL_POWER_UNIT, > + kvm_rdmsr_rapl_power_unit, NULL); > + if (!r) { > + error_report("Could not install MSR_RAPL_POWER_UNIT \ > + handler: %s", > + strerror(-ret)); > + exit(1); > + } > + > + r = kvm_filter_msr(s, MSR_PKG_POWER_LIMIT, > + kvm_rdmsr_pkg_power_limit, NULL); > + if (!r) { > + error_report("Could not install MSR_PKG_POWER_LIMIT \ > + handler: %s", > + strerror(-ret)); > + exit(1); > + } > + > + r = kvm_filter_msr(s, MSR_PKG_POWER_INFO, > + kvm_rdmsr_pkg_power_info, NULL); > + if (!r) { > + error_report("Could not install MSR_PKG_POWER_INFO \ > + handler: %s", > + strerror(-ret)); > + exit(1); > + } > + r = kvm_filter_msr(s, MSR_PKG_ENERGY_STATUS, > + kvm_rdmsr_pkg_energy_status, NULL); > + if (!r) { > + error_report("Could not install MSR_PKG_ENERGY_STATUS \ > + handler: %s", > + strerror(-ret)); > + exit(1); > + } > + r = kvm_msr_energy_thread_init(s, ms); > + if (r) { > + error_report("kvm : error RAPL feature requirement not meet"); > + exit(1); > + } > + > + } > } > > return 0; > diff --git a/target/i386/kvm/meson.build b/target/i386/kvm/meson.build > index 84d9143e6029..16010638df69 100644 > --- a/target/i386/kvm/meson.build > +++ b/target/i386/kvm/meson.build > @@ -3,6 +3,7 @@ i386_kvm_ss = ss.source_set() > i386_kvm_ss.add(files( > 'kvm.c', > 'kvm-cpu.c', > + 'vmsr_energy.c', > )) > > i386_kvm_ss.add(when: 'CONFIG_XEN_EMU', if_true: files('xen-emu.c')) > diff --git a/target/i386/kvm/vmsr_energy.c b/target/i386/kvm/vmsr_energy.c > new file mode 100644 > index 000000000000..20ac50c3f0b6 > --- /dev/null > +++ b/target/i386/kvm/vmsr_energy.c > @@ -0,0 +1,381 @@ > +/* > + * QEMU KVM support -- x86 virtual RAPL msr > + * > + * Copyright 2024 Red Hat, Inc. 2024 > + * > + * Author: > + * Anthony Harivel <aharivel@redhat.com> > + * > + * This work is licensed under the terms of the GNU GPL, version 2 or later. > + * See the COPYING file in the top-level directory. > + * > + */ > + > +#include "qemu/osdep.h" > +#include "qemu/error-report.h" > +#include "vmsr_energy.h" > +#include "io/channel.h" > +#include "io/channel-socket.h" > +#include "hw/boards.h" > +#include "cpu.h" > +#include "host-cpu.h" > + > +static char *compute_default_paths(void) > +{ > + g_autofree char *state = qemu_get_local_state_dir(); > + > + return g_build_filename(state, "run", "qemu-vmsr-helper.sock", NULL); > +} > + > +bool is_host_cpu_intel(void) > +{ > + int family, model, stepping; > + char vendor[CPUID_VENDOR_SZ + 1]; > + > + host_cpu_vendor_fms(vendor, &family, &model, &stepping); > + > + return strcmp(vendor, CPUID_VENDOR_INTEL); > +} > + > +int is_rapl_enabled(void) > +{ > + const char *path = "/sys/class/powercap/intel-rapl/enabled"; > + FILE *file = fopen(path, "r"); > + int value = 0; > + > + if (file != NULL) { > + if (fscanf(file, "%d", &value) != 1) { > + error_report("INTEL RAPL not enabled"); > + } > + fclose(file); > + } else { > + error_report("Error opening %s", path); > + } > + > + return value; > +} > + > +static int vmsr_helper_socket_write(QIOChannel *ioc, > + int fd, In the previous version I said to remove this parameter since you're not using it.... > + const void *buf, int sz, Error **errp) > +{ > + size_t nfds = (fd != -1); > + while (sz > 0) { > + struct iovec iov; > + ssize_t n_written; > + > + iov.iov_base = (void *)buf; > + iov.iov_len = sz; > + n_written = qio_channel_writev_full(QIO_CHANNEL(ioc), &iov, 1, > + nfds ? &fd : NULL, nfds, 0, errp); > + > + if (n_written <= 0) { > + assert(n_written != QIO_CHANNEL_ERR_BLOCK); > + object_unref(OBJECT(ioc)); Calling unref() here is unsafe - the caller will also unref resulting in ref count underflow and crash > + ioc = NULL; > + return n_written < 0 ? -EINVAL : 0; > + } > + > + nfds = 0; > + buf += n_written; > + sz -= n_written; > + } > + > + return 0; > +} Again, tthis whole method can be removed in favour of calling qio_channel_write_all() > + > +static int vmsr_helper_socket_read(QIOChannel *ioc, > + void *buf, int sz, Error **errp) > +{ > + ssize_t r = qio_channel_read_all(ioc, buf, sz, errp); > + > + if (r < 0) { > + object_unref(OBJECT(ioc)); > + ioc = NULL; This is unsafe as the caller unrefs already. > + return -EINVAL; > + } > + > + return 0; > +} This whole method should be removed - just call qio_channel_read_all directly below. > + > +uint64_t vmsr_read_msr(uint32_t reg, unsigned int cpu_id, uint32_t tid, > + const char *path) > +{ > + uint64_t data = 0; > + char *socket_path = NULL; > + char buffer[3]; > + > + if (path == NULL) { > + socket_path = compute_default_paths(); > + } else { > + socket_path = g_strdup(path); > + } Re-computing the socket path every time vmsr_read_msr is called is pretty mad - compute it once at startup and store the value. > + > + SocketAddress saddr = { > + .type = SOCKET_ADDRESS_TYPE_UNIX, > + .u.q_unix.path = socket_path > + }; > + QIOChannelSocket *sioc = qio_channel_socket_new(); > + Error *local_err = NULL; > + > + int r; > + > + qio_channel_set_name(QIO_CHANNEL(sioc), "vmsr-helper"); > + qio_channel_socket_connect_sync(sioc, > + &saddr, > + &local_err); > + g_free(socket_path); > + if (local_err) { > + goto out_close; > + } In the previous posting I suggested that connectiong to the helper again & again for every individual MSR read is a high overhead. Connect once, and then just keep the socket open forever. > + > + /* > + * Send the required arguments: > + * 1. RAPL MSR register to read > + * 2. On which CPU ID > + * 3. From which vCPU (Thread ID) > + */ > + buffer[0] = reg; > + buffer[1] = cpu_id; > + buffer[2] = tid; > + > + r = vmsr_helper_socket_write(QIO_CHANNEL(sioc), > + -1, > + &buffer, sizeof(buffer), > + &local_err); > + if (r < 0) { > + goto out_close; > + } > + > + r = vmsr_helper_socket_read(QIO_CHANNEL(sioc), > + &data, sizeof(data), > + &local_err); > + if (r < 0) { > + data = 0; > + goto out_close; > + } > + > +out_close: > + /* Close socket. */ > + qio_channel_close(QIO_CHANNEL(sioc), NULL); > + object_unref(OBJECT(sioc)); > + return data; > +} > + > +/* Retrieve the max number of physical package */ > +unsigned int vmsr_get_max_physical_package(unsigned int max_cpus) > +{ > + const char *dir = "/sys/devices/system/cpu/"; > + const char *topo_path = "topology/physical_package_id"; > + g_autofree int *uniquePackages = 0; NULL, not 0 for pointers but.... > + unsigned int packageCount = 0; > + FILE *file = NULL; > + > + uniquePackages = g_new0(int, max_cpus); just initialize this at the same time as declaring it. > + > + for (int i = 0; i < max_cpus; i++) { > + g_autofree char *filePath = NULL; > + g_autofree char *cpuid = g_strdup_printf("cpu%d", i); > + > + filePath = g_build_filename(dir, cpuid, topo_path, NULL); > + > + file = fopen(filePath, "r"); > + > + if (file == NULL) { > + error_report("Error opening physical_package_id file"); > + return 0; > + } > + > + char packageId[10]; > + if (fgets(packageId, sizeof(packageId), file) == NULL) { > + packageCount = 0; > + } > + > + fclose(file); > + > + int currentPackageId = atoi(packageId); > + > + bool isUnique = true; > + for (int j = 0; j < packageCount; j++) { > + if (uniquePackages[j] == currentPackageId) { > + isUnique = false; > + break; > + } > + } > + > + if (isUnique) { > + uniquePackages[packageCount] = currentPackageId; > + packageCount++; > + > + if (packageCount >= max_cpus) { > + break; > + } > + } > + } > + > + return (packageCount == 0) ? 1 : packageCount; > +} > + > +/* Retrieve the max number of physical cpu on the host */ > +unsigned int vmsr_get_maxcpus(void) > +{ > + GDir *dir; > + const gchar *entry_name; > + unsigned int cpu_count = 0; > + const char *path = "/sys/devices/system/cpu/"; > + > + dir = g_dir_open(path, 0, NULL); > + if (dir == NULL) { > + error_report("Unable to open cpu directory"); > + return -1; > + } > + > + while ((entry_name = g_dir_read_name(dir)) != NULL) { > + if (g_ascii_strncasecmp(entry_name, "cpu", 3) == 0 && > + isdigit(entry_name[3])) { > + cpu_count++; > + } > + } > + > + g_dir_close(dir); > + > + return cpu_count; > +} > + > +/* Count the number of physical cpu on each packages */ > +unsigned int vmsr_count_cpus_per_package(unsigned int *package_count, > + unsigned int max_pkgs) > +{ > + g_autofree char *file_contents = NULL; > + g_autofree char *path = NULL; > + gsize length; > + > + /* Iterate over cpus and count cpus in each package */ > + for (int cpu_id = 0; ; cpu_id++) { > + path = g_build_filename( > + g_strdup_printf("/sys/devices/system/cpu/cpu%d/" > + "topology/physical_package_id", cpu_id), NULL); > + > + if (!g_file_get_contents(path, &file_contents, &length, NULL)) { > + break; /* No more cpus */ > + } > + > + /* Get the physical package ID for this CPU */ > + int package_id = atoi(file_contents); > + > + /* Check if the package ID is within the known number of packages */ > + if (package_id >= 0 && package_id < max_pkgs) { > + /* If yes, count the cpu for this package*/ > + package_count[package_id]++; > + } > + } > + > + return 0; > +} > + > +/* Get the physical package id from a given cpu id */ > +int vmsr_get_physical_package_id(int cpu_id) > +{ > + g_autofree char *file_contents = NULL; > + g_autofree char *file_path = NULL; > + int package_id = -1; > + gsize length; > + > + file_path = g_strdup_printf("/sys/devices/system/cpu/cpu%d\ > + /topology/physical_package_id", cpu_id); > + > + if (!g_file_get_contents(file_path, &file_contents, &length, NULL)) { > + goto out; > + } > + > + package_id = atoi(file_contents); > + > +out: > + return package_id; > +} > + > +/* Read the scheduled time for a given thread of a give pid */ > +void vmsr_read_thread_stat(pid_t pid, > + unsigned int thread_id, > + unsigned long long *utime, > + unsigned long long *stime, > + unsigned int *cpu_id) > +{ > + g_autofree char *path = NULL; > + > + path = g_build_filename(g_strdup_printf("/proc/%u/task/%d/stat", pid, \ > + thread_id), NULL); > + > + FILE *file = fopen(path, "r"); > + if (file == NULL) { > + pid = -1; > + return; > + } > + > + if (fscanf(file, "%*d (%*[^)]) %*c %*d %*d %*d %*d %*d %*u %*u %*u %*u %*u" > + " %llu %llu %*d %*d %*d %*d %*d %*d %*u %*u %*d %*u %*u" > + " %*u %*u %*u %*u %*u %*u %*u %*u %*u %*d %*u %*u %u", > + utime, stime, cpu_id) != 3) > + { > + pid = -1; > + return; > + } > + > + fclose(file); > + return; > +} > + > +/* Read QEMU stat task folder to retrieve all QEMU threads ID */ > +pid_t *vmsr_get_thread_ids(pid_t pid, unsigned int *num_threads) > +{ > + g_autofree char *path = g_build_filename("/proc", > + g_strdup_printf("%d/task", pid), NULL); > + > + DIR *dir = opendir(path); > + if (dir == NULL) { > + error_report("Error opening /proc/qemu/task"); > + return NULL; > + } > + > + pid_t *thread_ids = NULL; > + unsigned int thread_count = 0; > + > + g_autofree struct dirent *ent = NULL; > + while ((ent = readdir(dir)) != NULL) { > + if (ent->d_name[0] == '.') { > + continue; > + } > + pid_t tid = atoi(ent->d_name); > + if (pid != tid) { > + thread_ids = g_renew(pid_t, thread_ids, (thread_count + 1)); > + thread_ids[thread_count] = tid; > + thread_count++; > + } > + } > + > + closedir(dir); > + > + *num_threads = thread_count; > + return thread_ids; > +} > + > +void vmsr_delta_ticks(thread_stat *thd_stat, int i) > +{ > + thd_stat[i].delta_ticks = (thd_stat[i].utime[1] + thd_stat[i].stime[1]) > + - (thd_stat[i].utime[0] + thd_stat[i].stime[0]); > +} > + > +double vmsr_get_ratio(uint64_t e_delta, > + unsigned long long delta_ticks, > + unsigned int maxticks) > +{ > + return (e_delta / 100.0) * ((100.0 / maxticks) * delta_ticks); > +} > + > +void vmsr_init_topo_info(X86CPUTopoInfo *topo_info, > + const MachineState *ms) > +{ > + topo_info->dies_per_pkg = ms->smp.dies; > + topo_info->cores_per_die = ms->smp.cores; > + topo_info->threads_per_core = ms->smp.threads; > +} With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v4 3/3] Add support for RAPL MSRs in KVM/Qemu 2024-03-21 13:35 ` Daniel P. Berrangé @ 2024-04-05 8:33 ` Anthony Harivel 0 siblings, 0 replies; 10+ messages in thread From: Anthony Harivel @ 2024-04-05 8:33 UTC (permalink / raw) To: Daniel P. Berrangé; +Cc: pbonzini, mtosatti, qemu-devel, vchundur, rjarry Hi Daniel, > > + SocketAddress saddr = { > > + .type = SOCKET_ADDRESS_TYPE_UNIX, > > + .u.q_unix.path = socket_path > > + }; > > + QIOChannelSocket *sioc = qio_channel_socket_new(); > > + Error *local_err = NULL; > > + > > + int r; > > + > > + qio_channel_set_name(QIO_CHANNEL(sioc), "vmsr-helper"); > > + qio_channel_socket_connect_sync(sioc, > > + &saddr, > > + &local_err); > > + g_free(socket_path); > > + if (local_err) { > > + goto out_close; > > + } > > In the previous posting I suggested that connectiong to the > helper again & again for every individual MSR read is a > high overhead. Connect once, and then just keep the socket > open forever. > Indeed, this would be way more efficient. Does that means that I should create the socket during the initialisation of the main loop (kvm_msr_energy_thread_init) and keep track of the context variable and then just give the QIOChannelSocket pointer has a parameter to the vmsr_read_msr() function to send the data? Regards, Anthony ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-04-05 8:34 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-03-18 15:12 [PATCH v4 0/3] Add support for the RAPL MSRs series Anthony Harivel 2024-03-18 15:12 ` [PATCH v4 1/3] qio: add support for SO_PEERCRED for socket channel Anthony Harivel 2024-03-20 17:18 ` Daniel P. Berrangé 2024-03-18 15:12 ` [PATCH v4 2/3] tools: build qemu-vmsr-helper Anthony Harivel 2024-03-21 11:33 ` Daniel P. Berrangé 2024-03-28 10:27 ` Anthony Harivel 2024-03-28 10:46 ` Daniel P. Berrangé 2024-03-18 15:12 ` [PATCH v4 3/3] Add support for RAPL MSRs in KVM/Qemu Anthony Harivel 2024-03-21 13:35 ` Daniel P. Berrangé 2024-04-05 8:33 ` Anthony Harivel
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).