* [PATCH v2 1/5] meson: Introduce 'qatzip' feature to the build system
2024-03-26 22:42 [PATCH v2 0/5] *** Implement using Intel QAT to offload ZLIB Bryan Zhang
@ 2024-03-26 22:42 ` Bryan Zhang
2024-03-26 22:42 ` [PATCH v2 2/5] migration: Add migration parameters for QATzip Bryan Zhang
` (4 subsequent siblings)
5 siblings, 0 replies; 13+ messages in thread
From: Bryan Zhang @ 2024-03-26 22:42 UTC (permalink / raw)
To: qemu-devel
Cc: peterx, farosas, yuan1.liu, berrange, nanhai.zou, hao.xiang,
Bryan Zhang
Add a 'qatzip' feature, which is automatically disabled, and which
depends on the QATzip library if enabled.
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
meson.build | 10 ++++++++++
meson_options.txt | 2 ++
scripts/meson-buildoptions.sh | 3 +++
3 files changed, 15 insertions(+)
diff --git a/meson.build b/meson.build
index 0ef1654e86..aa79ee85bd 100644
--- a/meson.build
+++ b/meson.build
@@ -1198,6 +1198,14 @@ if not get_option('zstd').auto() or have_block
required: get_option('zstd'),
method: 'pkg-config')
endif
+
+qatzip = not_found
+if get_option('qatzip').enabled()
+ qatzip = dependency('qatzip', version: '>=1.1.2',
+ required: get_option('qatzip'),
+ method: 'pkg-config')
+endif
+
virgl = not_found
have_vhost_user_gpu = have_tools and host_os == 'linux' and pixman.found()
@@ -2299,6 +2307,7 @@ config_host_data.set('CONFIG_MALLOC_TRIM', has_malloc_trim)
config_host_data.set('CONFIG_STATX', has_statx)
config_host_data.set('CONFIG_STATX_MNT_ID', has_statx_mnt_id)
config_host_data.set('CONFIG_ZSTD', zstd.found())
+config_host_data.set('CONFIG_QATZIP', qatzip.found())
config_host_data.set('CONFIG_FUSE', fuse.found())
config_host_data.set('CONFIG_FUSE_LSEEK', fuse_lseek.found())
config_host_data.set('CONFIG_SPICE_PROTOCOL', spice_protocol.found())
@@ -4439,6 +4448,7 @@ summary_info += {'snappy support': snappy}
summary_info += {'bzip2 support': libbzip2}
summary_info += {'lzfse support': liblzfse}
summary_info += {'zstd support': zstd}
+summary_info += {'qatzip support': qatzip}
summary_info += {'NUMA host support': numa}
summary_info += {'capstone': capstone}
summary_info += {'libpmem support': libpmem}
diff --git a/meson_options.txt b/meson_options.txt
index 0a99a059ec..106c540ce7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -259,6 +259,8 @@ option('xkbcommon', type : 'feature', value : 'auto',
description: 'xkbcommon support')
option('zstd', type : 'feature', value : 'auto',
description: 'zstd compression support')
+option('qatzip', type: 'feature', value: 'disabled',
+ description: 'QATzip compression support')
option('fuse', type: 'feature', value: 'auto',
description: 'FUSE block device export')
option('fuse_lseek', type : 'feature', value : 'auto',
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index 680fa3f581..1afd373606 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -164,6 +164,7 @@ meson_options_help() {
printf "%s\n" ' plugins TCG plugins via shared library loading'
printf "%s\n" ' png PNG support with libpng'
printf "%s\n" ' pvrdma Enable PVRDMA support'
+ printf "%s\n" ' qatzip QATzip compression support'
printf "%s\n" ' qcow1 qcow1 image format support'
printf "%s\n" ' qed qed image format support'
printf "%s\n" ' qga-vss build QGA VSS support (broken with MinGW)'
@@ -430,6 +431,8 @@ _meson_option_parse() {
--prefix=*) quote_sh "-Dprefix=$2" ;;
--enable-pvrdma) printf "%s" -Dpvrdma=enabled ;;
--disable-pvrdma) printf "%s" -Dpvrdma=disabled ;;
+ --enable-qatzip) printf "%s" -Dqatzip=enabled ;;
+ --disable-qatzip) printf "%s" -Dqatzip=disabled ;;
--enable-qcow1) printf "%s" -Dqcow1=enabled ;;
--disable-qcow1) printf "%s" -Dqcow1=disabled ;;
--enable-qed) printf "%s" -Dqed=enabled ;;
--
2.30.2
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH v2 2/5] migration: Add migration parameters for QATzip
2024-03-26 22:42 [PATCH v2 0/5] *** Implement using Intel QAT to offload ZLIB Bryan Zhang
2024-03-26 22:42 ` [PATCH v2 1/5] meson: Introduce 'qatzip' feature to the build system Bryan Zhang
@ 2024-03-26 22:42 ` Bryan Zhang
2024-03-28 7:23 ` Liu, Yuan1
2024-04-01 15:30 ` Fabiano Rosas
2024-03-26 22:42 ` [PATCH v2 3/5] migration: Introduce unimplemented 'qatzip' compression method Bryan Zhang
` (3 subsequent siblings)
5 siblings, 2 replies; 13+ messages in thread
From: Bryan Zhang @ 2024-03-26 22:42 UTC (permalink / raw)
To: qemu-devel
Cc: peterx, farosas, yuan1.liu, berrange, nanhai.zou, hao.xiang,
Bryan Zhang
Adds support for migration parameters to control QATzip compression
level and to enable/disable software fallback when QAT hardware is
unavailable. This is a preparatory commit for a subsequent commit that
will actually use QATzip compression.
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
Revision: This commit now includes a parameter for controlling software
fallback. Fallback is generally intended to be disabled, but having this
option available enables using software fallback for testing.
This commit also now has some glue code to properly set parameters.
migration/migration-hmp-cmds.c | 8 +++++
migration/options.c | 57 ++++++++++++++++++++++++++++++++++
migration/options.h | 2 ++
qapi/migration.json | 35 +++++++++++++++++++++
4 files changed, 102 insertions(+)
diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
index 99b49df5dd..4bd23ba14d 100644
--- a/migration/migration-hmp-cmds.c
+++ b/migration/migration-hmp-cmds.c
@@ -630,6 +630,14 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
p->has_multifd_zlib_level = true;
visit_type_uint8(v, param, &p->multifd_zlib_level, &err);
break;
+ case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL:
+ p->has_multifd_qatzip_level = true;
+ visit_type_uint8(v, param, &p->multifd_qatzip_level, &err);
+ break;
+ case MIGRATION_PARAMETER_MULTIFD_QATZIP_SW_FALLBACK:
+ p->has_multifd_qatzip_sw_fallback = true;
+ visit_type_bool(v, param, &p->multifd_qatzip_sw_fallback, &err);
+ break;
case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL:
p->has_multifd_zstd_level = true;
visit_type_uint8(v, param, &p->multifd_zstd_level, &err);
diff --git a/migration/options.c b/migration/options.c
index 3e3e0b93b4..1316ea605a 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -62,6 +62,15 @@
#define DEFAULT_MIGRATE_MULTIFD_COMPRESSION MULTIFD_COMPRESSION_NONE
/* 0: means nocompress, 1: best speed, ... 9: best compress ratio */
#define DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL 1
+/*
+ * 1: best speed, ... 9: best compress ratio
+ * There is some nuance here. Refer to QATzip documentation to understand
+ * the mapping of QATzip levels to standard deflate levels.
+ */
+#define DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL 1
+/* QATzip's SW fallback implementation is extremely slow, so avoid fallback */
+#define DEFAULT_MIGRATE_MULTIFD_QATZIP_SW_FALLBACK false
+
/* 0: means nocompress, 1: best speed, ... 20: best compress ratio */
#define DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL 1
@@ -143,6 +152,12 @@ Property migration_properties[] = {
DEFINE_PROP_UINT8("multifd-zlib-level", MigrationState,
parameters.multifd_zlib_level,
DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL),
+ DEFINE_PROP_UINT8("multifd-qatzip-level", MigrationState,
+ parameters.multifd_qatzip_level,
+ DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL),
+ DEFINE_PROP_BOOL("multifd-qatzip-sw-fallback", MigrationState,
+ parameters.multifd_qatzip_sw_fallback,
+ DEFAULT_MIGRATE_MULTIFD_QATZIP_SW_FALLBACK),
DEFINE_PROP_UINT8("multifd-zstd-level", MigrationState,
parameters.multifd_zstd_level,
DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL),
@@ -861,6 +876,20 @@ int migrate_multifd_zlib_level(void)
return s->parameters.multifd_zlib_level;
}
+int migrate_multifd_qatzip_level(void)
+{
+ MigrationState *s = migrate_get_current();
+
+ return s->parameters.multifd_qatzip_level;
+}
+
+bool migrate_multifd_qatzip_sw_fallback(void)
+{
+ MigrationState *s = migrate_get_current();
+
+ return s->parameters.multifd_qatzip_sw_fallback;
+}
+
int migrate_multifd_zstd_level(void)
{
MigrationState *s = migrate_get_current();
@@ -983,6 +1012,11 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
params->multifd_compression = s->parameters.multifd_compression;
params->has_multifd_zlib_level = true;
params->multifd_zlib_level = s->parameters.multifd_zlib_level;
+ params->has_multifd_qatzip_level = true;
+ params->multifd_qatzip_level = s->parameters.multifd_qatzip_level;
+ params->has_multifd_qatzip_sw_fallback = true;
+ params->multifd_qatzip_sw_fallback =
+ s->parameters.multifd_qatzip_sw_fallback;
params->has_multifd_zstd_level = true;
params->multifd_zstd_level = s->parameters.multifd_zstd_level;
params->has_xbzrle_cache_size = true;
@@ -1038,6 +1072,8 @@ void migrate_params_init(MigrationParameters *params)
params->has_multifd_channels = true;
params->has_multifd_compression = true;
params->has_multifd_zlib_level = true;
+ params->has_multifd_qatzip_level = true;
+ params->has_multifd_qatzip_sw_fallback = true;
params->has_multifd_zstd_level = true;
params->has_xbzrle_cache_size = true;
params->has_max_postcopy_bandwidth = true;
@@ -1147,6 +1183,14 @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
return false;
}
+ if (params->has_multifd_qatzip_level &&
+ ((params->multifd_qatzip_level > 9) ||
+ (params->multifd_qatzip_level < 1))) {
+ error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "multifd_qatzip_level",
+ "a value between 1 and 9");
+ return false;
+ }
+
if (params->has_multifd_zstd_level &&
(params->multifd_zstd_level > 20)) {
error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "multifd_zstd_level",
@@ -1312,6 +1356,12 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
if (params->has_multifd_compression) {
dest->multifd_compression = params->multifd_compression;
}
+ if (params->has_multifd_qatzip_level) {
+ dest->multifd_qatzip_level = params->multifd_qatzip_level;
+ }
+ if (params->has_multifd_qatzip_sw_fallback) {
+ dest->multifd_qatzip_sw_fallback = params->multifd_qatzip_sw_fallback;
+ }
if (params->has_xbzrle_cache_size) {
dest->xbzrle_cache_size = params->xbzrle_cache_size;
}
@@ -1447,6 +1497,13 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
if (params->has_multifd_compression) {
s->parameters.multifd_compression = params->multifd_compression;
}
+ if (params->has_multifd_qatzip_level) {
+ s->parameters.multifd_qatzip_level = params->multifd_qatzip_level;
+ }
+ if (params->has_multifd_qatzip_sw_fallback) {
+ s->parameters.multifd_qatzip_sw_fallback =
+ params->multifd_qatzip_sw_fallback;
+ }
if (params->has_xbzrle_cache_size) {
s->parameters.xbzrle_cache_size = params->xbzrle_cache_size;
xbzrle_cache_resize(params->xbzrle_cache_size, errp);
diff --git a/migration/options.h b/migration/options.h
index 246c160aee..94aee24d97 100644
--- a/migration/options.h
+++ b/migration/options.h
@@ -87,6 +87,8 @@ MigMode migrate_mode(void);
int migrate_multifd_channels(void);
MultiFDCompression migrate_multifd_compression(void);
int migrate_multifd_zlib_level(void);
+int migrate_multifd_qatzip_level(void);
+bool migrate_multifd_qatzip_sw_fallback(void);
int migrate_multifd_zstd_level(void);
uint8_t migrate_throttle_trigger_threshold(void);
const char *migrate_tls_authz(void);
diff --git a/qapi/migration.json b/qapi/migration.json
index 0b33a71ab4..66ea6d32fc 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -853,6 +853,16 @@
# speed, and 9 means best compression ratio which will consume
# more CPU. Defaults to 1. (Since 5.0)
#
+# @multifd-qatzip-level: Set the compression level to be used in live
+# migration. The level is an integer between 1 and 9, where 1 means
+# the best compression speed, and 9 means the best compression
+# ratio which will consume more CPU. Defaults to 1.
+#
+# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
+# is unavailable. Defaults to false. Software fallback performance
+# is very poor compared to regular zlib, so be cautious about
+# enabling this option.
+#
# @multifd-zstd-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0 and 20,
# where 0 means no compression, 1 means the best compression
@@ -915,6 +925,7 @@
'xbzrle-cache-size', 'max-postcopy-bandwidth',
'max-cpu-throttle', 'multifd-compression',
'multifd-zlib-level', 'multifd-zstd-level',
+ 'multifd-qatzip-level', 'multifd-qatzip-sw-fallback',
'block-bitmap-mapping',
{ 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] },
'vcpu-dirty-limit',
@@ -1045,6 +1056,16 @@
# speed, and 9 means best compression ratio which will consume
# more CPU. Defaults to 1. (Since 5.0)
#
+# @multifd-qatzip-level: Set the compression level to be used in live
+# migration. The level is an integer between 1 and 9, where 1 means
+# the best compression speed, and 9 means the best compression
+# ratio which will consume more CPU. Defaults to 1.
+#
+# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
+# is unavailable. Defaults to false. Software fallback performance
+# is very poor compared to regular zlib, so be cautious about
+# enabling this option.
+#
# @multifd-zstd-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0 and 20,
# where 0 means no compression, 1 means the best compression
@@ -1125,6 +1146,8 @@
'*max-cpu-throttle': 'uint8',
'*multifd-compression': 'MultiFDCompression',
'*multifd-zlib-level': 'uint8',
+ '*multifd-qatzip-level': 'uint8',
+ '*multifd-qatzip-sw-fallback': 'bool',
'*multifd-zstd-level': 'uint8',
'*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
'*x-vcpu-dirty-limit-period': { 'type': 'uint64',
@@ -1273,6 +1296,16 @@
# speed, and 9 means best compression ratio which will consume
# more CPU. Defaults to 1. (Since 5.0)
#
+# @multifd-qatzip-level: Set the compression level to be used in live
+# migration. The level is an integer between 1 and 9, where 1 means
+# the best compression speed, and 9 means the best compression
+# ratio which will consume more CPU. Defaults to 1.
+#
+# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
+# is unavailable. Defaults to false. Software fallback performance
+# is very poor compared to regular zlib, so be cautious about
+# enabling this option.
+#
# @multifd-zstd-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0 and 20,
# where 0 means no compression, 1 means the best compression
@@ -1350,6 +1383,8 @@
'*max-cpu-throttle': 'uint8',
'*multifd-compression': 'MultiFDCompression',
'*multifd-zlib-level': 'uint8',
+ '*multifd-qatzip-level': 'uint8',
+ '*multifd-qatzip-sw-fallback': 'bool',
'*multifd-zstd-level': 'uint8',
'*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
'*x-vcpu-dirty-limit-period': { 'type': 'uint64',
--
2.30.2
^ permalink raw reply related [flat|nested] 13+ messages in thread* RE: [PATCH v2 2/5] migration: Add migration parameters for QATzip
2024-03-26 22:42 ` [PATCH v2 2/5] migration: Add migration parameters for QATzip Bryan Zhang
@ 2024-03-28 7:23 ` Liu, Yuan1
2024-06-27 0:16 ` Yichen Wang
2024-04-01 15:30 ` Fabiano Rosas
1 sibling, 1 reply; 13+ messages in thread
From: Liu, Yuan1 @ 2024-03-28 7:23 UTC (permalink / raw)
To: Bryan Zhang, qemu-devel@nongnu.org
Cc: peterx@redhat.com, farosas@suse.de, berrange@redhat.com,
Zou, Nanhai, hao.xiang@linux.dev
> -----Original Message-----
> From: Bryan Zhang <bryan.zhang@bytedance.com>
> Sent: Wednesday, March 27, 2024 6:42 AM
> To: qemu-devel@nongnu.org
> Cc: peterx@redhat.com; farosas@suse.de; Liu, Yuan1 <yuan1.liu@intel.com>;
> berrange@redhat.com; Zou, Nanhai <nanhai.zou@intel.com>;
> hao.xiang@linux.dev; Bryan Zhang <bryan.zhang@bytedance.com>
> Subject: [PATCH v2 2/5] migration: Add migration parameters for QATzip
>
> Adds support for migration parameters to control QATzip compression
> level and to enable/disable software fallback when QAT hardware is
> unavailable. This is a preparatory commit for a subsequent commit that
> will actually use QATzip compression.
>
> Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> Revision: This commit now includes a parameter for controlling software
> fallback. Fallback is generally intended to be disabled, but having this
> option available enables using software fallback for testing.
>
> This commit also now has some glue code to properly set parameters.
>
> migration/migration-hmp-cmds.c | 8 +++++
> migration/options.c | 57 ++++++++++++++++++++++++++++++++++
> migration/options.h | 2 ++
> qapi/migration.json | 35 +++++++++++++++++++++
> 4 files changed, 102 insertions(+)
>
> diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-
> cmds.c
> index 99b49df5dd..4bd23ba14d 100644
> --- a/migration/migration-hmp-cmds.c
> +++ b/migration/migration-hmp-cmds.c
> @@ -630,6 +630,14 @@ void hmp_migrate_set_parameter(Monitor *mon, const
> QDict *qdict)
> p->has_multifd_zlib_level = true;
> visit_type_uint8(v, param, &p->multifd_zlib_level, &err);
> break;
> + case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL:
> + p->has_multifd_qatzip_level = true;
> + visit_type_uint8(v, param, &p->multifd_qatzip_level, &err);
> + break;
> + case MIGRATION_PARAMETER_MULTIFD_QATZIP_SW_FALLBACK:
> + p->has_multifd_qatzip_sw_fallback = true;
> + visit_type_bool(v, param, &p->multifd_qatzip_sw_fallback, &err);
> + break;
> case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL:
> p->has_multifd_zstd_level = true;
> visit_type_uint8(v, param, &p->multifd_zstd_level, &err);
> diff --git a/migration/options.c b/migration/options.c
> index 3e3e0b93b4..1316ea605a 100644
> --- a/migration/options.c
> +++ b/migration/options.c
> @@ -62,6 +62,15 @@
> #define DEFAULT_MIGRATE_MULTIFD_COMPRESSION MULTIFD_COMPRESSION_NONE
> /* 0: means nocompress, 1: best speed, ... 9: best compress ratio */
> #define DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL 1
> +/*
> + * 1: best speed, ... 9: best compress ratio
> + * There is some nuance here. Refer to QATzip documentation to understand
> + * the mapping of QATzip levels to standard deflate levels.
> + */
> +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL 1
> +/* QATzip's SW fallback implementation is extremely slow, so avoid
> fallback */
> +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_SW_FALLBACK false
> +
> /* 0: means nocompress, 1: best speed, ... 20: best compress ratio */
> #define DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL 1
Hi Bryan
The default compression level may be set higher, such as 6. I checked QAT throughput
If the data size is less than or equal to 64K, level 1 has much better throughput
performance than level 6 and level 9. But if the data size is greater than 128K, little
change in throughput, and the default MULTIFD_PACKET_SIZE is 512K, you can have a try
to use a high compression level, to get better compression performance without affecting
throughput.
In addition, if you change MULTIFD_PACKET_SIZE to 64K, you may have better throughput
with more multifd threads
> @@ -143,6 +152,12 @@ Property migration_properties[] = {
> DEFINE_PROP_UINT8("multifd-zlib-level", MigrationState,
> parameters.multifd_zlib_level,
> DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL),
> + DEFINE_PROP_UINT8("multifd-qatzip-level", MigrationState,
> + parameters.multifd_qatzip_level,
> + DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL),
> + DEFINE_PROP_BOOL("multifd-qatzip-sw-fallback", MigrationState,
> + parameters.multifd_qatzip_sw_fallback,
> + DEFAULT_MIGRATE_MULTIFD_QATZIP_SW_FALLBACK),
> DEFINE_PROP_UINT8("multifd-zstd-level", MigrationState,
> parameters.multifd_zstd_level,
> DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL),
> @@ -861,6 +876,20 @@ int migrate_multifd_zlib_level(void)
> return s->parameters.multifd_zlib_level;
> }
>
> +int migrate_multifd_qatzip_level(void)
> +{
> + MigrationState *s = migrate_get_current();
> +
> + return s->parameters.multifd_qatzip_level;
> +}
> +
> +bool migrate_multifd_qatzip_sw_fallback(void)
> +{
> + MigrationState *s = migrate_get_current();
> +
> + return s->parameters.multifd_qatzip_sw_fallback;
> +}
> +
> int migrate_multifd_zstd_level(void)
> {
> MigrationState *s = migrate_get_current();
> @@ -983,6 +1012,11 @@ MigrationParameters
> *qmp_query_migrate_parameters(Error **errp)
> params->multifd_compression = s->parameters.multifd_compression;
> params->has_multifd_zlib_level = true;
> params->multifd_zlib_level = s->parameters.multifd_zlib_level;
> + params->has_multifd_qatzip_level = true;
> + params->multifd_qatzip_level = s->parameters.multifd_qatzip_level;
> + params->has_multifd_qatzip_sw_fallback = true;
> + params->multifd_qatzip_sw_fallback =
> + s->parameters.multifd_qatzip_sw_fallback;
> params->has_multifd_zstd_level = true;
> params->multifd_zstd_level = s->parameters.multifd_zstd_level;
> params->has_xbzrle_cache_size = true;
> @@ -1038,6 +1072,8 @@ void migrate_params_init(MigrationParameters
> *params)
> params->has_multifd_channels = true;
> params->has_multifd_compression = true;
> params->has_multifd_zlib_level = true;
> + params->has_multifd_qatzip_level = true;
> + params->has_multifd_qatzip_sw_fallback = true;
> params->has_multifd_zstd_level = true;
> params->has_xbzrle_cache_size = true;
> params->has_max_postcopy_bandwidth = true;
> @@ -1147,6 +1183,14 @@ bool migrate_params_check(MigrationParameters
> *params, Error **errp)
> return false;
> }
>
> + if (params->has_multifd_qatzip_level &&
> + ((params->multifd_qatzip_level > 9) ||
> + (params->multifd_qatzip_level < 1))) {
> + error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
> "multifd_qatzip_level",
> + "a value between 1 and 9");
> + return false;
> + }
> +
> if (params->has_multifd_zstd_level &&
> (params->multifd_zstd_level > 20)) {
> error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
> "multifd_zstd_level",
> @@ -1312,6 +1356,12 @@ static void
> migrate_params_test_apply(MigrateSetParameters *params,
> if (params->has_multifd_compression) {
> dest->multifd_compression = params->multifd_compression;
> }
> + if (params->has_multifd_qatzip_level) {
> + dest->multifd_qatzip_level = params->multifd_qatzip_level;
> + }
> + if (params->has_multifd_qatzip_sw_fallback) {
> + dest->multifd_qatzip_sw_fallback = params-
> >multifd_qatzip_sw_fallback;
> + }
> if (params->has_xbzrle_cache_size) {
> dest->xbzrle_cache_size = params->xbzrle_cache_size;
> }
> @@ -1447,6 +1497,13 @@ static void
> migrate_params_apply(MigrateSetParameters *params, Error **errp)
> if (params->has_multifd_compression) {
> s->parameters.multifd_compression = params->multifd_compression;
> }
> + if (params->has_multifd_qatzip_level) {
> + s->parameters.multifd_qatzip_level = params-
> >multifd_qatzip_level;
> + }
> + if (params->has_multifd_qatzip_sw_fallback) {
> + s->parameters.multifd_qatzip_sw_fallback =
> + params->multifd_qatzip_sw_fallback;
> + }
> if (params->has_xbzrle_cache_size) {
> s->parameters.xbzrle_cache_size = params->xbzrle_cache_size;
> xbzrle_cache_resize(params->xbzrle_cache_size, errp);
> diff --git a/migration/options.h b/migration/options.h
> index 246c160aee..94aee24d97 100644
> --- a/migration/options.h
> +++ b/migration/options.h
> @@ -87,6 +87,8 @@ MigMode migrate_mode(void);
> int migrate_multifd_channels(void);
> MultiFDCompression migrate_multifd_compression(void);
> int migrate_multifd_zlib_level(void);
> +int migrate_multifd_qatzip_level(void);
> +bool migrate_multifd_qatzip_sw_fallback(void);
> int migrate_multifd_zstd_level(void);
> uint8_t migrate_throttle_trigger_threshold(void);
> const char *migrate_tls_authz(void);
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 0b33a71ab4..66ea6d32fc 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -853,6 +853,16 @@
> # speed, and 9 means best compression ratio which will consume
> # more CPU. Defaults to 1. (Since 5.0)
> #
> +# @multifd-qatzip-level: Set the compression level to be used in live
> +# migration. The level is an integer between 1 and 9, where 1 means
> +# the best compression speed, and 9 means the best compression
> +# ratio which will consume more CPU. Defaults to 1.
> +#
> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
> +# is unavailable. Defaults to false. Software fallback performance
> +# is very poor compared to regular zlib, so be cautious about
> +# enabling this option.
> +#
> # @multifd-zstd-level: Set the compression level to be used in live
> # migration, the compression level is an integer between 0 and 20,
> # where 0 means no compression, 1 means the best compression
> @@ -915,6 +925,7 @@
> 'xbzrle-cache-size', 'max-postcopy-bandwidth',
> 'max-cpu-throttle', 'multifd-compression',
> 'multifd-zlib-level', 'multifd-zstd-level',
> + 'multifd-qatzip-level', 'multifd-qatzip-sw-fallback',
> 'block-bitmap-mapping',
> { 'name': 'x-vcpu-dirty-limit-period', 'features':
> ['unstable'] },
> 'vcpu-dirty-limit',
> @@ -1045,6 +1056,16 @@
> # speed, and 9 means best compression ratio which will consume
> # more CPU. Defaults to 1. (Since 5.0)
> #
> +# @multifd-qatzip-level: Set the compression level to be used in live
> +# migration. The level is an integer between 1 and 9, where 1 means
> +# the best compression speed, and 9 means the best compression
> +# ratio which will consume more CPU. Defaults to 1.
> +#
> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
> +# is unavailable. Defaults to false. Software fallback performance
> +# is very poor compared to regular zlib, so be cautious about
> +# enabling this option.
> +#
> # @multifd-zstd-level: Set the compression level to be used in live
> # migration, the compression level is an integer between 0 and 20,
> # where 0 means no compression, 1 means the best compression
> @@ -1125,6 +1146,8 @@
> '*max-cpu-throttle': 'uint8',
> '*multifd-compression': 'MultiFDCompression',
> '*multifd-zlib-level': 'uint8',
> + '*multifd-qatzip-level': 'uint8',
> + '*multifd-qatzip-sw-fallback': 'bool',
> '*multifd-zstd-level': 'uint8',
> '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
> '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
> @@ -1273,6 +1296,16 @@
> # speed, and 9 means best compression ratio which will consume
> # more CPU. Defaults to 1. (Since 5.0)
> #
> +# @multifd-qatzip-level: Set the compression level to be used in live
> +# migration. The level is an integer between 1 and 9, where 1 means
> +# the best compression speed, and 9 means the best compression
> +# ratio which will consume more CPU. Defaults to 1.
> +#
> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
> +# is unavailable. Defaults to false. Software fallback performance
> +# is very poor compared to regular zlib, so be cautious about
> +# enabling this option.
> +#
> # @multifd-zstd-level: Set the compression level to be used in live
> # migration, the compression level is an integer between 0 and 20,
> # where 0 means no compression, 1 means the best compression
> @@ -1350,6 +1383,8 @@
> '*max-cpu-throttle': 'uint8',
> '*multifd-compression': 'MultiFDCompression',
> '*multifd-zlib-level': 'uint8',
> + '*multifd-qatzip-level': 'uint8',
> + '*multifd-qatzip-sw-fallback': 'bool',
> '*multifd-zstd-level': 'uint8',
> '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
> '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
> --
> 2.30.2
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: [PATCH v2 2/5] migration: Add migration parameters for QATzip
2024-03-28 7:23 ` Liu, Yuan1
@ 2024-06-27 0:16 ` Yichen Wang
2024-06-27 7:25 ` Liu, Yuan1
0 siblings, 1 reply; 13+ messages in thread
From: Yichen Wang @ 2024-06-27 0:16 UTC (permalink / raw)
To: Liu, Yuan1
Cc: Bryan Zhang, qemu-devel@nongnu.org, peterx@redhat.com,
farosas@suse.de, berrange@redhat.com, Zou, Nanhai,
hao.xiang@linux.dev
> On Mar 28, 2024, at 12:23 AM, Liu, Yuan1 <yuan1.liu@intel.com> wrote:
>
>> -----Original Message-----
>> From: Bryan Zhang <bryan.zhang@bytedance.com>
>> Sent: Wednesday, March 27, 2024 6:42 AM
>> To: qemu-devel@nongnu.org
>> Cc: peterx@redhat.com; farosas@suse.de; Liu, Yuan1 <yuan1.liu@intel.com>;
>> berrange@redhat.com; Zou, Nanhai <nanhai.zou@intel.com>;
>> hao.xiang@linux.dev; Bryan Zhang <bryan.zhang@bytedance.com>
>> Subject: [PATCH v2 2/5] migration: Add migration parameters for QATzip
>>
>> Adds support for migration parameters to control QATzip compression
>> level and to enable/disable software fallback when QAT hardware is
>> unavailable. This is a preparatory commit for a subsequent commit that
>> will actually use QATzip compression.
>>
>> Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
>> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
>> ---
>> Revision: This commit now includes a parameter for controlling software
>> fallback. Fallback is generally intended to be disabled, but having this
>> option available enables using software fallback for testing.
>>
>> This commit also now has some glue code to properly set parameters.
>>
>> migration/migration-hmp-cmds.c | 8 +++++
>> migration/options.c | 57 ++++++++++++++++++++++++++++++++++
>> migration/options.h | 2 ++
>> qapi/migration.json | 35 +++++++++++++++++++++
>> 4 files changed, 102 insertions(+)
>>
>> diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-
>> cmds.c
>> index 99b49df5dd..4bd23ba14d 100644
>> --- a/migration/migration-hmp-cmds.c
>> +++ b/migration/migration-hmp-cmds.c
>> @@ -630,6 +630,14 @@ void hmp_migrate_set_parameter(Monitor *mon, const
>> QDict *qdict)
>> p->has_multifd_zlib_level = true;
>> visit_type_uint8(v, param, &p->multifd_zlib_level, &err);
>> break;
>> + case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL:
>> + p->has_multifd_qatzip_level = true;
>> + visit_type_uint8(v, param, &p->multifd_qatzip_level, &err);
>> + break;
>> + case MIGRATION_PARAMETER_MULTIFD_QATZIP_SW_FALLBACK:
>> + p->has_multifd_qatzip_sw_fallback = true;
>> + visit_type_bool(v, param, &p->multifd_qatzip_sw_fallback, &err);
>> + break;
>> case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL:
>> p->has_multifd_zstd_level = true;
>> visit_type_uint8(v, param, &p->multifd_zstd_level, &err);
>> diff --git a/migration/options.c b/migration/options.c
>> index 3e3e0b93b4..1316ea605a 100644
>> --- a/migration/options.c
>> +++ b/migration/options.c
>> @@ -62,6 +62,15 @@
>> #define DEFAULT_MIGRATE_MULTIFD_COMPRESSION MULTIFD_COMPRESSION_NONE
>> /* 0: means nocompress, 1: best speed, ... 9: best compress ratio */
>> #define DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL 1
>> +/*
>> + * 1: best speed, ... 9: best compress ratio
>> + * There is some nuance here. Refer to QATzip documentation to understand
>> + * the mapping of QATzip levels to standard deflate levels.
>> + */
>> +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL 1
>> +/* QATzip's SW fallback implementation is extremely slow, so avoid
>> fallback */
>> +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_SW_FALLBACK false
>> +
>> /* 0: means nocompress, 1: best speed, ... 20: best compress ratio */
>> #define DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL 1
>
> Hi Bryan
>
> The default compression level may be set higher, such as 6. I checked QAT throughput
> If the data size is less than or equal to 64K, level 1 has much better throughput
> performance than level 6 and level 9. But if the data size is greater than 128K, little
> change in throughput, and the default MULTIFD_PACKET_SIZE is 512K, you can have a try
> to use a high compression level, to get better compression performance without affecting
> throughput.
>
> In addition, if you change MULTIFD_PACKET_SIZE to 64K, you may have better throughput
> with more multifd threads
Hi Yuan,
Thanks for your comment. I did a quick experiments on our CPU. With chunk size at 512K, for silesia dataset, the throughput degrades at level 6:
# for f in `seq 1 1 9`; do echo "Level $f:"; qzip -C 524288 -L $f -k silesia.8G | grep ":"; done
Level 1:
Time taken: 2230.772 ms
Throughput: 30402.186 Mbit/s
Space Savings: 65.712 %
Compression ratio: 2.916 : 1
Level 2:
Time taken: 2245.205 ms
Throughput: 30206.750 Mbit/s
Space Savings: 65.712 %
Compression ratio: 2.916 : 1
Level 3:
Time taken: 2217.789 ms
Throughput: 30580.161 Mbit/s
Space Savings: 65.712 %
Compression ratio: 2.916 : 1
Level 4:
Time taken: 2251.014 ms
Throughput: 30128.798 Mbit/s
Space Savings: 65.712 %
Compression ratio: 2.916 : 1
Level 5:
Time taken: 2200.991 ms
Throughput: 30813.550 Mbit/s
Space Savings: 65.712 %
Compression ratio: 2.916 : 1
Level 6:
Time taken: 2508.218 ms
Throughput: 27039.255 Mbit/s
Space Savings: 67.396 %
Compression ratio: 3.067 : 1
Level 7:
Time taken: 2510.847 ms
Throughput: 27010.943 Mbit/s
Space Savings: 67.396 %
Compression ratio: 3.067 : 1
Level 8:
Time taken: 2521.428 ms
Throughput: 26897.594 Mbit/s
Space Savings: 67.396 %
Compression ratio: 3.067 : 1
Level 9:
Time taken: 3071.055 ms
Throughput: 22083.729 Mbit/s
Space Savings: 67.664 %
Compression ratio: 3.092 : 1
For random text data, throughput actually improves at level 6.
# for f in `seq 1 1 9`; do echo "Level $f:"; qzip -C 524288 -L $f -k text.txt | grep ":"; done
Level 1:
Time taken: 1788.683 ms
Throughput: 8945.129 Mbit/s
Space Savings: 24.959 %
Compression ratio: 1.333 : 1
Level 2:
Time taken: 1786.135 ms
Throughput: 8957.890 Mbit/s
Space Savings: 24.959 %
Compression ratio: 1.333 : 1
Level 3:
Time taken: 1785.564 ms
Throughput: 8960.754 Mbit/s
Space Savings: 24.959 %
Compression ratio: 1.333 : 1
Level 4:
Time taken: 1787.351 ms
Throughput: 8951.795 Mbit/s
Space Savings: 24.959 %
Compression ratio: 1.333 : 1
Level 5:
Time taken: 1785.171 ms
Throughput: 8962.727 Mbit/s
Space Savings: 24.959 %
Compression ratio: 1.333 : 1
Level 6:
Time taken: 1752.000 ms
Throughput: 9132.420 Mbit/s
Space Savings: 24.873 %
Compression ratio: 1.331 : 1
Level 7:
Time taken: 1752.297 ms
Throughput: 9130.872 Mbit/s
Space Savings: 24.873 %
Compression ratio: 1.331 : 1
Level 8:
Time taken: 1752.538 ms
Throughput: 9129.617 Mbit/s
Space Savings: 24.873 %
Compression ratio: 1.331 : 1
Level 9:
Time taken: 1762.593 ms
Throughput: 9077.535 Mbit/s
Space Savings: 24.870 %
Compression ratio: 1.331 : 1
For random binary data, throughput remains the same from 0-9.
Given our live migration is mostly memory pages, and I am not very sure what scenario would better fit to above tests. I am OK with either default level (1 or 6). I can change to 6 in my next patchiest if you believe that is better.
Thanks very much!
Regards,
Yichen
>
>> @@ -143,6 +152,12 @@ Property migration_properties[] = {
>> DEFINE_PROP_UINT8("multifd-zlib-level", MigrationState,
>> parameters.multifd_zlib_level,
>> DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL),
>> + DEFINE_PROP_UINT8("multifd-qatzip-level", MigrationState,
>> + parameters.multifd_qatzip_level,
>> + DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL),
>> + DEFINE_PROP_BOOL("multifd-qatzip-sw-fallback", MigrationState,
>> + parameters.multifd_qatzip_sw_fallback,
>> + DEFAULT_MIGRATE_MULTIFD_QATZIP_SW_FALLBACK),
>> DEFINE_PROP_UINT8("multifd-zstd-level", MigrationState,
>> parameters.multifd_zstd_level,
>> DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL),
>> @@ -861,6 +876,20 @@ int migrate_multifd_zlib_level(void)
>> return s->parameters.multifd_zlib_level;
>> }
>>
>> +int migrate_multifd_qatzip_level(void)
>> +{
>> + MigrationState *s = migrate_get_current();
>> +
>> + return s->parameters.multifd_qatzip_level;
>> +}
>> +
>> +bool migrate_multifd_qatzip_sw_fallback(void)
>> +{
>> + MigrationState *s = migrate_get_current();
>> +
>> + return s->parameters.multifd_qatzip_sw_fallback;
>> +}
>> +
>> int migrate_multifd_zstd_level(void)
>> {
>> MigrationState *s = migrate_get_current();
>> @@ -983,6 +1012,11 @@ MigrationParameters
>> *qmp_query_migrate_parameters(Error **errp)
>> params->multifd_compression = s->parameters.multifd_compression;
>> params->has_multifd_zlib_level = true;
>> params->multifd_zlib_level = s->parameters.multifd_zlib_level;
>> + params->has_multifd_qatzip_level = true;
>> + params->multifd_qatzip_level = s->parameters.multifd_qatzip_level;
>> + params->has_multifd_qatzip_sw_fallback = true;
>> + params->multifd_qatzip_sw_fallback =
>> + s->parameters.multifd_qatzip_sw_fallback;
>> params->has_multifd_zstd_level = true;
>> params->multifd_zstd_level = s->parameters.multifd_zstd_level;
>> params->has_xbzrle_cache_size = true;
>> @@ -1038,6 +1072,8 @@ void migrate_params_init(MigrationParameters
>> *params)
>> params->has_multifd_channels = true;
>> params->has_multifd_compression = true;
>> params->has_multifd_zlib_level = true;
>> + params->has_multifd_qatzip_level = true;
>> + params->has_multifd_qatzip_sw_fallback = true;
>> params->has_multifd_zstd_level = true;
>> params->has_xbzrle_cache_size = true;
>> params->has_max_postcopy_bandwidth = true;
>> @@ -1147,6 +1183,14 @@ bool migrate_params_check(MigrationParameters
>> *params, Error **errp)
>> return false;
>> }
>>
>> + if (params->has_multifd_qatzip_level &&
>> + ((params->multifd_qatzip_level > 9) ||
>> + (params->multifd_qatzip_level < 1))) {
>> + error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
>> "multifd_qatzip_level",
>> + "a value between 1 and 9");
>> + return false;
>> + }
>> +
>> if (params->has_multifd_zstd_level &&
>> (params->multifd_zstd_level > 20)) {
>> error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
>> "multifd_zstd_level",
>> @@ -1312,6 +1356,12 @@ static void
>> migrate_params_test_apply(MigrateSetParameters *params,
>> if (params->has_multifd_compression) {
>> dest->multifd_compression = params->multifd_compression;
>> }
>> + if (params->has_multifd_qatzip_level) {
>> + dest->multifd_qatzip_level = params->multifd_qatzip_level;
>> + }
>> + if (params->has_multifd_qatzip_sw_fallback) {
>> + dest->multifd_qatzip_sw_fallback = params-
>>> multifd_qatzip_sw_fallback;
>> + }
>> if (params->has_xbzrle_cache_size) {
>> dest->xbzrle_cache_size = params->xbzrle_cache_size;
>> }
>> @@ -1447,6 +1497,13 @@ static void
>> migrate_params_apply(MigrateSetParameters *params, Error **errp)
>> if (params->has_multifd_compression) {
>> s->parameters.multifd_compression = params->multifd_compression;
>> }
>> + if (params->has_multifd_qatzip_level) {
>> + s->parameters.multifd_qatzip_level = params-
>>> multifd_qatzip_level;
>> + }
>> + if (params->has_multifd_qatzip_sw_fallback) {
>> + s->parameters.multifd_qatzip_sw_fallback =
>> + params->multifd_qatzip_sw_fallback;
>> + }
>> if (params->has_xbzrle_cache_size) {
>> s->parameters.xbzrle_cache_size = params->xbzrle_cache_size;
>> xbzrle_cache_resize(params->xbzrle_cache_size, errp);
>> diff --git a/migration/options.h b/migration/options.h
>> index 246c160aee..94aee24d97 100644
>> --- a/migration/options.h
>> +++ b/migration/options.h
>> @@ -87,6 +87,8 @@ MigMode migrate_mode(void);
>> int migrate_multifd_channels(void);
>> MultiFDCompression migrate_multifd_compression(void);
>> int migrate_multifd_zlib_level(void);
>> +int migrate_multifd_qatzip_level(void);
>> +bool migrate_multifd_qatzip_sw_fallback(void);
>> int migrate_multifd_zstd_level(void);
>> uint8_t migrate_throttle_trigger_threshold(void);
>> const char *migrate_tls_authz(void);
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index 0b33a71ab4..66ea6d32fc 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -853,6 +853,16 @@
>> # speed, and 9 means best compression ratio which will consume
>> # more CPU. Defaults to 1. (Since 5.0)
>> #
>> +# @multifd-qatzip-level: Set the compression level to be used in live
>> +# migration. The level is an integer between 1 and 9, where 1 means
>> +# the best compression speed, and 9 means the best compression
>> +# ratio which will consume more CPU. Defaults to 1.
>> +#
>> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
>> +# is unavailable. Defaults to false. Software fallback performance
>> +# is very poor compared to regular zlib, so be cautious about
>> +# enabling this option.
>> +#
>> # @multifd-zstd-level: Set the compression level to be used in live
>> # migration, the compression level is an integer between 0 and 20,
>> # where 0 means no compression, 1 means the best compression
>> @@ -915,6 +925,7 @@
>> 'xbzrle-cache-size', 'max-postcopy-bandwidth',
>> 'max-cpu-throttle', 'multifd-compression',
>> 'multifd-zlib-level', 'multifd-zstd-level',
>> + 'multifd-qatzip-level', 'multifd-qatzip-sw-fallback',
>> 'block-bitmap-mapping',
>> { 'name': 'x-vcpu-dirty-limit-period', 'features':
>> ['unstable'] },
>> 'vcpu-dirty-limit',
>> @@ -1045,6 +1056,16 @@
>> # speed, and 9 means best compression ratio which will consume
>> # more CPU. Defaults to 1. (Since 5.0)
>> #
>> +# @multifd-qatzip-level: Set the compression level to be used in live
>> +# migration. The level is an integer between 1 and 9, where 1 means
>> +# the best compression speed, and 9 means the best compression
>> +# ratio which will consume more CPU. Defaults to 1.
>> +#
>> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
>> +# is unavailable. Defaults to false. Software fallback performance
>> +# is very poor compared to regular zlib, so be cautious about
>> +# enabling this option.
>> +#
>> # @multifd-zstd-level: Set the compression level to be used in live
>> # migration, the compression level is an integer between 0 and 20,
>> # where 0 means no compression, 1 means the best compression
>> @@ -1125,6 +1146,8 @@
>> '*max-cpu-throttle': 'uint8',
>> '*multifd-compression': 'MultiFDCompression',
>> '*multifd-zlib-level': 'uint8',
>> + '*multifd-qatzip-level': 'uint8',
>> + '*multifd-qatzip-sw-fallback': 'bool',
>> '*multifd-zstd-level': 'uint8',
>> '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
>> '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
>> @@ -1273,6 +1296,16 @@
>> # speed, and 9 means best compression ratio which will consume
>> # more CPU. Defaults to 1. (Since 5.0)
>> #
>> +# @multifd-qatzip-level: Set the compression level to be used in live
>> +# migration. The level is an integer between 1 and 9, where 1 means
>> +# the best compression speed, and 9 means the best compression
>> +# ratio which will consume more CPU. Defaults to 1.
>> +#
>> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
>> +# is unavailable. Defaults to false. Software fallback performance
>> +# is very poor compared to regular zlib, so be cautious about
>> +# enabling this option.
>> +#
>> # @multifd-zstd-level: Set the compression level to be used in live
>> # migration, the compression level is an integer between 0 and 20,
>> # where 0 means no compression, 1 means the best compression
>> @@ -1350,6 +1383,8 @@
>> '*max-cpu-throttle': 'uint8',
>> '*multifd-compression': 'MultiFDCompression',
>> '*multifd-zlib-level': 'uint8',
>> + '*multifd-qatzip-level': 'uint8',
>> + '*multifd-qatzip-sw-fallback': 'bool',
>> '*multifd-zstd-level': 'uint8',
>> '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
>> '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
>> --
>> 2.30.2
^ permalink raw reply [flat|nested] 13+ messages in thread* RE: [PATCH v2 2/5] migration: Add migration parameters for QATzip
2024-06-27 0:16 ` Yichen Wang
@ 2024-06-27 7:25 ` Liu, Yuan1
0 siblings, 0 replies; 13+ messages in thread
From: Liu, Yuan1 @ 2024-06-27 7:25 UTC (permalink / raw)
To: Wang, Yichen
Cc: Bryan Zhang, qemu-devel@nongnu.org, peterx@redhat.com,
farosas@suse.de, berrange@redhat.com, Zou, Nanhai,
hao.xiang@linux.dev
> -----Original Message-----
> From: Yichen Wang <yichen.wang@bytedance.com>
> Sent: Thursday, June 27, 2024 8:17 AM
> To: Liu, Yuan1 <yuan1.liu@intel.com>
> Cc: Bryan Zhang <bryan.zhang@bytedance.com>; qemu-devel@nongnu.org;
> peterx@redhat.com; farosas@suse.de; berrange@redhat.com; Zou, Nanhai
> <nanhai.zou@intel.com>; hao.xiang@linux.dev
> Subject: Re: [PATCH v2 2/5] migration: Add migration parameters for QATzip
>
>
>
> > On Mar 28, 2024, at 12:23 AM, Liu, Yuan1 <yuan1.liu@intel.com> wrote:
> >
> >> -----Original Message-----
> >> From: Bryan Zhang <bryan.zhang@bytedance.com>
> >> Sent: Wednesday, March 27, 2024 6:42 AM
> >> To: qemu-devel@nongnu.org
> >> Cc: peterx@redhat.com; farosas@suse.de; Liu, Yuan1
> <yuan1.liu@intel.com>;
> >> berrange@redhat.com; Zou, Nanhai <nanhai.zou@intel.com>;
> >> hao.xiang@linux.dev; Bryan Zhang <bryan.zhang@bytedance.com>
> >> Subject: [PATCH v2 2/5] migration: Add migration parameters for QATzip
> >>
> >> Adds support for migration parameters to control QATzip compression
> >> level and to enable/disable software fallback when QAT hardware is
> >> unavailable. This is a preparatory commit for a subsequent commit that
> >> will actually use QATzip compression.
> >>
> >> Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
> >> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> >> ---
> >> Revision: This commit now includes a parameter for controlling software
> >> fallback. Fallback is generally intended to be disabled, but having
> this
> >> option available enables using software fallback for testing.
> >>
> >> This commit also now has some glue code to properly set parameters.
> >>
> >> migration/migration-hmp-cmds.c | 8 +++++
> >> migration/options.c | 57 ++++++++++++++++++++++++++++++++++
> >> migration/options.h | 2 ++
> >> qapi/migration.json | 35 +++++++++++++++++++++
> >> 4 files changed, 102 insertions(+)
> >>
> >> diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-
> >> cmds.c
> >> index 99b49df5dd..4bd23ba14d 100644
> >> --- a/migration/migration-hmp-cmds.c
> >> +++ b/migration/migration-hmp-cmds.c
> >> @@ -630,6 +630,14 @@ void hmp_migrate_set_parameter(Monitor *mon, const
> >> QDict *qdict)
> >> p->has_multifd_zlib_level = true;
> >> visit_type_uint8(v, param, &p->multifd_zlib_level, &err);
> >> break;
> >> + case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL:
> >> + p->has_multifd_qatzip_level = true;
> >> + visit_type_uint8(v, param, &p->multifd_qatzip_level, &err);
> >> + break;
> >> + case MIGRATION_PARAMETER_MULTIFD_QATZIP_SW_FALLBACK:
> >> + p->has_multifd_qatzip_sw_fallback = true;
> >> + visit_type_bool(v, param, &p->multifd_qatzip_sw_fallback,
> &err);
> >> + break;
> >> case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL:
> >> p->has_multifd_zstd_level = true;
> >> visit_type_uint8(v, param, &p->multifd_zstd_level, &err);
> >> diff --git a/migration/options.c b/migration/options.c
> >> index 3e3e0b93b4..1316ea605a 100644
> >> --- a/migration/options.c
> >> +++ b/migration/options.c
> >> @@ -62,6 +62,15 @@
> >> #define DEFAULT_MIGRATE_MULTIFD_COMPRESSION MULTIFD_COMPRESSION_NONE
> >> /* 0: means nocompress, 1: best speed, ... 9: best compress ratio */
> >> #define DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL 1
> >> +/*
> >> + * 1: best speed, ... 9: best compress ratio
> >> + * There is some nuance here. Refer to QATzip documentation to
> understand
> >> + * the mapping of QATzip levels to standard deflate levels.
> >> + */
> >> +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL 1
> >> +/* QATzip's SW fallback implementation is extremely slow, so avoid
> >> fallback */
> >> +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_SW_FALLBACK false
> >> +
> >> /* 0: means nocompress, 1: best speed, ... 20: best compress ratio */
> >> #define DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL 1
> >
> > Hi Bryan
> >
> > The default compression level may be set higher, such as 6. I checked
> QAT throughput
> > If the data size is less than or equal to 64K, level 1 has much better
> throughput
> > performance than level 6 and level 9. But if the data size is greater
> than 128K, little
> > change in throughput, and the default MULTIFD_PACKET_SIZE is 512K, you
> can have a try
> > to use a high compression level, to get better compression performance
> without affecting
> > throughput.
> >
> > In addition, if you change MULTIFD_PACKET_SIZE to 64K, you may have
> better throughput
> > with more multifd threads
> Hi Yuan,
>
> Thanks for your comment. I did a quick experiments on our CPU. With chunk
> size at 512K, for silesia dataset, the throughput degrades at level 6:
> # for f in `seq 1 1 9`; do echo "Level $f:"; qzip -C 524288 -L $f -k
> silesia.8G | grep ":"; done
> Level 1:
> Time taken: 2230.772 ms
> Throughput: 30402.186 Mbit/s
> Space Savings: 65.712 %
> Compression ratio: 2.916 : 1
> Level 2:
> Time taken: 2245.205 ms
> Throughput: 30206.750 Mbit/s
> Space Savings: 65.712 %
> Compression ratio: 2.916 : 1
> Level 3:
> Time taken: 2217.789 ms
> Throughput: 30580.161 Mbit/s
> Space Savings: 65.712 %
> Compression ratio: 2.916 : 1
> Level 4:
> Time taken: 2251.014 ms
> Throughput: 30128.798 Mbit/s
> Space Savings: 65.712 %
> Compression ratio: 2.916 : 1
> Level 5:
> Time taken: 2200.991 ms
> Throughput: 30813.550 Mbit/s
> Space Savings: 65.712 %
> Compression ratio: 2.916 : 1
> Level 6:
> Time taken: 2508.218 ms
> Throughput: 27039.255 Mbit/s
> Space Savings: 67.396 %
> Compression ratio: 3.067 : 1
> Level 7:
> Time taken: 2510.847 ms
> Throughput: 27010.943 Mbit/s
> Space Savings: 67.396 %
> Compression ratio: 3.067 : 1
> Level 8:
> Time taken: 2521.428 ms
> Throughput: 26897.594 Mbit/s
> Space Savings: 67.396 %
> Compression ratio: 3.067 : 1
> Level 9:
> Time taken: 3071.055 ms
> Throughput: 22083.729 Mbit/s
> Space Savings: 67.664 %
> Compression ratio: 3.092 : 1
>
> For random text data, throughput actually improves at level 6.
> # for f in `seq 1 1 9`; do echo "Level $f:"; qzip -C 524288 -L $f -k
> text.txt | grep ":"; done
> Level 1:
> Time taken: 1788.683 ms
> Throughput: 8945.129 Mbit/s
> Space Savings: 24.959 %
> Compression ratio: 1.333 : 1
> Level 2:
> Time taken: 1786.135 ms
> Throughput: 8957.890 Mbit/s
> Space Savings: 24.959 %
> Compression ratio: 1.333 : 1
> Level 3:
> Time taken: 1785.564 ms
> Throughput: 8960.754 Mbit/s
> Space Savings: 24.959 %
> Compression ratio: 1.333 : 1
> Level 4:
> Time taken: 1787.351 ms
> Throughput: 8951.795 Mbit/s
> Space Savings: 24.959 %
> Compression ratio: 1.333 : 1
> Level 5:
> Time taken: 1785.171 ms
> Throughput: 8962.727 Mbit/s
> Space Savings: 24.959 %
> Compression ratio: 1.333 : 1
> Level 6:
> Time taken: 1752.000 ms
> Throughput: 9132.420 Mbit/s
> Space Savings: 24.873 %
> Compression ratio: 1.331 : 1
> Level 7:
> Time taken: 1752.297 ms
> Throughput: 9130.872 Mbit/s
> Space Savings: 24.873 %
> Compression ratio: 1.331 : 1
> Level 8:
> Time taken: 1752.538 ms
> Throughput: 9129.617 Mbit/s
> Space Savings: 24.873 %
> Compression ratio: 1.331 : 1
> Level 9:
> Time taken: 1762.593 ms
> Throughput: 9077.535 Mbit/s
> Space Savings: 24.870 %
> Compression ratio: 1.331 : 1
>
> For random binary data, throughput remains the same from 0-9.
>
> Given our live migration is mostly memory pages, and I am not very sure
> what scenario would better fit to above tests. I am OK with either default
> level (1 or 6). I can change to 6 in my next patchiest if you believe that
> is better.
Please use level 1 as the default compression level, my reason below
1. The level 6 does not significantly improve the compression ratio than level 1 for Silesia dataset
But the throughput dropped by 10%. I also test Calgary Corpus dataset, same with Silesia.
2. Users can modify the actual compression level according to multifd_qatzip_level.
> >> @@ -143,6 +152,12 @@ Property migration_properties[] = {
> >> DEFINE_PROP_UINT8("multifd-zlib-level", MigrationState,
> >> parameters.multifd_zlib_level,
> >> DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL),
> >> + DEFINE_PROP_UINT8("multifd-qatzip-level", MigrationState,
> >> + parameters.multifd_qatzip_level,
> >> + DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL),
> >> + DEFINE_PROP_BOOL("multifd-qatzip-sw-fallback", MigrationState,
> >> + parameters.multifd_qatzip_sw_fallback,
> >> + DEFAULT_MIGRATE_MULTIFD_QATZIP_SW_FALLBACK),
> >> DEFINE_PROP_UINT8("multifd-zstd-level", MigrationState,
> >> parameters.multifd_zstd_level,
> >> DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL),
> >> @@ -861,6 +876,20 @@ int migrate_multifd_zlib_level(void)
> >> return s->parameters.multifd_zlib_level;
> >> }
> >>
> >> +int migrate_multifd_qatzip_level(void)
> >> +{
> >> + MigrationState *s = migrate_get_current();
> >> +
> >> + return s->parameters.multifd_qatzip_level;
> >> +}
> >> +
> >> +bool migrate_multifd_qatzip_sw_fallback(void)
> >> +{
> >> + MigrationState *s = migrate_get_current();
> >> +
> >> + return s->parameters.multifd_qatzip_sw_fallback;
> >> +}
> >> +
> >> int migrate_multifd_zstd_level(void)
> >> {
> >> MigrationState *s = migrate_get_current();
> >> @@ -983,6 +1012,11 @@ MigrationParameters
> >> *qmp_query_migrate_parameters(Error **errp)
> >> params->multifd_compression = s->parameters.multifd_compression;
> >> params->has_multifd_zlib_level = true;
> >> params->multifd_zlib_level = s->parameters.multifd_zlib_level;
> >> + params->has_multifd_qatzip_level = true;
> >> + params->multifd_qatzip_level = s->parameters.multifd_qatzip_level;
> >> + params->has_multifd_qatzip_sw_fallback = true;
> >> + params->multifd_qatzip_sw_fallback =
> >> + s->parameters.multifd_qatzip_sw_fallback;
> >> params->has_multifd_zstd_level = true;
> >> params->multifd_zstd_level = s->parameters.multifd_zstd_level;
> >> params->has_xbzrle_cache_size = true;
> >> @@ -1038,6 +1072,8 @@ void migrate_params_init(MigrationParameters
> >> *params)
> >> params->has_multifd_channels = true;
> >> params->has_multifd_compression = true;
> >> params->has_multifd_zlib_level = true;
> >> + params->has_multifd_qatzip_level = true;
> >> + params->has_multifd_qatzip_sw_fallback = true;
> >> params->has_multifd_zstd_level = true;
> >> params->has_xbzrle_cache_size = true;
> >> params->has_max_postcopy_bandwidth = true;
> >> @@ -1147,6 +1183,14 @@ bool migrate_params_check(MigrationParameters
> >> *params, Error **errp)
> >> return false;
> >> }
> >>
> >> + if (params->has_multifd_qatzip_level &&
> >> + ((params->multifd_qatzip_level > 9) ||
> >> + (params->multifd_qatzip_level < 1))) {
> >> + error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
> >> "multifd_qatzip_level",
> >> + "a value between 1 and 9");
> >> + return false;
> >> + }
> >> +
> >> if (params->has_multifd_zstd_level &&
> >> (params->multifd_zstd_level > 20)) {
> >> error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
> >> "multifd_zstd_level",
> >> @@ -1312,6 +1356,12 @@ static void
> >> migrate_params_test_apply(MigrateSetParameters *params,
> >> if (params->has_multifd_compression) {
> >> dest->multifd_compression = params->multifd_compression;
> >> }
> >> + if (params->has_multifd_qatzip_level) {
> >> + dest->multifd_qatzip_level = params->multifd_qatzip_level;
> >> + }
> >> + if (params->has_multifd_qatzip_sw_fallback) {
> >> + dest->multifd_qatzip_sw_fallback = params-
> >>> multifd_qatzip_sw_fallback;
> >> + }
> >> if (params->has_xbzrle_cache_size) {
> >> dest->xbzrle_cache_size = params->xbzrle_cache_size;
> >> }
> >> @@ -1447,6 +1497,13 @@ static void
> >> migrate_params_apply(MigrateSetParameters *params, Error **errp)
> >> if (params->has_multifd_compression) {
> >> s->parameters.multifd_compression = params-
> >multifd_compression;
> >> }
> >> + if (params->has_multifd_qatzip_level) {
> >> + s->parameters.multifd_qatzip_level = params-
> >>> multifd_qatzip_level;
> >> + }
> >> + if (params->has_multifd_qatzip_sw_fallback) {
> >> + s->parameters.multifd_qatzip_sw_fallback =
> >> + params->multifd_qatzip_sw_fallback;
> >> + }
> >> if (params->has_xbzrle_cache_size) {
> >> s->parameters.xbzrle_cache_size = params->xbzrle_cache_size;
> >> xbzrle_cache_resize(params->xbzrle_cache_size, errp);
> >> diff --git a/migration/options.h b/migration/options.h
> >> index 246c160aee..94aee24d97 100644
> >> --- a/migration/options.h
> >> +++ b/migration/options.h
> >> @@ -87,6 +87,8 @@ MigMode migrate_mode(void);
> >> int migrate_multifd_channels(void);
> >> MultiFDCompression migrate_multifd_compression(void);
> >> int migrate_multifd_zlib_level(void);
> >> +int migrate_multifd_qatzip_level(void);
> >> +bool migrate_multifd_qatzip_sw_fallback(void);
> >> int migrate_multifd_zstd_level(void);
> >> uint8_t migrate_throttle_trigger_threshold(void);
> >> const char *migrate_tls_authz(void);
> >> diff --git a/qapi/migration.json b/qapi/migration.json
> >> index 0b33a71ab4..66ea6d32fc 100644
> >> --- a/qapi/migration.json
> >> +++ b/qapi/migration.json
> >> @@ -853,6 +853,16 @@
> >> # speed, and 9 means best compression ratio which will consume
> >> # more CPU. Defaults to 1. (Since 5.0)
> >> #
> >> +# @multifd-qatzip-level: Set the compression level to be used in live
> >> +# migration. The level is an integer between 1 and 9, where 1
> means
> >> +# the best compression speed, and 9 means the best compression
> >> +# ratio which will consume more CPU. Defaults to 1.
> >> +#
> >> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT
> hardware
> >> +# is unavailable. Defaults to false. Software fallback performance
> >> +# is very poor compared to regular zlib, so be cautious about
> >> +# enabling this option.
> >> +#
> >> # @multifd-zstd-level: Set the compression level to be used in live
> >> # migration, the compression level is an integer between 0 and 20,
> >> # where 0 means no compression, 1 means the best compression
> >> @@ -915,6 +925,7 @@
> >> 'xbzrle-cache-size', 'max-postcopy-bandwidth',
> >> 'max-cpu-throttle', 'multifd-compression',
> >> 'multifd-zlib-level', 'multifd-zstd-level',
> >> + 'multifd-qatzip-level', 'multifd-qatzip-sw-fallback',
> >> 'block-bitmap-mapping',
> >> { 'name': 'x-vcpu-dirty-limit-period', 'features':
> >> ['unstable'] },
> >> 'vcpu-dirty-limit',
> >> @@ -1045,6 +1056,16 @@
> >> # speed, and 9 means best compression ratio which will consume
> >> # more CPU. Defaults to 1. (Since 5.0)
> >> #
> >> +# @multifd-qatzip-level: Set the compression level to be used in live
> >> +# migration. The level is an integer between 1 and 9, where 1
> means
> >> +# the best compression speed, and 9 means the best compression
> >> +# ratio which will consume more CPU. Defaults to 1.
> >> +#
> >> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT
> hardware
> >> +# is unavailable. Defaults to false. Software fallback performance
> >> +# is very poor compared to regular zlib, so be cautious about
> >> +# enabling this option.
> >> +#
> >> # @multifd-zstd-level: Set the compression level to be used in live
> >> # migration, the compression level is an integer between 0 and 20,
> >> # where 0 means no compression, 1 means the best compression
> >> @@ -1125,6 +1146,8 @@
> >> '*max-cpu-throttle': 'uint8',
> >> '*multifd-compression': 'MultiFDCompression',
> >> '*multifd-zlib-level': 'uint8',
> >> + '*multifd-qatzip-level': 'uint8',
> >> + '*multifd-qatzip-sw-fallback': 'bool',
> >> '*multifd-zstd-level': 'uint8',
> >> '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
> >> '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
> >> @@ -1273,6 +1296,16 @@
> >> # speed, and 9 means best compression ratio which will consume
> >> # more CPU. Defaults to 1. (Since 5.0)
> >> #
> >> +# @multifd-qatzip-level: Set the compression level to be used in live
> >> +# migration. The level is an integer between 1 and 9, where 1
> means
> >> +# the best compression speed, and 9 means the best compression
> >> +# ratio which will consume more CPU. Defaults to 1.
> >> +#
> >> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT
> hardware
> >> +# is unavailable. Defaults to false. Software fallback performance
> >> +# is very poor compared to regular zlib, so be cautious about
> >> +# enabling this option.
> >> +#
> >> # @multifd-zstd-level: Set the compression level to be used in live
> >> # migration, the compression level is an integer between 0 and 20,
> >> # where 0 means no compression, 1 means the best compression
> >> @@ -1350,6 +1383,8 @@
> >> '*max-cpu-throttle': 'uint8',
> >> '*multifd-compression': 'MultiFDCompression',
> >> '*multifd-zlib-level': 'uint8',
> >> + '*multifd-qatzip-level': 'uint8',
> >> + '*multifd-qatzip-sw-fallback': 'bool',
> >> '*multifd-zstd-level': 'uint8',
> >> '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
> >> '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
> >> --
> >> 2.30.2
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2 2/5] migration: Add migration parameters for QATzip
2024-03-26 22:42 ` [PATCH v2 2/5] migration: Add migration parameters for QATzip Bryan Zhang
2024-03-28 7:23 ` Liu, Yuan1
@ 2024-04-01 15:30 ` Fabiano Rosas
1 sibling, 0 replies; 13+ messages in thread
From: Fabiano Rosas @ 2024-04-01 15:30 UTC (permalink / raw)
To: Bryan Zhang, qemu-devel
Cc: peterx, yuan1.liu, berrange, nanhai.zou, hao.xiang, Bryan Zhang
Bryan Zhang <bryan.zhang@bytedance.com> writes:
> Adds support for migration parameters to control QATzip compression
> level and to enable/disable software fallback when QAT hardware is
> unavailable. This is a preparatory commit for a subsequent commit that
> will actually use QATzip compression.
>
> Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
> Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
> ---
> Revision: This commit now includes a parameter for controlling software
> fallback. Fallback is generally intended to be disabled, but having this
> option available enables using software fallback for testing.
>
> This commit also now has some glue code to properly set parameters.
>
> migration/migration-hmp-cmds.c | 8 +++++
> migration/options.c | 57 ++++++++++++++++++++++++++++++++++
> migration/options.h | 2 ++
> qapi/migration.json | 35 +++++++++++++++++++++
> 4 files changed, 102 insertions(+)
>
> diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
> index 99b49df5dd..4bd23ba14d 100644
> --- a/migration/migration-hmp-cmds.c
> +++ b/migration/migration-hmp-cmds.c
> @@ -630,6 +630,14 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
> p->has_multifd_zlib_level = true;
> visit_type_uint8(v, param, &p->multifd_zlib_level, &err);
> break;
> + case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL:
> + p->has_multifd_qatzip_level = true;
> + visit_type_uint8(v, param, &p->multifd_qatzip_level, &err);
> + break;
> + case MIGRATION_PARAMETER_MULTIFD_QATZIP_SW_FALLBACK:
> + p->has_multifd_qatzip_sw_fallback = true;
> + visit_type_bool(v, param, &p->multifd_qatzip_sw_fallback, &err);
> + break;
> case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL:
> p->has_multifd_zstd_level = true;
> visit_type_uint8(v, param, &p->multifd_zstd_level, &err);
> diff --git a/migration/options.c b/migration/options.c
> index 3e3e0b93b4..1316ea605a 100644
> --- a/migration/options.c
> +++ b/migration/options.c
> @@ -62,6 +62,15 @@
> #define DEFAULT_MIGRATE_MULTIFD_COMPRESSION MULTIFD_COMPRESSION_NONE
> /* 0: means nocompress, 1: best speed, ... 9: best compress ratio */
> #define DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL 1
> +/*
> + * 1: best speed, ... 9: best compress ratio
> + * There is some nuance here. Refer to QATzip documentation to understand
> + * the mapping of QATzip levels to standard deflate levels.
> + */
> +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL 1
> +/* QATzip's SW fallback implementation is extremely slow, so avoid fallback */
> +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_SW_FALLBACK false
> +
> /* 0: means nocompress, 1: best speed, ... 20: best compress ratio */
> #define DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL 1
>
> @@ -143,6 +152,12 @@ Property migration_properties[] = {
> DEFINE_PROP_UINT8("multifd-zlib-level", MigrationState,
> parameters.multifd_zlib_level,
> DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL),
> + DEFINE_PROP_UINT8("multifd-qatzip-level", MigrationState,
> + parameters.multifd_qatzip_level,
> + DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL),
> + DEFINE_PROP_BOOL("multifd-qatzip-sw-fallback", MigrationState,
> + parameters.multifd_qatzip_sw_fallback,
> + DEFAULT_MIGRATE_MULTIFD_QATZIP_SW_FALLBACK),
> DEFINE_PROP_UINT8("multifd-zstd-level", MigrationState,
> parameters.multifd_zstd_level,
> DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL),
> @@ -861,6 +876,20 @@ int migrate_multifd_zlib_level(void)
> return s->parameters.multifd_zlib_level;
> }
>
> +int migrate_multifd_qatzip_level(void)
> +{
> + MigrationState *s = migrate_get_current();
> +
> + return s->parameters.multifd_qatzip_level;
> +}
> +
> +bool migrate_multifd_qatzip_sw_fallback(void)
> +{
> + MigrationState *s = migrate_get_current();
> +
> + return s->parameters.multifd_qatzip_sw_fallback;
> +}
> +
> int migrate_multifd_zstd_level(void)
> {
> MigrationState *s = migrate_get_current();
> @@ -983,6 +1012,11 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
> params->multifd_compression = s->parameters.multifd_compression;
> params->has_multifd_zlib_level = true;
> params->multifd_zlib_level = s->parameters.multifd_zlib_level;
> + params->has_multifd_qatzip_level = true;
> + params->multifd_qatzip_level = s->parameters.multifd_qatzip_level;
> + params->has_multifd_qatzip_sw_fallback = true;
> + params->multifd_qatzip_sw_fallback =
> + s->parameters.multifd_qatzip_sw_fallback;
> params->has_multifd_zstd_level = true;
> params->multifd_zstd_level = s->parameters.multifd_zstd_level;
> params->has_xbzrle_cache_size = true;
> @@ -1038,6 +1072,8 @@ void migrate_params_init(MigrationParameters *params)
> params->has_multifd_channels = true;
> params->has_multifd_compression = true;
> params->has_multifd_zlib_level = true;
> + params->has_multifd_qatzip_level = true;
> + params->has_multifd_qatzip_sw_fallback = true;
> params->has_multifd_zstd_level = true;
> params->has_xbzrle_cache_size = true;
> params->has_max_postcopy_bandwidth = true;
> @@ -1147,6 +1183,14 @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
> return false;
> }
>
> + if (params->has_multifd_qatzip_level &&
> + ((params->multifd_qatzip_level > 9) ||
> + (params->multifd_qatzip_level < 1))) {
> + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "multifd_qatzip_level",
> + "a value between 1 and 9");
> + return false;
> + }
> +
> if (params->has_multifd_zstd_level &&
> (params->multifd_zstd_level > 20)) {
> error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "multifd_zstd_level",
> @@ -1312,6 +1356,12 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
> if (params->has_multifd_compression) {
> dest->multifd_compression = params->multifd_compression;
> }
> + if (params->has_multifd_qatzip_level) {
> + dest->multifd_qatzip_level = params->multifd_qatzip_level;
> + }
> + if (params->has_multifd_qatzip_sw_fallback) {
> + dest->multifd_qatzip_sw_fallback = params->multifd_qatzip_sw_fallback;
> + }
> if (params->has_xbzrle_cache_size) {
> dest->xbzrle_cache_size = params->xbzrle_cache_size;
> }
> @@ -1447,6 +1497,13 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
> if (params->has_multifd_compression) {
> s->parameters.multifd_compression = params->multifd_compression;
> }
> + if (params->has_multifd_qatzip_level) {
> + s->parameters.multifd_qatzip_level = params->multifd_qatzip_level;
> + }
> + if (params->has_multifd_qatzip_sw_fallback) {
> + s->parameters.multifd_qatzip_sw_fallback =
> + params->multifd_qatzip_sw_fallback;
> + }
> if (params->has_xbzrle_cache_size) {
> s->parameters.xbzrle_cache_size = params->xbzrle_cache_size;
> xbzrle_cache_resize(params->xbzrle_cache_size, errp);
> diff --git a/migration/options.h b/migration/options.h
> index 246c160aee..94aee24d97 100644
> --- a/migration/options.h
> +++ b/migration/options.h
> @@ -87,6 +87,8 @@ MigMode migrate_mode(void);
> int migrate_multifd_channels(void);
> MultiFDCompression migrate_multifd_compression(void);
> int migrate_multifd_zlib_level(void);
> +int migrate_multifd_qatzip_level(void);
> +bool migrate_multifd_qatzip_sw_fallback(void);
> int migrate_multifd_zstd_level(void);
> uint8_t migrate_throttle_trigger_threshold(void);
> const char *migrate_tls_authz(void);
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 0b33a71ab4..66ea6d32fc 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -853,6 +853,16 @@
> # speed, and 9 means best compression ratio which will consume
> # more CPU. Defaults to 1. (Since 5.0)
> #
> +# @multifd-qatzip-level: Set the compression level to be used in live
> +# migration. The level is an integer between 1 and 9, where 1 means
> +# the best compression speed, and 9 means the best compression
> +# ratio which will consume more CPU. Defaults to 1.
> +#
> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
> +# is unavailable. Defaults to false. Software fallback performance
> +# is very poor compared to regular zlib, so be cautious about
> +# enabling this option.
> +#
> # @multifd-zstd-level: Set the compression level to be used in live
> # migration, the compression level is an integer between 0 and 20,
> # where 0 means no compression, 1 means the best compression
> @@ -915,6 +925,7 @@
> 'xbzrle-cache-size', 'max-postcopy-bandwidth',
> 'max-cpu-throttle', 'multifd-compression',
> 'multifd-zlib-level', 'multifd-zstd-level',
> + 'multifd-qatzip-level', 'multifd-qatzip-sw-fallback',
> 'block-bitmap-mapping',
> { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] },
> 'vcpu-dirty-limit',
> @@ -1045,6 +1056,16 @@
> # speed, and 9 means best compression ratio which will consume
> # more CPU. Defaults to 1. (Since 5.0)
> #
> +# @multifd-qatzip-level: Set the compression level to be used in live
> +# migration. The level is an integer between 1 and 9, where 1 means
> +# the best compression speed, and 9 means the best compression
> +# ratio which will consume more CPU. Defaults to 1.
(Since 9.1)
> +#
> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
> +# is unavailable. Defaults to false. Software fallback performance
> +# is very poor compared to regular zlib, so be cautious about
> +# enabling this option.
(Since 9.1)
> +#
> # @multifd-zstd-level: Set the compression level to be used in live
> # migration, the compression level is an integer between 0 and 20,
> # where 0 means no compression, 1 means the best compression
> @@ -1125,6 +1146,8 @@
> '*max-cpu-throttle': 'uint8',
> '*multifd-compression': 'MultiFDCompression',
> '*multifd-zlib-level': 'uint8',
> + '*multifd-qatzip-level': 'uint8',
> + '*multifd-qatzip-sw-fallback': 'bool',
> '*multifd-zstd-level': 'uint8',
> '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
> '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
> @@ -1273,6 +1296,16 @@
> # speed, and 9 means best compression ratio which will consume
> # more CPU. Defaults to 1. (Since 5.0)
> #
> +# @multifd-qatzip-level: Set the compression level to be used in live
> +# migration. The level is an integer between 1 and 9, where 1 means
> +# the best compression speed, and 9 means the best compression
> +# ratio which will consume more CPU. Defaults to 1.
(Since 9.1)
> +#
> +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
> +# is unavailable. Defaults to false. Software fallback performance
> +# is very poor compared to regular zlib, so be cautious about
> +# enabling this option.
(Since 9.1)
> +#
> # @multifd-zstd-level: Set the compression level to be used in live
> # migration, the compression level is an integer between 0 and 20,
> # where 0 means no compression, 1 means the best compression
> @@ -1350,6 +1383,8 @@
> '*max-cpu-throttle': 'uint8',
> '*multifd-compression': 'MultiFDCompression',
> '*multifd-zlib-level': 'uint8',
> + '*multifd-qatzip-level': 'uint8',
> + '*multifd-qatzip-sw-fallback': 'bool',
> '*multifd-zstd-level': 'uint8',
> '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
> '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v2 3/5] migration: Introduce unimplemented 'qatzip' compression method
2024-03-26 22:42 [PATCH v2 0/5] *** Implement using Intel QAT to offload ZLIB Bryan Zhang
2024-03-26 22:42 ` [PATCH v2 1/5] meson: Introduce 'qatzip' feature to the build system Bryan Zhang
2024-03-26 22:42 ` [PATCH v2 2/5] migration: Add migration parameters for QATzip Bryan Zhang
@ 2024-03-26 22:42 ` Bryan Zhang
2024-03-26 22:42 ` [PATCH v2 4/5] migration: Implement 'qatzip' methods using QAT Bryan Zhang
` (2 subsequent siblings)
5 siblings, 0 replies; 13+ messages in thread
From: Bryan Zhang @ 2024-03-26 22:42 UTC (permalink / raw)
To: qemu-devel
Cc: peterx, farosas, yuan1.liu, berrange, nanhai.zou, hao.xiang,
Bryan Zhang
Adds support for 'qatzip' as an option for the multifd compression
method parameter, but copy-pastes the no-op logic to leave the actual
methods effectively unimplemented. This is in preparation of a subsequent
commit that will implement actually using QAT for compression and
decompression.
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
hw/core/qdev-properties-system.c | 6 +-
migration/meson.build | 1 +
migration/multifd-qatzip.c | 117 +++++++++++++++++++++++++++++++
migration/multifd.h | 1 +
qapi/migration.json | 5 +-
tests/qtest/meson.build | 4 ++
6 files changed, 132 insertions(+), 2 deletions(-)
create mode 100644 migration/multifd-qatzip.c
diff --git a/hw/core/qdev-properties-system.c b/hw/core/qdev-properties-system.c
index 1a396521d5..d8e48dcb0e 100644
--- a/hw/core/qdev-properties-system.c
+++ b/hw/core/qdev-properties-system.c
@@ -658,7 +658,11 @@ const PropertyInfo qdev_prop_fdc_drive_type = {
const PropertyInfo qdev_prop_multifd_compression = {
.name = "MultiFDCompression",
.description = "multifd_compression values, "
- "none/zlib/zstd",
+ "none/zlib/zstd"
+#ifdef CONFIG_QATZIP
+ "/qatzip"
+#endif
+ ,
.enum_table = &MultiFDCompression_lookup,
.get = qdev_propinfo_get_enum,
.set = qdev_propinfo_set_enum,
diff --git a/migration/meson.build b/migration/meson.build
index 92b1cc4297..e20f318379 100644
--- a/migration/meson.build
+++ b/migration/meson.build
@@ -40,6 +40,7 @@ if get_option('live_block_migration').allowed()
system_ss.add(files('block.c'))
endif
system_ss.add(when: zstd, if_true: files('multifd-zstd.c'))
+system_ss.add(when: qatzip, if_true: files('multifd-qatzip.c'))
specific_ss.add(when: 'CONFIG_SYSTEM_ONLY',
if_true: files('ram.c',
diff --git a/migration/multifd-qatzip.c b/migration/multifd-qatzip.c
new file mode 100644
index 0000000000..f66336a4a7
--- /dev/null
+++ b/migration/multifd-qatzip.c
@@ -0,0 +1,117 @@
+/*
+ * Multifd QATzip compression implementation
+ *
+ * Copyright (c) Bytedance
+ *
+ * Authors:
+ * Bryan Zhang <bryan.zhang@bytedance.com>
+ * Hao Xiang <hao.xiang@bytedance.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "exec/ramblock.h"
+#include "exec/target_page.h"
+#include "qapi/error.h"
+#include "migration.h"
+#include "options.h"
+#include "multifd.h"
+
+/*
+ * This is an intermediary file to introduce 'qatzip' as an option for multifd
+ * compression. The actual method implementations are no-ops.
+ */
+
+static int qatzip_send_setup(MultiFDSendParams *p, Error **errp)
+{
+ if (migrate_zero_copy_send()) {
+ p->write_flags |= QIO_CHANNEL_WRITE_FLAG_ZERO_COPY;
+ }
+
+ return 0;
+}
+
+static void qatzip_send_cleanup(MultiFDSendParams *p, Error **errp)
+{
+ return;
+}
+
+static int qatzip_send_prepare(MultiFDSendParams *p, Error **errp)
+{
+ bool use_zero_copy_send = migrate_zero_copy_send();
+ MultiFDPages_t *pages = p->pages;
+ int ret;
+
+ if (!use_zero_copy_send) {
+ /*
+ * Only !zerocopy needs the header in IOV; zerocopy will
+ * send it separately.
+ */
+ multifd_send_prepare_header(p);
+ }
+
+ for (int i = 0; i < pages->num; i++) {
+ p->iov[p->iovs_num].iov_base = pages->block->host + pages->offset[i];
+ p->iov[p->iovs_num].iov_len = p->page_size;
+ p->iovs_num++;
+ }
+
+ p->next_packet_size = pages->num * p->page_size;
+ p->flags |= MULTIFD_FLAG_NOCOMP;
+
+ multifd_send_fill_packet(p);
+
+ if (use_zero_copy_send) {
+ /* Send header first, without zerocopy */
+ ret = qio_channel_write_all(p->c, (void *)p->packet,
+ p->packet_len, errp);
+ if (ret != 0) {
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int qatzip_recv_setup(MultiFDRecvParams *p, Error **errp)
+{
+ return 0;
+}
+
+static void qatzip_recv_cleanup(MultiFDRecvParams *p)
+{
+}
+
+static int qatzip_recv_pages(MultiFDRecvParams *p, Error **errp)
+{
+ uint32_t flags = p->flags & MULTIFD_FLAG_COMPRESSION_MASK;
+
+ if (flags != MULTIFD_FLAG_NOCOMP) {
+ error_setg(errp, "multifd %u: flags received %x flags expected %x",
+ p->id, flags, MULTIFD_FLAG_NOCOMP);
+ return -1;
+ }
+ for (int i = 0; i < p->normal_num; i++) {
+ p->iov[i].iov_base = p->host + p->normal[i];
+ p->iov[i].iov_len = p->page_size;
+ }
+ return qio_channel_readv_all(p->c, p->iov, p->normal_num, errp);
+}
+
+static MultiFDMethods multifd_qatzip_ops = {
+ .send_setup = qatzip_send_setup,
+ .send_cleanup = qatzip_send_cleanup,
+ .send_prepare = qatzip_send_prepare,
+ .recv_setup = qatzip_recv_setup,
+ .recv_cleanup = qatzip_recv_cleanup,
+ .recv_pages = qatzip_recv_pages
+};
+
+static void multifd_qatzip_register(void)
+{
+ multifd_register_ops(MULTIFD_COMPRESSION_QATZIP, &multifd_qatzip_ops);
+}
+
+migration_init(multifd_qatzip_register);
diff --git a/migration/multifd.h b/migration/multifd.h
index b3fe27ae93..ae73f1713c 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -33,6 +33,7 @@ bool multifd_queue_page(RAMBlock *block, ram_addr_t offset);
#define MULTIFD_FLAG_NOCOMP (0 << 1)
#define MULTIFD_FLAG_ZLIB (1 << 1)
#define MULTIFD_FLAG_ZSTD (2 << 1)
+#define MULTIFD_FLAG_QATZIP (3 << 1)
/* This value needs to be a multiple of qemu_target_page_size() */
#define MULTIFD_PACKET_SIZE (512 * 1024)
diff --git a/qapi/migration.json b/qapi/migration.json
index 66ea6d32fc..9018166ac8 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -625,11 +625,14 @@
#
# @zstd: use zstd compression method.
#
+# @qatzip: use qatzip compression method.
+#
# Since: 5.0
##
{ 'enum': 'MultiFDCompression',
'data': [ 'none', 'zlib',
- { 'name': 'zstd', 'if': 'CONFIG_ZSTD' } ] }
+ { 'name': 'zstd', 'if': 'CONFIG_ZSTD' },
+ { 'name': 'qatzip', 'if': 'CONFIG_QATZIP'} ] }
##
# @MigMode:
diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index 6ea77893f5..539104c06d 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -315,6 +315,10 @@ if gnutls.found()
endif
endif
+if qatzip.found()
+ migration_files += [qatzip]
+endif
+
qtests = {
'bios-tables-test': [io, 'boot-sector.c', 'acpi-utils.c', 'tpm-emu.c'],
'cdrom-test': files('boot-sector.c'),
--
2.30.2
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH v2 4/5] migration: Implement 'qatzip' methods using QAT
2024-03-26 22:42 [PATCH v2 0/5] *** Implement using Intel QAT to offload ZLIB Bryan Zhang
` (2 preceding siblings ...)
2024-03-26 22:42 ` [PATCH v2 3/5] migration: Introduce unimplemented 'qatzip' compression method Bryan Zhang
@ 2024-03-26 22:42 ` Bryan Zhang
2024-04-01 15:46 ` Fabiano Rosas
2024-03-26 22:42 ` [PATCH v2 5/5] tests/migration: Add integration test for 'qatzip' compression method Bryan Zhang
2024-03-28 7:32 ` [PATCH v2 0/5] *** Implement using Intel QAT to offload ZLIB Liu, Yuan1
5 siblings, 1 reply; 13+ messages in thread
From: Bryan Zhang @ 2024-03-26 22:42 UTC (permalink / raw)
To: qemu-devel
Cc: peterx, farosas, yuan1.liu, berrange, nanhai.zou, hao.xiang,
Bryan Zhang
Uses QAT to offload deflate compression and decompression in the
'qatzip' compression method for multifd migration.
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
migration/multifd-qatzip.c | 331 +++++++++++++++++++++++++++++++++----
1 file changed, 298 insertions(+), 33 deletions(-)
diff --git a/migration/multifd-qatzip.c b/migration/multifd-qatzip.c
index f66336a4a7..13835cf76f 100644
--- a/migration/multifd-qatzip.c
+++ b/migration/multifd-qatzip.c
@@ -18,86 +18,351 @@
#include "migration.h"
#include "options.h"
#include "multifd.h"
+#include <qatzip.h>
-/*
- * This is an intermediary file to introduce 'qatzip' as an option for multifd
- * compression. The actual method implementations are no-ops.
- */
+struct qatzip_data {
+ /*
+ * Unique session for use with QATzip API
+ */
+ QzSession_T sess;
+
+ /*
+ * For compression: Buffer for pages to compress
+ * For decompression: Buffer for data to decompress
+ */
+ uint8_t *in_buf;
+ uint32_t in_len;
+ /*
+ * For compression: Output buffer of compressed data
+ * For decompression: Output buffer of decompressed data
+ */
+ uint8_t *out_buf;
+ uint32_t out_len;
+};
+
+/**
+ * qatzip_send_setup: Set up QATzip session and private buffers.
+ *
+ * @param p Multifd channel params
+ * @param errp Pointer to error, which will be set in case of error
+ * @return 0 on success, -1 on error (and *errp will be set)
+ */
static int qatzip_send_setup(MultiFDSendParams *p, Error **errp)
{
- if (migrate_zero_copy_send()) {
- p->write_flags |= QIO_CHANNEL_WRITE_FLAG_ZERO_COPY;
+ struct qatzip_data *q;
+ QzSessionParamsDeflate_T params;
+ const char *err_msg;
+ int ret;
+ int sw_fallback;
+
+ q = g_new0(struct qatzip_data, 1);
+ p->data = q;
+
+ sw_fallback = 0;
+ if (migrate_multifd_qatzip_sw_fallback()) {
+ sw_fallback = 1;
+ }
+
+ ret = qzInit(&q->sess, sw_fallback);
+ if (ret != QZ_OK && ret != QZ_DUPLICATE) {
+ err_msg = "qzInit failed";
+ goto err_free_q;
+ }
+
+ ret = qzGetDefaultsDeflate(¶ms);
+ if (ret != QZ_OK) {
+ err_msg = "qzGetDefaultsDeflate failed";
+ goto err_close;
+ }
+
+ /* Make sure to use configured QATzip compression level. */
+ params.common_params.comp_lvl = migrate_multifd_qatzip_level();
+
+ ret = qzSetupSessionDeflate(&q->sess, ¶ms);
+ if (ret != QZ_OK && ret != QZ_DUPLICATE) {
+ err_msg = "qzSetupSessionDeflate failed";
+ goto err_close;
+ }
+
+ /* TODO Add support for larger packets. */
+ if (MULTIFD_PACKET_SIZE > UINT32_MAX) {
+ err_msg = "packet size too large for QAT";
+ goto err_close;
+ }
+
+ q->in_len = MULTIFD_PACKET_SIZE;
+ q->in_buf = qzMalloc(q->in_len, 0, PINNED_MEM);
+ if (!q->in_buf) {
+ err_msg = "qzMalloc failed";
+ goto err_close;
+ }
+
+ q->out_len = qzMaxCompressedLength(MULTIFD_PACKET_SIZE, &q->sess);
+ q->out_buf = qzMalloc(q->out_len, 0, PINNED_MEM);
+ if (!q->out_buf) {
+ err_msg = "qzMalloc failed";
+ goto err_free_inbuf;
}
return 0;
+
+err_free_inbuf:
+ qzFree(q->in_buf);
+err_close:
+ qzClose(&q->sess);
+err_free_q:
+ g_free(q);
+ error_setg(errp, "multifd %u: %s", p->id, err_msg);
+ return -1;
}
+/**
+ * qatzip_send_cleanup: Tear down QATzip session and release private buffers.
+ *
+ * @param p Multifd channel params
+ * @param errp Pointer to error, which will be set in case of error
+ * @return None
+ */
static void qatzip_send_cleanup(MultiFDSendParams *p, Error **errp)
{
+ struct qatzip_data *q = p->data;
+ const char *err_msg;
+ int ret;
+
+ ret = qzTeardownSession(&q->sess);
+ if (ret != QZ_OK) {
+ err_msg = "qzTeardownSession failed";
+ goto err;
+ }
+
+ ret = qzClose(&q->sess);
+ if (ret != QZ_OK) {
+ err_msg = "qzClose failed";
+ goto err;
+ }
+
+ qzFree(q->in_buf);
+ q->in_buf = NULL;
+ qzFree(q->out_buf);
+ q->out_buf = NULL;
+ g_free(p->data);
+ p->data = NULL;
return;
+
+err:
+ error_setg(errp, "multifd %u: %s", p->id, err_msg);
}
+/**
+ * qatzip_send_prepare: Compress pages and update IO channel info.
+ *
+ * @param p Multifd channel params
+ * @param errp Pointer to error, which will be set in case of error
+ * @return 0 on success, -1 on error (and *errp will be set)
+ */
static int qatzip_send_prepare(MultiFDSendParams *p, Error **errp)
{
- bool use_zero_copy_send = migrate_zero_copy_send();
MultiFDPages_t *pages = p->pages;
+ struct qatzip_data *q = p->data;
int ret;
+ unsigned int in_len, out_len;
- if (!use_zero_copy_send) {
- /*
- * Only !zerocopy needs the header in IOV; zerocopy will
- * send it separately.
- */
- multifd_send_prepare_header(p);
- }
+ multifd_send_prepare_header(p);
+ /* memcpy all the pages into one buffer. */
for (int i = 0; i < pages->num; i++) {
- p->iov[p->iovs_num].iov_base = pages->block->host + pages->offset[i];
- p->iov[p->iovs_num].iov_len = p->page_size;
- p->iovs_num++;
+ memcpy(q->in_buf + (i * p->page_size),
+ p->pages->block->host + pages->offset[i],
+ p->page_size);
}
- p->next_packet_size = pages->num * p->page_size;
- p->flags |= MULTIFD_FLAG_NOCOMP;
-
- multifd_send_fill_packet(p);
+ in_len = pages->num * p->page_size;
+ if (in_len > q->in_len) {
+ error_setg(errp, "multifd %u: unexpectedly large input", p->id);
+ return -1;
+ }
+ out_len = q->out_len;
- if (use_zero_copy_send) {
- /* Send header first, without zerocopy */
- ret = qio_channel_write_all(p->c, (void *)p->packet,
- p->packet_len, errp);
- if (ret != 0) {
- return -1;
- }
+ /*
+ * Unlike other multifd compression implementations, we use a non-streaming
+ * API and place all the data into one buffer, rather than sending each page
+ * to the compression API at a time. Based on initial benchmarks, the
+ * non-streaming API outperforms the streaming API. Plus, the logic in QEMU
+ * is friendly to using the non-streaming API anyway. If either of these
+ * statements becomes no longer true, we can revisit adding a streaming
+ * implementation.
+ */
+ ret = qzCompress(&q->sess, q->in_buf, &in_len, q->out_buf, &out_len, 1);
+ if (ret != QZ_OK) {
+ error_setg(errp, "multifd %u: QATzip returned %d instead of QZ_OK",
+ p->id, ret);
+ return -1;
+ }
+ if (in_len != pages->num * p->page_size) {
+ error_setg(errp, "multifd %u: QATzip failed to compress all input",
+ p->id);
+ return -1;
}
+ p->iov[p->iovs_num].iov_base = q->out_buf;
+ p->iov[p->iovs_num].iov_len = out_len;
+ p->iovs_num++;
+ p->next_packet_size = out_len;
+ p->flags |= MULTIFD_FLAG_QATZIP;
+
+ multifd_send_fill_packet(p);
+
return 0;
}
+/**
+ * qatzip_recv_setup: Set up QATzip session and allocate private buffers.
+ *
+ * @param p Multifd channel params
+ * @param errp Pointer to error, which will be set in case of error
+ * @return 0 on success, -1 on error (and *errp will be set)
+ */
static int qatzip_recv_setup(MultiFDRecvParams *p, Error **errp)
{
+ struct qatzip_data *q;
+ QzSessionParamsDeflate_T params;
+ const char *err_msg;
+ int ret;
+ int sw_fallback;
+
+ q = g_new0(struct qatzip_data, 1);
+ p->data = q;
+
+ sw_fallback = 0;
+ if (migrate_multifd_qatzip_sw_fallback()) {
+ sw_fallback = 1;
+ }
+
+ ret = qzInit(&q->sess, sw_fallback);
+ if (ret != QZ_OK && ret != QZ_DUPLICATE) {
+ err_msg = "qzInit failed";
+ goto err_free_q;
+ }
+
+ ret = qzGetDefaultsDeflate(¶ms);
+ if (ret != QZ_OK) {
+ err_msg = "qzGetDefaultsDeflate failed";
+ goto err_close;
+ }
+
+ /* Make sure to use configured QATzip compression level. */
+ params.common_params.comp_lvl = migrate_multifd_qatzip_level();
+
+ ret = qzSetupSessionDeflate(&q->sess, ¶ms);
+ if (ret != QZ_OK && ret != QZ_DUPLICATE) {
+ err_msg = "qzSetupSessionDeflate failed";
+ goto err_close;
+ }
+
+ /*
+ * Mimic multifd-zlib, which reserves extra space for the
+ * incoming packet.
+ */
+ q->in_len = MULTIFD_PACKET_SIZE * 2;
+ q->in_buf = qzMalloc(q->in_len, 0, PINNED_MEM);
+ if (!q->in_buf) {
+ err_msg = "qzMalloc failed";
+ goto err_close;
+ }
+
+ q->out_len = MULTIFD_PACKET_SIZE;
+ q->out_buf = qzMalloc(q->out_len, 0, PINNED_MEM);
+ if (!q->out_buf) {
+ err_msg = "qzMalloc failed";
+ goto err_free_inbuf;
+ }
+
return 0;
+
+err_free_inbuf:
+ qzFree(q->in_buf);
+err_close:
+ qzClose(&q->sess);
+err_free_q:
+ g_free(q);
+ error_setg(errp, "multifd %u: %s", p->id, err_msg);
+ return -1;
}
+/**
+ * qatzip_recv_cleanup: Tear down QATzip session and release private buffers.
+ *
+ * @param p Multifd channel params
+ * @return None
+ */
static void qatzip_recv_cleanup(MultiFDRecvParams *p)
{
+ struct qatzip_data *q = p->data;
+
+ /* Ignoring return values here due to function signature. */
+ qzTeardownSession(&q->sess);
+ qzClose(&q->sess);
+ qzFree(q->in_buf);
+ qzFree(q->out_buf);
+ g_free(p->data);
}
+
+/**
+ * qatzip_recv_pages: Decompress pages and copy them to the appropriate
+ * locations.
+ *
+ * @param p Multifd channel params
+ * @param errp Pointer to error, which will be set in case of error
+ * @return 0 on success, -1 on error (and *errp will be set)
+ */
static int qatzip_recv_pages(MultiFDRecvParams *p, Error **errp)
{
+ struct qatzip_data *q = p->data;
+ int ret;
+ unsigned int in_len, out_len;
+ uint32_t in_size = p->next_packet_size;
+ uint32_t expected_size = p->normal_num * p->page_size;
uint32_t flags = p->flags & MULTIFD_FLAG_COMPRESSION_MASK;
- if (flags != MULTIFD_FLAG_NOCOMP) {
+ if (in_size > q->in_len) {
+ error_setg(errp, "multifd %u: received unexpectedly large packet",
+ p->id);
+ return -1;
+ }
+
+ if (flags != MULTIFD_FLAG_QATZIP) {
error_setg(errp, "multifd %u: flags received %x flags expected %x",
- p->id, flags, MULTIFD_FLAG_NOCOMP);
+ p->id, flags, MULTIFD_FLAG_QATZIP);
+ return -1;
+ }
+
+ ret = qio_channel_read_all(p->c, (void *)q->in_buf, in_size, errp);
+ if (ret != 0) {
+ return ret;
+ }
+
+ in_len = in_size;
+ out_len = q->out_len;
+ ret = qzDecompress(&q->sess, q->in_buf, &in_len, q->out_buf, &out_len);
+ if (ret != QZ_OK) {
+ error_setg(errp, "multifd %u: qzDecompress failed", p->id);
+ return -1;
+ }
+ if (out_len != expected_size) {
+ error_setg(errp, "multifd %u: packet size received %u size expected %u",
+ p->id, out_len, expected_size);
return -1;
}
+
+ /* Copy each page to its appropriate location. */
for (int i = 0; i < p->normal_num; i++) {
- p->iov[i].iov_base = p->host + p->normal[i];
- p->iov[i].iov_len = p->page_size;
+ memcpy(p->host + p->normal[i],
+ q->out_buf + p->page_size * i,
+ p->page_size);
}
- return qio_channel_readv_all(p->c, p->iov, p->normal_num, errp);
+ return 0;
}
static MultiFDMethods multifd_qatzip_ops = {
--
2.30.2
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH v2 5/5] tests/migration: Add integration test for 'qatzip' compression method
2024-03-26 22:42 [PATCH v2 0/5] *** Implement using Intel QAT to offload ZLIB Bryan Zhang
` (3 preceding siblings ...)
2024-03-26 22:42 ` [PATCH v2 4/5] migration: Implement 'qatzip' methods using QAT Bryan Zhang
@ 2024-03-26 22:42 ` Bryan Zhang
2024-04-01 15:40 ` Fabiano Rosas
2024-03-28 7:32 ` [PATCH v2 0/5] *** Implement using Intel QAT to offload ZLIB Liu, Yuan1
5 siblings, 1 reply; 13+ messages in thread
From: Bryan Zhang @ 2024-03-26 22:42 UTC (permalink / raw)
To: qemu-devel
Cc: peterx, farosas, yuan1.liu, berrange, nanhai.zou, hao.xiang,
Bryan Zhang
Adds an integration test for 'qatzip'.
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
---
Revision: This commit now does some parameter setting to test that
changing the 'multifd-qatzip-level' parameter works, and to enable
software fallback so that the QATzip test can be run even if the test
machine does not have QAT.
tests/qtest/migration-test.c | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index 83512bce85..997f0aa323 100644
--- a/tests/qtest/migration-test.c
+++ b/tests/qtest/migration-test.c
@@ -35,6 +35,10 @@
# endif /* CONFIG_TASN1 */
#endif /* CONFIG_GNUTLS */
+#ifdef CONFIG_QATZIP
+#include <qatzip.h>
+#endif /* CONFIG_QATZIP */
+
/* For dirty ring test; so far only x86_64 is supported */
#if defined(__linux__) && defined(HOST_X86_64)
#include "linux/kvm.h"
@@ -2676,6 +2680,22 @@ test_migrate_precopy_tcp_multifd_zstd_start(QTestState *from,
}
#endif /* CONFIG_ZSTD */
+#ifdef CONFIG_QATZIP
+static void *
+test_migrate_precopy_tcp_multifd_qatzip_start(QTestState *from,
+ QTestState *to)
+{
+ migrate_set_parameter_int(from, "multifd-qatzip-level", 2);
+ migrate_set_parameter_int(to, "multifd-qatzip-level", 2);
+
+ /* SW fallback is disabled by default, so enable it for testing. */
+ migrate_set_parameter_bool(from, "multifd-qatzip-sw-fallback", true);
+ migrate_set_parameter_bool(to, "multifd-qatzip-sw-fallback", true);
+
+ return test_migrate_precopy_tcp_multifd_start_common(from, to, "qatzip");
+}
+#endif
+
static void test_multifd_tcp_none(void)
{
MigrateCommon args = {
@@ -2711,6 +2731,17 @@ static void test_multifd_tcp_zstd(void)
}
#endif
+#ifdef CONFIG_QATZIP
+static void test_multifd_tcp_qatzip(void)
+{
+ MigrateCommon args = {
+ .listen_uri = "defer",
+ .start_hook = test_migrate_precopy_tcp_multifd_qatzip_start,
+ };
+ test_precopy_common(&args);
+}
+#endif
+
#ifdef CONFIG_GNUTLS
static void *
test_migrate_multifd_tcp_tls_psk_start_match(QTestState *from,
@@ -3600,6 +3631,10 @@ int main(int argc, char **argv)
migration_test_add("/migration/multifd/tcp/plain/zstd",
test_multifd_tcp_zstd);
#endif
+#ifdef CONFIG_QATZIP
+ migration_test_add("/migration/multifd/tcp/plain/qatzip",
+ test_multifd_tcp_qatzip);
+#endif
#ifdef CONFIG_GNUTLS
migration_test_add("/migration/multifd/tcp/tls/psk/match",
test_multifd_tcp_tls_psk_match);
--
2.30.2
^ permalink raw reply related [flat|nested] 13+ messages in thread* RE: [PATCH v2 0/5] *** Implement using Intel QAT to offload ZLIB
2024-03-26 22:42 [PATCH v2 0/5] *** Implement using Intel QAT to offload ZLIB Bryan Zhang
` (4 preceding siblings ...)
2024-03-26 22:42 ` [PATCH v2 5/5] tests/migration: Add integration test for 'qatzip' compression method Bryan Zhang
@ 2024-03-28 7:32 ` Liu, Yuan1
5 siblings, 0 replies; 13+ messages in thread
From: Liu, Yuan1 @ 2024-03-28 7:32 UTC (permalink / raw)
To: Bryan Zhang, qemu-devel@nongnu.org
Cc: peterx@redhat.com, farosas@suse.de, berrange@redhat.com,
Zou, Nanhai, hao.xiang@linux.dev
> -----Original Message-----
> From: Bryan Zhang <bryan.zhang@bytedance.com>
> Sent: Wednesday, March 27, 2024 6:42 AM
> To: qemu-devel@nongnu.org
> Cc: peterx@redhat.com; farosas@suse.de; Liu, Yuan1 <yuan1.liu@intel.com>;
> berrange@redhat.com; Zou, Nanhai <nanhai.zou@intel.com>;
> hao.xiang@linux.dev; Bryan Zhang <bryan.zhang@bytedance.com>
> Subject: [PATCH v2 0/5] *** Implement using Intel QAT to offload ZLIB
>
> v2:
> - Rebase changes on top of recent multifd code changes.
> - Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
> - Remove parameter tuning and use QATzip's defaults for better
> performance.
> - Add parameter to enable QAT software fallback.
>
> v1:
> https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg03761.html
>
> * Performance
>
> We present updated performance results. For circumstantial reasons, v1
> presented performance on a low-bandwidth (1Gbps) network.
>
> Here, we present updated results with a similar setup as before but with
> two main differences:
>
> 1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
> 2. We had a bug in our memory allocation causing us to only use ~1/2 of
> the VM's RAM. Now we properly allocate and fill nearly all of the VM's
> RAM.
>
> Thus, the test setup is as follows:
>
> We perform multifd live migration over TCP using a VM with 64GB memory.
> We prepare the machine's memory by powering it on, allocating a large
> amount of memory (60GB) as a single buffer, and filling the buffer with
> the repeated contents of the Silesia corpus[0]. This is in lieu of a more
> realistic memory snapshot, which proved troublesome to acquire.
>
> We analyze CPU usage by averaging the output of 'top' every second
> during migration. This is admittedly imprecise, but we feel that it
> accurately portrays the different degrees of CPU usage of varying
> compression methods.
>
> We present the latency, throughput, and CPU usage results for all of the
> compression methods, with varying numbers of multifd threads (4, 8, and
> 16).
>
> [0] The Silesia corpus can be accessed here:
> https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia
>
> ** Results
>
> 4 multifd threads:
>
> |---------------|---------------|----------------|---------|---------|
> |method |time(sec) |throughput(mbps)|send cpu%|recv cpu%|
> |---------------|---------------|----------------|---------|---------|
> |qatzip | 23.13 | 8749.94 |117.50 |186.49 |
> |---------------|---------------|----------------|---------|---------|
> |zlib |254.35 | 771.87 |388.20 |144.40 |
> |---------------|---------------|----------------|---------|---------|
> |zstd | 54.52 | 3442.59 |414.59 |149.77 |
> |---------------|---------------|----------------|---------|---------|
> |none | 12.45 |43739.60 |159.71 |204.96 |
> |---------------|---------------|----------------|---------|---------|
>
> 8 multifd threads:
>
> |---------------|---------------|----------------|---------|---------|
> |method |time(sec) |throughput(mbps)|send cpu%|recv cpu%|
> |---------------|---------------|----------------|---------|---------|
> |qatzip | 16.91 |12306.52 |186.37 |391.84 |
> |---------------|---------------|----------------|---------|---------|
> |zlib |130.11 | 1508.89 |753.86 |289.35 |
> |---------------|---------------|----------------|---------|---------|
> |zstd | 27.57 | 6823.23 |786.83 |303.80 |
> |---------------|---------------|----------------|---------|---------|
> |none | 11.82 |46072.63 |163.74 |238.56 |
> |---------------|---------------|----------------|---------|---------|
>
> 16 multifd threads:
>
> |---------------|---------------|----------------|---------|---------|
> |method |time(sec) |throughput(mbps)|send cpu%|recv cpu%|
> |---------------|---------------|----------------|---------|---------|
> |qatzip |18.64 |11044.52 | 573.61 |437.65 |
> |---------------|---------------|----------------|---------|---------|
> |zlib |66.43 | 2955.79 |1469.68 |567.47 |
> |---------------|---------------|----------------|---------|---------|
> |zstd |14.17 |13290.66 |1504.08 |615.33 |
> |---------------|---------------|----------------|---------|---------|
> |none |16.82 |32363.26 | 180.74 |217.17 |
> |---------------|---------------|----------------|---------|---------|
>
> ** Observations
I'm a little confused about the CPU utilization on the destination for
decompression, it seems the CPU is decompressing instead of QAT, I check
the code about qzDecompress, it is the same with qzCompress if the decompression
task is not completed, it will try to stay sleep state as much as possible.
Maybe I understand it incorrectly, but I think QAT should help save more CPU resources
in both compression and decompression.
Thank you very much for providing this version. I will set up an environment on your patch
set to test the performance.
> - In general, not using compression outperforms using compression in a
> non-network-bound environment.
> - 'qatzip' outperforms other compression workers with 4 and 8 workers,
> achieving a ~91% latency reduction over 'zlib' with 4 workers, and a
> ~58% latency reduction over 'zstd' with 4 workers.
> - 'qatzip' maintains comparable performance with 'zstd' at 16 workers,
> showing a ~32% increase in latency. This performance difference
> becomes more noticeable with more workers, as CPU compression is highly
> parallelizable.
> - 'qatzip' compression uses considerably less CPU than other compression
> methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
> compression CPU usage compared to 'zstd' and 'zlib'.
> - 'qatzip' decompression CPU usage is less impressive, and is even
> slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16 workers.
>
> Bryan Zhang (5):
> meson: Introduce 'qatzip' feature to the build system
> migration: Add migration parameters for QATzip
> migration: Introduce unimplemented 'qatzip' compression method
> migration: Implement 'qatzip' methods using QAT
> tests/migration: Add integration test for 'qatzip' compression method
>
> hw/core/qdev-properties-system.c | 6 +-
> meson.build | 10 +
> meson_options.txt | 2 +
> migration/meson.build | 1 +
> migration/migration-hmp-cmds.c | 8 +
> migration/multifd-qatzip.c | 382 +++++++++++++++++++++++++++++++
> migration/multifd.h | 1 +
> migration/options.c | 57 +++++
> migration/options.h | 2 +
> qapi/migration.json | 40 +++-
> scripts/meson-buildoptions.sh | 3 +
> tests/qtest/meson.build | 4 +
> tests/qtest/migration-test.c | 35 +++
> 13 files changed, 549 insertions(+), 2 deletions(-)
> create mode 100644 migration/multifd-qatzip.c
>
> --
> 2.30.2
^ permalink raw reply [flat|nested] 13+ messages in thread