* [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics
@ 2013-01-16 16:24 Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 01/14] qcow2: Add deduplication metrics structures Benoît Canet
` (13 more replies)
0 siblings, 14 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:24 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
This patchset implements QMP metrics for the qcow2 deduplication.
It applies on top of the "QCOW2 deduplication core functionality" patchset.
Benoît Canet (14):
qcow2: Add deduplication metrics structures.
qcow2: Initialize deduplication metrics.
qcow2: Collect unaligned writes missing data reads metric.
qcow2: Collect deduplicated cluster metric.
qcow2: Collect undeduplicated cluster metric.
qcow2: Count QCowHashNode creation metrics.
qcow2: Count QCowHashNode removal from tree for metrics.
qcow2: Count cluster deleted metric
qcow2: Count deduplication refcount overflow metric.
qapi: Add support for deduplication infos in qapi-schema.json.
block: Add deduplication metrics to BlockDriverInfo.
qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage.
qcow2: returns deduplication metrics and status via bdrv_get_info()
qapi: Return virtual block device deduplication metrics in QMP
block.c | 36 ++++++++++++++++++++++++++++++++++++
block/qcow2-dedup.c | 36 ++++++++++++++++++++++++++++++++----
block/qcow2.c | 4 ++++
block/qcow2.h | 4 +++-
include/block/block.h | 14 ++++++++++++++
qapi-schema.json | 40 +++++++++++++++++++++++++++++++++++++++-
6 files changed, 128 insertions(+), 6 deletions(-)
--
1.7.10.4
^ permalink raw reply [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 01/14] qcow2: Add deduplication metrics structures.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
@ 2013-01-16 16:24 ` Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 02/14] qcow2: Initialize deduplication metrics Benoît Canet
` (12 subsequent siblings)
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:24 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2.h | 3 ++-
include/block/block.h | 11 +++++++++++
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/block/qcow2.h b/block/qcow2.h
index 29267a9..0729ff2 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -237,9 +237,10 @@ typedef struct BDRVQcowState {
int32_t dedup_table_size;
GTree *dedup_tree_by_hash;
GTree *dedup_tree_by_sect;
+ CoMutex dedup_lock;
+ BlockDeduplicationMetrics dedup_metrics;
CoMutex lock;
- CoMutex dedup_lock;
uint32_t crypt_method; /* current crypt method, 0 if no key yet */
uint32_t crypt_method_header;
diff --git a/include/block/block.h b/include/block/block.h
index b81d200..16e1cf1 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -12,6 +12,17 @@
typedef struct BlockDriver BlockDriver;
typedef struct BlockJob BlockJob;
+typedef struct {
+ uint64_t deduplicated_clusters;
+ uint64_t non_deduplicated_clusters;
+ uint64_t missing_data_reads; /* reads used to complete partials clusters */
+ uint64_t ram_hash_creations; /* RAM based lookup */
+ uint64_t ram_hash_deletions; /* RAM based lookup */
+ uint64_t ram_usage; /* RAM usage in bytes */
+ uint64_t deleted_clusters; /* number of deleted clusters */
+ uint64_t refcount_overflows; /* number of refcount overflows */
+} BlockDeduplicationMetrics;
+
typedef struct BlockDriverInfo {
/* in bytes, 0 if irrelevant */
int cluster_size;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 02/14] qcow2: Initialize deduplication metrics.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 01/14] qcow2: Add deduplication metrics structures Benoît Canet
@ 2013-01-16 16:24 ` Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 03/14] qcow2: Collect unaligned writes missing data reads metric Benoît Canet
` (11 subsequent siblings)
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:24 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2-dedup.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
index 6cd1af4..997714b 100644
--- a/block/qcow2-dedup.c
+++ b/block/qcow2-dedup.c
@@ -1272,6 +1272,8 @@ int qcow2_dedup_init(BlockDriverState *bs)
s->has_dedup = true;
+ memset(&s->dedup_metrics, 0, sizeof(s->dedup_metrics));
+
ret = qcow2_dedup_alloc(bs);
if (ret < 0) {
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 03/14] qcow2: Collect unaligned writes missing data reads metric.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 01/14] qcow2: Add deduplication metrics structures Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 02/14] qcow2: Initialize deduplication metrics Benoît Canet
@ 2013-01-16 16:24 ` Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 04/14] qcow2: Collect deduplicated cluster metric Benoît Canet
` (10 subsequent siblings)
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:24 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2-dedup.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
index 997714b..e4920d4 100644
--- a/block/qcow2-dedup.c
+++ b/block/qcow2-dedup.c
@@ -130,6 +130,7 @@ int qcow2_dedup_read_missing_and_concatenate(BlockDriverState *bs,
/* read beginning */
if (cluster_beginning_nr) {
+ s->dedup_metrics.missing_data_reads++;
ret = qcow2_read_cluster_data(bs,
*data,
cluster_beginning_sector,
@@ -153,6 +154,7 @@ int qcow2_dedup_read_missing_and_concatenate(BlockDriverState *bs,
/* read and add ending */
if (cluster_ending_nr) {
+ s->dedup_metrics.missing_data_reads++;
ret = qcow2_read_cluster_data(bs,
*data +
(cluster_beginning_nr +
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 04/14] qcow2: Collect deduplicated cluster metric.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (2 preceding siblings ...)
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 03/14] qcow2: Collect unaligned writes missing data reads metric Benoît Canet
@ 2013-01-16 16:24 ` Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 05/14] qcow2: Collect undeduplicated " Benoît Canet
` (9 subsequent siblings)
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:24 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2-dedup.c | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
index e4920d4..716371c 100644
--- a/block/qcow2-dedup.c
+++ b/block/qcow2-dedup.c
@@ -400,10 +400,14 @@ static int qcow2_deduplicate_cluster(BlockDriverState *bs,
}
/* Increment the refcount of the cluster */
- return update_refcount(bs,
- (hash_node->physical_sect /
- s->cluster_sectors) << s->cluster_bits,
- 1, 1, true);
+ ret = update_refcount(bs,
+ (hash_node->physical_sect /
+ s->cluster_sectors) << s->cluster_bits,
+ 1, 1, true);
+
+ s->dedup_metrics.deduplicated_clusters++;
+
+ return ret;
}
/* This function tries to deduplicate a given cluster.
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 05/14] qcow2: Collect undeduplicated cluster metric.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (3 preceding siblings ...)
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 04/14] qcow2: Collect deduplicated cluster metric Benoît Canet
@ 2013-01-16 16:24 ` Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 06/14] qcow2: Count QCowHashNode creation metrics Benoît Canet
` (8 subsequent siblings)
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:24 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2-dedup.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
index 716371c..0f095a9 100644
--- a/block/qcow2-dedup.c
+++ b/block/qcow2-dedup.c
@@ -524,6 +524,7 @@ static int qcow2_count_next_non_dedupable_clusters(BlockDriverState *bs,
uint8_t *data,
int left_to_process)
{
+ BDRVQcowState *s = bs->opaque;
int i;
int ret = 0;
QCowHashNode *hash_node;
@@ -546,6 +547,7 @@ static int qcow2_count_next_non_dedupable_clusters(BlockDriverState *bs,
qcow2_build_and_insert_hash_node(bs, &ds->phash.hash);
add_hash_to_undedupable_list(bs, ds);
+ s->dedup_metrics.non_deduplicated_clusters++;
}
return i;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 06/14] qcow2: Count QCowHashNode creation metrics.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (4 preceding siblings ...)
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 05/14] qcow2: Collect undeduplicated " Benoît Canet
@ 2013-01-16 16:25 ` Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 07/14] qcow2: Count QCowHashNode removal from tree for metrics Benoît Canet
` (7 subsequent siblings)
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:25 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2-dedup.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
index 0f095a9..d22e2a4 100644
--- a/block/qcow2-dedup.c
+++ b/block/qcow2-dedup.c
@@ -289,6 +289,7 @@ static void qcow2_build_and_insert_hash_node(BlockDriverState *bs,
QCOW_FLAG_EMPTY,
QCOW_FLAG_EMPTY);
g_tree_insert(s->dedup_tree_by_hash, &hash_node->hash, hash_node);
+ s->dedup_metrics.ram_hash_creations++;
}
/*
@@ -1043,6 +1044,7 @@ static void qcow2_dedup_insert_hash_node(BlockDriverState *bs,
g_tree_insert(s->dedup_tree_by_hash, &hash_node->hash, hash_node);
g_tree_insert(s->dedup_tree_by_sect, &hash_node->physical_sect, hash_node);
+ s->dedup_metrics.ram_hash_creations++;
}
/* This load the QCowHashNode corresponding to a given cluster index into ram
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 07/14] qcow2: Count QCowHashNode removal from tree for metrics.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (5 preceding siblings ...)
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 06/14] qcow2: Count QCowHashNode creation metrics Benoît Canet
@ 2013-01-16 16:25 ` Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 08/14] qcow2: Count cluster deleted metric Benoît Canet
` (6 subsequent siblings)
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:25 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2-dedup.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
index d22e2a4..64e5a13 100644
--- a/block/qcow2-dedup.c
+++ b/block/qcow2-dedup.c
@@ -841,6 +841,7 @@ static void qcow2_remove_hash_node(BlockDriverState *bs,
BDRVQcowState *s = bs->opaque;
g_tree_remove(s->dedup_tree_by_sect, &hash_node->physical_sect);
g_tree_remove(s->dedup_tree_by_hash, &hash_node->hash);
+ s->dedup_metrics.ram_hash_deletions++;
}
/* This function removes a hash_node from the trees given a physical sector
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 08/14] qcow2: Count cluster deleted metric
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (6 preceding siblings ...)
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 07/14] qcow2: Count QCowHashNode removal from tree for metrics Benoît Canet
@ 2013-01-16 16:25 ` Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 09/14] qcow2: Count deduplication refcount overflow metric Benoît Canet
` (5 subsequent siblings)
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:25 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2-dedup.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
index 64e5a13..4a1b184 100644
--- a/block/qcow2-dedup.c
+++ b/block/qcow2-dedup.c
@@ -991,6 +991,7 @@ void qcow2_dedup_refcount_zero_reached(BlockDriverState *bs,
/* remove from ram if present so we won't dedup with it anymore */
qcow2_remove_hash_node_by_sector(bs, physical_sect);
+ s->dedup_metrics.deleted_clusters++;
}
/* Force to use a new physical cluster and QCowHashNode when the refcount pass
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 09/14] qcow2: Count deduplication refcount overflow metric.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (7 preceding siblings ...)
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 08/14] qcow2: Count cluster deleted metric Benoît Canet
@ 2013-01-16 16:25 ` Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 10/14] qapi: Add support for deduplication infos in qapi-schema.json Benoît Canet
` (4 subsequent siblings)
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:25 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2-dedup.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
index 4a1b184..db23b71 100644
--- a/block/qcow2-dedup.c
+++ b/block/qcow2-dedup.c
@@ -1024,6 +1024,7 @@ void qcow2_dedup_refcount_half_max_reached(BlockDriverState *bs,
/* remove the QCowHashNode from ram so we won't use it anymore for dedup */
qcow2_remove_hash_node(bs, hash_node);
+ s->dedup_metrics.refcount_overflows++;
}
bool qcow2_dedup_is_running(BlockDriverState *bs)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 10/14] qapi: Add support for deduplication infos in qapi-schema.json.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (8 preceding siblings ...)
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 09/14] qcow2: Count deduplication refcount overflow metric Benoît Canet
@ 2013-01-16 16:25 ` Benoît Canet
2013-01-16 19:35 ` Eric Blake
2013-01-17 12:15 ` Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 11/14] block: Add deduplication metrics to BlockDriverInfo Benoît Canet
` (3 subsequent siblings)
13 siblings, 2 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:25 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
qapi-schema.json | 40 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 39 insertions(+), 1 deletion(-)
diff --git a/qapi-schema.json b/qapi-schema.json
index 5dfa052..1a5014c 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -720,6 +720,40 @@
{ 'command': 'query-block', 'returns': ['BlockInfo'] }
##
+# @BlockDeviceDedupInfo
+#
+# Statistics of the deduplication on a virtual block device implementing it
+# since QEMU startup.
+#
+# @deduplicated-clusters: Number of clusters which where deduplicated.
+#
+# @non-deduplicated-clusters: Number of clusters which where not deduplicated.
+#
+# @missing-data-reads: Number of reads which where done to complete
+# unaligned or sub cluster sized writes.
+#
+# @ram-hash-creations: Number of cluster hash created in RAM.
+#
+# @ram-hash-deletions: Number of cluster hash deleted in RAM.
+#
+# @ram-usage: Number of bytes of RAM used.
+#
+# @deleted-clusters: Number of deleted cluster when refcount < 0
+#
+# @refcount-overflows: Number of refcount overflows
+#
+# @running: True if deduplication is running
+#
+# Since: 1.5.0
+##
+{ 'type': 'BlockDeviceDedupInfo',
+ 'data': {'deduplicated-clusters': 'int', 'non-deduplicated-clusters': 'int',
+ 'missing-data-reads': 'int', 'ram-hash-creations': 'int',
+ 'ram-hash-deletions': 'int', 'ram-usage': 'int',
+ 'deleted-clusters': 'int', 'refcount-overflows': 'int',
+ 'running': 'bool' } }
+
+##
# @BlockDeviceStats:
#
# Statistics of a virtual block device or a block backing device.
@@ -747,13 +781,17 @@
# growable sparse files (like qcow2) that are used on top
# of a physical device.
#
+# @deduplication: #optional @BlockDeviceDedupInfo describing deduplication
+# metrics (since 1.5)
+#
# Since: 0.14.0
##
{ 'type': 'BlockDeviceStats',
'data': {'rd_bytes': 'int', 'wr_bytes': 'int', 'rd_operations': 'int',
'wr_operations': 'int', 'flush_operations': 'int',
'flush_total_time_ns': 'int', 'wr_total_time_ns': 'int',
- 'rd_total_time_ns': 'int', 'wr_highest_offset': 'int' } }
+ 'rd_total_time_ns': 'int', 'wr_highest_offset': 'int',
+ '*deduplication': 'BlockDeviceDedupInfo' } }
##
# @BlockStats:
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 11/14] block: Add deduplication metrics to BlockDriverInfo.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (9 preceding siblings ...)
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 10/14] qapi: Add support for deduplication infos in qapi-schema.json Benoît Canet
@ 2013-01-16 16:25 ` Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 12/14] qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage Benoît Canet
` (2 subsequent siblings)
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:25 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
include/block/block.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/include/block/block.h b/include/block/block.h
index 16e1cf1..2043560 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -29,6 +29,9 @@ typedef struct BlockDriverInfo {
/* offset at which the VM state can be saved (0 if not possible) */
int64_t vm_state_offset;
bool is_dirty;
+ bool has_dedup;
+ bool dedup_running;
+ BlockDeduplicationMetrics dedup_metrics;
} BlockDriverInfo;
typedef struct BlockFragInfo {
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 12/14] qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage.
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (10 preceding siblings ...)
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 11/14] block: Add deduplication metrics to BlockDriverInfo Benoît Canet
@ 2013-01-16 16:25 ` Benoît Canet
2013-01-16 20:10 ` Eric Blake
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 13/14] qcow2: returns deduplication metrics and status via bdrv_get_info() Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 14/14] qapi: Return virtual block device deduplication metrics in QMP Benoît Canet
13 siblings, 1 reply; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:25 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2-dedup.c | 13 +++++++++++++
block/qcow2.h | 1 +
2 files changed, 14 insertions(+)
diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
index db23b71..4305746 100644
--- a/block/qcow2-dedup.c
+++ b/block/qcow2-dedup.c
@@ -1311,3 +1311,16 @@ void qcow2_dedup_close(BlockDriverState *bs)
{
qcow2_dedup_free(bs);
}
+
+#define GTREE_NODE_SIZE sizeof(int) * 5
+
+void qcow2_dedup_update_metrics(BlockDriverState *bs)
+{
+ BDRVQcowState *s = bs->opaque;
+
+ uint64_t nb_hashs = s->dedup_metrics.ram_hash_creations -
+ s->dedup_metrics.ram_hash_deletions;
+
+ s->dedup_metrics.ram_usage = nb_hashs * GTREE_NODE_SIZE * 2;
+ s->dedup_metrics.ram_usage += nb_hashs * sizeof(QCowHashNode);
+}
diff --git a/block/qcow2.h b/block/qcow2.h
index 0729ff2..d8e8539 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -510,5 +510,6 @@ void qcow2_dedup_refcount_half_max_reached(BlockDriverState *bs,
bool qcow2_dedup_is_running(BlockDriverState *bs);
int qcow2_dedup_init(BlockDriverState *bs);
void qcow2_dedup_close(BlockDriverState *bs);
+void qcow2_dedup_update_metrics(BlockDriverState *bs);
#endif
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 13/14] qcow2: returns deduplication metrics and status via bdrv_get_info()
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (11 preceding siblings ...)
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 12/14] qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage Benoît Canet
@ 2013-01-16 16:25 ` Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 14/14] qapi: Return virtual block device deduplication metrics in QMP Benoît Canet
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:25 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block/qcow2.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/block/qcow2.c b/block/qcow2.c
index 753fce0..e442268 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1868,6 +1868,10 @@ static int qcow2_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)
BDRVQcowState *s = bs->opaque;
bdi->cluster_size = s->cluster_size;
bdi->vm_state_offset = qcow2_vm_state_offset(s);
+ bdi->has_dedup = s->has_dedup;
+ bdi->dedup_running = s->dedup_status == QCOW_DEDUP_STARTED;
+ qcow2_dedup_update_metrics(bs);
+ memcpy(&bdi->dedup_metrics, &s->dedup_metrics, sizeof(bdi->dedup_metrics));
return 0;
}
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Qemu-devel] [RFC V1 14/14] qapi: Return virtual block device deduplication metrics in QMP
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
` (12 preceding siblings ...)
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 13/14] qcow2: returns deduplication metrics and status via bdrv_get_info() Benoît Canet
@ 2013-01-16 16:25 ` Benoît Canet
13 siblings, 0 replies; 21+ messages in thread
From: Benoît Canet @ 2013-01-16 16:25 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, Benoît Canet, stefanha
---
block.c | 36 ++++++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/block.c b/block.c
index 4e28c55..a245653 100644
--- a/block.c
+++ b/block.c
@@ -2921,6 +2921,40 @@ BlockInfoList *qmp_query_block(Error **errp)
return head;
}
+static void bdrv_get_dedup_metrics(const BlockDriverState *bs,
+ BlockDeviceStats *stats)
+{
+ BlockDriverInfo bdi;
+
+ if (bdrv_get_info((BlockDriverState *) bs, &bdi) < 0) {
+ return;
+ }
+
+ if (!bdi.has_dedup) {
+ return;
+ }
+
+ stats->has_deduplication = true;
+ stats->deduplication = g_malloc0(sizeof(*stats->deduplication));
+ stats->deduplication->deduplicated_clusters =
+ bdi.dedup_metrics.deduplicated_clusters;
+ stats->deduplication->non_deduplicated_clusters =
+ bdi.dedup_metrics.non_deduplicated_clusters;
+ stats->deduplication->missing_data_reads =
+ bdi.dedup_metrics.missing_data_reads;
+ stats->deduplication->ram_hash_creations =
+ bdi.dedup_metrics.ram_hash_creations;
+ stats->deduplication->ram_hash_deletions =
+ bdi.dedup_metrics.ram_hash_deletions;
+ stats->deduplication->ram_usage =
+ bdi.dedup_metrics.ram_usage;
+ stats->deduplication->deleted_clusters =
+ bdi.dedup_metrics.deleted_clusters;
+ stats->deduplication->refcount_overflows =
+ bdi.dedup_metrics.refcount_overflows;
+ stats->deduplication->running = bdi.dedup_running;
+}
+
BlockStats *bdrv_query_stats(const BlockDriverState *bs)
{
BlockStats *s;
@@ -2943,6 +2977,8 @@ BlockStats *bdrv_query_stats(const BlockDriverState *bs)
s->stats->rd_total_time_ns = bs->total_time_ns[BDRV_ACCT_READ];
s->stats->flush_total_time_ns = bs->total_time_ns[BDRV_ACCT_FLUSH];
+ bdrv_get_dedup_metrics(bs, s->stats);
+
if (bs->file) {
s->has_parent = true;
s->parent = bdrv_query_stats(bs->file);
--
1.7.10.4
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] [RFC V1 10/14] qapi: Add support for deduplication infos in qapi-schema.json.
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 10/14] qapi: Add support for deduplication infos in qapi-schema.json Benoît Canet
@ 2013-01-16 19:35 ` Eric Blake
2013-01-17 12:15 ` Benoît Canet
1 sibling, 0 replies; 21+ messages in thread
From: Eric Blake @ 2013-01-16 19:35 UTC (permalink / raw)
To: Benoît Canet; +Cc: kwolf, pbonzini, qemu-devel, stefanha
[-- Attachment #1: Type: text/plain, Size: 469 bytes --]
On 01/16/2013 09:25 AM, Benoît Canet wrote:
> ---
> qapi-schema.json | 40 +++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 39 insertions(+), 1 deletion(-)
>
> +#
> +# @deleted-clusters: Number of deleted cluster when refcount < 0
Is it really when refcount goes negative, or when it reaches zero?
Overall it seems nice.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] [RFC V1 12/14] qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage.
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 12/14] qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage Benoît Canet
@ 2013-01-16 20:10 ` Eric Blake
2013-01-17 11:11 ` Benoît Canet
0 siblings, 1 reply; 21+ messages in thread
From: Eric Blake @ 2013-01-16 20:10 UTC (permalink / raw)
To: Benoît Canet; +Cc: kwolf, pbonzini, qemu-devel, stefanha
[-- Attachment #1: Type: text/plain, Size: 1172 bytes --]
On 01/16/2013 09:25 AM, Benoît Canet wrote:
> ---
> block/qcow2-dedup.c | 13 +++++++++++++
> block/qcow2.h | 1 +
> 2 files changed, 14 insertions(+)
>
> diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
> index db23b71..4305746 100644
> --- a/block/qcow2-dedup.c
> +++ b/block/qcow2-dedup.c
> @@ -1311,3 +1311,16 @@ void qcow2_dedup_close(BlockDriverState *bs)
> {
> qcow2_dedup_free(bs);
> }
> +
> +#define GTREE_NODE_SIZE sizeof(int) * 5
Improperly parenthesized. Also, this feels like a magic number, is
there an actual sizeof(struct) you could use instead of hand-computing
how much is used per node?
> +
> +void qcow2_dedup_update_metrics(BlockDriverState *bs)
> +{
> + BDRVQcowState *s = bs->opaque;
> +
> + uint64_t nb_hashs = s->dedup_metrics.ram_hash_creations -
> + s->dedup_metrics.ram_hash_deletions;
> +
> + s->dedup_metrics.ram_usage = nb_hashs * GTREE_NODE_SIZE * 2;
But you got lucky that order of operations didn't care about the missing
() here.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] [RFC V1 12/14] qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage.
2013-01-16 20:10 ` Eric Blake
@ 2013-01-17 11:11 ` Benoît Canet
2013-01-17 15:40 ` Eric Blake
0 siblings, 1 reply; 21+ messages in thread
From: Benoît Canet @ 2013-01-17 11:11 UTC (permalink / raw)
To: Eric Blake; +Cc: kwolf, pbonzini, qemu-devel, stefanha
Le Wednesday 16 Jan 2013 à 13:10:12 (-0700), Eric Blake a écrit :
> On 01/16/2013 09:25 AM, Benoît Canet wrote:
> > ---
> > block/qcow2-dedup.c | 13 +++++++++++++
> > block/qcow2.h | 1 +
> > 2 files changed, 14 insertions(+)
> >
> > diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
> > index db23b71..4305746 100644
> > --- a/block/qcow2-dedup.c
> > +++ b/block/qcow2-dedup.c
> > @@ -1311,3 +1311,16 @@ void qcow2_dedup_close(BlockDriverState *bs)
> > {
> > qcow2_dedup_free(bs);
> > }
> > +
> > +#define GTREE_NODE_SIZE sizeof(int) * 5
>
> Improperly parenthesized. Also, this feels like a magic number, is
> there an actual sizeof(struct) you could use instead of hand-computing
> how much is used per node?
No the glib implementation totally hide it's structures.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] [RFC V1 10/14] qapi: Add support for deduplication infos in qapi-schema.json.
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 10/14] qapi: Add support for deduplication infos in qapi-schema.json Benoît Canet
2013-01-16 19:35 ` Eric Blake
@ 2013-01-17 12:15 ` Benoît Canet
2013-01-17 15:38 ` Eric Blake
1 sibling, 1 reply; 21+ messages in thread
From: Benoît Canet @ 2013-01-17 12:15 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, pbonzini, stefanha
> +# @running: True if deduplication is running
Internally QCOW2 deduplication state switch between STOPPED, STARTING, RUNNING
and STOPPING.
Should the running field be a status field reflecting all theses states in a
string for example ?
Or is the boolean ok ?
Regards
Benoît
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] [RFC V1 10/14] qapi: Add support for deduplication infos in qapi-schema.json.
2013-01-17 12:15 ` Benoît Canet
@ 2013-01-17 15:38 ` Eric Blake
0 siblings, 0 replies; 21+ messages in thread
From: Eric Blake @ 2013-01-17 15:38 UTC (permalink / raw)
To: Benoît Canet; +Cc: kwolf, pbonzini, qemu-devel, stefanha
[-- Attachment #1: Type: text/plain, Size: 544 bytes --]
On 01/17/2013 05:15 AM, Benoît Canet wrote:
>> +# @running: True if deduplication is running
>
> Internally QCOW2 deduplication state switch between STOPPED, STARTING, RUNNING
> and STOPPING.
>
> Should the running field be a status field reflecting all theses states in a
> string for example ?
> Or is the boolean ok ?
Outputting an enum of all four states may be the best for debugging
purposes.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Qemu-devel] [RFC V1 12/14] qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage.
2013-01-17 11:11 ` Benoît Canet
@ 2013-01-17 15:40 ` Eric Blake
0 siblings, 0 replies; 21+ messages in thread
From: Eric Blake @ 2013-01-17 15:40 UTC (permalink / raw)
To: Benoît Canet; +Cc: kwolf, pbonzini, qemu-devel, stefanha
[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]
On 01/17/2013 04:11 AM, Benoît Canet wrote:
> Le Wednesday 16 Jan 2013 à 13:10:12 (-0700), Eric Blake a écrit :
>> On 01/16/2013 09:25 AM, Benoît Canet wrote:
>>> ---
>>> block/qcow2-dedup.c | 13 +++++++++++++
>>> block/qcow2.h | 1 +
>>> 2 files changed, 14 insertions(+)
>>>
>>> diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c
>>> index db23b71..4305746 100644
>>> --- a/block/qcow2-dedup.c
>>> +++ b/block/qcow2-dedup.c
>>> @@ -1311,3 +1311,16 @@ void qcow2_dedup_close(BlockDriverState *bs)
>>> {
>>> qcow2_dedup_free(bs);
>>> }
>>> +
>>> +#define GTREE_NODE_SIZE sizeof(int) * 5
>>
>> Improperly parenthesized. Also, this feels like a magic number, is
>> there an actual sizeof(struct) you could use instead of hand-computing
>> how much is used per node?
>
> No the glib implementation totally hide it's structures.
Also, are you sure that this value is correct on both 32-bit and 64-bit
machines, or does a gtree node include a void* which changes how much
memory it uses based on platform? Even adding a comment pointing to the
internal glib implementation that you were copying from would be
helpful, so that it doesn't feel quite so magic.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2013-01-17 15:40 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-01-16 16:24 [Qemu-devel] [RFC V1 00/14] QCOW2 deduplication metrics Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 01/14] qcow2: Add deduplication metrics structures Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 02/14] qcow2: Initialize deduplication metrics Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 03/14] qcow2: Collect unaligned writes missing data reads metric Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 04/14] qcow2: Collect deduplicated cluster metric Benoît Canet
2013-01-16 16:24 ` [Qemu-devel] [RFC V1 05/14] qcow2: Collect undeduplicated " Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 06/14] qcow2: Count QCowHashNode creation metrics Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 07/14] qcow2: Count QCowHashNode removal from tree for metrics Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 08/14] qcow2: Count cluster deleted metric Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 09/14] qcow2: Count deduplication refcount overflow metric Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 10/14] qapi: Add support for deduplication infos in qapi-schema.json Benoît Canet
2013-01-16 19:35 ` Eric Blake
2013-01-17 12:15 ` Benoît Canet
2013-01-17 15:38 ` Eric Blake
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 11/14] block: Add deduplication metrics to BlockDriverInfo Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 12/14] qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage Benoît Canet
2013-01-16 20:10 ` Eric Blake
2013-01-17 11:11 ` Benoît Canet
2013-01-17 15:40 ` Eric Blake
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 13/14] qcow2: returns deduplication metrics and status via bdrv_get_info() Benoît Canet
2013-01-16 16:25 ` [Qemu-devel] [RFC V1 14/14] qapi: Return virtual block device deduplication metrics in QMP Benoît Canet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).