* [GIT PULL 0/2] KVM: s390: Changes for 6.7
@ 2023-10-30 9:08 Janosch Frank
2023-10-30 9:08 ` [GIT PULL 1/2] KVM: s390: add stat counter for shadow gmap events Janosch Frank
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Janosch Frank @ 2023-10-30 9:08 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, nrb,
imbrenda
Paolo, please pull these two patches for 6.7.
They introduce counters and a tracepoint into our nested VM page table
management code. Debuging nested page table management is notoriously
hard but the added low-overhead debug data will allow us to get a feel
for the problem before deploying deeper and higher overhead debugging
means.
I've held back the patches for a bit since we had suspicious CI fails
but they turned out to be unrelated and have been fixed at the end of
last week.
The following changes since commit ce9ecca0238b140b88f43859b211c9fdfd8e5b70:
Linux 6.6-rc2 (2023-09-17 14:40:24 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git tags/kvm-s390-next-6.7-1
for you to fetch changes up to 70fea30195168fd84e7076720c984f0ac1af5b09:
KVM: s390: add tracepoint in gmap notifier (2023-10-16 14:54:29 +0200)
Nico Boehr (2):
KVM: s390: add stat counter for shadow gmap events
KVM: s390: add tracepoint in gmap notifier
arch/s390/include/asm/kvm_host.h | 7 +++++++
arch/s390/kvm/gaccess.c | 7 +++++++
arch/s390/kvm/kvm-s390.c | 11 ++++++++++-
arch/s390/kvm/trace-s390.h | 23 +++++++++++++++++++++++
arch/s390/kvm/vsie.c | 5 ++++-
5 files changed, 51 insertions(+), 2 deletions(-)
--
2.41.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* [GIT PULL 1/2] KVM: s390: add stat counter for shadow gmap events
2023-10-30 9:08 [GIT PULL 0/2] KVM: s390: Changes for 6.7 Janosch Frank
@ 2023-10-30 9:08 ` Janosch Frank
2023-10-30 9:08 ` [GIT PULL 2/2] KVM: s390: add tracepoint in gmap notifier Janosch Frank
2023-10-31 14:10 ` [GIT PULL 0/2] KVM: s390: Changes for 6.7 Paolo Bonzini
2 siblings, 0 replies; 4+ messages in thread
From: Janosch Frank @ 2023-10-30 9:08 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, nrb,
imbrenda
From: Nico Boehr <nrb@linux.ibm.com>
The shadow gmap tracks memory of nested guests (guest-3). In certain
scenarios, the shadow gmap needs to be rebuilt, which is a costly operation
since it involves a SIE exit into guest-1 for every entry in the respective
shadow level.
Add kvm stat counters when new shadow structures are created at various
levels. Also add a counter gmap_shadow_create when a completely fresh
shadow gmap is created as well as a counter gmap_shadow_reuse when an
existing gmap is being reused.
Note that when several levels are shadowed at once, counters on all
affected levels will be increased.
Also note that not all page table levels need to be present and a ASCE
can directly point to e.g. a segment table. In this case, a new segment
table will always be equivalent to a new shadow gmap and hence will be
counted as gmap_shadow_create and not as gmap_shadow_segment.
Signed-off-by: Nico Boehr <nrb@linux.ibm.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Link: https://lore.kernel.org/r/20231009093304.2555344-2-nrb@linux.ibm.com
Message-Id: <20231009093304.2555344-2-nrb@linux.ibm.com>
---
arch/s390/include/asm/kvm_host.h | 7 +++++++
arch/s390/kvm/gaccess.c | 7 +++++++
arch/s390/kvm/kvm-s390.c | 9 ++++++++-
arch/s390/kvm/vsie.c | 5 ++++-
4 files changed, 26 insertions(+), 2 deletions(-)
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 427f9528a7b6..67a298b6cf6e 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -777,6 +777,13 @@ struct kvm_vm_stat {
u64 inject_service_signal;
u64 inject_virtio;
u64 aen_forward;
+ u64 gmap_shadow_create;
+ u64 gmap_shadow_reuse;
+ u64 gmap_shadow_r1_entry;
+ u64 gmap_shadow_r2_entry;
+ u64 gmap_shadow_r3_entry;
+ u64 gmap_shadow_sg_entry;
+ u64 gmap_shadow_pg_entry;
};
struct kvm_arch_memory_slot {
diff --git a/arch/s390/kvm/gaccess.c b/arch/s390/kvm/gaccess.c
index 6d6bc19b37dc..ff8349d17b33 100644
--- a/arch/s390/kvm/gaccess.c
+++ b/arch/s390/kvm/gaccess.c
@@ -1382,6 +1382,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr,
unsigned long *pgt, int *dat_protection,
int *fake)
{
+ struct kvm *kvm;
struct gmap *parent;
union asce asce;
union vaddress vaddr;
@@ -1390,6 +1391,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr,
*fake = 0;
*dat_protection = 0;
+ kvm = sg->private;
parent = sg->parent;
vaddr.addr = saddr;
asce.val = sg->orig_asce;
@@ -1450,6 +1452,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr,
rc = gmap_shadow_r2t(sg, saddr, rfte.val, *fake);
if (rc)
return rc;
+ kvm->stat.gmap_shadow_r1_entry++;
}
fallthrough;
case ASCE_TYPE_REGION2: {
@@ -1478,6 +1481,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr,
rc = gmap_shadow_r3t(sg, saddr, rste.val, *fake);
if (rc)
return rc;
+ kvm->stat.gmap_shadow_r2_entry++;
}
fallthrough;
case ASCE_TYPE_REGION3: {
@@ -1515,6 +1519,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr,
rc = gmap_shadow_sgt(sg, saddr, rtte.val, *fake);
if (rc)
return rc;
+ kvm->stat.gmap_shadow_r3_entry++;
}
fallthrough;
case ASCE_TYPE_SEGMENT: {
@@ -1548,6 +1553,7 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr,
rc = gmap_shadow_pgt(sg, saddr, ste.val, *fake);
if (rc)
return rc;
+ kvm->stat.gmap_shadow_sg_entry++;
}
}
/* Return the parent address of the page table */
@@ -1618,6 +1624,7 @@ int kvm_s390_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg,
pte.p |= dat_protection;
if (!rc)
rc = gmap_shadow_page(sg, saddr, __pte(pte.val));
+ vcpu->kvm->stat.gmap_shadow_pg_entry++;
ipte_unlock(vcpu->kvm);
mmap_read_unlock(sg->mm);
return rc;
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index b3f17e014cab..b42493110d76 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -66,7 +66,14 @@ const struct _kvm_stats_desc kvm_vm_stats_desc[] = {
STATS_DESC_COUNTER(VM, inject_pfault_done),
STATS_DESC_COUNTER(VM, inject_service_signal),
STATS_DESC_COUNTER(VM, inject_virtio),
- STATS_DESC_COUNTER(VM, aen_forward)
+ STATS_DESC_COUNTER(VM, aen_forward),
+ STATS_DESC_COUNTER(VM, gmap_shadow_reuse),
+ STATS_DESC_COUNTER(VM, gmap_shadow_create),
+ STATS_DESC_COUNTER(VM, gmap_shadow_r1_entry),
+ STATS_DESC_COUNTER(VM, gmap_shadow_r2_entry),
+ STATS_DESC_COUNTER(VM, gmap_shadow_r3_entry),
+ STATS_DESC_COUNTER(VM, gmap_shadow_sg_entry),
+ STATS_DESC_COUNTER(VM, gmap_shadow_pg_entry),
};
const struct kvm_stats_header kvm_vm_stats_header = {
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 61499293c2ac..02dcbe82a8e5 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -1214,8 +1214,10 @@ static int acquire_gmap_shadow(struct kvm_vcpu *vcpu,
* we're holding has been unshadowed. If the gmap is still valid,
* we can safely reuse it.
*/
- if (vsie_page->gmap && gmap_shadow_valid(vsie_page->gmap, asce, edat))
+ if (vsie_page->gmap && gmap_shadow_valid(vsie_page->gmap, asce, edat)) {
+ vcpu->kvm->stat.gmap_shadow_reuse++;
return 0;
+ }
/* release the old shadow - if any, and mark the prefix as unmapped */
release_gmap_shadow(vsie_page);
@@ -1223,6 +1225,7 @@ static int acquire_gmap_shadow(struct kvm_vcpu *vcpu,
if (IS_ERR(gmap))
return PTR_ERR(gmap);
gmap->private = vcpu->kvm;
+ vcpu->kvm->stat.gmap_shadow_create++;
WRITE_ONCE(vsie_page->gmap, gmap);
return 0;
}
--
2.41.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [GIT PULL 2/2] KVM: s390: add tracepoint in gmap notifier
2023-10-30 9:08 [GIT PULL 0/2] KVM: s390: Changes for 6.7 Janosch Frank
2023-10-30 9:08 ` [GIT PULL 1/2] KVM: s390: add stat counter for shadow gmap events Janosch Frank
@ 2023-10-30 9:08 ` Janosch Frank
2023-10-31 14:10 ` [GIT PULL 0/2] KVM: s390: Changes for 6.7 Paolo Bonzini
2 siblings, 0 replies; 4+ messages in thread
From: Janosch Frank @ 2023-10-30 9:08 UTC (permalink / raw)
To: pbonzini
Cc: kvm, frankja, david, borntraeger, cohuck, linux-s390, nrb,
imbrenda
From: Nico Boehr <nrb@linux.ibm.com>
The gmap notifier is called for changes in table entries with the
notifier bit set. To diagnose performance issues, it can be useful to
see what causes certain changes in the gmap.
Hence, add a tracepoint in the gmap notifier.
Signed-off-by: Nico Boehr <nrb@linux.ibm.com>
Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Link: https://lore.kernel.org/r/20231009093304.2555344-3-nrb@linux.ibm.com
Message-Id: <20231009093304.2555344-3-nrb@linux.ibm.com>
---
arch/s390/kvm/kvm-s390.c | 2 ++
arch/s390/kvm/trace-s390.h | 23 +++++++++++++++++++++++
2 files changed, 25 insertions(+)
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index b42493110d76..11676b81e6bf 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4060,6 +4060,8 @@ static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start,
unsigned long prefix;
unsigned long i;
+ trace_kvm_s390_gmap_notifier(start, end, gmap_is_shadow(gmap));
+
if (gmap_is_shadow(gmap))
return;
if (start >= 1UL << 31)
diff --git a/arch/s390/kvm/trace-s390.h b/arch/s390/kvm/trace-s390.h
index 6f0209d45164..9ac92dbf680d 100644
--- a/arch/s390/kvm/trace-s390.h
+++ b/arch/s390/kvm/trace-s390.h
@@ -333,6 +333,29 @@ TRACE_EVENT(kvm_s390_airq_suppressed,
__entry->id, __entry->isc)
);
+/*
+ * Trace point for gmap notifier calls.
+ */
+TRACE_EVENT(kvm_s390_gmap_notifier,
+ TP_PROTO(unsigned long start, unsigned long end, unsigned int shadow),
+ TP_ARGS(start, end, shadow),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, start)
+ __field(unsigned long, end)
+ __field(unsigned int, shadow)
+ ),
+
+ TP_fast_assign(
+ __entry->start = start;
+ __entry->end = end;
+ __entry->shadow = shadow;
+ ),
+
+ TP_printk("gmap notified (start:0x%lx end:0x%lx shadow:%d)",
+ __entry->start, __entry->end, __entry->shadow)
+ );
+
#endif /* _TRACE_KVMS390_H */
--
2.41.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [GIT PULL 0/2] KVM: s390: Changes for 6.7
2023-10-30 9:08 [GIT PULL 0/2] KVM: s390: Changes for 6.7 Janosch Frank
2023-10-30 9:08 ` [GIT PULL 1/2] KVM: s390: add stat counter for shadow gmap events Janosch Frank
2023-10-30 9:08 ` [GIT PULL 2/2] KVM: s390: add tracepoint in gmap notifier Janosch Frank
@ 2023-10-31 14:10 ` Paolo Bonzini
2 siblings, 0 replies; 4+ messages in thread
From: Paolo Bonzini @ 2023-10-31 14:10 UTC (permalink / raw)
To: Janosch Frank; +Cc: kvm, david, borntraeger, cohuck, linux-s390, nrb, imbrenda
On Mon, Oct 30, 2023 at 10:22 AM Janosch Frank <frankja@linux.ibm.com> wrote:
>
> Paolo, please pull these two patches for 6.7.
>
> They introduce counters and a tracepoint into our nested VM page table
> management code. Debuging nested page table management is notoriously
> hard but the added low-overhead debug data will allow us to get a feel
> for the problem before deploying deeper and higher overhead debugging
> means.
>
> I've held back the patches for a bit since we had suspicious CI fails
> but they turned out to be unrelated and have been fixed at the end of
> last week.
Pulled, thanks.
Paolo
>
>
> The following changes since commit ce9ecca0238b140b88f43859b211c9fdfd8e5b70:
>
> Linux 6.6-rc2 (2023-09-17 14:40:24 -0700)
>
> are available in the Git repository at:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git tags/kvm-s390-next-6.7-1
>
> for you to fetch changes up to 70fea30195168fd84e7076720c984f0ac1af5b09:
>
> KVM: s390: add tracepoint in gmap notifier (2023-10-16 14:54:29 +0200)
>
> Nico Boehr (2):
> KVM: s390: add stat counter for shadow gmap events
> KVM: s390: add tracepoint in gmap notifier
>
> arch/s390/include/asm/kvm_host.h | 7 +++++++
> arch/s390/kvm/gaccess.c | 7 +++++++
> arch/s390/kvm/kvm-s390.c | 11 ++++++++++-
> arch/s390/kvm/trace-s390.h | 23 +++++++++++++++++++++++
> arch/s390/kvm/vsie.c | 5 ++++-
> 5 files changed, 51 insertions(+), 2 deletions(-)
>
> --
> 2.41.0
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2023-10-31 14:11 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-30 9:08 [GIT PULL 0/2] KVM: s390: Changes for 6.7 Janosch Frank
2023-10-30 9:08 ` [GIT PULL 1/2] KVM: s390: add stat counter for shadow gmap events Janosch Frank
2023-10-30 9:08 ` [GIT PULL 2/2] KVM: s390: add tracepoint in gmap notifier Janosch Frank
2023-10-31 14:10 ` [GIT PULL 0/2] KVM: s390: Changes for 6.7 Paolo Bonzini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox