Re: [V2 PATCH 4/6] KVM: selftests: x86: Add helpers to execute VMs with private memory

[Sean Christopherson <seanjc@google.com>]

On Mon, Dec 05, 2022, Vishal Annapurve wrote:
> Introduce a set of APIs to execute VM with private memslots.
> 
> Host userspace APIs for:
> 1) Executing a vcpu run loop that handles MAPGPA hypercall
> 2) Backing/unbacking guest private memory
> 
> Guest APIs for:
> 1) Changing memory mapping type
> 
> Signed-off-by: Vishal Annapurve <vannapurve@google.com>
> ---
>  tools/testing/selftests/kvm/Makefile          |   1 +
>  .../kvm/include/x86_64/private_mem.h          |  24 +++
>  .../selftests/kvm/lib/x86_64/private_mem.c    | 139 ++++++++++++++++++
>  3 files changed, 164 insertions(+)
>  create mode 100644 tools/testing/selftests/kvm/include/x86_64/private_mem.h
>  create mode 100644 tools/testing/selftests/kvm/lib/x86_64/private_mem.c

Given that we _know_ private memory isn't always going to x86 specific, I don't
want to bury any helpers in x86_64 that aren't strictly x86 only.  E.g. helpers
for doing Intel+AMD hypercalls is ok, but "generic" private memory helpers belong
elsewhere.

I experimented with extracting memslots/mmu helpers out of kvm_util.c as prep work,
e.g. to avoid bloating kvm_util.c, but I couldn't come up with an obviously "good"
naming scheme and/or split.  At this time, the common bits are fairly small, so
I think the best approach for now is to simply put vm_mem_map_shared_or_private()
in kvm_util.c.

> +static inline uint64_t __kvm_hypercall_map_gpa_range(uint64_t gpa, uint64_t size,
> +	uint64_t flags)
> +{
> +	return kvm_hypercall(KVM_HC_MAP_GPA_RANGE, gpa, size >> PAGE_SHIFT, flags, 0);
> +}

This can go in tools/testing/selftests/kvm/include/x86_64/processor.h.

> +static inline void kvm_hypercall_map_gpa_range(uint64_t gpa, uint64_t size,
> +	uint64_t flags)
> +{
> +	uint64_t ret;
> +
> +	GUEST_ASSERT_2(IS_PAGE_ALIGNED(gpa) && IS_PAGE_ALIGNED(size), gpa, size);
> +
> +	ret = __kvm_hypercall_map_gpa_range(gpa, size, flags);
> +	GUEST_ASSERT_1(!ret, ret);
> +}
> +
> +void kvm_hypercall_map_shared(uint64_t gpa, uint64_t size)
> +{
> +	kvm_hypercall_map_gpa_range(gpa, size, KVM_MAP_GPA_RANGE_DECRYPTED);
> +}
> +
> +void kvm_hypercall_map_private(uint64_t gpa, uint64_t size)
> +{
> +	kvm_hypercall_map_gpa_range(gpa, size, KVM_MAP_GPA_RANGE_ENCRYPTED);
> +}
> +
> +static void vm_update_private_mem(struct kvm_vm *vm, uint64_t gpa, uint64_t size,
> +	bool unback_mem)

s/unback_mem/map_shared.  "unback memory" is going to be super confusing to someone
who isn't familiar with UPM.  map_private would be the obvious alternative, but
I like not having to invert the param in the helper.

> +{
> +	int restricted_fd;
> +	uint64_t restricted_fd_offset, guest_phys_base, fd_offset;
> +	struct kvm_memory_attributes attr;
> +	struct kvm_userspace_memory_region_ext *region_ext;
> +	struct kvm_userspace_memory_region *region;
> +	int fallocate_mode = 0;
> +	int ret;
> +
> +	region_ext = kvm_userspace_memory_region_ext_find(vm, gpa, gpa + size);

I forget if I've already mentioned this somewhere, but I'd prefer to use the
"private" userspace_mem_region_find() and delete the existing
kvm_userspace_memory_region_find().

> +	TEST_ASSERT(region_ext != NULL, "Region not found");
> +	region = &region_ext->region;
> +	TEST_ASSERT(region->flags & KVM_MEM_PRIVATE,
> +		"Can not update private memfd for non-private memslot\n");
> +	restricted_fd = region_ext->restricted_fd;
> +	restricted_fd_offset = region_ext->restricted_offset;
> +	guest_phys_base = region->guest_phys_addr;
> +	fd_offset = restricted_fd_offset + (gpa - guest_phys_base);
> +
> +	if (unback_mem)
> +		fallocate_mode = (FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE);
> +
> +	printf("restricted_fd %d fallocate_mode 0x%x for offset 0x%lx size 0x%lx\n",
> +		restricted_fd, fallocate_mode, fd_offset, size);

Don't put prints in common helpers, except _maybe_ for error paths.  It's fine
for development and/or debug, but for the final product it ends up being noise 99%
of the time.  If you really, really want printing checked in, then pr_debug() is an
option, but I would generally discourage even that for selftests.  E.g. strace can
give you all the information printed here without needing to rebuild the binary,
and without maintenance burden.

> +	ret = fallocate(restricted_fd, fallocate_mode, fd_offset, size);
> +	TEST_ASSERT(ret == 0, "fallocate failed\n");

Use whitespace to differntiate operations/blocks.

> +	attr.attributes = unback_mem ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE;
> +	attr.address = gpa;
> +	attr.size = size;
> +	attr.flags = 0;

Add a helper to do KVM_SET_MEMORY_ATTRIBUTES, e.g. to fill the appropriate struct.

> +	if (unback_mem)
> +		printf("undoing encryption for gpa 0x%lx size 0x%lx\n", gpa, size);
> +	else
> +		printf("doing encryption for gpa 0x%lx size 0x%lx\n", gpa, size);
> +
> +	vm_ioctl(vm, KVM_SET_MEMORY_ATTRIBUTES, &attr);
> +}


void vm_mem_map_shared_or_private(struct kvm_vm *vm, uint64_t gpa,
				  uint64_t size, bool map_shared)
{
	struct userspace_mem_region *region;
	uint64_t end = gpa + size - 1;
	off_t fd_offset;
	int mode, ret;

	region = userspace_mem_region_find(vm, gpa, gpa);
	TEST_ASSERT(region && region->region.flags & KVM_MEM_PRIVATE,
		    "Private memory region not found for GPA 0x%lx", gpa);

	TEST_ASSERT(region == userspace_mem_region_find(vm, end, end),
		    "Private/Shared conversions must act on a single memslot");

	fd_offset = region->region.restricted_offset +
		    (gpa - region->region.guest_phys_addr);

	/* To map shared, punch a hole.  To map private, allocate (no flags). */
	mode = map_shared ? (FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE) : 0;

	ret = fallocate(region->region.restricted_fd, mode, fd_offset, size);
	TEST_ASSERT(!ret, "fallocate() failed to map %lx[%lu] %s, fd = %d, mode = %x, offset = %lx\n",
		     gpa, size, map_shared ? "shared" : "private",
		     region->region.restricted_fd, mode, fd_offset);

	vm_set_memory_attributes(vm, gpa, size,
				 map_shared ? 0 : KVM_MEMORY_ATTRIBUTE_PRIVATE);
}