Re: [PATCH][RESEND v5 3/3] Add a Hyper-V Dynamic Memory Protocol driver (hv-balloon)

[David Hildenbrand <david@redhat.com>]

On 22.06.23 20:45, Maciej S. Szmigiero wrote:
> On 22.06.2023 14:52, David Hildenbrand wrote:
>> On 22.06.23 14:14, Maciej S. Szmigiero wrote:
>>> On 22.06.2023 14:06, David Hildenbrand wrote:
>>>> On 22.06.23 13:17, Maciej S. Szmigiero wrote:
>>>>> On 22.06.2023 13:15, David Hildenbrand wrote:
>>>>>> On 22.06.23 13:12, Maciej S. Szmigiero wrote:
>>>>>>> On 22.06.2023 13:01, David Hildenbrand wrote:
>>>>>>>> [...]
>>>>>>>>
>>>>>>>>>>>> We'd use a memory region container as device memory region (like [1]) and would have to handle the !memdev case (I can help with that). > Into that, you can map the RAM memory region on demand (and eventually even using multiple slots like [1]).
>>>>>>>>>>>>
>>>>>>>>>>>> (2) Use a single virtual DIMM and (un)plug that on demand. Let the machine code handle (un)plugging of the device.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> (1) feels cleanest to me, although it will require a bit more work.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I also think approach (1) makes more sense as it avoids memslot metadata
>>>>>>>>>>> overhead for not-yet-hot-added parts of the memory backing device.
>>>>>>>>>>>
>>>>>>>>>>> Not sure what you mean that the !memdev case would be problematic in this
>>>>>>>>>>> case - it is working in the current driver shape so why would adding
>>>>>>>>>>> potential memory subregions (used in the memdev case) change that?
>>>>>>>>>>
>>>>>>>>>> I'm thinking about the case where you have a hv-balloon device without a memdev.
>>>>>>>>>>
>>>>>>>>>> Without -m X,maxmem=y we don't currently expect to have memory devices around
>>>>>>>>>> (and especially them getting (un)plugged. But why should we "force" to set the
>>>>>>>>>> "maxmem" option
>>>>>>>>>
>>>>>>>>> I guess it's only a small change to QEMU to allow having hv-balloon
>>>>>>>>> device (without a memdev) even in the case where there's no "maxmem"
>>>>>>>>> option given on the QEMU command line.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I hope I'll find some time soonish to prototype what I have in mind, to see
>>>>>>>>>> if it could be made working.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Okay, so I'll wait for your prototype before commencing further work on
>>>>>>>>> the next version of this driver.
>>>>>>>>
>>>>>>>> About to have something simplistic running -- I think. Want to test with a Linux VM, but I don't seem to get it working (also without my changes).
>>>>>>>>
>>>>>>>>
>>>>>>>> #!/bin/bash
>>>>>>>>
>>>>>>>> build/qemu-system-x86_64 \
>>>>>>>>          --enable-kvm \
>>>>>>>>          -m 4G,maxmem=36G \
>>>>>>>>          -cpu host,hv-syndbg=on,hv-synic,hv-relaxed,hv-vpindex \
>>>>>>>>          -smp 16 \
>>>>>>>>          -nographic \
>>>>>>>>          -nodefaults \
>>>>>>>>          -net nic -net user \
>>>>>>>>          -chardev stdio,nosignal,id=serial \
>>>>>>>>          -hda Fedora-Cloud-Base-37-1.7.x86_64.qcow2 \
>>>>>>>>          -cdrom /home/dhildenb/git/cloud-init/cloud-init.iso \
>>>>>>>>          -device isa-serial,chardev=serial \
>>>>>>>>          -chardev socket,id=monitor,path=/var/tmp/mon_src,server,nowait \
>>>>>>>>          -mon chardev=monitor,mode=readline \
>>>>>>>>          -device vmbus-bridge \
>>>>>>>>          -object memory-backend-ram,size=2G,id=mem0 \
>>>>>>>>          -device hv-balloon,id=hv1,memdev=mem0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [root@vm-0 ~]# uname -r
>>>>>>>> 6.3.5-100.fc37.x86_64
>>>>>>>> [root@vm-0 ~]# modprobe hv_balloon
>>>>>>>> modprobe: ERROR: could not insert 'hv_balloon': No such device
>>>>>>>>
>>>>>>>>
>>>>>>>> Any magic flag I am missing? Or is there something preventing this to work with Linux VMs?
>>>>>>>>
>>>>>>>
>>>>>>> Haven't tested the driver with Linux guests in a long time (as it is
>>>>>>> targeting Windows), but I think you need to disable KVM PV interface for
>>>>>>> the Hyper-V one to be detected by Linux.
>>>>>>>
>>>>>>> Something like adding "kvm=off" to "-cpu" and seeing in the dmesg whether
>>>>>>> the detected hypervisor is now Hyper-V.
>>>>>>>
>>>>>>> Also, you need to disable S4 in the guest for hot-add capability to work
>>>>>>> (I'm adding "-global ICH9-LPC.disable_s4=1" with q35 machine for this).
>>>>>>>
>>>>>>> Would also suggest adding "--trace 'hv_balloon_*' --trace 'memory_device_*'"
>>>>>>> to QEMU command line to see what's happening.
>>>>>>
>>>>>> VM is not happy:
>>>>>>
>>>>>> [    1.908595] BUG: kernel NULL pointer dereference, address: 0000000000000007
>>>>>> [    1.908837] #PF: supervisor read access in kernel mode
>>>>>> [    1.908837] #PF: error_code(0x0000) - not-present page
>>>>>> [    1.908837] PGD 0 P4D 0
>>>>>> [    1.908837] Oops: 0000 [#1] PREEMPT SMP NOPTI
>>>>>> [    1.908837] CPU: 13 PID: 492 Comm: (udev-worker) Not tainted 6.3.5-100.fc37.x86_64 #1
>>>>>> [    1.908837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.2-0-gea1b7a073390-p4
>>>>>> [    1.908837] RIP: 0010:acpi_ns_lookup+0x8f/0x4c0
>>>>>> [    1.908837] Code: 8b 3d f5 eb 1c 03 83 05 52 ec 1c 03 01 48 85 ff 0f 84 51 03 00 00 44 89 c3 4c 89 cb
>>>>>> [    1.908837] RSP: 0018:ffff95b680ad7950 EFLAGS: 00010286
>>>>>> [    1.908837] RAX: ffff95b680ad79e0 RBX: 0000000000000002 RCX: 0000000000000003
>>>>>> [    1.908837] RDX: 0000000000000000 RSI: ffff8a0283a3c558 RDI: ffffffffa4b376e0
>>>>>> [    1.908837] RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000000
>>>>>> [    1.908837] R10: ffff8a02811034ec R11: 0000000000000000 R12: ffffffffffffffff
>>>>>> [    1.908837] R13: ffff8a02811034e8 R14: ffff8a02811034e8 R15: 0000000000000000
>>>>>> [    1.908837] FS:  00007f3bb2e7d0c0(0000) GS:ffff8a02bbd40000(0000) knlGS:0000000000000000
>>>>>> [    1.908837] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>>>> [    1.908837] CR2: 0000000000000007 CR3: 0000000100a58002 CR4: 0000000000770ee0
>>>>>> [    1.908837] PKRU: 55555554
>>>>>> [    1.908837] Call Trace:
>>>>>> [    1.908837]  <TASK>
>>>>>> [    1.908837]  ? __die+0x23/0x70
>>>>>> [    1.908837]  ? page_fault_oops+0x171/0x4e0
>>>>>> [    1.908837]  ? prepare_alloc_pages.constprop.0+0xf6/0x1a0
>>>>>> [    1.908837]  ? exc_page_fault+0x74/0x170
>>>>>> [    1.908837]  ? asm_exc_page_fault+0x26/0x30
>>>>>> [    1.908837]  ? acpi_ns_lookup+0x8f/0x4c0
>>>>>> [    1.908837]  acpi_ns_get_node_unlocked+0xdd/0x110
>>>>>> [    1.908837]  ? down_timeout+0x3e/0x60
>>>>>> [    1.908837]  ? acpi_ns_get_node+0x3e/0x60
>>>>>> [    1.908837]  acpi_ns_get_node+0x3e/0x60
>>>>>> [    1.908837]  acpi_ns_evaluate+0x1cb/0x2d0
>>>>>> [    1.908837]  acpi_ut_evaluate_object+0x68/0x1c0
>>>>>> [    1.908837]  acpi_rs_get_method_data+0x37/0x80
>>>>>> [    1.908837]  ? __pfx_vmbus_walk_resources+0x10/0x10 [hv_vmbus]
>>>>>> [    1.908837]  acpi_walk_resources+0x91/0xe0
>>>>>> [    1.908837]  vmbus_acpi_add+0x87/0x170 [hv_vmbus]
>>>>>> [    1.908837]  acpi_device_probe+0x47/0x160
>>>>>> [    1.908837]  really_probe+0x19f/0x400
>>>>>> [    1.908837]  ? __pfx___driver_attach+0x10/0x10
>>>>>> [    1.908837]  __driver_probe_device+0x78/0x160
>>>>>> [    1.908837]  driver_probe_device+0x1f/0x90
>>>>>> [    1.908837]  __driver_attach+0xd2/0x1c0
>>>>>> [    1.908837]  bus_for_each_dev+0x85/0xd0
>>>>>> [    1.908837]  bus_add_driver+0x116/0x220
>>>>>> [    1.908837]  driver_register+0x59/0x100
>>>>>> [    1.908837]  ? __pfx_init_module+0x10/0x10 [hv_vmbus]
>>>>>> [    1.908837]  hv_acpi_init+0x39/0xff0 [hv_vmbus]
>>>>>> [    1.908837]  ? __pfx_init_module+0x10/0x10 [hv_vmbus]
>>>>>> [    1.908837]  do_one_initcall+0x5a/0x240
>>>>>> [    1.908837]  do_init_module+0x4a/0x210
>>>>>> [    1.908837]  __do_sys_init_module+0x17f/0x1b0
>>>>>> [    1.908837]  do_syscall_64+0x5c/0x90
>>>>>> [    1.908837]  ? handle_mm_fault+0x11e/0x310
>>>>>> [    1.908837]  ? do_user_addr_fault+0x1e0/0x720
>>>>>> [    1.908837]  ? exc_page_fault+0x74/0x170
>>>>>> [    1.908837]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>>>>>>
>>>>>
>>>>> I guess *few* people run Linux with QEMU Hyper-V interfaces
>>>>> implementation..
>>>>>> Maybe I'll have to install a Windows guest :/
>>>>>>
>>>>> I think that makes more sense, since we're targeting Windows anyway.
>>>>>
>>>>
>>>> Having installed fairly recent Win10 and running with master+your patches, I still can't get it to work.
>>>>
>>>> Windows is stuck booting (before the little circle starts turning).
>>>>
>>>> Removing the hv-balloon device makes it work again (well, at least the circle spins again my windows installation now seems to be broken and I have to reinstall ... windows).
>>>>
>>>> Do you have a working cmdline for Windows I can try?
>>>>
>>>
>>> Do you get any tracing output from the driver when it is loaded or does its mere presence
>>> make your Windows installation fail to boot?
>>>
>>> I'm was testing using "qemu-system-x86_64 -machine q35,accel=kvm \
>>>              -cpu host,vmx,-mpx,-pmu,check,hv-crash,hv-relaxed,hv-time,hv-vapic,hv-spinlocks=0x1fff,hv-
>>
>> ^ That magic collection of flags seems to make Win10 happy.
> 
> I will try to determine the minimum set of Hyper-V flags required
> for Windows guests since having to manually specify the "right"
> set seems like a poor UX (and also try to determine why Linux
> guest crashes in ACPI when booted with above flags).
> 
> But I think both of these problems are related more to VMBus support
> itself rather than hv-balloon.

That's also my feeling. It would be great to extend

https://www.qemu.org/docs/master/system/i386/hyperv.html

and include some of the dirty details in you next submission, so it's 
easier to get running.

[...]

>> Maybe the VM needs some time after start-up to actually
>> process requests.
> 
> Yes, there's a waiting period immediately after the guest
> boot before executing any protocol operations in the guest:
>> /*
>>   * Some Windows versions (at least Server 2019) will crash with various
>>   * error codes when receiving DM protocol requests (at least
>>   * DM_MEM_HOT_ADD_REQUEST) immediately after boot.
>>   *
>>   * It looks like Hyper-V from Server 2016 uses a 50-second after-boot
>>   * delay, probably to workaround this issue, so we'll use this value, too.
>>   */
>> #define HV_BALLOON_POST_INIT_WAIT (50 * 1000)
> 
> 
> That's why I suggested running with tracing on, since then
> you can observe the device waiting in the "POST_INIT_WAIT" state
> (and also how things are proceeding in general).

Ah, makes sense.

-- 
Cheers,

David / dhildenb