Dovetail: Adding support for Hyper-v as hypervisor

Dovetail: Adding support for Hyper-v as hypervisor

[Florian Bezdeka]

Hi Philippe,

I'm currently experimenting with hyper-v as Hypervisor for dovetail. So
far so good, I'm able to boot the system and the system survives a
inband stress test. The necessary modifications can be found at [1].

But: Running the Xenomai 3 testsuite - especially the gdb test -
freezes the system immediately.

When enabling pipeline and dovetail debug options I get the following
splat report. Ideas welcome...

The following is a mix of two WARN() invocations. I'm assuming
everything around exc_invalid_op() is a follow up problem, as WARN()
triggers the invalid_op trap. Anyway, I copied it unmodified as I got
it over the UART to avoid any mistake while cleaning up.

I think it's worth to mention that there is a strange comment in
hv_qlock_wait() - which I already touched - that expresses that hyper-v
might inject an IRQ on a vcpu with hard IRQs disabled...

[    1.069032] ------------[ cut here ]------------
[    1.069035] Key type blacklist registered
[    1.070098] hard irqs on posting IRQ27 to Linux
[    1.070098] ------------[ cut here ]------------
[    1.070098] DEBUG_LOCKS_WARN_ON(running_inband() && lockdep_hardirq_context())
[    1.070098] WARNING: CPU: 0 PID: 28 at kernel/locking/lockdep.c:4457 lockdep_hardirqs_on_prepare+0x112/0x1b0
[    1.070098] Modules linked in:
[    1.070098] CPU: 0 UID: 0 PID: 28 Comm: kworker/u8:1 Tainted: G        W           6.15.0-hyprv-xenomai-0+ #49 PREEMPT(voluntary)
[    1.070098] Tainted: [W]=WARN
[    1.070098] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
[    1.070098] IRQ stage: Linux
[    1.070098] Workqueue: async async_run_entry_fn
[    1.070098] RIP: 0010:lockdep_hardirqs_on_prepare+0x112/0x1b0
[    1.070098] Code: 64 ff ff ff e8 cf 9f 5c 00 85 c0 74 ad 8b 15 35 21 bb 01 85 d2 75 a3 48 c7 c6 e0 2e 2c 93 48 c7 c7 90 32 2a 93 e8 ee 0d f6 ff <0f> 0b eb 8c be 06 00 00 00 48 89 df e8 4d fe ff ff e9 64 ff ff ff
[    1.070098] RSP: 0000:ffffb22680003ed0 EFLAGS: 00010082
[    1.070098] RAX: 0000000000000000 RBX: ffffb22680003f38 RCX: 0000000000000027
[    1.070098] RDX: ffff896c3ec1bc08 RSI: 0000000000000001 RDI: ffff896c3ec1bc00
[    1.070098] RBP: 0000000000000246 R08: 0000000000000486 R09: 0000000000000000
[    1.070098] R10: 00000000ffffffff R11: 0000000000000004 R12: 0000000000000000
[    1.070098] R13: ffffffff92184d7e R14: 0000000000000000 R15: 0000000000000000
[    1.070098] FS:  0000000000000000(0000) GS:ffff896caa667000(0000) knlGS:0000000000000000
[    1.070098] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.070098] CR2: ffff896c26202000 CR3: 000000002483a001 CR4: 00000000003706f0
[    1.070098] Call Trace:
[    1.070098]  <IRQ>
[    1.070098]  ? irq_post_stage+0x8c/0xc0
[    1.070098]  trace_hardirqs_on+0x53/0xc0
[    1.070098]  handle_bug+0x1ca/0x1e0
[    1.070098]  exc_invalid_op+0xae/0xc0
[    1.070098]  asm_exc_invalid_op+0x16/0x20
[    1.070098] RIP: 0010:irq_post_stage+0x8c/0xc0
[    1.070098] Code: 12 48 0f ab 30 c3 cc cc cc cc 80 3d 06 4e b4 01 00 75 f2 48 8b 57 08 48 c7 c7 98 81 2c 93 c6 05 f2 4d b4 01 01 e8 c4 3e f3 ff <0f> 0b c3 cc cc cc cc 80 3d de 4d b4 01 00 75 cb 48 8b 57 08 48 c7
[    1.070098] RSP: 0000:ffffb22680003fe0 EFLAGS: 00010286
[    1.070098] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000202
[    1.070098] RDX: 0000000080010002 RSI: 0000000000000000 RDI: 0000000000000001
[    1.070098] RBP: ffff896c011f3400 R08: 000000000000005b R09: 000000004b4c332f
[    1.070098] R10: 00000000ffffffff R11: 00000000ffffffff R12: 0000000000080105
[    1.070098] R13: 00000000000000ed R14: 0000000000000000 R15: 0000000000000000
[    1.070098]  __sysvec_hyperv_stimer0+0x2b/0x50
[    1.070098]  arch_do_IRQ_pipelined+0x23c/0x560
[    1.070098]  </IRQ>
[    1.070098]  <TASK>
[    1.070098]  sync_current_irq_stage+0x191/0x230
[    1.070098]  handle_irq_pipelined_finish+0x7b/0x1a0
[    1.070098]  arch_pipeline_entry+0xaa/0xf0
[    1.070098]  asm_sysvec_hyperv_stimer0+0x16/0x20
[    1.070098] RIP: 0010:ZSTD_decompressSequences_bmi2.constprop.0+0x62a/0xa80
[    1.070098] Code: 08 48 8b 58 08 48 83 c7 20 48 83 c0 20 48 89 4f e0 48 89 5f e8 48 8b 48 f0 48 8b 58 f8 48 89 4f f0 48 89 5f f8 48 39 d7 72 d4 <49> 83 fe 88 0f 87 25 fb ff ff 4d 01 f5 83 ad 34 ff ff ff 01 0f 85
[    1.070098] RSP: 0000:ffffb226800ffa10 EFLAGS: 00000283
[    1.070098] RAX: ffffb226807b4fc9 RBX: 000000000000000b RCX: ffffb226803fa41c
[    1.070098] RDX: 000000000038b951 RSI: ffffb226804296d8 RDI: ffffb226807b5029
[    1.070098] RBP: ffffb226800ffb50 R08: 040000000000003d R09: fc00000285000000
[    1.070098] R10: 0000000000000003 R11: 000000000000009f R12: 0000000000000060
[    1.070098] R13: ffffb226807b5028 R14: 000000000000000c R15: 0000000000055518
[    1.070098]  ZSTD_decompressContinue.part.0+0x359/0x490
[    1.070098]  ZSTD_decompressContinueStream+0x96/0x150
[    1.070098]  ZSTD_decompressStream+0x7c3/0xc00
[    1.070098]  ? __pfx_flush_buffer+0x10/0x10
[    1.070098]  ? __pfx_error+0x10/0x10
[    1.070098]  unzstd+0x340/0x570
[    1.070098]  ? __pfx_unzstd+0x10/0x10
[    1.070098]  unpack_to_rootfs+0x144/0x3a0
[    1.070098]  ? __pfx_error+0x10/0x10
[    1.070098]  ? _printk+0x5f/0x80
[    1.070098]  ? do_populate_rootfs+0x124/0x1d0
[    1.070098]  do_populate_rootfs+0x124/0x1d0
[    1.070098]  ? ktime_get+0x83/0x180
[    1.070098]  async_run_entry_fn+0x2d/0x130
[    1.070098]  process_one_work+0x22d/0x6d0
[    1.070098]  worker_thread+0x184/0x320
[    1.070098]  ? __pfx_worker_thread+0x10/0x10
[    1.070098]  kthread+0xfa/0x240
[    1.070098]  ? __pfx_kthread+0x10/0x10
[    1.070098]  ret_from_fork+0x2d/0x50
[    1.070098]  ? __pfx_kthread+0x10/0x10
[    1.070098]  ret_from_fork_asm+0x1a/0x30
[    1.070098]  </TASK>
[    1.070098] irq event stamp: 14192
[    1.070098] hardirqs last  enabled at (14191): [<ffffffff92d7c95a>] arch_pipeline_entry+0x8a/0xf0
[    1.070098] hardirqs last disabled at (14192): [<ffffffff92185fe1>] sync_current_irq_stage+0x201/0x230
[    1.070098] softirqs last  enabled at (12424): [<ffffffff920c3cfe>] handle_softirqs+0x33e/0x420
[    1.070098] softirqs last disabled at (12409): [<ffffffff920c3f48>] __irq_exit_rcu+0xe8/0x110
[    1.070098] ---[ end trace 0000000000000000 ]---
[    1.070098] WARNING: CPU: 0 PID: 28 at kernel/irq/pipeline.c:596 irq_post_stage+0x8c/0xc0
[    1.070098] workingset: timestamp_bits=36 max_order=18 bucket_order=0
[    1.070098] Modules linked in:
[    1.070098] integrity: Platform Keyring initialized
[    1.070098]
[    1.070098] CPU: 0 UID: 0 PID: 28 Comm: kworker/u8:1 Tainted: G        W           6.15.0-hyprv-xenomai-0+ #49 PREEMPT(voluntary)
[    1.070098] integrity: Machine keyring initialized
[    1.070098] Tainted: [W]=WARN
[    1.070098] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
[    1.070098] IRQ stage: Linux
[    1.070098] Workqueue: async async_run_entry_fn
[    1.070098] RIP: 0010:irq_post_stage+0x8c/0xc0
[    1.070098] Code: 12 48 0f ab 30 c3 cc cc cc cc 80 3d 06 4e b4 01 00 75 f2 48 8b 57 08 48 c7 c7 98 81 2c 93 c6 05 f2 4d b4 01 01 e8 c4 3e f3 ff <0f> 0b c3 cc cc cc cc 80 3d de 4d b4 01 00 75 cb 48 8b 57 08 48 c7
[    1.070098] RSP: 0000:ffffb22680003fe0 EFLAGS: 00010286
[    1.070098] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000202
[    1.070098] RDX: 0000000080010002 RSI: 0000000000000000 RDI: 0000000000000001
[    1.070098] RBP: ffff896c011f3400 R08: 000000000000005b R09: 000000004b4c332f
[    1.070098] R10: 00000000ffffffff R11: 00000000ffffffff R12: 0000000000080105
[    1.070098] R13: 00000000000000ed R14: 0000000000000000 R15: 0000000000000000
[    1.070098] FS:  0000000000000000(0000) GS:ffff896caa667000(0000) knlGS:0000000000000000
[    1.070098] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.070098] CR2: ffff896c26202000 CR3: 000000002483a001 CR4: 00000000003706f0
[    1.070098] Call Trace:
[    1.070098]  <IRQ>
[    1.070098]  __sysvec_hyperv_stimer0+0x2b/0x50
[    1.070098]  arch_do_IRQ_pipelined+0x23c/0x560
[    1.070098]  </IRQ>
[    1.070098]  <TASK>
[    1.070098]  sync_current_irq_stage+0x191/0x230
[    1.070098]  handle_irq_pipelined_finish+0x7b/0x1a0
[    1.070098]  arch_pipeline_entry+0xaa/0xf0
[    1.070098]  asm_sysvec_hyperv_stimer0+0x16/0x20
[    1.070098] RIP: 0010:ZSTD_decompressSequences_bmi2.constprop.0+0x62a/0xa80
[    1.070098] Code: 08 48 8b 58 08 48 83 c7 20 48 83 c0 20 48 89 4f e0 48 89 5f e8 48 8b 48 f0 48 8b 58 f8 48 89 4f f0 48 89 5f f8 48 39 d7 72 d4 <49> 83 fe 88 0f 87 25 fb ff ff 4d 01 f5 83 ad 34 ff ff ff 01 0f 85
[    1.070098] RSP: 0000:ffffb226800ffa10 EFLAGS: 00000283
[    1.070098] RAX: ffffb226807b4fc9 RBX: 000000000000000b RCX: ffffb226803fa41c
[    1.070098] RDX: 000000000038b951 RSI: ffffb226804296d8 RDI: ffffb226807b5029
[    1.070098] RBP: ffffb226800ffb50 R08: 040000000000003d R09: fc00000285000000
[    1.070098] R10: 0000000000000003 R11: 000000000000009f R12: 0000000000000060
[    1.070098] R13: ffffb226807b5028 R14: 000000000000000c R15: 0000000000055518
[    1.070098]  ZSTD_decompressContinue.part.0+0x359/0x490
[    1.070098]  ZSTD_decompressContinueStream+0x96/0x150
[    1.070098]  ZSTD_decompressStream+0x7c3/0xc00
[    1.070098]  ? __pfx_flush_buffer+0x10/0x10
[    1.070098]  ? __pfx_error+0x10/0x10
[    1.070098]  unzstd+0x340/0x570
[    1.070098]  ? __pfx_unzstd+0x10/0x10
[    1.070098]  unpack_to_rootfs+0x144/0x3a0
[    1.070098]  ? __pfx_error+0x10/0x10
[    1.070098]  ? _printk+0x5f/0x80
[    1.070098]  ? do_populate_rootfs+0x124/0x1d0
[    1.070098]  do_populate_rootfs+0x124/0x1d0
[    1.070098]  ? ktime_get+0x83/0x180
[    1.070098]  async_run_entry_fn+0x2d/0x130
[    1.070098]  process_one_work+0x22d/0x6d0
[    1.070098]  worker_thread+0x184/0x320
[    1.070098]  ? __pfx_worker_thread+0x10/0x10
[    1.070098]  kthread+0xfa/0x240
[    1.070098]  ? __pfx_kthread+0x10/0x10
[    1.070098]  ret_from_fork+0x2d/0x50
[    1.070098]  ? __pfx_kthread+0x10/0x10
[    1.070098]  ret_from_fork_asm+0x1a/0x30
[    1.070098]  </TASK>
[    1.070098] irq event stamp: 14192
[    1.070098] hardirqs last  enabled at (14191): [<ffffffff92d7c95a>] arch_pipeline_entry+0x8a/0xf0
[    1.070098] hardirqs last disabled at (14192): [<ffffffff92185fe1>] sync_current_irq_stage+0x201/0x230
[    1.070098] softirqs last  enabled at (12424): [<ffffffff920c3cfe>] handle_softirqs+0x33e/0x420
[    1.070098] softirqs last disabled at (12409): [<ffffffff920c3f48>] __irq_exit_rcu+0xe8/0x110
[    1.070098] ---[ end trace 0000000000000000 ]---

Have you - or someone else - ever tried running dovetail on top of
hyper-v?

The main purpose behind this exercise: Get a feeling about necessary
efforts to fully support hyper-v as hypervisor - if possible at all.

Best regards,
Florian

[1] https://gitlab.com/Xenomai/dovetail-hacker-space/-/tree/flo/add-hyperv-support-for-6-15?ref_type=heads

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: Dovetail: Adding support for Hyper-v as hypervisor

[Philippe Gerum]

Hi,

Florian Bezdeka <florian.bezdeka@siemens.com> writes:

> Hi Philippe,
>
> I'm currently experimenting with hyper-v as Hypervisor for dovetail. So
> far so good, I'm able to boot the system and the system survives a
> inband stress test. The necessary modifications can be found at [1].
>
> But: Running the Xenomai 3 testsuite - especially the gdb test -
> freezes the system immediately.
>
> When enabling pipeline and dovetail debug options I get the following
> splat report. Ideas welcome...

Since the splat happens because lockdep mistakenly thinks that we are
running in hard irq context although we are actually running a
workqueue, there must be an issue with the tracking of the
hardirq_context variable. Could there be some imbalance with
lockdep_hardirq_{enter, exit}() due to some specifics in routing the
STIMER0 vector?

>
>
> Have you - or someone else - ever tried running dovetail on top of
> hyper-v?
>

Nope.

-- 
Philippe.

Re: Dovetail: Adding support for Hyper-v as hypervisor

[Florian Bezdeka]

On Thu, 2025-06-05 at 09:58 +0200, Philippe Gerum wrote:
> Hi,
> 
> Florian Bezdeka <florian.bezdeka@siemens.com> writes:
> 
> > Hi Philippe,
> > 
> > I'm currently experimenting with hyper-v as Hypervisor for dovetail. So
> > far so good, I'm able to boot the system and the system survives a
> > inband stress test. The necessary modifications can be found at [1].
> > 
> > But: Running the Xenomai 3 testsuite - especially the gdb test -
> > freezes the system immediately.
> > 
> > When enabling pipeline and dovetail debug options I get the following
> > splat report. Ideas welcome...
> 
> Since the splat happens because lockdep mistakenly thinks that we are
> running in hard irq context although we are actually running a
> workqueue, there must be an issue with the tracking of the
> hardirq_context variable. Could there be some imbalance with
> lockdep_hardirq_{enter, exit}() due to some specifics in routing the
> STIMER0 vector?

I'm not sure yet, but yes, the STIMER0 vector seems to be involved
somehow. I end up in irq_post_check(), where the hard_irqs_enabled()
check fails.

Some instrumentation tells me that this happens when the STIMER0 vector
was the last vector handled by the CPU, somehow confused the IRQ
pipeline and the hard IRQ check fails while handling / posting the next
inband IRQ/vector.

Is there a trick available to find the code location that enabled
hard_irqs? Seems I need more instrumentation to get that information.


> 
> > 
> > 
> > Have you - or someone else - ever tried running dovetail on top of
> > hyper-v?
> > 
> 
> Nope.
>