[Question] Possible tlb_sync hang when remote SFENCE.VMA races with HSM hart_stop

[bin.yao@ingenic.com]

Hello OpenSBI maintainers,

We are seeing a hang during aging/stress testing and would like to check whether our analysis is correct.

Environment:
- OpenSBI base version: v1.8.1
- Base commit in our tree: 74434f255873d74e56cc50aa762d1caf24c099f8
- Downstream tree with platform-specific HSM hart_stop support
- Platform: downstream Ingenic platform

Observed symptom:
When one hart executes remote SFENCE.VMA while another hart is stopping, the sender can spin forever in tlb_sync() waiting for a completion that never arrives.

Our current understanding of the race is:

1. The source hart calls sbi_tlb_request() and then sbi_ipi_send_many().
2. sbi_ipi_send_many() builds the target mask using sbi_hsm_hart_interruptible_mask(), so a target hart can still be selected while it is in SBI_HSM_STATE_STARTED.
3. Before the source completes the send path, the target hart can execute sbi_hsm_hart_stop() and transition from STARTED to STOP_PENDING.
4. The source still queues the TLB request in tlb_update() and increments its local tlb_sync counter.
5. That counter is decremented only when the target hart later runs tlb_entry_process().
6. If the target hart has already passed the point in the stop path where pending IPIs are drained, the queued TLB request is never processed, so the source remains in tlb_sync() forever.

So this looks like a race between:
- the interruptible-mask snapshot in sbi_ipi_send_many(), and
- the STARTED -> STOP_PENDING transition in sbi_hsm_hart_stop()

In other words, tlb_sync itself does not appear to be wrong; the problem seems to be that a hart can be counted as a target, but then stop before it can process the queued TLB request.

To make the window easier to hit during debugging, we temporarily added an artificial delay in sbi_ipi_send_many() in our downstream tree. However, our understanding is that the delay only increases the reproduction probability, and the race exists in principle even without that modification.

Questions:
1. Does the analysis above look correct?
2. Is this race already known?
3. Is there an existing or preferred way to serialize remote rfence/IPI send against hart_stop?
4. Could this be related to issue #402 ("Lifelock condition under high load with many TLB shootdowns"), or is it better treated as a separate issue?

If useful, I can also send a minimal timeline or a proposed fix.

Thanks,
Bin Yao
Ingenic

-- 
opensbi mailing list
opensbi@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi