[REGRESSION] vhost_iotlb: bound map allocation in add_range breaks vhost-vdpa on hosts with fragmented memory

[REGRESSION] vhost_iotlb: bound map allocation in add_range breaks vhost-vdpa on hosts with fragmented memory

[Dragos Tatulea]

Hi,

We've started seeing regressions for a while in our verification CI for
vDPA Live Migration tests. Occasionally the FW returns a syndrome
related to invalid mappings during device reset.

We've narrowed this down to commit 1ed35ac7f3fe "vhost_iotlb: bound map
allocation in add_range" which limits the iotlbs to 2K by default. On a
host with fragmented memory you can get easily more than 2K entries.
This results in truncated/incomplete iotlbs mapped to the device and when
the rings happen to be outside of the first 2K mappings the FW will complain
because it got addresses that are not mapped.

I'm not sure how to fix this though:

- LLM suggests setting the 2K limit only in VA mode and using 0 as
  before in PA mode because PA mode is limited anyway. Does this
  work?

- Can we detect this and reject the device / stop the device at boot time?
  Probably not as this doesn't cover Live Migration.

Any thoughts?

Thanks,
Dragos

Re: [REGRESSION] vhost_iotlb: bound map allocation in add_range breaks vhost-vdpa on hosts with fragmented memory

[Michael S. Tsirkin]

On Wed, Oct 28, 2026 at 01:44:30PM +0100, Dragos Tatulea wrote:
> Hi,
> 
> We've started seeing regressions for a while in our verification CI for
> vDPA Live Migration tests. Occasionally the FW returns a syndrome
> related to invalid mappings during device reset.
> 
> We've narrowed this down to commit 1ed35ac7f3fe "vhost_iotlb: bound map
> allocation in add_range" which limits the iotlbs to 2K by default. On a
> host with fragmented memory you can get easily more than 2K entries.
> This results in truncated/incomplete iotlbs mapped to the device and when
> the rings happen to be outside of the first 2K mappings the FW will complain
> because it got addresses that are not mapped.
> 
> I'm not sure how to fix this though:
> 
> - LLM suggests setting the 2K limit only in VA mode and using 0 as
>   before in PA mode because PA mode is limited anyway. Does this
>   work?

Well no, we do need the limit in PA mode I think.

But we should maybe  bound from what's actually being mapped rather than
a fixed count. e.g. X entries per pinned page? Bounding by bytes instead
of entry count is another option.

Also 0 is now rejected.



> - Can we detect this and reject the device / stop the device at boot time?
>   Probably not as this doesn't cover Live Migration.

Exactly. Besides, failure is not at boot.

> 
> Any thoughts?

Does boosting max_iotlb_entries help?
vhost_vdpa.max_iotlb_entries=65536? Or sized to guest RAM?



> 
> Thanks,
> Dragos

Re: [REGRESSION] vhost_iotlb: bound map allocation in add_range breaks vhost-vdpa on hosts with fragmented memory

[Dragos Tatulea]

On 28.10.26 14:05, Michael S. Tsirkin wrote:
> On Wed, Oct 28, 2026 at 01:44:30PM +0100, Dragos Tatulea wrote:
>> Hi,
>>
>> We've started seeing regressions for a while in our verification CI for
>> vDPA Live Migration tests. Occasionally the FW returns a syndrome
>> related to invalid mappings during device reset.
>>
>> We've narrowed this down to commit 1ed35ac7f3fe "vhost_iotlb: bound map
>> allocation in add_range" which limits the iotlbs to 2K by default. On a
>> host with fragmented memory you can get easily more than 2K entries.
>> This results in truncated/incomplete iotlbs mapped to the device and when
>> the rings happen to be outside of the first 2K mappings the FW will complain
>> because it got addresses that are not mapped.
>>
>> I'm not sure how to fix this though:
>>
>> - LLM suggests setting the 2K limit only in VA mode and using 0 as
>>   before in PA mode because PA mode is limited anyway. Does this
>>   work?
> 
> Well no, we do need the limit in PA mode I think.
> 
> But we should maybe  bound from what's actually being mapped rather than
> a fixed count. e.g. X entries per pinned page?
You could have many 1 PAGE_SIZE mappings though. It wouldn't help
then.

> Bounding by bytes instead of entry count is another option.How to get this limit from userspace though? Would RLIMIT_MEMLOCK
be an option? Although `ulimit -l unlimited` would need to be
capped to some value...

Also, if the entry limit is still there some kind of precedence between
memory and entry limit needs to be decided.> 
> Also 0 is now rejected.
> 
> 
> 
>> - Can we detect this and reject the device / stop the device at boot time?
>>   Probably not as this doesn't cover Live Migration.
> 
> Exactly. Besides, failure is not at boot.
> 
>>
>> Any thoughts?
> 
> Does boosting max_iotlb_entries help?
> vhost_vdpa.max_iotlb_entries=65536? Or sized to guest RAM?
If set as module paramer? Yes, it works. But the user should see some
kind of error to know that something is off. Otherwise things will start
failing silently and randomly...

Thanks,
Dragos

Re: [REGRESSION] vhost_iotlb: bound map allocation in add_range breaks vhost-vdpa on hosts with fragmented memory

[Michael S. Tsirkin]

On Wed, Oct 28, 2026 at 03:46:05PM +0100, Dragos Tatulea wrote:
> On 28.10.26 14:05, Michael S. Tsirkin wrote:
> > On Wed, Oct 28, 2026 at 01:44:30PM +0100, Dragos Tatulea wrote:
> >> Hi,
> >>
> >> We've started seeing regressions for a while in our verification CI for
> >> vDPA Live Migration tests. Occasionally the FW returns a syndrome
> >> related to invalid mappings during device reset.
> >>
> >> We've narrowed this down to commit 1ed35ac7f3fe "vhost_iotlb: bound map
> >> allocation in add_range" which limits the iotlbs to 2K by default. On a
> >> host with fragmented memory you can get easily more than 2K entries.
> >> This results in truncated/incomplete iotlbs mapped to the device and when
> >> the rings happen to be outside of the first 2K mappings the FW will complain
> >> because it got addresses that are not mapped.
> >>
> >> I'm not sure how to fix this though:
> >>
> >> - LLM suggests setting the 2K limit only in VA mode and using 0 as
> >>   before in PA mode because PA mode is limited anyway. Does this
> >>   work?
> > 
> > Well no, we do need the limit in PA mode I think.
> > 
> > But we should maybe  bound from what's actually being mapped rather than
> > a fixed count. e.g. X entries per pinned page?
> You could have many 1 PAGE_SIZE mappings though. It wouldn't help
> then.

it still mostly is 1 entry=1 pinned page

sharing is possible but uncommon

> > Bounding by bytes instead of entry count is another option.How to get this limit from userspace though? Would RLIMIT_MEMLOCK
> be an option? Although `ulimit -l unlimited` would need to be
> capped to some value...

guest ram no?

> 
> Also, if the entry limit is still there some kind of precedence between
> memory and entry limit needs to be decided.

I'd steal a page from rlimit then use that for the entries, when out of
that - steal another page.

> 
> > Also 0 is now rejected.
> > 
> > 
> > 
> >> - Can we detect this and reject the device / stop the device at boot time?
> >>   Probably not as this doesn't cover Live Migration.
> > 
> > Exactly. Besides, failure is not at boot.
> > 
> >>
> >> Any thoughts?
> > 
> > Does boosting max_iotlb_entries help?
> > vhost_vdpa.max_iotlb_entries=65536? Or sized to guest RAM?
> If set as module paramer? Yes, it works. But the user should see some
> kind of error to know that something is off. Otherwise things will start
> failing silently and randomly...
> 
> Thanks,
> Dragos