From: Dragos Tatulea <dtatulea@nvidia.com>
To: "Michael S. Tsirkin" <mst@redhat.com>
Cc: "eperezma@redhat.com" <eperezma@redhat.com>,
Linfeng Sun <linfeng.sun.dev@gmail.com>,
"virtualization@lists.linux.dev" <virtualization@lists.linux.dev>,
Jason Wang <jasowangio@gmail.com>
Subject: Re: [REGRESSION] vhost_iotlb: bound map allocation in add_range breaks vhost-vdpa on hosts with fragmented memory
Date: Thu, 19 Nov 2026 19:40:11 +0100 [thread overview]
Message-ID: <9a901a79-6253-46ca-bbba-ca9bef7f0e4d@nvidia.com> (raw)
In-Reply-To: <ffef2498-eaba-4349-bdf5-7b035052ae80@nvidia.com>
On 30.10.26 14:47, Dragos Tatulea wrote:
> On 28.10.26 16:17, Michael S. Tsirkin wrote:
>> On Wed, Oct 28, 2026 at 03:46:05PM +0100, Dragos Tatulea wrote:
>>> On 28.10.26 14:05, Michael S. Tsirkin wrote:
>>>> On Wed, Oct 28, 2026 at 01:44:30PM +0100, Dragos Tatulea wrote:
>>>>> Hi,
>>>>>
>>>>> We've started seeing regressions for a while in our verification CI for
>>>>> vDPA Live Migration tests. Occasionally the FW returns a syndrome
>>>>> related to invalid mappings during device reset.
>>>>>
>>>>> We've narrowed this down to commit 1ed35ac7f3fe "vhost_iotlb: bound map
>>>>> allocation in add_range" which limits the iotlbs to 2K by default. On a
>>>>> host with fragmented memory you can get easily more than 2K entries.
>>>>> This results in truncated/incomplete iotlbs mapped to the device and when
>>>>> the rings happen to be outside of the first 2K mappings the FW will complain
>>>>> because it got addresses that are not mapped.
>>>>>
>>>>> I'm not sure how to fix this though:
>>>>>
>>>>> - LLM suggests setting the 2K limit only in VA mode and using 0 as
>>>>> before in PA mode because PA mode is limited anyway. Does this
>>>>> work?
>>>>
>>>> Well no, we do need the limit in PA mode I think.
>>>>
>>>> But we should maybe bound from what's actually being mapped rather than
>>>> a fixed count. e.g. X entries per pinned page?
>>> You could have many 1 PAGE_SIZE mappings though. It wouldn't help
>>> then.
>>
>> it still mostly is 1 entry=1 pinned page
>>
>> sharing is possible but uncommon
>>
>>>> Bounding by bytes instead of entry count is another option.How to get this limit from userspace though? Would RLIMIT_MEMLOCK
>>> be an option? Although `ulimit -l unlimited` would need to be
>>> capped to some value...
>>
>> guest ram no?
>>
>>>
>>> Also, if the entry limit is still there some kind of precedence between
>>> memory and entry limit needs to be decided.
>>
>> I'd steal a page from rlimit then use that for the entries, when out of
>> that - steal another page.
>>
> Do you mean something like this [1]?
>
> Remaining questions:
> 1) Is it ok to delete max_iotlb_entries? Or keep it initialized to max?
> 2) Is it ok to delete the module specific max_iotlb_entries param? For
> example mlx5_vdpa_max_iotlb_entries. I don't see much use for it ...
> 3) Shouldn't the core or the driver track holes encounter due to
> incomplete mappings? It would be good to not give bad addresses
> to the hw/fw...
>
> Thanks,
> Dragos
>
Ping. Did you have a moment to check?
Thanks,
Dragos
prev parent reply other threads:[~2026-11-19 18:40 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-28 12:44 [REGRESSION] vhost_iotlb: bound map allocation in add_range breaks vhost-vdpa on hosts with fragmented memory Dragos Tatulea
2026-10-28 13:05 ` Michael S. Tsirkin
2026-10-28 14:46 ` Dragos Tatulea
2026-10-28 15:17 ` Michael S. Tsirkin
2026-10-30 13:47 ` Dragos Tatulea
2026-11-19 18:40 ` Dragos Tatulea [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9a901a79-6253-46ca-bbba-ca9bef7f0e4d@nvidia.com \
--to=dtatulea@nvidia.com \
--cc=eperezma@redhat.com \
--cc=jasowangio@gmail.com \
--cc=linfeng.sun.dev@gmail.com \
--cc=mst@redhat.com \
--cc=virtualization@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox