Re: [PATCH v2 1/2] Resize the MAX_NR_IO_RANGES for ioreq server

[George Dunlap <george.dunlap@eu.citrix.com>]

On 07/06/2015 02:09 PM, Paul Durrant wrote:
>> -----Original Message-----
>> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of
>> George Dunlap
>> Sent: 06 July 2015 13:50
>> To: Paul Durrant
>> Cc: Yu Zhang; xen-devel@lists.xen.org; Keir (Xen.org); Jan Beulich; Andrew
>> Cooper; Kevin Tian; zhiyuan.lv@intel.com
>> Subject: Re: [Xen-devel] [PATCH v2 1/2] Resize the MAX_NR_IO_RANGES for
>> ioreq server
>>
>> On Mon, Jul 6, 2015 at 1:38 PM, Paul Durrant <Paul.Durrant@citrix.com>
>> wrote:
>>>> -----Original Message-----
>>>> From: dunlapg@gmail.com [mailto:dunlapg@gmail.com] On Behalf Of
>>>> George Dunlap
>>>> Sent: 06 July 2015 13:36
>>>> To: Yu Zhang
>>>> Cc: xen-devel@lists.xen.org; Keir (Xen.org); Jan Beulich; Andrew Cooper;
>>>> Paul Durrant; Kevin Tian; zhiyuan.lv@intel.com
>>>> Subject: Re: [Xen-devel] [PATCH v2 1/2] Resize the MAX_NR_IO_RANGES
>> for
>>>> ioreq server
>>>>
>>>> On Mon, Jul 6, 2015 at 7:25 AM, Yu Zhang <yu.c.zhang@linux.intel.com>
>>>> wrote:
>>>>> MAX_NR_IO_RANGES is used by ioreq server as the maximum
>>>>> number of discrete ranges to be tracked. This patch changes
>>>>> its value to 8k, so that more ranges can be tracked on next
>>>>> generation of Intel platforms in XenGT. Future patches can
>>>>> extend the limit to be toolstack tunable, and MAX_NR_IO_RANGES
>>>>> can serve as a default limit.
>>>>>
>>>>> Signed-off-by: Yu Zhang <yu.c.zhang@linux.intel.com>
>>>>
>>>> I said this at the Hackathon, and I'll say it here:  I think this is
>>>> the wrong approach.
>>>>
>>>> The problem here is not that you don't have enough memory ranges.  The
>>>> problem is that you are not tracking memory ranges, but individual
>>>> pages.
>>>>
>>>> You need to make a new interface that allows you to tag individual
>>>> gfns as p2m_mmio_write_dm, and then allow one ioreq server to get
>>>> notifications for all such writes.
>>>>
>>>
>>> I think that is conflating things. It's quite conceivable that more than one
>> ioreq server will handle write_dm pages. If we had enough types to have
>> two page types per server then I'd agree with you, but we don't.
>>
>> What's conflating things is using an interface designed for *device
>> memory ranges* to instead *track writes to gfns*.
> 
> What's the difference? Are you asserting that all device memory ranges have read side effects and therefore write_dm is not a reasonable optimization to use? I would not want to make that assertion.

The difference is that actual device memory is typically in big chunks,
and each device rarely has more than a handful; whereas guest pfns
allocated by the operating system to a driver are typically singleton
pages scattered all over the address space.

Which is why a handful of memory ranges is sufficient for tracking
devices with hundreds of megabytes of device memory, but thousands of
memory ranges are not sufficient for tracking a megabyte or so of GPU
pagetables: the former is one or two actual ranges of a significant
size; the latter are (apparently) thousands of ranges of one page each.

 -George