Re: [Qemu-devel] host physical address width issues/questions for x86_64

[Prasad Singamsetty <prasad.singamsetty@oracle.com>]

On 10/16/2017 8:56 PM, Peter Xu wrote:
> On Mon, Oct 16, 2017 at 10:02:25AM -0700, Prasad Singamsetty wrote:
>>
>>
>> On 10/14/2017 8:53 PM, Peter Xu wrote:
>>> On Fri, Oct 13, 2017 at 11:14:03AM -0600, Alex Williamson wrote:
>>>> On Fri, 13 Oct 2017 18:01:44 +0100
>>>> "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
>>>>
>>>>> * Prasad Singamsetty (prasad.singamsetty@oracle.com) wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I am new to the alias. I have some questions on this subject
>>>>>> and seek some clarifications from the experts in the team.
>>>>>> I ran into a couple of issues when I tried with large configuration
>>>>>> ( >= 1TB memory, > 255 CPUs) for x86_64 guest machine.
>>>>>>
>>>>>> 1. QEMU uses the default value of 40 (TCG_PHYS_ADDR_BITS) for address
>>>>>>     width if user has not specified phys-bits or host-phys-bits=true
>>>>>>     property. The default value is obviously not sufficient and
>>>>>>     causing guest kernel to crash if configured with >= 1TB
>>>>>>     memory. Depending on the linux kernel version in the guest the
>>>>>>     panic was in different code paths. The workaround is for the
>>>>>>     user to specify the phys-bits property or set the property
>>>>>>     host-phys-bits=true.
>>>>>>
>>>>>>     QUESTIONS:
>>>> ...
>>>>>> 2. host_address_width in DMAR table structure
>>>>>>
>>>>>>     In this case, the default value is set to 39
>>>>>>     (VTD_HOST_ADDRESS_WIDTH - 1). With interrupt remapping
>>>>>>     enabled for the intel iommu and the guest is configured
>>>>>>     with > 255 cpus and >= 1TB memory, the guest kernel hangs
>>>>>>     during boot up. This need to be fixed.
>>>>>>
>>>>>>     QUESTION:
>>>>>>     The question here again is can we fix this to use the
>>>>>>     real address width from the host as the default?
>>>>>
>>>>> I don't know DMAR stuff; chatting to Alex (cc'd) it does sound
>>>>> like that's an ommission that should be fixed.
>>>>
>>>> [CC +Peter]
>>>>
>>>> On physical hardware VT-d supports either 39 or 48 bit address widths
>>>> and generally you'd expect a sufficiently capable IOMMU to be matched
>>>> with the CPU.  Seems QEMU has only implemented a lower bit width and
>>>> it should probably be forcing phys bits of the VM to 39 to match until
>>>> the extended width can be implemented.  Thanks,
>>>>
>>>> Alex
>>>
>>> There were patches that tried to enable 48 bits GAW but it was
>>> not accepted somehow:
>>>
>>>    https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg01886.html
>>>
>>> Would this help in any way?
>>>
>>
>> Thanks Alex for the patch info. Just curious why the patch was not
>> accepted. Any way, I will try it.
> 
> I don't sure I know the reason.  Anyway, it originated from one of
> Fam's request for some NVMe tests.  If it can really help for your use
> case as well, please feel free to revive those patches, or let me know
> so that I can respin.  Thanks,
> 

Thanks Peter. I will start with your patch and see if I can get
it to work first.

A quick question. Looking at the code, it doesn't look like there
is a way to disable dma remapping. User may have a case where he
is interested only in interrupt remapping (for > 255 cpus) and
not DMA remapping. Is that scenario considered before?

Thanks.
--Prasad