Re: Why such a big difference in init-time PCI resource call-paths (x86 vs x86_64) ?

Re: Why such a big difference in init-time PCI resource call-paths (x86 vs x86_64) ?

[Jesse Barnes]

On Wednesday, April 30, 2008 9:07 am TJ wrote:
> In preparation for writing a Windows-style PCI resource allocation
> strategy
>
>  - use all e820 gaps for IOMEM resources; top-down allocation -
>
> and thus giving devices with large IOMEM requirements more chance of
> allocation in the 32-bit address space below 4GB (see bugzilla #10461),
> I'm mapping out the current implementation to ensure I understand the
> implications and guard against breaking any quirks or undocumented
> features.

Excellent, definitely a worthwhile change...

> There are significant differences between the x86 and x86_64
> implementations and I've not been able to find any explanation or deduce
> a reason for it.
>
> The first difference is mainly cosmetic. With one minor difference the
> source code of:
>
>   arch/x86/kernel/e820_32.c::e820_register_memory()
>   arch/x86/kernel/e820_64.c::e820_setup_gap()
>
> is identical. The purpose is to find the single largest gap in the e820
> map and point pci_mem_start to the start of the region (with a
> rounding-up adjustment).
>
> Why do these functions have such differing names?
> Which name is preferable?

Well e820_setup_gap is a bit more descriptive, but if you're factoring out 
common code in the x86 tree, it may as well be even more descriptive, e.g. 
e820_find_pci_gap or somesuch.

> The second difference appears more profound. In the x86_64 call path,
> the code that reserves resources and requests the ioport resource is
> called in:
>
>   arch/x86/kernel/setup_64.c::setup_arch()
>
> immediately *before* the call to e820_setup_gap():
>
> 	e820_reserve_resources();
> 	e820_mark_nosave_regions();
>
> 	/* request I/O space for devices used on all i[345]86 PCs */
> 	for (i = 0; i < ARRAY_SIZE(standard_io_resources); i++)
> 		request_resource(&ioport_resource, &standard_io_resources[i]);
>
> 	e820_setup_gap();
>
>
> On x86 however:
>
> 	e820_register_memory();
> 	e820_mark_nosave_regions();
>
> Although e820_register_memory() and e820_mark_nosave_regions() are
> called at basically the same point in:
>
>   arch/x86/kernel/setup_32.c::setup_arch()
>
> as the 64-bit setup_arch() call to e820_setup_gap(), the equivalent of
> the 64-bit e820_reserve_resources():
>
>   arch/x86/kernel/e820_32.c::init_iomem_resources()
>
> and the loop calling request_resource() is in:
>
>   arch/x86/kernel/setup_32.c::request_standard_resources()
>
> which is declared as:
>
>   subsys_initcall(request_standard_resources);
>
>
> Now, unless my call-path tracing has gone way wrong, this initcall won't
> happen until *much* later after a *lot* of other 'stuff' has been done.
> If my understanding of the initcall mechanism is correct, it finally
> gets called in:
>
>   init/main.c::do_initcalls()
>
> as part of the generic multi-level device init call-by-address
> functionality where each function is in the .initcallX.init section and
> they have names of the form __initcall_functionX (where X is the level
> id).
>
> So, why the big difference in implementations?
> What are the implications of each?
> Is one preferable to the other?

Andi may remember some of the history here.  It seems like we should be 
reserving those regions early on like the x86_64 code does, but it's possible 
that will create more conflicts later on for some platforms (I don't know of 
any offhand though).

> Any other observations or caveats?

For future patches and x86 specific questions, you should probably cc lkml and 
the x86 maintainers, they may be able to provide additional feedback.

Thanks,
Jesse

Re: Why such a big difference in init-time PCI resource call-paths (x86 vs x86_64) ?

[H. Peter Anvin]

Andi Kleen wrote:
> On Thu, May 01, 2008 at 11:16:31AM -0700, Jesse Barnes wrote:
>> On Wednesday, April 30, 2008 9:07 am TJ wrote:
>>> In preparation for writing a Windows-style PCI resource allocation
>>> strategy
>>>
>>>  - use all e820 gaps for IOMEM resources; top-down allocation -
>>>
>>> and thus giving devices with large IOMEM requirements more chance of
>>> allocation in the 32-bit address space below 4GB (see bugzilla #10461),
> 
> I tried that some time ago and it turned out that some systems have
> mappings in holes and don't boot anymore when you fill the holes too much. 
> But that was only considering e820. if you do this it might work if you 
> do it really like windows and consider all resources, including ACPI.

Yes, considering all possible reservation schemes is really critical 
here (including the magic knowledge of the legacy region).

>>> So, why the big difference in implementations?
>>> What are the implications of each?
>>> Is one preferable to the other?
> 
> I don't remember why it is different. Probably wasn't intentional.
> But in general x86-64 is less fragile than i386 here because it has the e820
> allocator and can deal better with conflicts.

Yes, and that's definitely the model we want on both sides.  Working on 
this is way high on my personal priority list at the moment.

	-hpa

Re: Why such a big difference in init-time PCI resource call-paths (x86 vs x86_64) ?

[Andi Kleen]

On Thu, May 01, 2008 at 11:16:31AM -0700, Jesse Barnes wrote:
> On Wednesday, April 30, 2008 9:07 am TJ wrote:
> > In preparation for writing a Windows-style PCI resource allocation
> > strategy
> >
> >  - use all e820 gaps for IOMEM resources; top-down allocation -
> >
> > and thus giving devices with large IOMEM requirements more chance of
> > allocation in the 32-bit address space below 4GB (see bugzilla #10461),

I tried that some time ago and it turned out that some systems have
mappings in holes and don't boot anymore when you fill the holes too much. 
But that was only considering e820. if you do this it might work if you 
do it really like windows and consider all resources, including ACPI.

> > So, why the big difference in implementations?
> > What are the implications of each?
> > Is one preferable to the other?

I don't remember why it is different. Probably wasn't intentional.
But in general x86-64 is less fragile than i386 here because it has the e820
allocator and can deal better with conflicts.

-Andi

Re: Why such a big difference in init-time PCI resource call-paths (x86 vs x86_64) ?

[Jesse Barnes]

On Thursday, May 01, 2008 1:11 pm H. Peter Anvin wrote:
> Andi Kleen wrote:
> > On Thu, May 01, 2008 at 11:16:31AM -0700, Jesse Barnes wrote:
> >> On Wednesday, April 30, 2008 9:07 am TJ wrote:
> >>> In preparation for writing a Windows-style PCI resource allocation
> >>> strategy
> >>>
> >>>  - use all e820 gaps for IOMEM resources; top-down allocation -
> >>>
> >>> and thus giving devices with large IOMEM requirements more chance of
> >>> allocation in the 32-bit address space below 4GB (see bugzilla #10461),
> >
> > I tried that some time ago and it turned out that some systems have
> > mappings in holes and don't boot anymore when you fill the holes too
> > much. But that was only considering e820. if you do this it might work if
> > you do it really like windows and consider all resources, including ACPI.
>
> Yes, considering all possible reservation schemes is really critical
> here (including the magic knowledge of the legacy region).

Yeah, good point.  There's code to check for ACPI reserved regions when 
verifying MCFG space, so at least we're starting to do this...

Thanks,
Jesse

Re: Why such a big difference in init-time PCI resource call-paths (x86 vs x86_64) ?

[Yinghai Lu]

On Thu, May 1, 2008 at 1:11 PM, Andi Kleen <andi@firstfloor.org> wrote:
> On Thu, May 01, 2008 at 11:16:31AM -0700, Jesse Barnes wrote:
>  > On Wednesday, April 30, 2008 9:07 am TJ wrote:
>  > > In preparation for writing a Windows-style PCI resource allocation
>  > > strategy
>  > >
>  > >  - use all e820 gaps for IOMEM resources; top-down allocation -
>  > >
>  > > and thus giving devices with large IOMEM requirements more chance of
>  > > allocation in the 32-bit address space below 4GB (see bugzilla #10461),
>
>  I tried that some time ago and it turned out that some systems have
>  mappings in holes and don't boot anymore when you fill the holes too much.
>  But that was only considering e820. if you do this it might work if you
>  do it really like windows and consider all resources, including ACPI.

wonder if using holes in MTRR AND e820 could help...

YH

Re: Why such a big difference in init-time PCI resource call-paths (x86 vs x86_64) ?

[H. Peter Anvin]

Yinghai Lu wrote:
> On Thu, May 1, 2008 at 1:11 PM, Andi Kleen <andi@firstfloor.org> wrote:
>> On Thu, May 01, 2008 at 11:16:31AM -0700, Jesse Barnes wrote:
>>  > On Wednesday, April 30, 2008 9:07 am TJ wrote:
>>  > > In preparation for writing a Windows-style PCI resource allocation
>>  > > strategy
>>  > >
>>  > >  - use all e820 gaps for IOMEM resources; top-down allocation -
>>  > >
>>  > > and thus giving devices with large IOMEM requirements more chance of
>>  > > allocation in the 32-bit address space below 4GB (see bugzilla #10461),
>>
>>  I tried that some time ago and it turned out that some systems have
>>  mappings in holes and don't boot anymore when you fill the holes too much.
>>  But that was only considering e820. if you do this it might work if you
>>  do it really like windows and consider all resources, including ACPI.
> 
> wonder if using holes in MTRR AND e820 could help...
> 

Typically not, since the MTRRs won't tell you what is free address space 
and what is occupied by non-BAR I/O devices.

	-hpa

Re: Why such a big difference in init-time PCI resource call-paths (x86 vs x86_64) ?

[H. Peter Anvin]

Robert Hancock wrote:
>>
>> Yes, considering all possible reservation schemes is really critical 
>> here (including the magic knowledge of the legacy region).
> 
> FYI, from what I've read, Windows ignores e820 for detecting resource 
> reservations and looks at ACPI reservations only (at least if it's 
> running in ACPI mode which these days is almost universal). It seems 
> Windows really only uses e820 for locating RAM areas..

There is no doubt ACPI is critical.  Using E820 is good practice too, 
though.  Windows actually uses (only) one of several methods, depending 
on which HAL it chooses to use.

	-hpa

Re: Why such a big difference in init-time PCI resource call-paths (x86 vs x86_64) ?

[Robert Hancock]

H. Peter Anvin wrote:
> Andi Kleen wrote:
>> On Thu, May 01, 2008 at 11:16:31AM -0700, Jesse Barnes wrote:
>>> On Wednesday, April 30, 2008 9:07 am TJ wrote:
>>>> In preparation for writing a Windows-style PCI resource allocation
>>>> strategy
>>>>
>>>>  - use all e820 gaps for IOMEM resources; top-down allocation -
>>>>
>>>> and thus giving devices with large IOMEM requirements more chance of
>>>> allocation in the 32-bit address space below 4GB (see bugzilla #10461),
>>
>> I tried that some time ago and it turned out that some systems have
>> mappings in holes and don't boot anymore when you fill the holes too 
>> much. But that was only considering e820. if you do this it might work 
>> if you do it really like windows and consider all resources, including 
>> ACPI.
> 
> Yes, considering all possible reservation schemes is really critical 
> here (including the magic knowledge of the legacy region).

FYI, from what I've read, Windows ignores e820 for detecting resource 
reservations and looks at ACPI reservations only (at least if it's 
running in ACPI mode which these days is almost universal). It seems 
Windows really only uses e820 for locating RAM areas..

Re: Why such a big difference in init-time PCI resource call-paths (x86 vs x86_64) ?

[Andi Kleen]

> FYI, from what I've read, Windows ignores e820 for detecting resource 
> reservations and looks at ACPI reservations only (at least if it's 
> running in ACPI mode which these days is almost universal). It seems 
> Windows really only uses e820 for locating RAM areas..

The usual problem is that they don't use that on older systems
but a different HAL (or XP doesn't run at all on these older systems).
But Linux still wants to run on older systems, ideally without
special settings too.

So if it's done like this it likely needs some date or other heuristic to 
decide if it should trust ACPI or should not. 

-Andi