Questions about logic_pio

Questions about logic_pio

[Jiaxun Yang]

Hi there,

Logic PIO gives us a way to make indirect PIO access, however,
the way it handles direct (MMIO) I/O access confused me.

I was trying to create a PCI controller Driver and noticed that I/O range parsed
from DeviceTree will be added to the Logic PIO range by logic_pio_register_range. 
And than PCI subsystem will use the ioport obtained from `logic_pio_trans_cpuaddr`
to allocate resources for the host bridge. In my case, the range added to the logic pio
 was set as hw_start 0x4000, size 0x4000. Later, `logic_pio_trans_cpuaddr` called
by `pci_address_to_pio` gives a ioport of 0x0, which is totally wrong.

After dig into logic pio logic, I found that logic pio is trying to "allocate" an io_start
for MMIO ranges, the allocation starts from 0x0. And later the io_start is used to calculate
cpu_address.  In my opinion, for direct MMIO access, logic_pio address should always
equal to hw address, because there is no way to translate address from logic pio address
to actual hw address in {in,out}{b,sb,w,sb,l,sl} operations.

How this mechanism intends to work? What is the reason that we are trying to
allocate a io_start for MMIO rather than take their hw_start ioport directly?

Thanks. 

--
Jiaxun Yang

Re: Questions about logic_pio

[Wei Xu]

Hi Jiaxun,

On 2020/2/19 21:58, Jiaxun Yang wrote:
> Hi there,
> 
> Logic PIO gives us a way to make indirect PIO access, however,
> the way it handles direct (MMIO) I/O access confused me.
> 
> I was trying to create a PCI controller Driver and noticed that I/O range parsed
> from DeviceTree will be added to the Logic PIO range by logic_pio_register_range. 
> And than PCI subsystem will use the ioport obtained from `logic_pio_trans_cpuaddr`
> to allocate resources for the host bridge. In my case, the range added to the logic pio
>  was set as hw_start 0x4000, size 0x4000. Later, `logic_pio_trans_cpuaddr` called
> by `pci_address_to_pio` gives a ioport of 0x0, which is totally wrong.
> 
> After dig into logic pio logic, I found that logic pio is trying to "allocate" an io_start
> for MMIO ranges, the allocation starts from 0x0. And later the io_start is used to calculate
> cpu_address.  In my opinion, for direct MMIO access, logic_pio address should always
> equal to hw address, because there is no way to translate address from logic pio address
> to actual hw address in {in,out}{b,sb,w,sb,l,sl} operations.
> 
> How this mechanism intends to work? What is the reason that we are trying to
> allocate a io_start for MMIO rather than take their hw_start ioport directly?
> 
> Thanks. 

Corrected John's mail address.
Maybe he can help.

Best Regards,
Wei

> 
> --
> Jiaxun Yang
> 
> 
> 
> .
> 

Re: Questions about logic_pio

[John Garry]

+ Arnd (also remove some defunct e-mail addresses)

On 20/02/2020 09:48, Wei Xu wrote:
> Hi Jiaxun,
> 
> On 2020/2/19 21:58, Jiaxun Yang wrote:
>> Hi there,
>>
>> Logic PIO gives us a way to make indirect PIO access, however,
>> the way it handles direct (MMIO) I/O access confused me.
>>
>> I was trying to create a PCI controller Driver and noticed that I/O range parsed
>> from DeviceTree will be added to the Logic PIO range by logic_pio_register_range.
>> And than PCI subsystem will use the ioport obtained from `logic_pio_trans_cpuaddr`
>> to allocate resources for the host bridge. In my case, the range added to the logic pio
>>   was set as hw_start 0x4000, size 0x4000.

FYI, For when CONFIG_INDIRECT_PIO is defined, in logical PIO space, we 
reserve the last 0x4000 bytes for Indirectio, while the rest is 
available for MMIO. When CONFIG_INDIRECT_PIO is not defined, all Logical 
PIO space is available for MMIO.

  Later, `logic_pio_trans_cpuaddr` called
>> by `pci_address_to_pio` gives a ioport of 0x0, which is totally wrong.

ioport of 0x0 seems fine. Here is my IO port listing for my host (which 
defines PCI_IOBASE):

00000000-0000ffff : PCI Bus 0002:f8
   00001000-00001fff : PCI Bus 0002:f9
     00001000-00001007 : 0002:f9:00.0
       00001000-00001007 : serial
     00001008-0000100f : 0002:f9:00.1
       00001008-0000100f : serial
     00001010-00001017 : 0002:f9:00.2
     00001018-0000101f : 0002:f9:00.2
00010000-0001ffff : PCI Bus 0004:88
00020000-0002ffff : PCI Bus 0005:78
00030000-0003ffff : PCI Bus 0006:c0
00040000-0004ffff : PCI Bus 0007:90
00050000-0005ffff : PCI Bus 000a:10
00060000-0006ffff : PCI Bus 000c:20
00070000-0007ffff : PCI Bus 000d:30
00ffc0e3-00ffc0e7 : hisi-lpc-ipmi.5.auto
00ffc2f7-00ffffff : serial8250.6.auto
   00ffc2f7-00ffc2fe : serial

>>
>> After dig into logic pio logic, I found that logic pio is trying to "allocate" an io_start
>> for MMIO ranges, the allocation starts from 0x0. And later the io_start is used to calculate
>> cpu_address.  In my opinion, for direct MMIO access, logic_pio address should always
>> equal to hw address,

I'm not sure what you mean by simply the hw address.

So there is the physical cpu address of the host bridge IO ports, and 
these host bridge IO ports are mapped in pci_remap_iospace() to some 
part of logical PIO space. The base of the logical PIO space corresponds 
to PCI_IOBASE virtual address.

  because there is no way to translate address from logic pio address
>> to actual hw address in {in,out}{b,sb,w,sb,l,sl} operations.

Please check 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/logic_pio.c?h=v5.6-rc2#n231 
for reference:

#if defined(CONFIG_INDIRECT_PIO) && defined(PCI_IOBASE)
#define BUILD_LOGIC_IO(bw, type)					\
type logic_in##bw(unsigned long addr)					\{									\	type ret = 
(type)~0;						\									\	if (addr < MMIO_UPPER_LIMIT) {					\		ret = 
read##bw(PCI_IOBASE + addr);			\	} else if (addr >= MMIO_UPPER_LIMIT && 
addr < IO_SPACE_LIMIT) { \		struct logic_pio_hwaddr *entry = 
find_io_range(addr);	\									\		if (entry)						\			ret = 
entry->ops->in(entry->hostdata,		\					addr, sizeof(type));		\		else				 
		\			WARN_ON_ONCE(1);				\	}								\	return ret;							\}											

For when the IO port address is within the MMIO range (addr < 
MMIO_UPPER_LIMIT), we use readl(PCI_IOBASE + addr); please remember that 
PCI_IOBASE virtual address is mapped to the physical host bridge IO port 
address.

For when CONFIG_INDIRECT_PIO is not defined, we use the asm generic version:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/asm-generic/io.h?h=v5.6-rc2#n461

As an aside, I do notice that the definitions here have changed here in 
87fe2d543f81 ("io: change inX() to have their own IO barrier overrides") 
since we introduced logic PIO.

>>
>> How this mechanism intends to work? 

I hope the above description makes this clearer.

What is the reason that we are trying to
>> allocate a io_start for MMIO rather than take their hw_start ioport directly?

For PCI MMIO, the io_start is the Logical PIO range start address.

>>
>> Thanks.
> 
> Corrected John's mail address.
> Maybe he can help.

Thanks to xuwei

> 
> Best Regards,
> Wei
> 
>>
>> --
>> Jiaxun Yang
>>
>>
>>
>> .
>>
> 
> .
> 

Re: Questions about logic_pio

[Jiaxun Yang]

 ---- 在 星期四, 2020-02-20 18:52:38 John Garry <john.garry@huawei.com> 撰写 ----
Also Cc MIPS list to check other's opinions.

Hi John.

Thanks for your kind explanation, however, I think this way is
violating how I/O ports supposed to work, at least in MIPS world.

 > >>
 > >> After dig into logic pio logic, I found that logic pio is trying to "allocate" an io_start
 > >> for MMIO ranges, the allocation starts from 0x0. And later the io_start is used to calculate
 > >> cpu_address.  In my opinion, for direct MMIO access, logic_pio address should always
 > >> equal to hw address,
 > 
 > I'm not sure what you mean by simply the hw address.
 > 

I meant  hw_start should always equal to io_start.


MIPS have their own wrapped inl/outl functions, doing the samething with
PCI_IOBASE enabled one. I was just trying to use PCI_IOBASE instead.

Originally, the I/O ports layout seems like this:

00000020-00000021 : pic1
00000060-0000006f : i8042
00000070-00000077 : rtc0
000000a0-000000a1 : pic2
00000170-00000177 : pata_atiixp
000001f0-000001f7 : pata_atiixp
00000376-00000376 : pata_atiixp
000003f6-000003f6 : pata_atiixp
00000800-000008ff : acpi
00001000-00001008 : piix4_smbus
00004000-0003ffff : pci io space
  00004000-00004fff : PCI Bus 0000:01
    00004000-000040ff : 0000:01:05.0
  00005000-00005fff : PCI Bus 0000:03
    00005000-0000501f : 0000:03:00.0

But with PCI_IOBASE defined, I got this:

host bridge /bus@10000000/pci@10000000 ranges:
      MEM 0x0040000000..0x007fffffff -> 0x0040000000
       IO 0x0000004000..0x0000007fff -> 0x0000004000
resource collision: [io  0x0000-0x3fff] conflicts with pic1 [io  0x0020-0x0021]

Because io_start was allocated to 0x0 by Logic PIO.

There are a lot of devices that have fixed ioports thanks to x86's legacy.
For example, in my hardware, ioports for RTC, PIC, I8042 are unmoveable,
and they can't be managed by logic pio subsystem.
Also, the PCI Hostbridge got implied by DeviceTree that it's I/O range
started from 0x4000 in bus side, but then, Logic PIO remapped to PCI_IOBASE + 0x0.
The real address should be PCI_IOBASE + 0x4000,
hardware never got correctly informed about that. And there is still no way to
transform to correct address as it's inside the MMIO_LIMIT.

So the question comes to why we're allocating io_start for MMIO PCI_IOBASE
rather than just check the range provided doesn't overlap each other or exceed
the MMIO_LIMIT.

Thanks.

--
Jiaxun Yang

Re: Questions about logic_pio

[John Garry]

> Also Cc MIPS list to check other's opinions.
> 
> Hi John.
> 

Hi Jiaxun Yang,

> Thanks for your kind explanation, however, I think this way is
> violating how I/O ports supposed to work, at least in MIPS world.

For a bit more history, please understand that the core PCI code was 
managing non-native IO port space in the same way before we added the 
logic PIO framework. The only real functional change here was that we 
introduced the indirect-io region within the IO port space, under 
CONFIG_INDIRECT_PIO.

> 
>   > >>
>   > >> After dig into logic pio logic, I found that logic pio is trying to "allocate" an io_start
>   > >> for MMIO ranges, the allocation starts from 0x0. And later the io_start is used to calculate
>   > >> cpu_address.  In my opinion, for direct MMIO access, logic_pio address should always
>   > >> equal to hw address,
>   >
>   > I'm not sure what you mean by simply the hw address.
>   >
> 
> I meant  hw_start should always equal to io_start.
> 
> 
> MIPS have their own wrapped inl/outl functions, 

Can you please point me to these? I could not find them in arch/mips

I will also note that arch/mips/include/asm/io.h does not include 
asm-generic io.h today

doing the samething with
> PCI_IOBASE enabled one. I was just trying to use PCI_IOBASE instead.
> 
> Originally, the I/O ports layout seems like this:
> 
> 00000020-00000021 : pic1
> 00000060-0000006f : i8042
> 00000070-00000077 : rtc0
> 000000a0-000000a1 : pic2
> 00000170-00000177 : pata_atiixp
> 000001f0-000001f7 : pata_atiixp
> 00000376-00000376 : pata_atiixp
> 000003f6-000003f6 : pata_atiixp
> 00000800-000008ff : acpi
> 00001000-00001008 : piix4_smbus
> 00004000-0003ffff : pci io space
>    00004000-00004fff : PCI Bus 0000:01
>      00004000-000040ff : 0000:01:05.0
>    00005000-00005fff : PCI Bus 0000:03
>      00005000-0000501f : 0000:03:00.0
> 
> But with PCI_IOBASE defined, I got this:
> 
> host bridge /bus@10000000/pci@10000000 ranges:
>        MEM 0x0040000000..0x007fffffff -> 0x0040000000
>         IO 0x0000004000..0x0000007fff -> 0x0000004000
> resource collision: [io  0x0000-0x3fff] conflicts with pic1 [io  0x0020-0x0021]
> 
> Because io_start was allocated to 0x0 by Logic PIO.
> 
> There are a lot of devices that have fixed ioports thanks to x86's legacy.

Well, yes, I'm not so surprised.

So if MIPS does not have native IO port access, then surely you need 
some host bridge to translate host CPU MMIO accesses to port I/O 
accesses, right? Where are these CPU addresses defined?

> For example, in my hardware, ioports for RTC, PIC, I8042 are unmoveable,
> and they can't be managed by logic pio subsystem. > Also, the PCI Hostbridge got implied by DeviceTree that it's I/O range
> started from 0x4000 in bus side

which bus is this?

, but then, Logic PIO remapped to PCI_IOBASE + 0x0.
> The real address should be PCI_IOBASE + 0x4000,

You seem to be using two methods to manage IO port space, and they seem 
to be conflicting.

> hardware never got correctly informed about that. And there is still no way to
> transform to correct address as it's inside the MMIO_LIMIT.
> 
> So the question comes to why we're allocating io_start for MMIO PCI_IOBASE
> rather than just check the range provided doesn't overlap each other or exceed
> the MMIO_LIMIT.

When PCI_IOBASE is defined, we work on the basis that any IO port range 
in the system is registered for a logical PIO region, which manages the 
actual IO port addresses - see logic_pio_trans_cpuaddr().

Thanks,
John

Re: Questions about logic_pio

[Jiaxun Yang]

 ---- 在 星期四, 2020-02-20 22:23:57 John Garry <john.garry@huawei.com> 撰写 ----
 > > Also Cc MIPS list to check other's opinions.
 > > 
 > > Hi John.
 > > 
 > 
 > Hi Jiaxun Yang,
 > 
 > > Thanks for your kind explanation, however, I think this way is
 > > violating how I/O ports supposed to work, at least in MIPS world.
 > 
 > For a bit more history, please understand that the core PCI code was 
 > managing non-native IO port space in the same way before we added the 
 > logic PIO framework. The only real functional change here was that we 
 > introduced the indirect-io region within the IO port space, under 
 > CONFIG_INDIRECT_PIO.

I'm going to do more investigation. Thanks. 

 > 
 > > 
 > >   > >>
 > >   > >> After dig into logic pio logic, I found that logic pio is trying to "allocate" an io_start
 > >   > >> for MMIO ranges, the allocation starts from 0x0. And later the io_start is used to calculate
 > >   > >> cpu_address.  In my opinion, for direct MMIO access, logic_pio address should always
 > >   > >> equal to hw address,
 > >   >
 > >   > I'm not sure what you mean by simply the hw address.
 > >   >
 > > 
 > > I meant  hw_start should always equal to io_start.
 > > 
 > > 
 > > MIPS have their own wrapped inl/outl functions, 
 > 
 > Can you please point me to these? I could not find them in arch/mips

They are built by __BUILD_IOPORT_PFX(bus, bwlq, type) macro.
Just using mips_io_port_base + offset to handle inl/outl, the same way PCI_IOBASE.

 > 
 > I will also note that arch/mips/include/asm/io.h does not include 
 > asm-generic io.h today

Yes, and I'm attempting to take advantage of asm-generic.

 > 
 > doing the samething with
 > > PCI_IOBASE enabled one. I was just trying to use PCI_IOBASE instead.
 > > 
 > > Originally, the I/O ports layout seems like this:
 > > 
 > > 00000020-00000021 : pic1
 > > 00000060-0000006f : i8042
 > > 00000070-00000077 : rtc0
 > > 000000a0-000000a1 : pic2
 > > 00000170-00000177 : pata_atiixp
 > > 000001f0-000001f7 : pata_atiixp
 > > 00000376-00000376 : pata_atiixp
 > > 000003f6-000003f6 : pata_atiixp
 > > 00000800-000008ff : acpi
 > > 00001000-00001008 : piix4_smbus
 > > 00004000-0003ffff : pci io space
 > >    00004000-00004fff : PCI Bus 0000:01
 > >      00004000-000040ff : 0000:01:05.0
 > >    00005000-00005fff : PCI Bus 0000:03
 > >      00005000-0000501f : 0000:03:00.0
 > > 
 > > But with PCI_IOBASE defined, I got this:
 > > 
 > > host bridge /bus@10000000/pci@10000000 ranges:
 > >        MEM 0x0040000000..0x007fffffff -> 0x0040000000
 > >         IO 0x0000004000..0x0000007fff -> 0x0000004000
 > > resource collision: [io  0x0000-0x3fff] conflicts with pic1 [io  0x0020-0x0021]
 > > 
 > > Because io_start was allocated to 0x0 by Logic PIO.
 > > 
 > > There are a lot of devices that have fixed ioports thanks to x86's legacy.
 > 
 > Well, yes, I'm not so surprised.
 > 
 > So if MIPS does not have native IO port access, then surely you need 
 > some host bridge to translate host CPU MMIO accesses to port I/O 
 > accesses, right? Where are these CPU addresses defined?

It is defined by the variable mips_io_port_base.

 > 
 > > For example, in my hardware, ioports for RTC, PIC, I8042 are unmoveable,
 > > and they can't be managed by logic pio subsystem. > Also, the PCI Hostbridge got implied by DeviceTree that it's I/O range
 > > started from 0x4000 in bus side
 > 
 > which bus is this?

They're all located under "ISA Range".  Just an MMIO range that will resend
the request to ISA I/O. --ioports for both PCI and some legacy devices.

In that range, base + 0x0000 to 0x4000 is preserved for PIO devices (e.g.) I8259
and base + 0x4000 to MMIO_LIMIT are for PCI devices under host bridge.
For the host bridge, ioports it can decode starts from 0x4000.

My intentional behavior is that when I'm specifying in dts that the IO Range of PCI host
bridge is 0x4000 to 0x7fff, it would request the IO_RESOURCE start from 0x4000
to 0x7fff, also tell the host driver to decode  0x4000 to 0x7fff in IO BAR, And let the drivers
access 0x4000 to 0x7fff via inl/outl, rather than allocate from PIO 0x0 to 0x3fff.

 > 
 > , but then, Logic PIO remapped to PCI_IOBASE + 0x0.
 > > The real address should be PCI_IOBASE + 0x4000,
 > 
 > You seem to be using two methods to manage IO port space, and they seem 
 > to be conflicting.

So... Are there any way to handle these unmoveable devices in logic pio world?

 > 
 > > hardware never got correctly informed about that. And there is still no way to
 > > transform to correct address as it's inside the MMIO_LIMIT.
 > > 
 > > So the question comes to why we're allocating io_start for MMIO PCI_IOBASE
 > > rather than just check the range provided doesn't overlap each other or exceed
 > > the MMIO_LIMIT.
 > 
 > When PCI_IOBASE is defined, we work on the basis that any IO port range 
 > in the system is registered for a logical PIO region, which manages the 
 > actual IO port addresses - see logic_pio_trans_cpuaddr().

The port is not the actual port.. It makes me confusing about what it's actually doing..
Sorry but probably I'm still thinking in a vintage way -- need some hints about how to
deal with these legacy cases in a modern way.

Thanks.

 > 
 > Thanks,
 > John
 >

Re: Questions about logic_pio

[John Garry]

On 20/02/2020 15:12, Jiaxun Yang wrote:
> 
>   ---- 在 星期四, 2020-02-20 22:23:57 John Garry <john.garry@huawei.com> 撰写 ----
>   > > Also Cc MIPS list to check other's opinions.
>   > >
>   > > Hi John.
>   > >
>   >
>   > Hi Jiaxun Yang,
>   >
>   > > Thanks for your kind explanation, however, I think this way is
>   > > violating how I/O ports supposed to work, at least in MIPS world.
>   >
>   > For a bit more history, please understand that the core PCI code was
>   > managing non-native IO port space in the same way before we added the
>   > logic PIO framework. The only real functional change here was that we
>   > introduced the indirect-io region within the IO port space, under
>   > CONFIG_INDIRECT_PIO.
> 
> I'm going to do more investigation. Thanks.
> 
>   >
>   > >
>   > >   > >>
>   > >   > >> After dig into logic pio logic, I found that logic pio is trying to "allocate" an io_start
>   > >   > >> for MMIO ranges, the allocation starts from 0x0. And later the io_start is used to calculate
>   > >   > >> cpu_address.  In my opinion, for direct MMIO access, logic_pio address should always
>   > >   > >> equal to hw address,
>   > >   >
>   > >   > I'm not sure what you mean by simply the hw address.
>   > >   >
>   > >
>   > > I meant  hw_start should always equal to io_start.
>   > >
>   > >
>   > > MIPS have their own wrapped inl/outl functions,
>   >
>   > Can you please point me to these? I could not find them in arch/mips
> 
> They are built by __BUILD_IOPORT_PFX(bus, bwlq, type) macro.
> Just using mips_io_port_base + offset to handle inl/outl, the same way PCI_IOBASE.

Right, so I had a glance through the code and mips has it own management 
of this IO port space. And, like you say, mips_io_port_base is 
equivalent to PCI_IOBASE.

> 
>   >
>   > I will also note that arch/mips/include/asm/io.h does not include
>   > asm-generic io.h today
> 
> Yes, and I'm attempting to take advantage of asm-generic.

I just don't think it's as simple as saying we want to take advantage of 
asm-generic. asm-generic io.h includes logic_pio.h, which uses logical 
PIO to manage IO port space and relies on PIO_IOBASE. This is 
incompatible with having some other framework - like mips_io_port_base - 
managing IO port space at the same time.

The core PCI code relies on logical PIO to manage IO port space for when 
PCI_IOBASE is defined.

> 
>   >
>   > doing the samething with
>   > > PCI_IOBASE enabled one. I was just trying to use PCI_IOBASE instead.
>   > >
>   > > Originally, the I/O ports layout seems like this:
>   > >
>   > > 00000020-00000021 : pic1
>   > > 00000060-0000006f : i8042
>   > > 00000070-00000077 : rtc0
>   > > 000000a0-000000a1 : pic2
>   > > 00000170-00000177 : pata_atiixp
>   > > 000001f0-000001f7 : pata_atiixp
>   > > 00000376-00000376 : pata_atiixp
>   > > 000003f6-000003f6 : pata_atiixp
>   > > 00000800-000008ff : acpi
>   > > 00001000-00001008 : piix4_smbus
>   > > 00004000-0003ffff : pci io space
>   > >    00004000-00004fff : PCI Bus 0000:01
>   > >      00004000-000040ff : 0000:01:05.0
>   > >    00005000-00005fff : PCI Bus 0000:03
>   > >      00005000-0000501f : 0000:03:00.0
>   > >
>   > > But with PCI_IOBASE defined, I got this:
>   > >
>   > > host bridge /bus@10000000/pci@10000000 ranges:
>   > >        MEM 0x0040000000..0x007fffffff -> 0x0040000000
>   > >         IO 0x0000004000..0x0000007fff -> 0x0000004000
>   > > resource collision: [io  0x0000-0x3fff] conflicts with pic1 [io  0x0020-0x0021]
>   > >
>   > > Because io_start was allocated to 0x0 by Logic PIO.
>   > >
>   > > There are a lot of devices that have fixed ioports thanks to x86's legacy.
>   >
>   > Well, yes, I'm not so surprised.
>   >
>   > So if MIPS does not have native IO port access, then surely you need
>   > some host bridge to translate host CPU MMIO accesses to port I/O
>   > accesses, right? Where are these CPU addresses defined?
> 
> It is defined by the variable mips_io_port_base.
> 
>   >
>   > > For example, in my hardware, ioports for RTC, PIC, I8042 are unmoveable,
>   > > and they can't be managed by logic pio subsystem. > Also, the PCI Hostbridge got implied by DeviceTree that it's I/O range
>   > > started from 0x4000 in bus side
>   >
>   > which bus is this?
> 
> They're all located under "ISA Range".  Just an MMIO range that will resend
> the request to ISA I/O. --ioports for both PCI and some legacy devices.
> 
> In that range, base + 0x0000 to 0x4000 is preserved for PIO devices (e.g.) I8259
> and base + 0x4000 to MMIO_LIMIT are for PCI devices under host bridge.
> For the host bridge, ioports it can decode starts from 0x4000.
> 
> My intentional behavior is that when I'm specifying in dts that the IO Range of PCI host
> bridge is 0x4000 to 0x7fff, it would request the IO_RESOURCE start from 0x4000
> to 0x7fff, also tell the host driver to decode  0x4000 to 0x7fff in IO BAR, And let the drivers
> access 0x4000 to 0x7fff via inl/outl, rather than allocate from PIO 0x0 to 0x3fff.
> 
>   >
>   > , but then, Logic PIO remapped to PCI_IOBASE + 0x0.
>   > > The real address should be PCI_IOBASE + 0x4000,
>   >
>   > You seem to be using two methods to manage IO port space, and they seem
>   > to be conflicting.
> 
> So... Are there any way to handle these unmoveable devices in logic pio world?

When you say that they are unmovable, they are at a fixed address on 
this "ISA Range", right? If so, yes, you should be able to handle it in 
logical PIO. You just need to deal with translating logical PIO 
addresses to ISA bus addresses. We do this very thing in our LPC driver 
- see drivers/bus/hisi_lpc.c

This driver deals with legacy IO ports where we need to bitbang 
accesses, i.e. we don't support MMIO for this.
> 
>   >
>   > > hardware never got correctly informed about that. And there is still no way to
>   > > transform to correct address as it's inside the MMIO_LIMIT.
>   > >
>   > > So the question comes to why we're allocating io_start for MMIO PCI_IOBASE
>   > > rather than just check the range provided doesn't overlap each other or exceed
>   > > the MMIO_LIMIT.
>   >
>   > When PCI_IOBASE is defined, we work on the basis that any IO port range
>   > in the system is registered for a logical PIO region, which manages the
>   > actual IO port addresses - see logic_pio_trans_cpuaddr().
> 
> The port is not the actual port.. It makes me confusing about what it's actually doing..
> Sorry but probably I'm still thinking in a vintage way -- need some hints about how to
> deal with these legacy cases in a modern way.
> 
> Thanks.
> 
>   >
>   > Thanks,
>   > John
>   >
> 

Re: Questions about logic_pio

[John Garry]

On 20/02/2020 17:39, John Garry wrote:
> On 20/02/2020 15:12, Jiaxun Yang wrote:
>>
>>   ---- 在 星期四, 2020-02-20 22:23:57 John Garry 
>> <john.garry@huawei.com> 撰写 ----
>>   > > Also Cc MIPS list to check other's opinions.
>>   > >
>>   > > Hi John.
>>   > >
>>   >
>>   > Hi Jiaxun Yang,
>>   >
>>   > > Thanks for your kind explanation, however, I think this way is
>>   > > violating how I/O ports supposed to work, at least in MIPS world.
>>   >
>>   > For a bit more history, please understand that the core PCI code was
>>   > managing non-native IO port space in the same way before we added the
>>   > logic PIO framework. The only real functional change here was that we
>>   > introduced the indirect-io region within the IO port space, under
>>   > CONFIG_INDIRECT_PIO.
>>
>> I'm going to do more investigation. Thanks.
>>
>>   >
>>   > >
>>   > >   > >>
>>   > >   > >> After dig into logic pio logic, I found that logic pio is 
>> trying to "allocate" an io_start
>>   > >   > >> for MMIO ranges, the allocation starts from 0x0. And 
>> later the io_start is used to calculate
>>   > >   > >> cpu_address.  In my opinion, for direct MMIO access, 
>> logic_pio address should always
>>   > >   > >> equal to hw address,
>>   > >   >
>>   > >   > I'm not sure what you mean by simply the hw address.
>>   > >   >
>>   > >
>>   > > I meant  hw_start should always equal to io_start.
>>   > >
>>   > >
>>   > > MIPS have their own wrapped inl/outl functions,
>>   >
>>   > Can you please point me to these? I could not find them in arch/mips
>>
>> They are built by __BUILD_IOPORT_PFX(bus, bwlq, type) macro.
>> Just using mips_io_port_base + offset to handle inl/outl, the same way 
>> PCI_IOBASE.
> 
> Right, so I had a glance through the code and mips has it own management 
> of this IO port space. And, like you say, mips_io_port_base is 
> equivalent to PCI_IOBASE.
> 
>>
>>   >
>>   > I will also note that arch/mips/include/asm/io.h does not include
>>   > asm-generic io.h today
>>
>> Yes, and I'm attempting to take advantage of asm-generic.
> 
> I just don't think it's as simple as saying we want to take advantage of 
> asm-generic. asm-generic io.h includes logic_pio.h, which uses logical 
> PIO to manage IO port space and relies on PIO_IOBASE. This is 
> incompatible with having some other framework - like mips_io_port_base - 
> managing IO port space at the same time.
> 
> The core PCI code relies on logical PIO to manage IO port space for when 
> PCI_IOBASE is defined.
> 
>>
>>   >
>>   > doing the samething with
>>   > > PCI_IOBASE enabled one. I was just trying to use PCI_IOBASE 
>> instead.
>>   > >
>>   > > Originally, the I/O ports layout seems like this:
>>   > >
>>   > > 00000020-00000021 : pic1
>>   > > 00000060-0000006f : i8042
>>   > > 00000070-00000077 : rtc0
>>   > > 000000a0-000000a1 : pic2
>>   > > 00000170-00000177 : pata_atiixp
>>   > > 000001f0-000001f7 : pata_atiixp
>>   > > 00000376-00000376 : pata_atiixp
>>   > > 000003f6-000003f6 : pata_atiixp
>>   > > 00000800-000008ff : acpi
>>   > > 00001000-00001008 : piix4_smbus
>>   > > 00004000-0003ffff : pci io space
>>   > >    00004000-00004fff : PCI Bus 0000:01
>>   > >      00004000-000040ff : 0000:01:05.0
>>   > >    00005000-00005fff : PCI Bus 0000:03
>>   > >      00005000-0000501f : 0000:03:00.0
>>   > >
>>   > > But with PCI_IOBASE defined, I got this:
>>   > >
>>   > > host bridge /bus@10000000/pci@10000000 ranges:
>>   > >        MEM 0x0040000000..0x007fffffff -> 0x0040000000
>>   > >         IO 0x0000004000..0x0000007fff -> 0x0000004000
>>   > > resource collision: [io  0x0000-0x3fff] conflicts with pic1 [io  
>> 0x0020-0x0021]
>>   > >
>>   > > Because io_start was allocated to 0x0 by Logic PIO.
>>   > >
>>   > > There are a lot of devices that have fixed ioports thanks to 
>> x86's legacy.
>>   >
>>   > Well, yes, I'm not so surprised.
>>   >
>>   > So if MIPS does not have native IO port access, then surely you need
>>   > some host bridge to translate host CPU MMIO accesses to port I/O
>>   > accesses, right? Where are these CPU addresses defined?
>>
>> It is defined by the variable mips_io_port_base.
>>
>>   >
>>   > > For example, in my hardware, ioports for RTC, PIC, I8042 are 
>> unmoveable,
>>   > > and they can't be managed by logic pio subsystem. > Also, the 
>> PCI Hostbridge got implied by DeviceTree that it's I/O range
>>   > > started from 0x4000 in bus side
>>   >
>>   > which bus is this?
>>
>> They're all located under "ISA Range".  Just an MMIO range that will 
>> resend
>> the request to ISA I/O. --ioports for both PCI and some legacy devices.
>>
>> In that range, base + 0x0000 to 0x4000 is preserved for PIO devices 
>> (e.g.) I8259
>> and base + 0x4000 to MMIO_LIMIT are for PCI devices under host bridge.
>> For the host bridge, ioports it can decode starts from 0x4000.
>>
>> My intentional behavior is that when I'm specifying in dts that the IO 
>> Range of PCI host
>> bridge is 0x4000 to 0x7fff, it would request the IO_RESOURCE start 
>> from 0x4000
>> to 0x7fff, also tell the host driver to decode  0x4000 to 0x7fff in IO 
>> BAR, And let the drivers
>> access 0x4000 to 0x7fff via inl/outl, rather than allocate from PIO 
>> 0x0 to 0x3fff.
>>
>>   >
>>   > , but then, Logic PIO remapped to PCI_IOBASE + 0x0.
>>   > > The real address should be PCI_IOBASE + 0x4000,
>>   >
>>   > You seem to be using two methods to manage IO port space, and they 
>> seem
>>   > to be conflicting.
>>
>> So... Are there any way to handle these unmoveable devices in logic 
>> pio world?
> 
> When you say that they are unmovable, they are at a fixed address on 
> this "ISA Range", right? If so, yes, you should be able to handle it in 
> logical PIO. You just need to deal with translating logical PIO 
> addresses to ISA bus addresses. We do this very thing in our LPC driver 
> - see drivers/bus/hisi_lpc.c

I will add this may not cover your need, as you probably cannot deal 
with any logical PIO <-> ISA translation without modifying the device 
driver. For this, we may need to reserve the first 0x4000 in logical PIO 
space for this sort of legacy host.

That would not be a bad thing - see 
https://lore.kernel.org/linux-pci/1560262374-67875-1-git-send-email-john.garry@huawei.com/


> 
> This driver deals with legacy IO ports where we need to bitbang 
> accesses, i.e. we don't support MMIO for this.
>>
>>   >
>>   > > hardware never got correctly informed about that. And there is 
>> still no way to
>>   > > transform to correct address as it's inside the MMIO_LIMIT.
>>   > >
>>   > > So the question comes to why we're allocating io_start for MMIO 
>> PCI_IOBASE
>>   > > rather than just check the range provided doesn't overlap each 
>> other or exceed
>>   > > the MMIO_LIMIT.
>>   >
>>   > When PCI_IOBASE is defined, we work on the basis that any IO port 
>> range
>>   > in the system is registered for a logical PIO region, which 
>> manages the
>>   > actual IO port addresses - see logic_pio_trans_cpuaddr().
>>
>> The port is not the actual port.. It makes me confusing about what 
>> it's actually doing..
>> Sorry but probably I'm still thinking in a vintage way -- need some 
>> hints about how to
>> deal with these legacy cases in a modern way.
>>
>> Thanks.
>>
>>   >
>>   > Thanks,
>>   > John
>>   >
>>
> 
> 

Re: Questions about logic_pio

[Jiaxun Yang]

 ---- 在 星期五, 2020-02-21 02:32:39 John Garry <john.garry@huawei.com> 撰写 ----
 > On 20/02/2020 17:39, John Garry wrote:
 > > On 20/02/2020 15:12, Jiaxun Yang wrote:
 > >>
 > >>   ---- 在 星期四, 2020-02-20 22:23:57 John Garry 
 > >> <john.garry@huawei.com> 撰写 ----
 > >>   > > Also Cc MIPS list to check other's opinions.
 > >>   > >
 > >>   > > Hi John.
 > >>   > >
 > >>   >
 > >>   > Hi Jiaxun Yang,
 > >>   >
 > >>   > > Thanks for your kind explanation, however, I think this way is
 > >>   > > violating how I/O ports supposed to work, at least in MIPS world.
 > >>   >
 > >>   > For a bit more history, please understand that the core PCI code was
 > >>   > managing non-native IO port space in the same way before we added the
 > >>   > logic PIO framework. The only real functional change here was that we
 > >>   > introduced the indirect-io region within the IO port space, under
 > >>   > CONFIG_INDIRECT_PIO.
 > >>
 > >> I'm going to do more investigation. Thanks.
 > >>
 > >>   >
 > >>   > >
 > >>   > >   > >>
 > >>   > >   > >> After dig into logic pio logic, I found that logic pio is 
 > >> trying to "allocate" an io_start
 > >>   > >   > >> for MMIO ranges, the allocation starts from 0x0. And 
 > >> later the io_start is used to calculate
 > >>   > >   > >> cpu_address.  In my opinion, for direct MMIO access, 
 > >> logic_pio address should always
 > >>   > >   > >> equal to hw address,
 > >>   > >   >
 > >>   > >   > I'm not sure what you mean by simply the hw address.
 > >>   > >   >
 > >>   > >
 > >>   > > I meant  hw_start should always equal to io_start.
 > >>   > >
 > >>   > >
 > >>   > > MIPS have their own wrapped inl/outl functions,
 > >>   >
 > >>   > Can you please point me to these? I could not find them in arch/mips
 > >>
 > >> They are built by __BUILD_IOPORT_PFX(bus, bwlq, type) macro.
 > >> Just using mips_io_port_base + offset to handle inl/outl, the same way 
 > >> PCI_IOBASE.
 > > 
 > > Right, so I had a glance through the code and mips has it own management 
 > > of this IO port space. And, like you say, mips_io_port_base is 
 > > equivalent to PCI_IOBASE.
 > > 
 > >>
 > >>   >
 > >>   > I will also note that arch/mips/include/asm/io.h does not include
 > >>   > asm-generic io.h today
 > >>
 > >> Yes, and I'm attempting to take advantage of asm-generic.
 > > 
 > > I just don't think it's as simple as saying we want to take advantage of 
 > > asm-generic. asm-generic io.h includes logic_pio.h, which uses logical 
 > > PIO to manage IO port space and relies on PIO_IOBASE. This is 
 > > incompatible with having some other framework - like mips_io_port_base - 
 > > managing IO port space at the same time.
 > > 
 > > The core PCI code relies on logical PIO to manage IO port space for when 
 > > PCI_IOBASE is defined.
 > > 
 > >>
 > >>   >
 > >>   > doing the samething with
 > >>   > > PCI_IOBASE enabled one. I was just trying to use PCI_IOBASE 
 > >> instead.
 > >>   > >
 > >>   > > Originally, the I/O ports layout seems like this:
 > >>   > >
 > >>   > > 00000020-00000021 : pic1
 > >>   > > 00000060-0000006f : i8042
 > >>   > > 00000070-00000077 : rtc0
 > >>   > > 000000a0-000000a1 : pic2
 > >>   > > 00000170-00000177 : pata_atiixp
 > >>   > > 000001f0-000001f7 : pata_atiixp
 > >>   > > 00000376-00000376 : pata_atiixp
 > >>   > > 000003f6-000003f6 : pata_atiixp
 > >>   > > 00000800-000008ff : acpi
 > >>   > > 00001000-00001008 : piix4_smbus
 > >>   > > 00004000-0003ffff : pci io space
 > >>   > >    00004000-00004fff : PCI Bus 0000:01
 > >>   > >      00004000-000040ff : 0000:01:05.0
 > >>   > >    00005000-00005fff : PCI Bus 0000:03
 > >>   > >      00005000-0000501f : 0000:03:00.0
 > >>   > >
 > >>   > > But with PCI_IOBASE defined, I got this:
 > >>   > >
 > >>   > > host bridge /bus@10000000/pci@10000000 ranges:
 > >>   > >        MEM 0x0040000000..0x007fffffff -> 0x0040000000
 > >>   > >         IO 0x0000004000..0x0000007fff -> 0x0000004000
 > >>   > > resource collision: [io  0x0000-0x3fff] conflicts with pic1 [io  
 > >> 0x0020-0x0021]
 > >>   > >
 > >>   > > Because io_start was allocated to 0x0 by Logic PIO.
 > >>   > >
 > >>   > > There are a lot of devices that have fixed ioports thanks to 
 > >> x86's legacy.
 > >>   >
 > >>   > Well, yes, I'm not so surprised.
 > >>   >
 > >>   > So if MIPS does not have native IO port access, then surely you need
 > >>   > some host bridge to translate host CPU MMIO accesses to port I/O
 > >>   > accesses, right? Where are these CPU addresses defined?
 > >>
 > >> It is defined by the variable mips_io_port_base.
 > >>
 > >>   >
 > >>   > > For example, in my hardware, ioports for RTC, PIC, I8042 are 
 > >> unmoveable,
 > >>   > > and they can't be managed by logic pio subsystem. > Also, the 
 > >> PCI Hostbridge got implied by DeviceTree that it's I/O range
 > >>   > > started from 0x4000 in bus side
 > >>   >
 > >>   > which bus is this?
 > >>
 > >> They're all located under "ISA Range".  Just an MMIO range that will 
 > >> resend
 > >> the request to ISA I/O. --ioports for both PCI and some legacy devices.
 > >>
 > >> In that range, base + 0x0000 to 0x4000 is preserved for PIO devices 
 > >> (e.g.) I8259
 > >> and base + 0x4000 to MMIO_LIMIT are for PCI devices under host bridge.
 > >> For the host bridge, ioports it can decode starts from 0x4000.
 > >>
 > >> My intentional behavior is that when I'm specifying in dts that the IO 
 > >> Range of PCI host
 > >> bridge is 0x4000 to 0x7fff, it would request the IO_RESOURCE start 
 > >> from 0x4000
 > >> to 0x7fff, also tell the host driver to decode  0x4000 to 0x7fff in IO 
 > >> BAR, And let the drivers
 > >> access 0x4000 to 0x7fff via inl/outl, rather than allocate from PIO 
 > >> 0x0 to 0x3fff.
 > >>
 > >>   >
 > >>   > , but then, Logic PIO remapped to PCI_IOBASE + 0x0.
 > >>   > > The real address should be PCI_IOBASE + 0x4000,
 > >>   >
 > >>   > You seem to be using two methods to manage IO port space, and they 
 > >> seem
 > >>   > to be conflicting.
 > >>
 > >> So... Are there any way to handle these unmoveable devices in logic 
 > >> pio world?
 > > 
 > > When you say that they are unmovable, they are at a fixed address on 
 > > this "ISA Range", right? If so, yes, you should be able to handle it in 
 > > logical PIO. You just need to deal with translating logical PIO 
 > > addresses to ISA bus addresses. We do this very thing in our LPC driver 
 > > - see drivers/bus/hisi_lpc.c
 > 
 > I will add this may not cover your need, as you probably cannot deal 
 > with any logical PIO <-> ISA translation without modifying the device 
 > driver. For this, we may need to reserve the first 0x4000 in logical PIO 
 > space for this sort of legacy host.

Hi,

After thinking twice, I realized that the most convenient way for me is
adding an option to get rid of the mess of logic PIO. MIPS system is emulating
x86's behavior, while logic PIO isn't designed for such platform.

Or probably I need a variation of Logic PIO, which leave MMIO space AS-IS
(not try to reallocate it) but still preserving higher 0x4000 for indirect access.

Thanks a lot!
 > 
 > That would not be a bad thing - see 
 > https://lore.kernel.org/linux-pci/1560262374-67875-1-git-send-email-john.garry@huawei.com/
 > 
 > 
 > > 
 > > This driver deals with legacy IO ports where we need to bitbang 
 > > accesses, i.e. we don't support MMIO for this.
 > >>
 > 

Re: Questions about logic_pio

[John Garry]

On 21/02/2020 00:42, Jiaxun Yang wrote:
>   >
>   > I will add this may not cover your need, as you probably cannot deal
>   > with any logical PIO <-> ISA translation without modifying the device
>   > driver. For this, we may need to reserve the first 0x4000 in logical PIO
>   > space for this sort of legacy host.
> 
> Hi,
> 
> After thinking twice, I realized that the most convenient way for me is
> adding an option to get rid of the mess of logic PIO. MIPS system is emulating
> x86's behavior, while logic PIO isn't designed for such platform.

It was designed for archs which define PCI_IOBASE for PCI MMIO-based or 
IndirectIO-based IO port access.

> 
> Or probably I need a variation of Logic PIO, which leave MMIO space AS-IS
> (not try to reallocate it)

That does not work if add a PCI host with MMIO-based IO port regions 
into the mix.

It only so happens today that for mips you have a single MMIO-based IO 
port region, and you have IO port base for that region conveniently @ 
0x0. Then your drivers can have fixed IO port addresses.

For dealing with multiple MMIO-based IO ports regions - which is the 
case for PCI host bridges - then you need to map those MMIO-based IO 
port regions to different regions in IO port space.

> but still preserving higher 0x4000 for indirect access.

Then there is no space for PCI MMIO-based IO ports.

> 
> Thanks a lot!
>   >
>   > That would not be a bad thing - see
>   >https://lore.kernel.org/linux-pci/1560262374-67875-1-git-send-email-john.garry@huawei.com/
>   >
>   >
>   > >
>   > > This driver deals with legacy IO ports where we need to bitbang

Re: Questions about logic_pio

[Jiaxun Yang]

于 2020年2月21日 GMT+08:00 下午7:49:45, John Garry <john.garry@huawei.com> 写到:
>On 21/02/2020 00:42, Jiaxun Yang wrote:
>>   >
>>   > I will add this may not cover your need, as you probably cannot
>deal
>>   > with any logical PIO <-> ISA translation without modifying the
>device
>>   > driver. For this, we may need to reserve the first 0x4000 in
>logical PIO
>>   > space for this sort of legacy host.
>> 
>> Hi,
>> 
>> After thinking twice, I realized that the most convenient way for me
>is
>> adding an option to get rid of the mess of logic PIO. MIPS system is
>emulating
>> x86's behavior, while logic PIO isn't designed for such platform.
>
>It was designed for archs which define PCI_IOBASE for PCI MMIO-based or
>
>IndirectIO-based IO port access.
>
>> 
>> Or probably I need a variation of Logic PIO, which leave MMIO space
>AS-IS
>> (not try to reallocate it)
>
>That does not work if add a PCI host with MMIO-based IO port regions 
>into the mix.
>
>It only so happens today that for mips you have a single MMIO-based IO 
>port region, and you have IO port base for that region conveniently @ 
>0x0. Then your drivers can have fixed IO port addresses.

In MIPS, PCI Host Bridge share PIO region with legacy devices.

Probably the better solution is to add a kind of logic pio region called FIXED_MMIO,
so we can occupy some low ioports for these fixed devices at boot time.

A ISA bridge node in DeviceTree can be used
to express this region.

I'm going to work on this and try to make a RFC patch later.

Thanks for your help again!
>
>For dealing with multiple MMIO-based IO ports regions - which is the 
>case for PCI host bridges - then you need to map those MMIO-based IO 
>port regions to different regions in IO port space.
>
>> but still preserving higher 0x4000 for indirect access.
>
>Then there is no space for PCI MMIO-based IO ports.

Well, it can be higher or lower, whatever.

>
>> 
>> Thanks a lot!
>>   >
>>   > That would not be a bad thing - see
>>  
>>https://lore.kernel.org/linux-pci/1560262374-67875-1-git-send-email-john.garry@huawei.com/
>>   >
>>   >
>>   > >
>>   > > This driver deals with legacy IO ports where we need to bitbang

-- 
Jiaxun Yang