can't access PCIe card under sbc8548

can't access PCIe card under sbc8548

[wolfking]

hi, all
  I'm doing some developing on the windriver's sbc8548 board. The kernel I
use
is 3.6.10 and the u-boot version is 2012-10. I changed the board's
configuration:
the board now boot from the 64MB SODIMM Flash (not the default 8MB on-board
Flash
memory), and the PCI clock rate is changed to 33MHZ.
  Now the trouble I am in is that: the PCI card (a NIC card rtl8139) can be 
accessed OK, while the PCIe card can't work, that is, the kernel can't
access
its internal register. The kernel can correctly probe the PCIe card. its
BAR0 
is a I/O mapped register, I use ioport_map to map the BAR0 to kernel's
address
space, then use ioread8/iowrite8 to access its internal register, it doesn't
work.
I analyse the ioport_map function and find it just add the input parameter
to
a fixed _IO_BASE value, below is the function:
void __iomem *ioport_map(unsigned long port, unsigned int len)
{
	return (void __iomem *) (port + _IO_BASE);
}
the _IO_BASE value under sbc8548 is 0xfd7fd000, the value of ioport_map
paramenter 
"port" is 0xfefff000. Obviously the add overflows, so the follow-up
operations 
can't succeed. The value of "port" is got from the function
pci_resource_start.
So I guess the kernel allocate a bad address to this PCIe card. How can I
fix this?
  I also plug this PCIe card into a freescale's board mpc8641-hpcn, try the
same driver,
I noticed ioport_map also meet overflow, but it does work fine. So I'm
messed, :<



--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.

Re: can't access PCIe card under sbc8548

[tiejun.chen]

On 05/30/2013 11:42 AM, wolfking wrote:
> hi, all
>    I'm doing some developing on the windriver's sbc8548 board. The kernel I
> use
> is 3.6.10 and the u-boot version is 2012-10. I changed the board's
> configuration:
> the board now boot from the 64MB SODIMM Flash (not the default 8MB on-board
> Flash
> memory), and the PCI clock rate is changed to 33MHZ.
>    Now the trouble I am in is that: the PCI card (a NIC card rtl8139) can be
> accessed OK, while the PCIe card can't work, that is, the kernel can't
> access
> its internal register. The kernel can correctly probe the PCIe card. its
> BAR0
> is a I/O mapped register, I use ioport_map to map the BAR0 to kernel's
> address
> space, then use ioread8/iowrite8 to access its internal register, it doesn't
> work.
> I analyse the ioport_map function and find it just add the input parameter
> to
> a fixed _IO_BASE value, below is the function:
> void __iomem *ioport_map(unsigned long port, unsigned int len)
> {
> 	return (void __iomem *) (port + _IO_BASE);
> }
> the _IO_BASE value under sbc8548 is 0xfd7fd000, the value of ioport_map
> paramenter
> "port" is 0xfefff000. Obviously the add overflows, so the follow-up
> operations
> can't succeed. The value of "port" is got from the function
> pci_resource_start.

In PPC case I/O is memory-mapped, so you should use ioremap() instead of 
ioport_map()..

Tiejun

Re: can't access PCIe card under sbc8548

[wolfking]

hi, tiejun.chen:
  Thanks for replying.
  I tried to use ioremap too, but it doesn't work. The ioport_map method
succeeded
under mpc8641d which is also a ppc.
  In fact, I also tried using pci_iomap by imitating rtl8139's codes. The
rtl8139's BAR0 
is also a I/O mapped register and its code succeeded in accessing its
registers by 
using pci_iomap under sbc8548. But when I used pci_iomap for my PCIe card,
its
output is just the same with the method of using ioport_map. Any suggestion?



--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775p71782.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.

Re: can't access PCIe card under sbc8548

[tiejun.chen]

On 05/30/2013 03:19 PM, wolfking wrote:
> hi, tiejun.chen:
>    Thanks for replying.
>    I tried to use ioremap too, but it doesn't work. The ioport_map method

What happened?

Tiejun

Re: can't access PCIe card under sbc8548

[wolfking]

hi, tiejun.chen:
  When I use ioremap, the card seems to work fine. That is, I can access
part of all register. My PCIe card is a rs232 expand card, it has some
standard UART register, for example the SCR(scratch register). My driver 
can access the SCR(write and read) normally, but the other registers
behave odd. For example, the DLM should be 0, but it reads 5. The card
has a software reset bit, when it is set to 1, the card reset itself. When
it finished reset, this reset bit should be back to 0. But In sbc8548, when
I set this
bit, it remains high. So I guess, the area I accessed is not the PCIe card,
instead it maybe some RAM in the system. :>
I'm sure the card hardware is OK, I insert it into the 8641d board, it works
ok.



--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775p71815.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.

Re: can't access PCIe card under sbc8548

[tiejun.chen]

On 05/30/2013 05:15 PM, wolfking wrote:
> hi, tiejun.chen:
>    When I use ioremap, the card seems to work fine. That is, I can access

Yes, ioremap() should work for MMIO.

> part of all register. My PCIe card is a rs232 expand card, it has some
> standard UART register, for example the SCR(scratch register). My driver
> can access the SCR(write and read) normally, but the other registers
> behave odd. For example, the DLM should be 0, but it reads 5. The card
> has a software reset bit, when it is set to 1, the card reset itself. When
> it finished reset, this reset bit should be back to 0. But In sbc8548, when
> I set this
> bit, it remains high. So I guess, the area I accessed is not the PCIe card,

I suspect you're missing some load/storage sync code, so what is your R/W 
function exactly?

Tiejun

Re: can't access PCIe card under sbc8548

[wolfking]

(continued)
  I traced the 8139too.c when it uses pci_iomap, the pci_iomap called the 
ioport_map. The difference between 8139 and my PCIe card lies in the
"port" value :
void __iomem *ioport_map(unsigned long port, unsigned int len)
{
	return (void __iomem *) (port + _IO_BASE);
}
  in 8139too.c, the "port" value is 0x1000; for my PCIe card, the "port"
value
is 0xfefff000. And the value is got from pci_resource_start. So you see, the 
8139 case doesn't overflow, my PICe overflows when _IO_BASE's value is 
0xfd7fd000.



--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775p71783.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.

Re: can't access PCIe card under sbc8548

[tiejun.chen]

On 05/30/2013 03:32 PM, wolfking wrote:
> (continued)
>    I traced the 8139too.c when it uses pci_iomap, the pci_iomap called the
> ioport_map. The difference between 8139 and my PCIe card lies in the
> "port" value :
> void __iomem *ioport_map(unsigned long port, unsigned int len)
> {
> 	return (void __iomem *) (port + _IO_BASE);

_IO_BASE is equal to isa_io_base. So if this is not zero, I think there's a isa 
bridge in your platform. So you can access these I/O ports based on that isa 
bridge/bus with ioreadx/iowritex.

> }
>    in 8139too.c, the "port" value is 0x1000; for my PCIe card, the "port"
> value
> is 0xfefff000. And the value is got from pci_resource_start. So you see, the

But this means the port is as memory-mapped so ioremap() should be workable in 
this case. Then out_bex/in_bex should be fine.

Tiejun

Re: can't access PCIe card under sbc8548

[wolfking]

tiejun.chen wrote
> On 05/30/2013 03:32 PM, wolfking wrote:
>> (continued)
>>    I traced the 8139too.c when it uses pci_iomap, the pci_iomap called
>> the
>> ioport_map. The difference between 8139 and my PCIe card lies in the
>> "port" value :
>> void __iomem *ioport_map(unsigned long port, unsigned int len)
>> {
>> 	return (void __iomem *) (port + _IO_BASE);
> 
> _IO_BASE is equal to isa_io_base. So if this is not zero, I think there's
> a isa 
> bridge in your platform. So you can access these I/O ports based on that
> isa 
> bridge/bus with ioreadx/iowritex.
> 
> I tried ioread8/iowriet8 after ioremap, it doesn't work
> 
>> }
>>    in 8139too.c, the "port" value is 0x1000; for my PCIe card, the "port"
>> value
>> is 0xfefff000. And the value is got from pci_resource_start. So you see,
>> the
> 
> But this means the port is as memory-mapped so ioremap() should be
> workable in 
> this case. Then out_bex/in_bex should be fine.
> 
> Tiejun
> 
> _______________________________________________
> Linuxppc-dev mailing list

> Linuxppc-dev@.ozlabs

> https://lists.ozlabs.org/listinfo/linuxppc-dev





--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775p71827.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.

Re: can't access PCIe card under sbc8548

[Scott Wood]

On 05/30/2013 07:49:48 AM, wolfking wrote:
> tiejun.chen wrote
> > On 05/30/2013 03:32 PM, wolfking wrote:
> >> (continued)
> >>    I traced the 8139too.c when it uses pci_iomap, the pci_iomap =20
> called
> >> the
> >> ioport_map. The difference between 8139 and my PCIe card lies in =20
> the
> >> "port" value :
> >> void __iomem *ioport_map(unsigned long port, unsigned int len)
> >> {
> >> 	return (void __iomem *) (port + _IO_BASE);
> >
> > _IO_BASE is equal to isa_io_base. So if this is not zero, I think =20
> there's
> > a isa
> > bridge in your platform. So you can access these I/O ports based on =20
> that
> > isa
> > bridge/bus with ioreadx/iowritex.
> >
> > I tried ioread8/iowriet8 after ioremap, it doesn't work
> >
> >> }
> >>    in 8139too.c, the "port" value is 0x1000; for my PCIe card, the =20
> "port"
> >> value
> >> is 0xfefff000. And the value is got from pci_resource_start. So =20
> you see,
> >> the
> >
> > But this means the port is as memory-mapped

Are you sure?  It could mean that it's on a non-primary bus and I/O for =20
this bus is mapped at a lower address than the primary.  Just because =20
the addition is wrapping around doesn't mean it's wrong.

> > so ioremap() should be workable in this case. Then out_bex/in_bex =20
> should be fine.

ioremap() and out_bex/in_bex are not appropriate for PCI I/O regions =20
(and presumably that's what it is, if pci_iomap is calling =20
ioport_map).  Big-endian is not appropriate for PCI in any case.

The whole point of pci_iomap() appears to be that the driver doesn't =20
need to care whether it's MMIO or PIO, and can use ioread/writeX on the =20
resulting cookie.  If PPC is messing this up it's not the driver's =20
fault.

-Scott=

Re: can't access PCIe card under sbc8548

[tiejun.chen]

On 05/31/2013 12:24 AM, Scott Wood wrote:
> On 05/30/2013 07:49:48 AM, wolfking wrote:
>> tiejun.chen wrote
>> > On 05/30/2013 03:32 PM, wolfking wrote:
>> >> (continued)
>> >>    I traced the 8139too.c when it uses pci_iomap, the pci_iomap called
>> >> the
>> >> ioport_map. The difference between 8139 and my PCIe card lies in the
>> >> "port" value :
>> >> void __iomem *ioport_map(unsigned long port, unsigned int len)
>> >> {
>> >>     return (void __iomem *) (port + _IO_BASE);
>> >
>> > _IO_BASE is equal to isa_io_base. So if this is not zero, I think there's
>> > a isa
>> > bridge in your platform. So you can access these I/O ports based on that
>> > isa
>> > bridge/bus with ioreadx/iowritex.
>> >
>> > I tried ioread8/iowriet8 after ioremap, it doesn't work
>> >
>> >> }
>> >>    in 8139too.c, the "port" value is 0x1000; for my PCIe card, the "port"
>> >> value
>> >> is 0xfefff000. And the value is got from pci_resource_start. So you see,
>> >> the
>> >
>> > But this means the port is as memory-mapped
>
> Are you sure?  It could mean that it's on a non-primary bus and I/O for this bus

We can take a further look at '/proc/ioports', '/proc/iomem' and other message.

> is mapped at a lower address than the primary.  Just because the addition is
> wrapping around doesn't mean it's wrong.
>
>> > so ioremap() should be workable in this case. Then out_bex/in_bex should be
>> fine.
>
> ioremap() and out_bex/in_bex are not appropriate for PCI I/O regions (and
> presumably that's what it is, if pci_iomap is calling ioport_map).  Big-endian
> is not appropriate for PCI in any case.

Yes, I should correct that PCI interprets all accesses as little-endian.

Tiejun

Re: can't access PCIe card under sbc8548

[wolfking]

hi, tiejun:
  When I don't compile my pcie card driver(in kernel or as a driver), the
kernel 
auto allocate the resoures to my pcie card, here is the output:

root@generic-powerpc-e500v2:~# cat /proc/ioports
00000000-007fffff : /pci@e0008000
  00000000-00000fff : Legacy IO
  00001000-000010ff : 0000:00:11.0
    00001000-000010ff : 8139too
feffe000-ffffdfff : /pcie@e000a000
  feffe000-ffffdfff : PCI Bus 0001:02
    fefff000-fefff0ff : 0001:02:00.0
    fefff100-fefff103 : 0001:02:00.0

the "feffe000-ffffdfff " is the resources that the kernel allocates to my
card.



--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775p71850.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.

Re: can't access PCIe card under sbc8548

[Benjamin Herrenschmidt]

On Thu, 2013-05-30 at 11:24 -0500, Scott Wood wrote:
> ioremap() and out_bex/in_bex are not appropriate for PCI I/O regions  
> (and presumably that's what it is, if pci_iomap is calling  
> ioport_map).  Big-endian is not appropriate for PCI in any case.
> 
> The whole point of pci_iomap() appears to be that the driver doesn't  
> need to care whether it's MMIO or PIO, and can use ioread/writeX on the  
> resulting cookie.  If PPC is messing this up it's not the driver's  
> fault.

We are not messing this up and it should work.

Cheers,
Ben.

Re: can't access PCIe card under sbc8548

[wolfking]

I tried several R/W functions: inb/outb and ioread8/iowrite8. The inb/outb 
doesn't report any error while the ioread8 freeze. The load/storage sync 
code that I have used include mb and something else.



--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775p71817.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.

Re: can't access PCIe card under sbc8548

[tiejun.chen]

On 05/30/2013 06:02 PM, wolfking wrote:
> I tried several R/W functions: inb/outb and ioread8/iowrite8. The inb/outb

What about out_be32/in_be32?

Tiejun

Re: can't access PCIe card under sbc8548

[wolfking]

tiejun.chen wrote
> On 05/30/2013 06:02 PM, wolfking wrote:
>> I tried several R/W functions: inb/outb and ioread8/iowrite8. The
>> inb/outb
> 
> What about out_be32/in_be32?
> 
> my PCIe card's internal register is only 8 bit. So I tried ioremap,
> out_8/in_8.
> It still doesn't work.
> 
> Tiejun
> _______________________________________________
> Linuxppc-dev mailing list

> Linuxppc-dev@.ozlabs

> https://lists.ozlabs.org/listinfo/linuxppc-dev





--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775p71826.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.

Re: can't access PCIe card under sbc8548

[Scott Wood]

On 05/29/2013 10:42:01 PM, wolfking wrote:
> hi, all
>   I'm doing some developing on the windriver's sbc8548 board. The =20
> kernel I
> use
> is 3.6.10 and the u-boot version is 2012-10. I changed the board's
> configuration:
> the board now boot from the 64MB SODIMM Flash (not the default 8MB =20
> on-board
> Flash
> memory), and the PCI clock rate is changed to 33MHZ.
>   Now the trouble I am in is that: the PCI card (a NIC card rtl8139) =20
> can be
> accessed OK, while the PCIe card can't work, that is, the kernel can't
> access
> its internal register. The kernel can correctly probe the PCIe card. =20
> its
> BAR0
> is a I/O mapped register, I use ioport_map to map the BAR0 to kernel's
> address
> space, then use ioread8/iowrite8 to access its internal register, it =20
> doesn't
> work.

In what specific way does it not work?

> I analyse the ioport_map function and find it just add the input =20
> parameter
> to
> a fixed _IO_BASE value, below is the function:
> void __iomem *ioport_map(unsigned long port, unsigned int len)
> {
> 	return (void __iomem *) (port + _IO_BASE);
> }
> the _IO_BASE value under sbc8548 is 0xfd7fd000, the value of =20
> ioport_map
> paramenter
> "port" is 0xfefff000. Obviously the add overflows, so the follow-up
> operations
> can't succeed.

Why can't it succeed?  Is there nothing mapped at 0xfc7fc000?

> The value of "port" is got from the function
> pci_resource_start.
> So I guess the kernel allocate a bad address to this PCIe card. How =20
> can I
> fix this?
>   I also plug this PCIe card into a freescale's board mpc8641-hpcn, =20
> try the
> same driver,
> I noticed ioport_map also meet overflow, but it does work fine.

So don't focus on the overflow, but rather on the actual breakage.

-Scott=

Re: can't access PCIe card under sbc8548

[wolfking]

hi, scott:
  Thanks for replying!
> In what specific way does it not work?
  when I use iowrite8 to write, things seem OK, the codes continue running,
  when I use ioread8 to read, the console I use freezes. The console stops
  responding.
> Why can't it succeed?  Is there nothing mapped at 0xfc7fc000?
  My PCIe card's local bus register is at BAR0+0xC7, so the driver should
access 0xfc7fc027 and the following 7 bytes. But When I access the areas:
ioread8 just freezes. 



--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775p71847.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.

Re: can't access PCIe card under sbc8548

[Benjamin Herrenschmidt]

On Thu, 2013-05-30 at 17:40 -0700, wolfking wrote:
> hi, scott:
>   Thanks for replying!
> > In what specific way does it not work?
>   when I use iowrite8 to write, things seem OK, the codes continue running,
>   when I use ioread8 to read, the console I use freezes. The console stops
>   responding.
> > Why can't it succeed?  Is there nothing mapped at 0xfc7fc000?
>   My PCIe card's local bus register is at BAR0+0xC7, so the driver should
> access 0xfc7fc027 and the following 7 bytes. But When I access the areas:
> ioread8 just freezes. 

That looks more like a HW error to me unless you are hitting completely
the wrong system addresses.

What would be useful would be to add printk's to check what exact
physical address pci_iomap ends up using and whether that matches
to the iobase_phys of the PCI bridge + the BAR value of the card.

Cheers,
Ben.

Re: can't access PCIe card under sbc8548

[wolfking]

hi, Herrenschmidt
 Thanks for replying to this topic!

> That looks more like a HW error to me unless you are hitting completely 
> the wrong system addresses.
The HW is OK. If I plug this card into another ppc board: mpc86641-hpcn,
it works fine. 

> What would be useful would be to add printk's to check what exact 
> physical address pci_iomap ends up using and whether that matches 
> to the iobase_phys of the PCI bridge + the BAR value of the card.
Would you mind telling me at what position should I add the printk?
I can only print pci_iomap's return address which is mentioned in 
the previous mail, it is 0xfc7fc000. The following is part of the
linux boot message:

PCI host bridge /pcie@e000a000  ranges:
 MEM 0x00000000a0000000..0x00000000afffffff -> 0x00000000a0000000 
  IO 0x00000000e2800000..0x00000000ea7fffff -> 0x0000000000000000
/pcie@e000a000: PCICSRBAR @ 0xfff00000

 Is 0xe2800000 the iobase_phys of the PCI bridge?

So any suggestion is appreciated!

regards,
wolfking.




--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775p71874.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.

Re: can't access PCIe card under sbc8548

[wolfking]

hi, all:
  The problem is now solved.
  In the sbc8548.dts file, the resource that is allocated to pcie controller
is not
suitable for my card. Actually, I think it is a typo, the range that is
allocated to
pcie io is 0x08000000, now I change it to 0x00800000. Then my serial card
work OK!
  Is this is a bug in sbc8548.dts?



--
View this message in context: http://linuxppc.10917.n7.nabble.com/can-t-access-PCIe-card-under-sbc8548-tp71775p72164.html
Sent from the linuxppc-dev mailing list archive at Nabble.com.