Re: PCI Memory mapping

Re: PCI Memory mapping

[koteswararaom]

[-- Attachment #1: Type: text/plain, Size: 574 bytes --]

Hi, 

 

I have come across this forum, and I have a question. I am working on PCI
device driver, I am trying to insert the  pci card in to the pci slot of the
main computer, I am finding a problem here, when I insert the card in to the
pci slot, my pci device ID and vendor Id are not getting displayed. I am
using windows xp o/s.

 

I am trying to map pci memory. Can you suggest me how can we map the pci
memory. What steps should I take in order to make my Device Id and Vendor Id
displayed when I insert the card in to the pci slot.

 

With Kind regards,

Ajith.

 


[-- Attachment #2: Type: text/html, Size: 2426 bytes --]

Re: PCI Memory mapping

[David Hawkins]

Hi Ajith,

> I have come across this forum, and I have a question. I am working on
> PCI device driver, I am trying to insert the pci card in to the pci slot
> of the main computer, I am finding a problem here, when I insert the
> card in to the pci slot, my pci device ID and vendor Id are not getting
> displayed. I am using windows xp o/s.
>
> I am trying to map pci memory. Can you suggest me how can we map the pci
> memory. What steps should I take in order to make my Device Id and
> Vendor Id displayed when I insert the card in to the pci slot.

This is a PowerPC Linux developer list so your question is
way off-topic here.

If your PCI card is not showing up at all, it could be due to the
fact that it is not being powered. I have found old motherboards
that do not supply both the 5V and 3.3V rails to PCI slots.
Check the power supplies on your PCI board, or at least
measure the PCI power pins in the slots. Then if power is present,
check there is a PCI clock in the slot. After that, boot Linux
from a bootable CD-ROM and type lspci to see if your device is
listed.

Cheers,
Dave

Re: PCI Memory mapping

[Michael Neuling]

> I have come across this forum, and I have a question. I am working on PCI
> device driver, I am trying to insert the  pci card in to the pci slot of the
> main computer, I am finding a problem here, when I insert the card in to the
> pci slot, my pci device ID and vendor Id are not getting displayed. I am
> using windows xp o/s.

Err, so err, humm... er...  how do I put this..?!

This is the "Linux on PowerPC" mailing list.  Your question is neither
Linux or PowerPC related.  

I'm not sure how you expect anyone here to help.

> I am trying to map pci memory. Can you suggest me how can we map the pci
> memory. What steps should I take in order to make my Device Id and Vendor Id
> displayed when I insert the card in to the pci slot.


Mikey

PCI memory mapping

[Renaud Barbier]

Hello,
Barebox version: 2024-09

I am porting the Linux PCIE driver for Broadcom Cortex-A9 (ARMv7) chip.
So far I am able to detect the bridge and NVME device attach to it:

pci: pci_scan_bus for bus 0
pci:  last_io = 0x00000000, last_mem = 0x20000000, last_mem_pref = 0x00000000
pci: class = 00000604, hdr_type = 00000001
pci: 00:00 [14e4:b170]

pci: pci_scan_bus for bus 1
pci:  last_io = 0x00000000, last_mem = 0x20000000, last_mem_pref = 0x00000000
pci: class = 00000108, hdr_type = 00000000
pci: 01:00 [126f:2263]
ERROR: pci: last_mem = 0x20000000, 16384
pci: pbar0: mask=ffffc004 NP-MEM 16384 bytes
...
pci: class = 00000108, hdr_type = 00000000
pci: 01:f8 [126f:2263]
ERROR: pci: last_mem = 0x2007c000, 16384
pci: pbar0: mask=ffffc004 NP-MEM 16384 bytes
pci: pci_scan_bus returning with max=02
pci: bridge NP limit at 0x20100000

The PCI memory address is assigned to the device and map (pci_iomap from nvme_dev_map)  but access to this PCI space crashes the system:

pci: pci_scan_bus returning with max=02
ERROR: nvme_dev_map: bar = 0x20000000
nvme pci-126f:2263.0: enabling bus mastering
ERROR: pci:  __pci_set_master
ERROR: nvme_pci_enable: address: 0x2000001c
unable to handle paging request at address 0x2000001c
pc : [<9fe3d6e4>]    lr : [<9fe3d6d8>]
sp : 9fff7f38  ip : 00000002  fp : 00000000
r10: 0006b52c  r9 : 40000000  r8 : 9fea9b88
r7 : 7fe18500  r6 : 7fe295a8  r5 : 9fe8e5b8  r4 : 7fe29538
r3 : 20000000  r2 : 00000000  r1 : 0000000a  r0 : 00000027
Flags: nZCv  IRQs off  FIQs off  Mode SVC_32
[<9fe3d6e4>] (nvme_probe+0xd8/0x4c4) from [<9fe14650>] (device_probe+0x64/0x15c)
[<9fe14650>] (device_probe+0x64/0x15c) from [<9fe14778>] (match+0x30/0x68)

Tracing the problem, I found that if I limit the scanning in pci_scan_bus to 1 device/function
-       for (devfn = 0; devfn < 0xff; ++devfn) {
+       for (devfn = 0; devfn < 0x8; ++devfn) { ==> At 0x10 I see the crash.

Then, the NVME device is detected:
nvme pci-126f:2263.0: serial: A012410180620000000
nvme pci-126f:2263.0: model: SM681GEF AGS
nvme pci-126f:2263.0: firmware: TFX7GB

Has anybody seen this problem before (I havenoyt seen that on the LS1046A)?
I am not seeing much diff with the master branch in /drivers/pci and drivers/nvme

Cheers,
Renaud

Re: PCI memory mapping

[Lucas Stach]

Hi Renaud,

Am Montag, dem 07.04.2025 um 14:55 +0000 schrieb Renaud Barbier:
> Hello,
> Barebox version: 2024-09
> 
> I am porting the Linux PCIE driver for Broadcom Cortex-A9 (ARMv7) chip.
> So far I am able to detect the bridge and NVME device attach to it:
> 
> pci: pci_scan_bus for bus 0
> pci:  last_io = 0x00000000, last_mem = 0x20000000, last_mem_pref = 0x00000000
> pci: class = 00000604, hdr_type = 00000001
> pci: 00:00 [14e4:b170]
> 
> pci: pci_scan_bus for bus 1
> pci:  last_io = 0x00000000, last_mem = 0x20000000, last_mem_pref = 0x00000000
> pci: class = 00000108, hdr_type = 00000000
> pci: 01:00 [126f:2263]
> ERROR: pci: last_mem = 0x20000000, 16384
> pci: pbar0: mask=ffffc004 NP-MEM 16384 bytes
> ...
> pci: class = 00000108, hdr_type = 00000000
> pci: 01:f8 [126f:2263]
> ERROR: pci: last_mem = 0x2007c000, 16384
> pci: pbar0: mask=ffffc004 NP-MEM 16384 bytes
> pci: pci_scan_bus returning with max=02
> pci: bridge NP limit at 0x20100000
> 
I highly doubt that your NVMe device actually occupies all those BDF
addresses.

Either you host driver isn't properly reporting bus timeouts on the
PCI_VENDOR_ID config space access, to make it appear like there are
multiple devices on the bus to the topology walk, or more likely from
the symptoms you report, your host driver doesn't properly set up the
DF part of the BDF for the config space requests. In that case the
first device on the bus may correctly answer to all the config space
requests, which again will make it appear like there are multiple
devices, but the endpoint will actually get configured with the BAR
setup from the last "device" on the bus. If you then try to access the
MMIO space of the first device, there is no EP configured to handle the
request, causing a bus abort.

Regards,
Lucas

RE: PCI memory mapping

[Renaud Barbier]

For information, I had a look at the Linux (6.11.3) PCI driver , the PCI bus (PCIe in my case) probing in drivers/pci/probe.c is limited by this function due to PCIe specification.
static int only_one_child(struct pci_bus *bus)
  {
          struct pci_dev *bridge = bus->self;

          /*
           * Systems with unusual topologies set PCI_SCAN_ALL_PCIE_DEVS so
           * we scan for all possible devices, not just Device 0.
           */
          if (pci_has_flag(PCI_SCAN_ALL_PCIE_DEVS))
                  return 0;

          /*
           * A PCIe Downstream Port normally leads to a Link with only Device
           * 0 on it (PCIe spec r3.1, sec 7.3.1).  As an optimization, scan
           * only for Device 0 in that situation.
           */
          if (bridge && pci_is_pcie(bridge) && pcie_downstream_port(bridge))
                  return 1;

          return 0;
  }

int pci_scan_slot(struct pci_bus *bus, int devfn)
  {
          struct pci_dev *dev;
          int fn = 0, nr = 0;

~         if (only_one_child(bus) && (devfn > 0)) {
+                 pr_err("XXX: %s one child devfn = %d\n", __func__, devfn);
                  return 0; /* Already scanned the entire slot */
+         }
...


> -----Original Message-----
> From: Lucas Stach <l.stach@pengutronix.de>
> Sent: 08 April 2025 10:54
> To: Renaud Barbier <Renaud.Barbier@ametek.com>; Barebox List
> <barebox@lists.infradead.org>
> Subject: Re: PCI memory mapping
> 
> ***NOTICE*** This came from an external source. Use caution when replying,
> clicking links, or opening attachments.
> 
> Hi Renaud,
> 
> Am Montag, dem 07.04.2025 um 14:55 +0000 schrieb Renaud Barbier:
> > Hello,
> > Barebox version: 2024-09
> >
> > I am porting the Linux PCIE driver for Broadcom Cortex-A9 (ARMv7) chip.
> > So far I am able to detect the bridge and NVME device attach to it:
> >
> > pci: pci_scan_bus for bus 0
> > pci:  last_io = 0x00000000, last_mem = 0x20000000, last_mem_pref =
> > 0x00000000
> > pci: class = 00000604, hdr_type = 00000001
> > pci: 00:00 [14e4:b170]
> >
> > pci: pci_scan_bus for bus 1
> > pci:  last_io = 0x00000000, last_mem = 0x20000000, last_mem_pref =
> > 0x00000000
> > pci: class = 00000108, hdr_type = 00000000
> > pci: 01:00 [126f:2263]
> > ERROR: pci: last_mem = 0x20000000, 16384
> > pci: pbar0: mask=ffffc004 NP-MEM 16384 bytes ...
> > pci: class = 00000108, hdr_type = 00000000
> > pci: 01:f8 [126f:2263]
> > ERROR: pci: last_mem = 0x2007c000, 16384
> > pci: pbar0: mask=ffffc004 NP-MEM 16384 bytes
> > pci: pci_scan_bus returning with max=02
> > pci: bridge NP limit at 0x20100000
> >
> I highly doubt that your NVMe device actually occupies all those BDF
> addresses.
> 
> Either you host driver isn't properly reporting bus timeouts on the
> PCI_VENDOR_ID config space access, to make it appear like there are multiple
> devices on the bus to the topology walk, or more likely from the symptoms
> you report, your host driver doesn't properly set up the DF part of the BDF for
> the config space requests. In that case the first device on the bus may correctly
> answer to all the config space requests, which again will make it appear like
> there are multiple devices, but the endpoint will actually get configured with
> the BAR setup from the last "device" on the bus. If you then try to access the
> MMIO space of the first device, there is no EP configured to handle the
> request, causing a bus abort.
> 
> Regards,
> Lucas

Re: PCI Memory mapping

[Marc Leeman]

I now realise I sent to the wrong mailing list :-/
(thread was on linuxppc-dev instead of linuxppc-embedded).

Hi,

I know I've been flooding the mailing list with this particular little
topic and thanks to some valuable comments [1], we were able to
pinpoint our problem fairly quickly.

Partly based on a number of related known silicon bugs, I have drafted
the following description (and work-around) that explains our data
corruption.

http://scorpius.homelinux.org/~marc/masterwrite/

[1] There is still one comment which is in the queue to be investigated.

 marc.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

PCI Memory mapping

[Marc Leeman]

I am currently trying the following

 +-----+        +-----+
 | mem |        | mem |
 +-----+        +-----+
    |              |
 +-----+        +-----+
 | ppc |        | dsp |
 +-----+        +-----+
    |     pci      |
    +--------------+

For a number of reasons (load on ppc, dsp silicon bug), we are trying to
transform the ppc dma transfers (ppc master write) to transfers
initiated by the dsp; where the memory block that is the result of
processing by the ppc is fetched by the dsp (ppc slave read).

Mapping pci devices registers and memory  into ppc kernel memory is well
documented, but I seem to be missing information about the other way
around: I (ppc) want to signal the dsp that data is ready at address
0xXXXXXXXX, come and get it (virt_to_bus(0xXXXXXXXX)?).

Most likely I have been searching archives with the wrong keywords since
I haven't found a good reference next to some notes in IO-mapping.txt
which seem to indicate that this is possible.

Any pointers about making cpu memory visible on the pci bus would greatly
be appreciated.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Jeff Angielski]

On Tue, 2004-03-16 at 06:40, Marc Leeman wrote:
> Any pointers about making cpu memory visible on the pci bus would greatly
> be appreciated.

Check your PPC documentation for the configuration of the PITARx,
PIBARx, and PICMRx.  Once you read about these it should all make sense.

Then look at your bootloader source code (hopefully you are using
u-boot) to see how these registers are being configured.

HTH,
Jeff Angielski
The PTR Group


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

> Check your PPC documentation for the configuration of the PITARx,
> PIBARx, and PICMRx. Once you read about these it should all make
> sense. Then look at your bootloader source code (hopefully you are
> using u-boot) to see how these registers are being configured.

Tnx for the info.

I am working on a solution, so instead of just posing a question on
the list, maybe the current working solution is a good contribution to
the lesser divine kernel hackers on the list :)

pci_map_single(driver_data.dev,driver_data.device[i]->\
    pciaddress,PCI_TRANSFER_SIZE,PCI_DMA_TODEVICE)

and

pci_unmap_single(driver_data.dev,driver_data.device[i]->\
    bus_addr,PCI_TRANSFER_SIZE,PCI_DMA_TODEVICE);

The data is transferred in blocks of 2k and first copied into a kernel
buffer from user space and then mapped on the PCI address range. Ideally
the user-space buffer would be mapped onto PCI (don't know if this is
possible, I did a quick, negative, test, but I suspect that I should
check the addresses again). However, this would put some logic at user
level, which is currently (in the transformation process) not desired.

The bus address (result of pci_map_single) is transferred to the DSP,
which it uses to fetch the data.

  +------+       +--------+     +------------+
  | user |------>| kernel |---->| pci-mapped |
  +------+       +--------+     +------------+

In the current kernel code (2.4.17 mvl2.1), pci_sync_single does not
seem to be implemented. In the code I checked, the successive mapping
and unmapping of the buffers does result in the same addresses.

I would assume that pci_sync_single would allow me to pass the mapped
address only once to the DSP, awaking it with an appropriate interrupt
when new data is available. Due to the absence of pci_sync_single, the
pci-mapped address needs to be passed to the DSP each time.

The observation based question is: to what degree can I be certain that
in a streaming data application, the successive mapping and unmapping of
the pci mapped addresses will result in the same bus addresses?

--
  Marc Leeman                                          Hardware R&D Engineer
  Barco Controlrooms Division                Noordlaan 5, B-8520 Kuurne (BE)
  Tel. +32 56 368 428 http://www.barcocontrolrooms.com marc.leeman@barco.com

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

> the list, maybe the current working solution is a good contribution to
> the lesser divine kernel hackers on the list :)

euhm, which include me :)

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

>   +------+       +--------+     +------------+
>   | user |------>| kernel |---->| pci-mapped |
>   +------+       +--------+     +------------+

Hm, I'm not quite out of the woods yet it seems :-/

The system contains (amongst other components) a MPC8245 PPC and a
TMS320C6415 DSP. The PPC handles incoming data and passes it along to
the DSP. Partially due to a problem on the DSP, we want the DMA to be
initiated by the DSP.

As such, we prepare a kernel buffer:
if(!(driver_data.device[i]->kern_addr= \
  pci_alloc_consistent(driver_data.dev,PCI_TRANSFER_SIZE,\
  &(driver_data.device[i]->bus_addr)))){...}

that links/maps it on the PCI memory range. This dma_t (bus address) is
passed to the DSP. The DSP then reads the data over the PCI buss and
acknowledges this to the (currently polling) PPC.

The PPC prepares the data, and calls an ioctl that copies part of the
user buffer into kernel space (driver_data.device[i]->kern_addr);
following this, the DSP is warned that new data is available.

 ioctl(...,MAP_CONSISTENT,...);

 while(ptr<end){
   movebuftokernel(ptr,size); /* ioctl */
   ptr+=size;
 }

 ioctl(...,MAPOFF_CONSISTENT,...);

This process is repeated until the entire userspace buffer has been
processed.

Even though documentation indicates that consistent DMA mapping should
not have caching issues we have a problem that reflects this very much:

When data is passed to the DSP and every buffer is confirmed with an
[Enter] (i.e. __very__ slow_), data is passed correctly. When we try to
pass the same data without user intervention (comment out
waitforenter()),
parts of the transferred data are copied from other locations: most of
the time, part of the buffer (e.g. 2048 bytes) is not update yet (old
data).

e.g.

8 * 32 bit: correct
96 * 32 bit: wrong
... rest is OK.

Any ideas what might go wrong since consistent mappings 'should' not
have problems with caching...

--
  Marc Leeman                          Hardware R&D Engineer
  Barco Controlrooms Division Noordlaan 5, B-8520 Kuurne (BE)
  Tel. +32 56 368 428        http://www.barcocontrolrooms.com=20

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

> Any ideas what might go wrong since consistent mappings 'should' not
> have problems with caching...

OK, more tests.

I tried to make certain that cache was disabled for these pages by
including CONFIG_NOT_COHERENT_CACHE (which defaults back to

  consistent_alloc
instead of
  __get_free_pages

But we did not notice a difference. Adding a user space buffer of 2 Megs
that just got filled with a counter makes the DMA transfers OK (this
should have purged the cache I think).

But so does adding a delay of 1 second between 2 transfers.

Between two transfers, I even filled the kernel buffer inbetween with
0xCA in kernel space and in user space (by copying a user buffer into
the kernel buffer again).

The strange thing is that the second buffer is always corrupted with
seemingly 'old' data, at an offset of 4 words (32 bit) and this for only
24 words, the rest of the buffer is fine.

The common factor seems to be 'timing', but unfortunately, this does not
yet pinpoint the problem to either the DSP or the PPC side. Any
experienced insight or well educated guesses?

--
  Marc Leeman                          Hardware R&D Engineer
  Barco Controlrooms Division Noordlaan 5, B-8520 Kuurne (BE)
  Tel. +32 56 368 428        http://www.barcocontrolrooms.com

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Michael R. Zucca]

On Mar 23, 2004, at 11:01 AM, Marc Leeman wrote:

> The strange thing is that the second buffer is always corrupted with
> seemingly 'old' data, at an offset of 4 words (32 bit) and this for
> only
> 24 words, the rest of the buffer is fine.
>
> The common factor seems to be 'timing', but unfortunately, this does
> not
> yet pinpoint the problem to either the DSP or the PPC side. Any
> experienced insight or well educated guesses?

Are you putting in sync instructions where appropriate?

--
----------------------------------------------
  Michael Zucca - mrz5149@acm.org
----------------------------------------------
  "I'm too old to use Emacs." -- Rod MacDonald
----------------------------------------------


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Benjamin Herrenschmidt]

On Tue, 2004-03-23 at 22:17, Marc Leeman wrote:

> When data is passed to the DSP and every buffer is confirmed with an
> [Enter] (i.e. __very__ slow_), data is passed correctly. When we try to
> pass the same data without user intervention (comment out
> waitforenter()),
> parts of the transferred data are copied from other locations: most of
> the time, part of the buffer (e.g. 2048 bytes) is not update yet (old
> data).

One thing is: after having written your data to the consistent buffer,
you should make sure you have proper ordering with the MMIO write you
do to kick the DSP. On most CPUs, eieio appear to be enough and I
wouldn't suppose the 8245 to heavily reorder, but just in case, try
adding a sync.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

I just added a [1]
  flush_cache_all();

after copying the buffer from user space, no change.

The same for
__asm__ __volatile__("sync\n");

However, more and more things seem to point to either the DSP or 'the
hardware':

For testing purposes, I allocate a 4k buffer and pass the address of the
buffer +0x400 to the DSP (the corrupted data always occur in the firsts
28 words: from word 5 to word 28).

In userspace, I fill the 0x400 bytes with 0xFF and only use the last
0xC00 bytes with 'userful' data. The 4k buffer is passed to kernel space
where it is mapped on the PCI.

  PPC user
  +-------+
  | 0x400 |
  +-------+
  |       |
  | 0xC00 |
  |       |
  +-------+

     || copy_from_user
     \/

  PPC kern            DSP
  +-------+
  | 0x400 |
  +-------+        +-------+
  |       |  PCI   |       |
  | 0xC00 |  ===>  | 0xC00 |
  |       |        |       |
  +-------+        +-------+



The DSP only reads the last 3k bytes but still, it gets the corrupted
data from word 5 to word 32 when no (excessive) delays are used.

Next to this, consistent_alloc() already uses _PAGE_NO_CACHE to avoid
caching of the buffer (called by pci_consistent_alloc()) in cachemap.c
(yes, the 8245 kernel configuration does not enable
CONFIG_NOT_COHERENT_CACHE normally [2], but I added it but the behaviour
is still identical).

Filling the user buffer with 0xCA after copy_from_user and re-copying it
into the kernel buffer (without interrupting the DSP this time) still
has no effect: the 'corrupted' data is still old data from the last
PCI transferred data on the DSP side (after a subsequent copy_from_user
with useful data and DSP interrupt).

However the first PCI transferred buffer is ALWAYS correct.

On the other hand, this DSP-type did read via PCI master read
operations and a PCI plugin card memory from w32 on an x86 arch (instead
of our internal PCI bus) a couple of yrs ago I'm told.

[1] hmmmmm, this is helpful code for ppc :)

/*
 * No cache flushing is required when address mappings are
 * changed, because the caches on PowerPCs are physically
 * addressed.  -- paulus
 * Also, when SMP we use the coherency (M) bit of the
 * BATs and PTEs.  -- Cort
 */
#define flush_cache_all()               do { } while (0)


[2]
6xx/7xx/74xx/8260      CONFIG_6xx
instead of
CONFIG_8xx

as Processor Type.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

This is getting weird, I found a solution to the problem, well kind of.

Let's introduce the major players in this drama:

 /* communication struct between user and kernel space , in driver.h
  * kernel include */
 typedef struct SStreamPci {
         unsigned int pciaddress;
         unsigned int useraddress;
         unsigned char stream;
 }TSStreamPci;

 /* user space program */
 /* as long as not all data is transferred */
 while(userptr < (userbuffer + (TRANSFER_SIZE>>2))){
   /* currentstream is a struct that is passed to the kernel of
    * type TSStreamPci */
   currentstream.useraddress = (unsigned)userptr;

   /* copy our buffer to the kernel (PCI_TRANSFER_SIZE), 4k */
   ioctl(dsphandle,PPC2DSP_CONSISTENT_COPY_TO_KERNEL,&currentstream);

   /* Interrupt the DSP */
   WriteToDSP(PacketReady,0x1);
   ioctl(dsphandle,PPC2DSP_SET_HDCR,&interrupt);

   /* Wait until the DSP acknowledges transfer */
   while(ReadFromDSP(PacketReady)){}

   /* Read kernel buffer back */
   ioctl(dsphandle,PPC2DSP_CONSISTENT_COPY_FROM_KERNEL,&currentstream);

   /* increment pointer */
   userptr += (BLOCK_TRANSFER_SIZE >> 2);
 }

The strange thing is that this works only and only if I do a transfer
back from kernel space into the userbuffer AFTER the DSP has
acknowledged transferring the data. Note that this data is never used
again. Doing this copy_to_user before DSP acknowledgement still results
in the afore mentioned data corruption.

The associated ioctls are:
case PPC2DSP_CONSISTENT_COPY_TO_KERNEL:
  {
    TSStreamPci currentstream;
    if(copy_from_user(&currentstream,(TSStreamPci*) arg,sizeof(TSStreamPci))){
      return -EFAULT;
    }
    if(copy_from_user(driver_data.device[currentstream.stream]->kern_addr,(char*)currentstream.useraddress, PCI_TRANSFER_SIZE)){
      return -EFAULT;
    }
    break;
  }
case PPC2DSP_CONSISTENT_COPY_FROM_KERNEL:
  {
    TSStreamPci currentstream;
    if(copy_from_user(&currentstream,(TSStreamPci*) arg, sizeof(TSStreamPci))){
      return -EFAULT;
    }
    if(copy_to_user((char*)currentstream.useraddress,driver_data.device[currentstream.stream]->kern_addr, PCI_TRANSFER_SIZE)){
      return -EFAULT;
    }
    break;
  }

There is about a 10% bandwith reduction (due to PPC utilisation I would
assume) over the PCI, so I can live with that; but what bothers me is
not knowing _why_, ... Especially since I want my first module not to
contain too large beginners mistakes :)

  marc.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[linas]

On Wed, Mar 24, 2004 at 03:25:24PM +0100, Marc Leeman wrote:
>
> The strange thing is that this works only and only if I do a transfer
> back from kernel space into the userbuffer AFTER the DSP has
> acknowledged transferring the data. Note that this data is never used
> again. Doing this copy_to_user before DSP acknowledgement still results
> in the afore mentioned data corruption.

Excuse my total ignorance on this topic, but ... is it possible that the
DSP is using some sort of oddball bookmark mechanism to read the data?
This would be a rather poor design decision on the part of whomever
wrote the DSP code, but not out of the realms of impossible.  (I've
seen equally bizarre code in the past whose only excuse for existance
was stupidity.)  It could also be a bug in the DSP code.  The fact that
'everything is OK after DSP acks' is telling: in the end, the DSP *did*
do the right thing, by the time it ack'ed even if it did something weird
while data was flying around.

--linas

p.s. I did not read your code to see if it made sense, nor do I know
the ppc dma code.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

> Excuse my total ignorance on this topic, but ... is it possible that
> the DSP is using some sort of oddball bookmark mechanism to read the
> data?

Not at all, we're at the point that we appreciate external comments,
clues, ...
The DSP gets the PCI address, configures a DMA transfer and fetches the
data to some predefined memory location. There is not further
functionality (next to checking the counter in the memory for data
consistency purposes) because we want to verify and debug the technique.

> This would be a rather poor design decision on the part of whomever
> wrote the DSP code, but not out of the realms of impossible. (I've
> seen equally bizarre code in the past whose only excuse for existence
> was stupidity.) It could also be a bug in the DSP code. The fact
> that 'everything is OK after DSP acks' is telling: in the end, the
> DSP *did* do the right thing, by the time it ack'ed even if it did
> something weird while data was flying around.

The DSP code is fairly simple, but I would not put it past a DSP/Bios
bug, it has happened before (unfortunately). The annoying thing is that
we are not able to pinpoint a location where it is happening (on the DSP
or the PPC side). Next to this, I have a bit more confidence in the guy
writing the code than that :)

The only pattern we notice is that the first 8 words (32 bit words) are
fine, the following 24 words are not, the rest of the buffer is fine
(tested with buffers of 512, 1024, 2048 and 4096).

We did some further experimentation and the minimum of data we need to
read back after a DSP ACK is 97 bytes. You will note that 24*4=96. I
don't believe in coincidences :) If we read 96 bytes or less, we get the
corrupted data again... I even tested and tried to verify that the code
which is being used by the kernel is the one from the 2.4.17 mvl for
memory management and not 'tainted' code.

Anyway, this is a workaround until we have some more time to try to
figure out where the problem lies (PPC/PCI/Kernel/DSP) and what it is...
As ever, insights are welcome :-/

  marc.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[linas]

On Thu, Mar 25, 2004 at 04:48:46PM +0100, Marc Leeman wrote:
> As ever, insights are welcome :-/

OK, another wild guess: are the cache lines properly invalidated?
I think you said that the corrpution was a copy of earlier data ...
well, where could 'earlier data' be hiding?  The offsets you're
reporting sound all wrong, but hey ...

Don't know how hard it would be for you to run the test with cachine disabled,
but it might be worth a try.

--linas

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[linas]

On Thu, Mar 25, 2004 at 10:34:14AM -0600, linas@austin.ibm.com wrote:
>
> On Thu, Mar 25, 2004 at 04:48:46PM +0100, Marc Leeman wrote:
> > As ever, insights are welcome :-/
>
> OK, another wild guess: are the cache lines properly invalidated?
> I think you said that the corrpution was a copy of earlier data ...
> well, where could 'earlier data' be hiding?  The offsets you're
> reporting sound all wrong, but hey ...

actually, if you have 32-byte cache lines on your cpu, and you
have, umm, something maybe 4-way set associative (I've forgotten
how it works) that might explain it.   I don't know what cpu's
have what cache sizes.

> Don't know how hard it would be for you to run the test with cachine disabled,
> but it might be worth a try.
>
> --linas
>
>

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

> actually, if you have 32-byte cache lines on your cpu, and you have,
> umm, something maybe 4-way set associative (I've forgotten how it
> works) that might explain it. I don't know what cpu's have what cache
> sizes.

This was also one of the paths we were considering, but the following
tests have been performed in order to validate or invalidate this
assumption:
1. I was under the impression that 'consistent_alloc' (which I
activited by defining CONFIG_NOT_COHERENT_CACHE in the kernel
configuration for our board) assured that the pages were not using
cache, at least the following lines lead me to this:

[malm@bpscltpd linux-2.4.17_bl21]$ vim ./arch/ppc/mm/cachemap.c

<...>
        /*
         * we need to ensure that there are no cachelines in use,
         * or worse dirty in this area.
         */
        invalidate_dcache_range(page, page + size);
<...>

and

<...>
        flags = _PAGE_KERNEL | _PAGE_NO_CACHE;

        err = 0;
        for (i = 0; i < size && err == 0; i += PAGE_SIZE)
                err = map_page(va+i, pa+i, flags);
<...>

As told before, no change with or without this define enabled that uses
consistent_alloc.

2. Next I tried to invalidate the cache the naive way:
I refilled the used buffer with 0xCA in userspace and copied it back to
kernelspace (in the PCI mappded buffer). Next I transferred the next
'useful' data from user to kernel space. Still completely the same: the
data showing up (corrupted) on the DSP side was the 'old' lines of the
buffer last transferred (i.e. the buffer before bringing 0xCA into
kernel space).

3. I took the previous point one step further: I filled the kernel
buffer with 0xCA before copying in the userbuffer. Still the same old
data is showing up on the DSP side. Still, I understand that none of
these two techniques assure purging the cache.

4. I added sync's and the like which should sync cache and mem, still no
change.

Apparently, the closest I can come to pinpointing the location of the
problem at the moment is that 'something' must be happening at the point
where the DSP starts reading from the PPC memory. When no such read is
performed, the problem does not occur. The other way around, the DSP can
write into the PPC memory without corrupting data.

Maybe reading of the DSP over PCI and the resulting DMA setup causes a
[hardware|software] timing problem with purging the cache (which should
not be used for these pages to start with (see [1]), or the DSP ack's
too early on and reading these 97 bytes after the DSP ACK causes some
hardware lock to be released, giving the DSP just enough time to
correctly fetch the data before the PPC starts pumping in new, ...

  marc.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Jeff Angielski]

Marc,

How is the PICMRx configured?  Do you have snoop enabled or disabled?

This will sound obvious, but have you stuck a PCI bus analyzer into the
system to watch the actual PCI transactions?  They more than make up for
their cost in time savings.  :)

Jeff


On Fri, 2004-03-26 at 03:00, Marc Leeman wrote:
> > actually, if you have 32-byte cache lines on your cpu, and you have,
> > umm, something maybe 4-way set associative (I've forgotten how it
> > works) that might explain it. I don't know what cpu's have what cache
> > sizes.
>
> This was also one of the paths we were considering, but the following
> tests have been performed in order to validate or invalidate this
> assumption:
> 1. I was under the impression that 'consistent_alloc' (which I
> activited by defining CONFIG_NOT_COHERENT_CACHE in the kernel
> configuration for our board) assured that the pages were not using
> cache, at least the following lines lead me to this:
>
> [malm@bpscltpd linux-2.4.17_bl21]$ vim ./arch/ppc/mm/cachemap.c
>
> <...>
>         /*
>          * we need to ensure that there are no cachelines in use,
>          * or worse dirty in this area.
>          */
>         invalidate_dcache_range(page, page + size);
> <...>
>
> and
>
> <...>
>         flags = _PAGE_KERNEL | _PAGE_NO_CACHE;
>
>         err = 0;
>         for (i = 0; i < size && err == 0; i += PAGE_SIZE)
>                 err = map_page(va+i, pa+i, flags);
> <...>
>
> As told before, no change with or without this define enabled that uses
> consistent_alloc.
>
> 2. Next I tried to invalidate the cache the naive way:
> I refilled the used buffer with 0xCA in userspace and copied it back to
> kernelspace (in the PCI mappded buffer). Next I transferred the next
> 'useful' data from user to kernel space. Still completely the same: the
> data showing up (corrupted) on the DSP side was the 'old' lines of the
> buffer last transferred (i.e. the buffer before bringing 0xCA into
> kernel space).
>
> 3. I took the previous point one step further: I filled the kernel
> buffer with 0xCA before copying in the userbuffer. Still the same old
> data is showing up on the DSP side. Still, I understand that none of
> these two techniques assure purging the cache.
>
> 4. I added sync's and the like which should sync cache and mem, still no
> change.
>
> Apparently, the closest I can come to pinpointing the location of the
> problem at the moment is that 'something' must be happening at the point
> where the DSP starts reading from the PPC memory. When no such read is
> performed, the problem does not occur. The other way around, the DSP can
> write into the PPC memory without corrupting data.
>
> Maybe reading of the DSP over PCI and the resulting DMA setup causes a
> [hardware|software] timing problem with purging the cache (which should
> not be used for these pages to start with (see [1]), or the DSP ack's
> too early on and reading these 97 bytes after the DSP ACK causes some
> hardware lock to be released, giving the DSP just enough time to
> correctly fetch the data before the PPC starts pumping in new, ...
>
>   marc.
>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

> How is the PICMRx configured? Do you have snoop enabled or disabled?

This is what I like about my job: problem based digging deeper into code
internals :)

Anyway, the time of day difference between us enables me to try to come
up with some kind of founded answer. We are using ppcboot (yeah, I know,
but unfortunately, there are too many things in the todo queue to
upgraced to uboot).

After some digging into kernel code, ppcboot code and the mpc8245
documentation, I assume you are trying to point me to the following:
In Section 7.5:
"Because a snoop transaction is required to establish a lock, the
MPC8245 does not honour the assertion of a LOCK when PICR1[NO_SNOOP_EN]
is set."; when describing exclusive access to data.

In cpu/mpc824x/cpu_init.c:
    CONFIG_WRITE_HALFWORD(PCICR, 0x06);
    /* bus master, respond to PCI accesses */
    CONFIG_READ_WORD(PICR1, val);
    CONFIG_WRITE_WORD( PICR1,
       (val & (PICR1_RCS0)) |
               PICR1_PROC_TYPE_603E |
               PICR1_FLASH_WR_EN | PICR1_MCP_EN |
               PICR1_CF_DPARK | PICR1_NO_BUSW_CK |
               PICR1_DEC| PICR1_CF_APARK | 0x10);
/* 8245 UM says bit 4 must be set */

#define PICR1_RCS0              0x00100000
#define PICR1_PROC_TYPE_603E    0x00040000
#define PICR1_FLASH_WR_EN       0x00001000
#define PICR1_MCP_EN            0x00000800
#define PICR1_CF_DPARK          0x00000200
#define PICR1_NO_BUSW_CK        0x00000080 /* no bus width check for
flash writes */
#define PICR1_DEC               0x00000100 /* Time Base enable on
8245/8241 */
#define PICR1_CF_APARK          0x00000008

And for PICR2:
    CONFIG_READ_WORD(PICR2, val);
    val= val & ~ (PICR2_CF_SNOOP_WS_MASK | PICR2_CF_APHASE_WS_MASK);
    /*mask off waitstate bits*/

#define PICR2_CF_APHASE_WS_MASK 0x0000000c
#define PICR2_CF_SNOOP_WS_MASK  0x000c0000

k. I got the (relevant) registers states at bootup now, I am still
looking for the NO_SNOOP_EN bit, it must be somewhere in the docs...

4.7: Speculative PCI reads see Chapter 13;

but NO_SNOOP_EN is defined for PICR2 on bit 27, could it be that the
the description in section (note) 7.5 is referring to PICR2 instead of
the mentioned PICR1?

which should be 0x8000000

I'll masking PICR2 with this tomorrow...

> will sound obvious, but have you stuck a PCI bus analyzer into the
> system to watch the actual PCI transactions? They more than make up
> for their cost in time savings. :)

Not yet, mainly because we want to interactively PPC/DSP rule out other
possibilities (an infamous DSP reset which is extremely difficult to
pinpoint); combined with the usual growing to do list with continuous
interrupts for 'paramount' things to do...

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

> which should be 0x8000000 I'll masking PICR2 with this tomorrow...

Of course, I should't enable this bit, but at least I'll be able to see
if the situation gets worse by enabling it...

I need more coffee...

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

> I'll masking PICR2 with this tomorrow...

I have obtained some results by setting some 8245 registers at bootup.

I tested the following:

. enabling NO_SNOOP_EN (27 in PICR2): nfs does not boot anymore
. increasing the snoop and address wait cycles to 5 (default ppc bootup),
no real change.
. disable speculative reads  (2 in PICR1): doesn't boot nfs anymore
(via eepro100?)

No real success, until I came accross the PCMBCR register which defines
the number of PCI to Local Memory Read and Write Buffers. When I set
both to 1 ( & 0xF0), the data copy back is no longer needed to assure
data consistency between the PPC and DSP (default is 4 32 byte buffers).

include/mpc824x.h:
#define PCMBCR          0x800000e1  /* PCI/Memory Buffer Configuration
Register */

cpu/mpc824x/cpu_init.c:
CONFIG_READ_BYTE(PCMBCR,val);
CONFIG_WRITE_BYTE(PCMBCR,(val | 0xF0));

Double checking this by setting both on 4 again (during cpu_init of
ppcboot) resulted in severe MPEG data corruption. These results at least
seems to point in the direction you suggested.

This is the only way I seem to be able to assure data consistency (the
kernel copy back is no longer required), but the documentation also
suggests that (obviously) this degrades performance and is mainly used
for debugging.0

Still, when reading the explanations that could case this behaviour
(copy back buffer and filling the PCMRBs) , they point to cache
operations, which, I thought, were disabled in the linux kernel for
these particular PCI mapped buffers.

By adding __asm__ __volatile__("eieio"); in user and kernel space,
which lets the CCU buffers to be flushed, no change is observed (sect.
5.4.3.1; CCU Responses to the Processor Transactions).

It looks to me as if I should disable the bus snooping, but as mentioned
before, initialising this in ppc boot inhibits the NFS boot process.

Can this be disabled at runtime (possibly with re-enabling it) and if
so, is this a good practice to do so?

  marc.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Benjamin Herrenschmidt]

On Thu, 2004-04-01 at 22:33, Marc Leeman wrote:

> No real success, until I came accross the PCMBCR register which defines
> the number of PCI to Local Memory Read and Write Buffers. When I set
> both to 1 ( & 0xF0), the data copy back is no longer needed to assure
> data consistency between the PPC and DSP (default is 4 32 byte buffers).
>
> include/mpc824x.h:
> #define PCMBCR          0x800000e1  /* PCI/Memory Buffer Configuration
> Register */
>
> cpu/mpc824x/cpu_init.c:
> CONFIG_READ_BYTE(PCMBCR,val);
> CONFIG_WRITE_BYTE(PCMBCR,(val | 0xF0));
>
> Double checking this by setting both on 4 again (during cpu_init of
> ppcboot) resulted in severe MPEG data corruption. These results at least
> seems to point in the direction you suggested.
>
> This is the only way I seem to be able to assure data consistency (the
> kernel copy back is no longer required), but the documentation also
> suggests that (obviously) this degrades performance and is mainly used
> for debugging.0
>
> Still, when reading the explanations that could case this behaviour
> (copy back buffer and filling the PCMRBs) , they point to cache
> operations, which, I thought, were disabled in the linux kernel for
> these particular PCI mapped buffers.
>
> By adding __asm__ __volatile__("eieio"); in user and kernel space,
> which lets the CCU buffers to be flushed, no change is observed (sect.
> 5.4.3.1; CCU Responses to the Processor Transactions).
>
> It looks to me as if I should disable the bus snooping, but as mentioned
> before, initialising this in ppc boot inhibits the NFS boot process.
>
> Can this be disabled at runtime (possibly with re-enabling it) and if
> so, is this a good practice to do so?

Looks like your north bridge is horribly buggy...

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Adrian Cox]

On Thu, 2004-04-01 at 13:33, Marc Leeman wrote:

> By adding __asm__ __volatile__("eieio"); in user and kernel space,
> which lets the CCU buffers to be flushed, no change is observed (sect.
> 5.4.3.1; CCU Responses to the Processor Transactions).
>
> It looks to me as if I should disable the bus snooping, but as mentioned
> before, initialising this in ppc boot inhibits the NFS boot process.

Here's one possible scenario:

Your user space buffer is cleared with dcbz operations. This may be done
by standard library functions, rather than explicitly in your code. (If
that's not true, what follows could still bite someone else.)

The 603e core will only broadcast the dcbz when the page is marked
coherent. The page is only marked coherent when the kernel is compiled
for SMP.

If the above hypothesis is correct, the 824x CPUs will need to have
CPU_FTR_NEED_COHERENT set in this patch:

 http://lists.linuxppc.org/linuxppc-embedded/200403/msg00218.html

- Adrian Cox
http://www.humboldt.co.uk/


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: PCI Memory mapping

[Marc Leeman]

Hi,

I know I've been flooding the mailing list with this particular little
topic and thanks to some valuable comments [1], we were able to
pinpoint our problem fairly quickly.

Partly based on a number of related known silicon bugs, I have drafted
the following description (and work-around) that explains our data
corruption.

http://scorpius.homelinux.org/~marc/masterwrite/

[1] There is still one comment which is in the queue to be investigated.

 marc.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/