From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: Loading PCIe Device Driver at Dom0 Date: Thu, 24 May 2012 21:34:21 -0400 Message-ID: <20120525013421.GC20947@phenom.dumpdata.com> References: <20120524001712.GA6285@phenom.dumpdata.com> <20120524061806.GX2058@reaktio.net> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Kenneth Wong Cc: "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On Thu, May 24, 2012 at 04:21:16PM -0700, Kenneth Wong wrote: > Hi all, > = > I am now using dma_alloc_coherent(). > = > However, the first parameter, pdev->dev required, which is "struct device= ", does not seem to have initialized. = > = > When and who is suppose to initialize it? Um, does your driver have a PCI vendor and model? It would do it from the struct pci_driver->probe function. > = > In Linux, I can pass "NULL" and it just works. In Xen, it crashes. = NULL in that case is incorrect. > = > Sometimes insmod pass, sometimes hangs the system. > = > Please advise! Look at how other drivers do it. You might also want to pick up an Linux Device Drivers book and read the chapter about PCI devices. > = > Kenneth > ________________________________________ > From: Pasi K=E4rkk=E4inen [pasik@iki.fi] > Sent: Wednesday, May 23, 2012 11:18 PM > To: Kenneth Wong > Cc: xen-devel@lists.xen.org > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 > = > On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote: > > Hi Konrad and others, > > > > Oh, there are Xen/dom0 specific APIs for PCI and DMA? > > > = > PCI/DMA APIs are the Linux kernel APIs, not Xen specific. > = > = > > May I ask the names and where can I can more info on the APIs? > > > = > Quick googling reveals: > = > http://www.mjmwired.net/kernel/Documentation/DMA-API.txt > http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt > = > = > -- Pasi > = > > Many thanks! > > > > Kenneth > > > > ________________________________________ > > From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com] > > Sent: Wednesday, May 23, 2012 5:17 PM > > To: Kenneth Wong > > Cc: xen-devel@lists.xen.org > > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0 > > > > On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote: > > > Dear all, > > > > > > I have a PCIe device driver that I have been using on various Linux d= istributions and Kernel versions (2.6.x - 3.x.y) successfully all along. > > > > > > I recently set up a Xen environment with Linux Mint 12 and Xen Hyperv= isor 4.1. When I boot to Linux Mint, my driver still load (via insmod manu= ally) successfully at Dom0 without any issue. I can do reads and write to = the hardware device. But once booted to Xen, the driver failed to complete= the driver load (via insmod manually) at Dom 0 and the console just hangs. > > > > > > >From my debug messages, it appears it hangs because the driver doesn= 't receive any interrupt after a command is sent to the hardware device by = writing a parameter to the mapped register. Once that register is written,= the device is expected to DMA the command from the buffer allocated by the= driver. > > > > > > The things that I can only think of that might have caused the proble= m are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure. > > > > The 2). > > > > > > > > > What the driver does: > > > > You do need to use the PCI API (or the DMA API). > > > > > > > > Set up a command buffer: > > > Buf_t *buf =3D kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL); > > > unsigned long buf_addr =3D __pa(buf); > > > unsigned int buf_addr_low =3D (unsigned int)buf_addr; > > > > > > Tell device about the buffer: > > > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW); > > > > > > > > Set up IRQ: > > > if (pci_find_capability(dev, PCI_CAP_ID_MSI) && > > > (!pci_enable_msi(dev))) > > > { > > > if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DR= IVER_NAME, my_dev)) > > > { > > > return -ENODEV; > > > } > > > my_dev->intr_mode =3D INTERRUPT_MSI; > > > } > > > > > > Ask device to fetch command from buffer (Expect interrupt after this = after device fetched the command from buf. But interrupt did not happen.): > > > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG); > > > > > > > > > >From dmesg, it looks like IRQ initialization is complete. > > > [ 241.743769] My_driver initialization > > > [ 241.743787] xen: registering gsi 16 triggering 0 polarity 1 > > > [ 241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16 > > > [ 241.743795] xen: --> pirq=3D16 -> irq=3D16 (gsi=3D16) > > > [ 241.743801] Already setup the GSI :16 > > > [ 241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, lo= w) -> IRQ 16 > > > [ 241.743815] my-driver 0000:02:00.0: setting latency timer to 64 > > > > > > /proc/interrupts: > > > CPU0 CPU1 CPU2 CPU3 CPU4 CP= U5 CPU6 CPU7 > > > ...... > > > ...... > > > 339: 0 0 0 0 0 = 0 0 0 xen-pirq-msi my-driver > > > ...... > > > ...... > > > > > > Any idea what might cause the problem? > > > > > > Is there anything we have to be enable/disable, use different functio= ns, or do differently in drivers written for Xen Dom0 environment regarding= the following? > > > 1) Allocating a DMA buffer in driver to allow the device to DMA= stuffs. > > > 2) Requesting MSI irq. > > > > > > Please advise! > > > > > > Thanks a lot in advance!! > > > > > > Kenneth > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xen.org > > > http://lists.xen.org/xen-devel > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > = > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel