Loading PCIe Device Driver at Dom0

Loading PCIe Device Driver at Dom0

[Kenneth Wong]

[-- Attachment #1.1: Type: text/plain, Size: 2904 bytes --]

Dear all,

I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.

I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1.  When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue.  I can do reads and write to the hardware device.  But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs.

>From my debug messages, it appears it hangs because the driver doesn't receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register.  Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver.

The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.


What the driver does:

Set up a command buffer:
Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
unsigned long buf_addr = __pa(buf);
unsigned int buf_addr_low = (unsigned int)buf_addr;

Tell device about the buffer:
iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);

Set up IRQ:
    if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
        (!pci_enable_msi(dev)))
    {
        if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev))
        {
            return  -ENODEV;
        }
        my_dev->intr_mode = INTERRUPT_MSI;
    }

Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf.  But interrupt did not happen.):
iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);


>From dmesg, it looks like IRQ initialization is complete.
[  241.743769] My_driver initialization
[  241.743787] xen: registering gsi 16 triggering 0 polarity 1
[  241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
[  241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
[  241.743801] Already setup the GSI :16
[  241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[  241.743815] my-driver 0000:02:00.0: setting latency timer to 64

/proc/interrupts:
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
......
......
339:          0          0          0          0          0          0          0          0  xen-pirq-msi       my-driver
......
......

Any idea what might cause the problem?

Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following?
1)       Allocating a DMA buffer in driver to allow the device to DMA stuffs.
2)       Requesting MSI irq.

Please advise!

Thanks a lot in advance!!

Kenneth

[-- Attachment #1.2: Type: text/html, Size: 16105 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Loading PCIe Device Driver at Dom0

[Konrad Rzeszutek Wilk]

On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:
> Dear all,
> 
> I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.
> 
> I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1.  When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue.  I can do reads and write to the hardware device.  But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs.
> 
> >From my debug messages, it appears it hangs because the driver doesn't receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register.  Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver.
> 
> The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.

The 2).
> 
> 
> What the driver does:

You do need to use the PCI API (or the DMA API).

> 
> Set up a command buffer:
> Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
> unsigned long buf_addr = __pa(buf);
> unsigned int buf_addr_low = (unsigned int)buf_addr;
> 
> Tell device about the buffer:
> iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);

> 
> Set up IRQ:
>     if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
>         (!pci_enable_msi(dev)))
>     {
>         if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev))
>         {
>             return  -ENODEV;
>         }
>         my_dev->intr_mode = INTERRUPT_MSI;
>     }
> 
> Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf.  But interrupt did not happen.):
> iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);
> 
> 
> >From dmesg, it looks like IRQ initialization is complete.
> [  241.743769] My_driver initialization
> [  241.743787] xen: registering gsi 16 triggering 0 polarity 1
> [  241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
> [  241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
> [  241.743801] Already setup the GSI :16
> [  241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> [  241.743815] my-driver 0000:02:00.0: setting latency timer to 64
> 
> /proc/interrupts:
>             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
> ......
> ......
> 339:          0          0          0          0          0          0          0          0  xen-pirq-msi       my-driver
> ......
> ......
> 
> Any idea what might cause the problem?
> 
> Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following?
> 1)       Allocating a DMA buffer in driver to allow the device to DMA stuffs.
> 2)       Requesting MSI irq.
> 
> Please advise!
> 
> Thanks a lot in advance!!
> 
> Kenneth

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Loading PCIe Device Driver at Dom0

[Kenneth Wong]

Hi Konrad and others,

Oh, there are Xen/dom0 specific APIs for PCI and DMA?  

May I ask the names and where can I can more info on the APIs?

Many thanks!

Kenneth

________________________________________
From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com]
Sent: Wednesday, May 23, 2012 5:17 PM
To: Kenneth Wong
Cc: xen-devel@lists.xen.org
Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0

On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:
> Dear all,
>
> I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.
>
> I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1.  When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue.  I can do reads and write to the hardware device.  But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs.
>
> >From my debug messages, it appears it hangs because the driver doesn't receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register.  Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver.
>
> The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.

The 2).
>
>
> What the driver does:

You do need to use the PCI API (or the DMA API).

>
> Set up a command buffer:
> Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
> unsigned long buf_addr = __pa(buf);
> unsigned int buf_addr_low = (unsigned int)buf_addr;
>
> Tell device about the buffer:
> iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);

>
> Set up IRQ:
>     if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
>         (!pci_enable_msi(dev)))
>     {
>         if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev))
>         {
>             return  -ENODEV;
>         }
>         my_dev->intr_mode = INTERRUPT_MSI;
>     }
>
> Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf.  But interrupt did not happen.):
> iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);
>
>
> >From dmesg, it looks like IRQ initialization is complete.
> [  241.743769] My_driver initialization
> [  241.743787] xen: registering gsi 16 triggering 0 polarity 1
> [  241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
> [  241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
> [  241.743801] Already setup the GSI :16
> [  241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> [  241.743815] my-driver 0000:02:00.0: setting latency timer to 64
>
> /proc/interrupts:
>             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
> ......
> ......
> 339:          0          0          0          0          0          0          0          0  xen-pirq-msi       my-driver
> ......
> ......
>
> Any idea what might cause the problem?
>
> Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following?
> 1)       Allocating a DMA buffer in driver to allow the device to DMA stuffs.
> 2)       Requesting MSI irq.
>
> Please advise!
>
> Thanks a lot in advance!!
>
> Kenneth

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Loading PCIe Device Driver at Dom0

[Pasi Kärkkäinen]

On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote:
> Hi Konrad and others,
> 
> Oh, there are Xen/dom0 specific APIs for PCI and DMA?  
> 

PCI/DMA APIs are the Linux kernel APIs, not Xen specific.


> May I ask the names and where can I can more info on the APIs?
> 

Quick googling reveals:

http://www.mjmwired.net/kernel/Documentation/DMA-API.txt
http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt


-- Pasi

> Many thanks!
> 
> Kenneth
> 
> ________________________________________
> From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com]
> Sent: Wednesday, May 23, 2012 5:17 PM
> To: Kenneth Wong
> Cc: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0
> 
> On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:
> > Dear all,
> >
> > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.
> >
> > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1.  When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue.  I can do reads and write to the hardware device.  But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs.
> >
> > >From my debug messages, it appears it hangs because the driver doesn't receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register.  Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver.
> >
> > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.
> 
> The 2).
> >
> >
> > What the driver does:
> 
> You do need to use the PCI API (or the DMA API).
> 
> >
> > Set up a command buffer:
> > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
> > unsigned long buf_addr = __pa(buf);
> > unsigned int buf_addr_low = (unsigned int)buf_addr;
> >
> > Tell device about the buffer:
> > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);
> 
> >
> > Set up IRQ:
> >     if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
> >         (!pci_enable_msi(dev)))
> >     {
> >         if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev))
> >         {
> >             return  -ENODEV;
> >         }
> >         my_dev->intr_mode = INTERRUPT_MSI;
> >     }
> >
> > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf.  But interrupt did not happen.):
> > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);
> >
> >
> > >From dmesg, it looks like IRQ initialization is complete.
> > [  241.743769] My_driver initialization
> > [  241.743787] xen: registering gsi 16 triggering 0 polarity 1
> > [  241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
> > [  241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
> > [  241.743801] Already setup the GSI :16
> > [  241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> > [  241.743815] my-driver 0000:02:00.0: setting latency timer to 64
> >
> > /proc/interrupts:
> >             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
> > ......
> > ......
> > 339:          0          0          0          0          0          0          0          0  xen-pirq-msi       my-driver
> > ......
> > ......
> >
> > Any idea what might cause the problem?
> >
> > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following?
> > 1)       Allocating a DMA buffer in driver to allow the device to DMA stuffs.
> > 2)       Requesting MSI irq.
> >
> > Please advise!
> >
> > Thanks a lot in advance!!
> >
> > Kenneth
> 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Loading PCIe Device Driver at Dom0

[Kenneth Wong]

Hi Pasi,

Thanks a lot for the info!  I will try it.

Kenneth
________________________________________
From: Pasi Kärkkäinen [pasik@iki.fi]
Sent: Wednesday, May 23, 2012 11:18 PM
To: Kenneth Wong
Cc: xen-devel@lists.xen.org
Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0

On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote:
> Hi Konrad and others,
>
> Oh, there are Xen/dom0 specific APIs for PCI and DMA?
>

PCI/DMA APIs are the Linux kernel APIs, not Xen specific.


> May I ask the names and where can I can more info on the APIs?
>

Quick googling reveals:

http://www.mjmwired.net/kernel/Documentation/DMA-API.txt
http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt


-- Pasi

> Many thanks!
>
> Kenneth
>
> ________________________________________
> From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com]
> Sent: Wednesday, May 23, 2012 5:17 PM
> To: Kenneth Wong
> Cc: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0
>
> On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:
> > Dear all,
> >
> > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.
> >
> > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1.  When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue.  I can do reads and write to the hardware device.  But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs.
> >
> > >From my debug messages, it appears it hangs because the driver doesn't receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register.  Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver.
> >
> > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.
>
> The 2).
> >
> >
> > What the driver does:
>
> You do need to use the PCI API (or the DMA API).
>
> >
> > Set up a command buffer:
> > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
> > unsigned long buf_addr = __pa(buf);
> > unsigned int buf_addr_low = (unsigned int)buf_addr;
> >
> > Tell device about the buffer:
> > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);
>
> >
> > Set up IRQ:
> >     if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
> >         (!pci_enable_msi(dev)))
> >     {
> >         if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev))
> >         {
> >             return  -ENODEV;
> >         }
> >         my_dev->intr_mode = INTERRUPT_MSI;
> >     }
> >
> > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf.  But interrupt did not happen.):
> > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);
> >
> >
> > >From dmesg, it looks like IRQ initialization is complete.
> > [  241.743769] My_driver initialization
> > [  241.743787] xen: registering gsi 16 triggering 0 polarity 1
> > [  241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
> > [  241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
> > [  241.743801] Already setup the GSI :16
> > [  241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> > [  241.743815] my-driver 0000:02:00.0: setting latency timer to 64
> >
> > /proc/interrupts:
> >             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
> > ......
> > ......
> > 339:          0          0          0          0          0          0          0          0  xen-pirq-msi       my-driver
> > ......
> > ......
> >
> > Any idea what might cause the problem?
> >
> > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following?
> > 1)       Allocating a DMA buffer in driver to allow the device to DMA stuffs.
> > 2)       Requesting MSI irq.
> >
> > Please advise!
> >
> > Thanks a lot in advance!!
> >
> > Kenneth
>
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Loading PCIe Device Driver at Dom0

[Kenneth Wong]

Hi all,

I am now using dma_alloc_coherent().

However, the first parameter, pdev->dev required, which is "struct device", does not seem to have initialized.  

When and who is suppose to initialize it?

In Linux, I can pass "NULL" and it just works.  In Xen, it crashes.  

Sometimes insmod pass, sometimes hangs the system.

Please advise!

Kenneth
________________________________________
From: Pasi Kärkkäinen [pasik@iki.fi]
Sent: Wednesday, May 23, 2012 11:18 PM
To: Kenneth Wong
Cc: xen-devel@lists.xen.org
Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0

On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote:
> Hi Konrad and others,
>
> Oh, there are Xen/dom0 specific APIs for PCI and DMA?
>

PCI/DMA APIs are the Linux kernel APIs, not Xen specific.


> May I ask the names and where can I can more info on the APIs?
>

Quick googling reveals:

http://www.mjmwired.net/kernel/Documentation/DMA-API.txt
http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt


-- Pasi

> Many thanks!
>
> Kenneth
>
> ________________________________________
> From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com]
> Sent: Wednesday, May 23, 2012 5:17 PM
> To: Kenneth Wong
> Cc: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0
>
> On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:
> > Dear all,
> >
> > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.
> >
> > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1.  When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue.  I can do reads and write to the hardware device.  But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs.
> >
> > >From my debug messages, it appears it hangs because the driver doesn't receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register.  Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver.
> >
> > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.
>
> The 2).
> >
> >
> > What the driver does:
>
> You do need to use the PCI API (or the DMA API).
>
> >
> > Set up a command buffer:
> > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
> > unsigned long buf_addr = __pa(buf);
> > unsigned int buf_addr_low = (unsigned int)buf_addr;
> >
> > Tell device about the buffer:
> > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);
>
> >
> > Set up IRQ:
> >     if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
> >         (!pci_enable_msi(dev)))
> >     {
> >         if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev))
> >         {
> >             return  -ENODEV;
> >         }
> >         my_dev->intr_mode = INTERRUPT_MSI;
> >     }
> >
> > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf.  But interrupt did not happen.):
> > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);
> >
> >
> > >From dmesg, it looks like IRQ initialization is complete.
> > [  241.743769] My_driver initialization
> > [  241.743787] xen: registering gsi 16 triggering 0 polarity 1
> > [  241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
> > [  241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
> > [  241.743801] Already setup the GSI :16
> > [  241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> > [  241.743815] my-driver 0000:02:00.0: setting latency timer to 64
> >
> > /proc/interrupts:
> >             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
> > ......
> > ......
> > 339:          0          0          0          0          0          0          0          0  xen-pirq-msi       my-driver
> > ......
> > ......
> >
> > Any idea what might cause the problem?
> >
> > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following?
> > 1)       Allocating a DMA buffer in driver to allow the device to DMA stuffs.
> > 2)       Requesting MSI irq.
> >
> > Please advise!
> >
> > Thanks a lot in advance!!
> >
> > Kenneth
>
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Loading PCIe Device Driver at Dom0

[Kenneth Wong]

Dear all,

The driver now loaded.  But got the following exception after a few commands:

May 24 17:05:27 marvell-desktop kernel: [   84.804412] Xorg[1234]: segfault at 34 ip 00000000005067b1 sp 00007fff37a82f40 error 4 in Xorg[400000+1d4000]
May 24 17:05:29 marvell-desktop kernel: [   86.721658] rtkit-daemon[1708]: segfault at ffffffffffffff80 ip 00007fe23abee61a sp 00007fff9c249410 error 4 in libdbus-1.so.3.5.7[7fe23abc3000+42000]

What IRQF flags  I should use when setting up MSI IRQ, etc?

What did I do wrong??  Please help!

Kenneth


________________________________________
From: Kenneth Wong
Sent: Thursday, May 24, 2012 4:21 PM
To: xen-devel@lists.xen.org
Subject: RE: [Xen-devel] Loading PCIe Device Driver at Dom0

Hi all,

I am now using dma_alloc_coherent().

However, the first parameter, pdev->dev required, which is "struct device", does not seem to have initialized.

When and who is suppose to initialize it?

In Linux, I can pass "NULL" and it just works.  In Xen, it crashes.

Sometimes insmod pass, sometimes hangs the system.

Please advise!

Kenneth
________________________________________
From: Pasi Kärkkäinen [pasik@iki.fi]
Sent: Wednesday, May 23, 2012 11:18 PM
To: Kenneth Wong
Cc: xen-devel@lists.xen.org
Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0

On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote:
> Hi Konrad and others,
>
> Oh, there are Xen/dom0 specific APIs for PCI and DMA?
>

PCI/DMA APIs are the Linux kernel APIs, not Xen specific.


> May I ask the names and where can I can more info on the APIs?
>

Quick googling reveals:

http://www.mjmwired.net/kernel/Documentation/DMA-API.txt
http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt


-- Pasi

> Many thanks!
>
> Kenneth
>
> ________________________________________
> From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com]
> Sent: Wednesday, May 23, 2012 5:17 PM
> To: Kenneth Wong
> Cc: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0
>
> On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:
> > Dear all,
> >
> > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.
> >
> > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1.  When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue.  I can do reads and write to the hardware device.  But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs.
> >
> > >From my debug messages, it appears it hangs because the driver doesn't receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register.  Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver.
> >
> > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.
>
> The 2).
> >
> >
> > What the driver does:
>
> You do need to use the PCI API (or the DMA API).
>
> >
> > Set up a command buffer:
> > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
> > unsigned long buf_addr = __pa(buf);
> > unsigned int buf_addr_low = (unsigned int)buf_addr;
> >
> > Tell device about the buffer:
> > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);
>
> >
> > Set up IRQ:
> >     if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
> >         (!pci_enable_msi(dev)))
> >     {
> >         if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev))
> >         {
> >             return  -ENODEV;
> >         }
> >         my_dev->intr_mode = INTERRUPT_MSI;
> >     }
> >
> > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf.  But interrupt did not happen.):
> > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);
> >
> >
> > >From dmesg, it looks like IRQ initialization is complete.
> > [  241.743769] My_driver initialization
> > [  241.743787] xen: registering gsi 16 triggering 0 polarity 1
> > [  241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
> > [  241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
> > [  241.743801] Already setup the GSI :16
> > [  241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> > [  241.743815] my-driver 0000:02:00.0: setting latency timer to 64
> >
> > /proc/interrupts:
> >             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
> > ......
> > ......
> > 339:          0          0          0          0          0          0          0          0  xen-pirq-msi       my-driver
> > ......
> > ......
> >
> > Any idea what might cause the problem?
> >
> > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following?
> > 1)       Allocating a DMA buffer in driver to allow the device to DMA stuffs.
> > 2)       Requesting MSI irq.
> >
> > Please advise!
> >
> > Thanks a lot in advance!!
> >
> > Kenneth
>
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Loading PCIe Device Driver at Dom0

[Konrad Rzeszutek Wilk]

On Thu, May 24, 2012 at 04:21:16PM -0700, Kenneth Wong wrote:
> Hi all,
> 
> I am now using dma_alloc_coherent().
> 
> However, the first parameter, pdev->dev required, which is "struct device", does not seem to have initialized.  
> 
> When and who is suppose to initialize it?

Um, does your driver have a PCI vendor and model? It would
do it from the struct pci_driver->probe function.

> 
> In Linux, I can pass "NULL" and it just works.  In Xen, it crashes.  

NULL in that case is incorrect.
> 
> Sometimes insmod pass, sometimes hangs the system.
> 
> Please advise!

Look at how other drivers do it. You might also want to
pick up an Linux Device Drivers book and read the chapter
about PCI devices.

> 
> Kenneth
> ________________________________________
> From: Pasi Kärkkäinen [pasik@iki.fi]
> Sent: Wednesday, May 23, 2012 11:18 PM
> To: Kenneth Wong
> Cc: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0
> 
> On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote:
> > Hi Konrad and others,
> >
> > Oh, there are Xen/dom0 specific APIs for PCI and DMA?
> >
> 
> PCI/DMA APIs are the Linux kernel APIs, not Xen specific.
> 
> 
> > May I ask the names and where can I can more info on the APIs?
> >
> 
> Quick googling reveals:
> 
> http://www.mjmwired.net/kernel/Documentation/DMA-API.txt
> http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt
> 
> 
> -- Pasi
> 
> > Many thanks!
> >
> > Kenneth
> >
> > ________________________________________
> > From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com]
> > Sent: Wednesday, May 23, 2012 5:17 PM
> > To: Kenneth Wong
> > Cc: xen-devel@lists.xen.org
> > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0
> >
> > On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:
> > > Dear all,
> > >
> > > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.
> > >
> > > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1.  When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue.  I can do reads and write to the hardware device.  But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs.
> > >
> > > >From my debug messages, it appears it hangs because the driver doesn't receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register.  Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver.
> > >
> > > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.
> >
> > The 2).
> > >
> > >
> > > What the driver does:
> >
> > You do need to use the PCI API (or the DMA API).
> >
> > >
> > > Set up a command buffer:
> > > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
> > > unsigned long buf_addr = __pa(buf);
> > > unsigned int buf_addr_low = (unsigned int)buf_addr;
> > >
> > > Tell device about the buffer:
> > > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);
> >
> > >
> > > Set up IRQ:
> > >     if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
> > >         (!pci_enable_msi(dev)))
> > >     {
> > >         if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev))
> > >         {
> > >             return  -ENODEV;
> > >         }
> > >         my_dev->intr_mode = INTERRUPT_MSI;
> > >     }
> > >
> > > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf.  But interrupt did not happen.):
> > > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);
> > >
> > >
> > > >From dmesg, it looks like IRQ initialization is complete.
> > > [  241.743769] My_driver initialization
> > > [  241.743787] xen: registering gsi 16 triggering 0 polarity 1
> > > [  241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
> > > [  241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
> > > [  241.743801] Already setup the GSI :16
> > > [  241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> > > [  241.743815] my-driver 0000:02:00.0: setting latency timer to 64
> > >
> > > /proc/interrupts:
> > >             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
> > > ......
> > > ......
> > > 339:          0          0          0          0          0          0          0          0  xen-pirq-msi       my-driver
> > > ......
> > > ......
> > >
> > > Any idea what might cause the problem?
> > >
> > > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following?
> > > 1)       Allocating a DMA buffer in driver to allow the device to DMA stuffs.
> > > 2)       Requesting MSI irq.
> > >
> > > Please advise!
> > >
> > > Thanks a lot in advance!!
> > >
> > > Kenneth
> >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > http://lists.xen.org/xen-devel
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Loading PCIe Device Driver at Dom0

[Kenneth Wong]

Hi Konrad,

> Um, does your driver have a PCI vendor and model? It would
> do it from the struct pci_driver->probe function.

Yeah, it has all that.  It has been running fine on regular Linux, just that problems start coming up when porting to Xen env.  I think because it is now less forgiving due to the virtualization layer.

Any idea on the following messages?

        Xorg[1234]: segfault at 34 ip 00000000005067b1 sp 00007fff37a82f40 error 4 in Xorg[400000+1d4000]
        rtkit-daemon[1708]: segfault at ffffffffffffff80 ip 00007fe23abee61a sp 00007fff9c249410 error 4 in libdbus-1.so.3.5.7[7fe23abc3000+42000]

I think this might be cause by some other things in the driver.


Thanks,
Kenneth
________________________________________
From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com]
Sent: Thursday, May 24, 2012 6:34 PM
To: Kenneth Wong
Cc: xen-devel@lists.xen.org
Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0

On Thu, May 24, 2012 at 04:21:16PM -0700, Kenneth Wong wrote:
> Hi all,
>
> I am now using dma_alloc_coherent().
>
> However, the first parameter, pdev->dev required, which is "struct device", does not seem to have initialized.
>
> When and who is suppose to initialize it?

Um, does your driver have a PCI vendor and model? It would
do it from the struct pci_driver->probe function.

>
> In Linux, I can pass "NULL" and it just works.  In Xen, it crashes.

NULL in that case is incorrect.
>
> Sometimes insmod pass, sometimes hangs the system.
>
> Please advise!

Look at how other drivers do it. You might also want to
pick up an Linux Device Drivers book and read the chapter
about PCI devices.

>
> Kenneth
> ________________________________________
> From: Pasi Kärkkäinen [pasik@iki.fi]
> Sent: Wednesday, May 23, 2012 11:18 PM
> To: Kenneth Wong
> Cc: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0
>
> On Wed, May 23, 2012 at 07:41:34PM -0700, Kenneth Wong wrote:
> > Hi Konrad and others,
> >
> > Oh, there are Xen/dom0 specific APIs for PCI and DMA?
> >
>
> PCI/DMA APIs are the Linux kernel APIs, not Xen specific.
>
>
> > May I ask the names and where can I can more info on the APIs?
> >
>
> Quick googling reveals:
>
> http://www.mjmwired.net/kernel/Documentation/DMA-API.txt
> http://www.mjmwired.net/kernel/Documentation/DMA-API-HOWTO.txt
>
>
> -- Pasi
>
> > Many thanks!
> >
> > Kenneth
> >
> > ________________________________________
> > From: Konrad Rzeszutek Wilk [konrad.wilk@oracle.com]
> > Sent: Wednesday, May 23, 2012 5:17 PM
> > To: Kenneth Wong
> > Cc: xen-devel@lists.xen.org
> > Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0
> >
> > On Wed, May 23, 2012 at 04:43:55PM -0700, Kenneth Wong wrote:
> > > Dear all,
> > >
> > > I have a PCIe device driver that I have been using on various Linux distributions and Kernel versions (2.6.x - 3.x.y) successfully all along.
> > >
> > > I recently set up a Xen environment with Linux Mint 12 and Xen Hypervisor 4.1.  When I boot to Linux Mint, my driver still load (via insmod manually) successfully at Dom0 without any issue.  I can do reads and write to the hardware device.  But once booted to Xen, the driver failed to complete the driver load (via insmod manually) at Dom 0 and the console just hangs.
> > >
> > > >From my debug messages, it appears it hangs because the driver doesn't receive any interrupt after a command is sent to the hardware device by writing a parameter to the mapped register.  Once that register is written, the device is expected to DMA the command from the buffer allocated by the driver.
> > >
> > > The things that I can only think of that might have caused the problem are 1) IRQ mapping issue, or 2) DMA mapping issue, which I am not sure.
> >
> > The 2).
> > >
> > >
> > > What the driver does:
> >
> > You do need to use the PCI API (or the DMA API).
> >
> > >
> > > Set up a command buffer:
> > > Buf_t *buf = kmalloc(BUF_SIZE*sizeof(buf_t), GFP_KERNEL);
> > > unsigned long buf_addr = __pa(buf);
> > > unsigned int buf_addr_low = (unsigned int)buf_addr;
> > >
> > > Tell device about the buffer:
> > > iowrite32(buf_addr_low, dev->pci_reg_map + BUF_ADR__LOW);
> >
> > >
> > > Set up IRQ:
> > >     if (pci_find_capability(dev, PCI_CAP_ID_MSI) &&
> > >         (!pci_enable_msi(dev)))
> > >     {
> > >         if (request_irq(dev->irq, func_msi_interrupt, IRQF_SHARED, DRIVER_NAME, my_dev))
> > >         {
> > >             return  -ENODEV;
> > >         }
> > >         my_dev->intr_mode = INTERRUPT_MSI;
> > >     }
> > >
> > > Ask device to fetch command from buffer (Expect interrupt after this after device fetched the command from buf.  But interrupt did not happen.):
> > > iowrite32(buf_offset, dev->pci_reg_map + FETCH_CMD_REG);
> > >
> > >
> > > >From dmesg, it looks like IRQ initialization is complete.
> > > [  241.743769] My_driver initialization
> > > [  241.743787] xen: registering gsi 16 triggering 0 polarity 1
> > > [  241.743793] xen_map_pirq_gsi: returning irq 16 for gsi 16
> > > [  241.743795] xen: --> pirq=16 -> irq=16 (gsi=16)
> > > [  241.743801] Already setup the GSI :16
> > > [  241.743805] my-driver 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> > > [  241.743815] my-driver 0000:02:00.0: setting latency timer to 64
> > >
> > > /proc/interrupts:
> > >             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
> > > ......
> > > ......
> > > 339:          0          0          0          0          0          0          0          0  xen-pirq-msi       my-driver
> > > ......
> > > ......
> > >
> > > Any idea what might cause the problem?
> > >
> > > Is there anything we have to be enable/disable, use different functions, or do differently in drivers written for Xen Dom0 environment regarding the following?
> > > 1)       Allocating a DMA buffer in driver to allow the device to DMA stuffs.
> > > 2)       Requesting MSI irq.
> > >
> > > Please advise!
> > >
> > > Thanks a lot in advance!!
> > >
> > > Kenneth
> >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > http://lists.xen.org/xen-devel
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Loading PCIe Device Driver at Dom0

[Konrad Rzeszutek Wilk]

On Thu, May 24, 2012 at 07:37:44PM -0700, Kenneth Wong wrote:
> Hi Konrad,
> 
> > Um, does your driver have a PCI vendor and model? It would
> > do it from the struct pci_driver->probe function.
> 
> Yeah, it has all that.  It has been running fine on regular Linux, just that problems start coming up when porting to Xen env.  I think because it is now less forgiving due to the virtualization layer.

Sure. It also means that your driver would not work with IOMMU's properly.

> 
> Any idea on the following messages?

Some, but without any details (like machine type, userspace version, Xorg version,
kernel version, Xorg.0.log, etc) I've no clue.

> 
>         Xorg[1234]: segfault at 34 ip 00000000005067b1 sp 00007fff37a82f40 error 4 in Xorg[400000+1d4000]
>         rtkit-daemon[1708]: segfault at ffffffffffffff80 ip 00007fe23abee61a sp 00007fff9c249410 error 4 in libdbus-1.so.3.5.7[7fe23abc3000+42000]
> 
> I think this might be cause by some other things in the driver.

So you see this only after you load your driver?

Re: Loading PCIe Device Driver at Dom0

[Kenneth Wong]

Hi Konrad,

It's Xen 4.1.1.

It did not occur immediately after loading. It occurs after a few I/Os.

Best Regards,
Kenneth

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] 
Sent: Friday, May 25, 2012 1:31 PM
To: Kenneth Wong
Cc: xen-devel@lists.xen.org
Subject: Re: [Xen-devel] Loading PCIe Device Driver at Dom0

On Thu, May 24, 2012 at 07:37:44PM -0700, Kenneth Wong wrote:
> Hi Konrad,
> 
> > Um, does your driver have a PCI vendor and model? It would
> > do it from the struct pci_driver->probe function.
> 
> Yeah, it has all that.  It has been running fine on regular Linux, just that problems start coming up when porting to Xen env.  I think because it is now less forgiving due to the virtualization layer.

Sure. It also means that your driver would not work with IOMMU's properly.

> 
> Any idea on the following messages?

Some, but without any details (like machine type, userspace version, Xorg version,
kernel version, Xorg.0.log, etc) I've no clue.

> 
>         Xorg[1234]: segfault at 34 ip 00000000005067b1 sp 00007fff37a82f40 error 4 in Xorg[400000+1d4000]
>         rtkit-daemon[1708]: segfault at ffffffffffffff80 ip 00007fe23abee61a sp 00007fff9c249410 error 4 in libdbus-1.so.3.5.7[7fe23abc3000+42000]
> 
> I think this might be cause by some other things in the driver.

So you see this only after you load your driver?

Re: Xorg crashes... with what distro?

[Konrad Rzeszutek Wilk]

On Fri, May 25, 2012 at 02:12:48PM -0700, Kenneth Wong wrote:
> Hi Konrad,

Please do not top-post.
> 
> It's Xen 4.1.1.
> 
> It did not occur immediately after loading. It occurs after a few I/Os.


.. snip..

Please provide the details I've asked for.
> > Any idea on the following messages?
> 
> Some, but without any details (like machine type, userspace version, Xorg version,
> kernel version, Xorg.0.log, etc) I've no clue.
> 
> > 
> >         Xorg[1234]: segfault at 34 ip 00000000005067b1 sp 00007fff37a82f40 error 4 in Xorg[400000+1d4000]
> >         rtkit-daemon[1708]: segfault at ffffffffffffff80 ip 00007fe23abee61a sp 00007fff9c249410 error 4 in libdbus-1.so.3.5.7[7fe23abc3000+42000]
> > 
> > I think this might be cause by some other things in the driver.
> 
> So you see this only after you load your driver?
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel