xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

[Michael Kapp]

[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]

Hi,

i have tested the following setup:

System: gentoo_2008, xen-sources-2.6.18-r12, x86_64

1. Hide one port from the d-link quad nic via pci.hide in dom0.

module: sundance

grub.conf: pciback.hide=(06:07.0)

2. Use it within a domU.

domu.conf: pci = [ '06:07.0' ]

Tools:
------
Gnu C                  4.1.2
Gnu make               3.81
binutils               Binutils
util-linux             2.13.1.1
mount                  2.13.1.1
module-init-tools      3.4
e2fsprogs              1.40.9
xfsprogs               2.9.7
Linux C Library        6.1
Dynamic linker (ldd)   2.6.1
Procps                 3.2.7
Net-tools              1.60
Kbd                    1.13
Sh-utils               6.10
udev                   124
Modules Loaded         sundance

Result see attachment.

I have tested the same setup with the original xen
linux-2.6.18-xen-3.3.0 upstream kernel and it crashed too. With the
vanilla 2.6.27 kernel it doesn't crashed, but the device is not
available in the booted domU.

Is there any chance to get such a setup working?
Do you need more informations?

TIA
Michael

[-- Attachment #2: sundance.d-link.kernel.bug.txt --]
[-- Type: text/plain, Size: 2018 bytes --]

 * Starting eth0
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:377
invalid opcode: 0000 [1] SMP
CPU 0
Modules linked in: sundance
Pid: 3730, comm: ip Not tainted 2.6.18.8-U #1
RIP: e030:[<ffffffff81010bf1>]  [<ffffffff81010bf1>] dma_map_single+0x15a/0x1aa
RSP: e02b:ffff88000d709c78  EFLAGS: 00010292
RAX: 000000000000002f RBX: 00000000b4a5aa72 RCX: ffff88000d708000
RDX: ffffffffff5fd000 RSI: 0000000000000001 RDI: ffffffff81690880
RBP: ffff88000dccea72 R08: ffffffff81690888 R09: 00000000000029fd
R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000600
R13: ffff880001fccc90 R14: 0000000000000003 R15: ffff88000e832000
FS:  0000000000000000(0000) GS:ffffffff8178f000(0063) knlGS:0000000000000000
CS:  e033 DS: 002b ES: 002b
Process ip (pid: 3730, threadinfo ffff88000d708000, task ffff88000f95a040)
Stack:  ffff88000e832500 ffff88000e832528 ffff88000e832500 ffff88000e031000
 0000000000000030 ffffffff880022a0 00000009f308c122 0000000000000003
 0000000000000000 000000000001e480 00000000816971d0 ffffffff8105a335
Call Trace:
 [<ffffffff880022a0>] :sundance:netdev_open+0x1aa/0x48d
 [<ffffffff8105a335>] get_page_from_freelist+0x9c/0x47f
 [<ffffffff81429677>] dev_open+0x2d/0x6c
 [<ffffffff8142761a>] dev_change_flags+0x5c/0x11b
 [<ffffffff8148127f>] devinet_ioctl+0x217/0x598
 [<ffffffff810587a5>] filemap_nopage+0x187/0x316
 [<ffffffff8141e10b>] sock_ioctl+0x1b1/0x1d6
 [<ffffffff8108b8cd>] do_ioctl+0x21/0x6b
 [<ffffffff8108bb54>] vfs_ioctl+0x23d/0x256
 [<ffffffff8108bbc6>] sys_ioctl+0x59/0x7b
 [<ffffffff810a5438>] dev_ifsioc+0xd1/0x1f4
 [<ffffffff8107870d>] fd_install+0x25/0x5f
 [<ffffffff810a384b>] compat_sys_ioctl+0x26e/0x2b2
 [<ffffffff8101b05e>] sysenter_do_call+0x1b/0x25


Code: 0f 0b 68 9b 98 5a 81 c2 79 01 4d 85 ed 74 11 49 8b 85 f0 01
RIP  [<ffffffff81010bf1>] dma_map_single+0x15a/0x1aa
 RSP <ffff88000d709c78>
 /lib/rcscripts/net/iproute2.sh: line 7:  3730 Segmentation fault      LC_ALL=C /sbin/ip "$@"

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

[Keir Fraser]

On 25/10/08 21:05, "Michael Kapp" <tquarkk@isis-it.de> wrote:

> ----------- [cut here ] --------- [please bite here ] ---------
> Kernel BUG at arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:377

What's at line 377 in that source file in your kernel source tree?

 -- Keir

Re: xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

[Michael Kapp]

[-- Attachment #1: Type: text/plain, Size: 386 bytes --]

Keir Fraser wrote:
> On 25/10/08 21:05, "Michael Kapp" <tquarkk@isis-it.de> wrote:
> 
>> ----------- [cut here ] --------- [please bite here ] ---------
>> Kernel BUG at arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:377
> 
> What's at line 377 in that source file in your kernel source tree?

See attachment of the specific part from pci-dma-xen.c.

>  -- Keir

Regards,
Michael



[-- Attachment #2: pci-dma-xen.c.sources.xen.txt --]
[-- Type: text/plain, Size: 655 bytes --]

362 dma_addr_t
363 dma_map_single(struct device *dev, void *ptr, size_t size,
364            enum dma_data_direction direction)
365 {
366     dma_addr_t dma;
367
368     if (direction == DMA_NONE)
369         BUG();
370     WARN_ON(size == 0);
371
372     if (swiotlb) {
373         dma = swiotlb_map_single(dev, ptr, size, direction);
374     } else {
375         dma = gnttab_dma_map_page(virt_to_page(ptr)) +
376               offset_in_page(ptr);
377         IOMMU_BUG_ON(range_straddles_page_boundary(__pa(ptr), size));
378         IOMMU_BUG_ON(address_needs_mapping(dev, dma));
379     }
380
381     flush_write_buffers();
382     return dma;
383 }

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

[Keir Fraser]

On 26/10/08 08:40, "Michael Kapp" <tquarkk@isis-it.de> wrote:

> Keir Fraser wrote:
>> On 25/10/08 21:05, "Michael Kapp" <tquarkk@isis-it.de> wrote:
>> 
>>> ----------- [cut here ] --------- [please bite here ] ---------
>>> Kernel BUG at arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:377
>> 
>> What's at line 377 in that source file in your kernel source tree?
> 
> See attachment of the specific part from pci-dma-xen.c.

Add swiotlb=force to your domain's kernel command line.

 -- Keir

Re: xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

[Michael Kapp]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

Keir Fraser wrote:
> On 26/10/08 08:40, "Michael Kapp" <tquarkk@isis-it.de> wrote:
> 
>> Keir Fraser wrote:
>>> On 25/10/08 21:05, "Michael Kapp" <tquarkk@isis-it.de> wrote:
>>>
>>>> ----------- [cut here ] --------- [please bite here ] ---------
>>>> Kernel BUG at arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:377
>>> What's at line 377 in that source file in your kernel source tree?
>> See attachment of the specific part from pci-dma-xen.c.
> 
> Add swiotlb=force to your domain's kernel command line.

I set swiotlb=force in the domU conf, that doesn't fix it - also with
noirqdebug.

Michael

[-- Attachment #2: sundance.d-link.swiotlb.kernel.bug.txt --]
[-- Type: text/plain, Size: 2001 bytes --]

----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:377
invalid opcode: 0000 [1] SMP
CPU 0
Modules linked in: sundance
Pid: 3697, comm: ip Not tainted 2.6.18.8-U #1
RIP: e030:[<ffffffff81010bf1>]  [<ffffffff81010bf1>] dma_map_single+0x15a/0x1aa
RSP: e02b:ffff88000dba3c78  EFLAGS: 00010292
RAX: 000000000000002f RBX: 00000000b4858a72 RCX: ffff88000dba2000
RDX: ffffffffff5fd000 RSI: 0000000000000001 RDI: ffffffff81690880
RBP: ffff88000ded0a72 R08: ffffffff81690888 R09: 0000000000002a4d
R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000600
R13: ffff880001fccc90 R14: 0000000000000000 R15: ffff88000e832000
FS:  0000000000000000(0000) GS:ffffffff8178f000(0063) knlGS:0000000000000000
CS:  e033 DS: 002b ES: 002b
Process ip (pid: 3697, threadinfo ffff88000dba2000, task ffff88000f962800)
Stack:  ffff88000e832500 ffff88000e832510 ffff88000e832500 ffff88000e21f000
 0000000000000000 ffffffff880022a0 00000009f308c122 0000000000000000
 0000000000000000 000000000001e480 00000000816971d0 ffffffff8105a335
Call Trace:
 [<ffffffff880022a0>] :sundance:netdev_open+0x1aa/0x48d
 [<ffffffff8105a335>] get_page_from_freelist+0x9c/0x47f
 [<ffffffff81429677>] dev_open+0x2d/0x6c
 [<ffffffff8142761a>] dev_change_flags+0x5c/0x11b
 [<ffffffff8148127f>] devinet_ioctl+0x217/0x598
 [<ffffffff810587a5>] filemap_nopage+0x187/0x316
 [<ffffffff8141e10b>] sock_ioctl+0x1b1/0x1d6
 [<ffffffff8108b8cd>] do_ioctl+0x21/0x6b
 [<ffffffff8108bb54>] vfs_ioctl+0x23d/0x256
 [<ffffffff8108bbc6>] sys_ioctl+0x59/0x7b
 [<ffffffff810a5438>] dev_ifsioc+0xd1/0x1f4
 [<ffffffff8107870d>] fd_install+0x25/0x5f
 [<ffffffff810a384b>] compat_sys_ioctl+0x26e/0x2b2
 [<ffffffff8101b05e>] sysenter_do_call+0x1b/0x25


Code: 0f 0b 68 9b 98 5a 81 c2 79 01 4d 85 ed 74 11 49 8b 85 f0 01
RIP  [<ffffffff81010bf1>] dma_map_single+0x15a/0x1aa
 RSP <ffff88000dba3c78>
 /lib/rcscripts/net/iproute2.sh: line 7:  3697 Segmentation fault      LC_ALL=C /sbin/ip "$@"

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

[Keir Fraser]

On 26/10/08 08:58, "Michael Kapp" <tquarkk@isis-it.de> wrote:

>> Add swiotlb=force to your domain's kernel command line.
> 
> I set swiotlb=force in the domU conf, that doesn't fix it - also with
> noirqdebug.

That should be impossible since the line you are bugging on will not be
executed if the swiotlb has been successfully enabled. Do you get a line in
dmesg stating "Software IO TLB enabled"? If not then is CONFIG_SWIOTLB
enabled in your kernel config (should happen by default). The file
lib/swiotlb-xen.c should be getting built and you'll see there is code near
the top of that file to parse swiotlb=force which sets swiotlb_force and
should cause swiotlb_init() to do the right thing (which should get called
via the path mem_init->pci_iommu_alloc->pci_swiotlb_init->swiotlb_init.

 -- Keir

Re: xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

[Michael Kapp]

Keir Fraser wrote:
> On 26/10/08 08:58, "Michael Kapp" <tquarkk@isis-it.de> wrote:
> 
>>> Add swiotlb=force to your domain's kernel command line.
>> I set swiotlb=force in the domU conf, that doesn't fix it - also with
>> noirqdebug.
> 
> That should be impossible since the line you are bugging on will not be
> executed if the swiotlb has been successfully enabled. Do you get a line in
> dmesg stating "Software IO TLB enabled"? If not then is CONFIG_SWIOTLB
> enabled in your kernel config (should happen by default). The file
> lib/swiotlb-xen.c should be getting built and you'll see there is code near
> the top of that file to parse swiotlb=force which sets swiotlb_force and
> should cause swiotlb_init() to do the right thing (which should get called
> via the path mem_init->pci_iommu_alloc->pci_swiotlb_init->swiotlb_init.

yeah - your right! My domU config was wrong, with swiotlb=force it's
working, thanks!

Can you tell me what this parameter is doing in detail, or where i can
find docs about it?

Regards,
Michael

Re: xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

[Keir Fraser]

On 26/10/08 10:48, "Michael Kapp" <tquarkk@isis-it.de> wrote:

>>>> Add swiotlb=force to your domain's kernel command line.
>>> I set swiotlb=force in the domU conf, that doesn't fix it - also with
>>> noirqdebug.
>> 
>> That should be impossible since the line you are bugging on will not be
>> executed if the swiotlb has been successfully enabled. Do you get a line in
>> dmesg stating "Software IO TLB enabled"? If not then is CONFIG_SWIOTLB
>> enabled in your kernel config (should happen by default). The file
>> lib/swiotlb-xen.c should be getting built and you'll see there is code near
>> the top of that file to parse swiotlb=force which sets swiotlb_force and
>> should cause swiotlb_init() to do the right thing (which should get called
>> via the path mem_init->pci_iommu_alloc->pci_swiotlb_init->swiotlb_init.
> 
> yeah - your right! My domU config was wrong, with swiotlb=force it's
> working, thanks!
> 
> Can you tell me what this parameter is doing in detail, or where i can
> find docs about it?

The network card is only capable of 32-bit DMA. So for addresses above 4GB a
bounce buffer is required (DMA to/from bounce buffer; then copy to/from
proper high-memory buffer using host CPU). The swiotlb is basically a
pre-allocated bounce-buffer area. Because it is pre-allocated it is a waste
of memory if it isn't used and, since domUs do not normally perform real
physical I/O directly, we therefore by default only create a swiotlb for
dom0. You then have to override on the cmdline if this default doesn't
actually work (as in this case).

 -- Keir

Re: xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

[Michael Kapp]

Keir Fraser wrote:
>> Can you tell me what this parameter is doing in detail, or where i can
>> find docs about it?
> 
> The network card is only capable of 32-bit DMA. So for addresses above 4GB a
> bounce buffer is required (DMA to/from bounce buffer; then copy to/from
> proper high-memory buffer using host CPU). The swiotlb is basically a
> pre-allocated bounce-buffer area. Because it is pre-allocated it is a waste
> of memory if it isn't used and, since domUs do not normally perform real
> physical I/O directly, we therefore by default only create a swiotlb for
> dom0. You then have to override on the cmdline if this default doesn't
> actually work (as in this case).

...and why is it not possible (from the dom0 point of view) to detect
this behaviour while starting the domU, reading it's conf and stop the
boot process with a warning?

Michael

Re: xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

[Keir Fraser]

On 26/10/08 19:17, "Michael Kapp" <tquarkk@isis-it.de> wrote:

>> The network card is only capable of 32-bit DMA. So for addresses above 4GB a
>> bounce buffer is required (DMA to/from bounce buffer; then copy to/from
>> proper high-memory buffer using host CPU). The swiotlb is basically a
>> pre-allocated bounce-buffer area. Because it is pre-allocated it is a waste
>> of memory if it isn't used and, since domUs do not normally perform real
>> physical I/O directly, we therefore by default only create a swiotlb for
>> dom0. You then have to override on the cmdline if this default doesn't
>> actually work (as in this case).
> 
> ...and why is it not possible (from the dom0 point of view) to detect
> this behaviour while starting the domU, reading it's conf and stop the
> boot process with a warning?

IOMMU_BUG_ON() is supposed to printk a suitable warnign before actually
crashing. Not sure why that didn't work for you.

 -- Keir