[BUG?] cavium cn56xx and dma_map_single warning

[BUG?] cavium cn56xx and dma_map_single warning

[Zhuang Yuyao]

Hi,

I've got a strange dma error on my cavium cn56xx board.

......
dma_map_single: Warning: Mapping memory address that might conflict
with devices 0xfbb8f000-0xfbb8ffff
dma_map_single: Warning: Mapping memory address that might conflict
with devices 0xfb2ef000-0xfb2f0fff
dma_map_single: Warning: Mapping memory address that might conflict
with devices 0xfb3de000-0xfb3defff
dma_map_single: Warning: Mapping memory address that might conflict
with devices 0xfb3dd000-0xfb3ddfff
dma_map_single: Warning: Mapping memory address that might conflict
with devices 0xfac28000-0xfac29fff
dma_map_single: Warning: Mapping memory address that might conflict
with devices 0xfad1a000-0xfad1bfff
.....

this error appeared when I plugged 4G or 8G ram on the board, if there
is only 2G ram on the board, the error never happened.

an adaptec 3405 pci-e raid card is plugged on the board, and a
harddisk attached. the program i am running is a postgresql server
with more than 20 million record, so do a count(*) will let postgresql
consume almost all of the available memory. while there is 4G ram, the
'free' command reported that the free memory is below 18M.

then the error kept printing on the console for a while, then kernel panic.

the warning message is come from arch/mips/cavium-octeon/dma-octeon.c

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=arch/mips/cavium-octeon/dma-octeon.c;h=be531ec1f2064b590b58dfe8b4db4f5534999bab;hb=HEAD

i tried several raid card from different producer, including LSI and
Adaptec. the same error always happened while there are more than or
equal to 4G memory on the board.

Would you please give some advice on this issue? any help will be appreciated.

Thanks.

Zhuang Yuyao

Re: [BUG?] cavium cn56xx and dma_map_single warning

[Zhuang Yuyao]

Here are some additional information.I added a dump_stack() where the
warning happened and changed pr_warning() to panic(). here is the
output. I am using linux 2.6.32.9.

Is it scsi drivers where i should look into, to make the scsi  do not
request for memory reserved by octeon?

[<ffffffff8110e718>] dump_stack+0x8/0x34
[<ffffffff811144ec>] octeon_map_dma_mem+0x4dc/0x538
[<ffffffff81126a50>] dma_map_sg+0xa0/0xe0
[<ffffffff812e1c98>] scsi_dma_map+0x40/0x50
[<ffffffff8130297c>] aac_build_sgraw+0x44/0x1a0
[<ffffffff81302cd0>] aac_read_raw_io+0x98/0x120
[<ffffffff81300f50>] aac_scsi_cmd+0xb48/0x14b8
[<ffffffff812ff39c>] aac_queuecommand+0x94/0xa8
[<ffffffff812d9c74>] scsi_dispatch_cmd+0x104/0x260
[<ffffffff812dff94>] scsi_request_fn+0x374/0x430
[<ffffffff81279e48>] generic_unplug_device+0x38/0x50
[<ffffffff8118328c>] sync_page+0x4c/0x70
[<ffffffff811832c0>] sync_page_killable+0x10/0x48
[<ffffffff8110fb2c>] __wait_on_bit_lock+0xcc/0x158
[<ffffffff811831b0>] __lock_page_killable+0x50/0x60
[<ffffffff8118549c>] generic_file_aio_read+0x444/0x6c0
[<ffffffff811b81dc>] do_sync_read+0xbc/0x120
[<ffffffff811b8fc4>] vfs_read+0xb4/0x178
[<ffffffff811b9170>] SyS_read+0x48/0xa0
[<ffffffff81102bc4>] handle_sys64+0x44/0x60

Kernel panic - not syncing: dma_map_single: conflict 0xffc00000


On Mon, Mar 22, 2010 at 3:01 PM, Zhuang Yuyao <mlistz@gmail.com> wrote:
> Hi,
>
> I've got a strange dma error on my cavium cn56xx board.
>
> ......
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfbb8f000-0xfbb8ffff
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfb2ef000-0xfb2f0fff
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfb3de000-0xfb3defff
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfb3dd000-0xfb3ddfff
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfac28000-0xfac29fff
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfad1a000-0xfad1bfff
> .....
>
> this error appeared when I plugged 4G or 8G ram on the board, if there
> is only 2G ram on the board, the error never happened.
>
> an adaptec 3405 pci-e raid card is plugged on the board, and a
> harddisk attached. the program i am running is a postgresql server
> with more than 20 million record, so do a count(*) will let postgresql
> consume almost all of the available memory. while there is 4G ram, the
> 'free' command reported that the free memory is below 18M.
>
> then the error kept printing on the console for a while, then kernel panic.
>
> the warning message is come from arch/mips/cavium-octeon/dma-octeon.c
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=arch/mips/cavium-octeon/dma-octeon.c;h=be531ec1f2064b590b58dfe8b4db4f5534999bab;hb=HEAD
>
> i tried several raid card from different producer, including LSI and
> Adaptec. the same error always happened while there are more than or
> equal to 4G memory on the board.
>
> Would you please give some advice on this issue? any help will be appreciated.
>
> Thanks.
>
> Zhuang Yuyao
>

Re: [BUG?] cavium cn56xx and dma_map_single warning

[David Daney]

On 03/22/2010 12:01 AM, Zhuang Yuyao wrote:
> Hi,
>
> I've got a strange dma error on my cavium cn56xx board.
>
> ......
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfbb8f000-0xfbb8ffff
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfb2ef000-0xfb2f0fff
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfb3de000-0xfb3defff
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfb3dd000-0xfb3ddfff
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfac28000-0xfac29fff
> dma_map_single: Warning: Mapping memory address that might conflict
> with devices 0xfad1a000-0xfad1bfff
> .....
>
> this error appeared when I plugged 4G or 8G ram on the board, if there
> is only 2G ram on the board, the error never happened.
>
> an adaptec 3405 pci-e raid card is plugged on the board, and a
> harddisk attached. the program i am running is a postgresql server
> with more than 20 million record, so do a count(*) will let postgresql
> consume almost all of the available memory. while there is 4G ram, the
> 'free' command reported that the free memory is below 18M.
>
> then the error kept printing on the console for a while, then kernel panic.
>
> the warning message is come from arch/mips/cavium-octeon/dma-octeon.c
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=arch/mips/cavium-octeon/dma-octeon.c;h=be531ec1f2064b590b58dfe8b4db4f5534999bab;hb=HEAD
>
> i tried several raid card from different producer, including LSI and
> Adaptec. the same error always happened while there are more than or
> equal to 4G memory on the board.
>
> Would you please give some advice on this issue? any help will be appreciated.

This is a known issue.

passing mem==3072M will restrict kernel memory usage thus avoiding the 
issue.

David Daney



>
> Thanks.
>
> Zhuang Yuyao
>

Re: [BUG?] cavium cn56xx and dma_map_single warning

[Zhuang Yuyao]

Thanks for your reply. Will this issue be solved soon? Is it a
hardware issue or a software one? Can I make the 4G-256M memory
reserved so that kernel will not try to allocate memory in this area?

can not use more than 4G ram is bothersome since the memory is so cheap now. :-)

Thanks very much.

On Tue, Mar 23, 2010 at 12:44 AM, David Daney <ddaney@caviumnetworks.com> wrote:
>
> This is a known issue.
>
> passing mem==3072M will restrict kernel memory usage thus avoiding the
> issue.
>
> David Daney
>

Re: [BUG?] cavium cn56xx and dma_map_single warning

[David Daney]

On 03/22/2010 06:30 PM, Zhuang Yuyao wrote:
> Thanks for your reply. Will this issue be solved soon? Is it a
> hardware issue or a software one?

A combination.  I think a Software fix is possible.

> Can I make the 4G-256M memory
> reserved so that kernel will not try to allocate memory in this area?
>

I have never tried it.

The issue is maintaining mappings for 32-bit PCI devices.  If you only 
want to support 64-bit devices, it would be easier to address the issue.

David Daney

> can not use more than 4G ram is bothersome since the memory is so cheap now. :-)
>
> Thanks very much.
>
> On Tue, Mar 23, 2010 at 12:44 AM, David Daney<ddaney@caviumnetworks.com>  wrote:
>>
>> This is a known issue.
>>
>> passing mem==3072M will restrict kernel memory usage thus avoiding the
>> issue.
>>
>> David Daney
>>
>

Re: [BUG?] cavium cn56xx and dma_map_single warning

[Zhuang Yuyao]

On Tue, Mar 23, 2010 at 9:39 AM, David Daney <ddaney@caviumnetworks.com> wrote:

> The issue is maintaining mappings for 32-bit PCI devices.  If you only want
> to support 64-bit devices, it would be easier to address the issue.
>
> David Daney
>

Wow, that is a great news, I do not known if my adaptec 3045e raid
card is a 64bit device, but if I only want to support 64bit devices,
Is there a quick fix for it?

Zhuang Yuyao

Re: [BUG?] cavium cn56xx and dma_map_single warning

[David Daney]

On 03/22/2010 06:45 PM, Zhuang Yuyao wrote:
> On Tue, Mar 23, 2010 at 9:39 AM, David Daney<ddaney@caviumnetworks.com>  wrote:
>
>> The issue is maintaining mappings for 32-bit PCI devices.  If you only want
>> to support 64-bit devices, it would be easier to address the issue.
>>
>> David Daney
>>
>
> Wow, that is a great news, I do not known if my adaptec 3045e raid
> card is a 64bit device, but if I only want to support 64bit devices,
> Is there a quick fix for it?
>

I don't know.  You could try to make all DMA accesses go via BAR2.  That 
would break many 32-bit devices though.

David Daney

> Zhuang Yuyao
>