[BUG] Bad page map in process ibv_devinfo

[BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

Hello Mr. Sokolovsky,

i've built OFED-1.5.3.2 for a linux-2.6.39.4 sparc64 (vanilla) kernel. The IPoIB stuff works fine.

If I run 'ibv_devinfo' I get:
---

hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.6.628
        node_guid:                      0003:ba00:0100:b1d8
        sys_image_guid:                 0003:ba00:0100:b1db
        vendor_id:                      0x03ba
        vendor_part_id:                 25418
        hw_ver:                         0xA0
        board_id:                       SUN0070000001
        phys_port_cnt:                  2
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 1
                        port_lid:               6
                        port_lmc:               0x00
                        link_layer:             IB[...]
---

But I also get this BUG message from the kernel:
---
[ 9305.698663] swap_free: Bad swap file entry 100005e000061800
[ 9305.698791] BUG: Bad page map in process ibv_devinfo  pte:bc0000c300104848 pmd:00f38054
[ 9305.698908] addr:fffff80100114000 vm_flags:000844fa anon_vma:          (null) mapping:fffff807f313a410 index:6180082
[ 9305.699087] vma->vm_file->f_op->mmap: ib_uverbs_mmap+0x8/0x38 [ib_uverbs]
[ 9305.699135] Call Trace:
[ 9305.699174]  [00000000004cd558] unmap_vmas+0x514/0x7f4
[ 9305.699302]  [00000000004d1274] unmap_region+0xb4/0x164
[ 9305.699383]  [00000000004d22c0] do_munmap+0x2a8/0x31c
[ 9305.699467]  [000000000042d350] SyS_64_munmap+0x88/0xa8
[ 9305.699550]  [0000000000406154] linux_sparc_syscall+0x34/0x44
---


I've the following
- System: Sun T5120 (SPARC64)

- OS: Debian 6.0.3
- IB-card:
# lspci -s 12:00

12:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0)


And same BUG messages with:
- OFED-1.5.3.2 with linux-2.6.38 (vanilla)

- OFED-1.5.4-rc4 with linux-2.6.39.4 (vanilla)


The problem is that I also get such messages when I run openmpi over openib (naturally with another process name than 'ibv_devinfo'). Then mpirun dies with an error message. Hence I can't use MPI over the Infiniband cards...


So is there something I could try to do to get these cards working?
Can I give you more debug information?


I'm very happy about any help!


Regards,
Lukas



PS:
My kernel-config:
http://net.razik.de/linux/T5120/config-2.6.39.4-razik-2011-11-05


The OFED-packets I've installed:

---

kernel-ib
ofed-scripts
libibverbs
libibverbs-devel
libibverbs-utils
libmlx4
libmlx4-devel
libibumad
libibumad-devel
libibmad
libibmad-devel
librdmacm
librdmacm-utils
librdmacm-devel
opensm-libs
ibutils
infiniband-diags
ofed-docs
mpi-selector
openmpi_gcc
mpitests_openmpi_gcc
---

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Vladimir Sokolovsky]

On 11/12/2011 05:15 PM, Lukas Razik wrote:
> Hello Mr. Sokolovsky,
>
> i've built OFED-1.5.3.2 for a linux-2.6.39.4 sparc64 (vanilla) kernel. The IPoIB stuff works fine.
>
> If I run 'ibv_devinfo' I get:
> ---
>
> hca_id: mlx4_0
>          transport:                      InfiniBand (0)
>          fw_ver:                         2.6.628
>          node_guid:                      0003:ba00:0100:b1d8
>          sys_image_guid:                 0003:ba00:0100:b1db
>          vendor_id:                      0x03ba
>          vendor_part_id:                 25418
>          hw_ver:                         0xA0
>          board_id:                       SUN0070000001
>          phys_port_cnt:                  2
>                  port:   1
>                          state:                  PORT_ACTIVE (4)
>                          max_mtu:                2048 (4)
>                          active_mtu:             2048 (4)
>                          sm_lid:                 1
>                          port_lid:               6
>                          port_lmc:               0x00
>                          link_layer:             IB[...]
> ---
>


Hello Lukas,
Try to update HCA's firmware to the latest version (http://www.mellanox.com/content/pages.php?pg=firmware_download).

Regards,
Vladimir



> But I also get this BUG message from the kernel:
> ---
> [ 9305.698663] swap_free: Bad swap file entry 100005e000061800
> [ 9305.698791] BUG: Bad page map in process ibv_devinfo  pte:bc0000c300104848 pmd:00f38054
> [ 9305.698908] addr:fffff80100114000 vm_flags:000844fa anon_vma:          (null) mapping:fffff807f313a410 index:6180082
> [ 9305.699087] vma->vm_file->f_op->mmap: ib_uverbs_mmap+0x8/0x38 [ib_uverbs]
> [ 9305.699135] Call Trace:
> [ 9305.699174]  [00000000004cd558] unmap_vmas+0x514/0x7f4
> [ 9305.699302]  [00000000004d1274] unmap_region+0xb4/0x164
> [ 9305.699383]  [00000000004d22c0] do_munmap+0x2a8/0x31c
> [ 9305.699467]  [000000000042d350] SyS_64_munmap+0x88/0xa8
> [ 9305.699550]  [0000000000406154] linux_sparc_syscall+0x34/0x44
> ---
>
>
> I've the following
> - System: Sun T5120 (SPARC64)
>
> - OS: Debian 6.0.3
> - IB-card:
> # lspci -s 12:00
>
> 12:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0)
>
>
> And same BUG messages with:
> - OFED-1.5.3.2 with linux-2.6.38 (vanilla)
>
> - OFED-1.5.4-rc4 with linux-2.6.39.4 (vanilla)
>
>
> The problem is that I also get such messages when I run openmpi over openib (naturally with another process name than 'ibv_devinfo'). Then mpirun dies with an error message. Hence I can't use MPI over the Infiniband cards...
>
>
> So is there something I could try to do to get these cards working?
> Can I give you more debug information?
>
>
> I'm very happy about any help!
>
>
> Regards,
> Lukas
>
>
>
> PS:
> My kernel-config:
> http://net.razik.de/linux/T5120/config-2.6.39.4-razik-2011-11-05
>
>
> The OFED-packets I've installed:
>
> ---
>
> kernel-ib
> ofed-scripts
> libibverbs
> libibverbs-devel
> libibverbs-utils
> libmlx4
> libmlx4-devel
> libibumad
> libibumad-devel
> libibmad
> libibmad-devel
> librdmacm
> librdmacm-utils
> librdmacm-devel
> opensm-libs
> ibutils
> infiniband-diags
> ofed-docs
> mpi-selector
> openmpi_gcc
> mpitests_openmpi_gcc
> ---
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

Vladimir Sokolovsky <vlad-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> On 11/12/2011 05:15 PM, Lukas Razik wrote:

>>  Hello Mr. Sokolovsky,
>> 
>>  i've built OFED-1.5.3.2 for a linux-2.6.39.4 sparc64 (vanilla) kernel. 
> The IPoIB stuff works fine.
>> 
>>  If I run 'ibv_devinfo' I get:
>>  ---
>> 
>>  hca_id: mlx4_0
>>           transport:                      InfiniBand (0)
>>           fw_ver:                         2.6.628
>>           node_guid:                      0003:ba00:0100:b1d8
>>           sys_image_guid:                 0003:ba00:0100:b1db
>>           vendor_id:                      0x03ba
>>           vendor_part_id:                 25418
>>           hw_ver:                         0xA0
>>           board_id:                       SUN0070000001
>>           phys_port_cnt:                  2
>>                   port:   1
>>                           state:                  PORT_ACTIVE (4)
>>                           max_mtu:                2048 (4)
>>                           active_mtu:             2048 (4)
>>                           sm_lid:                 1
>>                           port_lid:               6
>>                           port_lmc:               0x00
>>                           link_layer:             IB[...]
>>  ---
>> 
> 
> 
> Hello Lukas,
> Try to update HCA's firmware to the latest version 
> (http://www.mellanox.com/content/pages.php?pg=firmware_download).


Hello Vladimir,

many, many thanks for your fast answer!
But now I have two big problems:

1. problem: The board ID is SUN0070000001 .
If I compare the ibv_devinfo output (mstflint doesn't show me the chip rev.) with the Mellanox table I think it's a ConnectX adapter, isn't it?
http://www.mellanox.com/content/pages.php?pg=firmware_HCA_FW_identification

So do I get the INI file _and_ the FW file from Oracle?
If I get the INI file only - do I need the following FW file then?
http://www.mellanox.com/downloads/firmware/disclaimer.php?download=/downloads/firmware/ConnectX-rel-2_9_1000.tgz

2. problem: For burning the firmware I need the MFT package, right?
But Mellanox doesn't provide it for sparc64.
I could build the kernel modules of kernel-mft-2.7.1-7.src.rpm without any problems.
But the user space stuff is ppc64, x86 and x86_64 only...
Is there maybe a possibility to get sparc64 binaries?

Best regards,
Lukas


PS: That's the output of mstflint
# mstflint -d 12:00.0 q
Image type:      ConnectX
FW Version:      2.6.628
Device ID:       25418
Description:     Node             Port1            Port2            Sys image
GUIDs:           0003ba000100b1d8 0003ba000100b1d9 0003ba000100b1da 0003ba000100b1db 
MACs:                                 0002c9000001     0002c9000002
Board ID:        n/a (SUN0070000001)
VSD:             n/a
PSID:            SUN0070000001
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

>>    hca_id: mlx4_0

>>             transport:                      InfiniBand (0)
>>             fw_ver:                         2.6.628
>>             node_guid:                      0003:ba00:0100:b1d8
>>             sys_image_guid:                 0003:ba00:0100:b1db
>>             vendor_id:                      0x03ba
>>             vendor_part_id:                 25418
>>             hw_ver:                         0xA0
>>             board_id:                       SUN0070000001
>>             phys_port_cnt:                  2
>>                     port:   1
>>                             state:                  PORT_ACTIVE (4)
>>                             max_mtu:                2048 (4)
>>                             active_mtu:             2048 (4)
>>                             sm_lid:                 1
>>                             port_lid:               6
>>                             port_lmc:               0x00
>>                             link_layer:             IB[...]
>>  
>>
>>  2. problem: For burning the firmware I need the MFT package, right?
>>  But Mellanox doesn't provide it for sparc64.
>>  I could build the kernel modules of kernel-mft-2.7.1-7.src.rpm without any 
> problems.
>>  But the user space stuff is ppc64, x86 and x86_64 only...
>>  Is there maybe a possibility to get sparc64 binaries?
>
> You can burn FW binary image using mstflint.
> Use MFT (mlxburn) to create the binary image from FW mlx file and ini file. You 
> can install MFT on some Linux or Windows machine.


Hello Vladimir,

thanks again for this tip!

I need a "/dev/mst/mt25418_pci_cr0" when I use mstflint.
Therefore I've extracted the "/etc/init.d/mst" script from the "mft-2.7.1-7.x86.rpm"
which creates a "/dev/mst/mt25418_pci_cr0".

But the script outputs errors and I can't use the created device:
# mstflint -d /dev/mst/mt25418_pci_cr0 q
Unable to parse device name /dev/mst/mt25418_pci_cr0
-E- Can not open /dev/mst/mt25418_pci_cr0: Invalid argument MFE_CR_ERROR

Maybe it works when I create "/dev/mst/mt25418_pci_cr0" manually but I don't know how.
So could you maybe write a short instruction list for me what I have to do after loading the mst_* modules to get the needed device file? Or do you maybe have another idea how I could flash the firmware?

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

>>>     hca_id: mlx4_0
>>>              transport:                      InfiniBand (0)

>>>              fw_ver:                         2.6.628
>>>              node_guid:                      0003:ba00:0100:b1d8
>>>              sys_image_guid:                 0003:ba00:0100:b1db
>>>              vendor_id:                      0x03ba
>>>              vendor_part_id:                 25418
>>>              hw_ver:                         0xA0
>>>              board_id:                       SUN0070000001
>>>              phys_port_cnt:                  2
>>>                      port:   1
>>>                              state:                  PORT_ACTIVE (4)
>>>                              max_mtu:                2048 (4)
>>>                              active_mtu:             2048 (4)
>>>                              sm_lid:                 1
>>>                              port_lid:               6
>>>                              port_lmc:               0x00
>>>                              link_layer:             IB[...]
>>>   
>>> 
>>  You can burn FW binary image using mstflint.
>
> I need a "/dev/mst/mt25418_pci_cr0" when I use mstflint.
> Therefore I've extracted the "/etc/init.d/mst" script from the 
> "mft-2.7.1-7.x86.rpm"
> which creates a "/dev/mst/mt25418_pci_cr0".


Hello again, Vladimir!

To set up "/dev/mst/mt25418_pci_cr0" manually I've followed up what "/etc/init.d/mst start" does.
I've seen that it uses the "minit" tool which is part of the arch dependent mft-package.

Is there maybe a possibility to get "minit" for sparc64?
Otherwise I can't init "/dev/mst/mt25418_pci_cr0"... :-(

Many thanks for your time!

Best regards,
Lukas


PS: These are my steps:
# mkdir /dev/mst
# modprobe mst_pci
# modprobe mst_pciconf

I've looked up the pcimajor and piconfmajor :
# cat /proc/devices | grep mst_
251 mst_pciconf
252 mst_pci

Then the script finds the follwing PCI-card from its 'dev_id_database' :
# lspci -m -n -d 15b3:634a
12:00.0 "0c06" "15b3" "634a" -ra0 "15b3" "634a"
which is the ConnectX controller:
# lspci -s 12:00.0
12:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0)

With all these information it creates the device file:
# mknod -m 666 /dev/mst/mt25418_pci_cr0 c 252 0

And now I need the "minit" command from the mft package.
# /usr/bin/minit /dev/mst/mt25418_pci_cr0 12:00.0 0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

> Hello again, Vladimir!

> 
> To set up "/dev/mst/mt25418_pci_cr0" manually I've followed up 
> what "/etc/init.d/mst start" does.
> I've seen that it uses the "minit" tool which is part of the arch 
> dependent mft-package.
> 
> Is there maybe a possibility to get "minit" for sparc64?
> Otherwise I can't init "/dev/mst/mt25418_pci_cr0"... :-(
> 

Oh, I'm sorry! I've read this file from Mellanox:
http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf
which I've found here:
http://www.mellanox.com/content/pages.php?pg=management_tools&menu_section=34
where the firmware flashing procedure is described for flint.

And I thought flint == mstflint because mstflint shows it's flint:
# mstflint 
            FLINT - FLash INTerface
SYNOPSIS
    flint [switches...] <command> [parameters...]

Now I've seen that I don't need "/dev/mst/mt25418_pci_cr0" for mstflint.
So if I want to burn a firmware on the adapter I only have to write:
mstflint -d 12:00.0 -i fw.bin burn
Will the guid and the mac stay the same as before?

A big sorry again!

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Roland Dreier]

On Sun, Nov 13, 2011 at 12:26 AM, Vladimir Sokolovsky
<vlad-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> Try to update HCA's firmware to the latest version
> (http://www.mellanox.com/content/pages.php?pg=firmware_download).

Independent of a firmware update, have you tried with an unpatched (upstream)
kernel?  2.6.39.4 would be fine, or you could try 3.0 or 3.1.

We shouldn't BUG no matter what FW version is burned on an HCA.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:

> On Sun, Nov 13, 2011 at 12:26 AM, Vladimir Sokolovsky
> <vlad-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
>>  Try to update HCA's firmware to the latest version
>>  (http://www.mellanox.com/content/pages.php?pg=firmware_download).
> 
> Independent of a firmware update, have you tried with an unpatched (upstream)
> kernel?  2.6.39.4 would be fine, or you could try 3.0 or 3.1.
> 
> We shouldn't BUG no matter what FW version is burned on an HCA.

Hello Roland!

Yes, I've tested:
- OFED-1.5.3.2 with linux-2.6.39.4 (vanilla)
- OFED-1.5.3.2 with linux-2.6.38 (vanilla)
- OFED-1.5.4-rc4 with linux-2.6.39.4 (vanilla)

That's the original post:
http://thread.gmane.org/gmane.linux.drivers.rdma/10157

I want(ed) to test linux-3.1.1 (vanilla) but I had no time.
(I've tried to get MFT from Mellanox working - now I know I don't need it).
linux-3.1.1 will be the next I try while waiting for the new firmware.

Thanks for your answer!

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Vladimir Sokolovsky]

On 11/15/2011 12:42 AM, Lukas Razik wrote:
>> Hello again, Vladimir!
>
>>
>> To set up "/dev/mst/mt25418_pci_cr0" manually I've followed up
>> what "/etc/init.d/mst start" does.
>> I've seen that it uses the "minit" tool which is part of the arch
>> dependent mft-package.
>>
>> Is there maybe a possibility to get "minit" for sparc64?
>> Otherwise I can't init "/dev/mst/mt25418_pci_cr0"... :-(
>>
>
> Oh, I'm sorry! I've read this file from Mellanox:
> http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf
> which I've found here:
> http://www.mellanox.com/content/pages.php?pg=management_tools&menu_section=34
> where the firmware flashing procedure is described for flint.
>
> And I thought flint == mstflint because mstflint shows it's flint:
> # mstflint
>              FLINT - FLash INTerface
> SYNOPSIS
>      flint [switches...]<command>  [parameters...]
>
> Now I've seen that I don't need "/dev/mst/mt25418_pci_cr0" for mstflint.
> So if I want to burn a firmware on the adapter I only have to write:
> mstflint -d 12:00.0 -i fw.bin burn
> Will the guid and the mac stay the same as before?
>
> A big sorry again!
>
> Regards,
> Lukas
>

Hi Lukas,
GUID and MAC will stay the same as before.

Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Roland Dreier]

On Sat, Nov 12, 2011 at 7:15 AM, Lukas Razik <linux-GLgANOly0l1BDLzU/O5InQ@public.gmane.org> wrote:
> But I also get this BUG message from the kernel:
> ---
> [ 9305.698663] swap_free: Bad swap file entry 100005e000061800
> [ 9305.698791] BUG: Bad page map in process ibv_devinfo  pte:bc0000c300104848 pmd:00f38054
> [ 9305.698908] addr:fffff80100114000 vm_flags:000844fa anon_vma:          (null) mapping:fffff807f313a410 index:6180082
> [ 9305.699087] vma->vm_file->f_op->mmap: ib_uverbs_mmap+0x8/0x38 [ib_uverbs]
> [ 9305.699135] Call Trace:
> [ 9305.699174]  [00000000004cd558] unmap_vmas+0x514/0x7f4
> [ 9305.699302]  [00000000004d1274] unmap_region+0xb4/0x164
> [ 9305.699383]  [00000000004d22c0] do_munmap+0x2a8/0x31c
> [ 9305.699467]  [000000000042d350] SyS_64_munmap+0x88/0xa8
> [ 9305.699550]  [0000000000406154] linux_sparc_syscall+0x34/0x44

Hi Sparc Linux hackers!

This message is coming from someone using a ConnectX (mlx4) adapter on
a Sun T5120 (SPARC64).

I'm wondering if the bug is in the mlx4 code or in the sparc
io_remap_pfn_range()
code.  From the "Bad page map" pte value, if I understand sparc mm correctly,
the PTE is seen as not present.

The mapping is coming from the code in drivers/infiniband/hw/mlx4/main.c:

static int mlx4_ib_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
{
	struct mlx4_ib_dev *dev = to_mdev(context->device);

	if (vma->vm_end - vma->vm_start != PAGE_SIZE)
		return -EINVAL;

	if (vma->vm_pgoff == 0) {
		vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);

		if (io_remap_pfn_range(vma, vma->vm_start,
				       to_mucontext(context)->uar.pfn,
				       PAGE_SIZE, vma->vm_page_prot))
			return -EAGAIN;
	} else if (vma->vm_pgoff == 1 && dev->dev->caps.bf_reg_size != 0) {
		vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);

		if (io_remap_pfn_range(vma, vma->vm_start,
				       to_mucontext(context)->uar.pfn +
				       dev->dev->caps.num_uars,
				       PAGE_SIZE, vma->vm_page_prot))
			return -EAGAIN;
	} else
		return -EINVAL;

	return 0;
}

which is just taking one page -- the pfn comes from the PCI BAR
resource value in

int mlx4_uar_alloc(struct mlx4_dev *dev, struct mlx4_uar *uar)
{
	uar->index = mlx4_bitmap_alloc(&mlx4_priv(dev)->uar_table.bitmap);
	if (uar->index == -1)
		return -ENOMEM;

	uar->pfn = (pci_resource_start(dev->pdev, 2) >> PAGE_SHIFT) + uar->index;
	uar->map = NULL;

	return 0;
}

and calling io_remap_pfn_range on it.

Is there some VMA flag we're forgetting to set that only affects sparc?
Or could this possible be a bug in the sparc handling of PFN maps?

Thanks!
  Roland
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Roland Dreier]

On Mon, Nov 14, 2011 at 4:03 PM, Lukas Razik <linux-GLgANOly0l1BDLzU/O5InQ@public.gmane.org> wrote:
> That's the original post:
> http://thread.gmane.org/gmane.linux.drivers.rdma/10157
>


Sorry, I missed the "vanilla" in your original email and just saw "OFED".

Anyway yes, looks like a bug in the mlx4 driver.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Roland Dreier]

On Tue, Nov 15, 2011 at 11:10 AM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> Sorry, I missed the "vanilla" in your original email and just saw "OFED".
>
> Anyway yes, looks like a bug in the mlx4 driver.

Actually where is your mlx4 driver coming from?  The upstream kernel or OFED?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:

> On Tue, Nov 15, 2011 at 11:10 AM, Roland Dreier <roland@purestorage.com> 
> wrote:
>>  Sorry, I missed the "vanilla" in your original email and just saw 
> "OFED".
>> 
>>  Anyway yes, looks like a bug in the mlx4 driver.
> 
> Actually where is your mlx4 driver coming from?  The upstream kernel or OFED?

It comes from OFED.
Here it's from OFED-1.5.4 (although you see v1.0-ofed1.5.3 in the kernel log).I think at this point it's useful to show all relevant (i.e. ofed regarding) parts of the kernel log:

[  195.010076] mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.3 (January 19, 2011)
[  195.010208] mlx4_core: Initializing 0000:12:00.0
[  195.010294] PCI: Enabling device: (0000:12:00.0), cmd 2
[  195.010356] mlx4_core 0000:12:00.0: Warning: couldn't set 64-bit PCI DMA mask.
[  195.010469] mlx4_core 0000:12:00.0: Warning: couldn't set 64-bit consistent PCI DMA mask.
<snip>
[  197.479244] Kernel unaligned access at TPC[10061624] mlx4_QUERY_ADAPTER+0xf0/0x11c [mlx4_core]
[  197.479424] Kernel unaligned access at TPC[10061624] mlx4_QUERY_ADAPTER+0xf0/0x11c [mlx4_core]
[  197.479594] Kernel unaligned access at TPC[10061624] mlx4_QUERY_ADAPTER+0xf0/0x11c [mlx4_core]
[  197.479764] Kernel unaligned access at TPC[10061624] mlx4_QUERY_ADAPTER+0xf0/0x11c [mlx4_core]

<snip>

[  231.377790] mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.3 (January 19, 2011)
[  232.217620] ADDRCONF(NETDEV_UP): ib0: link is not ready
[  233.782679] ib0: enabling connected mode will cause multicast packet drops
[  233.782810] ib0: mtu > 2044 will cause multicast packet drops.
[  233.782916] ib0: mtu > 2044 will cause multicast packet drops.
[  234.860368] ib1: enabling connected mode will cause multicast packet drops
[  234.860502] ib1: mtu > 2044 will cause multicast packet drops.
[  234.860613] ib1: mtu > 2044 will cause multicast packet drops.
[  235.225652] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
[  246.210384] ib0: no IPv6 routers present
[  650.857233] swap_free: Bad swap file entry 100005e000061800
[  650.857363] BUG: Bad page map in process ibv_devinfo  pte:bc0000c300104848 pmd:00fc9bc4
[  650.857481] addr:fffff80100114000 vm_flags:000844fa anon_vma:          (null) mapping:fffff807f6e06b38 index:6180082
[  650.857661] vma->vm_file->f_op->mmap: ib_uverbs_mmap+0x8/0x38 [ib_uverbs]
[  650.857708] Call Trace:
[  650.857747]  [00000000004cd430] unmap_vmas+0x514/0x7f4
[  650.857826]  [00000000004d114c] unmap_region+0xb4/0x164
[  650.857875]  [00000000004d2198] do_munmap+0x2a8/0x31c
[  650.857958]  [000000000042d340] SyS_64_munmap+0x88/0xa8
[  650.858042]  [0000000000406154] linux_sparc_syscall+0x34/0x44
[  650.858116] Disabling lock debugging due to kernel taint


Please ask if you need more information. I'll provide them promptly.

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Roland Dreier]

On Tue, Nov 15, 2011 at 12:03 PM, Lukas Razik <linux-GLgANOly0l1BDLzU/O5InQ@public.gmane.org> wrote:
>> Actually where is your mlx4 driver coming from?  The upstream kernel or OFED?
>
> It comes from OFED.

If it's possible, it would be useful to try with the vanilla kernel
and all upstream
modules too.  Otherwise I can't rule out the possiblity that we're chasing a
bug that OFED introduces.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:

> If it's possible, it would be useful to try with the vanilla kernel
> and all upstream
> modules too.  Otherwise I can't rule out the possiblity that we're 
> chasing a
> bug that OFED introduces.

OK, I've tested the modules from the kernel.

What I've done:
I've built linux-2.6.39.4 (vanilla) with the belonging Infiniband drivers.
I've installed OFED-1.5.4-rc4 as usual and deleted the new modules in '/lib/modules/<version>/update/drivers' (because we want to use the linux-2.6.39.4 modules).
I've run 'depmod -a'.

When I executed '/etc/init.d/openib start' I've got this kernel log:
[ 1438.479977] mlx4_core: Mellanox ConnectX core driver v0.01 (May 1, 2007)
[ 1438.480106] mlx4_core: Initializing 0000:12:00.0
[ 1438.480229] mlx4_core 0000:12:00.0: Warning: couldn't set 64-bit PCI DMA mask.
[ 1438.480348] mlx4_core 0000:12:00.0: Warning: couldn't set 64-bit consistent PCI DMA mask.
[ 1440.546533] Kernel unaligned access at TPC[102df650] mlx4_QUERY_ADAPTER+0xf0/0x11c [mlx4_core]
[ 1440.546704] Kernel unaligned access at TPC[102df650] mlx4_QUERY_ADAPTER+0xf0/0x11c [mlx4_core]
[ 1440.546865] Kernel unaligned access at TPC[102df650] mlx4_QUERY_ADAPTER+0xf0/0x11c [mlx4_core]
[ 1440.547026] Kernel unaligned access at TPC[102df650] mlx4_QUERY_ADAPTER+0xf0/0x11c [mlx4_core]
[ 1440.771361] mlx4_core 0000:12:00.0: Sense command failed for port: 1
[ 1440.771926] mlx4_core 0000:12:00.0: Sense command failed for port: 2
[ 1440.884909] mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
[ 1441.469341] ADDRCONF(NETDEV_UP): ib0: link is not ready
[ 1442.860385] ib0: enabling connected mode will cause multicast packet drops
[ 1442.886875] ib0: mtu > 2044 will cause multicast packet drops.
[ 1443.932557] ib1: enabling connected mode will cause multicast packet drops
[ 1443.938749] ib1: mtu > 2044 will cause multicast packet drops.
[ 1444.481266] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready


Then I executed 'ibv_devinfo' and got erros (which I hadn't with the kernel modules from OFED):
# ibv_devinfo 
mlx4: There is a mismatch between the kernel and the userspace libraries: Kernel does not support XRC. Exiting.
Failed to open device

And the error in the kernel log was existent, too:
[ 1464.503632] swap_free: Bad swap file entry 100005e000061800
[ 1464.503759] BUG: Bad page map in process ibv_devinfo  pte:bc0000c300104848 pmd:00fe46f0
[ 1464.503877] addr:fffff80100114000 vm_flags:000844fa anon_vma:          (null) mapping:fffff807f401b6a0 index:6180082
[ 1464.504054] vma->vm_file->f_op->mmap: ib_uverbs_mmap+0x8/0x38 [ib_uverbs]
[ 1464.504102] Call Trace:
[ 1464.504141]  [00000000004cd430] unmap_vmas+0x514/0x7f4
[ 1464.504191]  [00000000004d114c] unmap_region+0xb4/0x164
[ 1464.504269]  [00000000004d2198] do_munmap+0x2a8/0x31c
[ 1464.504323]  [000000000042d340] SyS_64_munmap+0x88/0xa8
[ 1464.504378]  [0000000000406154] linux_sparc_syscall+0x34/0x44
[ 1464.504423] Disabling lock debugging due to kernel taint
[ 1464.504478] swap_free: Bad swap file entry 100005e000061802
[ 1464.504526] BUG: Bad page map in process ibv_devinfo  pte:bc0000c300504848 pmd:00fe46f0
[ 1464.504585] addr:fffff8010011a000 vm_flags:000844fa anon_vma:          (null) mapping:fffff807f401b6a0 index:6180282
[ 1464.504665] vma->vm_file->f_op->mmap: ib_uverbs_mmap+0x8/0x38 [ib_uverbs]
[ 1464.504711] Call Trace:
[ 1464.504742]  [00000000004cd430] unmap_vmas+0x514/0x7f4
[ 1464.504790]  [00000000004d114c] unmap_region+0xb4/0x164
[ 1464.504838]  [00000000004d2198] do_munmap+0x2a8/0x31c
[ 1464.504888]  [000000000042d340] SyS_64_munmap+0x88/0xa8
[ 1464.504967]  [0000000000406154] linux_sparc_syscall+0x34/0x44

BTW: I'm still trying to get the new firmware for my adapters. Maybe it clicks tomorrow...

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

Lukas Razik <linux-GLgANOly0l1BDLzU/O5InQ@public.gmane.org> wrote
> Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:

> 
>>  If it's possible, it would be useful to try with the vanilla kernel
>>  and all upstream
>>  modules too.  Otherwise I can't rule out the possiblity that we're 
>>  chasing a
>>  bug that OFED introduces.
>
> BTW: I'm still trying to get the new firmware for my adapters. Maybe it 
> clicks tomorrow...
> 

Hello Vladimir and Roland,

with kindly help of the Mellanox support team I updated the FW of my adapters.
It seems that the firmware wasn't the problem. I want to summarize the facts:

I've a
- Sun Enterprise T5120 SPARC Server
- Debian 6.0.3
- linux-2.6.39.4 vanilla and sparc64
- OFED-1.5.4-rc4

If I execute 
# ibv_devinfo 
hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.9.1000
        node_guid:                      0003:ba00:0100:b1f0
        sys_image_guid:                 0003:ba00:0100:b1f3
        vendor_id:                      0x03ba
        vendor_part_id:                 25418
        hw_ver:                         0xA0
        board_id:                       SUN0070000001
        phys_port_cnt:                  2 

I get this BUG message from the kernel:
[ 1032.739077] swap_free: Bad swap file entry 100005e000061800
[ 1032.739206] BUG: Bad page map in process ibv_devinfo  pte:bc0000c300104848 pmd:00fc947c
[ 1032.739324] addr:fffff80100114000 vm_flags:000844fa anon_vma:          (null) mapping:fffff807f5d1c930 index:6180082
[ 1032.739503] vma->vm_file->f_op->mmap: ib_uverbs_mmap+0x8/0x38 [ib_uverbs]
[ 1032.739551] Call Trace:
[ 1032.739589]  [00000000004cd430] unmap_vmas+0x514/0x7f4
[ 1032.739641]  [00000000004d114c] unmap_region+0xb4/0x164
[ 1032.739690]  [00000000004d2198] do_munmap+0x2a8/0x31c
[ 1032.739744]  [000000000042d340] SyS_64_munmap+0x88/0xa8
[ 1032.739800]  [0000000000406154] linux_sparc_syscall+0x34/0x44
[ 1032.739845] Disabling lock debugging due to kernel taint

I also get such BUG messages when I try to use OpenMPI over Infiniband and mpirun dies.
Same with:
- OFED-1.5.3.2 and linux-2.6.39.4 vanilla
- OFED-1.5.3.2 and linux-2.6.38

That's the whole conversation:
http://thread.gmane.org/gmane.linux.drivers.rdma/10157/focus=10210

---

BTW:
I can't build ofa-1.5.4 kernel modules for linux-3.1.1:
http://thread.gmane.org/gmane.linux.drivers.rdma/10192

Regards
Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

Lukas Razik <linux-GLgANOly0l1BDLzU/O5InQ@public.gmane.org> wrote:

> I get this BUG message from the kernel:
> [ 1032.739077] swap_free: Bad swap file entry 100005e000061800
> [ 1032.739206] BUG: Bad page map in process ibv_devinfo  pte:bc0000c300104848 
> pmd:00fc947c
> [ 1032.739324] addr:fffff80100114000 vm_flags:000844fa anon_vma:          (null) 
> mapping:fffff807f5d1c930 index:6180082
> [ 1032.739503] vma->vm_file->f_op->mmap: ib_uverbs_mmap+0x8/0x38 
> [ib_uverbs]
> [ 1032.739551] Call Trace:
> [ 1032.739589]  [00000000004cd430] unmap_vmas+0x514/0x7f4
> [ 1032.739641]  [00000000004d114c] unmap_region+0xb4/0x164
> [ 1032.739690]  [00000000004d2198] do_munmap+0x2a8/0x31c
> [ 1032.739744]  [000000000042d340] SyS_64_munmap+0x88/0xa8
> [ 1032.739800]  [0000000000406154] linux_sparc_syscall+0x34/0x44
> [ 1032.739845] Disabling lock debugging due to kernel taint
> 
> I also get such BUG messages when I try to use OpenMPI over Infiniband and 
> mpirun dies.
> Same with:
> - OFED-1.5.3.2 and linux-2.6.39.4 vanilla
> - OFED-1.5.3.2 and linux-2.6.38
> 

I've tried a rather old kernel, too. I get same BUG messages with:
- OFED-1.5.3.2 and linux-2.6.29 vanilla
- linux-2.6.39.4 vanilla (with the included Infiniband modules - not the modules from OFED)

That's the whole conversation:
http://thread.gmane.org/gmane.linux.drivers.rdma/10157

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

That's a forwarding of Roland's email he sent yesterday to you (the sparclinux list) because you haven't received it.
I thank you for any help!


That's his original email:
Roland Dreier <roland@purestorage.com> wrote:

On Sat, Nov 12, 2011 at 7:15 AM, Lukas Razik <linux@...> wrote:
> But I also get this BUG message from the kernel:
> ---
> [ 9305.698663] swap_free: Bad swap file entry 100005e000061800
> [ 9305.698791] BUG: Bad page map in process ibv_devinfo  pte:bc0000c300104848 pmd:00f38054
> [ 9305.698908] addr:fffff80100114000 vm_flags:000844fa anon_vma:          (null)
mapping:fffff807f313a410 index:6180082
> [ 9305.699087] vma->vm_file->f_op->mmap: ib_uverbs_mmap+0x8/0x38 [ib_uverbs]
> [ 9305.699135] Call Trace:
> [ 9305.699174]  [00000000004cd558] unmap_vmas+0x514/0x7f4
> [ 9305.699302]  [00000000004d1274] unmap_region+0xb4/0x164
> [ 9305.699383]  [00000000004d22c0] do_munmap+0x2a8/0x31c
> [ 9305.699467]  [000000000042d350] SyS_64_munmap+0x88/0xa8
> [ 9305.699550]  [0000000000406154] linux_sparc_syscall+0x34/0x44


Hi Sparc Linux hackers!

This message is coming from someone using a ConnectX (mlx4) adapter on
a Sun T5120 (SPARC64).

I'm wondering if the bug is in the mlx4 code or in the sparc
io_remap_pfn_range()
code.  From the "Bad page map" pte value, if I understand sparc mm correctly,
the PTE is seen as not present.

The mapping is coming from the code in drivers/infiniband/hw/mlx4/main.c:

static int mlx4_ib_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
{
  struct mlx4_ib_dev *dev = to_mdev(context->device);

  if (vma->vm_end - vma->vm_start != PAGE_SIZE)
    return -EINVAL;

  if (vma->vm_pgoff == 0) {
    vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);

    if (io_remap_pfn_range(vma, vma->vm_start,
               to_mucontext(context)->uar.pfn,
               PAGE_SIZE, vma->vm_page_prot))
      return -EAGAIN;
  } else if (vma->vm_pgoff == 1 && dev->dev->caps.bf_reg_size != 0) {
    vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);

    if (io_remap_pfn_range(vma, vma->vm_start,
               to_mucontext(context)->uar.pfn +
               dev->dev->caps.num_uars,
               PAGE_SIZE, vma->vm_page_prot))
      return -EAGAIN;
  } else
    return -EINVAL;

  return 0;
}

which is just taking one page -- the pfn comes from the PCI BAR
resource value in

int mlx4_uar_alloc(struct mlx4_dev *dev, struct mlx4_uar *uar)
{
  uar->index = mlx4_bitmap_alloc(&mlx4_priv(dev)->uar_table.bitmap);
  if (uar->index == -1)
    return -ENOMEM;

  uar->pfn = (pci_resource_start(dev->pdev, 2) >> PAGE_SHIFT) + uar->index;
  uar->map = NULL;

  return 0;
}

and calling io_remap_pfn_range on it.

Is there some VMA flag we're forgetting to set that only affects sparc?
Or could this possible be a bug in the sparc handling of PFN maps?

Thanks!
  Roland
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[David Miller]

From: Lukas Razik <linux-GLgANOly0l1BDLzU/O5InQ@public.gmane.org>
Date: Wed, 16 Nov 2011 19:22:34 +0000 (GMT)

> I'm wondering if the bug is in the mlx4 code or in the sparc
> io_remap_pfn_range()
> code.  From the "Bad page map" pte value, if I understand sparc mm correctly,
> the PTE is seen as not present.

The not-present test is essentially testing the PTE against zero.

This bug check is making sure that there are not already existing mappings
at that address when the io_remap_pfn_range() call occurs.

x86 uses remap_pfn_range() as the implementation of it's
io_remap_pfn_range() and it also has the same exact assertion
check, in remap_pte_range().

I can't see any obvious bug in the mlx4 mmap method.

The only thing I can come up with is that perhaps there is an assumption
hiding around somewhere that PAGE_SIZE is 4096.  Such an assumption would
go undetected on both 32-bit and 64-bit x86 platforms since PAGE_SIZE is
4096 there.  Whereas on sparc64 it is 8192.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Roland Dreier]

On Thu, Nov 17, 2011 at 1:49 PM, David Miller <davem@davemloft.net> wrote:
>> I'm wondering if the bug is in the mlx4 code or in the sparc
>> io_remap_pfn_range()
>> code.  From the "Bad page map" pte value, if I understand sparc mm correctly,
>> the PTE is seen as not present.

> The not-present test is essentially testing the PTE against zero.

Isn't the pte_present assembly checking if the _PAGE_PRESENT_4V bit
is set, not whether it's completely 0?

> This bug check is making sure that there are not already existing mappings
> at that address when the io_remap_pfn_range() call occurs.

But the call trace is in the munmap() call -- it's just printing where the
bad map was set up initially.  We're somewhere in unmap_vmas(),
probably one of the tests in zap_pte_range().  Since we got

    [ 9305.698663] swap_free: Bad swap file entry 100005e000061800

I'm guessing somehow we reached

		} else {
			swp_entry_t entry = pte_to_swp_entry(ptent);

			if (!non_swap_entry(entry))
				rss[MM_SWAPENTS]--;
			if (unlikely(!free_swap_and_cache(entry)))
				print_bad_pte(vma, addr, ptent, NULL);
		}

which shouldn't be possible if pte_present() were set.

 > The only thing I can come up with is that perhaps there is an assumption
 > hiding around somewhere that PAGE_SIZE is 4096.

It's a good idea.  But I don't see how we get past:

static int mlx4_ib_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
{
	struct mlx4_ib_dev *dev = to_mdev(context->device);

	if (vma->vm_end - vma->vm_start != PAGE_SIZE)
		return -EINVAL;

 - R.
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[David Miller]

From: Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org>
Date: Thu, 17 Nov 2011 15:30:19 -0800

> But the call trace is in the munmap() call -- it's just printing where the
> bad map was set up initially.  We're somewhere in unmap_vmas(),
> probably one of the tests in zap_pte_range().  Since we got
> 
>     [ 9305.698663] swap_free: Bad swap file entry 100005e000061800
> 
> I'm guessing somehow we reached
> 
> 		} else {
> 			swp_entry_t entry = pte_to_swp_entry(ptent);
> 
> 			if (!non_swap_entry(entry))
> 				rss[MM_SWAPENTS]--;
> 			if (unlikely(!free_swap_and_cache(entry)))
> 				print_bad_pte(vma, addr, ptent, NULL);
> 		}
> 
> which shouldn't be possible if pte_present() were set.

I suspect this is an issue about who is responsible for setting the
various VM_* flags in the VMA.

It at least used to be the case that the caller was responsible, but
it seems that this has changed in some regard.  And if you look at
remap_pfn_range(), which is what x86 uses, it sets things like VM_IO
explicitly.

In fact, sparc is one of the few platforms not using remap_pfn_range()
as it's io_remap_pfn_range() implementation.

Sparc overrides the implementation for two reasons:

1) To deal with encoding of upper bits into the PFN properly on 32-bit
   machines.  This is similar to what MIPS is dealing with in it's
   wrapper around remap_pfn_range().

2) To use large page mappings when possible on 64-bit.  This used to
   matter a lot with framebuffers about 10 years ago.

I'd rather give up an optimization of dubious value than to have to
constantly stay on top of every single change ever made to
remap_pfn_range().

Therefore I'll code up a patch that handles the PFN encoding issue
the way that MIPS does, with a small wrapper around remap_pfn_range().

Patch coming up for testing in a bit.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Roland Dreier]

On Thu, Nov 17, 2011 at 6:03 PM, David Miller <davem@davemloft.net> wrote:
> I suspect this is an issue about who is responsible for setting the
> various VM_* flags in the VMA.
>
> It at least used to be the case that the caller was responsible, but
> it seems that this has changed in some regard.  And if you look at
> remap_pfn_range(), which is what x86 uses, it sets things like VM_IO
> explicitly.
>
> In fact, sparc is one of the few platforms not using remap_pfn_range()
> as it's io_remap_pfn_range() implementation.

Somewhat makes sense but OTOH I don't see the difference between
arch/sparc/mm/generic_64.c:

int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long from,
                unsigned long pfn, unsigned long size, pgprot_t prot)
{
...
        vma->vm_flags |= VM_IO | VM_RESERVED | VM_PFNMAP;

and mm/memory.c:

int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr,
                    unsigned long pfn, unsigned long size, pgprot_t prot)
{
...
        if (addr == vma->vm_start && end == vma->vm_end) {
                vma->vm_pgoff = pfn;
                vma->vm_flags |= VM_PFN_AT_MMAP;
        } else if (is_cow_mapping(vma->vm_flags))
                return -EINVAL;

        vma->vm_flags |= VM_IO | VM_RESERVED | VM_PFNMAP;

The only thing sparc seems to be missing is the VM_PFN_AT_MMAP
but that only gets tested in the x86 PAT code and in the transparent
hugepage code (which I suspect doesn't get used on sparc either).

Anyway definitely makes sense as something to check.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[David Miller]

From: David Miller <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
Date: Thu, 17 Nov 2011 21:03:16 -0500 (EST)

> Patch coming up for testing in a bit.

Can someone with the effected hardware please test this?

--------------------
[PATCH] sparc: Kill custom io_remap_pfn_range().

To handle the large physical addresses, just make a simple wrapper
around remap_pfn_range() like MIPS does.

Signed-off-by: David S. Miller <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
---
 arch/sparc/include/asm/pgtable_32.h |   20 ++++-
 arch/sparc/include/asm/pgtable_64.h |   20 ++++-
 arch/sparc/mm/Makefile              |    1 -
 arch/sparc/mm/generic_32.c          |   99 ---------------------
 arch/sparc/mm/generic_64.c          |  165 -----------------------------------
 5 files changed, 32 insertions(+), 273 deletions(-)
 delete mode 100644 arch/sparc/mm/generic_32.c
 delete mode 100644 arch/sparc/mm/generic_64.c

diff --git a/arch/sparc/include/asm/pgtable_32.h b/arch/sparc/include/asm/pgtable_32.h
index 5b31a8e..a790cc6 100644
--- a/arch/sparc/include/asm/pgtable_32.h
+++ b/arch/sparc/include/asm/pgtable_32.h
@@ -431,10 +431,6 @@ extern unsigned long *sparc_valid_addr_bitmap;
 #define kern_addr_valid(addr) \
 	(test_bit(__pa((unsigned long)(addr))>>20, sparc_valid_addr_bitmap))
 
-extern int io_remap_pfn_range(struct vm_area_struct *vma,
-			      unsigned long from, unsigned long pfn,
-			      unsigned long size, pgprot_t prot);
-
 /*
  * For sparc32&64, the pfn in io_remap_pfn_range() carries <iospace> in
  * its high 4 bits.  These macros/functions put it there or get it from there.
@@ -443,6 +439,22 @@ extern int io_remap_pfn_range(struct vm_area_struct *vma,
 #define GET_IOSPACE(pfn)		(pfn >> (BITS_PER_LONG - 4))
 #define GET_PFN(pfn)			(pfn & 0x0fffffffUL)
 
+extern int remap_pfn_range(struct vm_area_struct *, unsigned long, unsigned long,
+			   unsigned long, pgprot_t);
+
+static inline int io_remap_pfn_range(struct vm_area_struct *vma,
+				     unsigned long from, unsigned long pfn,
+				     unsigned long size, pgprot_t prot)
+{
+	unsigned long long offset, space, phys_base;
+
+	offset = ((unsigned long long) GET_PFN(pfn)) << PAGE_SHIFT;
+	space = GET_IOSPACE(pfn);
+	phys_base = offset | (space << 32ULL);
+
+	return remap_pfn_range(vma, from, phys_base >> PAGE_SHIFT, size, prot);
+}
+
 #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS
 #define ptep_set_access_flags(__vma, __address, __ptep, __entry, __dirty) \
 ({									  \
diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index adf8932..38ebb2c 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -757,10 +757,6 @@ static inline bool kern_addr_valid(unsigned long addr)
 
 extern int page_in_phys_avail(unsigned long paddr);
 
-extern int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long from,
-			       unsigned long pfn,
-			       unsigned long size, pgprot_t prot);
-
 /*
  * For sparc32&64, the pfn in io_remap_pfn_range() carries <iospace> in
  * its high 4 bits.  These macros/functions put it there or get it from there.
@@ -769,6 +765,22 @@ extern int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long from,
 #define GET_IOSPACE(pfn)		(pfn >> (BITS_PER_LONG - 4))
 #define GET_PFN(pfn)			(pfn & 0x0fffffffffffffffUL)
 
+extern int remap_pfn_range(struct vm_area_struct *, unsigned long, unsigned long,
+			   unsigned long, pgprot_t);
+
+static inline int io_remap_pfn_range(struct vm_area_struct *vma,
+				     unsigned long from, unsigned long pfn,
+				     unsigned long size, pgprot_t prot)
+{
+	unsigned long offset = GET_PFN(pfn) << PAGE_SHIFT;
+	int space = GET_IOSPACE(pfn);
+	unsigned long phys_base;
+
+	phys_base = offset | (((unsigned long) space) << 32UL);
+
+	return remap_pfn_range(vma, from, phys_base >> PAGE_SHIFT, size, prot);
+}
+
 #include <asm-generic/pgtable.h>
 
 /* We provide our own get_unmapped_area to cope with VA holes and
diff --git a/arch/sparc/mm/Makefile b/arch/sparc/mm/Makefile
index e3cda21..301421c 100644
--- a/arch/sparc/mm/Makefile
+++ b/arch/sparc/mm/Makefile
@@ -8,7 +8,6 @@ obj-$(CONFIG_SPARC64)   += ultra.o tlb.o tsb.o gup.o
 obj-y                   += fault_$(BITS).o
 obj-y                   += init_$(BITS).o
 obj-$(CONFIG_SPARC32)   += loadmmu.o
-obj-y                   += generic_$(BITS).o
 obj-$(CONFIG_SPARC32)   += extable.o btfixup.o srmmu.o iommu.o io-unit.o
 obj-$(CONFIG_SPARC32)   += hypersparc.o viking.o tsunami.o swift.o
 obj-$(CONFIG_SPARC_LEON)+= leon_mm.o
diff --git a/arch/sparc/mm/generic_32.c b/arch/sparc/mm/generic_32.c
deleted file mode 100644
index 6ca39a6..0000000
--- a/arch/sparc/mm/generic_32.c
+++ /dev/null
@@ -1,99 +0,0 @@
-/*
- * generic.c: Generic Sparc mm routines that are not dependent upon
- *            MMU type but are Sparc specific.
- *
- * Copyright (C) 1996 David S. Miller (davem-CVlFQpJstH5+CIkdHa1UOg@public.gmane.org)
- */
-
-#include <linux/kernel.h>
-#include <linux/mm.h>
-#include <linux/swap.h>
-#include <linux/pagemap.h>
-#include <linux/export.h>
-
-#include <asm/pgalloc.h>
-#include <asm/pgtable.h>
-#include <asm/page.h>
-#include <asm/cacheflush.h>
-#include <asm/tlbflush.h>
-
-/* Remap IO memory, the same way as remap_pfn_range(), but use
- * the obio memory space.
- *
- * They use a pgprot that sets PAGE_IO and does not check the
- * mem_map table as this is independent of normal memory.
- */
-static inline void io_remap_pte_range(struct mm_struct *mm, pte_t * pte, unsigned long address, unsigned long size,
-	unsigned long offset, pgprot_t prot, int space)
-{
-	unsigned long end;
-
-	address &= ~PMD_MASK;
-	end = address + size;
-	if (end > PMD_SIZE)
-		end = PMD_SIZE;
-	do {
-		set_pte_at(mm, address, pte, mk_pte_io(offset, prot, space));
-		address += PAGE_SIZE;
-		offset += PAGE_SIZE;
-		pte++;
-	} while (address < end);
-}
-
-static inline int io_remap_pmd_range(struct mm_struct *mm, pmd_t * pmd, unsigned long address, unsigned long size,
-	unsigned long offset, pgprot_t prot, int space)
-{
-	unsigned long end;
-
-	address &= ~PGDIR_MASK;
-	end = address + size;
-	if (end > PGDIR_SIZE)
-		end = PGDIR_SIZE;
-	offset -= address;
-	do {
-		pte_t *pte = pte_alloc_map(mm, NULL, pmd, address);
-		if (!pte)
-			return -ENOMEM;
-		io_remap_pte_range(mm, pte, address, end - address, address + offset, prot, space);
-		address = (address + PMD_SIZE) & PMD_MASK;
-		pmd++;
-	} while (address < end);
-	return 0;
-}
-
-int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long from,
-		       unsigned long pfn, unsigned long size, pgprot_t prot)
-{
-	int error = 0;
-	pgd_t * dir;
-	unsigned long beg = from;
-	unsigned long end = from + size;
-	struct mm_struct *mm = vma->vm_mm;
-	int space = GET_IOSPACE(pfn);
-	unsigned long offset = GET_PFN(pfn) << PAGE_SHIFT;
-
-	/* See comment in mm/memory.c remap_pfn_range */
-	vma->vm_flags |= VM_IO | VM_RESERVED | VM_PFNMAP;
-	vma->vm_pgoff = (offset >> PAGE_SHIFT) |
-		((unsigned long)space << 28UL);
-
-	offset -= from;
-	dir = pgd_offset(mm, from);
-	flush_cache_range(vma, beg, end);
-
-	while (from < end) {
-		pmd_t *pmd = pmd_alloc(mm, dir, from);
-		error = -ENOMEM;
-		if (!pmd)
-			break;
-		error = io_remap_pmd_range(mm, pmd, from, end - from, offset + from, prot, space);
-		if (error)
-			break;
-		from = (from + PGDIR_SIZE) & PGDIR_MASK;
-		dir++;
-	}
-
-	flush_tlb_range(vma, beg, end);
-	return error;
-}
-EXPORT_SYMBOL(io_remap_pfn_range);
diff --git a/arch/sparc/mm/generic_64.c b/arch/sparc/mm/generic_64.c
deleted file mode 100644
index 9b357dd..0000000
--- a/arch/sparc/mm/generic_64.c
+++ /dev/null
@@ -1,165 +0,0 @@
-/*
- * generic.c: Generic Sparc mm routines that are not dependent upon
- *            MMU type but are Sparc specific.
- *
- * Copyright (C) 1996 David S. Miller (davem-CVlFQpJstH5+CIkdHa1UOg@public.gmane.org)
- */
-
-#include <linux/kernel.h>
-#include <linux/mm.h>
-#include <linux/swap.h>
-#include <linux/export.h>
-#include <linux/pagemap.h>
-
-#include <asm/pgalloc.h>
-#include <asm/pgtable.h>
-#include <asm/page.h>
-#include <asm/tlbflush.h>
-
-/* Remap IO memory, the same way as remap_pfn_range(), but use
- * the obio memory space.
- *
- * They use a pgprot that sets PAGE_IO and does not check the
- * mem_map table as this is independent of normal memory.
- */
-static inline void io_remap_pte_range(struct mm_struct *mm, pte_t * pte,
-				      unsigned long address,
-				      unsigned long size,
-				      unsigned long offset, pgprot_t prot,
-				      int space)
-{
-	unsigned long end;
-
-	/* clear hack bit that was used as a write_combine side-effect flag */
-	offset &= ~0x1UL;
-	address &= ~PMD_MASK;
-	end = address + size;
-	if (end > PMD_SIZE)
-		end = PMD_SIZE;
-	do {
-		pte_t entry;
-		unsigned long curend = address + PAGE_SIZE;
-		
-		entry = mk_pte_io(offset, prot, space, PAGE_SIZE);
-		if (!(address & 0xffff)) {
-			if (PAGE_SIZE < (4 * 1024 * 1024) &&
-			    !(address & 0x3fffff) &&
-			    !(offset & 0x3ffffe) &&
-			    end >= address + 0x400000) {
-				entry = mk_pte_io(offset, prot, space,
-						  4 * 1024 * 1024);
-				curend = address + 0x400000;
-				offset += 0x400000;
-			} else if (PAGE_SIZE < (512 * 1024) &&
-				   !(address & 0x7ffff) &&
-				   !(offset & 0x7fffe) &&
-				   end >= address + 0x80000) {
-				entry = mk_pte_io(offset, prot, space,
-						  512 * 1024 * 1024);
-				curend = address + 0x80000;
-				offset += 0x80000;
-			} else if (PAGE_SIZE < (64 * 1024) &&
-				   !(offset & 0xfffe) &&
-				   end >= address + 0x10000) {
-				entry = mk_pte_io(offset, prot, space,
-						  64 * 1024);
-				curend = address + 0x10000;
-				offset += 0x10000;
-			} else
-				offset += PAGE_SIZE;
-		} else
-			offset += PAGE_SIZE;
-
-		if (pte_write(entry))
-			entry = pte_mkdirty(entry);
-		do {
-			BUG_ON(!pte_none(*pte));
-			set_pte_at(mm, address, pte, entry);
-			address += PAGE_SIZE;
-			pte_val(entry) += PAGE_SIZE;
-			pte++;
-		} while (address < curend);
-	} while (address < end);
-}
-
-static inline int io_remap_pmd_range(struct mm_struct *mm, pmd_t * pmd, unsigned long address, unsigned long size,
-	unsigned long offset, pgprot_t prot, int space)
-{
-	unsigned long end;
-
-	address &= ~PGDIR_MASK;
-	end = address + size;
-	if (end > PGDIR_SIZE)
-		end = PGDIR_SIZE;
-	offset -= address;
-	do {
-		pte_t *pte = pte_alloc_map(mm, NULL, pmd, address);
-		if (!pte)
-			return -ENOMEM;
-		io_remap_pte_range(mm, pte, address, end - address, address + offset, prot, space);
-		pte_unmap(pte);
-		address = (address + PMD_SIZE) & PMD_MASK;
-		pmd++;
-	} while (address < end);
-	return 0;
-}
-
-static inline int io_remap_pud_range(struct mm_struct *mm, pud_t * pud, unsigned long address, unsigned long size,
-	unsigned long offset, pgprot_t prot, int space)
-{
-	unsigned long end;
-
-	address &= ~PUD_MASK;
-	end = address + size;
-	if (end > PUD_SIZE)
-		end = PUD_SIZE;
-	offset -= address;
-	do {
-		pmd_t *pmd = pmd_alloc(mm, pud, address);
-		if (!pud)
-			return -ENOMEM;
-		io_remap_pmd_range(mm, pmd, address, end - address, address + offset, prot, space);
-		address = (address + PUD_SIZE) & PUD_MASK;
-		pud++;
-	} while (address < end);
-	return 0;
-}
-
-int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long from,
-		unsigned long pfn, unsigned long size, pgprot_t prot)
-{
-	int error = 0;
-	pgd_t * dir;
-	unsigned long beg = from;
-	unsigned long end = from + size;
-	struct mm_struct *mm = vma->vm_mm;
-	int space = GET_IOSPACE(pfn);
-	unsigned long offset = GET_PFN(pfn) << PAGE_SHIFT;
-	unsigned long phys_base;
-
-	phys_base = offset | (((unsigned long) space) << 32UL);
-
-	/* See comment in mm/memory.c remap_pfn_range */
-	vma->vm_flags |= VM_IO | VM_RESERVED | VM_PFNMAP;
-	vma->vm_pgoff = phys_base >> PAGE_SHIFT;
-
-	offset -= from;
-	dir = pgd_offset(mm, from);
-	flush_cache_range(vma, beg, end);
-
-	while (from < end) {
-		pud_t *pud = pud_alloc(mm, dir, from);
-		error = -ENOMEM;
-		if (!pud)
-			break;
-		error = io_remap_pud_range(mm, pud, from, end - from, offset + from, prot, space);
-		if (error)
-			break;
-		from = (from + PGDIR_SIZE) & PGDIR_MASK;
-		dir++;
-	}
-
-	flush_tlb_range(vma, beg, end);
-	return error;
-}
-EXPORT_SYMBOL(io_remap_pfn_range);
-- 
1.7.6.401.g6a319

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[David Miller]

From: Roland Dreier <roland@purestorage.com>
Date: Thu, 17 Nov 2011 18:17:08 -0800

> Anyway definitely makes sense as something to check.

Another difference is remap_pfn_range() uses pte_mkspecial().

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

David Miller <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org> wrote:

> Can someone with the effected hardware please test this?
> 
> --------------------
> [PATCH] sparc: Kill custom io_remap_pfn_range().
> 
> To handle the large physical addresses, just make a simple wrapper
> around remap_pfn_range() like MIPS does.
> 
> Signed-off-by: David S. Miller <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
> ---
> arch/sparc/include/asm/pgtable_32.h |   20 ++++-
> arch/sparc/include/asm/pgtable_64.h |   20 ++++-
> arch/sparc/mm/Makefile              |    1 -
> arch/sparc/mm/generic_32.c          |   99 ---------------------
> arch/sparc/mm/generic_64.c          |  165 -----------------------------------
> 5 files changed, 32 insertions(+), 273 deletions(-)
> delete mode 100644 arch/sparc/mm/generic_32.c
> delete mode 100644 arch/sparc/mm/generic_64.c
> 
> diff --git a/arch/sparc/include/asm/pgtable_32.h 
> b/arch/sparc/include/asm/pgtable_32.h
> index 5b31a8e..a790cc6 100644
> --- a/arch/sparc/include/asm/pgtable_32.h
> +++ b/arch/sparc/include/asm/pgtable_32.h
> @@ -431,10 +431,6 @@ extern unsigned long *sparc_valid_addr_bitmap;
> #define kern_addr_valid(addr) \
>     (test_bit(__pa((unsigned long)(addr))>>20, sparc_valid_addr_bitmap))
> 
> -extern int io_remap_pfn_range(struct vm_area_struct *vma,
> -                  unsigned long from, unsigned long pfn,
> -                  unsigned long size, pgprot_t prot);
> -
> /*
>   * For sparc32&64, the pfn in io_remap_pfn_range() carries <iospace> 
> in
>   * its high 4 bits.  These macros/functions put it there or get it from there.
> @@ -443,6 +439,22 @@ extern int io_remap_pfn_range(struct vm_area_struct *vma,
> #define GET_IOSPACE(pfn)        (pfn >> (BITS_PER_LONG - 4))
> #define GET_PFN(pfn)            (pfn & 0x0fffffffUL)
> 
> +extern int remap_pfn_range(struct vm_area_struct *, unsigned long, unsigned 
> long,
> +               unsigned long, pgprot_t);
> +
> +static inline int io_remap_pfn_range(struct vm_area_struct *vma,
> +                     unsigned long from, unsigned long pfn,
> +                     unsigned long size, pgprot_t prot)
> +{
> +    unsigned long long offset, space, phys_base;
> +
> +    offset = ((unsigned long long) GET_PFN(pfn)) << PAGE_SHIFT;
> +    space = GET_IOSPACE(pfn);
> +    phys_base = offset | (space << 32ULL);
> +
> +    return remap_pfn_range(vma, from, phys_base >> PAGE_SHIFT, size, 
> prot);
> +}
> +
> #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS
> #define ptep_set_access_flags(__vma, __address, __ptep, __entry, __dirty) \
> ({                                      \
> diff --git a/arch/sparc/include/asm/pgtable_64.h 
> b/arch/sparc/include/asm/pgtable_64.h
> index adf8932..38ebb2c 100644
> --- a/arch/sparc/include/asm/pgtable_64.h
> +++ b/arch/sparc/include/asm/pgtable_64.h
> @@ -757,10 +757,6 @@ static inline bool kern_addr_valid(unsigned long addr)
> 
> extern int page_in_phys_avail(unsigned long paddr);
> 
> -extern int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long from,
> -                   unsigned long pfn,
> -                   unsigned long size, pgprot_t prot);
> -
> /*
>   * For sparc32&64, the pfn in io_remap_pfn_range() carries <iospace> 
> in
>   * its high 4 bits.  These macros/functions put it there or get it from there.
> @@ -769,6 +765,22 @@ extern int io_remap_pfn_range(struct vm_area_struct *vma, 
> unsigned long from,
> #define GET_IOSPACE(pfn)        (pfn >> (BITS_PER_LONG - 4))
> #define GET_PFN(pfn)            (pfn & 0x0fffffffffffffffUL)
> 
> +extern int remap_pfn_range(struct vm_area_struct *, unsigned long, unsigned 
> long,
> +               unsigned long, pgprot_t);
> +
> +static inline int io_remap_pfn_range(struct vm_area_struct *vma,
> +                     unsigned long from, unsigned long pfn,
> +                     unsigned long size, pgprot_t prot)
> +{
> +    unsigned long offset = GET_PFN(pfn) << PAGE_SHIFT;
> +    int space = GET_IOSPACE(pfn);
> +    unsigned long phys_base;
> +
> +    phys_base = offset | (((unsigned long) space) << 32UL);
> +
> +    return remap_pfn_range(vma, from, phys_base >> PAGE_SHIFT, size, 
> prot);
> +}
> +
> #include <asm-generic/pgtable.h>
> 
> /* We provide our own get_unmapped_area to cope with VA holes and
> diff --git a/arch/sparc/mm/Makefile b/arch/sparc/mm/Makefile
> index e3cda21..301421c 100644
> --- a/arch/sparc/mm/Makefile
> +++ b/arch/sparc/mm/Makefile
> @@ -8,7 +8,6 @@ obj-$(CONFIG_SPARC64)   += ultra.o tlb.o tsb.o gup.o
> obj-y                   += fault_$(BITS).o
> obj-y                   += init_$(BITS).o
> obj-$(CONFIG_SPARC32)   += loadmmu.o
> -obj-y                   += generic_$(BITS).o
> obj-$(CONFIG_SPARC32)   += extable.o btfixup.o srmmu.o iommu.o io-unit.o
> obj-$(CONFIG_SPARC32)   += hypersparc.o viking.o tsunami.o swift.o
> obj-$(CONFIG_SPARC_LEON)+= leon_mm.o
> diff --git a/arch/sparc/mm/generic_32.c b/arch/sparc/mm/generic_32.c
> deleted file mode 100644
> index 6ca39a6..0000000
> --- a/arch/sparc/mm/generic_32.c
> +++ /dev/null
> @@ -1,99 +0,0 @@
> -/*
> - * generic.c: Generic Sparc mm routines that are not dependent upon
> - *            MMU type but are Sparc specific.
> - *
> - * Copyright (C) 1996 David S. Miller (davem-CVlFQpJstH5+CIkdHa1UOg@public.gmane.org)
> - */
> -
> -#include <linux/kernel.h>
> -#include <linux/mm.h>
> -#include <linux/swap.h>
> -#include <linux/pagemap.h>
> -#include <linux/export.h>
> -
> -#include <asm/pgalloc.h>
> -#include <asm/pgtable.h>
> -#include <asm/page.h>
> -#include <asm/cacheflush.h>
> -#include <asm/tlbflush.h>
> -
> -/* Remap IO memory, the same way as remap_pfn_range(), but use
> - * the obio memory space.
> - *
> - * They use a pgprot that sets PAGE_IO and does not check the
> - * mem_map table as this is independent of normal memory.
> - */
> -static inline void io_remap_pte_range(struct mm_struct *mm, pte_t * pte, 
> unsigned long address, unsigned long size,
> -    unsigned long offset, pgprot_t prot, int space)
> -{
> -    unsigned long end;
> -
> -    address &= ~PMD_MASK;
> -    end = address + size;
> -    if (end > PMD_SIZE)
> -        end = PMD_SIZE;
> -    do {
> -        set_pte_at(mm, address, pte, mk_pte_io(offset, prot, space));
> -        address += PAGE_SIZE;
> -        offset += PAGE_SIZE;
> -        pte++;
> -    } while (address < end);
> -}
> -
> -static inline int io_remap_pmd_range(struct mm_struct *mm, pmd_t * pmd, 
> unsigned long address, unsigned long size,
> -    unsigned long offset, pgprot_t prot, int space)
> -{
> -    unsigned long end;
> -
> -    address &= ~PGDIR_MASK;
> -    end = address + size;
> -    if (end > PGDIR_SIZE)
> -        end = PGDIR_SIZE;
> -    offset -= address;
> -    do {
> -        pte_t *pte = pte_alloc_map(mm, NULL, pmd, address);
> -        if (!pte)
> -            return -ENOMEM;
> -        io_remap_pte_range(mm, pte, address, end - address, address + offset, 
> prot, space);
> -        address = (address + PMD_SIZE) & PMD_MASK;
> -        pmd++;
> -    } while (address < end);
> -    return 0;
> -}
> -
> -int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long from,
> -               unsigned long pfn, unsigned long size, pgprot_t prot)
> -{
> -    int error = 0;
> -    pgd_t * dir;
> -    unsigned long beg = from;
> -    unsigned long end = from + size;
> -    struct mm_struct *mm = vma->vm_mm;
> -    int space = GET_IOSPACE(pfn);
> -    unsigned long offset = GET_PFN(pfn) << PAGE_SHIFT;
> -
> -    /* See comment in mm/memory.c remap_pfn_range */
> -    vma->vm_flags |= VM_IO | VM_RESERVED | VM_PFNMAP;
> -    vma->vm_pgoff = (offset >> PAGE_SHIFT) |
> -        ((unsigned long)space << 28UL);
> -
> -    offset -= from;
> -    dir = pgd_offset(mm, from);
> -    flush_cache_range(vma, beg, end);
> -
> -    while (from < end) {
> -        pmd_t *pmd = pmd_alloc(mm, dir, from);
> -        error = -ENOMEM;
> -        if (!pmd)
> -            break;
> -        error = io_remap_pmd_range(mm, pmd, from, end - from, offset + from, 
> prot, space);
> -        if (error)
> -            break;
> -        from = (from + PGDIR_SIZE) & PGDIR_MASK;
> -        dir++;
> -    }
> -
> -    flush_tlb_range(vma, beg, end);
> -    return error;
> -}
> -EXPORT_SYMBOL(io_remap_pfn_range);
> diff --git a/arch/sparc/mm/generic_64.c b/arch/sparc/mm/generic_64.c
> deleted file mode 100644
> index 9b357dd..0000000
> --- a/arch/sparc/mm/generic_64.c
> +++ /dev/null
> @@ -1,165 +0,0 @@
> -/*
> - * generic.c: Generic Sparc mm routines that are not dependent upon
> - *            MMU type but are Sparc specific.
> - *
> - * Copyright (C) 1996 David S. Miller (davem-CVlFQpJstH5+CIkdHa1UOg@public.gmane.org)
> - */
> -
> -#include <linux/kernel.h>
> -#include <linux/mm.h>
> -#include <linux/swap.h>
> -#include <linux/export.h>
> -#include <linux/pagemap.h>
> -
> -#include <asm/pgalloc.h>
> -#include <asm/pgtable.h>
> -#include <asm/page.h>
> -#include <asm/tlbflush.h>
> -
> -/* Remap IO memory, the same way as remap_pfn_range(), but use
> - * the obio memory space.
> - *
> - * They use a pgprot that sets PAGE_IO and does not check the
> - * mem_map table as this is independent of normal memory.
> - */
> -static inline void io_remap_pte_range(struct mm_struct *mm, pte_t * pte,
> -                      unsigned long address,
> -                      unsigned long size,
> -                      unsigned long offset, pgprot_t prot,
> -                      int space)
> -{
> -    unsigned long end;
> -
> -    /* clear hack bit that was used as a write_combine side-effect flag */
> -    offset &= ~0x1UL;
> -    address &= ~PMD_MASK;
> -    end = address + size;
> -    if (end > PMD_SIZE)
> -        end = PMD_SIZE;
> -    do {
> -        pte_t entry;
> -        unsigned long curend = address + PAGE_SIZE;
> -        
> -        entry = mk_pte_io(offset, prot, space, PAGE_SIZE);
> -        if (!(address & 0xffff)) {
> -            if (PAGE_SIZE < (4 * 1024 * 1024) &&
> -                !(address & 0x3fffff) &&
> -                !(offset & 0x3ffffe) &&
> -                end >= address + 0x400000) {
> -                entry = mk_pte_io(offset, prot, space,
> -                          4 * 1024 * 1024);
> -                curend = address + 0x400000;
> -                offset += 0x400000;
> -            } else if (PAGE_SIZE < (512 * 1024) &&
> -                   !(address & 0x7ffff) &&
> -                   !(offset & 0x7fffe) &&
> -                   end >= address + 0x80000) {
> -                entry = mk_pte_io(offset, prot, space,
> -                          512 * 1024 * 1024);
> -                curend = address + 0x80000;
> -                offset += 0x80000;
> -            } else if (PAGE_SIZE < (64 * 1024) &&
> -                   !(offset & 0xfffe) &&
> -                   end >= address + 0x10000) {
> -                entry = mk_pte_io(offset, prot, space,
> -                          64 * 1024);
> -                curend = address + 0x10000;
> -                offset += 0x10000;
> -            } else
> -                offset += PAGE_SIZE;
> -        } else
> -            offset += PAGE_SIZE;
> -
> -        if (pte_write(entry))
> -            entry = pte_mkdirty(entry);
> -        do {
> -            BUG_ON(!pte_none(*pte));
> -            set_pte_at(mm, address, pte, entry);
> -            address += PAGE_SIZE;
> -            pte_val(entry) += PAGE_SIZE;
> -            pte++;
> -        } while (address < curend);
> -    } while (address < end);
> -}
> -
> -static inline int io_remap_pmd_range(struct mm_struct *mm, pmd_t * pmd, 
> unsigned long address, unsigned long size,
> -    unsigned long offset, pgprot_t prot, int space)
> -{
> -    unsigned long end;
> -
> -    address &= ~PGDIR_MASK;
> -    end = address + size;
> -    if (end > PGDIR_SIZE)
> -        end = PGDIR_SIZE;
> -    offset -= address;
> -    do {
> -        pte_t *pte = pte_alloc_map(mm, NULL, pmd, address);
> -        if (!pte)
> -            return -ENOMEM;
> -        io_remap_pte_range(mm, pte, address, end - address, address + offset, 
> prot, space);
> -        pte_unmap(pte);
> -        address = (address + PMD_SIZE) & PMD_MASK;
> -        pmd++;
> -    } while (address < end);
> -    return 0;
> -}
> -
> -static inline int io_remap_pud_range(struct mm_struct *mm, pud_t * pud, 
> unsigned long address, unsigned long size,
> -    unsigned long offset, pgprot_t prot, int space)
> -{
> -    unsigned long end;
> -
> -    address &= ~PUD_MASK;
> -    end = address + size;
> -    if (end > PUD_SIZE)
> -        end = PUD_SIZE;
> -    offset -= address;
> -    do {
> -        pmd_t *pmd = pmd_alloc(mm, pud, address);
> -        if (!pud)
> -            return -ENOMEM;
> -        io_remap_pmd_range(mm, pmd, address, end - address, address + offset, 
> prot, space);
> -        address = (address + PUD_SIZE) & PUD_MASK;
> -        pud++;
> -    } while (address < end);
> -    return 0;
> -}
> -
> -int io_remap_pfn_range(struct vm_area_struct *vma, unsigned long from,
> -        unsigned long pfn, unsigned long size, pgprot_t prot)
> -{
> -    int error = 0;
> -    pgd_t * dir;
> -    unsigned long beg = from;
> -    unsigned long end = from + size;
> -    struct mm_struct *mm = vma->vm_mm;
> -    int space = GET_IOSPACE(pfn);
> -    unsigned long offset = GET_PFN(pfn) << PAGE_SHIFT;
> -    unsigned long phys_base;
> -
> -    phys_base = offset | (((unsigned long) space) << 32UL);
> -
> -    /* See comment in mm/memory.c remap_pfn_range */
> -    vma->vm_flags |= VM_IO | VM_RESERVED | VM_PFNMAP;
> -    vma->vm_pgoff = phys_base >> PAGE_SHIFT;
> -
> -    offset -= from;
> -    dir = pgd_offset(mm, from);
> -    flush_cache_range(vma, beg, end);
> -
> -    while (from < end) {
> -        pud_t *pud = pud_alloc(mm, dir, from);
> -        error = -ENOMEM;
> -        if (!pud)
> -            break;
> -        error = io_remap_pud_range(mm, pud, from, end - from, offset + from, 
> prot, space);
> -        if (error)
> -            break;
> -        from = (from + PGDIR_SIZE) & PGDIR_MASK;
> -        dir++;
> -    }
> -
> -    flush_tlb_range(vma, beg, end);
> -    return error;
> -}
> -EXPORT_SYMBOL(io_remap_pfn_range);
> -- 
> 1.7.6.401.g6a319
> 

I'll try them immediately and write back ASAP.
Thanks once already!

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

David Miller <davem@davemloft.net> wrote:

> Can someone with the effected hardware please test this?
> 
> --------------------
> [PATCH] sparc: Kill custom io_remap_pfn_range().
> 
> To handle the large physical addresses, just make a simple wrapper
> around remap_pfn_range() like MIPS does.
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>
> ---
> arch/sparc/include/asm/pgtable_32.h |   20 ++++-
> arch/sparc/include/asm/pgtable_64.h |   20 ++++-
> arch/sparc/mm/Makefile              |    1 -
> arch/sparc/mm/generic_32.c          |   99 ---------------------
> arch/sparc/mm/generic_64.c          |  165 -----------------------------------
> 5 files changed, 32 insertions(+), 273 deletions(-)
> delete mode 100644 arch/sparc/mm/generic_32.c
> delete mode 100644 arch/sparc/mm/generic_64.c
> 

Hello David!

After applying your patch I still get the errors:
[ 1117.556582] swap_free: Bad swap file entry 100004e000061800
[ 1117.556710] BUG: Bad page map in process ibv_devinfo  pte:9c0000c300104608 pmd:00f882d0
[ 1117.556827] addr:fffff80100114000 vm_flags:400844fa anon_vma:          (null) mapping:fffff807f623a410 index:6180082
[ 1117.557007] vma->vm_file->f_op->mmap: ib_uverbs_mmap+0x8/0x38 [ib_uverbs]
[ 1117.557054] Call Trace:
[ 1117.557093]  [00000000004ccff8] unmap_vmas+0x514/0x7f4
[ 1117.557143]  [00000000004d0d14] unmap_region+0xb4/0x164
[ 1117.557192]  [00000000004d1d60] do_munmap+0x2a8/0x31c
[ 1117.557247]  [000000000042d340] SyS_64_munmap+0x88/0xa8
[ 1117.557302]  [0000000000406154] linux_sparc_syscall+0x34/0x44
[ 1117.557347] Disabling lock debugging due to kernel taint


Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[David Miller]

From: Lukas Razik <linux-GLgANOly0l1BDLzU/O5InQ@public.gmane.org>
Date: Fri, 18 Nov 2011 05:02:57 +0000 (GMT)

> Hello David!
> 
> After applying your patch I still get the errors:

Thanks for testing, while I continue to look into this please double
check to make sure you really did run a kernel with the patch applied
:-)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[David Miller]

From: Lukas Razik <linux@razik.name>
Date: Fri, 18 Nov 2011 05:02:57 +0000 (GMT)

> After applying your patch I still get the errors:
> [ 1117.556582] swap_free: Bad swap file entry 100004e000061800
> [ 1117.556710] BUG: Bad page map in process ibv_devinfo  pte:9c0000c300104608 pmd:00f882d0

This looks almost like it's actually a mapping to physical memory, or
at least it is a PTE which hasn't had pgprot_noncached() applied to
it's protections.

Another hint is that pte_mkspecial() hasn't been performed on this PTE
either.

The PTE is:			0x9c0000c300104608

which amounts to:

	_PAGE_VALID		0x8000000000000000
	_PAGE_ACCESSED_4V	0x1000000000000000
	_PAGE_READ_4V		0x0800000000000000
	_PAGE_WRITE_4V		0x0400000000000000
	physical page		0x000000c300104000
	_PAGE_CP_4V		0x0000000000000400
	_PAGE_CV_4V		0x0000000000000200
	_PAGE_RESV_4V		0x0000000000000008

That last bit is mysterious, there shouldn't be anything setting
the reserved bit 0x8.

But that just so happens to be equal to the SUN4U PTE's "side effect"
bit, _PAGE_E_4U.

Hmmm...

That explains everything.  The problem is that we don't do the sparc64
PTE handling code patching in modules.  So it's left at the default 4U
versions.

So the PTE did have pgprot_noncached() applied to it's protections,
it's just that this function is inlined into the driver module and
hasn't been patched properly by the sun4v code patcher when the module
got loaded.

I'll work on a fix for this.
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Fri, 18 Nov 2011 01:17:08 -0500 (EST)

> That explains everything.  The problem is that we don't do the sparc64
> PTE handling code patching in modules.  So it's left at the default 4U
> versions.
 ...
> I'll work on a fix for this.

Ok, please test this out, thanks!

--------------------
[PATCH] sparc64: Patch sun4v code sequences properly on module load.

Some of the sun4v code patching occurs in inline functions visible
to, and usable by, modules.

Therefore we have to patch them up during module load.

Signed-off-by: David S. Miller <davem@davemloft.net>
---
 arch/sparc/kernel/entry.h    |    7 ++++++
 arch/sparc/kernel/module.c   |   27 +++++++++++++++++++++++
 arch/sparc/kernel/setup_64.c |   48 +++++++++++++++++++++++++----------------
 3 files changed, 63 insertions(+), 19 deletions(-)

diff --git a/arch/sparc/kernel/entry.h b/arch/sparc/kernel/entry.h
index e27f8ea..0c218e4 100644
--- a/arch/sparc/kernel/entry.h
+++ b/arch/sparc/kernel/entry.h
@@ -42,6 +42,9 @@ extern void fpsave(unsigned long *fpregs, unsigned long *fsr,
 extern void fpload(unsigned long *fpregs, unsigned long *fsr);
 
 #else /* CONFIG_SPARC32 */
+
+#include <asm/trap_block.h>
+
 struct popc_3insn_patch_entry {
 	unsigned int	addr;
 	unsigned int	insns[3];
@@ -57,6 +60,10 @@ extern struct popc_6insn_patch_entry __popc_6insn_patch,
 	__popc_6insn_patch_end;
 
 extern void __init per_cpu_patch(void);
+extern void sun4v_patch_1insn_range(struct sun4v_1insn_patch_entry *,
+				    struct sun4v_1insn_patch_entry *);
+extern void sun4v_patch_2insn_range(struct sun4v_2insn_patch_entry *,
+				    struct sun4v_2insn_patch_entry *);
 extern void __init sun4v_patch(void);
 extern void __init boot_cpu_id_too_large(int cpu);
 extern unsigned int dcache_parity_tl1_occurred;
diff --git a/arch/sparc/kernel/module.c b/arch/sparc/kernel/module.c
index da0c6c7..e551987 100644
--- a/arch/sparc/kernel/module.c
+++ b/arch/sparc/kernel/module.c
@@ -17,6 +17,8 @@
 #include <asm/processor.h>
 #include <asm/spitfire.h>
 
+#include "entry.h"
+
 #ifdef CONFIG_SPARC64
 
 #include <linux/jump_label.h>
@@ -203,6 +205,29 @@ int apply_relocate_add(Elf_Shdr *sechdrs,
 }
 
 #ifdef CONFIG_SPARC64
+static void do_patch_sections(const Elf_Ehdr *hdr,
+			      const Elf_Shdr *sechdrs)
+{
+	const Elf_Shdr *s, *sun4v_1insn = NULL, *sun4v_2insn = NULL;
+	char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
+
+	for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
+		if (!strcmp(".sun4v_1insn_patch", secstrings + s->sh_name))
+			sun4v_1insn = s;
+		if (!strcmp(".sun4v_2insn_patch", secstrings + s->sh_name))
+			sun4v_2insn = s;
+	}
+
+	if (sun4v_1insn && tlb_type == hypervisor) {
+		void *p = (void *) sun4v_1insn->sh_addr;
+		sun4v_patch_1insn_range(p, p + sun4v_1insn->sh_size);
+	}
+	if (sun4v_2insn && tlb_type == hypervisor) {
+		void *p = (void *) sun4v_2insn->sh_addr;
+		sun4v_patch_2insn_range(p, p + sun4v_2insn->sh_size);
+	}
+}
+
 int module_finalize(const Elf_Ehdr *hdr,
 		    const Elf_Shdr *sechdrs,
 		    struct module *me)
@@ -210,6 +235,8 @@ int module_finalize(const Elf_Ehdr *hdr,
 	/* make jump label nops */
 	jump_label_apply_nops(me);
 
+	do_patch_sections(hdr, sechdrs);
+
 	/* Cheetah's I-cache is fully coherent.  */
 	if (tlb_type == spitfire) {
 		unsigned long va;
diff --git a/arch/sparc/kernel/setup_64.c b/arch/sparc/kernel/setup_64.c
index c965595a..a854a1c 100644
--- a/arch/sparc/kernel/setup_64.c
+++ b/arch/sparc/kernel/setup_64.c
@@ -234,40 +234,50 @@ void __init per_cpu_patch(void)
 	}
 }
 
-void __init sun4v_patch(void)
+void sun4v_patch_1insn_range(struct sun4v_1insn_patch_entry *start,
+			     struct sun4v_1insn_patch_entry *end)
 {
-	extern void sun4v_hvapi_init(void);
-	struct sun4v_1insn_patch_entry *p1;
-	struct sun4v_2insn_patch_entry *p2;
-
-	if (tlb_type != hypervisor)
-		return;
+	while (start < end) {
+		unsigned long addr = start->addr;
 
-	p1 = &__sun4v_1insn_patch;
-	while (p1 < &__sun4v_1insn_patch_end) {
-		unsigned long addr = p1->addr;
-
-		*(unsigned int *) (addr +  0) = p1->insn;
+		*(unsigned int *) (addr +  0) = start->insn;
 		wmb();
 		__asm__ __volatile__("flush	%0" : : "r" (addr +  0));
 
-		p1++;
+		start++;
 	}
+}
 
-	p2 = &__sun4v_2insn_patch;
-	while (p2 < &__sun4v_2insn_patch_end) {
-		unsigned long addr = p2->addr;
+void sun4v_patch_2insn_range(struct sun4v_2insn_patch_entry *start,
+			     struct sun4v_2insn_patch_entry *end)
+{
+	while (start < end) {
+		unsigned long addr = start->addr;
 
-		*(unsigned int *) (addr +  0) = p2->insns[0];
+		*(unsigned int *) (addr +  0) = start->insns[0];
 		wmb();
 		__asm__ __volatile__("flush	%0" : : "r" (addr +  0));
 
-		*(unsigned int *) (addr +  4) = p2->insns[1];
+		*(unsigned int *) (addr +  4) = start->insns[1];
 		wmb();
 		__asm__ __volatile__("flush	%0" : : "r" (addr +  4));
 
-		p2++;
+		start++;
 	}
+}
+
+void __init sun4v_patch(void)
+{
+	extern void sun4v_hvapi_init(void);
+
+	if (tlb_type != hypervisor)
+		return;
+
+	sun4v_patch_1insn_range(&__sun4v_1insn_patch,
+				&__sun4v_1insn_patch_end);
+
+	sun4v_patch_2insn_range(&__sun4v_2insn_patch,
+				&__sun4v_2insn_patch_end);
 
 	sun4v_hvapi_init();
 }
-- 
1.7.6.401.g6a319

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

David Miller <davem@davemloft.net> wrote:
>>  That explains everything.  The problem is that we don't do the sparc64
>>  PTE handling code patching in modules.  So it's left at the default 4U
>>  versions.
> ...
>>  I'll work on a fix for this.
> 
> Ok, please test this out, thanks!
> 
> --------------------
> [PATCH] sparc64: Patch sun4v code sequences properly on module load.
> 
> Some of the sun4v code patching occurs in inline functions visible
> to, and usable by, modules.
> 
> Therefore we have to patch them up during module load.
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>
> ---
> arch/sparc/kernel/entry.h    |    7 ++++++
> arch/sparc/kernel/module.c   |   27 +++++++++++++++++++++++
> arch/sparc/kernel/setup_64.c |   48 +++++++++++++++++++++++++----------------
> 3 files changed, 63 insertions(+), 19 deletions(-)
> 

Hi David,

sorry that it took so long but I had to sleep to avoid mistakes - it was 6am here.
Thanks again for this fast patch!!!

Your first patch I could apply against linux-2.6.39.4. Your second patch I can't.
I've tried to apply it manually but I've seen too many differences and gave up.

Therefore I applied _both_ patches against linux-3.1.1.
Then I've tested ibv_devinfo with the builtin mlx drivers from linux-3.1.1 and it seems that the BUG message is away.

But I can't do further testing because I would need the mlx modules from OFED-1.5.4-rc4 (newest version) which doesn't compile with linux-3.1.1:
http://thread.gmane.org/gmane.linux.drivers.rdma/10192

Hence I would need your patch against 2.6.39.4 to say for sure if it works now. :-/
Please don't feel forced by me! It would be nice to have a patch but I see that you work on many areas needing improvement. Then I must wait for an OFED version that is compatible with linux-3.1.1 ...

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

----- Ursprüngliche Message -----

> Von: David Miller <davem@davemloft.net>
> An: linux@razik.name
> Cc: roland@purestorage.com; sparclinux@vger.kernel.org; linux-rdma@vger.kernel.org
> Gesendet: 7:55 Freitag, 18.November 2011
> Betreff: Re: [BUG] Bad page map in process ibv_devinfo
> 
> From: David Miller <davem@davemloft.net>
> Date: Fri, 18 Nov 2011 01:17:08 -0500 (EST)
> 
>>  That explains everything.  The problem is that we don't do the sparc64
>>  PTE handling code patching in modules.  So it's left at the default 4U
>>  versions.
> ...
>>  I'll work on a fix for this.
> 
> Ok, please test this out, thanks!
> 
> --------------------
> [PATCH] sparc64: Patch sun4v code sequences properly on module load.
> 
> Some of the sun4v code patching occurs in inline functions visible
> to, and usable by, modules.
> 
> Therefore we have to patch them up during module load.
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>
> ---
> arch/sparc/kernel/entry.h    |    7 ++++++
> arch/sparc/kernel/module.c   |   27 +++++++++++++++++++++++
> arch/sparc/kernel/setup_64.c |   48 +++++++++++++++++++++++++----------------
> 3 files changed, 63 insertions(+), 19 deletions(-)
> 

Hi David,

I could apply also your second patch to linux-2.6.39.4 manually. So you don't need to write another version.
Now I'll try to test it with the new OFED drivers...

Regards,
Lukas
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] Bad page map in process ibv_devinfo

[David Miller]

From: Lukas Razik <linux@razik.name>
Date: Fri, 18 Nov 2011 17:42:30 +0000 (GMT)

> I could apply also your second patch to linux-2.6.39.4 manually. So
> you don't need to write another version.  Now I'll try to test it
> with the new OFED drivers...

Thanks for all of your testing so far.

Re: [BUG] Bad page map in process ibv_devinfo

[Lukas Razik]

David Miller <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org> wrote:
> From: Lukas Razik <linux-GLgANOly0l1BDLzU/O5InQ@public.gmane.org>

> Date: Fri, 18 Nov 2011 17:42:30 +0000 (GMT)
> 
>>  I could apply also your second patch to linux-2.6.39.4 manually. So
>>  you don't need to write another version.  Now I'll try to test it
>>  with the new OFED drivers...
> 
> Thanks for all of your testing so far.

Hello David!

Oh, no! Please don't thank me. That's as if my grandma would have thanked after I've eaten her tasty cake... :)
I thank you and Roland very much for the fast and professional help! You did a very good job!
I still can't use OpenMPI over the ConnectX adapter (another error) but that's not for the SPARC list I think.

I think you've eliminated the BUG we talk about. :)
Again: I've applied the patch against 2.6.39.4 manually. I can't say if it also works with linux-3.1.1 because of OFED-1.5.4-rc4 .

Now if I run 'ibv_devinfo' I get no kernel BUG messages anymore and my 
system doesn't hang if I use one of the ibv_*_pingpong utilities:
---
cluster1:~# ibv_rc_pingpong -n 1000000
  local address:  LID 0x0006, QPN 0x04004e, PSN 0x411c25, GID ::
  remote address: LID 0x0003, QPN 0x0c004a, PSN 0xc6aa3d, GID ::
8192000000 bytes in 19.59 seconds = 3345.20 Mbit/sec
1000000 iters in 19.59 seconds = 19.59 usec/iter

cluster2:~# ibv_rc_pingpong -n 1000000 cluster1
  local address:  LID 0x0003, QPN 0x0c004a, PSN 0xc6aa3d, GID ::
  remote address: LID 0x0006, QPN 0x04004e, PSN 0x411c25, GID ::
8192000000 bytes in 19.60 seconds = 3344.02 Mbit/sec
1000000 iters in 19.60 seconds = 19.60 usec/iter
---

BTW: If you need someone in the future for testing a SPARC64 patch - just write me!

Regards and have a nice weekend,
Lukas


PS: That's the whole conversation:
http://thread.gmane.org/gmane.linux.drivers.rdma/10157
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html