[PATCH v2 0/3] ntb_hw_switchtec: Added support of >=4G memory windows

[PATCH v2 0/3] ntb_hw_switchtec: Added support of >=4G memory windows

[Wesley Sheng]

Hi, Everyone,

This patch series adds support of >=4G memory windows.

Current Switchtec's BAR setup registers are limited to 32bits,
corresponding to the maximum MW (memory window) size is <4G.
Increase the MW sizes with the addition of the BAR Setup Extension
Register for the upper 32bits of a 64bits MW size. This increases the MW
range to between 4K and 2^63.

Additionally, we've made the following changes:

* debug print 64bit aligned crosslink BAR numbers
* Fix the array size of NT req id mapping table

Tested with ntb_test.sh successfully based on NTB fixes series from
Logan Gunthorpe <logang@deltatee.com> at 
https://github.com/sbates130272/linux-p2pmem on branch of
ntb_multiport_fixes

Regards,
Wesley

--

Changed since v1:
  - Using upper_32_bits() and lower_32_bits() marcos makes it easier
    to read and avoids compiler warning on 32-bit arch
  - Reorder the patches to make the bug fixes first and add a "Fixes"
    line to the commit messages

--
Paul Selles (2):
  ntb_hw_switchtec: debug print 64bit aligned crosslink BAR Numbers
  ntb_hw_switchtec: Added support of >=4G memory windows

Wesley Sheng (1):
  ntb_hw_switchtec: NT req id mapping table register entry number should
    be 512

 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 11 ++++++++---
 include/linux/switchtec.h              | 10 +++++++---
 2 files changed, 15 insertions(+), 6 deletions(-)

-- 
2.7.4

[PATCH v2 1/3] ntb_hw_switchtec: debug print 64bit aligned crosslink BAR Numbers

[Wesley Sheng]

From: Paul Selles <paul.selles@microchip.com>

Switchtec NTB crosslink BARs are 64bit addressed but they are printed as
32bit addressed BARs. Fix debug log to increment the BAR numbers by 2 to
reflect the 64bit address alignment.

Fixes: 017525018202 ("ntb_hw_switchtec: Add initialization code for crosslink")
Signed-off-by: Paul Selles <paul.selles@microchip.com>
Signed-off-by: Wesley Sheng <wesley.sheng@microchip.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 5ee5f40..9916bc5 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -1120,7 +1120,7 @@ static int crosslink_enum_partition(struct switchtec_ntb *sndev,
 
 		dev_dbg(&sndev->stdev->dev,
 			"Crosslink BAR%d addr: %llx\n",
-			i, bar_addr);
+			i*2, bar_addr);
 
 		if (bar_addr != bar_space * i)
 			continue;
-- 
2.7.4

[PATCH v2 2/3] ntb_hw_switchtec: NT req id mapping table register entry number should be 512

[Wesley Sheng]

The number of available NT req id mapping table entries per NTB control
register is 512. The driver mistakenly limits the number to 256.

Fix the array size of NT req id mapping table.

Fixes: c082b04c9d40 ("NTB: switchtec: Add NTB hardware register definitions")
Signed-off-by: Wesley Sheng <wesley.sheng@microchip.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
---
 include/linux/switchtec.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/switchtec.h b/include/linux/switchtec.h
index ab400af..623719c9 100644
--- a/include/linux/switchtec.h
+++ b/include/linux/switchtec.h
@@ -244,8 +244,8 @@ struct ntb_ctrl_regs {
 		u64 xlate_addr;
 	} bar_entry[6];
 	u32 reserved2[216];
-	u32 req_id_table[256];
-	u32 reserved3[512];
+	u32 req_id_table[512];
+	u32 reserved3[256];
 	u64 lut_entry[512];
 } __packed;
 
-- 
2.7.4

[PATCH v2 3/3] ntb_hw_switchtec: Added support of >=4G memory windows

[Wesley Sheng]

From: Paul Selles <paul.selles@microchip.com>

Current Switchtec's BAR setup registers are limited to 32bits,
corresponding to the maximum MW (memory window) size is <4G.

Increase the MW sizes with the addition of the BAR Setup Extension
Register for the upper 32bits of a 64bits MW size. This increases the MW
range to between 4K and 2^63.

Reported-by: Boris Glimcher <boris.glimcher@emc.com>
Signed-off-by: Paul Selles <paul.selles@microchip.com>
Signed-off-by: Wesley Sheng <wesley.sheng@microchip.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
---
 drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 9 +++++++--
 include/linux/switchtec.h              | 6 +++++-
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
index 9916bc5..f6f0035 100644
--- a/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
+++ b/drivers/ntb/hw/mscc/ntb_hw_switchtec.c
@@ -264,6 +264,7 @@ static void switchtec_ntb_mw_clr_direct(struct switchtec_ntb *sndev, int idx)
 	ctl_val &= ~NTB_CTRL_BAR_DIR_WIN_EN;
 	iowrite32(ctl_val, &ctl->bar_entry[bar].ctl);
 	iowrite32(0, &ctl->bar_entry[bar].win_size);
+	iowrite32(0, &ctl->bar_ext_entry[bar].win_size);
 	iowrite64(sndev->self_partition, &ctl->bar_entry[bar].xlate_addr);
 }
 
@@ -286,7 +287,9 @@ static void switchtec_ntb_mw_set_direct(struct switchtec_ntb *sndev, int idx,
 	ctl_val |= NTB_CTRL_BAR_DIR_WIN_EN;
 
 	iowrite32(ctl_val, &ctl->bar_entry[bar].ctl);
-	iowrite32(xlate_pos | size, &ctl->bar_entry[bar].win_size);
+	iowrite32(xlate_pos | (lower_32_bits(size) & 0xFFFFF000),
+		  &ctl->bar_entry[bar].win_size);
+	iowrite32(upper_32_bits(size), &ctl->bar_ext_entry[bar].win_size);
 	iowrite64(sndev->self_partition | addr,
 		  &ctl->bar_entry[bar].xlate_addr);
 }
@@ -1053,7 +1056,9 @@ static int crosslink_setup_mws(struct switchtec_ntb *sndev, int ntb_lut_idx,
 		ctl_val |= NTB_CTRL_BAR_DIR_WIN_EN;
 
 		iowrite32(ctl_val, &ctl->bar_entry[bar].ctl);
-		iowrite32(xlate_pos | size, &ctl->bar_entry[bar].win_size);
+		iowrite32(xlate_pos | (lower_32_bits(size) & 0xFFFFF000),
+			  &ctl->bar_entry[bar].win_size);
+		iowrite32(upper_32_bits(size), &ctl->bar_ext_entry[bar].win_size);
 		iowrite64(sndev->peer_partition | addr,
 			  &ctl->bar_entry[bar].xlate_addr);
 	}
diff --git a/include/linux/switchtec.h b/include/linux/switchtec.h
index 623719c9..64aa25e 100644
--- a/include/linux/switchtec.h
+++ b/include/linux/switchtec.h
@@ -243,7 +243,11 @@ struct ntb_ctrl_regs {
 		u32 win_size;
 		u64 xlate_addr;
 	} bar_entry[6];
-	u32 reserved2[216];
+	struct {
+		u32 win_size;
+		u32 reserved[3];
+	} bar_ext_entry[6];
+	u32 reserved2[192];
 	u32 req_id_table[512];
 	u32 reserved3[256];
 	u64 lut_entry[512];
-- 
2.7.4

Re: [PATCH v2 0/3] ntb_hw_switchtec: Added support of >=4G memory windows

[Jon Mason]

On Thu, Dec 6, 2018 at 1:47 AM Wesley Sheng <wesley.sheng@microchip.com> wrote:
>
> Hi, Everyone,
>
> This patch series adds support of >=4G memory windows.
>
> Current Switchtec's BAR setup registers are limited to 32bits,
> corresponding to the maximum MW (memory window) size is <4G.
> Increase the MW sizes with the addition of the BAR Setup Extension
> Register for the upper 32bits of a 64bits MW size. This increases the MW
> range to between 4K and 2^63.
>
> Additionally, we've made the following changes:
>
> * debug print 64bit aligned crosslink BAR numbers
> * Fix the array size of NT req id mapping table
>
> Tested with ntb_test.sh successfully based on NTB fixes series from
> Logan Gunthorpe <logang@deltatee.com> at
> https://github.com/sbates130272/linux-p2pmem on branch of
> ntb_multiport_fixes

So, you based your patches on a series of patches not in the
ntb/ntb-next branch?  Please don't do this.  I see nothing in these
patches which requires that series, which makes this even more
unnecessary.  Since these are fairly trivial, I'm taking them and
pushing to the ntb-next branch to give these more time to be tested
(due to not being tested on the proper branch).  I would really
appreciate you testing the ntb-next branch as a sanity check.

Thanks,
Jon

>
> Regards,
> Wesley
>
> --
>
> Changed since v1:
>   - Using upper_32_bits() and lower_32_bits() marcos makes it easier
>     to read and avoids compiler warning on 32-bit arch
>   - Reorder the patches to make the bug fixes first and add a "Fixes"
>     line to the commit messages
>
> --
> Paul Selles (2):
>   ntb_hw_switchtec: debug print 64bit aligned crosslink BAR Numbers
>   ntb_hw_switchtec: Added support of >=4G memory windows
>
> Wesley Sheng (1):
>   ntb_hw_switchtec: NT req id mapping table register entry number should
>     be 512
>
>  drivers/ntb/hw/mscc/ntb_hw_switchtec.c | 11 ++++++++---
>  include/linux/switchtec.h              | 10 +++++++---
>  2 files changed, 15 insertions(+), 6 deletions(-)
>
> --
> 2.7.4
>

Re: [PATCH v2 0/3] ntb_hw_switchtec: Added support of >=4G memory windows

[Logan Gunthorpe]

On 2018-12-12 4:00 p.m., Jon Mason wrote:
> So, you based your patches on a series of patches not in the
> ntb/ntb-next branch?  Please don't do this.  I see nothing in these
> patches which requires that series, which makes this even more
> unnecessary.  Since these are fairly trivial, I'm taking them and
> pushing to the ntb-next branch to give these more time to be tested
> (due to not being tested on the proper branch).  I would really
> appreciate you testing the ntb-next branch as a sanity check.

The NTB test tools don't work with switchtec hardware without that patch
set, so there's no way to test the changes without that branch.

You're right that there is no compile-time dependency.

Logan

Re: [PATCH v2 0/3] ntb_hw_switchtec: Added support of >=4G memory windows

[Jon Mason]

On Wed, Dec 12, 2018 at 6:42 PM Logan Gunthorpe <logang@deltatee.com> wrote:
>
>
>
> On 2018-12-12 4:00 p.m., Jon Mason wrote:
> > So, you based your patches on a series of patches not in the
> > ntb/ntb-next branch?  Please don't do this.  I see nothing in these
> > patches which requires that series, which makes this even more
> > unnecessary.  Since these are fairly trivial, I'm taking them and
> > pushing to the ntb-next branch to give these more time to be tested
> > (due to not being tested on the proper branch).  I would really
> > appreciate you testing the ntb-next branch as a sanity check.
>
> The NTB test tools don't work with switchtec hardware without that patch
> set, so there's no way to test the changes without that branch.

Then let's get those patches in.  IIRC, I asked you to split up the
patch series to be bugfixes and features (or at least reorder the
series so I can split it up that way in my branches).  Also, I think
Serge had some comments that may/may not need to be addressed.  Could
you please reorder and resend (and Serge can comment as needed on the
resend)?

Thanks,
Jon


>
> You're right that there is no compile-time dependency.
>
> Logan
>
> --
> You received this message because you are subscribed to the Google Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com.
> To post to this group, send email to linux-ntb@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/f22b6112-920b-37f1-68fe-f43164a18f05%40deltatee.com.
> For more options, visit https://groups.google.com/d/optout.

Re: [PATCH v2 0/3] ntb_hw_switchtec: Added support of >=4G memory windows

[Logan Gunthorpe]

On 2018-12-12 4:57 p.m., Jon Mason wrote:
> On Wed, Dec 12, 2018 at 6:42 PM Logan Gunthorpe <logang@deltatee.com> wrote:
>>
>>
>>
>> On 2018-12-12 4:00 p.m., Jon Mason wrote:
>>> So, you based your patches on a series of patches not in the
>>> ntb/ntb-next branch?  Please don't do this.  I see nothing in these
>>> patches which requires that series, which makes this even more
>>> unnecessary.  Since these are fairly trivial, I'm taking them and
>>> pushing to the ntb-next branch to give these more time to be tested
>>> (due to not being tested on the proper branch).  I would really
>>> appreciate you testing the ntb-next branch as a sanity check.
>>
>> The NTB test tools don't work with switchtec hardware without that patch
>> set, so there's no way to test the changes without that branch.
> 
> Then let's get those patches in.  IIRC, I asked you to split up the
> patch series to be bugfixes and features (or at least reorder the
> series so I can split it up that way in my branches).  Also, I think
> Serge had some comments that may/may not need to be addressed.  Could
> you please reorder and resend (and Serge can comment as needed on the
> resend)?

I resent a while back and responded to all the feedback. Every patch in
that series fixes a bug. None of them add features.

Logan

Re: [PATCH v2 0/3] ntb_hw_switchtec: Added support of >=4G memory windows

[Jon Mason]

On Wed, Dec 12, 2018 at 7:01 PM Logan Gunthorpe <logang@deltatee.com> wrote:
>
>
>
> On 2018-12-12 4:57 p.m., Jon Mason wrote:
> > On Wed, Dec 12, 2018 at 6:42 PM Logan Gunthorpe <logang@deltatee.com> wrote:
> >>
> >>
> >>
> >> On 2018-12-12 4:00 p.m., Jon Mason wrote:
> >>> So, you based your patches on a series of patches not in the
> >>> ntb/ntb-next branch?  Please don't do this.  I see nothing in these
> >>> patches which requires that series, which makes this even more
> >>> unnecessary.  Since these are fairly trivial, I'm taking them and
> >>> pushing to the ntb-next branch to give these more time to be tested
> >>> (due to not being tested on the proper branch).  I would really
> >>> appreciate you testing the ntb-next branch as a sanity check.
> >>
> >> The NTB test tools don't work with switchtec hardware without that patch
> >> set, so there's no way to test the changes without that branch.
> >
> > Then let's get those patches in.  IIRC, I asked you to split up the
> > patch series to be bugfixes and features (or at least reorder the
> > series so I can split it up that way in my branches).  Also, I think
> > Serge had some comments that may/may not need to be addressed.  Could
> > you please reorder and resend (and Serge can comment as needed on the
> > resend)?
>
> I resent a while back and responded to all the feedback. Every patch in
> that series fixes a bug. None of them add features.

Per https://lkml.org/lkml/2018/6/12/552, I asked the series be split
up and the comments to be cleaned-up.  You repushed without addressing
this (or Serge's comments), which caused me to ignore the series.
Again, I'm happy to take it as a single series and split it up on my
end, I just need the patches reordered to have the bugfixes I
specified in the front to allow for this to be easily done.

Alternatively, you can rebase and resend them as-is to the ntb mailing
list and we can rehash it out there.

Thanks,
Jon

>
> Logan
>
> --
> You received this message because you are subscribed to the Google Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com.
> To post to this group, send email to linux-ntb@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/ff7019a0-bac2-2de2-d06c-d54a82286b90%40deltatee.com.
> For more options, visit https://groups.google.com/d/optout.

Re: [PATCH v2 0/3] ntb_hw_switchtec: Added support of >=4G memory windows

[Logan Gunthorpe]

On 2018-12-12 5:25 p.m., Jon Mason wrote:
> On Wed, Dec 12, 2018 at 7:01 PM Logan Gunthorpe <logang@deltatee.com> wrote:
>>
>>
>>
>> On 2018-12-12 4:57 p.m., Jon Mason wrote:
>>> On Wed, Dec 12, 2018 at 6:42 PM Logan Gunthorpe <logang@deltatee.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2018-12-12 4:00 p.m., Jon Mason wrote:
>>>>> So, you based your patches on a series of patches not in the
>>>>> ntb/ntb-next branch?  Please don't do this.  I see nothing in these
>>>>> patches which requires that series, which makes this even more
>>>>> unnecessary.  Since these are fairly trivial, I'm taking them and
>>>>> pushing to the ntb-next branch to give these more time to be tested
>>>>> (due to not being tested on the proper branch).  I would really
>>>>> appreciate you testing the ntb-next branch as a sanity check.
>>>>
>>>> The NTB test tools don't work with switchtec hardware without that patch
>>>> set, so there's no way to test the changes without that branch.
>>>
>>> Then let's get those patches in.  IIRC, I asked you to split up the
>>> patch series to be bugfixes and features (or at least reorder the
>>> series so I can split it up that way in my branches).  Also, I think
>>> Serge had some comments that may/may not need to be addressed.  Could
>>> you please reorder and resend (and Serge can comment as needed on the
>>> resend)?
>>
>> I resent a while back and responded to all the feedback. Every patch in
>> that series fixes a bug. None of them add features.
> 
> Per https://lkml.org/lkml/2018/6/12/552, I asked the series be split
> up and the comments to be cleaned-up.  You repushed without addressing
> this (or Serge's comments), which caused me to ignore the series.
> Again, I'm happy to take it as a single series and split it up on my
> end, I just need the patches reordered to have the bugfixes I
> specified in the front to allow for this to be easily done.

And what was not clear about my response?

https://lkml.org/lkml/2018/6/12/577

That commit is absolutely *not* a feature request. Without it, none of
the other fixes will fix anything and are thus worse than useless.

I responded to Serge's comments and then responded *again* in the cover
letter of v2. What he was asking for was, and still is, physically
impossible.

Logan