linux rdma 3.14 merge plans

linux rdma 3.14 merge plans

[Or Gerlitz]

Hi Roland,

Currently there is single patch for 3.14 on your for-next branch, the
usnic driver. With 3.13 being on rc7 and likely to be released next
week, are you planning any other merges for 3.14? we have patches
waiting for weeks and months without any comment from you.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Roland Dreier]

On Tue, Jan 7, 2014 at 1:02 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Currently there is single patch for 3.14 on your for-next branch, the
> usnic driver. With 3.13 being on rc7 and likely to be released next
> week, are you planning any other merges for 3.14? we have patches
> waiting for weeks and months without any comment from you.

I am definitely planning on merging the new IBoE IP addressing stuff,
since we seem to have solved the ABI issues.

The UD flow steering patches seem good and I will take a closer look soon.

And there are quite a few usnic patches still to pick up.

I'm confident that will all make it.


The data integrity stuff I'm not so sure about.  Sean raised some I
think legitimate questions about whether all this should be added to
the verbs API and I want to see more discussion or at least have a
deep think about this myself before comitting.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Sagi Grimberg]

On 1/8/2014 2:51 AM, Roland Dreier wrote:
> On Tue, Jan 7, 2014 at 1:02 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> Currently there is single patch for 3.14 on your for-next branch, the
>> usnic driver. With 3.13 being on rc7 and likely to be released next
>> week, are you planning any other merges for 3.14? we have patches
>> waiting for weeks and months without any comment from you.
> I am definitely planning on merging the new IBoE IP addressing stuff,
> since we seem to have solved the ABI issues.
>
> The UD flow steering patches seem good and I will take a closer look soon.
>
> And there are quite a few usnic patches still to pick up.
>
> I'm confident that will all make it.
>
>
> The data integrity stuff I'm not so sure about.  Sean raised some I
> think legitimate questions about whether all this should be added to
> the verbs API and I want to see more discussion or at least have a
> deep think about this myself before comitting.

Hey Roland,

I don't think that Sean didn't question weather data-integrity support 
should or shouldn't be added to Verbs API (Sean correct me if I'm 
wrong), but rather the way it should be added.
 From our discussion on this, the only conflict that Sean and I had was 
weather the protection setup should "ride" on ib_post_send.
Sean suggested a separate routine that would post on the SQ. I think 
that in the current framework where
placing a fast-path operation is done via ib_post_send, we keep current 
implementation, and open a discussion
if it is a good idea to migrate "non-send" work-requests out of 
ib_post_send (also fast-registration and memory-windows).

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On 08/01/2014 02:51, Roland Dreier wrote:
> I am definitely planning on merging the new IBoE IP addressing stuff,
> since we seem to have solved the ABI issues.
>
> The UD flow steering patches seem good and I will take a closer look soon.

This sounds good,  repeating myself, with 3.13-rc7 being out, 3.13 might 
be released Monday, so in case something need to be fixed, can you take
this closer look asap so we have enough time for changes? also when you 
expect this merge to happen?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On 08/01/2014 02:51, Roland Dreier wrote:
> The data integrity stuff I'm not so sure about.  Sean raised some I think legitimate questions about whether all this should be added to the verbs API and I want to see more discussion or at least have a deep think about this myself before comitting.

Well, re the deep think thing, we have problem here -- the patches were 
posted three months ago, on Oct 15th 2013, and so far not a single bit 
of input from you. Can you take a look on that as soon as possible and 
let us know your thoughts?

As for the questions posed by Sean -- Sagi, Tzahi and myself provided 
detailed answers. Taking into account the small scale of the rdma 
community, I don't see any other choice other than the guy wearing the 
maintainer hat to step in and provide his say or follow up questions 
that would cut things out or  revive the discussion.

Or.





--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Nicholas A. Bellinger]

Hi Roland & Co,

Just curious if your planning to take a look at this series soon for
v3.14 code..?

As you might imagine, I'd like to see it merged this round in order to
move forward on iser-target protection for v3.15.

Are there any specific issues that you'd like to see addressed for an
initial merge..?

Thank you,

--nab

On Wed, 2014-01-08 at 11:37 +0200, Or Gerlitz wrote:
> On 08/01/2014 02:51, Roland Dreier wrote:
> > The data integrity stuff I'm not so sure about.  Sean raised some I
> think legitimate questions about whether all this should be added to
> the verbs API and I want to see more discussion or at least have a
> deep think about this myself before comitting.
> 
> Well, re the deep think thing, we have problem here -- the patches were 
> posted three months ago, on Oct 15th 2013, and so far not a single bit 
> of input from you. Can you take a look on that as soon as possible and 
> let us know your thoughts?
> 
> As for the questions posed by Sean -- Sagi, Tzahi and myself provided 
> detailed answers. Taking into account the small scale of the rdma 
> community, I don't see any other choice other than the guy wearing the 
> maintainer hat to step in and provide his say or follow up questions 
> that would cut things out or  revive the discussion.
> 
> Or.
> 
> 
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Nicholas A. Bellinger]

On Mon, 2014-01-13 at 12:32 -0800, Nicholas A. Bellinger wrote:
> Hi Roland & Co,
> 
> Just curious if your planning to take a look at this series soon for
> v3.14 code..?
> 
> As you might imagine, I'd like to see it merged this round in order to
> move forward on iser-target protection for v3.15.
> 
> Are there any specific issues that you'd like to see addressed for an
> initial merge..?

Hi again Roland,

Ping^2..?  We're quite eager to see this code merged this round.

Is there a specific issue on these patches preventing a v3.14 merge..?

Thank you,

--nab

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Nicholas A. Bellinger]

On Thu, 2014-01-16 at 12:13 -0800, Roland Dreier wrote:
> 
> > On Mon, 2014-01-13 at 12:32 -0800, Nicholas A. Bellinger wrote:
> > > Hi Roland & Co,
> > >
> > > Just curious if your planning to take a look at this series soon for
> > > v3.14 code..?
> > >
> > > As you might imagine, I'd like to see it merged this round in order to
> > > move forward on iser-target protection for v3.15.
> > >
> > > Are there any specific issues that you'd like to see addressed for an
> > > initial merge..?
> 
> I haven't really had a chance to look at the new API and decide if it
> makes sense as the way to expose these features.  Have you looked at
> the new kernel API?  Any thoughts?

I've reviewed the API from the perspective of what's required for
implementing protection support in iser, and currently don't have any
recommendations or objections beyond what has been proposed by Sagi & Co
in PATCH-v4 code.

> How does it compare with how other subsystems have exposed protection
> info?
> 

So there is not a interface in SCSI land for interacting (directly) with
hardware protection support, as it's primarily just telling SCSI what
protection modes are supported while the rest is implemented in vendor
specific firmware interfaces.  (CC'ing MKP)

> Right now I'm dealing with fixing the fallout from picking up the "IP
> addressing for IBoE" and other patch sets that were supposedly "ready
> to merge for a long time" so I'm not sure I'll get to the protection
> changes in time for 3.14.
> 

Pretty please for v3.14..?

Sagi & Co are committed to supporting the changes, and are ready to do
what's necessary for a merge.  AFAICT the series has gone through enough
list review over the last months without (serious) objections, so I'm
not sure what else is necessary to move forward for a merge, aside from
your input of course.

:-)

--nab

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Greg Kroah-Hartman]

On Thu, Jan 16, 2014 at 01:14:12PM -0800, Nicholas A. Bellinger wrote:
> On Thu, 2014-01-16 at 12:13 -0800, Roland Dreier wrote:
> > 
> > > On Mon, 2014-01-13 at 12:32 -0800, Nicholas A. Bellinger wrote:
> > > > Hi Roland & Co,
> > > >
> > > > Just curious if your planning to take a look at this series soon for
> > > > v3.14 code..?
> > > >
> > > > As you might imagine, I'd like to see it merged this round in order to
> > > > move forward on iser-target protection for v3.15.
> > > >
> > > > Are there any specific issues that you'd like to see addressed for an
> > > > initial merge..?
> > 
> > I haven't really had a chance to look at the new API and decide if it
> > makes sense as the way to expose these features.  Have you looked at
> > the new kernel API?  Any thoughts?
> 
> I've reviewed the API from the perspective of what's required for
> implementing protection support in iser, and currently don't have any
> recommendations or objections beyond what has been proposed by Sagi & Co
> in PATCH-v4 code.
> 
> > How does it compare with how other subsystems have exposed protection
> > info?
> > 
> 
> So there is not a interface in SCSI land for interacting (directly) with
> hardware protection support, as it's primarily just telling SCSI what
> protection modes are supported while the rest is implemented in vendor
> specific firmware interfaces.  (CC'ing MKP)
> 
> > Right now I'm dealing with fixing the fallout from picking up the "IP
> > addressing for IBoE" and other patch sets that were supposedly "ready
> > to merge for a long time" so I'm not sure I'll get to the protection
> > changes in time for 3.14.
> > 
> 
> Pretty please for v3.14..?

It's _really_ late in the development cycle for new stuff for 3.14.  My
trees have been closed for almost a week now, for major stuff, and for
anything else for a few days now.  It would be good if you could get
your patchsets into the 0-day testing bot earlier to shake out any build
issues that might happen with them, please work with that developer to
help make the merging of the code easier.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On Thu, Jan 16, 2014 at 11:37 PM, Greg Kroah-Hartman
<gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r@public.gmane.org> wrote:
> On Thu, Jan 16, 2014 at 01:14:12PM -0800, Nicholas A. Bellinger wrote:
>> On Thu, 2014-01-16 at 12:13 -0800, Roland Dreier wrote:
>> > > On Mon, 2014-01-13 at 12:32 -0800, Nicholas A. Bellinger wrote:
>> > > > Hi Roland & Co,
>> > > > Just curious if your planning to take a look at this series soon for
>> > > > v3.14 code..?
>> > > >
>> > > > As you might imagine, I'd like to see it merged this round in order to
>> > > > move forward on iser-target protection for v3.15.
>> > > >
>> > > > Are there any specific issues that you'd like to see addressed for an
>> > > > initial merge..?
>> >
>> > I haven't really had a chance to look at the new API and decide if it
>> > makes sense as the way to expose these features.  Have you looked at
>> > the new kernel API?  Any thoughts?
>>
>> I've reviewed the API from the perspective of what's required for
>> implementing protection support in iser, and currently don't have any
>> recommendations or objections beyond what has been proposed by Sagi & Co
>> in PATCH-v4 code.
>>
>> > How does it compare with how other subsystems have exposed protection
>> > info?
>> >
>>
>> So there is not a interface in SCSI land for interacting (directly) with
>> hardware protection support, as it's primarily just telling SCSI what
>> protection modes are supported while the rest is implemented in vendor
>> specific firmware interfaces.  (CC'ing MKP)
>>
>> > Right now I'm dealing with fixing the fallout from picking up the "IP
>> > addressing for IBoE" and other patch sets that were supposedly "ready
>> > to merge for a long time" so I'm not sure I'll get to the protection
>> > changes in time for 3.14.
>> >
>>
>> Pretty please for v3.14..?
>
> It's _really_ late in the development cycle for new stuff for 3.14.  My
> trees have been closed for almost a week now, for major stuff, and for
> anything else for a few days now.  It would be good if you could get
> your patchsets into the 0-day testing bot earlier to shake out any build
> issues that might happen with them, please work with that developer to
> help make the merging of the code easier.

Greg,

The T10 patches were posted whole three months ago (V0 Oct 15th). I
don't see why another cycle should get lost just because there was no
maintainer feedback on them throughout this whole period. There's
enough time for us to fix things that will show up in the testing bot
before Roland sends his pull request over the two weeks merge window.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Greg Kroah-Hartman]

On Thu, Jan 16, 2014 at 11:42:19PM +0200, Or Gerlitz wrote:
> On Thu, Jan 16, 2014 at 11:37 PM, Greg Kroah-Hartman
> <gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r@public.gmane.org> wrote:
> > On Thu, Jan 16, 2014 at 01:14:12PM -0800, Nicholas A. Bellinger wrote:
> >> On Thu, 2014-01-16 at 12:13 -0800, Roland Dreier wrote:
> >> > > On Mon, 2014-01-13 at 12:32 -0800, Nicholas A. Bellinger wrote:
> >> > > > Hi Roland & Co,
> >> > > > Just curious if your planning to take a look at this series soon for
> >> > > > v3.14 code..?
> >> > > >
> >> > > > As you might imagine, I'd like to see it merged this round in order to
> >> > > > move forward on iser-target protection for v3.15.
> >> > > >
> >> > > > Are there any specific issues that you'd like to see addressed for an
> >> > > > initial merge..?
> >> >
> >> > I haven't really had a chance to look at the new API and decide if it
> >> > makes sense as the way to expose these features.  Have you looked at
> >> > the new kernel API?  Any thoughts?
> >>
> >> I've reviewed the API from the perspective of what's required for
> >> implementing protection support in iser, and currently don't have any
> >> recommendations or objections beyond what has been proposed by Sagi & Co
> >> in PATCH-v4 code.
> >>
> >> > How does it compare with how other subsystems have exposed protection
> >> > info?
> >> >
> >>
> >> So there is not a interface in SCSI land for interacting (directly) with
> >> hardware protection support, as it's primarily just telling SCSI what
> >> protection modes are supported while the rest is implemented in vendor
> >> specific firmware interfaces.  (CC'ing MKP)
> >>
> >> > Right now I'm dealing with fixing the fallout from picking up the "IP
> >> > addressing for IBoE" and other patch sets that were supposedly "ready
> >> > to merge for a long time" so I'm not sure I'll get to the protection
> >> > changes in time for 3.14.
> >> >
> >>
> >> Pretty please for v3.14..?
> >
> > It's _really_ late in the development cycle for new stuff for 3.14.  My
> > trees have been closed for almost a week now, for major stuff, and for
> > anything else for a few days now.  It would be good if you could get
> > your patchsets into the 0-day testing bot earlier to shake out any build
> > issues that might happen with them, please work with that developer to
> > help make the merging of the code easier.
> 
> Greg,
> 
> The T10 patches were posted whole three months ago (V0 Oct 15th). I
> don't see why another cycle should get lost just because there was no
> maintainer feedback on them throughout this whole period. There's
> enough time for us to fix things that will show up in the testing bot
> before Roland sends his pull request over the two weeks merge window.

There's the rule that things need to be in linux-next for a week or so
to shake things out before going to Linus, and that new things can't be
added to a maintainer tree after Linus does a release before -rc1 is
out.  We can't break those rules unless you want to answer to the
linux-next developer and Linus for it.

Yes, I know your patches have been pending for a long time, this merge
window was tough in that it covered two major holidays, your patches
aren't the only thing that is going to miss 3.14 by a long-shot, there
are lots of other developers who also didn't get stuff in due to
maintainers taking long vacations.

Which is fine, and is why we have 3 month release cycles, there's
nothing wrong with waiting for 3.15 if there are build issues that are
cropping up this late in the cycle.  That's why I suggest you get your
trees into the 0-day bot test systems, so that a maintainer has a
semblance of confidence that the build will not break, and the "obvious"
security issues are resolved, if they take your patches.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Nicholas A. Bellinger]

On Thu, 2014-01-16 at 23:42 +0200, Or Gerlitz wrote:
> On Thu, Jan 16, 2014 at 11:37 PM, Greg Kroah-Hartman
> <gregkh-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r@public.gmane.org> wrote:
> > On Thu, Jan 16, 2014 at 01:14:12PM -0800, Nicholas A. Bellinger wrote:
> >> On Thu, 2014-01-16 at 12:13 -0800, Roland Dreier wrote:

<SNIP>

> >> > I haven't really had a chance to look at the new API and decide if it
> >> > makes sense as the way to expose these features.  Have you looked at
> >> > the new kernel API?  Any thoughts?
> >>
> >> I've reviewed the API from the perspective of what's required for
> >> implementing protection support in iser, and currently don't have any
> >> recommendations or objections beyond what has been proposed by Sagi & Co
> >> in PATCH-v4 code.
> >>
> >> > How does it compare with how other subsystems have exposed protection
> >> > info?
> >> >
> >>
> >> So there is not a interface in SCSI land for interacting (directly) with
> >> hardware protection support, as it's primarily just telling SCSI what
> >> protection modes are supported while the rest is implemented in vendor
> >> specific firmware interfaces.  (CC'ing MKP)
> >>
> >> > Right now I'm dealing with fixing the fallout from picking up the "IP
> >> > addressing for IBoE" and other patch sets that were supposedly "ready
> >> > to merge for a long time" so I'm not sure I'll get to the protection
> >> > changes in time for 3.14.
> >> >
> >>
> >> Pretty please for v3.14..?
> >
> > It's _really_ late in the development cycle for new stuff for 3.14.  My
> > trees have been closed for almost a week now, for major stuff, and for
> > anything else for a few days now.  It would be good if you could get
> > your patchsets into the 0-day testing bot earlier to shake out any build
> > issues that might happen with them, please work with that developer to
> > help make the merging of the code easier.
> 
> Greg,
> 
> The T10 patches were posted whole three months ago (V0 Oct 15th). I
> don't see why another cycle should get lost just because there was no
> maintainer feedback on them throughout this whole period. There's
> enough time for us to fix things that will show up in the testing bot
> before Roland sends his pull request over the two weeks merge window.
> 

So as Greg noted, it's still useful to get this into the 0-day build
testing now.

That said, pushing the -v4 series into target-pending/rdma-dif, and
Fengguang's scripts should be picking it up shortly.

--nab

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Martin K. Petersen]

>>>>> "nab" == Nicholas A Bellinger <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org> writes:

nab> So there is not a interface in SCSI land for interacting (directly)
nab> with hardware protection support, as it's primarily just telling
nab> SCSI what protection modes are supported while the rest is
nab> implemented in vendor specific firmware interfaces.  (CC'ing MKP)

I've worked on getting my DIX1.1 changes ready for submission the last
couple of days. That'll take the magic out of firmware and into the SCSI
stack.

I'll get those patches out early next week. They are fairly trivial but
do require some changes to mptNsas, lpfc and qla2xxx.

-- 
Martin K. Petersen	Oracle Linux Engineering
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Roland Dreier]

On Thu, Jan 16, 2014 at 1:14 PM, Nicholas A. Bellinger
<nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org> wrote:
> I've reviewed the API from the perspective of what's required for
> implementing protection support in iser, and currently don't have any
> recommendations or objections beyond what has been proposed by Sagi & Co
> in PATCH-v4 code.

I guess I'm a little confused about why we need verbs support for this
to implement DIF/DIX in iser.  Isn't the whole point of protection to
have end-to-end checksums, rather than having checksums computed by
the transport after there's a chance for corruption?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Nicholas A. Bellinger]

On Sat, 2014-01-18 at 13:42 -0800, Roland Dreier wrote:
> On Thu, Jan 16, 2014 at 1:14 PM, Nicholas A. Bellinger
> <nab@linux-iscsi.org> wrote:
> > I've reviewed the API from the perspective of what's required for
> > implementing protection support in iser, and currently don't have any
> > recommendations or objections beyond what has been proposed by Sagi & Co
> > in PATCH-v4 code.
> 
> I guess I'm a little confused about why we need verbs support for this
> to implement DIF/DIX in iser.  Isn't the whole point of protection to
> have end-to-end checksums, rather than having checksums computed by
> the transport after there's a chance for corruption?
> 

So to my knowledge, there are three target side DIX HBA modes of
operation:

  - TARGET PASS: Fabric + Backend support PI
  - TARGET INSERT: Fabric does not support PI, backend supports PI
  - TARGET STRIP: Fabric supports PI, backend does not support PI

The scenario your thinking about above is the 'TARGET INSERT' case,
where the initiator does not generate PI, but the backend device on the
target side expects PI, so the target fabric ends up generating PI on
incoming WRITEs, and verifying + striping PI on outgoing READs.

The scenario for 'TARGET STRIP' is when the initiator generates PI but
the backend device does not support/process PI, so the target verifies +
strips PI on incoming WRITESs, and inserts PI on outgoing READs.

Your correct that both of these modes don't provide true end-to-end
protection, and my understanding is that they are provided as a way to
accommodate existing fabrics + backend devices where PI is not supported
all the way through the stack.

The 'TARGET PASS' is the scenario that provides true end-to-end
guarantees, where for WRITEs PI is generated by the Host OS, verified +
passed on the initiator side HBA, verified + passed on the target HBA,
and verified + stored on the device backend.  For READs, PI is retrieved
from the backend device, verified + passed on the target HBA, verified +
passed on the initiator HBA, and finally verified on the Host OS.

So in the proposed RDMA VERBs changes these three modes of target DIX
operation are supported.  Also it's my understanding (Sagi & Co, please
correct me), that the proposed changes are implemented to be independent
of target/initiator mode DIX operation.

--nab

Re: linux rdma 3.14 merge plans

[sagi grimberg]

On 1/19/2014 5:42 AM, Nicholas A. Bellinger wrote:
> On Sat, 2014-01-18 at 13:42 -0800, Roland Dreier wrote:
>> On Thu, Jan 16, 2014 at 1:14 PM, Nicholas A. Bellinger
>> <nab@linux-iscsi.org> wrote:
>>> I've reviewed the API from the perspective of what's required for
>>> implementing protection support in iser, and currently don't have any
>>> recommendations or objections beyond what has been proposed by Sagi & Co
>>> in PATCH-v4 code.
>> I guess I'm a little confused about why we need verbs support for this
>> to implement DIF/DIX in iser.  Isn't the whole point of protection to
>> have end-to-end checksums, rather than having checksums computed by
>> the transport after there's a chance for corruption?
>>
> So to my knowledge, there are three target side DIX HBA modes of
> operation:
>
>    - TARGET PASS: Fabric + Backend support PI
>    - TARGET INSERT: Fabric does not support PI, backend supports PI
>    - TARGET STRIP: Fabric supports PI, backend does not support PI
>
> The scenario your thinking about above is the 'TARGET INSERT' case,
> where the initiator does not generate PI, but the backend device on the
> target side expects PI, so the target fabric ends up generating PI on
> incoming WRITEs, and verifying + striping PI on outgoing READs.
>
> The scenario for 'TARGET STRIP' is when the initiator generates PI but
> the backend device does not support/process PI, so the target verifies +
> strips PI on incoming WRITESs, and inserts PI on outgoing READs.
>
> Your correct that both of these modes don't provide true end-to-end
> protection, and my understanding is that they are provided as a way to
> accommodate existing fabrics + backend devices where PI is not supported
> all the way through the stack.
>
> The 'TARGET PASS' is the scenario that provides true end-to-end
> guarantees, where for WRITEs PI is generated by the Host OS, verified +
> passed on the initiator side HBA, verified + passed on the target HBA,
> and verified + stored on the device backend.  For READs, PI is retrieved
> from the backend device, verified + passed on the target HBA, verified +
> passed on the initiator HBA, and finally verified on the Host OS.
>
> So in the proposed RDMA VERBs changes these three modes of target DIX
> operation are supported.  Also it's my understanding (Sagi & Co, please
> correct me), that the proposed changes are implemented to be independent
> of target/initiator mode DIX operation.

Correct. Verbs API allow all supported protection operations without any 
peer dependencies.

>
> --nab
>

Thanks Nic,  let me elaborate on this,

It is true that T10-PI aims for end-to-end data-integrity, the verbs API 
offer HW offload for protection
information processing which is VERY expensive for CPU computation (CRC 
verify for each block). T10-PI
is intended to be offloaded, both on the backend devices which supports 
this feature and for fabric
transports (such as Qlogic/Emulex FC drivers), Verbs API just adds RDMA 
to the game...

T10-PI specifies protection information scheme in a way that over the 
wire each protection interval is
followed by 8 bytes of data-integrity (interleaved). In the memory on 
the other hand, the data and
protection block-guards may lie in separated buffers (for example that 
is the preferred approach in block
and SCSI layers).

Newly introduced REG_SIG_MR work request allows to (fast) register a 
"signature" memory key which
incorporates the protection method and pattern used:
1. Where is the data? (data sge)
2. Where is the protection block-guards? (protection sge)
3. What are the signature attributes? (T10-PI method, crc/reftag/apptag 
seeds, verify mask, memory pattern, wire pattern)
When doing the actual data-transfer the HCA will enforce T10-PI scheme 
(See my cover-letter for a more detailed explanation).

If you take a look in SCSI implementation you will see that SCSI signals 
the transport of protection attributes in
scsi_cmnd (prot SG-list, protection type, guard type, protection 
operation). Verbs API allows full offload of all
T10-PI operations:
1. INSERT - HCA computes/generates data-integrity block-guards and 
writes them according to the specified
     pattern (interleaved with the data or in a separated buffer).
2. STRIP (and VERIFY) - HCA verifies incoming data-integrity 
block-guards and strip them from the data stream.
3. PASS (and VERIFY) - HCA verifies incoming data-integrity block-guards 
and passes them forward according to
     the specified pattern (interleaved/separated).

In addition, Verbs API can be easily extended to support other 
data-integrity methods (XOR-32, CRC-32, etc...)
so that an application interested in data-integrity has signature verbs 
in its tool-box. This is why we use "Signature"
notation and refer to T10-PI as a specific signature method.

Hope this helps,
Sagi.

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On Sun, Jan 19, 2014 at 1:20 PM, sagi grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
> Thanks Nic,  let me elaborate on this,
>
> It is true that T10-PI aims for end-to-end data-integrity, the verbs API offer HW offload for protection
> information processing which is VERY expensive for CPU computation (CRC verify for each block). T10-PI
> is intended to be offloaded, both on the backend devices which supports this feature and for fabric
> transports (such as Qlogic/Emulex FC drivers), Verbs API just adds RDMA to the game...
>
> T10-PI specifies protection information scheme in a way that over the wire each protection interval is
> followed by 8 bytes of data-integrity (interleaved). In the memory on the other hand, the data and
> protection block-guards may lie in separated buffers (for example that is the preferred approach in block
> and SCSI layers).
>
> Newly introduced REG_SIG_MR work request allows to (fast) register a "signature" memory key which
> incorporates the protection method and pattern used:
> 1. Where is the data? (data sge)
> 2. Where is the protection block-guards? (protection sge)
> 3. What are the signature attributes? (T10-PI method, crc/reftag/apptag seeds, verify mask, memory pattern, wire pattern)
> When doing the actual data-transfer the HCA will enforce T10-PI scheme (See my cover-letter for a more detailed explanation).
>
> If you take a look in SCSI implementation you will see that SCSI signals the transport of protection attributes in
> scsi_cmnd (prot SG-list, protection type, guard type, protection operation). Verbs API allows full offload of all
> T10-PI operations:
> 1. INSERT - HCA computes/generates data-integrity block-guards and writes them according to the specified
>     pattern (interleaved with the data or in a separated buffer).
> 2. STRIP (and VERIFY) - HCA verifies incoming data-integrity block-guards and strip them from the data stream.
> 3. PASS (and VERIFY) - HCA verifies incoming data-integrity block-guards and passes them forward according to
>     the specified pattern (interleaved/separated).
>
> In addition, Verbs API can be easily extended to support other data-integrity methods (XOR-32, CRC-32, etc...)
> so that an application interested in data-integrity has signature verbs in its tool-box. This is why we use "Signature"
> notation and refer to T10-PI as a specific signature method.
>
> Hope this helps,

Hi Roland,

So with Nic's && Sagi's answers @ hand, were your questions resolved?

Also, Nic put the V4 patches in a branch on his tree for 0-day testing
and we had one hit from Dan Carpenter which Sagi addressed with
incremental fix he sent you.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On Mon, Jan 20, 2014, Or Gerlitz <or.gerlitz@gmail.com> wrote:
> On Sun, Jan 19, 2014, sagi grimberg <sagig@mellanox.com> wrote:
>> Thanks Nic,  let me elaborate on this,
[...]
>> Hope this helps,

> Hi Roland, with Nic's && Sagi's answers @ hand, were your questions resolved?

Roland, ping! the signature patches were posted > three months ago. We
deserve a response from the maintainer that goes beyond "I need to
think on that".

Responsiveness was stated by Linus to be the #1 requirement from
kernel maintainers.

Or.

Re: linux rdma 3.14 merge plans

[Roland Dreier]

On Tue, Jan 21, 2014 at 2:00 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Roland, ping! the signature patches were posted > three months ago. We
> deserve a response from the maintainer that goes beyond "I need to
> think on that".
>
> Responsiveness was stated by Linus to be the #1 requirement from
> kernel maintainers.

Or, I'm not sure what response you're after from me.  Linus has also
said that maintainers should say "no" a lot more
(http://lwn.net/Articles/571995/) so maybe you want me to say, "No, I
won't merge this patch set, since it adds a bunch of complexity to
support a feature no one really cares about."  Is that it?  (And yes I
am skeptical about this stuff — I work at an enterprise storage
company and even here it's hard to find anyone who cares about
DIF/DIX, especially offload features that stop it from being
end-to-end)

I'm sure you're not expecting me to say, "Sure, I'll merge it without
understanding the problem it's solving or how it's doing that,"
especially given the your recent history of pushing me to merge stuff
like the IP-RoCE patches back when they broke the userspace ABI.

I'd really rather spend my time on something actually useful like
cleaning up softroce.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On Wed, Jan 22, 2014 at 2:43 AM, Roland Dreier <roland@kernel.org> wrote:
> On Tue, Jan 21, 2014 at 2:00 PM, Or Gerlitz <or.gerlitz@gmail.com> wrote:
>> Roland, ping! the signature patches were posted > three months ago. We
>> deserve a response from the maintainer that goes beyond "I need to
>> think on that".
>>
>> Responsiveness was stated by Linus to be the #1 requirement from
>> kernel maintainers.
>
> Or, I'm not sure what response you're after from me.

Roland, what I am after is a r-e-s-p-o-n-s-e from you, and let it
contain what ever justified and/or unjustified mud as below. We posted
the V0 series on Oct 15 2013 and since that time not a word from you,
except for an "I need to think on that" comment last week after we
nudged million times.

You can't leave us clueless in the air for whole three months without
any concrete or unconcrete comment. There's no way to carry kernel
development like that. I am old enough to hear and face "no" and "wTF
is this" or "yTF you do it this way" etc etc, this happened few times
with e.g with networking patches we sent  and we either improved
things or did them differently or whatever needed to be done.

There's no way on earth to face plain ignoring of your work, and this
is what happens here. I had no way to get your below response except
for going to LKML, why?


> Linus has also said that maintainers should say "no" a lot more
> (http://lwn.net/Articles/571995/) so maybe you want me to say, "No, I
> won't merge this patch set, since it adds a bunch of complexity to
> support a feature no one really cares about."  Is that it?  (And yes I
> am skeptical about this stuff — I work at an enterprise storage
> company and even here it's hard to find anyone who cares about
> DIF/DIX, especially offload features that stop it from being
> end-to-end)
>
> I'm sure you're not expecting me to say, "Sure, I'll merge it without
> understanding the problem it's solving or how it's doing that,"
> especially given the your recent history of pushing me to merge stuff
> like the IP-RoCE patches back when they broke the userspace ABI.
>
> I'd really rather spend my time on something actually useful like
> cleaning up softroce.
>
>  - R.

Re: linux rdma 3.14 merge plans

[Nicholas A. Bellinger]

Roland & Co,

On Tue, 2014-01-21 at 16:43 -0800, Roland Dreier wrote:
> On Tue, Jan 21, 2014 at 2:00 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > Roland, ping! the signature patches were posted > three months ago. We
> > deserve a response from the maintainer that goes beyond "I need to
> > think on that".
> >
> > Responsiveness was stated by Linus to be the #1 requirement from
> > kernel maintainers.
> 
> Or, I'm not sure what response you're after from me.  Linus has also
> said that maintainers should say "no" a lot more
> (http://lwn.net/Articles/571995/) so maybe you want me to say, "No, I
> won't merge this patch set, since it adds a bunch of complexity to
> support a feature no one really cares about."  Is that it? 

The patch set proposed by Sagi + Or is modest in terms of LOC to core IB
code, and includes mostly mlx5 specific driver changes that enables HW
offloads.

> (And yes I
> am skeptical about this stuff — I work at an enterprise storage
> company and even here it's hard to find anyone who cares about
> DIF/DIX, especially offload features that stop it from being
> end-to-end)
> 

My understanding is most HBAs capable of T10 PI offload in DIX PASS +
VERIFY mode are already implementing DIX INSERT + STRIP modes in various
capacities to support legacy environments.

Beyond the DIX INSERT + STRIP case for enterprise storage, the amount of
FC + SAS HBAs that already support T10 PI metadata is substantial.

> I'm sure you're not expecting me to say, "Sure, I'll merge it without
> understanding the problem it's solving or how it's doing that,"
> especially given the your recent history of pushing me to merge stuff
> like the IP-RoCE patches back when they broke the userspace ABI.

With the merge window now upon us, there is a understandable reluctance
to merge new features.  Given the amount of time the series has spent on
the list, it is however a good candidate to consider for an exception.

Short of that, are you planning to accept the series for the next round
once the current merge window closes..?

We'd really like to start enabling fabrics with these types of offloads
for v3.15. 

> I'd really rather spend my time on something actually useful like
> cleaning up softroce.
> 

+1 for softroce + T10 PI support!

--nab

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Nicholas A. Bellinger]

Hi Roland,

On Tue, 2014-01-21 at 23:27 -0800, Nicholas A. Bellinger wrote:
> Roland & Co,
> 
> On Tue, 2014-01-21 at 16:43 -0800, Roland Dreier wrote:
> > On Tue, Jan 21, 2014 at 2:00 PM, Or Gerlitz <or.gerlitz@gmail.com> wrote:
> > > Roland, ping! the signature patches were posted > three months ago. We
> > > deserve a response from the maintainer that goes beyond "I need to
> > > think on that".
> > >
> > > Responsiveness was stated by Linus to be the #1 requirement from
> > > kernel maintainers.
> > 
> > Or, I'm not sure what response you're after from me.  Linus has also
> > said that maintainers should say "no" a lot more
> > (http://lwn.net/Articles/571995/) so maybe you want me to say, "No, I
> > won't merge this patch set, since it adds a bunch of complexity to
> > support a feature no one really cares about."  Is that it? 
> 
> The patch set proposed by Sagi + Or is modest in terms of LOC to core IB
> code, and includes mostly mlx5 specific driver changes that enables HW
> offloads.
> 
> > (And yes I
> > am skeptical about this stuff — I work at an enterprise storage
> > company and even here it's hard to find anyone who cares about
> > DIF/DIX, especially offload features that stop it from being
> > end-to-end)
> > 
> 
> My understanding is most HBAs capable of T10 PI offload in DIX PASS +
> VERIFY mode are already implementing DIX INSERT + STRIP modes in various
> capacities to support legacy environments.
> 
> Beyond the DIX INSERT + STRIP case for enterprise storage, the amount of
> FC + SAS HBAs that already support T10 PI metadata is substantial.
> 
> > I'm sure you're not expecting me to say, "Sure, I'll merge it without
> > understanding the problem it's solving or how it's doing that,"
> > especially given the your recent history of pushing me to merge stuff
> > like the IP-RoCE patches back when they broke the userspace ABI.
> 
> With the merge window now upon us, there is a understandable reluctance
> to merge new features.  Given the amount of time the series has spent on
> the list, it is however a good candidate to consider for an exception.
> 
> Short of that, are you planning to accept the series for the next round
> once the current merge window closes..?
> 
> We'd really like to start enabling fabrics with these types of offloads
> for v3.15. 
> 

Now with the initial DIF backend taraget support in place for v3.14-rc1
code, we'd like to move forward on iser-target related pieces for T10
PI.

Can you give us an estimate of when you'll have some time to give
feedback on the outstanding patches..?

--nab

Re: linux rdma 3.14 merge plans

[Roland Dreier]

On Thu, Feb 6, 2014 at 4:02 PM, Nicholas A. Bellinger
<nab@linux-iscsi.org> wrote:
> Can you give us an estimate of when you'll have some time to give
> feedback on the outstanding patches..?

I hope to get to it in the next few weeks.

 - R.

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On Fri, Feb 7, 2014 at 2:04 AM, Roland Dreier <roland@kernel.org> wrote:
> On Thu, Feb 6, 2014 at 4:02 PM, Nicholas A. Bellinger
> <nab@linux-iscsi.org> wrote:
>> Can you give us an estimate of when you'll have some time to give
>> feedback on the outstanding patches..?
>
> I hope to get to it in the next few weeks.

Hi Roland,

So days, weeks and months are passing by and we still didn't get any
real feedback from you on the patches which now make a complete story:
verbs, driver, target and initiator nor on the detailed response/s
sent to you as replies to the few on the surface quick questions and
doubts you raised. 3.15 is coming soon and there's no reason to miss
it just for that lack of feedback as happened for 3.13 and 3.14 -- are
you picking on this or maybe prefer that Nic will push it upstream
through his tree? all the patches are now on the rdma-dif of the
target-pending tree, please let us know.




>
>  - R.

Re: linux rdma 3.14 merge plans

[Nicholas A. Bellinger]

On Thu, 2014-02-06 at 16:04 -0800, Roland Dreier wrote:
> On Thu, Feb 6, 2014 at 4:02 PM, Nicholas A. Bellinger
> <nab@linux-iscsi.org> wrote:
> > Can you give us an estimate of when you'll have some time to give
> > feedback on the outstanding patches..?
> 
> I hope to get to it in the next few weeks.
> 
>  - R.

Hi Roland,

We'd very much like to move forward with the verbs + mlx5 + LIO target
PI related patches for v3.15 code.

Given the verbs + mlx5 changes have undergone ~5 months of review at
this point, I'd like to go ahead and get them in target-pending/for-next
ASAP to continue making forward progress towards a mainline merge.

I'm happy with the state of the iser-target related changes, and Sagi &
Co are making good progress on the iser-initiator related patches with
Mike.

That all said, do you have an objection wrt taking this bits through
target-pending..?  Given the dependencies involved, that would seem the
most logical path to take.

--nab 

Re: linux rdma 3.14 merge plans

[Roland Dreier]

On Wed, Mar 5, 2014 at 1:54 AM, Nicholas A. Bellinger
<nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org> wrote:
> That all said, do you have an objection wrt taking this bits through
> target-pending..?  Given the dependencies involved, that would seem the
> most logical path to take.

Perhaps not surprisingly, I would prefer to get a chance to review a
major change to the core RDMA midlayer rather than having you merge it
through your tree.  So yes I do object.  Please give me a chance to
review and merge it.  I am working on that this week.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On 05/03/2014 17:18, Roland Dreier wrote:
> On Wed, Mar 5, 2014 at 1:54 AM, Nicholas A. Bellinger
> <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org>  wrote:
>> >That all said, do you have an objection wrt taking this bits through
>> >target-pending..?  Given the dependencies involved, that would seem the
>> >most logical path to take.
> Perhaps not surprisingly, I would prefer to get a chance to review a
> major change to the core RDMA midlayer rather than having you merge it
> through your tree.  So yes I do object.  Please give me a chance to
> review and merge it.  I am working on that this week.

Hi Roland, we're very happy to hear that!!

As you know, we are soon marking whole five months!! (patches posted Oct 
15th/2013) of sitting and waiting to your feedback, lost few upstream 
releases on the way. Let's get this into the works.

Or.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Nicholas A. Bellinger]

On Wed, 2014-03-05 at 07:18 -0800, Roland Dreier wrote:
> On Wed, Mar 5, 2014 at 1:54 AM, Nicholas A. Bellinger
> <nab@linux-iscsi.org> wrote:
> > That all said, do you have an objection wrt taking this bits through
> > target-pending..?  Given the dependencies involved, that would seem the
> > most logical path to take.
> 
> Perhaps not surprisingly, I would prefer to get a chance to review a
> major change to the core RDMA midlayer rather than having you merge it
> through your tree.  So yes I do object.  Please give me a chance to
> review and merge it.  I am working on that this week.
> 

Great.  We'll be looking for a response by the end of the week.

Otherwise if you end up not having time, we'd still like to move forward
for v3.15 given the amount of review the series has already gotten on
the list.

Thank you,

--nab

RE: linux rdma 3.14 merge plans

[Devesh Sharma]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1756 bytes --]

Hi Roland,

Is it okay to send next series of patches even if previous series is not accepted yet in your tree? Off-course I will cut patches on top of previous series of patches.

-Regards
 Devesh

-----Original Message-----
From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma-owner@vger.kernel.org] On Behalf Of Nicholas A. Bellinger
Sent: Thursday, March 06, 2014 12:34 AM
To: Roland Dreier
Cc: Or Gerlitz; Hefty Sean; linux-rdma; Martin K. Petersen; target-devel; Sagi Grimberg; linux-kernel
Subject: Re: linux rdma 3.14 merge plans

On Wed, 2014-03-05 at 07:18 -0800, Roland Dreier wrote:
> On Wed, Mar 5, 2014 at 1:54 AM, Nicholas A. Bellinger 
> <nab@linux-iscsi.org> wrote:
> > That all said, do you have an objection wrt taking this bits through 
> > target-pending..?  Given the dependencies involved, that would seem 
> > the most logical path to take.
> 
> Perhaps not surprisingly, I would prefer to get a chance to review a 
> major change to the core RDMA midlayer rather than having you merge it 
> through your tree.  So yes I do object.  Please give me a chance to 
> review and merge it.  I am working on that this week.
> 

Great.  We'll be looking for a response by the end of the week.

Otherwise if you end up not having time, we'd still like to move forward for v3.15 given the amount of review the series has already gotten on the list.

Thank you,

--nab



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

Re: linux rdma 3.14 merge plans

[Roland Dreier]

Sure, no problem.

Do you have a git tree with the latest versions of all the changes you
want for 3.15 in a branch?  That would be helpful as I catch up on
applying things, so that I don't miss anything.

If you don't have one, taking a little time to set one up on github or
wherever would be nice.  You can base your set of changes on Linus's
latest tree.

Thanks!
  Roland

On Thu, Mar 6, 2014 at 9:07 PM, Devesh Sharma <Devesh.Sharma-laKkSmNT4hbQT0dZR+AlfA@public.gmane.org> wrote:
> Hi Roland,
>
> Is it okay to send next series of patches even if previous series is not accepted yet in your tree? Off-course I will cut patches on top of previous series of patches.
>
> -Regards
>  Devesh
>
> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Nicholas A. Bellinger
> Sent: Thursday, March 06, 2014 12:34 AM
> To: Roland Dreier
> Cc: Or Gerlitz; Hefty Sean; linux-rdma; Martin K. Petersen; target-devel; Sagi Grimberg; linux-kernel
> Subject: Re: linux rdma 3.14 merge plans
>
> On Wed, 2014-03-05 at 07:18 -0800, Roland Dreier wrote:
>> On Wed, Mar 5, 2014 at 1:54 AM, Nicholas A. Bellinger
>> <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org> wrote:
>> > That all said, do you have an objection wrt taking this bits through
>> > target-pending..?  Given the dependencies involved, that would seem
>> > the most logical path to take.
>>
>> Perhaps not surprisingly, I would prefer to get a chance to review a
>> major change to the core RDMA midlayer rather than having you merge it
>> through your tree.  So yes I do object.  Please give me a chance to
>> review and merge it.  I am working on that this week.
>>
>
> Great.  We'll be looking for a response by the end of the week.
>
> Otherwise if you end up not having time, we'd still like to move forward for v3.15 given the amount of review the series has already gotten on the list.
>
> Thank you,
>
> --nab
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: linux rdma 3.14 merge plans

[Devesh Sharma]

Thanks Roland,
My response is inline below.

-Regards
 Devesh
-----Original Message-----
From: roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org [mailto:roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org] On Behalf Of Roland Dreier
Sent: Saturday, March 08, 2014 1:02 AM
To: Devesh Sharma
Cc: Nicholas A. Bellinger; Or Gerlitz; Hefty Sean; linux-rdma; Martin K. Petersen; target-devel; Sagi Grimberg; linux-kernel
Subject: Re: linux rdma 3.14 merge plans

Sure, no problem.

Do you have a git tree with the latest versions of all the changes you want for 3.15 in a branch?  That would be helpful as I catch up on applying things, so that I don't miss anything.
[DS]: Yes, I do have a cloned copy of your git tree migrated to for-next branch, with all the patches we have submitted to linux-rdma (patch series: [PATCH for-next 00/17] ocrdma driver code sync-up). However, it is on my local server.
In case of urgency (and if OFED EWG permits me) I can set it up on benay dot openfabrics dot org, for now and give a link to pull.

If you don't have one, taking a little time to set one up on github or wherever would be nice.  You can base your set of changes on Linus's latest tree.
[DS]: Currently we do not have such tree. I will have to discuss within my organization on how to go about this request. 

Thanks!
  Roland

On Thu, Mar 6, 2014 at 9:07 PM, Devesh Sharma <Devesh.Sharma-laKkSmNT4hbQT0dZR+AlfA@public.gmane.org> wrote:
> Hi Roland,
>
> Is it okay to send next series of patches even if previous series is not accepted yet in your tree? Off-course I will cut patches on top of previous series of patches.
>
> -Regards
>  Devesh
>
> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org 
> [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Nicholas A. 
> Bellinger
> Sent: Thursday, March 06, 2014 12:34 AM
> To: Roland Dreier
> Cc: Or Gerlitz; Hefty Sean; linux-rdma; Martin K. Petersen; 
> target-devel; Sagi Grimberg; linux-kernel
> Subject: Re: linux rdma 3.14 merge plans
>
> On Wed, 2014-03-05 at 07:18 -0800, Roland Dreier wrote:
>> On Wed, Mar 5, 2014 at 1:54 AM, Nicholas A. Bellinger 
>> <nab-IzHhD5pYlfBP7FQvKIMDCQ@public.gmane.org> wrote:
>> > That all said, do you have an objection wrt taking this bits 
>> > through target-pending..?  Given the dependencies involved, that 
>> > would seem the most logical path to take.
>>
>> Perhaps not surprisingly, I would prefer to get a chance to review a 
>> major change to the core RDMA midlayer rather than having you merge 
>> it through your tree.  So yes I do object.  Please give me a chance 
>> to review and merge it.  I am working on that this week.
>>
>
> Great.  We'll be looking for a response by the end of the week.
>
> Otherwise if you end up not having time, we'd still like to move forward for v3.15 given the amount of review the series has already gotten on the list.
>
> Thank you,
>
> --nab
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" 
> in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On Wed, Jan 22, 2014 at 3:43 AM, Roland Dreier <roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
[...]
> I'd really rather spend my time on something actually useful like cleaning up softroce.

Hi Roland,

Were you referring to the code that was posted here few years ago, or
you're working on something new? two comments

1. make sure it uses IP based GIDs...

2. I think we don't want to have a 3rd SW implementation in the kernel
of IB transport, (1=ipath, 2=qib) right? so the relevant
qib code could/should be refactored  into a module which implements
the IB transport.

thoughts?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Sagi Grimberg]

On 1/22/2014 2:43 AM, Roland Dreier wrote:
> On Tue, Jan 21, 2014 at 2:00 PM, Or Gerlitz <or.gerlitz@gmail.com> wrote:
>> Roland, ping! the signature patches were posted > three months ago. We
>> deserve a response from the maintainer that goes beyond "I need to
>> think on that".
>>
>> Responsiveness was stated by Linus to be the #1 requirement from
>> kernel maintainers.

Hi Roland, I'll try to respond here.
removing LKML and adding Linux-scsi.

> Or, I'm not sure what response you're after from me.  Linus has also
> said that maintainers should say "no" a lot more
> (http://lwn.net/Articles/571995/) so maybe you want me to say, "No, I
> won't merge this patch set, since it adds a bunch of complexity to
> support a feature no one really cares about."

1. I disagree about no-one cares about DIF/DIX. We are witnessing growing
interests in this especially for RDMA.
2. We put a lot of efforts to avoid complexity here and plug-in as 
simple as possible.
Application that will choose to use DIF will implement only 3 steps:
a. allocate signature enabled MR.
b. register signature enabled MR with DIF attributes (via post_send) and 
then do RDMA.
c. check MR status after transaction is completed (_lightweight_ verb 
that can be called from interrupt context).

>    Is that it?  (And yes I
> am skeptical about this stuff — I work at an enterprise storage
> company and even here it's hard to find anyone who cares about
> DIF/DIX, especially offload features that stop it from being
> end-to-end)

1. RDMA verbs are _NOT_ stopping DIF from being end-to-end.
OS (or SCSI in our specific case) passes LLD 2 scatterlists: data 
{block1, block2, block3,...}, and protection {DIF1, DIF2, DIF3}.
LLD is required to verify the data integrity (block guards) and to 
interleave over the wire {block1, DIF1, block2, DIF2....}.
You must support that in HW, you rather iSER/SRP will use giant copy's 
to interleave by itself? or in case OS asked LLD
to INSERT DIF iSER/SRP will compute CRC for each data-block? RDMA 
storage ULPs are transports - they should have no business with
data processing.

2. HW DIF offload also gives you protection across the PCI. the 
data-validation is done (hopefully offloaded) also
when data+protection are written to the back-end device. end-to-end is 
preserved.

3. SAS & FC have T10-PI offload. This is just adding RDMA into the game.
With this set of verbs iSER, SRP, FCoE Initiators and targets will be able
to support T10-PI.

> I'm sure you're not expecting me to say, "Sure, I'll merge it without
> understanding the problem it's solving

Problem: T10-PI offload support for RDMA based initiators. Supporting 
end-to-end data integrity
while sustaining high RDMA performance.

>   or how it's doing that,"

How it's doing that:
- We introduce a new type of memory region that posses protection 
attributes suited for data integrity offload.
- We Introduce a new fast registration method that can bind all the 
relevant info for verify/generate of protection information:
   * describe if/how to interleave data with protection.
   * describe what method of data integrity is used (DIF type X, CRC, 
XOR...) and the seeds that HW should start calculation from.
   * describe how to verify the data.
- We Introduce a new lightweight check of the data-integrity status to 
check if there were any integrity errors and get information on them.

Note: We made MR allocation routine generic enough to lay a framework to 
unite all MR allocation
methods (get_dma_mr, alloc_fast_reg_mr, reg_phys, reg_user_mr, fmrs, and 
probably more in the future...).
We defined ib_create_mr that can actually get mr_init_attr which can be 
easily extended as opposed to the specific calls exists today.
So I would say this even reduces complexity.

Hope this helps,

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On Wed, Jan 22, 2014, Sagi Grimberg <sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> On 1/22/2014 2:43 AM, Roland Dreier wrote:
>> On Tue, Jan 21, 2014, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

>>> Roland, ping! the signature patches were posted > three months ago. We
>>> deserve a response from the maintainer that goes beyond "I need to
>>> think on that". Responsiveness was stated by Linus to be the #1 requirement
>>> from kernel maintainers.

> Hi Roland, I'll try to respond here. removing LKML and adding Linux-scsi.

Sorry, it seems we will not getting responses unless coping LKML, so
lets do that again -- Roland, below is the detailed response Sagi
wrote you following your "adds a bunch of complexity to support a
feature no one really cares about" comment, can we get you to respond
on that?

>> Or, I'm not sure what response you're after from me.  Linus has also
>> said that maintainers should say "no" a lot more
>> (http://lwn.net/Articles/571995/) so maybe you want me to say, "No, I
>> won't merge this patch set, since it adds a bunch of complexity to
>> support a feature no one really cares about."

> 1. I disagree about no-one cares about DIF/DIX. We are witnessing growing
> interests in this especially for RDMA.
> 2. We put a lot of efforts to avoid complexity here and plug-in as simple as
> possible.
> Application that will choose to use DIF will implement only 3 steps:
> a. allocate signature enabled MR.
> b. register signature enabled MR with DIF attributes (via post_send) and
> then do RDMA.
> c. check MR status after transaction is completed (_lightweight_ verb that
> can be called from interrupt context).

>> Is that it?  (And yes I am skeptical about this stuff -- I work at an enterprise
>> storage company and even here it's hard to find anyone who cares about
>> DIF/DIX, especially offload features that stop it from being end-to-end)


> 1. RDMA verbs are _NOT_ stopping DIF from being end-to-end.
> OS (or SCSI in our specific case) passes LLD 2 scatterlists: data {block1,
> block2, block3,...}, and protection {DIF1, DIF2, DIF3}.
> LLD is required to verify the data integrity (block guards) and to
> interleave over the wire {block1, DIF1, block2, DIF2....}.
> You must support that in HW, you rather iSER/SRP will use giant copy's to
> interleave by itself? or in case OS asked LLD
> to INSERT DIF iSER/SRP will compute CRC for each data-block? RDMA storage
> ULPs are transports - they should have no business with
> data processing.

> 2. HW DIF offload also gives you protection across the PCI. the
> data-validation is done (hopefully offloaded) also
> when data+protection are written to the back-end device. end-to-end is
> preserved.

> 3. SAS & FC have T10-PI offload. This is just adding RDMA into the game.
> With this set of verbs iSER, SRP, FCoE Initiators and targets will be able
> to support T10-PI.


>> I'm sure you're not expecting me to say, "Sure, I'll merge it without
>> understanding the problem it's solving

> Problem: T10-PI offload support for RDMA based initiators. Supporting
> end-to-end data integrity while sustaining high RDMA performance.


>>   or how it's doing that,"

> How it's doing that:
> - We introduce a new type of memory region that posses protection attributes
> suited for data integrity offload.
> - We Introduce a new fast registration method that can bind all the relevant
> info for verify/generate of protection information:
>   * describe if/how to interleave data with protection.
>   * describe what method of data integrity is used (DIF type X, CRC, XOR...)
> and the seeds that HW should start calculation from.
>   * describe how to verify the data.
> - We Introduce a new lightweight check of the data-integrity status to check
> if there were any integrity errors and get information on them.

> Note: We made MR allocation routine generic enough to lay a framework to
> unite all MR allocation
> methods (get_dma_mr, alloc_fast_reg_mr, reg_phys, reg_user_mr, fmrs, and
> probably more in the future...).
> We defined ib_create_mr that can actually get mr_init_attr which can be
> easily extended as opposed to the specific calls exists today.
> So I would say this even reduces complexity.

> Hope this helps,
> Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Bart Van Assche]

On 01/28/14 22:02, Or Gerlitz wrote:
>>>> Roland, ping! the signature patches were posted > three months ago.

I have a question about RDMA and T10-DIF support. The Linux block layer,
the Linux SCSI core, Linux filesystems and block drivers all support
discontiguous buffers. These are buffers that cannot be mapped onto a
single contiguous range of virtual memory addresses. I think the RDMA
FMR and FR memory registration mechanisms only support mapping a range
of addresses that can be mapped onto a contiguous region of virtual
addresses. This means that either multiple RDMA memory regions are
needed to map a discontiguous buffer or that the buffer contents has to
be copied before associating it with a memory region. What I understood
from the mlx5 T10-DIF patch series is that the API introduced in that
patch series is such that only a single memory region per data transfer
operation is supported. This made me wonder how to support T10-DIF for
discontiguous buffers. Would one of the options below perhaps be an
appropriate solution ?
- Add a flag in struct request_queue that tells the block layer to use a
  bounce buffer (data copying) for I/O requests with a discontiguous
  data buffer. The block layer already supports copying data buffers for
  which e.g. DMA alignment requirements are not met. Rework patch
  "IB/iser: Introduce fast memory registration model" (commit
  5587856c9659ac2d6ab201141aa8a5c2ff3be4cd) such that it takes advantage
  of that new flag. See also blk_rq_map_user_iov() for an example of
  how the block layer decides when copying a data buffer is necessary.
- Modify the patches that add T10-DIF such that discontiguous buffers
  are supported.

Sorry that I had not realized this before.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Sagi Grimberg]

On 1/29/2014 4:13 PM, Bart Van Assche wrote:
> On 01/28/14 22:02, Or Gerlitz wrote:
>>>>> Roland, ping! the signature patches were posted > three months ago.
> I have a question about RDMA and T10-DIF support. The Linux block layer,
> the Linux SCSI core, Linux filesystems and block drivers all support
> discontiguous buffers. These are buffers that cannot be mapped onto a
> single contiguous range of virtual memory addresses. I think the RDMA
> FMR and FR memory registration mechanisms only support mapping a range
> of addresses that can be mapped onto a contiguous region of virtual
> addresses. This means that either multiple RDMA memory regions are
> needed to map a discontiguous buffer or that the buffer contents has to
> be copied before associating it with a memory region. What I understood
> from the mlx5 T10-DIF patch series is that the API introduced in that
> patch series is such that only a single memory region per data transfer
> operation is supported. This made me wonder how to support T10-DIF for
> discontiguous buffers.

Hey Bart, Thanks for the observation,

You are correct, the T10-PI RDMA offload API requires a single memory 
region that describes the data
and a single memory region that describes the protection information.

Non-contiguous buffers are a well known problem/challenge for RDMA 
transports, each handles them
differently: iSER uses bounce buffers, SRP registers multiple contiguous 
regions. The T10-PI offload API does
not impose any other constraint on this situation, each ULP will act the 
same: iSER will copy to contiguous
bounce buffers and SRP will register multiple "Signature" memory regions 
(each for a contiguous chunk of data)
and send the rkeys to the target.

> Would one of the options below perhaps be an
> appropriate solution ?
> - Add a flag in struct request_queue that tells the block layer to use a
>    bounce buffer (data copying) for I/O requests with a discontiguous
>    data buffer. The block layer already supports copying data buffers for
>    which e.g. DMA alignment requirements are not met. Rework patch
>    "IB/iser: Introduce fast memory registration model" (commit
>    5587856c9659ac2d6ab201141aa8a5c2ff3be4cd) such that it takes advantage
>    of that new flag. See also blk_rq_map_user_iov() for an example of
>    how the block layer decides when copying a data buffer is necessary.

Didn't understand why should it matter where the copy is done (iser/block)?

> - Modify the patches that add T10-DIF such that discontiguous buffers
>    are supported.

I thought about adding some kind of logic to T10-PI RDMA API to try and 
solve the alignment
issue at the same time, but the fact is that the two are unrelated, and 
I figured it is not
such a good idea. Non-contiguous memory registration is separate problem 
(hint: under
development) and should be addressed separately, T10-PI RDMA offload 
should play well
with it.

> Sorry that I had not realized this before.
>
> Bart.
>

Hops this helps,

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On 29/01/2014 17:06, Sagi Grimberg wrote:
> Non-contiguous buffers are a well known problem/challenge for RDMA 
> transports, each handles them differently: iSER uses bounce buffers, 
> SRP registers multiple contiguous regions.

To be precise, we do it only on very special cases, normally the payload 
**each** SCSI command of size > PAGE_SIZE is scattered between random 
set of pages allocated by the kernel buffer cache and this set is mapped 
through fast-reg or fmr mechanism to one RDMA virtual address.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Bart Van Assche]

On 01/29/14 16:06, Sagi Grimberg wrote:
> On 1/29/2014 4:13 PM, Bart Van Assche wrote:
>> Would one of the options below perhaps be an
>> appropriate solution ?
>> - Add a flag in struct request_queue that tells the block layer to use a
>>    bounce buffer (data copying) for I/O requests with a discontiguous
>>    data buffer. The block layer already supports copying data buffers for
>>    which e.g. DMA alignment requirements are not met. Rework patch
>>    "IB/iser: Introduce fast memory registration model" (commit
>>    5587856c9659ac2d6ab201141aa8a5c2ff3be4cd) such that it takes advantage
>>    of that new flag. See also blk_rq_map_user_iov() for an example of
>>    how the block layer decides when copying a data buffer is necessary.
> 
> Didn't understand why should it matter where the copy is done (iser/block)?

In the Linux kernel community it is considered important to avoid code
duplication. Hence the proposal to keep code that copies data buffers in
the block layer core and to avoid that such functionality has to be
reimplemented in every block driver or SCSI LLD.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On 29/01/2014 19:56, Bart Van Assche wrote:
> On 01/29/14 16:06, Sagi Grimberg wrote:
>>
>> Didn't understand why should it matter where the copy is done (iser/block)?
> In the Linux kernel community it is considered important to avoid code duplication. Hence the proposal to keep code that copies data buffers in the block layer core and to avoid that such functionality has to be reimplemented in every block driver or SCSI LLD.

Hi Bart,

Thanks for narrowing this down, I see your point, however the solution I 
propose if to remove this copy altogether... for those rare cases where 
fast-registration can't be done -- in SRP I think the code goes to 
indirect mode under which there no copy (right?). As for iSER, we will 
add to our TODO enhancing the code to avoid using bounce-buffer, few 
designs might be possible, one way would be to create set of 
FMRS/Fast-reg element which are capable to map chunks which are < 
PAGE_SIZE, e.g in the order of single block, and hence can map what ever 
arbitrary set of blocks provided by the upper layer.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Bart Van Assche]

On 01/30/14 09:19, Or Gerlitz wrote:
> On 29/01/2014 19:56, Bart Van Assche wrote:
>> On 01/29/14 16:06, Sagi Grimberg wrote:
>>> Didn't understand why should it matter where the copy is done
>>> (iser/block)?
>> In the Linux kernel community it is considered important to avoid code
>> duplication. Hence the proposal to keep code that copies data buffers
>> in the block layer core and to avoid that such functionality has to be
>> reimplemented in every block driver or SCSI LLD.
> 
> Thanks for narrowing this down, I see your point, however the solution I
> propose if to remove this copy altogether... for those rare cases where
> fast-registration can't be done -- in SRP I think the code goes to
> indirect mode under which there no copy (right?). As for iSER, we will
> add to our TODO enhancing the code to avoid using bounce-buffer, few
> designs might be possible, one way would be to create set of
> FMRS/Fast-reg element which are capable to map chunks which are <
> PAGE_SIZE, e.g in the order of single block, and hence can map what ever
> arbitrary set of blocks provided by the upper layer.

I'm not sure how important this issue is on x86 systems since the page
size on these systems is 4 KB and since many applications use an I/O
size >= 4 KB. So if the I/O buffers are aligned on page boundaries the
buffer memory can be registered as a single memory region without having
to copy any data. However, on PPC systems, which have a page size of 64
KB, if e.g. a database issues a readv() or writev() call or if an I/O
scheduler coalesces several small I/O requests into a single I/O request
the data buffer of these I/O requests is discontiguous. I think it is
important that data copying can be avoided for such I/O requests.

By the way, is it documented somewhere what the alignment requirements
are for FMR and FR requests ? As far as I can see the mlx4 driver
requires page alignment for FMR requests but not for FR requests (see
e.g. mlx4_check_fmr()). I haven't found a page aligment requirement in
e.g. ehca_fmr_check_page_list(). Does this mean that alignment
restrictions for FMR are HCA-dependent ? Would it be a good idea to add
an attribute in e.g. ib_device_attr that allows ULP drivers to retrieve
FMR alignment requirements in a generic way ?

Thanks,

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Or Gerlitz]

On 30/01/2014 12:07, Bart Van Assche wrote:
> On 01/30/14 09:19, Or Gerlitz wrote:
>> Thanks for narrowing this down, I see your point, however the solution I
>> propose if to remove this copy altogether... for those rare cases where
>> fast-registration can't be done -- in SRP I think the code goes to
>> indirect mode under which there no copy (right?). As for iSER, we will
>> add to our TODO enhancing the code to avoid using bounce-buffer, few
>> designs might be possible, one way would be to create set of
>> FMRS/Fast-reg element which are capable to map chunks which are <
>> PAGE_SIZE, e.g in the order of single block, and hence can map what ever
>> arbitrary set of blocks provided by the upper layer.
> I'm not sure how important this issue is on x86 systems since the page
> size on these systems is 4 KB and since many applications use an I/O
> size >= 4 KB. So if the I/O buffers are aligned on page boundaries the
> buffer memory can be registered as a single memory region without having
> to copy any data. However, on PPC systems, which have a page size of 64
> KB, if e.g. a database issues a readv() or writev() call or if an I/O
> scheduler coalesces several small I/O requests into a single I/O request
> the data buffer of these I/O requests is discontiguous. I think it is
> important that data copying can be avoided for such I/O requests.

We know that... and hence iser uses fast registration in chucks of 4KB 
even when the page size is 64KB

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: linux rdma 3.14 merge plans

[Bart Van Assche]

On 01/08/14 01:51, Roland Dreier wrote:
> On Tue, Jan 7, 2014 at 1:02 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Currently there is single patch for 3.14 on your for-next branch, the
>> usnic driver. With 3.13 being on rc7 and likely to be released next
>> week, are you planning any other merges for 3.14? we have patches
>> waiting for weeks and months without any comment from you.
> 
> I am definitely planning on merging the new IBoE IP addressing stuff,
> since we seem to have solved the ABI issues.
> 
> The UD flow steering patches seem good and I will take a closer look soon.
> 
> And there are quite a few usnic patches still to pick up.
> 
> I'm confident that will all make it.
> 
> 
> The data integrity stuff I'm not so sure about.  Sean raised some I
> think legitimate questions about whether all this should be added to
> the verbs API and I want to see more discussion or at least have a
> deep think about this myself before comitting.

Hi Roland,

In his role as SRP maintainer Dave Dillow had acked a few SRP initiator
patches three weeks ago (see also
http://thread.gmane.org/gmane.linux.scsi/86532/focus=87013). As far as I
can see these patches are not yet in your for-next branch ? Do you see
any chance to get these included in kernel 3.14 ?

Thanks,

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html