Re: Topic: Remote DMA network technologies

Re: Topic: Remote DMA network technologies

[Gerrit Huizenga]

[ Can we start updating the Subject line occasionally when we have
  a specific topic, like above.  And no, I'm not top-posting, I'm
  commenting on the Subject:.  ;-)  --gerrit ]

On Mon, 28 Mar 2005 11:45:19 PST, Roland Dreier wrote:
> Let me slightly hijack this thread to throw out another topic that I
> think is worth talking about at the kernel summit: handling remote DMA
> (RDMA) network technologies.
> 
> As some of you might know, I'm one of the main authors of the
> InfiniBand support in the kernel, and I think we have things fairly
> well in hand there, although handling direct userspace access to RDMA
> capabilities may raise some issues worth talking about.
> 
> However, there is also RDMA-over-TCP hardware beginning to be used,
> based on the specs from the IETF rddp working group and the RDMA
> Consortium.  I would hope that we can abstract out the common pieces
> for InfiniBand and RDMA NIC (RNIC) support and morph
> drivers/infiniband into a more general drivers/rdma.
> 
> This is not _that_ offtopic, since RDMA NICs provide another way of
> handling OOM for iSCSI.  By having the NIC handle the network
> transport through something like iSER, you avoid a lot of the issues
> in this thread.  Having to reconnect to a target while OOM is still a
> problem, but it seems no worse in principal than the issues with a
> dump FC card that needs the host driver to handling fabric login.
> 
> I know that in the InfiniBand world, people have been able to run
> stress tests of storage over SCSI RDMA Protocol (SRP) with very heavy
> swapping going on and no deadlocks.  SRP is in effect network storage
> with the transport handled by the IB hardware.
> 
> However there are some sticky points that I would be interested in
> discussing.  For example, the IETF rddp drafts envisage what they call
> a "dual stack" model: TCP connections are set up by the usual network
> stack and run for a while in "streaming" mode until the application is
> ready to start using RDMA.  At that point there is an "MPA"
> negotiation and then the socket is handed over to the RNIC.  Clearly
> moving the state from the kernel's stack to the RNIC is not trivial.
> 
> Other developers who have more direct experience with RNIC hardware or
> perhaps just strong opinions may have other things in this area that
> they'd like to talk about.
> 
> Thanks,
>   Roland
> 
> 
> 

Re: Topic: Remote DMA network technologies

[Roland Dreier]

    Gerrit> [ Can we start updating the Subject line occasionally when
    Gerrit> we have a specific topic, like above.  And no, I'm not
    Gerrit> top-posting, I'm commenting on the Subject:.  ;-) --gerrit ]

Sorry -- your point is well taken, but everyone else was having so
much fun discussing iSCSI specifics under the old Subject line that I
just went with the flow ;)

 - Roland

RE: [Ksummit-2005-discuss] Summary of 2005 Kernel Summit ProposedTopics

[Asgeir Eiriksson]

> -----Original Message-----
> From: netdev-bounce@oss.sgi.com [mailto:netdev-bounce@oss.sgi.com] On
> Behalf Of Dmitry Yusupov
> Sent: Monday, March 28, 2005 12:49 PM
> To: open-iscsi@googlegroups.com
> Cc: David S. Miller; mpm@selenic.com; andrea@suse.de;
> michaelc@cs.wisc.edu; James.Bottomley@HansenPartnership.com;
ksummit-2005-
> discuss@thunk.org; netdev@oss.sgi.com
> Subject: Re: [Ksummit-2005-discuss] Summary of 2005 Kernel Summit
> ProposedTopics
> 
> Basically, HW offloading all kind of is a different subject.
> Yes, iSER/RDMA/RNIC will help to avoid bunch of problems but at the
same
> time will add bunch of new problems. OOM/deadlock problem we are
> discussing is a software, *not* hardware related.
> 
> If you have plans to start new project such as SoftRDMA than yes. lets
> discuss it since set of problems will be similar to what we've got
with
> software iSCSI Initiators.
> 
> I'm not a believer in any HW state-full protocol offloading
technologies
> and that was one of my motivations to initiate Open-iSCSI project to
> prove that performance is not an issue anymore. And we succeeded, by
> showing comparable to iSCSI HW Initiator's numbers.
> 

Dmitry

Care to be more specific about the performance you achieved?

You might want to contrast your numbers to veritest verified numbers of
800+ MBps and 600+KOPS achieved by Chelsio HBA with stateful offload
using either 1500B or 9KB MTU (for full detail see Veritest report at
http://www.chelsio.com/technology/Chelsio10GbE_iSCSI_report.pdf)

'Asgeir

> Though, for me, RDMA over TCP is an interesting topic from software
> implementation point of view. I was thinking about organizing new
> project. If someone knows that related work is already started - let
me
> know since I might be interested to help.
> 
> Dmitry
> 
> On Mon, 2005-03-28 at 11:45 -0800, Roland Dreier wrote:
> > Let me slightly hijack this thread to throw out another topic that I
> > think is worth talking about at the kernel summit: handling remote
DMA
> > (RDMA) network technologies.
> >
> > As some of you might know, I'm one of the main authors of the
> > InfiniBand support in the kernel, and I think we have things fairly
> > well in hand there, although handling direct userspace access to
RDMA
> > capabilities may raise some issues worth talking about.
> >
> > However, there is also RDMA-over-TCP hardware beginning to be used,
> > based on the specs from the IETF rddp working group and the RDMA
> > Consortium.  I would hope that we can abstract out the common pieces
> > for InfiniBand and RDMA NIC (RNIC) support and morph
> > drivers/infiniband into a more general drivers/rdma.
> >
> > This is not _that_ offtopic, since RDMA NICs provide another way of
> > handling OOM for iSCSI.  By having the NIC handle the network
> > transport through something like iSER, you avoid a lot of the issues
> > in this thread.  Having to reconnect to a target while OOM is still
a
> > problem, but it seems no worse in principal than the issues with a
> > dump FC card that needs the host driver to handling fabric login.
> >
> > I know that in the InfiniBand world, people have been able to run
> > stress tests of storage over SCSI RDMA Protocol (SRP) with very
heavy
> > swapping going on and no deadlocks.  SRP is in effect network
storage
> > with the transport handled by the IB hardware.
> >
> > However there are some sticky points that I would be interested in
> > discussing.  For example, the IETF rddp drafts envisage what they
call
> > a "dual stack" model: TCP connections are set up by the usual
network
> > stack and run for a while in "streaming" mode until the application
is
> > ready to start using RDMA.  At that point there is an "MPA"
> > negotiation and then the socket is handed over to the RNIC.  Clearly
> > moving the state from the kernel's stack to the RNIC is not trivial.
> >
> > Other developers who have more direct experience with RNIC hardware
or
> > perhaps just strong opinions may have other things in this area that
> > they'd like to talk about.
> >
> > Thanks,
> >   Roland
> 

Re: Topic: Remote DMA network technologies

[Dmitry Yusupov]

On Mon, 2005-03-28 at 16:44 -0800, Asgeir Eiriksson wrote:
> 
> 
> > -----Original Message-----
> > From: netdev-bounce@oss.sgi.com [mailto:netdev-bounce@oss.sgi.com] On
> > Behalf Of Dmitry Yusupov
> > Sent: Monday, March 28, 2005 12:49 PM
> > To: open-iscsi@googlegroups.com
> > Cc: David S. Miller; mpm@selenic.com; andrea@suse.de;
> > michaelc@cs.wisc.edu; James.Bottomley@HansenPartnership.com;
> ksummit-2005-
> > discuss@thunk.org; netdev@oss.sgi.com
> > Subject: Re: [Ksummit-2005-discuss] Summary of 2005 Kernel Summit
> > ProposedTopics
> > 
> > Basically, HW offloading all kind of is a different subject.
> > Yes, iSER/RDMA/RNIC will help to avoid bunch of problems but at the
> same
> > time will add bunch of new problems. OOM/deadlock problem we are
> > discussing is a software, *not* hardware related.
> > 
> > If you have plans to start new project such as SoftRDMA than yes. lets
> > discuss it since set of problems will be similar to what we've got
> with
> > software iSCSI Initiators.
> > 
> > I'm not a believer in any HW state-full protocol offloading
> technologies
> > and that was one of my motivations to initiate Open-iSCSI project to
> > prove that performance is not an issue anymore. And we succeeded, by
> > showing comparable to iSCSI HW Initiator's numbers.
> > 
> 
> Dmitry
> 
> Care to be more specific about the performance you achieved?
> 
> You might want to contrast your numbers to veritest verified numbers of
> 800+ MBps and 600+KOPS achieved by Chelsio HBA with stateful offload
> using either 1500B or 9KB MTU (for full detail see Veritest report at
> http://www.chelsio.com/technology/Chelsio10GbE_iSCSI_report.pdf)
> 
> 'Asgeir

I changed the subject, btw.

In your paper. Its 28 M$ Initiator against one iSCSI Target on top of
Chelsio HBA. Now prove me that Chelsio HBA as an Initiator can do
somewhat close to 100K IOPS with regular Linux SCSI-MidLayer...

Regards.
Dmitry

RE: Topic: Remote DMA network technologies

[Leonid Grossman]

> -----Original Message-----
> From: netdev-bounce@oss.sgi.com [mailto:netdev-bounce@oss.sgi.com] On
> Behalf Of Dmitry Yusupov
> Sent: Monday, March 28, 2005 4:55 PM
> To: open-iscsi@googlegroups.com
> Cc: David S. Miller; mpm@selenic.com; andrea@suse.de;
> michaelc@cs.wisc.edu; James.Bottomley@HansenPartnership.com; ksummit-2005-
> discuss@thunk.org; netdev@oss.sgi.com
> Subject: Re: Topic: Remote DMA network technologies
> 
> On Mon, 2005-03-28 at 16:44 -0800, Asgeir Eiriksson wrote:
> > Care to be more specific about the performance you achieved?
> >
> > You might want to contrast your numbers to veritest verified numbers of
> > 800+ MBps and 600+KOPS achieved by Chelsio HBA with stateful offload
> > using either 1500B or 9KB MTU (for full detail see Veritest report at
> > http://www.chelsio.com/technology/Chelsio10GbE_iSCSI_report.pdf)
> >
> > 'Asgeir

FYI, 
http://biz.yahoo.com/prnews/050329/cgtu006.html?.v=4

The actual whitepaper with performance results is on our site. In short, it
shows that it's fine to run sw iSCSI over 10GbE NIC with Linux-supported
stateless offloads (receive side could use more of these though :-)), on a
generic Opteron box. 
  
Leonid

> 
> I changed the subject, btw.
> 
> In your paper. Its 28 M$ Initiator against one iSCSI Target on top of
> Chelsio HBA. Now prove me that Chelsio HBA as an Initiator can do
> somewhat close to 100K IOPS with regular Linux SCSI-MidLayer...
> 
> Regards.
> Dmitry
> 
>