Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

[Or Gerlitz]

Liran Liss wrote:
> >> all the rdmaoe materials saying the lossless traffic class is a 
> must,  are you saying that this works well also  >> without it? then 
> why from  architect point of view you have posed this requirement?
>
> lossless traffic can be achieved today using global pause, for 
> example.  PFC is still important; we will submit initial patches that 
> support it next wee
Liran, I would say that OTOH global pause isn't the way to go and OTHO 
IB RC functions quite bad when many packets are lost. As such RDMAoE 
without PFC and mapping priorities into TCs (the Ethernet VLs) isn't 
really for production, for any non trivial environment involving more 
then one hop. Also, this email is from one month ago, any news on the 
patches?

Yevgeny, I took a look, and there are patches to support pfc for the 
mlx4_en driver, but they were never submitted upstream, which means that 
even if rdmaoe goes upstream, mainline users will not be able even to 
really test it. Also,  the pfc in these patches configuration seems to 
be done with sysfs and not through the Netlink APIs defined in 
include/net/dcbnl.c, did you had any specific reason not to integrate 
with the mainline method of pfc/tc configuration?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

[Roland Dreier]

 > Liran, I would say that OTOH global pause isn't the way to go and OTHO
 > IB RC functions quite bad when many packets are lost. As such RDMAoE
 > without PFC and mapping priorities into TCs (the Ethernet VLs) isn't
 > really for production, for any non trivial environment involving more
 > then one hop. Also, this email is from one month ago, any news on the
 > patches?

I agree that implementing DCB is important for IBoE, but why do you say
that a classical ethernet fabric with global pause isn't usable?  That
should be roughly equivalent to an IB fabric that uses only a single VL,
which is the case for many production IB fabrics.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

[Or Gerlitz]

Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:

> I agree that implementing DCB is important for IBoE, but why do you say
> that a classical ethernet fabric with global pause isn't usable?  That
> should be roughly equivalent to an IB fabric that uses only a single VL,
> which is the case for many production IB fabrics.

To start with, no matter how many data VLs are used (e.g one), all the
crucial management traffic (SMPs) go on VL15 which is on the one hand
lossy and on the other hand not subject to congestion when other VLs
are. Now how would you manage your Cisco switch --remotely-- on a
globally paused fabric when some multicast receiver hasn't had its
breakfast and now slows the sender while filling the queues throughout
the congestion tree where this switch is part of?

To continue with, lossless is good, but to make your cluster usable
under congestion, you need congestion control, that is QCN, which is
designed/optimized to the case of multiple TCs.

Also, IBoE can potentially find its way to much more complex
environments than IB has, specifically, to clusters whose hosts are
acting as hypervisors running many many VMs and the underlying fabrics
does consolidates many types of traffic, globally pausing a port can
dramatically reduce the efficiency of such computing center which
probably was built originally to increase efficiency.

I believe that the ixgbe team well understand that, and hence their
continued DCB efforts can make the combination of RXE with
Niantic/ixgbe very intresting to test.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

[Or Gerlitz]

Paul Grun <pgrun-klaOcWyJdxkshyMvu7JE4pqQE7yCjDx5@public.gmane.org> wrote:
> there doesn't appear to be an argument in favor of requiring DCB with RoCEE

Interesting, the ofa server is down now, so I don't have access to ofa
IBoE materials, from my memory I recall that in ALL of them you have
made the IBoE/CEE bundling very clear & evident, e.g this  IBTA
presentation made to T11 @
http://www.t11.org/ftp/t11/pub/fc/study/09-543v0.pdf

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

[Paul Grun]

Or -
The emergence of standards-based lossless Ethernet (DCB) was the
precipitating factor that allowed us for the first time to talk about
running the IB message transport on what had formerly been a lossy network.

That said, there is a distinction to be made between 'works better' and 'is
required in order to work'.  Again, I don't think there is an argument to be
made for REQUIRING the use of DCB as a prerequisite to RoCEE.
Make sense?
-Paul

-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
[mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Or Gerlitz
Sent: Wednesday, December 23, 2009 2:18 PM
To: Paul Grun
Cc: Roland Dreier; Or Gerlitz; Liran Liss; Yevgeny Petrilin; Richard Frank;
Sean Hefty; Linux RDMA list
Subject: Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

Paul Grun <pgrun-klaOcWyJdxkshyMvu7JE4pqQE7yCjDx5@public.gmane.org> wrote:
> there doesn't appear to be an argument in favor of requiring DCB with
RoCEE

Interesting, the ofa server is down now, so I don't have access to ofa
IBoE materials, from my memory I recall that in ALL of them you have
made the IBoE/CEE bundling very clear & evident, e.g this  IBTA
presentation made to T11 @
http://www.t11.org/ftp/t11/pub/fc/study/09-543v0.pdf

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

[Roland Dreier]

 > To start with, no matter how many data VLs are used (e.g one), all the
 > crucial management traffic (SMPs) go on VL15 which is on the one hand
 > lossy and on the other hand not subject to congestion when other VLs
 > are. Now how would you manage your Cisco switch --remotely-- on a
 > globally paused fabric when some multicast receiver hasn't had its
 > breakfast and now slows the sender while filling the queues throughout
 > the congestion tree where this switch is part of?

There's not really an analog of QP0/VL15 traffic in IBoE (no SM, etc).
The analog of switch management traffic would either be on a separate
management network (and I wouldn't be surprised if many IBoE fabrics
have 100 meg management networks next to the 10/40G data fabric), or
would be QP1 traffic on the same data VL.  Yes this leads to problems if
the fabric is congested but many IB production fabrics seem to cope.

As I said, DCB is definitely useful for IBoE and also has many
advantages even for non-RDMA deployments, but conversely I think IBoE
may be useful in production, even in non-DCB classical ethernet fabrics.

 > To continue with, lossless is good, but to make your cluster usable
 > under congestion, you need congestion control, that is QCN, which is
 > designed/optimized to the case of multiple TCs.

I am not aware of a single production deployment of IB congestion
management.  So clearly it's a "nice to have" but again not a prereq for
production use.

 > Also, IBoE can potentially find its way to much more complex
 > environments than IB has, specifically, to clusters whose hosts are
 > acting as hypervisors running many many VMs and the underlying fabrics
 > does consolidates many types of traffic, globally pausing a port can
 > dramatically reduce the efficiency of such computing center which
 > probably was built originally to increase efficiency.

Sure, DCB is very useful, in many environments.  And maybe even a
requirement sometimes.  I'm simply trying to say that IBoE with
classical ethernet is at least as useful as standard IB in many cases
(IBoE without DCB is roughly equivalent to IB without QoS, and most IB
deployments still don't use QoS).

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

[Liran Liss]

I second...

Note that even with a "single VL", no endpoint can freeze the fabric -
if a multicast receiver has gone to breakfast it would just loose its
own packets rather than introducing congestion.
The only way an end-node can cause congestion is if its internal buses
don't match the IB link's BW, but this is unrelated to (lack of)
transport-level flow control.
--Liran


-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
[mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Roland Dreier
Sent: Thursday, December 24, 2009 2:47 AM
To: Or Gerlitz
Cc: Or Gerlitz; Liran Liss; Yevgeny Petrilin; Richard Frank; Sean Hefty;
Linux RDMA list; Paul Grun
Subject: Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)


 > To start with, no matter how many data VLs are used (e.g one), all
the  > crucial management traffic (SMPs) go on VL15 which is on the one
hand  > lossy and on the other hand not subject to congestion when other
VLs  > are. Now how would you manage your Cisco switch --remotely-- on a
> globally paused fabric when some multicast receiver hasn't had its  >
breakfast and now slows the sender while filling the queues throughout
> the congestion tree where this switch is part of?

There's not really an analog of QP0/VL15 traffic in IBoE (no SM, etc).
The analog of switch management traffic would either be on a separate
management network (and I wouldn't be surprised if many IBoE fabrics
have 100 meg management networks next to the 10/40G data fabric), or
would be QP1 traffic on the same data VL.  Yes this leads to problems if
the fabric is congested but many IB production fabrics seem to cope.

As I said, DCB is definitely useful for IBoE and also has many
advantages even for non-RDMA deployments, but conversely I think IBoE
may be useful in production, even in non-DCB classical ethernet fabrics.

 > To continue with, lossless is good, but to make your cluster usable
> under congestion, you need congestion control, that is QCN, which is
> designed/optimized to the case of multiple TCs.

I am not aware of a single production deployment of IB congestion
management.  So clearly it's a "nice to have" but again not a prereq for
production use.

 > Also, IBoE can potentially find its way to much more complex  >
environments than IB has, specifically, to clusters whose hosts are  >
acting as hypervisors running many many VMs and the underlying fabrics
> does consolidates many types of traffic, globally pausing a port can
> dramatically reduce the efficiency of such computing center which  >
probably was built originally to increase efficiency.

Sure, DCB is very useful, in many environments.  And maybe even a
requirement sometimes.  I'm simply trying to say that IBoE with
classical ethernet is at least as useful as standard IB in many cases
(IBoE without DCB is roughly equivalent to IB without QoS, and most IB
deployments still don't use QoS).

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info
at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

[Or Gerlitz]

Roland Dreier wrote:
> Sure, DCB is very useful, in many environments. And maybe even a requirement sometimes.  I'm simply trying to say that IBoE with classical ethernet is at least as useful as standard IB in many cases
Roland, Paul,

Putting a side for a moment the detailed discussion we've started and 
looking on the concluding remarks you have made, I wasn't sure to 
follow:  if DCB isn't available (even from a silly reason of hw 
supporting pfc but patches not being pushed to the kernel...) what you 
think would function better (or function at all) for IBoE, lossy or 
globally paused Ethernet? I haven't managed so far to convince you that 
both aren't applicable for IBoE, but I also didn't manage to see what 
are you suggesting in the absence of DCB.

Or.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

[Or Gerlitz]

Liran Liss wrote:
> I second...
>   
fair-enough, so now (A) everyone agrees that DCB is good for IBoE and 
(B) mlx4 supports pfc, any reason not to push the pfc patches into the 
kernel and have mlx4_en comply with the mainline dcbnl code?
> The only way an end-node can cause congestion is if its internal buses don't match the IB link's BW, but this is unrelated to (lack of) transport-level flow control.
>   
thanks for clarifying this

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMAoE / lossless Ethernet (ewg: SC'09 BOF - Meeting notes)

[Roland Dreier]

 > Putting a side for a moment the detailed discussion we've started and
 > looking on the concluding remarks you have made, I wasn't sure to
 > follow:  if DCB isn't available (even from a silly reason of hw
 > supporting pfc but patches not being pushed to the kernel...) what you
 > think would function better (or function at all) for IBoE, lossy or
 > globally paused Ethernet? I haven't managed so far to convince you
 > that both aren't applicable for IBoE, but I also didn't manage to see
 > what are you suggesting in the absence of DCB.

I think you would have to enable pause to avoid dropping packets.  The
IB RC transport doesn't seem to be designed to recover from packet loss.

And my main point is that if you're only using a single VL, then there's
no real difference between DCB/PFC and classical ethernet flow control.
The fundamental difference with IB remains, namely credit-based
vs. pause-based flow control, so you'll need more buffering on ethernet.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html