* Re: Prioritize Heartbeat packets [not found] ` <ltmnkp$tod$1@ger.gmane.org> @ 2015-03-07 0:20 ` Robert LeBlanc [not found] ` <CAANLjFrHdDNJnMnWQk3s2cT0LZxsCGp63xHBJw7HNJocfSahzw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2015-03-07 23:23 ` Daniel Swarbrick 0 siblings, 2 replies; 9+ messages in thread From: Robert LeBlanc @ 2015-03-07 0:20 UTC (permalink / raw) To: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, ceph-devel [-- Attachment #1.1: Type: text/plain, Size: 1446 bytes --] I see that Jian Wen has done work on this for 0.94. I tried looking through the code to see if I can figure out how to configure this new option, but it all went over my head pretty quick. Can I get a brief summary on how to set the priority of heartbeat packets or where to look in the code to figure it out? Thanks, On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick < daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> wrote: > On 28/08/14 02:56, Sage Weil wrote: > > I seem to remember someone telling me there were hooks/hints you could > > call that would tag either a socket or possibly data on that socket with > a > > label for use by iptables and such.. but I forget what it was. > > > > Something like setsockopt() SO_MARK? > > *SO_MARK *(since Linux 2.6.25) > Set the mark for each packet sent through this socket > (similar > to the netfilter MARK target but socket-based). Changing the > mark can be used for mark-based routing without netfilter or > for packet filtering. Setting this option requires the > *CAP_NET_ADMIN *capability. > > Alternatively, directly set IP_TOS options on the socket, or SO_PRIORITY > which sets the IP TOS bits as well. > > > _______________________________________________ > ceph-users mailing list > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > [-- Attachment #1.2: Type: text/html, Size: 2320 bytes --] [-- Attachment #2: Type: text/plain, Size: 178 bytes --] _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <CAANLjFrHdDNJnMnWQk3s2cT0LZxsCGp63xHBJw7HNJocfSahzw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Fwd: Prioritize Heartbeat packets [not found] ` <CAANLjFrHdDNJnMnWQk3s2cT0LZxsCGp63xHBJw7HNJocfSahzw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2015-03-07 0:22 ` Robert LeBlanc 0 siblings, 0 replies; 9+ messages in thread From: Robert LeBlanc @ 2015-03-07 0:22 UTC (permalink / raw) To: ceph-devel, ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Hidden HTML ... trying agin... ---------- Forwarded message ---------- From: Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> Date: Fri, Mar 6, 2015 at 5:20 PM Subject: Re: [ceph-users] Prioritize Heartbeat packets To: "ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org" <ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org>, ceph-devel <ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> I see that Jian Wen has done work on this for 0.94. I tried looking through the code to see if I can figure out how to configure this new option, but it all went over my head pretty quick. Can I get a brief summary on how to set the priority of heartbeat packets or where to look in the code to figure it out? Thanks, On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick <daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org> wrote: > > On 28/08/14 02:56, Sage Weil wrote: > > I seem to remember someone telling me there were hooks/hints you could > > call that would tag either a socket or possibly data on that socket with a > > label for use by iptables and such.. but I forget what it was. > > > > Something like setsockopt() SO_MARK? > > *SO_MARK *(since Linux 2.6.25) > Set the mark for each packet sent through this socket (similar > to the netfilter MARK target but socket-based). Changing the > mark can be used for mark-based routing without netfilter or > for packet filtering. Setting this option requires the > *CAP_NET_ADMIN *capability. > > Alternatively, directly set IP_TOS options on the socket, or SO_PRIORITY > which sets the IP TOS bits as well. > > > _______________________________________________ > ceph-users mailing list > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Prioritize Heartbeat packets 2015-03-07 0:20 ` Prioritize Heartbeat packets Robert LeBlanc [not found] ` <CAANLjFrHdDNJnMnWQk3s2cT0LZxsCGp63xHBJw7HNJocfSahzw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2015-03-07 23:23 ` Daniel Swarbrick 2015-03-09 19:36 ` Robert LeBlanc 1 sibling, 1 reply; 9+ messages in thread From: Daniel Swarbrick @ 2015-03-07 23:23 UTC (permalink / raw) To: ceph-devel; +Cc: ceph-users Judging by the commit, this ought to do the trick: osd heartbeat use min delay socket = true On 07/03/15 01:20, Robert LeBlanc wrote: > I see that Jian Wen has done work on this for 0.94. I tried looking > through the code to see if I can figure out how to configure this new > option, but it all went over my head pretty quick. > > Can I get a brief summary on how to set the priority of heartbeat > packets or where to look in the code to figure it out? > > Thanks, > > On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick > <daniel.swarbrick@profitbricks.com > <mailto:daniel.swarbrick@profitbricks.com>> wrote: > > On 28/08/14 02:56, Sage Weil wrote: > > I seem to remember someone telling me there were hooks/hints you could > > call that would tag either a socket or possibly data on that socket with a > > label for use by iptables and such.. but I forget what it was. > > > > Something like setsockopt() SO_MARK? > > *SO_MARK *(since Linux 2.6.25) > Set the mark for each packet sent through this socket > (similar > to the netfilter MARK target but socket-based). > Changing the > mark can be used for mark-based routing without > netfilter or > for packet filtering. Setting this option requires the > *CAP_NET_ADMIN *capability. > > Alternatively, directly set IP_TOS options on the socket, or SO_PRIORITY > which sets the IP TOS bits as well. > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > <mailto:ceph-users@lists.ceph.com> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Prioritize Heartbeat packets 2015-03-07 23:23 ` Daniel Swarbrick @ 2015-03-09 19:36 ` Robert LeBlanc 2015-03-10 2:58 ` [ceph-users] " Jian Wen 0 siblings, 1 reply; 9+ messages in thread From: Robert LeBlanc @ 2015-03-09 19:36 UTC (permalink / raw) To: Daniel Swarbrick Cc: ceph-devel, ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see that when 'osd heartbeat use min delay socket = true' it will mark the packet with DSCP CS6. Based on the setting of the socket in msg/simple/Pipe.cc is it possible that this can apply to both OSD and monitor? I don't understand the code enough to know how the set_socket_options() is called from the OSD and monitor. If this applies to both monitor and OSD, would it be better to rename the option to a more generic name? Thanks, On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick <daniel.swarbrick-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > Judging by the commit, this ought to do the trick: > > osd heartbeat use min delay socket = true > > On 07/03/15 01:20, Robert LeBlanc wrote: >> >> I see that Jian Wen has done work on this for 0.94. I tried looking >> through the code to see if I can figure out how to configure this new >> option, but it all went over my head pretty quick. >> >> Can I get a brief summary on how to set the priority of heartbeat >> packets or where to look in the code to figure it out? >> >> Thanks, >> >> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick >> <daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org >> <mailto:daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>> wrote: >> >> On 28/08/14 02:56, Sage Weil wrote: >> > I seem to remember someone telling me there were hooks/hints you >> could >> > call that would tag either a socket or possibly data on that socket >> with a >> > label for use by iptables and such.. but I forget what it was. >> > >> >> Something like setsockopt() SO_MARK? >> >> *SO_MARK *(since Linux 2.6.25) >> Set the mark for each packet sent through this socket >> (similar >> to the netfilter MARK target but socket-based). >> Changing the >> mark can be used for mark-based routing without >> netfilter or >> for packet filtering. Setting this option requires the >> *CAP_NET_ADMIN *capability. >> >> Alternatively, directly set IP_TOS options on the socket, or >> SO_PRIORITY >> which sets the IP TOS bits as well. >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org >> <mailto:ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [ceph-users] Prioritize Heartbeat packets 2015-03-09 19:36 ` Robert LeBlanc @ 2015-03-10 2:58 ` Jian Wen 2015-03-10 5:16 ` Robert LeBlanc 0 siblings, 1 reply; 9+ messages in thread From: Jian Wen @ 2015-03-10 2:58 UTC (permalink / raw) To: Robert LeBlanc; +Cc: Daniel Swarbrick, ceph-devel, ceph-users@lists.ceph.com Only OSD calls set_socket_priority(). See https://github.com/ceph/ceph/pull/3353 On Tue, Mar 10, 2015 at 3:36 AM, Robert LeBlanc <robert@leblancnet.us> wrote: > I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see > that when 'osd heartbeat use min delay socket = true' it will mark the > packet with DSCP CS6. Based on the setting of the socket in > msg/simple/Pipe.cc is it possible that this can apply to both OSD and > monitor? I don't understand the code enough to know how the > set_socket_options() is called from the OSD and monitor. > > If this applies to both monitor and OSD, would it be better to rename > the option to a more generic name? > > Thanks, > > On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick > <daniel.swarbrick@gmail.com> wrote: >> Judging by the commit, this ought to do the trick: >> >> osd heartbeat use min delay socket = true >> >> On 07/03/15 01:20, Robert LeBlanc wrote: >>> >>> I see that Jian Wen has done work on this for 0.94. I tried looking >>> through the code to see if I can figure out how to configure this new >>> option, but it all went over my head pretty quick. >>> >>> Can I get a brief summary on how to set the priority of heartbeat >>> packets or where to look in the code to figure it out? >>> >>> Thanks, >>> >>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick >>> <daniel.swarbrick@profitbricks.com >>> <mailto:daniel.swarbrick@profitbricks.com>> wrote: >>> >>> On 28/08/14 02:56, Sage Weil wrote: >>> > I seem to remember someone telling me there were hooks/hints you >>> could >>> > call that would tag either a socket or possibly data on that socket >>> with a >>> > label for use by iptables and such.. but I forget what it was. >>> > >>> >>> Something like setsockopt() SO_MARK? >>> >>> *SO_MARK *(since Linux 2.6.25) >>> Set the mark for each packet sent through this socket >>> (similar >>> to the netfilter MARK target but socket-based). >>> Changing the >>> mark can be used for mark-based routing without >>> netfilter or >>> for packet filtering. Setting this option requires the >>> *CAP_NET_ADMIN *capability. >>> >>> Alternatively, directly set IP_TOS options on the socket, or >>> SO_PRIORITY >>> which sets the IP TOS bits as well. >>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> <mailto:ceph-users@lists.ceph.com> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Best, Jian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [ceph-users] Prioritize Heartbeat packets 2015-03-10 2:58 ` [ceph-users] " Jian Wen @ 2015-03-10 5:16 ` Robert LeBlanc [not found] ` <CAANLjFrwqr-znyMSBw=wCgiVkwqRT69ud3n0yAnhWUH=PZRTTw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 9+ messages in thread From: Robert LeBlanc @ 2015-03-10 5:16 UTC (permalink / raw) To: Jian Wen; +Cc: Daniel Swarbrick, ceph-devel, ceph-users@lists.ceph.com Jian, Thanks for the clarification. I'll mark traffic destined for the monitors as well. We are getting ready to put our first cluster into production. If you are interested we will be testing the heartbeat priority to see if we can saturate the network (not an easy task for 40 Gb) and keep the cluster from falling apart. Our network team is marking COS based on the DSCP and enforcing priority. We have three VLANs on bonded 40 GbE, management, storage (monitors, clients, OSDs), and cluster (replication). We have three priority classes management (heartbeats on all VLANs, SSH, DNS, etc), storage traffic (no marking), and replication (scavenger class). We are interested to see how things pan out. Thanks, Robert On Mon, Mar 9, 2015 at 8:58 PM, Jian Wen <wenjianhn@gmail.com> wrote: > Only OSD calls set_socket_priority(). > See https://github.com/ceph/ceph/pull/3353 > > On Tue, Mar 10, 2015 at 3:36 AM, Robert LeBlanc <robert@leblancnet.us> wrote: >> I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see >> that when 'osd heartbeat use min delay socket = true' it will mark the >> packet with DSCP CS6. Based on the setting of the socket in >> msg/simple/Pipe.cc is it possible that this can apply to both OSD and >> monitor? I don't understand the code enough to know how the >> set_socket_options() is called from the OSD and monitor. >> >> If this applies to both monitor and OSD, would it be better to rename >> the option to a more generic name? >> >> Thanks, >> >> On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick >> <daniel.swarbrick@gmail.com> wrote: >>> Judging by the commit, this ought to do the trick: >>> >>> osd heartbeat use min delay socket = true >>> >>> On 07/03/15 01:20, Robert LeBlanc wrote: >>>> >>>> I see that Jian Wen has done work on this for 0.94. I tried looking >>>> through the code to see if I can figure out how to configure this new >>>> option, but it all went over my head pretty quick. >>>> >>>> Can I get a brief summary on how to set the priority of heartbeat >>>> packets or where to look in the code to figure it out? >>>> >>>> Thanks, >>>> >>>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick >>>> <daniel.swarbrick@profitbricks.com >>>> <mailto:daniel.swarbrick@profitbricks.com>> wrote: >>>> >>>> On 28/08/14 02:56, Sage Weil wrote: >>>> > I seem to remember someone telling me there were hooks/hints you >>>> could >>>> > call that would tag either a socket or possibly data on that socket >>>> with a >>>> > label for use by iptables and such.. but I forget what it was. >>>> > >>>> >>>> Something like setsockopt() SO_MARK? >>>> >>>> *SO_MARK *(since Linux 2.6.25) >>>> Set the mark for each packet sent through this socket >>>> (similar >>>> to the netfilter MARK target but socket-based). >>>> Changing the >>>> mark can be used for mark-based routing without >>>> netfilter or >>>> for packet filtering. Setting this option requires the >>>> *CAP_NET_ADMIN *capability. >>>> >>>> Alternatively, directly set IP_TOS options on the socket, or >>>> SO_PRIORITY >>>> which sets the IP TOS bits as well. >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@lists.ceph.com >>>> <mailto:ceph-users@lists.ceph.com> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@lists.ceph.com >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>> >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Best, > > Jian ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <CAANLjFrwqr-znyMSBw=wCgiVkwqRT69ud3n0yAnhWUH=PZRTTw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Prioritize Heartbeat packets [not found] ` <CAANLjFrwqr-znyMSBw=wCgiVkwqRT69ud3n0yAnhWUH=PZRTTw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2015-04-10 6:39 ` Jian Wen [not found] ` <CAMXzGWJJpDbV0M0rtTkoRU=tSkgi3tM=x6Sy+wKhfZ=oPNy2hw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2015-04-10 17:46 ` [ceph-users] " Robert LeBlanc 0 siblings, 2 replies; 9+ messages in thread From: Jian Wen @ 2015-04-10 6:39 UTC (permalink / raw) To: Robert LeBlanc Cc: Daniel Swarbrick, ceph-devel, ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Robert, 60 days ago, we enabled this feature in our production environment. It works for us. Did you see any flapping OSDs? On Tue, Mar 10, 2015 at 1:16 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: > Jian, > > Thanks for the clarification. I'll mark traffic destined for the > monitors as well. We are getting ready to put our first cluster into > production. If you are interested we will be testing the heartbeat > priority to see if we can saturate the network (not an easy task for > 40 Gb) and keep the cluster from falling apart. Our network team is > marking COS based on the DSCP and enforcing priority. We have three > VLANs on bonded 40 GbE, management, storage (monitors, clients, OSDs), > and cluster (replication). We have three priority classes management > (heartbeats on all VLANs, SSH, DNS, etc), storage traffic (no > marking), and replication (scavenger class). We are interested to see > how things pan out. > > Thanks, > Robert > > On Mon, Mar 9, 2015 at 8:58 PM, Jian Wen <wenjianhn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> Only OSD calls set_socket_priority(). >> See https://github.com/ceph/ceph/pull/3353 >> >> On Tue, Mar 10, 2015 at 3:36 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote: >>> I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see >>> that when 'osd heartbeat use min delay socket = true' it will mark the >>> packet with DSCP CS6. Based on the setting of the socket in >>> msg/simple/Pipe.cc is it possible that this can apply to both OSD and >>> monitor? I don't understand the code enough to know how the >>> set_socket_options() is called from the OSD and monitor. >>> >>> If this applies to both monitor and OSD, would it be better to rename >>> the option to a more generic name? >>> >>> Thanks, >>> >>> On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick >>> <daniel.swarbrick-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >>>> Judging by the commit, this ought to do the trick: >>>> >>>> osd heartbeat use min delay socket = true >>>> >>>> On 07/03/15 01:20, Robert LeBlanc wrote: >>>>> >>>>> I see that Jian Wen has done work on this for 0.94. I tried looking >>>>> through the code to see if I can figure out how to configure this new >>>>> option, but it all went over my head pretty quick. >>>>> >>>>> Can I get a brief summary on how to set the priority of heartbeat >>>>> packets or where to look in the code to figure it out? >>>>> >>>>> Thanks, >>>>> >>>>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick >>>>> <daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org >>>>> <mailto:daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>> wrote: >>>>> >>>>> On 28/08/14 02:56, Sage Weil wrote: >>>>> > I seem to remember someone telling me there were hooks/hints you >>>>> could >>>>> > call that would tag either a socket or possibly data on that socket >>>>> with a >>>>> > label for use by iptables and such.. but I forget what it was. >>>>> > >>>>> >>>>> Something like setsockopt() SO_MARK? >>>>> >>>>> *SO_MARK *(since Linux 2.6.25) >>>>> Set the mark for each packet sent through this socket >>>>> (similar >>>>> to the netfilter MARK target but socket-based). >>>>> Changing the >>>>> mark can be used for mark-based routing without >>>>> netfilter or >>>>> for packet filtering. Setting this option requires the >>>>> *CAP_NET_ADMIN *capability. >>>>> >>>>> Alternatively, directly set IP_TOS options on the socket, or >>>>> SO_PRIORITY >>>>> which sets the IP TOS bits as well. >>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org >>>>> <mailto:ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org> >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>> >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> -- >> Best, >> >> Jian -- Best, Jian ^ permalink raw reply [flat|nested] 9+ messages in thread
[parent not found: <CAMXzGWJJpDbV0M0rtTkoRU=tSkgi3tM=x6Sy+wKhfZ=oPNy2hw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Prioritize Heartbeat packets [not found] ` <CAMXzGWJJpDbV0M0rtTkoRU=tSkgi3tM=x6Sy+wKhfZ=oPNy2hw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2015-04-10 14:49 ` Robert LeBlanc 0 siblings, 0 replies; 9+ messages in thread From: Robert LeBlanc @ 2015-04-10 14:49 UTC (permalink / raw) To: Jian Wen; +Cc: Daniel Swarbrick, ceph-users-idqoXFIVOFJgJs9I8MT0rw, ceph-devel [-- Attachment #1.1: Type: text/plain, Size: 5340 bytes --] We have had some delays. Our to be production cluster is currently on firefly but should be on hammer next week. Once upgraded we will preform the network saturation test and verify results. Robert LeBlanc Sent from a mobile device please excuse any typos. On Apr 10, 2015 12:39 AM, "Jian Wen" <wenjianhn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > Robert, > 60 days ago, we enabled this feature in our production environment. > It works for us. > > Did you see any flapping OSDs? > > On Tue, Mar 10, 2015 at 1:16 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> > wrote: > > Jian, > > > > Thanks for the clarification. I'll mark traffic destined for the > > monitors as well. We are getting ready to put our first cluster into > > production. If you are interested we will be testing the heartbeat > > priority to see if we can saturate the network (not an easy task for > > 40 Gb) and keep the cluster from falling apart. Our network team is > > marking COS based on the DSCP and enforcing priority. We have three > > VLANs on bonded 40 GbE, management, storage (monitors, clients, OSDs), > > and cluster (replication). We have three priority classes management > > (heartbeats on all VLANs, SSH, DNS, etc), storage traffic (no > > marking), and replication (scavenger class). We are interested to see > > how things pan out. > > > > Thanks, > > Robert > > > > On Mon, Mar 9, 2015 at 8:58 PM, Jian Wen <wenjianhn-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >> Only OSD calls set_socket_priority(). > >> See https://github.com/ceph/ceph/pull/3353 > >> > >> On Tue, Mar 10, 2015 at 3:36 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> > wrote: > >>> I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see > >>> that when 'osd heartbeat use min delay socket = true' it will mark the > >>> packet with DSCP CS6. Based on the setting of the socket in > >>> msg/simple/Pipe.cc is it possible that this can apply to both OSD and > >>> monitor? I don't understand the code enough to know how the > >>> set_socket_options() is called from the OSD and monitor. > >>> > >>> If this applies to both monitor and OSD, would it be better to rename > >>> the option to a more generic name? > >>> > >>> Thanks, > >>> > >>> On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick > >>> <daniel.swarbrick-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >>>> Judging by the commit, this ought to do the trick: > >>>> > >>>> osd heartbeat use min delay socket = true > >>>> > >>>> On 07/03/15 01:20, Robert LeBlanc wrote: > >>>>> > >>>>> I see that Jian Wen has done work on this for 0.94. I tried looking > >>>>> through the code to see if I can figure out how to configure this new > >>>>> option, but it all went over my head pretty quick. > >>>>> > >>>>> Can I get a brief summary on how to set the priority of heartbeat > >>>>> packets or where to look in the code to figure it out? > >>>>> > >>>>> Thanks, > >>>>> > >>>>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick > >>>>> <daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org > >>>>> <mailto:daniel.swarbrick-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>> wrote: > >>>>> > >>>>> On 28/08/14 02:56, Sage Weil wrote: > >>>>> > I seem to remember someone telling me there were hooks/hints > you > >>>>> could > >>>>> > call that would tag either a socket or possibly data on that > socket > >>>>> with a > >>>>> > label for use by iptables and such.. but I forget what it was. > >>>>> > > >>>>> > >>>>> Something like setsockopt() SO_MARK? > >>>>> > >>>>> *SO_MARK *(since Linux 2.6.25) > >>>>> Set the mark for each packet sent through this > socket > >>>>> (similar > >>>>> to the netfilter MARK target but socket-based). > >>>>> Changing the > >>>>> mark can be used for mark-based routing without > >>>>> netfilter or > >>>>> for packet filtering. Setting this option > requires the > >>>>> *CAP_NET_ADMIN *capability. > >>>>> > >>>>> Alternatively, directly set IP_TOS options on the socket, or > >>>>> SO_PRIORITY > >>>>> which sets the IP TOS bits as well. > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> ceph-users mailing list > >>>>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > >>>>> <mailto:ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org> > >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> ceph-users mailing list > >>>>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>>>> > >>>> > >>>> > >>>> -- > >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in > >>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>> _______________________________________________ > >>> ceph-users mailing list > >>> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > >> > >> > >> -- > >> Best, > >> > >> Jian > > > > -- > Best, > > Jian > [-- Attachment #1.2: Type: text/html, Size: 8471 bytes --] [-- Attachment #2: Type: text/plain, Size: 178 bytes --] _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [ceph-users] Prioritize Heartbeat packets 2015-04-10 6:39 ` Jian Wen [not found] ` <CAMXzGWJJpDbV0M0rtTkoRU=tSkgi3tM=x6Sy+wKhfZ=oPNy2hw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2015-04-10 17:46 ` Robert LeBlanc 1 sibling, 0 replies; 9+ messages in thread From: Robert LeBlanc @ 2015-04-10 17:46 UTC (permalink / raw) To: Jian Wen; +Cc: Daniel Swarbrick, ceph-devel, ceph-users@lists.ceph.com We have had some delays. Our to be production cluster is currently on firefly but should be on hammer next week. Once upgraded we will preform the network saturation test and verify results. On Fri, Apr 10, 2015 at 12:39 AM, Jian Wen <wenjianhn@gmail.com> wrote: > Robert, > 60 days ago, we enabled this feature in our production environment. > It works for us. > > Did you see any flapping OSDs? > > On Tue, Mar 10, 2015 at 1:16 PM, Robert LeBlanc <robert@leblancnet.us> wrote: >> Jian, >> >> Thanks for the clarification. I'll mark traffic destined for the >> monitors as well. We are getting ready to put our first cluster into >> production. If you are interested we will be testing the heartbeat >> priority to see if we can saturate the network (not an easy task for >> 40 Gb) and keep the cluster from falling apart. Our network team is >> marking COS based on the DSCP and enforcing priority. We have three >> VLANs on bonded 40 GbE, management, storage (monitors, clients, OSDs), >> and cluster (replication). We have three priority classes management >> (heartbeats on all VLANs, SSH, DNS, etc), storage traffic (no >> marking), and replication (scavenger class). We are interested to see >> how things pan out. >> >> Thanks, >> Robert >> >> On Mon, Mar 9, 2015 at 8:58 PM, Jian Wen <wenjianhn@gmail.com> wrote: >>> Only OSD calls set_socket_priority(). >>> See https://github.com/ceph/ceph/pull/3353 >>> >>> On Tue, Mar 10, 2015 at 3:36 AM, Robert LeBlanc <robert@leblancnet.us> wrote: >>>> I've found commit 9b9a682fe035c985e416ee1c112fa58f9045a27c and I see >>>> that when 'osd heartbeat use min delay socket = true' it will mark the >>>> packet with DSCP CS6. Based on the setting of the socket in >>>> msg/simple/Pipe.cc is it possible that this can apply to both OSD and >>>> monitor? I don't understand the code enough to know how the >>>> set_socket_options() is called from the OSD and monitor. >>>> >>>> If this applies to both monitor and OSD, would it be better to rename >>>> the option to a more generic name? >>>> >>>> Thanks, >>>> >>>> On Sat, Mar 7, 2015 at 4:23 PM, Daniel Swarbrick >>>> <daniel.swarbrick@gmail.com> wrote: >>>>> Judging by the commit, this ought to do the trick: >>>>> >>>>> osd heartbeat use min delay socket = true >>>>> >>>>> On 07/03/15 01:20, Robert LeBlanc wrote: >>>>>> >>>>>> I see that Jian Wen has done work on this for 0.94. I tried looking >>>>>> through the code to see if I can figure out how to configure this new >>>>>> option, but it all went over my head pretty quick. >>>>>> >>>>>> Can I get a brief summary on how to set the priority of heartbeat >>>>>> packets or where to look in the code to figure it out? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> On Thu, Aug 28, 2014 at 2:01 AM, Daniel Swarbrick >>>>>> <daniel.swarbrick@profitbricks.com >>>>>> <mailto:daniel.swarbrick@profitbricks.com>> wrote: >>>>>> >>>>>> On 28/08/14 02:56, Sage Weil wrote: >>>>>> > I seem to remember someone telling me there were hooks/hints you >>>>>> could >>>>>> > call that would tag either a socket or possibly data on that socket >>>>>> with a >>>>>> > label for use by iptables and such.. but I forget what it was. >>>>>> > >>>>>> >>>>>> Something like setsockopt() SO_MARK? >>>>>> >>>>>> *SO_MARK *(since Linux 2.6.25) >>>>>> Set the mark for each packet sent through this socket >>>>>> (similar >>>>>> to the netfilter MARK target but socket-based). >>>>>> Changing the >>>>>> mark can be used for mark-based routing without >>>>>> netfilter or >>>>>> for packet filtering. Setting this option requires the >>>>>> *CAP_NET_ADMIN *capability. >>>>>> >>>>>> Alternatively, directly set IP_TOS options on the socket, or >>>>>> SO_PRIORITY >>>>>> which sets the IP TOS bits as well. >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list >>>>>> ceph-users@lists.ceph.com >>>>>> <mailto:ceph-users@lists.ceph.com> >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list >>>>>> ceph-users@lists.ceph.com >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>> >>>>> >>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@lists.ceph.com >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >>> >>> -- >>> Best, >>> >>> Jian > > > > -- > Best, > > Jian ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2015-04-10 17:46 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1985102496.575.1409186929716.JavaMail.root@thunderbeast.private.linuxbox.com>
[not found] ` <alpine.DEB.2.00.1408271755160.966@cobra.newdream.net>
[not found] ` <ltmnkp$tod$1@ger.gmane.org>
2015-03-07 0:20 ` Prioritize Heartbeat packets Robert LeBlanc
[not found] ` <CAANLjFrHdDNJnMnWQk3s2cT0LZxsCGp63xHBJw7HNJocfSahzw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-03-07 0:22 ` Fwd: " Robert LeBlanc
2015-03-07 23:23 ` Daniel Swarbrick
2015-03-09 19:36 ` Robert LeBlanc
2015-03-10 2:58 ` [ceph-users] " Jian Wen
2015-03-10 5:16 ` Robert LeBlanc
[not found] ` <CAANLjFrwqr-znyMSBw=wCgiVkwqRT69ud3n0yAnhWUH=PZRTTw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-04-10 6:39 ` Jian Wen
[not found] ` <CAMXzGWJJpDbV0M0rtTkoRU=tSkgi3tM=x6Sy+wKhfZ=oPNy2hw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-04-10 14:49 ` Robert LeBlanc
2015-04-10 17:46 ` [ceph-users] " Robert LeBlanc
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.