* Re: Understanding the number of TCP connections between clients and OSDs [not found] ` <590F8B9C-DA15-41C2-9E37-8847F79D7F6B@schermer.cz> @ 2015-10-27 0:41 ` hzwulibin-Re5JQEeQqe8AvxtiuMwx3w 2015-10-27 1:12 ` Re: [ceph-users] " hzwulibin 1 sibling, 0 replies; 4+ messages in thread From: hzwulibin-Re5JQEeQqe8AvxtiuMwx3w @ 2015-10-27 0:41 UTC (permalink / raw) To: Jan Schermer, Rick Balsano Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [-- Attachment #1.1: Type: text/plain, Size: 2994 bytes --] Hi, I also concerns about this problem. And my problem is how many threads will the qemu-system-x86 has. From what i tested, it could between 100 to 800, yeah, maybe it has relationship with the osd number. But it seems affect the performance when it has many threads. From what i tested, 4k randwrite will reduce from 15k to 4k. That's really unacceptable! My evnironment: 1. nine OSD storage servers with two intel DC 3500 SSD on each 2. hammer 0.94.3 3. QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u4~bpo70+1) Thanks! hzwulibin@gmail.com From: Jan Schermer Date: 2015-10-27 05:48 To: Rick Balsano CC: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Understanding the number of TCP connections between clients and OSDs If we're talking about RBD clients (qemu) then the number also grows with number of volumes attached to the client. With a single volume it was <1000. It grows when there's heavy IO happening in the guest. I had to bump up the file open limits to several thusands (8000 was it?) to accomodate client with 10 volumes in our cluster. We just scaled the number of OSDs down so hopefully I could have a graph of that. But I just guesstimated what it could become, and that's not necessarily what the theoretical limit is. Very bad things happen when you reach that threshold. It could also depend on the guest settings (like queue depth), and how much it seeks over the drive (how many different PGs it hits), but knowing the upper bound is most critical. Jan On 26 Oct 2015, at 21:32, Rick Balsano <rick@opower.com> wrote: We've run into issues with the number of open TCP connections from a single client to the OSDs in our Ceph cluster. We can (& have) increased the open file limit to work around this, but we're looking to understand what determines the number of open connections maintained between a client and a particular OSD. Our naive assumption was 1 open TCP connection per OSD or per port made available by the Ceph node. There are many more than this, presumably to allow parallel connections, because we see 1-4 connections from each client per open port on a Ceph node. Here is some background on our cluster: * still running Firefly 0.80.8 * 414 OSDs, 35 nodes, one massive pool * clients are KVM processes, accessing Ceph RBD images using virtio * total number of open TCP connections from one client to all nodes between 500-1000 Is there any way to either know or cap the maximum number of connections we should expect? I can provide more info as required. I've done some searches and found references to "huge number of TCP connections" but nothing concrete to tell me how to predict how that scales. Thanks, Rick -- Rick Balsano Senior Software Engineer Opower O +1 571 384 1210 We're Hiring! See jobs here. _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [-- Attachment #1.2: Type: text/html, Size: 6822 bytes --] [-- Attachment #2: Type: text/plain, Size: 178 bytes --] _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Re: [ceph-users] Understanding the number of TCP connections between clients and OSDs [not found] ` <590F8B9C-DA15-41C2-9E37-8847F79D7F6B@schermer.cz> 2015-10-27 0:41 ` Understanding the number of TCP connections between clients and OSDs hzwulibin-Re5JQEeQqe8AvxtiuMwx3w @ 2015-10-27 1:12 ` hzwulibin 2015-10-27 5:36 ` Haomai Wang 1 sibling, 1 reply; 4+ messages in thread From: hzwulibin @ 2015-10-27 1:12 UTC (permalink / raw) To: ceph-devel Hi, develops I also concerns about this problem. And my problem is how many threads will the qemu-system-x86 has? When will it cut down the threads? From what i tested, it could between 100 to 800, yeah, maybe it has relationship with the osd number. But it seems affect the performance when it has many threads. From what i tested, 4k randwrite will reduce from 15k to 4k. That's really unacceptable! My evnironment: 1. nine OSD storage servers with two intel DC 3500 SSD on each 2. hammer 0.94.3 3. QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u4~bpo70+1) Thanks! ------------------ hzwulibin 2015-10-27 ------------------------------------------------------------- 发件人：Jan Schermer <jan@schermer.cz> 发送日期：2015-10-27 05:48 收件人：Rick Balsano 抄送：ceph-users@lists.ceph.com 主题：Re: [ceph-users] Understanding the number of TCP connections between clients and OSDs If we're talking about RBD clients (qemu) then the number also grows with number of volumes attached to the client. With a single volume it was <1000. It grows when there's heavy IO happening in the guest. I had to bump up the file open limits to several thusands (8000 was it?) to accomodate client with 10 volumes in our cluster. We just scaled the number of OSDs down so hopefully I could have a graph of that. But I just guesstimated what it could become, and that's not necessarily what the theoretical limit is. Very bad things happen when you reach that threshold. It could also depend on the guest settings (like queue depth), and how much it seeks over the drive (how many different PGs it hits), but knowing the upper bound is most critical. Jan > On 26 Oct 2015, at 21:32, Rick Balsano <rick@opower.com> wrote: > > We've run into issues with the number of open TCP connections from a single client to the OSDs in our Ceph cluster. > > We can (& have) increased the open file limit to work around this, but we're looking to understand what determines the number of open connections maintained between a client and a particular OSD. Our naive assumption was 1 open TCP connection per OSD or per port made available by the Ceph node. There are many more than this, presumably to allow parallel connections, because we see 1-4 connections from each client per open port on a Ceph node. > > Here is some background on our cluster: > * still running Firefly 0.80.8 > * 414 OSDs, 35 nodes, one massive pool > * clients are KVM processes, accessing Ceph RBD images using virtio > * total number of open TCP connections from one client to all nodes between 500-1000 > > Is there any way to either know or cap the maximum number of connections we should expect? > > I can provide more info as required. I've done some searches and found references to "huge number of TCP connections" but nothing concrete to tell me how to predict how that scales. > > Thanks, > Rick > -- > Rick Balsano > Senior Software Engineer > Opower <http://www.opower.com/> > > O +1 571 384 1210 > We're Hiring! See jobs here <http://www.opower.com/careers>. > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Re: [ceph-users] Understanding the number of TCP connections between clients and OSDs 2015-10-27 1:12 ` Re: [ceph-users] " hzwulibin @ 2015-10-27 5:36 ` Haomai Wang 2015-10-27 6:31 ` hzwulibin 0 siblings, 1 reply; 4+ messages in thread From: Haomai Wang @ 2015-10-27 5:36 UTC (permalink / raw) To: hzwulibin; +Cc: ceph-devel On Tue, Oct 27, 2015 at 9:12 AM, hzwulibin <hzwulibin@gmail.com> wrote: > Hi, develops > > I also concerns about this problem. And my problem is how many threads will the qemu-system-x86 has? > When will it cut down the threads? It's because of network model, each connection will has two threads. We are actually working on this to avoid. BTW, for client level, maybe we can add a proxy for ceph message to avoid too much tcp socket on client host. But it need let us improve single connection's performance. > > From what i tested, it could between 100 to 800, yeah, maybe it has relationship with the osd number. But it > seems affect the performance when it has many threads. From what i tested, 4k randwrite will reduce from 15k > to 4k. That's really unacceptable! > > My evnironment: > > 1. nine OSD storage servers with two intel DC 3500 SSD on each > 2. hammer 0.94.3 > 3. QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u4~bpo70+1) > > Thanks! > > ------------------ > hzwulibin > 2015-10-27 > > ------------------------------------------------------------- > 发件人：Jan Schermer <jan@schermer.cz> > 发送日期：2015-10-27 05:48 > 收件人：Rick Balsano > 抄送：ceph-users@lists.ceph.com > 主题：Re: [ceph-users] Understanding the number of TCP connections > between clients and OSDs > > If we're talking about RBD clients (qemu) then the number also grows with number of volumes attached to the client. With a single volume it was <1000. It grows when there's heavy IO happening in the guest. > I had to bump up the file open limits to several thusands (8000 was it?) to accomodate client with 10 volumes in our cluster. We just scaled the number of OSDs down so hopefully I could have a graph of that. > But I just guesstimated what it could become, and that's not necessarily what the theoretical limit is. Very bad things happen when you reach that threshold. It could also depend on the guest settings (like queue depth), and how much it seeks over the drive (how many different PGs it hits), but knowing the upper bound is most critical. > > Jan > >> On 26 Oct 2015, at 21:32, Rick Balsano <rick@opower.com> wrote: >> >> We've run into issues with the number of open TCP connections from a single client to the OSDs in our Ceph cluster. >> >> We can (& have) increased the open file limit to work around this, but we're looking to understand what determines the number of open connections maintained between a client and a particular OSD. Our naive assumption was 1 open TCP connection per OSD or per port made available by the Ceph node. There are many more than this, presumably to allow parallel connections, because we see 1-4 connections from each client per open port on a Ceph node. >> >> Here is some background on our cluster: >> * still running Firefly 0.80.8 >> * 414 OSDs, 35 nodes, one massive pool >> * clients are KVM processes, accessing Ceph RBD images using virtio >> * total number of open TCP connections from one client to all nodes between 500-1000 >> >> Is there any way to either know or cap the maximum number of connections we should expect? >> >> I can provide more info as required. I've done some searches and found references to "huge number of TCP connections" but nothing concrete to tell me how to predict how that scales. >> >> Thanks, >> Rick >> -- >> Rick Balsano >> Senior Software Engineer >> Opower <http://www.opower.com/> >> >> O +1 571 384 1210 >> We're Hiring! See jobs here <http://www.opower.com/careers>. >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- Best Regards, Wheat -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Re: Re: [ceph-users] Understanding the number of TCP connections between clients and OSDs 2015-10-27 5:36 ` Haomai Wang @ 2015-10-27 6:31 ` hzwulibin 0 siblings, 0 replies; 4+ messages in thread From: hzwulibin @ 2015-10-27 6:31 UTC (permalink / raw) To: Haomai Wang; +Cc: ceph-devel Hi, Haomai Two many threads and connection is fine. But will it impact performance? From our test environment, the VM with more than 700 threads has poor IOPS performance. The qemu-system-x86 process's cpu usage is high(1420%): 15801 libvirt- 20 0 33.7g 1.4g 11m R 1420 0.6 1322:26 qemu-system-x86 we use perf to trace the pid(15801), it seems context switch occupy the many CPU cycle and impact the performance (because i see do_raw_spin_lock will be called in sched.c in the kernel): Samples: 1M of event 'cycles', Event count (approx.): 1057109744252 - 75.23% qemu-system-x86 [kernel.kallsyms] [k] do_raw_spin_lock So, i hope i could get an answer here, will the threads number impact performance so much? If so, any solution for it? Change different LTS version of ceph? my server: 256G RAM 32 CPU (model name : Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz) Two pools with 9 osd storage servers ssd pool: 2 OSD (intel DC 3500 SSD) on each server SAS pool 8 OSD (sas ) on each server. The VM has volumes from both pools, and we test on those volumes. The performance of volume from SSD cut down from 15K to 5K when the VM become abornormal(more than 700 threads as describe above). We are really stuck here! Thanks! ------------------ hzwulibin 2015-10-27 ------------------------------------------------------------- 发件人：Haomai Wang <haomaiwang@gmail.com> 发送日期：2015-10-27 13:36 收件人：hzwulibin 抄送：ceph-devel 主题：Re: Re: [ceph-users] Understanding the number of TCP connections between clients and OSDs On Tue, Oct 27, 2015 at 9:12 AM, hzwulibin <hzwulibin@gmail.com> wrote: > Hi, develops > > I also concerns about this problem. And my problem is how many threads will the qemu-system-x86 has? > When will it cut down the threads? It's because of network model, each connection will has two threads. We are actually working on this to avoid. BTW, for client level, maybe we can add a proxy for ceph message to avoid too much tcp socket on client host. But it need let us improve single connection's performance. > > From what i tested, it could between 100 to 800, yeah, maybe it has relationship with the osd number. But it > seems affect the performance when it has many threads. From what i tested, 4k randwrite will reduce from 15k > to 4k. That's really unacceptable! > > My evnironment: > > 1. nine OSD storage servers with two intel DC 3500 SSD on each > 2. hammer 0.94.3 > 3. QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u4~bpo70+1) > > Thanks! > > ------------------ > hzwulibin > 2015-10-27 > > ------------------------------------------------------------- > 发件人：Jan Schermer <jan@schermer.cz> > 发送日期：2015-10-27 05:48 > 收件人：Rick Balsano > 抄送：ceph-users@lists.ceph.com > 主题：Re: [ceph-users] Understanding the number of TCP connections > between clients and OSDs > > If we're talking about RBD clients (qemu) then the number also grows with number of volumes attached to the client. With a single volume it was <1000. It grows when there's heavy IO happening in the guest. > I had to bump up the file open limits to several thusands (8000 was it?) to accomodate client with 10 volumes in our cluster. We just scaled the number of OSDs down so hopefully I could have a graph of that. > But I just guesstimated what it could become, and that's not necessarily what the theoretical limit is. Very bad things happen when you reach that threshold. It could also depend on the guest settings (like queue depth), and how much it seeks over the drive (how many different PGs it hits), but knowing the upper bound is most critical. > > Jan > >> On 26 Oct 2015, at 21:32, Rick Balsano <rick@opower.com> wrote: >> >> We've run into issues with the number of open TCP connections from a single client to the OSDs in our Ceph cluster. >> >> We can (& have) increased the open file limit to work around this, but we're looking to understand what determines the number of open connections maintained between a client and a particular OSD. Our naive assumption was 1 open TCP connection per OSD or per port made available by the Ceph node. There are many more than this, presumably to allow parallel connections, because we see 1-4 connections from each client per open port on a Ceph node. >> >> Here is some background on our cluster: >> * still running Firefly 0.80.8 >> * 414 OSDs, 35 nodes, one massive pool >> * clients are KVM processes, accessing Ceph RBD images using virtio >> * total number of open TCP connections from one client to all nodes between 500-1000 >> >> Is there any way to either know or cap the maximum number of connections we should expect? >> >> I can provide more info as required. I've done some searches and found references to "huge number of TCP connections" but nothing concrete to tell me how to predict how that scales. >> >> Thanks, >> Rick >> -- >> Rick Balsano >> Senior Software Engineer >> Opower <http://www.opower.com/> >> >> O +1 571 384 1210 >> We're Hiring! See jobs here <http://www.opower.com/careers>. >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- Best Regards, Wheat ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-10-27 6:31 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAD76J01iYt1wVZZR655=KF+kRZATMc_yEB62QJOyRmRhMD8pFg@mail.gmail.com>
[not found] ` <590F8B9C-DA15-41C2-9E37-8847F79D7F6B@schermer.cz>
2015-10-27 0:41 ` Understanding the number of TCP connections between clients and OSDs hzwulibin-Re5JQEeQqe8AvxtiuMwx3w
2015-10-27 1:12 ` Re: [ceph-users] " hzwulibin
2015-10-27 5:36 ` Haomai Wang
2015-10-27 6:31 ` hzwulibin
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.