From mboxrd@z Thu Jan 1 00:00:00 1970 From: Joe Jin Subject: Re: Lots of connections led oxenstored stuck Date: Mon, 11 Aug 2014 08:35:37 +0800 Message-ID: <53E80FD9.3010002@oracle.com> References: <53E475C8.2070806@oracle.com> <0E6BCB61859D7F4EB9CAC75FC6EE6FF84573109A@SZXEMA502-MBX.china.huawei.com> <56117839-54D7-410D-9008-25F5F18514FA@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <56117839-54D7-410D-9008-25F5F18514FA@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Dave Scott , "Liuqiming (John)" Cc: Zheng Li , "Luis R. Rodriguez" , Luonengjun , xen-devel , Fanhenglong , Ian Jackson List-Id: xen-devel@lists.xenproject.org On 08/08/14 17:37, Dave Scott wrote: > = > On 8 Aug 2014, at 09:35, Liuqiming (John) wro= te: > = >> In oxenstored it use "select" for incoming socket, so I don't think it c= an handle more than 1024 socket connections. = > = > That=92s true. The problem is when oxenstored does not respond any request anymore even all thread exited, with my reproducer, when you executed it and all threads exi= ted, "xm list -l" will stuck. Thanks, Joe > = > In the long term I=92d like to use Lwt which internally uses libev and ha= s a more scalable event loop. > = > In the short term I think Zheng Li (cc:d) may have a prototype patch to w= ork around this issue. Is this right, Zheng? > = > Cheers, > Dave > = >> >>> -----Original Message----- >>> From: xen-devel-bounces@lists.xen.org >>> [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Joe Jin >>> Sent: Friday, August 08, 2014 3:01 PM >>> To: David Scott; Luis R. Rodriguez; Ian Jackson >>> Cc: xen-devel >>> Subject: [Xen-devel] Lots of connections led oxenstored stuck >>> >>> Hi, >>> >>> During internal test on Xen-4.3-stable we found sometime when restarted >>> Xen, it stuck and does not response any request, xenstored.log filled >>> out below stuff: >>> [20140702T21:00:41.564Z|error|xenstored] caught exception >>> Unix.Unix_error(15, "accept", "") >>> >>> I created reproducer which will create 2000 connections to oxenstored, >>> after >>> ran the reproducer, "xm list --long" will stuck, oxenstored does not >>> response anymore, same test case passed when use xenstored, any input >>> will appreciate! >>> >>> /* >>> * This program used to test oxenstored connections stuck issue. >>> * please compile by below command: >>> * gcc -o client client.c -lpthread >>> */ >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> #include >>> >>> >>> void *main_thread(void *arg) >>> { >>> struct sockaddr_un address; >>> int socket_fd, nbytes; >>> char buffer[256]; >>> int i; >>> extern int errno; >>> >>> memcpy(&i, arg, sizeof(i)); >>> socket_fd =3D socket(PF_UNIX, SOCK_STREAM, 0); >>> if (socket_fd < 0) { >>> fprintf(stderr, "socket() %dth failed, errno=3D%d\n", i, errno); >>> return; >>> } >>> fprintf(stderr, "socket() %dth ok!\n", i); >>> >>> /* start with a clean address structure */ >>> memset(&address, 0, sizeof(struct sockaddr_un)); >>> >>> address.sun_family =3D AF_UNIX; >>> snprintf(address.sun_path, 1024, "/var/run/xenstored/socket"); >>> >>> if (connect(socket_fd, >>> (struct sockaddr *) &address, >>> sizeof(struct sockaddr_un)) !=3D 0) { >>> fprintf(stderr, "connect() %d failed, error=3D%d", i, errno); >>> return; >>> } >>> fprintf(stderr, "connec() %dth ok!\n", i); >>> >>> while (1) >>> sleep(1); >>> if (arg) { >>> free(arg); >>> arg =3D NULL; >>> } >>> >>> return; >>> } >>> >>> int main(void) >>> { >>> int i; >>> for (i =3D 0; i < 2000; i++) { >>> void *arg =3D malloc(sizeof(i)); >>> memset(arg, 0, sizeof(i)); >>> memcpy(arg, &i, sizeof(i)); >>> pthread_t thread; >>> if (pthread_create(&thread, NULL, main_thread, arg) !=3D 0) { >>> perror("pthread_create:"); >>> break; >>> } >>> } >>> /* Wait all children exit */ >>> sleep(3); >>> return 0; >>> } >>> /* end */ >>> >>> Thanks, >>> Joe >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel > = > = > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > =