From mboxrd@z Thu Jan  1 00:00:00 1970
From: Joe Jin <joe.jin@oracle.com>
Subject: Lots of connections led oxenstored stuck
Date: Fri, 08 Aug 2014 15:01:28 +0800
Message-ID: <53E475C8.2070806@oracle.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: David Scott <dave.scott@eu.citrix.com>, "Luis R. Rodriguez" <mcgrof@suse.com>, Ian Jackson <ian.jackson@eu.citrix.com>
Cc: xen-devel <xen-devel@lists.xen.org>
List-Id: xen-devel@lists.xenproject.org

Hi,

During internal test on Xen-4.3-stable we found sometime when restarted
Xen, it stuck and does not response any request, xenstored.log filled
out below stuff:
[20140702T21:00:41.564Z|error|xenstored] caught exception Unix.Unix_error(15, "accept", "")

I created reproducer which will create 2000 connections to oxenstored, after
ran the reproducer, "xm list --long" will stuck, oxenstored does not
response anymore, same test case passed when use xenstored, any input
will appreciate! 

/* 
 * This program used to test oxenstored connections stuck issue.
 * please compile by below command:
 *	gcc -o client client.c -lpthread
 */
#include <stdio.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
#include <string.h>
#include <pthread.h>
#include <stdlib.h>
#include <errno.h>


void *main_thread(void *arg)
{
	struct sockaddr_un address;
	int socket_fd, nbytes;
	char buffer[256];
	int i;
	extern int errno;

	memcpy(&i, arg, sizeof(i));
	socket_fd = socket(PF_UNIX, SOCK_STREAM, 0);
	if (socket_fd < 0) {
		fprintf(stderr, "socket() %dth failed, errno=%d\n", i, errno);
		return;
	}
	fprintf(stderr, "socket() %dth ok!\n", i);

	/* start with a clean address structure */
	memset(&address, 0, sizeof(struct sockaddr_un));

	address.sun_family = AF_UNIX;
	snprintf(address.sun_path, 1024, "/var/run/xenstored/socket");

	if (connect(socket_fd,
		    (struct sockaddr *) &address,
		    sizeof(struct sockaddr_un)) != 0) {
		fprintf(stderr, "connect() %d failed, error=%d", i, errno);
		return;
	}
	fprintf(stderr, "connec() %dth ok!\n", i);

	while (1)
		sleep(1);
	if (arg) {
		free(arg);
		arg = NULL;
	}

	return;
}

int main(void)
{
	int i;
	for (i = 0; i < 2000; i++) {
		void *arg = malloc(sizeof(i));
		memset(arg, 0, sizeof(i));
		memcpy(arg, &i, sizeof(i));
		pthread_t thread;
		if (pthread_create(&thread, NULL, main_thread, arg) != 0) {
			perror("pthread_create:");
			break;
		}
	}
	/* Wait all children exit */
	sleep(3);
	return 0;
}
/* end */

Thanks,
Joe