From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeff Garzik Subject: Re: Metadata replication in tabled Date: Mon, 28 Jun 2010 19:21:21 -0400 Message-ID: <4C292E71.1090504@garzik.org> References: <20100624183123.08248d80@lembas.zaitcev.lan> <4C28979F.1030306@redhat.com> <20100628121056.16dfc2f8@lembas.zaitcev.lan> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; bh=AXd49gcJhHsUxftdp/PslLdcWgzKz2IgJ2iPqJUJpCk=; b=MGesIx7JLZUh6iGK6qTJ824hkUFco+lXK5mOmraNkyjP0gf30rN9a4Tmdrriix3HPC 5uAHf7yFT4uD9S6A45ZItSOk5BMOSFuxeyqlisCBLev6wLjVAB3TrMxsNUIysSpOFyxN VN/WZYlBlahxWr3lwyW96dU1O5/kq7uDqTjF0= In-Reply-To: <20100628121056.16dfc2f8@lembas.zaitcev.lan> Sender: hail-devel-owner@vger.kernel.org List-ID: Content-Type: text/plain; charset="us-ascii"; format="flowed" To: Pete Zaitcev Cc: Jeff Darcy , Project Hail List On 06/28/2010 02:10 PM, Pete Zaitcev wrote: > Well, yeah... I guess I was too lazy and reluctant to create yet > another state machine for this. Maybe I should just bite the bullet > and make tabled fully multi-threaded. It was likely to come next > anyway since you complained about the abysmal performance (I do not > know yet what the issues with performance are, but threads are > likely to participate). But if so, a thread may just easily loop, > as ncld API intends. With my wikipedia corpus testing, part of the issue seems to be a lack of ready-and-waiting-to-write/read chunkd connections. Ideally you want to have an already-open, initialized connection to a chunkd server ready. Back in the days when I slung NNTP news across the galaxy, having multiple TCP connections to a single server was critical for performance, in particular mitigating the overhead of connection and authentication initialization. Jeff