From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeff Garzik Subject: Re: Design challenges in chunkd self-checking Date: Tue, 22 Dec 2009 17:43:58 -0500 Message-ID: <4B314BAE.4010805@garzik.org> References: <20091222144111.789a5b91@redhat.com> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; bh=+sSwA7N0iqx5Ta+VCPRaRJFbKeqIL6ROsRThzTAkXfs=; b=NvsJugLkqMwUdw3iETQYDfnWJluczfAxJCTj/pLrtIMNNBshgGz5rWtAv1jSYRCpPZ axjKTzibCTCpFjXND0ZboI1o6gFDyntzPDqe/FbaGiPEO0XD8NMebnWEOil85d9RMQpy 0X3BzzXPKxfAndjxBXYqQ6KET+UZbXia1HNxY= In-Reply-To: <20091222144111.789a5b91@redhat.com> Sender: hail-devel-owner@vger.kernel.org List-ID: Content-Type: text/plain; charset="us-ascii"; format="flowed" To: Pete Zaitcev Cc: Project Hail List On 12/22/2009 04:41 PM, Pete Zaitcev wrote: > I'm looking into adding self-checking to chunkd. This involves basically > a process that re-reads everything stored in the chunkserver and verifies > that it's still ok. Nothing can be simpler, right? > > So, current problems for which I'd like input are: > > - Scheduling and deconflicting with normal operation. > > Run "genisofs" in your Fedora desktop and your Firefox is DEAD. > It is also the reason why everyone does rpm -e mlocate the first thing > after the installation. The effect of massive data access blowing > away caches is very drastic in a regular Linux. > So, I have to have a good way to keep self-checkig from interfering > with normal service of a chunkserver. > Also, need to save power instead of burning it on re-reading data. The problem seems to revolve around two variables: * last-checked time. You wouldn't want to check a single individual object more than once every N hours|days|weeks. * maximum bytes-per-second. You wouldn't want to exceed a useful bound for throughput. Perhaps the last variable could be calculated by observing disk throughput over time, in conjunction with the number of objects and their sizes, resulting in an idea of the total time required to check the entire dataset. And if we start keeping data like this, we might want to move metadata from the beginning of each object to a TC database. That might speed up fs_list_objs and a couple other operations, too. > - Consistency. > > Returning wrong checksums for an object that is being updated may > lead to us deciding to drop a perfectly good object, which is > unacceptable (especially when redundancy is impaired already). > So, I need some kind of locking, or logging, or invalidation... It is normal and reasonable to maintain global information about all in-progress operations. Caching systems do that, for example, to ensure multiple cache requests for object A do not initiate multiple simultaneous back-end requests for object A. For the purposes of verification, I would just skip objects that are actively being written-to. Those are, by definition, too new to probably need verification anyway. BTW, in case this is helpful, chunkd's backend writes a zeroed metadata header to the beginning of each object. The metadata header is only updated with "real" values after the final data byte is written. Jeff