From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stan Hoeppner Subject: Re: high throughput storage server? Date: Mon, 28 Feb 2011 16:22:08 -0600 Message-ID: <4D6C2010.1030406@hardwarefreak.com> References: <20110215044434.GA9186@septictank.raw-sewage.fake> <4D6AC288.20101@wildgooses.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4D6AC288.20101@wildgooses.com> Sender: linux-raid-owner@vger.kernel.org To: Ed W Cc: Matt Garman , Mdadm List-Id: linux-raid.ids Ed W put forth on 2/27/2011 3:30 PM: > Your application appears to be an implementation of a queue processing > system? ie each machine: pulls a file down, processes it, gets the next > file, etc? > > Can you share some information on > - the size of files you pull down (I saw something in another post) > - how long each machine takes to process each file > - whether there is any dependency between the processing machines? eg > can each machine operate completely independently of the others and > start it's job when it wishes (or does it need to sync?) > > Given the tentative assumption that > - processing each file takes many multiples of the time needed to > download the file, and > - files are processed independently > > It would appear that you can use a much lower powered system to > basically push jobs out to the processing machines in advance, this way > your bandwidth basically only needs to be: > size_of_job * num_machines / time_to_process_jobs > > So if the time to process jobs is significant then you have quite some > time to push out the next job to local storage ready? > > Firstly is this architecture workable? If so then you have some new > performance parameters to target for the storage architecture? > > Good luck Ed, you stated this thought much more thoroughly and eloquently than I did in my last rambling post. Thank you. -- Stan