From mboxrd@z Thu Jan 1 00:00:00 1970 From: wangdi Subject: Re: [LSF/MM TOPIC] Parallelize file operation (like creation, unlink) under large shared directory Date: Sun, 22 Jan 2012 12:31:31 -0800 Message-ID: <4F1C7223.3040301@whamcloud.com> References: <4F1B2612.7020001@whamcloud.com> <4F1C1556.7010908@panasas.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: lsf-pc@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, Jinshan Xiong To: Boaz Harrosh Return-path: Received: from mail-iy0-f174.google.com ([209.85.210.174]:46824 "EHLO mail-iy0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752192Ab2AVUbe (ORCPT ); Sun, 22 Jan 2012 15:31:34 -0500 Received: by iacb35 with SMTP id b35so2653961iac.19 for ; Sun, 22 Jan 2012 12:31:33 -0800 (PST) In-Reply-To: <4F1C1556.7010908@panasas.com> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On 01/22/2012 05:55 AM, Boaz Harrosh wrote: > On 01/21/2012 10:54 PM, wangdi wrote: >> Single directory performance is a critical in some use cases. For >> example the multiple application threads might create hundreds of >> thousands of files in a single directory simultaneously within a short >> window of time. >> > read NFSD here ;-) > >> Currently, both filename lookup and file system modifying operations >> (such as create and unlink) are protected with a single lock for the >> entire directory. It might be useful to remove this lock, so multiple >> application threads can access the directory simultaneously. >> > I agree about create, unlink, and so on. But don't we have some lockless > look up in place since a few Kernels ago? The lockless lookup is only for lookups in the dcache and not in the backing filesystem, which is serialized by i_mutex in the VFS(real_lookup). > But yes the topic is very interesting, though I'd suspect its hard to > implement. We actually already implemented this for ext4, and we saw a lot performance improvement(at least 30% improvements for open/create in a single directory)for lustre stack, but we want to make this improvement accessible through the VFS. Probably XFS and Btrfs could also benefit from this. Thanks WangDi >> Thanks >> WangDi >> > Thanks > Boaz