From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Siddhartha Jain" Subject: RE: Replicating directories - Intercepting write/modify system calls Date: Fri, 13 Feb 2004 10:30:54 +0530 Sender: linux-fsdevel-owner@vger.kernel.org Message-ID: References: <000301c3f18c$a42bb870$0201a8c0@joe> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8BIT Return-path: Received: from mymail.netmagicians.com ([202.87.39.126]:34224 "HELO mymail.netmagicians.com") by vger.kernel.org with SMTP id S266723AbUBMFGF convert rfc822-to-8bit (ORCPT ); Fri, 13 Feb 2004 00:06:05 -0500 To: In-Reply-To: <000301c3f18c$a42bb870$0201a8c0@joe> List-Id: linux-fsdevel.vger.kernel.org Hey Joseph, >Too complicated. Too invasive. In what way? Will it slow down the FS too much? What other approach can be adopted to keep the implementation FS-independant and still track the changes in real-time. >Replicating the entire partition rather than just a specific directory is more feasible. How would I do that? Will the implementation still be FS-independant? >Does it need to be replicated in real-time, or would an incremental synchronization be acceptable (i.e. synced every 15 >>minutes or so)? Synced every fifteen minutes is ok but anything that scans the whole partition/directory file by file would be nighmarish given that the mail toasters I am mirroring have tens of thousands of files. Therefore, IMHO, the solution needs to track changes as and when they happen to the file and mirror them immiediately. For eg, a rsync/mirrordir takes upto half-an-hour to sync about 100,000 files. Not to mention the 40% CPU load on the file server while the rsync runs (a host mounts the NFS folder from a NetApp filer and rsyncs it to a local dir). This is just one server. If I were to put all my 6 mail toasters on the filer and mirror the filer's main volume to a host, I guess either would croak. >Does it need to be a two-way synchronization, or is a one-way synchronization acceptable? In other words, do changes on the >replica need to be synced back to the original? For my purpose, one-way is fine. But two-way would be cool and in the greater good of humankind ;) >Are you willing to commission someone (i.e. pay them money) to do this work? 8-D Yep, I will give you a free mail account on my mail servers and if the code is really good, maybe a rksh account too :P Btw, I am willing to do all the dirty work of wrestling with C code and testing but I need guidance. Sorry, if my questions sound too un-informed or stupid. I feel like a mere human roaming amongst gods of FS. :) Siddhartha Jain (CISSP) > -----Original Message----- > From: Siddhartha Jain [mailto:sid@netmagicsolutions.com] > Sent: Thursday, February 12, 2004 6:39 AM > To: linux-fsdevel@vger.kernel.org > Subject: Replicating directories - Intercepting write/modify system calls > > Hello, > > I need a module/tool to replicate directories locally or across a network. > Locally, the master and the mirror directories may not belong to the same > FS. My immiediate need is to replicate a NFS-mounted directory to another > local/NFS directory. So here is what I figured: > > 1. A config file contains the master and mirror paths > 2. Expand the vnode structure to include two things: a bit-flag to > indicate > that the vnode has a replica and a pointer to the replica vnode. > 3. When an open() is called, > a. check if the read/modify option is set in the open() (I am not > interested in replicating access-time at the moment). > b. If yes, check the path of the file against the known masters in > the > config. > c. If it matches, copy the file to a destination specified by the > mirror's > path. > d. Set the flag in the master vnode for replica and place a pointer > to the > replica vnode in the master's structure. > e. After this replicate all write() calls for the master vnode to > the > replica vnode. > f. close() the replica vnode when the master is close()-ed. > > > Being a sysadmin and not really a C/Kernel programmer, I do not know how > much is this feasible and practical. > > Given that it is ok, how do I avoid the copy at open() and do only > incremental changes everytime a file is closed and re-opened. > > Regards, > > Siddhartha > > - > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" > in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html