From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wu Fengguang Subject: Re: NFS page states & writeback Date: Fri, 25 Mar 2011 15:11:50 +0800 Message-ID: <20110325071149.GB5970@localhost> References: <20110325012803.GA25052@quack.suse.cz> <20110325044754.GK26611@dastard> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Jan Kara , "linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Andrew Morton , Christoph Hellwig To: Dave Chinner Return-path: Content-Disposition: inline In-Reply-To: <20110325044754.GK26611@dastard> Sender: linux-nfs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: linux-fsdevel.vger.kernel.org On Fri, Mar 25, 2011 at 12:47:54PM +0800, Dave Chinner wrote: > On Fri, Mar 25, 2011 at 02:28:03AM +0100, Jan Kara wrote: > > Hi, > > > > while working on changes to balance_dirty_pages() I was investigating why > > NFS writeback is *so* bumpy when I do not call writeback_inodes_wb() from > > balance_dirty_pages(). Take a single dd writing to NFS. What I can > > see is that we quickly accumulate dirty pages upto limit - ~700 MB on that > > machine. So flusher thread starts working and in an instant all these ~700 > > MB transition from Dirty state to Writeback state. Then, as server acks > > writes, Writeback pages slowly change to Unstable pages (at 100 MB/s rate > > let's say) and then at one moment (commit to server happens) all pages > > transition from Unstable to Clean state - the cycle begins from the start. > > > > The reason for this behavior seems to be a flaw in the logic in > > over_bground_thresh() which checks: > > global_page_state(NR_FILE_DIRTY) + > > global_page_state(NR_UNSTABLE_NFS) > background_thresh > > So at the moment all pages are turned Writeback, flusher thread goes to > > sleep and doesn't do any background writeback, until we have accumulated > > enough Stable pages to get over background_thresh. But NFS needs to have > > ->write_inode() called so that it can sent commit requests to the server. > > So effectively we end up sending commit only when background_thresh Unstable > > pages have accumulated which creates the bumpyness. Previously this wasn't > > a problem because balance_dirty_pages() ended up calling ->write_inode() > > often enough for NFS to send commit requests reasonably often. > > > > Now I wouldn't write so long email about this if I knew how to cleanly fix > > the check ;-). One way to "fix" the check would be to add there Writeback > > pages: > > NR_FILE_DIRTY + NR_WRITEBACK + NR_UNSTABLE_NFS > background_thresh > > > > This would work in the sense that it would keep flusher thread working but > > a) for normal filesystems it would be working even if there's potentially > > nothing to do (or it is not necessary to do anything) > > b) NFS is picky when it sends commit requests (inode has to have more > > Stable pages than Writeback pages if I'm reading the code in > > nfs_commit_unstable_pages() right) so flusher thread may be working but > > nothing really happens until enough stable pages accumulate. > > > > A check which kind of works but looks a bit hacky and is not perfect when > > there are multiple files is: > > NR_FILE_DIRTY + NR_UNSTABLE_NFS > background_thresh || > > NR_UNSTABLE_NFS > NR_WRITEBACK (to match what NFS does) > > > > Any better idea for a fix? > > Have NFS account for it's writeback pages to also be accounted as > NR_UNSTABLE_NFS pages? i.e. rather than incrementing NR_UNSTABLE_NFS > at the writeback->unstable transition, account it at the > dirty->writeback transition.... This increases the opportunity for the NFS flusher to busy loop. Maybe not a big problem as long as we add some sleep in the loop. writeback: sleep for 10ms when nothing is written http://linux.derkeiler.com/Mailing-Lists/Kernel/2010-12/msg06391.html Thanks, Fengguang -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html