From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexandre Oliva Subject: Re: corruption of active mmapped files in btrfs snapshots Date: Fri, 22 Mar 2013 11:17:30 -0300 Message-ID: References: <20130318225245.25964.22219@localhost.localdomain> <20130319120940.29898.80124@localhost.localdomain> <20130321180614.30044.29961@localhost.localdomain> <20130321230621.30044.80640@localhost.localdomain> <20130322120754.27874.14389@localhost.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Received: from linux-libre.fsfla.org ([208.118.235.54]:38961 "EHLO linux-libre.fsfla.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932116Ab3CVORu (ORCPT ); Fri, 22 Mar 2013 10:17:50 -0400 In-Reply-To: <20130322120754.27874.14389@localhost.localdomain> (Chris Mason's message of "Fri, 22 Mar 2013 08:07:54 -0400") Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Chris Mason Cc: "linux-btrfs@vger.kernel.org" , "ceph-devel@vger.kernel.org" On Mar 22, 2013, Chris Mason wrote: > Are you using compression in btrfs or just in leveldb? btrfs lzo compression. > I'd like to take snapshots out of the picture for a minute. That's understandable, I guess, but I don't know that anyone has ever got the problem without snapshots. I mean, even when the master copy of the database got corrupted, snapshots of the subvol containing it were being taken every now and again, because that's the way ceph works. Even back when I noticed corruption of firefox _CACHE_* files, snapshots taken for archival were involved. So, unless the program happens to trigger the problem with the -DNOSNAPS option about as easily as it did without it, I guess we may not have a choice but to keep snapshots in the picture. > We need some way to synchronize the leveldb with snapshotting I purposefully refrained from doing that, because AFAICT ceph doesn't do that. Once I failed to trigger the problem with Sync calls, and determined ceph only syncs the leveldb logs before taking its snapshots, I went without syncing and finally succeeded in triggering the bug in snapshots, by simulating very similar snapshotting and mmaping conditions to those generated by ceph. I haven't managed to trigger the corruption of the master subvol yet with the test program, but I already knew its corruption didn't occur as often as that of the snapshots, and since it smells like two slightly different symptoms of the same bug, I decided to leave the test program at that. -- Alexandre Oliva, freedom fighter http://FSFLA.org/~lxoliva/ You must be the change you wish to see in the world. -- Gandhi Be Free! -- http://FSFLA.org/ FSF Latin America board member Free Software Evangelist Red Hat Brazil Compiler Engineer From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from linux-libre.fsfla.org ([208.118.235.54]:38961 "EHLO linux-libre.fsfla.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932116Ab3CVORu (ORCPT ); Fri, 22 Mar 2013 10:17:50 -0400 From: Alexandre Oliva To: Chris Mason Cc: "linux-btrfs\@vger.kernel.org" , "ceph-devel\@vger.kernel.org" Subject: Re: corruption of active mmapped files in btrfs snapshots References: <20130318225245.25964.22219@localhost.localdomain> <20130319120940.29898.80124@localhost.localdomain> <20130321180614.30044.29961@localhost.localdomain> <20130321230621.30044.80640@localhost.localdomain> <20130322120754.27874.14389@localhost.localdomain> Date: Fri, 22 Mar 2013 11:17:30 -0300 In-Reply-To: <20130322120754.27874.14389@localhost.localdomain> (Chris Mason's message of "Fri, 22 Mar 2013 08:07:54 -0400") Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: linux-btrfs-owner@vger.kernel.org List-ID: On Mar 22, 2013, Chris Mason wrote: > Are you using compression in btrfs or just in leveldb? btrfs lzo compression. > I'd like to take snapshots out of the picture for a minute. That's understandable, I guess, but I don't know that anyone has ever got the problem without snapshots. I mean, even when the master copy of the database got corrupted, snapshots of the subvol containing it were being taken every now and again, because that's the way ceph works. Even back when I noticed corruption of firefox _CACHE_* files, snapshots taken for archival were involved. So, unless the program happens to trigger the problem with the -DNOSNAPS option about as easily as it did without it, I guess we may not have a choice but to keep snapshots in the picture. > We need some way to synchronize the leveldb with snapshotting I purposefully refrained from doing that, because AFAICT ceph doesn't do that. Once I failed to trigger the problem with Sync calls, and determined ceph only syncs the leveldb logs before taking its snapshots, I went without syncing and finally succeeded in triggering the bug in snapshots, by simulating very similar snapshotting and mmaping conditions to those generated by ceph. I haven't managed to trigger the corruption of the master subvol yet with the test program, but I already knew its corruption didn't occur as often as that of the snapshots, and since it smells like two slightly different symptoms of the same bug, I decided to leave the test program at that. -- Alexandre Oliva, freedom fighter http://FSFLA.org/~lxoliva/ You must be the change you wish to see in the world. -- Gandhi Be Free! -- http://FSFLA.org/ FSF Latin America board member Free Software Evangelist Red Hat Brazil Compiler Engineer