Re: btrfs csum failed on git .pack file

[Markus Trippelsdorf <markus@trippelsdorf.de>]

[-- Attachment #1: Type: text/plain, Size: 2506 bytes --]

On Thu, Sep 17, 2009 at 02:15:01PM +0200, Markus Trippelsdorf wrote:
> On Thu, Sep 17, 2009 at 11:05:49AM +0200, Jens Axboe wrote:
> > On Thu, Sep 17 2009, Markus Trippelsdorf wrote:
> > > On Thu, Sep 17, 2009 at 08:44:56AM +0200, Jens Axboe wrote:
> > > > On Thu, Sep 17 2009, Markus Trippelsdorf wrote:
> > > > > On Tue, Sep 08, 2009 at 10:00:42PM +0200, Jens Axboe wrote:
> > > > > > On Mon, Sep 07 2009, Markus Trippelsdorf wrote:
> > > > > > > Just got this error today in my dmesg:
> > > > > > > btrfs csum failed ino 1483065 off 158482432 csum 4283543305 private 43905798
> > > > > > > 
> > > > > > > linux % find . -inum 1483065
> > > > > > > ./.git/objects/pack/pack-f9251bcc6a8afe3c92193e14d1d742f2f0182ce5.pack
> > > > > > > 
> > > > > > > It's the main pack file from my git linux kernel tree:
> > > > > > > 
> > > > > > 
> > > > > > Hmm, I ran into something very similar. Care to check what the corrupted
> > > > > > block of data looks like (and how big it is)?
> > > > > 
> > > > > I've hit the same problem again today:
> > > > > 
> > > > > btrfs csum failed ino 1826333 off 150208512 csum 4148434891 private 1660028275
> > > > > 
> > > > > The file in question is:
> > > > > ./.git/objects/pack/pack-a2330b703d5a7fd62626b39a5fdfb6eecf739d0d.pack
> > > > > 
> > > > > I can't read the file directly, because of the csum mismatch:
> > > > 
> > > > Chris, is there a way to force reading the file? Seems like that would
> > > > be a very handy feature.
> > > > 
> > > > Markus, not sure if that works, but you could always try and remount
> > > > with data checksumming disabled.
> > > > 
> > > > mount /dev/fooX -o remount,rw,nodatasum
> > > > 
> > > > should do the trick.
> > > 
> > > That doesn't work unfortunately, btrfs still calculates and compares the
> > > checksums (it won't write new ones I guess).
> > 
> > Ah ok, as mentioned I wasn't sure whether that would work or not. I'll
> > defer to Chris :-)
> 
> Understood.
> 
> I did some further investigations and was able to reconstruct exactly
> the same pack file in question by starting from an older backup copy of
> my git repro and then running the same git commands as previous. 
> Then I did a binary comparison between this reconstructed file and a
> corrupted backup copy from the time before the csum errors occured (I
> automatically backup every 4h).
> 
Thanks to Chris' patch (from IRC) I was able to compare the file with
the csum error to the reconstructed one. You'll find the reults as
attachments.

-- 
Markus

[-- Attachment #2: hex_bad1 --]
[-- Type: text/plain, Size: 114 bytes --]

08F403A0   5D 8E B3 32  7D 8F 5D E7  54 B6 9D 1E  E6 0C 9B 0D  BE 1D 9D 0C  34 BA 7F FE  7F D4 E5 1A  0A 16 29 96

[-- Attachment #3: hex_bad2 --]
[-- Type: text/plain, Size: 342 bytes --]

105AC3A0   76 80 1E 0A  3F 8A 7E FC  B3 2E 2B 9E  9E 53 82 10  C3 F6 4B C1  C0 12 FC 61  A5 0E 63 70  B0 A4 7B 27
105AC3C0   DC AE 26 CE  48 5D CA 07  B7 26 B6 3C  BC 91 AD 00  55 97 BF E4  8C D7 EF AA  28 B7 54 65  30 DB 78 A6
105AC3E0   26 90 18 88  8F F4 25 91  48 5F 9C F6  4F 0D 46 72  A2 04 77 1A  AF FB 88 23  93 AF FB AA  B9 82 BC CC

[-- Attachment #4: hex_good1 --]
[-- Type: text/plain, Size: 114 bytes --]

08F403A0   5D 8E B3 32  7D 8F 5D E7  54 B4 9D 1E  E6 0C 9B 0D  BE 1D 9D 0C  34 BA 7F FE  7F D4 E5 1A  0A 16 29 96

[-- Attachment #5: hex_good2 --]
[-- Type: text/plain, Size: 342 bytes --]

105AC3A0   76 80 1E 0A  3F 8A 7E FC  B3 2E 2B 9E  9E 53 82 10  C3 F7 4B C1  C0 12 FC 61  A5 0E 63 70  B0 A4 7B 27
105AC3C0   DC AE 26 CE  48 5D CA 07  B7 77 B6 3C  BC 91 AD 00  55 97 BF E4  8C D7 EF AA  28 A7 54 65  30 DB 78 A6
105AC3E0   26 90 18 88  8F F4 25 91  48 5F 9C F6  4F 0D 46 72  A2 04 77 1A  AF FB 88 23  93 AF FB AA  B9 82 BC CC