* Re: Two persistent problems
2014-11-14 22:00 ` Josef Bacik
@ 2014-11-17 10:59 ` Konstantin
2014-11-17 11:36 ` Hugo Mills
2014-11-17 11:10 ` Hugo Mills
2014-11-26 18:35 ` Marc Joliet
2 siblings, 1 reply; 6+ messages in thread
From: Konstantin @ 2014-11-17 10:59 UTC (permalink / raw)
To: Josef Bacik, Hugo Mills, Btrfs mailing list, Chris Mason
Josef Bacik wrote on 14.11.2014 at 23:00:
> On 11/14/2014 04:51 PM, Hugo Mills wrote:
>> Chris, Josef, anyone else who's interested,
>>
>> On IRC, I've been seeing reports of two persistent unsolved
>> problems. Neither is showing up very often, but both have turned up
>> often enough to indicate that there's something specific going on
>> worthy of investigation.
>>
>> One of them is definitely a btrfs problem. The other may be btrfs,
>> or something in the block layer, or just broken hardware; it's hard to
>> tell from where I sit.
>>
>> Problem 1: ENOSPC on balance
>>
>> This has been going on since about March this year. I can
>> reasonably certainly recall 8-10 cases, possibly a number more. When
>> running a balance, the operation fails with ENOSPC when there's plenty
>> of space remaining unallocated. This happens on full balance, filtered
>> balance, and device delete. Other than the ENOSPC on balance, the FS
>> seems to work OK. It seems to be more prevalent on filesystems
>> converted from ext*. The first few or more reports of this didn't make
>> it to bugzilla, but a few of them since then have gone in.
>>
>> Problem 2: Unexplained zeroes
>>
>> Failure to mount. Transid failure, "expected xyz, have 0". Chris
>> looked at an early one of these (for Ke, on IRC) back in September
>> (the 27th -- sadly, the public IRC logs aren't there for it, but I can
>> supply a copy of the private log). He rapidly came to the conclusion
>> that it was something bad going on with TRIM, replacing some blocks
>> with zeroes. Since then, I've seen a bunch of these coming past on
>> IRC. It seems to be a 3.17 thing. I can successfully predict the
>> presence of an SSD and -odiscard from the "have 0". I've successfully
>> persuaded several people to put this into bugzilla and capture
>> btrfs-images. btrfs recover doesn't generally seem to be helpful in
>> recovering data.
>>
>>
>> I think Josef had problem 1 in his sights, but I don't know if
>> additional images or reports are helpful at this point. For problem 2,
>> there's obviously something bad going on, but there's not much else to
>> go on -- and the inability to recover data isn't good.
>>
>> For each of these, what more information should I be trying to
>> collect from any future reporters?
>>
>>
>
> So for #2 I've been looking at that the last two weeks. I'm always
> paranoid we're screwing up one of our data integrity sort of things,
> either not waiting on IO to complete properly or something like that.
> I've built a dm target to be as evil as possible and have been running
> it trying to make bad things happen. I got slightly side tracked
> since my stress test exposed a bug in the tree log stuff an csums
> which I just fixed. Now that I've fixed that I'm going back to try
> and make the "expected blah, have 0" type errors happen.
>
> As for the ENOSPC I keep meaning to look into it and I keep getting
> distracted with other more horrible things. Ideally I'd like to
> reproduce it myself, so more info on that front would be good, like do
> all reports use RAID/compression/some other odd set of features?
> Thanks for taking care of this stuff Hugo, #2 is the worst one and I'd
> like to be absolutely sure it's not our bug, once I'm happy we aren't
> I'll look at the balance thing.
>
> Josef
For #2, I had a strangely damaged BTRFS I reported a week or so ago
which may have similar background. Dmesg gives:
parent transid verify failed on 586239082496 wanted 13329746340512024838
found 588
BTRFS: open_ctree failed
The thing is that btrfsck crashes when trying to check this. As nobody
seemed to be interested I reformatted this disk today.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Two persistent problems
2014-11-17 10:59 ` Konstantin
@ 2014-11-17 11:36 ` Hugo Mills
0 siblings, 0 replies; 6+ messages in thread
From: Hugo Mills @ 2014-11-17 11:36 UTC (permalink / raw)
To: Konstantin; +Cc: Josef Bacik, Btrfs mailing list, Chris Mason
[-- Attachment #1: Type: text/plain, Size: 2469 bytes --]
On Mon, Nov 17, 2014 at 11:59:48AM +0100, Konstantin wrote:
> Josef Bacik wrote on 14.11.2014 at 23:00:
> > On 11/14/2014 04:51 PM, Hugo Mills wrote:
[snip]
> >> Problem 2: Unexplained zeroes
> >>
> >> Failure to mount. Transid failure, "expected xyz, have 0". Chris
> >> looked at an early one of these (for Ke, on IRC) back in September
> >> (the 27th -- sadly, the public IRC logs aren't there for it, but I can
> >> supply a copy of the private log). He rapidly came to the conclusion
> >> that it was something bad going on with TRIM, replacing some blocks
> >> with zeroes. Since then, I've seen a bunch of these coming past on
> >> IRC. It seems to be a 3.17 thing. I can successfully predict the
> >> presence of an SSD and -odiscard from the "have 0". I've successfully
> >> persuaded several people to put this into bugzilla and capture
> >> btrfs-images. btrfs recover doesn't generally seem to be helpful in
> >> recovering data.
[snip]
> > So for #2 I've been looking at that the last two weeks. I'm always
> > paranoid we're screwing up one of our data integrity sort of things,
> > either not waiting on IO to complete properly or something like that.
> > I've built a dm target to be as evil as possible and have been running
> > it trying to make bad things happen. I got slightly side tracked
> > since my stress test exposed a bug in the tree log stuff an csums
> > which I just fixed. Now that I've fixed that I'm going back to try
> > and make the "expected blah, have 0" type errors happen.
[snip]
> For #2, I had a strangely damaged BTRFS I reported a week or so ago
> which may have similar background. Dmesg gives:
>
> parent transid verify failed on 586239082496 wanted 13329746340512024838
> found 588
> BTRFS: open_ctree failed
>
> The thing is that btrfsck crashes when trying to check this. As nobody
> seemed to be interested I reformatted this disk today.
Whilst that's a genuine problem, it's not specifically the one I
was referring to here, which shows up with "want=X, have=0" from btrfs
check, and seems to be related to TRIM on SSDs.
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Turning, pages turning in the widening bath, / The spine ---
cannot bear the humidity. / Books fall apart; the binding
cannot hold. / Page 129 is loosed upon the world.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Two persistent problems
2014-11-14 22:00 ` Josef Bacik
2014-11-17 10:59 ` Konstantin
@ 2014-11-17 11:10 ` Hugo Mills
2014-11-26 18:35 ` Marc Joliet
2 siblings, 0 replies; 6+ messages in thread
From: Hugo Mills @ 2014-11-17 11:10 UTC (permalink / raw)
To: Josef Bacik; +Cc: Btrfs mailing list, Chris Mason
[-- Attachment #1: Type: text/plain, Size: 4363 bytes --]
On Fri, Nov 14, 2014 at 05:00:26PM -0500, Josef Bacik wrote:
> On 11/14/2014 04:51 PM, Hugo Mills wrote:
> > Chris, Josef, anyone else who's interested,
> >
> > On IRC, I've been seeing reports of two persistent unsolved
> >problems. Neither is showing up very often, but both have turned up
> >often enough to indicate that there's something specific going on
> >worthy of investigation.
> >
> > One of them is definitely a btrfs problem. The other may be btrfs,
> >or something in the block layer, or just broken hardware; it's hard to
> >tell from where I sit.
> >
> >Problem 1: ENOSPC on balance
> >
> > This has been going on since about March this year. I can
> >reasonably certainly recall 8-10 cases, possibly a number more. When
> >running a balance, the operation fails with ENOSPC when there's plenty
> >of space remaining unallocated. This happens on full balance, filtered
> >balance, and device delete. Other than the ENOSPC on balance, the FS
> >seems to work OK. It seems to be more prevalent on filesystems
> >converted from ext*. The first few or more reports of this didn't make
> >it to bugzilla, but a few of them since then have gone in.
> >
> >Problem 2: Unexplained zeroes
> >
> > Failure to mount. Transid failure, "expected xyz, have 0". Chris
> >looked at an early one of these (for Ke, on IRC) back in September
> >(the 27th -- sadly, the public IRC logs aren't there for it, but I can
> >supply a copy of the private log). He rapidly came to the conclusion
> >that it was something bad going on with TRIM, replacing some blocks
> >with zeroes. Since then, I've seen a bunch of these coming past on
> >IRC. It seems to be a 3.17 thing. I can successfully predict the
> >presence of an SSD and -odiscard from the "have 0". I've successfully
> >persuaded several people to put this into bugzilla and capture
> >btrfs-images. btrfs recover doesn't generally seem to be helpful in
> >recovering data.
> >
> >
> > I think Josef had problem 1 in his sights, but I don't know if
> >additional images or reports are helpful at this point. For problem 2,
> >there's obviously something bad going on, but there's not much else to
> >go on -- and the inability to recover data isn't good.
> >
> > For each of these, what more information should I be trying to
> >collect from any future reporters?
> >
> >
>
> So for #2 I've been looking at that the last two weeks. I'm always
> paranoid we're screwing up one of our data integrity sort of things,
> either not waiting on IO to complete properly or something like
> that. I've built a dm target to be as evil as possible and have been
> running it trying to make bad things happen. I got slightly side
> tracked since my stress test exposed a bug in the tree log stuff an
> csums which I just fixed. Now that I've fixed that I'm going back
> to try and make the "expected blah, have 0" type errors happen.
I've searched the bugzilla archive and found the two reports that I
know of (87061 and 87021); I couldn't see any others. I've requested
more information on both -- nothing obviously in common, except SSD
and (probably) discard. I tried to tag them both with "trim" for easy
finding, but that seems to have been lost somewhere. I'll try that
again when I get home this evening and have access to my password.
> As for the ENOSPC I keep meaning to look into it and I keep getting
> distracted with other more horrible things. Ideally I'd like to
> reproduce it myself, so more info on that front would be good, like
> do all reports use RAID/compression/some other odd set of features?
> Thanks for taking care of this stuff Hugo, #2 is the worst one and
> I'd like to be absolutely sure it's not our bug, once I'm happy we
> aren't I'll look at the balance thing.
OK, good to know you're on both of these. I think the "easy"
solution to reproduce the ENOSPC is to convert an ext4 filesystem. It
doesn't seem to be a unique characteristic, but it is a frequent
correlation. We had another one today, after an FS conversion -- I've
asked them to attach a btrfs-image dump and the enospc_debug log to
the bugzilla report.
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- 2 + 2 = 5, for sufficiently large values of 2. ---
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Two persistent problems
2014-11-14 22:00 ` Josef Bacik
2014-11-17 10:59 ` Konstantin
2014-11-17 11:10 ` Hugo Mills
@ 2014-11-26 18:35 ` Marc Joliet
2 siblings, 0 replies; 6+ messages in thread
From: Marc Joliet @ 2014-11-26 18:35 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 3569 bytes --]
Am Fri, 14 Nov 2014 17:00:26 -0500
schrieb Josef Bacik <jbacik@fb.com>:
> On 11/14/2014 04:51 PM, Hugo Mills wrote:
> > Chris, Josef, anyone else who's interested,
> >
> > On IRC, I've been seeing reports of two persistent unsolved
> > problems. Neither is showing up very often, but both have turned up
> > often enough to indicate that there's something specific going on
> > worthy of investigation.
> >
> > One of them is definitely a btrfs problem. The other may be btrfs,
> > or something in the block layer, or just broken hardware; it's hard to
> > tell from where I sit.
> >
> > Problem 1: ENOSPC on balance
> >
> > This has been going on since about March this year. I can
> > reasonably certainly recall 8-10 cases, possibly a number more. When
> > running a balance, the operation fails with ENOSPC when there's plenty
> > of space remaining unallocated. This happens on full balance, filtered
> > balance, and device delete. Other than the ENOSPC on balance, the FS
> > seems to work OK. It seems to be more prevalent on filesystems
> > converted from ext*. The first few or more reports of this didn't make
> > it to bugzilla, but a few of them since then have gone in.
> >
> > Problem 2: Unexplained zeroes
> >
> > Failure to mount. Transid failure, "expected xyz, have 0". Chris
> > looked at an early one of these (for Ke, on IRC) back in September
> > (the 27th -- sadly, the public IRC logs aren't there for it, but I can
> > supply a copy of the private log). He rapidly came to the conclusion
> > that it was something bad going on with TRIM, replacing some blocks
> > with zeroes. Since then, I've seen a bunch of these coming past on
> > IRC. It seems to be a 3.17 thing. I can successfully predict the
> > presence of an SSD and -odiscard from the "have 0". I've successfully
> > persuaded several people to put this into bugzilla and capture
> > btrfs-images. btrfs recover doesn't generally seem to be helpful in
> > recovering data.
> >
> >
> > I think Josef had problem 1 in his sights, but I don't know if
> > additional images or reports are helpful at this point. For problem 2,
> > there's obviously something bad going on, but there's not much else to
> > go on -- and the inability to recover data isn't good.
> >
> > For each of these, what more information should I be trying to
> > collect from any future reporters?
> >
> >
>
> So for #2 I've been looking at that the last two weeks. I'm always
> paranoid we're screwing up one of our data integrity sort of things,
> either not waiting on IO to complete properly or something like that.
> I've built a dm target to be as evil as possible and have been running
> it trying to make bad things happen. I got slightly side tracked since
> my stress test exposed a bug in the tree log stuff an csums which I just
> fixed. Now that I've fixed that I'm going back to try and make the
> "expected blah, have 0" type errors happen.
Just a quick question from a user: does Filipe's patch "Btrfs: fix race between
fs trimming and block group remove/allocation" fix this? Judging by the commit
message, it looks like it. If so, can you say whether it will make it into
3.17.x?
Maybe I'm being overly paranoid, but I stuck with 3.16.7 because of this. (I
mean, I have backups, but there's no need to provoke a situation where I will
need them ;-) .)
--
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread