btrfs-balance causes system-freeze on full disk

btrfs-balance causes system-freeze on full disk

[Jakob Schürz]

Hi there!

Is it possible, what i've recognized now. My system (debian) runs on
btrfs, and i have a lot of snapshots on my hard-disk.
Since some days my system freezes totally. I recognized, it always
happens during btrfs-balance.

So i deleted some of the old snapshots and tried another balance-run.
Nothing happened... No system-freeze.

System-freeze means: No Keyboard-action. The Mouse is frozen, the screen
is frozen, no magic-sysreq, no ssh-login.

Can btrfs cause such a freeze??

greez

jakob
-- 
http://xundeenergie.at
http://verkehrsloesungen.wordpress.com/
http://cogitationum.wordpress.com/

Re: btrfs-balance causes system-freeze on full disk

[Kyle Manna]

I had a number of similar btrfs balance crashes in the past few days,
but the disk wasn't full.  You should try tailing the system logs from
a remote machine when it happens. You'll likely see some bug info
before the system dies and becomes unusable.

The issue I encountered is described @
https://bugzilla.kernel.org/show_bug.cgi?id=105681
ᐧ

On Wed, Oct 21, 2015 at 12:38 PM, Jakob Schürz
<wertstoffe@nurfuerspam.de> wrote:
> Hi there!
>
> Is it possible, what i've recognized now. My system (debian) runs on
> btrfs, and i have a lot of snapshots on my hard-disk.
> Since some days my system freezes totally. I recognized, it always
> happens during btrfs-balance.
>
> So i deleted some of the old snapshots and tried another balance-run.
> Nothing happened... No system-freeze.
>
> System-freeze means: No Keyboard-action. The Mouse is frozen, the screen
> is frozen, no magic-sysreq, no ssh-login.
>
> Can btrfs cause such a freeze??
>
> greez
>
> jakob
> --
> http://xundeenergie.at
> http://verkehrsloesungen.wordpress.com/
> http://cogitationum.wordpress.com/
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-balance causes system-freeze on full disk

[Jakob Schürz]

Thx!

It seems to be the same bug.

Am 2015-10-21 um 22:51 schrieb Kyle Manna:
> I had a number of similar btrfs balance crashes in the past few days,
> but the disk wasn't full.  You should try tailing the system logs from
> a remote machine when it happens. You'll likely see some bug info
> before the system dies and becomes unusable.
> 
> The issue I encountered is described @
> https://bugzilla.kernel.org/show_bug.cgi?id=105681
> ᐧ
> 
> On Wed, Oct 21, 2015 at 12:38 PM, Jakob Schürz
> <wertstoffe@nurfuerspam.de> wrote:
>> Hi there!
>>
>> Is it possible, what i've recognized now. My system (debian) runs on
>> btrfs, and i have a lot of snapshots on my hard-disk.
>> Since some days my system freezes totally. I recognized, it always
>> happens during btrfs-balance.
>>
>> So i deleted some of the old snapshots and tried another balance-run.
>> Nothing happened... No system-freeze.
>>
>> System-freeze means: No Keyboard-action. The Mouse is frozen, the screen
>> is frozen, no magic-sysreq, no ssh-login.
>>
>> Can btrfs cause such a freeze??
>>
>> greez
>>
>> jakob
>> --
>> http://xundeenergie.at
>> http://verkehrsloesungen.wordpress.com/
>> http://cogitationum.wordpress.com/
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
http://xundeenergie.at
http://verkehrsloesungen.wordpress.com/
http://cogitationum.wordpress.com/

Re: btrfs-balance causes system-freeze on full disk

[Duncan]

Kyle Manna posted on Wed, 21 Oct 2015 13:51:22 -0700 as excerpted:

> The issue I encountered is described @
> https://bugzilla.kernel.org/show_bug.cgi?id=105681

FWIW...

I won't try to deal with the issue reported there, but I can help clear 
something up that's mentioned on the bug[1].

The question (comment 5 and 6) refers to btrfs device usage output, for a 
three-device btrfs raid1, both data/metadata.  The question was why only 
two of the three devices listed a system chunk.

Btrfs raid1, unlike say mdraid1, is strictly pair-mirror, exactly two 
copies of the chunk, one each on two different devices.  More devices 
adds to the space available, not to the number of redundant copies.

As it happens, the two devices that got a copy of the system chunk were 
sdb and sdd, sdc didn't get a copy, as there are only two copies to 
distribute, no matter the number of devices in the raid1.


And as it happens, I've been personally interested in and thus following 
the roadmapped btrfs N-way-mirroring, the feature that would put a copy 
on all three devices, this being my most hotly anticipated btrfs feature 
since 3-way-mirroring is about the perfect balance between cost and 
reliability due to device redundancy, for me.

For quite some time now, a new N-way-mirroring feature has been on the 
roadmap, to be worked on after raid56 mode, as the planned implementation 
was to use some of the same code.  Raid56 mode is complete now, tho it 
took far longer than initially expected, so hopefully n-way-mirroring is 
already in development.  However, given the time raid56 took, 2-3 years 
of development, it's likely to be some time before n-way-mirroring 
actually appears.  And again, if it follows the pattern of other btrfs 
features, it'll take a couple kernel cycles after initial release to 
stabilize to actual usability, and a full year (five cycles) to stabilize 
to approximately the same maturity/stability as the rest of btrfs in 
general.

For raid56, nominally code-complete in 3.19, the last critical bug was in 
the early 4.1 code, fixed by 4.1 release.  But my recommendation has been 
to wait another couple cycles just to be sure nothing else "interesting" 
comes up, basically a full year, five kernel cycles, after nominal code-
complete release.  That would be 4.4...

Back to N-way-mirroring, assuming the work doesn't get delayed by 
something else, I'd EWAG (educated WAG) an 18 month to 2 year development 
time to nominally complete.  That would put initial release around 
4.7-4.9, actual usability at 4.9-4.11, and year-on stability at 4.12-4.14.

So altho we're nearing a year since raid56 nominal-completion, I don't 
expect N-way-mirroring code release for another year or so yet, don't 
expect it to be really usable for another five months (two kernel cycles) 
after that, and even then, wouldn't expect it to be as stable as the rest 
of btrfs for another further three kernels or so, thus putting actual 
reasonable stability (compared to the already stable 2-way raid1 code) 
two years out...

So it's coming, and at least now it's close enough there's /some/ 
estimate of when it might be available, but it's going to be some time 
yet before I'd expect even nominal code-completion release, and some time 
after that before it reaches the stability benefit that I'm actually 
hotly anticipating the feature for.  Very roughly two years from now, tho 
I'd not be surprised at all to see that slide another six months to a 
year, and that's assuming nothing else shoves it out of the way, priority-
wise.

---
[1] I do have a kernel-bugs login but didn't want to bother logging in 
just to add the comment there, when I had just clicked a link here to get 
there, and could simply reply here instead.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs-balance causes system-freeze on full disk

[Jakob Schürz]

[-- Attachment #1: Type: text/plain, Size: 2167 bytes --]

Found a new habit...

I have a lot Snapshots on my drive. (Take every 10 Min a new one, every
houry one, every day one, every system-update, every plugin after
external HD...)

If there are to much snapshots (didn't find out the count how much
exactly) balancing fails with a system-freeze.
if i delete some of my snapshots, balancing is working...

Maybe this is a hint for bugfixing!!

Do you think?

greez

Jakob

Am 2015-10-21 um 22:51 schrieb Kyle Manna:
> I had a number of similar btrfs balance crashes in the past few days,
> but the disk wasn't full.  You should try tailing the system logs from
> a remote machine when it happens. You'll likely see some bug info
> before the system dies and becomes unusable.
> 
> The issue I encountered is described @
> https://bugzilla.kernel.org/show_bug.cgi?id=105681
> ᐧ
> 
> On Wed, Oct 21, 2015 at 12:38 PM, Jakob Schürz
> <wertstoffe@nurfuerspam.de> wrote:
>> Hi there!
>>
>> Is it possible, what i've recognized now. My system (debian) runs on
>> btrfs, and i have a lot of snapshots on my hard-disk.
>> Since some days my system freezes totally. I recognized, it always
>> happens during btrfs-balance.
>>
>> So i deleted some of the old snapshots and tried another balance-run.
>> Nothing happened... No system-freeze.
>>
>> System-freeze means: No Keyboard-action. The Mouse is frozen, the screen
>> is frozen, no magic-sysreq, no ssh-login.
>>
>> Can btrfs cause such a freeze??
>>
>> greez
>>
>> jakob
>> --
>> http://xundeenergie.at
>> http://verkehrsloesungen.wordpress.com/
>> http://cogitationum.wordpress.com/
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
http://xundeenergie.at
http://verkehrsloesungen.wordpress.com/
http://cogitationum.wordpress.com/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: btrfs-balance causes system-freeze on full disk

[Filipe Manana]

On Tue, Oct 27, 2015 at 4:05 PM, Jakob Schürz <wertstoffe@nurfuerspam.de> wrote:
> Found a new habit...
>
> I have a lot Snapshots on my drive. (Take every 10 Min a new one, every
> houry one, every day one, every system-update, every plugin after
> external HD...)
>
> If there are to much snapshots (didn't find out the count how much
> exactly) balancing fails with a system-freeze.
> if i delete some of my snapshots, balancing is working...
>
> Maybe this is a hint for bugfixing!!

A patch set to fix this issue was already sent yesterday and tested
over the weekend by Stéphane:

http://thread.gmane.org/gmane.comp.file-systems.btrfs/49630

cheers

>
> Do you think?
>
> greez
>
> Jakob
>
> Am 2015-10-21 um 22:51 schrieb Kyle Manna:
>> I had a number of similar btrfs balance crashes in the past few days,
>> but the disk wasn't full.  You should try tailing the system logs from
>> a remote machine when it happens. You'll likely see some bug info
>> before the system dies and becomes unusable.
>>
>> The issue I encountered is described @
>> https://bugzilla.kernel.org/show_bug.cgi?id=105681
>> ᐧ
>>
>> On Wed, Oct 21, 2015 at 12:38 PM, Jakob Schürz
>> <wertstoffe@nurfuerspam.de> wrote:
>>> Hi there!
>>>
>>> Is it possible, what i've recognized now. My system (debian) runs on
>>> btrfs, and i have a lot of snapshots on my hard-disk.
>>> Since some days my system freezes totally. I recognized, it always
>>> happens during btrfs-balance.
>>>
>>> So i deleted some of the old snapshots and tried another balance-run.
>>> Nothing happened... No system-freeze.
>>>
>>> System-freeze means: No Keyboard-action. The Mouse is frozen, the screen
>>> is frozen, no magic-sysreq, no ssh-login.
>>>
>>> Can btrfs cause such a freeze??
>>>
>>> greez
>>>
>>> jakob
>>> --
>>> http://xundeenergie.at
>>> http://verkehrsloesungen.wordpress.com/
>>> http://cogitationum.wordpress.com/
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
> --
> http://xundeenergie.at
> http://verkehrsloesungen.wordpress.com/
> http://cogitationum.wordpress.com/
>



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: btrfs-balance causes system-freeze on full disk

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2255 bytes --]

On Tue, Oct 27, 2015 at 05:05:55PM +0100, Jakob Schürz wrote:
> Found a new habit...
> 
> I have a lot Snapshots on my drive. (Take every 10 Min a new one, every
> houry one, every day one, every system-update, every plugin after
> external HD...)
> 
> If there are to much snapshots (didn't find out the count how much
> exactly) balancing fails with a system-freeze.
> if i delete some of my snapshots, balancing is working...

   The whole machine comes to a halt, or just the balancing? If you
have lots of snapshots, balance can take *insane* amounts of time --
my big storage array, for example, takes about 60 seconds to balance a
data block group, and something like 4+ hours to balance some of the
metadata block groups. While it's "stuck" like that, it is actually
making progress, but it doesn't look like it.

   Hugo.

> Maybe this is a hint for bugfixing!!
> 
> Do you think?
> 
> greez
> 
> Jakob
> 
> Am 2015-10-21 um 22:51 schrieb Kyle Manna:
> > I had a number of similar btrfs balance crashes in the past few days,
> > but the disk wasn't full.  You should try tailing the system logs from
> > a remote machine when it happens. You'll likely see some bug info
> > before the system dies and becomes unusable.
> > 
> > The issue I encountered is described @
> > https://bugzilla.kernel.org/show_bug.cgi?id=105681
> > ᐧ
> > 
> > On Wed, Oct 21, 2015 at 12:38 PM, Jakob Schürz
> > <wertstoffe@nurfuerspam.de> wrote:
> >> Hi there!
> >>
> >> Is it possible, what i've recognized now. My system (debian) runs on
> >> btrfs, and i have a lot of snapshots on my hard-disk.
> >> Since some days my system freezes totally. I recognized, it always
> >> happens during btrfs-balance.
> >>
> >> So i deleted some of the old snapshots and tried another balance-run.
> >> Nothing happened... No system-freeze.
> >>
> >> System-freeze means: No Keyboard-action. The Mouse is frozen, the screen
> >> is frozen, no magic-sysreq, no ssh-login.
> >>
> >> Can btrfs cause such a freeze??
> >>
> >> greez
> >>
> >> jakob

-- 
Hugo Mills             | Darkling's First Law of Filesystems:
hugo@... carfax.org.uk | The user hates their data
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: btrfs-balance causes system-freeze on full disk

[Jakob Schürz]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

Am 2015-10-27 um 18:09 schrieb Hugo Mills:
> On Tue, Oct 27, 2015 at 05:05:55PM +0100, Jakob Schürz wrote:
>> Found a new habit...
>>
>> I have a lot Snapshots on my drive. (Take every 10 Min a new one, every
>> houry one, every day one, every system-update, every plugin after
>> external HD...)
>>
>> If there are to much snapshots (didn't find out the count how much
>> exactly) balancing fails with a system-freeze.
>> if i delete some of my snapshots, balancing is working...
> 
>    The whole machine comes to a halt, or just the balancing? 

The whole machine. Even magic-sysreq-keys are not reacting... I have to
reboot my machine using the power-button...

> If you
> have lots of snapshots, balance can take *insane* amounts of time --
> my big storage array, for example, takes about 60 seconds to balance a
> data block group, and something like 4+ hours to balance some of the
> metadata block groups. While it's "stuck" like that, it is actually
> making progress, but it doesn't look like it.

I know, balancint takes a huge amount of time. So my command is:

/bin/btrfs balance start -dusage=55 -v <mountpoint>

When balancing is running, the machine reactes a little slower than
normal... but in my case, the machine is freezing. completly and
totally... :(

greez jakob




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]