Re: Lots of trouble hanging when rm files with many extents

Re: Lots of trouble hanging when rm files with many extents

[Tomasz Chmielewski]

> but I am not sure if that was the right thing to do.  This is
> producing OOM issues and leading to system crashes (including
> eventual panics) with such alarming frequency that I wonder if
> perhaps there is something different about my setup than others.

FYI, I'm seeing similar / random hangs on systems with btrfs,
compress-force=zlib, qgroups enabled. The systems have 32 GB RAM.
Sometimes, I can review old logs (after force reboot) and see OOM,
allocation failures and such.

The scenario there are snapshots, rsync with --inplace flag, which can
fragment files a lot.

I'll do some more tests with lots of extents to see if it's
reproducible here as well.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: Lots of trouble hanging when rm files with many extents

[John Goerzen]

On 10/20/2013 09:51 AM, Tomasz Chmielewski wrote:
>> but I am not sure if that was the right thing to do.  This is
>> producing OOM issues and leading to system crashes (including
>> eventual panics) with such alarming frequency that I wonder if
>> perhaps there is something different about my setup than others.
> FYI, I'm seeing similar / random hangs on systems with btrfs,
> compress-force=zlib, qgroups enabled. The systems have 32 GB RAM.
> Sometimes, I can review old logs (after force reboot) and see OOM,
> allocation failures and such.
>
> The scenario there are snapshots, rsync with --inplace flag, which can
> fragment files a lot.
>
> I'll do some more tests with lots of extents to see if it's
> reproducible here as well.
>
>
Interestingly, I've generally had qgroups enabled here as well, possibly 
on all of these systems.  Could that be the culprit?

rm with quotas can cause panic (was Re: Lots of trouble hanging when rm files with many extents)

[John Goerzen]

John Goerzen <jgoerzen <at> complete.org> writes:

> 
> On 10/20/2013 09:51 AM, Tomasz Chmielewski wrote:
> > I'll do some more tests with lots of extents to see if it's
> > reproducible here as well.
> >
> >
> Interestingly, I've generally had qgroups enabled here as well, possibly 
> on all of these systems.  Could that be the culprit?


Updating the subject here, because it seems increasingly likely that quotas
are the problem causing OOM panics.

I was able to reproduce the problem on a test filesystem with quotas
enabled.  After quotas were disabled and the system was rebooted, the
problem no longer occurred.  The dangerous combination seems to be many
extents plus quotas.

Sadly, quotas are the only way to have btrfs keep track of how much space
subvols are using, so it's very useful to be able to use that feature.

I also noted this at https://bugzilla.kernel.org/show_bug.cgi?id=63071

John

Re: Lots of trouble hanging when rm files with many extents

[Tomasz Chmielewski]

> Another approach could probably be to write a C program that writes
> 20GB to a file, then seeks to random places within it, writing chunks
> of data within them.
> 
> I could probably whip up such a thing if the debuggers need it.

That would be very useful, I guess!

-- 
Tomasz Chmielewski
http://wpkg.org

Re: Lots of trouble hanging when rm files with many extents

[Tomasz Chmielewski]

> > I'll do some more tests with lots of extents to see if it's
> > reproducible here as well.
> 
> Interestingly, I've generally had qgroups enabled here as well,
> possibly on all of these systems.  Could that be the culprit?

Possibly.

Is there a reliable way to create files with many extents? I've tried
playing with fallocate, but couldn't create files with too many extents
with it.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: Lots of trouble hanging when rm files with many extents

[Duncan]

Tomasz Chmielewski posted on Mon, 21 Oct 2013 13:36:44 +0900 as excerpted:

>> > I'll do some more tests with lots of extents to see if it's
>> > reproducible here as well.
>> 
>> Interestingly, I've generally had qgroups enabled here as well,
>> possibly on all of these systems.  Could that be the culprit?
> 
> Possibly.
> 
> Is there a reliable way to create files with many extents? I've tried
> playing with fallocate, but couldn't create files with too many extents
> with it.

While the absense of both qgroups and the problem here doesn't prove 
anything, it's a datapoint.  However, I /am/ a bit suspicious of them, as 
it just seems people keep reporting weird problems and the common thread 
often seems to be qgroups.  I'm not a coder let alone a kernel/btrfs 
coder so take this with a suitably sized grain of salt, but my intuition 
says qgroups add more stress to the system and likely some as yet 
unresolved race-points, and they raise the possibility of triggering 
otherwise unrelated problems that already exist, along with any issues 
they themselves might have.

That would fit with the many-thousands-of-extents issues here as well, 
since especially on spinning rust that's going to create all sorts of 
stress on the drive and filesystem and all sorts of potential to trigger 
race conditions.  If both of them increase stress and open up the 
critical sections during which races are possible, putting the two 
together is an effective torture-test likely indeed to bring out 
otherwise extremely small window race conditions that might otherwise 
very seldom trigger.

Meanwhile, on the extents issue I believe creating a large enough file 
and then writing into random locations within it, saving, writing some 
more, should do it, given btrfs' copy-on-write defaults.

Working with an almost full filesystem is likely to create worse 
fragmentation too, since at some point btrfs just gives up trying to 
create single-extent files and writes anywhere it can find space.

Finally, btrfs compression is said to look like fragmentation to 
filefrag.  I'm not sure whether it /acts/ like fragmentation or not, but 
any suitably large file that gets btrfs compression will look fragmented 
I believe because the compression blocks (128KB IIRC) are reported to 
filefrag as individual extents due to the way btrfs handles and tracks 
them.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Lots of trouble hanging when rm files with many extents

[John Goerzen]

On 10/20/2013 11:36 PM, Tomasz Chmielewski wrote:
>>> I'll do some more tests with lots of extents to see if it's
>>> reproducible here as well.
>> Interestingly, I've generally had qgroups enabled here as well,
>> possibly on all of these systems.  Could that be the culprit?
> Possibly.
>
> Is there a reliable way to create files with many extents? I've tried
> playing with fallocate, but couldn't create files with too many extents
> with it.
I could do it quite reliably by storing a VirtualBox VM hard drive image 
there, then running a Windows Update inside the VM.  Instant 
fragmentation ;-)

Another approach could probably be to write a C program that writes 20GB 
to a file, then seeks to random places within it, writing chunks of data 
within them.

I could probably whip up such a thing if the debuggers need it.

John

Lots of trouble hanging when rm files with many extents

[John Goerzen]

Hello folks,

I reported a bug here:

https://bugzilla.kernel.org/show_bug.cgi?id=63071

but I am not sure if that was the right thing to do.  This is producing OOM
issues and leading to system crashes (including eventual panics) with such
alarming frequency that I wonder if perhaps there is something different
about my setup than others.

In a nutshell, I originally made the mistake of storing a VirtualBox 40G
.vdi file on a btrfs partition (kernel 3.10).  I  managed to copy it to a
file marked chattr +C (I read the wiki *NOW*), but I couldn't delete the
original file -- rm hangs the system.

(I actually have the original file and another made with cp --reflink of it,
while trying to diagnose the issue.  I need to rm both of them)

rm starts out with lots of hard disk activity, then that stops.  I
eventually get the attached kernel BUG, then a little bit later it panics
due to out of memory.

But later I had the virtually identical problem on another system, this one
on just a 5GB file with about 25,000 extents (it was the restoresymtable
from restore(8) from the e2fs dump/restore pair).   That took down my
server, trying to delete it.

I would have expected btrfs to be more resilient by this point.  Any ideas?

Thanks,

John