From: Liu Bo <liub.liubo@gmail.com>
To: Mitch Harder <mitch.harder@sabayonlinux.org>
Cc: linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: Btrfs Intermittent ENOSPC Issues
Date: Wed, 01 Aug 2012 20:40:52 +0800 [thread overview]
Message-ID: <501923D4.5070607@gmail.com> (raw)
In-Reply-To: <CAKcLGm-JYUUTc+NodFHFi2SLpE7=JSwRigygt5x8xgy_bYzzoQ@mail.gmail.com>
On 08/01/2012 03:37 AM, Mitch Harder wrote:
> I've been working on running down intermittent ENOSPC issues.
>
> I can only seem to replicate ENOSPC errors when running zlib
> compression. However, I have been seeing similar ENOSPC errors to a
> lesser extent when playing with the LZ4HC patches.
>
> I apologize for not following up on this sooner, but I had drifted
> away from using zlib, and didn't notice there was still an issue.
>
> My test case involves un-archiving linux git sources to a freshly
> formatted btrfs partition, mounted with compress-force=zlib. I am
> using a 16 GB partition on a 250 GB Western Digital SATA Hard Disk.
> My current kernel is x86_64 linux-3.5.0 merged with Chris' for-linus
> branch (for 3.6_rc). This includes Josef's "Btrfs: flush delayed
> inodes if we're short on space" patch.
>
> I haven't isolated a root cause, but here's the feedback I have so far.
>
> (1) My test case won't generate ENOSPC issues with lzo compression or
> no compression.
>
> (2) I've inserted some trace_printk debugging statements to trace
> back the call stack, and the ENOSPC errors only seem to occur on a new
> transaction: vfs_create -> btrfs_create -> btrfs_start_transaction ->
> start_transaction -> btrfs_block_rsv_add -> reserve_metadata_bytes.
>
> (3) The ENOSPC condition will usually clear in a few seconds,
> allowing writes to proceed.
>
> (4) I've added a loop to the reserve_metadata_bytes() function to
> loop back with 'flush_state = FLUSH_DELALLOC (1)' for 1024 retries.
> This reduces and/or eliminates the ENOSPC errors, as if we're waiting
> on something else that is trying to complete.
>
> (5) I've been heavily debugging the reserve_metadata_bytes()
> function, and I'm seeing problems with the way
> space_info->bytes_may_use is handled. The space_info->bytes_may_use
> value is important in determining if we're in an over-commit state.
> But space_info->bytes_may_use value is often increased arbitrarily
> without any mechanism for correcting the value. Subsequently,
> space_info->bytes_may_use quickly increases in size to the point where
> we are always in fallback allocation as if we're overcommitted. In my
> trials, it was hard to capture a point where space_info->bytes_may_use
> wasn't larger than the available size.
>
Interesting results.
IIRC, space_info->bytes_may_use seems not to be arbitrarily increased:
Block_rsv wants NUM bytes
-> space_info's bytes_may_use += NUM
Block_rsv uses SOME bytes and release itself
-> space_info's bytes_may_use -= (NUM - SOME)
So IMO it is 'over-reserve' that causes ENOSPC.
Maybe we can try to find why more bytes need to be reserved with
compress=zlib/compress=LZ4HC.
thanks,
liubo
> (6) Even though reserve_metadata_bytes() is almost always in fallback
> overcommitted mode, it is still working pretty well, and I've
> developed the perception that the problem is something that needs to
> finish elsewhere.
>
> Sorry for not having a patch to fix the issue. I'll try to keep
> banging on it as time allows.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
next prev parent reply other threads:[~2012-08-01 12:40 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-07-31 19:37 Btrfs Intermittent ENOSPC Issues Mitch Harder
2012-08-01 12:40 ` Liu Bo [this message]
2012-08-27 20:44 ` Mitch Harder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=501923D4.5070607@gmail.com \
--to=liub.liubo@gmail.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=mitch.harder@sabayonlinux.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).