* Re: 3.10.11 scrub OOM crash
2013-11-09 23:48 3.10.11 scrub OOM crash Russell Coker
@ 2013-11-10 10:29 ` Duncan
2013-11-10 13:01 ` Russell Coker
0 siblings, 1 reply; 3+ messages in thread
From: Duncan @ 2013-11-10 10:29 UTC (permalink / raw)
To: linux-btrfs
Russell Coker posted on Sun, 10 Nov 2013 10:48:17 +1100 as excerpted:
> I've got an AMD64 system with 8G of RAM and 1G of swap. It runs as a
> home file server with 2*3TB disks in a RAID-1 array and a 120G SSD for
> root and /home. It also does some light desktop work (running KDE and
> web browsing with Chromium).
>
> When a btrfs scrub is run from cron the system gives an OOM and then
> locks up after apparently killing some processes (after a reboot I see
> syslog entries about some processes being killed - even though it didn't
> appear to kill X or anything the system is hung).
>
> The system really shouldn't have a OOM. For light desktop use a total
> of 8G of RAM and 1G of swap should be more than enough. Most of the
> time swap is hardly used and there are several gigs of RAM used for
> cache.
>
> The kernel is Debian package 3.10.11-1.
>
> This is the same system about which I reported the 3.11.5 kernel
> infinite loop bug. I had this crash on scrub issue before I upgraded to
> 3.11.5. I'm not certain that 3.11.5 fixed the crash on scrub problem,
> maybe 3.11.5 just crashed the system before it could be up for long
> enough.
Do you have disk quotas (btrfs qgroups) enabled? There's a known current
problem with some really nasty memory usage and/or leaks somewhere with
them enabled, and even 16-gig systems are often not enough to avoid OOMs
in certain cases with qgroups enabled. As I'm sure you know given that
previous experience, btrfs remains in general classified as experimental,
but qgroups are known to make things MUCH worse currently and are
definitely negative-recommended. If at all possible, disable quotes on
btrfs and reboot (without a reboot, some qgroups structures remain in
memory and the problems remain with them). If your use-case requires
quotas, the STRONG recommendation is to use a different filesystem where
they're considered stable, at this point.
If it's not quotas/qgroups, I'm not sure, and simply fall back to the
general recommendations on the wiki, etc.
Meanwhile, the threads getting the OOMs are going to be btrfs-worker
threads. Killing them leaves open I/O and the locks for that I/O won't
get cleaned up, thus effectively stalling I/O for anything on that btrfs
volume (AFAIK the entire volume, not just that subvolume), tho I'm not
sure whether it'll affect all I/O (on other btrfs volumes and non-btrfs
filesystems) or not. Existing tasks will continue to operate as long as
they don't do any I/O to the locked-up volumes, and from my experience
with similar lockups under somewhat different circumstances (and with a
btrfs rootfs but mounted read-only, so it's not directly affected), if
something's already in cache you can read it, but you can't write
anything out to the affected filesystem, and if you try to read anything
in that's not already cached, the process trying to do that read will
lockup as well.
Which generally means that altho the system often isn't entirely dead,
it's running pretty crippled, and depending on where the actual lockup is
(especially if it's on root, but /home tends to be bad too, and if
they're on the same btrfs filesystem...), it can appear dead.
And while to the extent possible a controlled shut down and after that,
magic-sysrequests, does appear to safely save data on non-affected
volumes (either non-btrfs or entirely separate btrfs filesystems, again,
subvolumes on the same filesystem are locked up too I believe), since
writes to the affected filesystem are locked up, it won't properly
unmount or remount-read-only, so for it you gotta take the hard shutdown
and hope there's not too much damage. (Tho of course with btrfs being
experimental, if you're running it without tested backups to restore from
if worse comes to worse, you're doing it wrong, which means that any
damage there might be simply means restoring from that backup, at worst.)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 3+ messages in thread