Re: [Bug 199931] New: systemd/rtorrent file data corruption when using echo 3 >/proc/sys/vm/drop_caches

Andrew Morton
2018-06-05T20:03:31Z

(switched to email. Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Tue, 05 Jun 2018 18:01:36 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=199931
>
> Bug ID: 199931
> Summary: systemd/rtorrent file data corruption when using echo
> 3 >/proc/sys/vm/drop_caches

A long tale of woe here. Chris, do you think the pagecache corruption
is a general thing, or is it possible that btrfs is contributing?

Also, that 4.4 oom-killer regression sounds very serious.

> Product: Memory Management
> Version: 2.5
> Kernel Version: 4.14.33
> Hardware: All
> OS: Linux
> Tree: Mainline
> Status: NEW
> Severity: normal
> Priority: P1
> Component: Other
> Assignee: akpm@linux-foundation.org
> Reporter: bugzilla.kernel.org@plan9.de
> Regression: No
>
> We found that
>
> echo 3 >/proc/sys/vm/drop_caches
>
> causes file data corruption. We found this because we saw systemd journal
> corruption (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=897266) and
> tracked this to a cron job dropping caches every hour. The filesystem in use is
> btrfs, but I don't know if it only happens with this filesystem. btrfs scrub
> reports no problems, so this is not filesystem metdata corruption.
>
> Basically:
>
> # journalctl --verify
> [everything fine at this point]
> # echo 3 >/proc/sys/vm/drop_caches
> # journalctl --verify
> [journalctl now reporting corruption problems]
>
> This is not always reproducible, but when deleting our journal, creating log
> messages for a few hours and then doing the above manually has a ~50% chance of
> corrupting the journal.
>
> After investigating we found that rtorrent also suffers from corrupted
> downloads when using the above echo - basically, downloading torrents is fine,
> except when executing the above echo a few times during a download, after which
> rtorrent very likely reports a failed hash check.
>
> All of this is reproducible on two different boxes, so is unlikely to be a
> hardware issue.
>
> On one affected server we have over 50TB of files, many that have been created
> with the cronjob in place, and none of them are corrupted (we have md5sums of
> everything), so it seems to be related to something that systemd and rtorrent
> do, rather than a generic file corruption issue.
>
> I also was able to "cmp -l" two corrupted files with their correct version, and
> the corruption manifests itself as streaks of ~100-3000 zero bytes instead of
> the real data. The start offset sems random, but the end offset seems to be
> always aligned to a 4K offset - speculating without the hindrance of knowledge
> this feels like a race somewhere between writing to a mmapped area and freeing
> it, or so.
>
> Here is the output of cmp -l between a working and a corrupted file, for two
> files:
>
> http://data.plan9.de/01.cmp.txt
> http://data.plan9.de/02.cmp.txt
>
> We also have a mysql database with hundreds of gigabytes of writes per day on
> one server which also does not seem to suffer from any corruption.
>
> As for why we would do something silly as dropping the caches every hour (in a
> cronjob), we started doing this recently because after kernel 4.4, we got
> frequent OOM kills despite having gigabytes of available memory (e.g. 12GB in
> use, 20GB page cache and 16GB empty swap and bang, mysql gets killed). We found
> that that the debian 4.9 kernel is unusable, and 4.14 works, *iff* we use the
> above as an hourly cron job, so we did that, and afterwards run into
> rtorrent/journald corruption issues. Without the echo in place, mysql usually
> gets oom-killed after a few days of uptime.
>
> --
> You are receiving this mail because:
> You are the assignee for the bug.