missing checksums on reboot

missing checksums on reboot

[Blake Lewis]

Hi, all, this is my first posting to the mailing list.  I am a
long-time file system guy who is just starting to take a serious
interest in btrfs.

My company's product uses btrfs for its backing storage.  We
maintain a log file to let us synchronize after reboots.  In
testing, we find that when the system panics and we read the
file after coming back up, we intermittently (but fairly often)
get "no csum found for inode X start Y" messages and from our
point of view, the log is corrupt.

Here are a few pertinent details:

1) When we see this, the device is always an SSD.
2) We reproduce it easily with 3.10 kernels but we have not
been able to reproduce it in 4.8.
3) The log file is opened with O_SYNC|O_DIRECT; its size is 128MB
and we are appending to it.
4) No other activity in the file system except a generated sequential
write workload
5) Panics are induced with "echo c > /proc/sysrq-trigger".

We filed a bug (https://bugzilla.kernel.org/show_bug.cgi?id=188051)
but I wanted to see if anyone here recognized these symptoms and
could point me in the right direction, especially since the problem
seems to have gone away in more recent releases.  We can't realistically
make our customers run newer kernels, so I want to understand the
nature of the bug and whether it might be able to fix it in the older
code.

Thanks for any guidance.

Blake

Re: missing checksums on reboot

[Lionel Bouton]

Hi,

Le 02/12/2016 à 20:07, Blake Lewis a écrit :
> Hi, all, this is my first posting to the mailing list.  I am a
> long-time file system guy who is just starting to take a serious
> interest in btrfs.
>
> My company's product uses btrfs for its backing storage.  We
> maintain a log file to let us synchronize after reboots.  In
> testing, we find that when the system panics and we read the
> file after coming back up, we intermittently (but fairly often)
> get "no csum found for inode X start Y" messages and from our
> point of view, the log is corrupt.
>
> Here are a few pertinent details:
>
> 1) When we see this, the device is always an SSD.
> 2) We reproduce it easily with 3.10 kernels 

Wow. That's ancient and certainly full of various bugs fixed since its
release.

> but we have not
> been able to reproduce it in 4.8.
> 3) The log file is opened with O_SYNC|O_DIRECT; its size is 128MB
> and we are appending to it.
> 4) No other activity in the file system except a generated sequential
> write workload
> 5) Panics are induced with "echo c > /proc/sysrq-trigger".
>
> We filed a bug (https://bugzilla.kernel.org/show_bug.cgi?id=188051)
> but I wanted to see if anyone here recognized these symptoms and
> could point me in the right direction, especially since the problem
> seems to have gone away in more recent releases.  We can't realistically
> make our customers run newer kernels,

Why ? If the kernel has a bug they have to update it to get the fix,
there's no way around it. Unless they use exotic software (proprietary
kernel modules typically) which won't work with later kernels, updating
to a simple patch or a much newer kernel doesn't make much of a
difference (it may be hard to package a recent kernel for an ancient
distribution, but it's definitely doable and usually transparent for its
users).

Btrfs and old kernels don't mix *at all*. I wouldn't advise using it in
any environment where updating the kernel to the latest mainline isn't
possible.

AFAIK from my reading of this mailing list :
- btrfs developers don't backport patches, at least not to anything but
the latest stable kernel version (currently 4.4.x).
- distributions aren't known to backport patches either (you should ask
your distribution support for specifics to make sure). Note : I'm not
sure why they compile btrfs support making users think it's OK to use it
as-is but don't actually support it.

I think 3.10 is pretty much unmaintained btrfs-wise. So much work as
been done on btrfs the last 3 years (3.10 is more than 3 years old) that
applying patches to a 3.10 distribution kernel is probably orders of
magnitude more complex than packaging a recent kernel for an old
distribution.

Best regards,

Lionel

Re: missing checksums on reboot

[Blake Lewis]

Well, 3.10 is what you get with the RHEL7.x distributions, so that's
why people are running it.
Apparently, it is "good enough" for many purposes.

My real goal here is to understand the scope of the bug and whether
any mitigation is
possible.  Of course, I don't expect anyone else to make a patch for
me (or even to accept
my patch), but if I knew what the bug is and what was done to fix it,
I'd be in a much better
position to decide what to do.  If anyone can shed any light on this,
I'd be very grateful.


On Fri, Dec 2, 2016 at 11:47 AM, Lionel Bouton
<lionel-subscription@bouton.name> wrote:
> Hi,
>
> Le 02/12/2016 à 20:07, Blake Lewis a écrit :
>> Hi, all, this is my first posting to the mailing list.  I am a
>> long-time file system guy who is just starting to take a serious
>> interest in btrfs.
>>
>> My company's product uses btrfs for its backing storage.  We
>> maintain a log file to let us synchronize after reboots.  In
>> testing, we find that when the system panics and we read the
>> file after coming back up, we intermittently (but fairly often)
>> get "no csum found for inode X start Y" messages and from our
>> point of view, the log is corrupt.
>>
>> Here are a few pertinent details:
>>
>> 1) When we see this, the device is always an SSD.
>> 2) We reproduce it easily with 3.10 kernels
>
> Wow. That's ancient and certainly full of various bugs fixed since its
> release.
>
>> but we have not
>> been able to reproduce it in 4.8.
>> 3) The log file is opened with O_SYNC|O_DIRECT; its size is 128MB
>> and we are appending to it.
>> 4) No other activity in the file system except a generated sequential
>> write workload
>> 5) Panics are induced with "echo c > /proc/sysrq-trigger".
>>
>> We filed a bug (https://bugzilla.kernel.org/show_bug.cgi?id=188051)
>> but I wanted to see if anyone here recognized these symptoms and
>> could point me in the right direction, especially since the problem
>> seems to have gone away in more recent releases.  We can't realistically
>> make our customers run newer kernels,
>
> Why ? If the kernel has a bug they have to update it to get the fix,
> there's no way around it. Unless they use exotic software (proprietary
> kernel modules typically) which won't work with later kernels, updating
> to a simple patch or a much newer kernel doesn't make much of a
> difference (it may be hard to package a recent kernel for an ancient
> distribution, but it's definitely doable and usually transparent for its
> users).
>
> Btrfs and old kernels don't mix *at all*. I wouldn't advise using it in
> any environment where updating the kernel to the latest mainline isn't
> possible.
>
> AFAIK from my reading of this mailing list :
> - btrfs developers don't backport patches, at least not to anything but
> the latest stable kernel version (currently 4.4.x).
> - distributions aren't known to backport patches either (you should ask
> your distribution support for specifics to make sure). Note : I'm not
> sure why they compile btrfs support making users think it's OK to use it
> as-is but don't actually support it.
>
> I think 3.10 is pretty much unmaintained btrfs-wise. So much work as
> been done on btrfs the last 3 years (3.10 is more than 3 years old) that
> applying patches to a 3.10 distribution kernel is probably orders of
> magnitude more complex than packaging a recent kernel for an old
> distribution.
>
> Best regards,
>
> Lionel

Re: missing checksums on reboot

[Duncan]

Blake Lewis posted on Fri, 02 Dec 2016 12:36:29 -0800 as excerpted:

> Well, 3.10 is what you get with the RHEL7.x distributions, so that's why
> people are running it.
> Apparently, it is "good enough" for many purposes.
> 
> My real goal here is to understand the scope of the bug and whether any
> mitigation is possible.  Of course, I don't expect anyone else to make a
> patch for me (or even to accept my patch), but if I knew what the bug is
> and what was done to fix it,
> I'd be in a much better position to decide what to do.  If anyone can
> shed any light on this,
> I'd be very grateful.

I'm going to try to make several points with this post, including a 
(perhaps too simplistic, but it works for me and apparently enough others 
to have a dedicated tool) suggestion for finding the problem, along with 
others basically agreeing with LB's reply but with a bit more 
background.  YMMV but it's posted in the hope that it is of help.

1) I am a list regular and btrfs using admin myself, but not a dev, so 
don't look here for the code-level stuff.

2) This is the btrfs development list, kernel level, not a distro list, 
and the viewpoints generally held here may or may not correspond to that 
of various distros and their support teams.

3) In particular, while btrfs has been officially out of experimental for 
some time now, on this list btrfs is held (by both btrfs devs and list-
regular users) to be still stabilizing, not fully stable and mature, as I 
normally describe it.  Both on this list and on the btrfs wiki (at 
https://btrfs.wiki.kernel.org ), the strong recommendation is to keep 
current on the kernel in particular, because bugs are still being found 
and fixed, and naturally, this list being development-focused, the view 
tends to be rather more forward leaning than particularly "enterprise-
stale^H^Hble" distros.

Also strongly recommended is keeping backups, tested and ready-to-use, 
because there /are/ still serious bugs being fixed, and sometimes the 
problems they trigger are, for non-devs at least, simply easiest to fix 
by blowing away the existing filesystem and starting over with a clean 
one and backups to recover from.

4) OTOH, some distros and other product vendors (including your company) 
obviously consider btrfs if not yet /entirely/ stable, stable /enough/ to 
build and ship product on.  While this is accepted as a questionable but 
already-in-the-wild-so-get-used-to-it position on the list, for kernels 
outside our normal support range (next point), the standard answer to 
users asking here for support is to say that while we recognize some 
distros support it on older kernels and with older btrfs userspace, we on 
this list tend to be forward looking and don't track what patches distros 
and vendors may have backported and which ones they haven't.  Thus, 
there's a 3-way choice they need to make, either (a) upgrade to something 
withing our recommended support range so we can best help, or (b) take 
the distro/vendor up on the support they offer and that the user may in 
fact be paying for, since they're best positioned to provide that support 
for older kernels they've done their own patching to, or (c) stay where 
they are and muddle thru the best they can with the limited support we 
/can/ offer -- we'll still do what we can, but honestly, the "impedance 
mismatch" with code that old is going to make it difficult to apply what 
limited support we can offer.

5) The versions we support best on the list, keeping in mind the above 
"current kernel" recommendation, are the two latest kernel series in 
either the current or LTS series.  On the current side, kernel 4.9 is 
very close to out now, so we like to see 4.8, tho 4.7 is still current 
enough that people on-list are likely to be able to reply sanely in terms 
of whether we recognize a bug and whether it's still current or has 
already been fixed.

On the LTS side, contradicting LB slightly, btrfs does try to backport 
bugfixes when we know they're needed in LTS series -- no effort is made 
to backport to non-LTS beyond the relatively short mainline non-LTS 
current kernel series support period, and as I said, they basically go 
out of support two kernels back.

The two most recent LTS kernel series are 4.4 and 4.1, with 3.18 before 
that.  Based on the two-most-recent policy, 4.4 is definitely still in 
focus, and we do still try for 4.1 as well, tho it's getting long enough 
in the tooth now that if the bug isn't recognized, an early question/
recommendation is going to be to try with a newer kernel.

The 3.18 LTS series actually ended up reasonably stable for btrfs, while 
4.4 may have taken a bit longer than normal to mature (in non-btrfs areas 
as well), so while it's back too far for much active support, it was 
working and is likely still working quite well for many.

3.16 was the LTS series before that, but that was a rough period for 
btrfs, and honestly, btrfs LTS support was new enough back then that we 
were only really trying to support the latest one, so 3.18 is really the 
practical horizon in terms of list support.  Before that there were some 
pretty bad bugs that nobody wants to even think about any more.

6) Meanwhile, what a lot of people don't realize is that until 3.12 (IIRC) 
stripped off the experimental label, btrfs remained officially 
experimental, with a pretty strong warning on both the kernel option and 
mkfs.btrfs on the user side.  In list support terms that's seriously 
ancient history now (think electricity in the 18th century maturity, it 
was a toy people could see was going to do great things one day and 
people were doing stuff with it, but it really /was/ experimental in 
anything even close to modern-day terms) and you're pretty much on your 
own.

Given that you're asking about 3.10, well... like I said, that's still 
btrfs experimental era, and I don't think I'm alone when I seriously 
question the sanity of anyone still attempting to support a product 
running btrfs on /that/!  If you're doing it, and people are willing to 
pay for it, well, I can't argue with that, but in all honesty, it's your 
customer's data you're putting at risk, and were I to see a product still 
running btrfs on a 3.10 kernel (or really, earlier than 3.18, for the 
reasons explained above, but 3.10 really /was/ still labeled experimental!
), I'd immediately mark everything that company sold as highly 
experimental and thus questionable, as well.  <shrug>  Just being honest.


So you can see why you're getting told to upgrade to /something/ semi-
recent, 3.18 LTS, at absolute minimum.  Some of those bugs in the early 
3.15-3.16 era tie my stomach in knots thinking about people still running 
those versions, they were that bad, even if in theory those bugs are long 
patched, by now.  Like I said, we were glad to see 3.18 LTS and it 
really /did/ end up surprisingly stable, up thru early 4.4 at least, when 
we stopped tracking it.

And FWIW, I /could/ be wrong here, but I /believe/ I saw that even Red 
Hat now actively recommends that people move off of btrfs on 3.10 era 
RHEL-7.x.  (At least there was a post on this list that I interpreted to 
that effect, tho if it actually affected me I'd be double-checking.)  I 
don't know what their actual support status is for those that don't.  You 
might want to look that up, because if it's true, it /would/ give your 
company a bit of an out in terms of supporting then-experimental btrfs, 
as well.


All that said, if you insist, again keeping in mind that I'm not a coder, 
so it's unsurprising that I don't have much in the way of specific 
commits to point you at, but at least here's a way to help you find them, 
yourself. =:^)

7) Incremental problem/fix bisect.  Recursively break the problem space 
roughly in half and test to see which half the problem/fix is in, then 
recurse by breaking that half in half and testing again.  You can use 
either git bisect, which has been popularized as a way for even non-
coders like me to nail down bugs (or in your case fixes) to specific 
commits, or perhaps first, to narrow down the range you need to git 
bisect, simply by doing a manual bisect of the release series between 
3.10 and 4.8, to find where the problem goes away, and then looking at 
that commit or the commits around that area to see what might have 
changed that broke, or in your case, fixed, the problem.

Bisect may be dumb and brute force, but it works surprisingly well, 
especially since git bisect automated most of the process, and as I said, 
it has allowed many non-coders to help in tracing and ultimately fixing 
their bugs.  I know it has worked that way for me.  And a modern git 
bisect better stays with release and then rc tags, then big merge points, 
as long as possible before diving down into individual commits, as well, 
making the incremental-bisect process a bit nicer and less "black-box" 
than it used to be, too.

Given what I said about 3.18 being a really good LTS in btrfs terms, I 
might suggest you start by testing it.  If it fixes the problem for you, 
then you can decide whether to try to push it as an upgrade, or try to 
bisect the problem further in ordered to properly backport the fix.  If 
it doesn't, of course the 4.1 and 4.4 LTS kernels are other major test 
points you can try.


So indeed, some of that was rehash, but with hopefully helpful additional 
detail now, and the bisect suggestion may be too simplistic.  But hope 
/some/ of it was helpful, anyway. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: missing checksums on reboot

[Blake Lewis]

Thanks, Duncan.  Your points are very reasonable and I will be doing
a bisect, as you suggest (though I don't know that the problem is fixed
in later releases, just that we couldn't reproduce it readily).  Just to be
clear, I am not looking for support.  I just hoped that perhaps this had
been a conspicuous enough problem that someone would be able to
give me a little history.


On Fri, Dec 2, 2016 at 9:58 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Blake Lewis posted on Fri, 02 Dec 2016 12:36:29 -0800 as excerpted:
>
>> Well, 3.10 is what you get with the RHEL7.x distributions, so that's why
>> people are running it.
>> Apparently, it is "good enough" for many purposes.
>>
>> My real goal here is to understand the scope of the bug and whether any
>> mitigation is possible.  Of course, I don't expect anyone else to make a
>> patch for me (or even to accept my patch), but if I knew what the bug is
>> and what was done to fix it,
>> I'd be in a much better position to decide what to do.  If anyone can
>> shed any light on this,
>> I'd be very grateful.
>
> I'm going to try to make several points with this post, including a
> (perhaps too simplistic, but it works for me and apparently enough others
> to have a dedicated tool) suggestion for finding the problem, along with
> others basically agreeing with LB's reply but with a bit more
> background.  YMMV but it's posted in the hope that it is of help.
>
> 1) I am a list regular and btrfs using admin myself, but not a dev, so
> don't look here for the code-level stuff.
>
> 2) This is the btrfs development list, kernel level, not a distro list,
> and the viewpoints generally held here may or may not correspond to that
> of various distros and their support teams.
>
> 3) In particular, while btrfs has been officially out of experimental for
> some time now, on this list btrfs is held (by both btrfs devs and list-
> regular users) to be still stabilizing, not fully stable and mature, as I
> normally describe it.  Both on this list and on the btrfs wiki (at
> https://btrfs.wiki.kernel.org ), the strong recommendation is to keep
> current on the kernel in particular, because bugs are still being found
> and fixed, and naturally, this list being development-focused, the view
> tends to be rather more forward leaning than particularly "enterprise-
> stale^H^Hble" distros.
>
> Also strongly recommended is keeping backups, tested and ready-to-use,
> because there /are/ still serious bugs being fixed, and sometimes the
> problems they trigger are, for non-devs at least, simply easiest to fix
> by blowing away the existing filesystem and starting over with a clean
> one and backups to recover from.
>
> 4) OTOH, some distros and other product vendors (including your company)
> obviously consider btrfs if not yet /entirely/ stable, stable /enough/ to
> build and ship product on.  While this is accepted as a questionable but
> already-in-the-wild-so-get-used-to-it position on the list, for kernels
> outside our normal support range (next point), the standard answer to
> users asking here for support is to say that while we recognize some
> distros support it on older kernels and with older btrfs userspace, we on
> this list tend to be forward looking and don't track what patches distros
> and vendors may have backported and which ones they haven't.  Thus,
> there's a 3-way choice they need to make, either (a) upgrade to something
> withing our recommended support range so we can best help, or (b) take
> the distro/vendor up on the support they offer and that the user may in
> fact be paying for, since they're best positioned to provide that support
> for older kernels they've done their own patching to, or (c) stay where
> they are and muddle thru the best they can with the limited support we
> /can/ offer -- we'll still do what we can, but honestly, the "impedance
> mismatch" with code that old is going to make it difficult to apply what
> limited support we can offer.
>
> 5) The versions we support best on the list, keeping in mind the above
> "current kernel" recommendation, are the two latest kernel series in
> either the current or LTS series.  On the current side, kernel 4.9 is
> very close to out now, so we like to see 4.8, tho 4.7 is still current
> enough that people on-list are likely to be able to reply sanely in terms
> of whether we recognize a bug and whether it's still current or has
> already been fixed.
>
> On the LTS side, contradicting LB slightly, btrfs does try to backport
> bugfixes when we know they're needed in LTS series -- no effort is made
> to backport to non-LTS beyond the relatively short mainline non-LTS
> current kernel series support period, and as I said, they basically go
> out of support two kernels back.
>
> The two most recent LTS kernel series are 4.4 and 4.1, with 3.18 before
> that.  Based on the two-most-recent policy, 4.4 is definitely still in
> focus, and we do still try for 4.1 as well, tho it's getting long enough
> in the tooth now that if the bug isn't recognized, an early question/
> recommendation is going to be to try with a newer kernel.
>
> The 3.18 LTS series actually ended up reasonably stable for btrfs, while
> 4.4 may have taken a bit longer than normal to mature (in non-btrfs areas
> as well), so while it's back too far for much active support, it was
> working and is likely still working quite well for many.
>
> 3.16 was the LTS series before that, but that was a rough period for
> btrfs, and honestly, btrfs LTS support was new enough back then that we
> were only really trying to support the latest one, so 3.18 is really the
> practical horizon in terms of list support.  Before that there were some
> pretty bad bugs that nobody wants to even think about any more.
>
> 6) Meanwhile, what a lot of people don't realize is that until 3.12 (IIRC)
> stripped off the experimental label, btrfs remained officially
> experimental, with a pretty strong warning on both the kernel option and
> mkfs.btrfs on the user side.  In list support terms that's seriously
> ancient history now (think electricity in the 18th century maturity, it
> was a toy people could see was going to do great things one day and
> people were doing stuff with it, but it really /was/ experimental in
> anything even close to modern-day terms) and you're pretty much on your
> own.
>
> Given that you're asking about 3.10, well... like I said, that's still
> btrfs experimental era, and I don't think I'm alone when I seriously
> question the sanity of anyone still attempting to support a product
> running btrfs on /that/!  If you're doing it, and people are willing to
> pay for it, well, I can't argue with that, but in all honesty, it's your
> customer's data you're putting at risk, and were I to see a product still
> running btrfs on a 3.10 kernel (or really, earlier than 3.18, for the
> reasons explained above, but 3.10 really /was/ still labeled experimental!
> ), I'd immediately mark everything that company sold as highly
> experimental and thus questionable, as well.  <shrug>  Just being honest.
>
>
> So you can see why you're getting told to upgrade to /something/ semi-
> recent, 3.18 LTS, at absolute minimum.  Some of those bugs in the early
> 3.15-3.16 era tie my stomach in knots thinking about people still running
> those versions, they were that bad, even if in theory those bugs are long
> patched, by now.  Like I said, we were glad to see 3.18 LTS and it
> really /did/ end up surprisingly stable, up thru early 4.4 at least, when
> we stopped tracking it.
>
> And FWIW, I /could/ be wrong here, but I /believe/ I saw that even Red
> Hat now actively recommends that people move off of btrfs on 3.10 era
> RHEL-7.x.  (At least there was a post on this list that I interpreted to
> that effect, tho if it actually affected me I'd be double-checking.)  I
> don't know what their actual support status is for those that don't.  You
> might want to look that up, because if it's true, it /would/ give your
> company a bit of an out in terms of supporting then-experimental btrfs,
> as well.
>
>
> All that said, if you insist, again keeping in mind that I'm not a coder,
> so it's unsurprising that I don't have much in the way of specific
> commits to point you at, but at least here's a way to help you find them,
> yourself. =:^)
>
> 7) Incremental problem/fix bisect.  Recursively break the problem space
> roughly in half and test to see which half the problem/fix is in, then
> recurse by breaking that half in half and testing again.  You can use
> either git bisect, which has been popularized as a way for even non-
> coders like me to nail down bugs (or in your case fixes) to specific
> commits, or perhaps first, to narrow down the range you need to git
> bisect, simply by doing a manual bisect of the release series between
> 3.10 and 4.8, to find where the problem goes away, and then looking at
> that commit or the commits around that area to see what might have
> changed that broke, or in your case, fixed, the problem.
>
> Bisect may be dumb and brute force, but it works surprisingly well,
> especially since git bisect automated most of the process, and as I said,
> it has allowed many non-coders to help in tracing and ultimately fixing
> their bugs.  I know it has worked that way for me.  And a modern git
> bisect better stays with release and then rc tags, then big merge points,
> as long as possible before diving down into individual commits, as well,
> making the incremental-bisect process a bit nicer and less "black-box"
> than it used to be, too.
>
> Given what I said about 3.18 being a really good LTS in btrfs terms, I
> might suggest you start by testing it.  If it fixes the problem for you,
> then you can decide whether to try to push it as an upgrade, or try to
> bisect the problem further in ordered to properly backport the fix.  If
> it doesn't, of course the 4.1 and 4.4 LTS kernels are other major test
> points you can try.
>
>
> So indeed, some of that was rehash, but with hopefully helpful additional
> detail now, and the bisect suggestion may be too simplistic.  But hope
> /some/ of it was helpful, anyway. =:^)
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: missing checksums on reboot

[Chris Murphy]

On Fri, Dec 2, 2016 at 1:36 PM, Blake Lewis <blake@portworx.com> wrote:
> Well, 3.10 is what you get with the RHEL7.x distributions, so that's
> why people are running it.
> Apparently, it is "good enough" for many purposes.

Nope. Btrfs in RHEL 7 is a technology preview, and ...

"Technology Preview features are currently unsupported, may not be
functionally complete, and are not suitable for deployment in
production. "


> My real goal here is to understand the scope of the bug and whether
> any mitigation is
> possible.  Of course, I don't expect anyone else to make a patch for
> me (or even to accept
> my patch), but if I knew what the bug is and what was done to fix it,
> I'd be in a much better
> position to decide what to do.  If anyone can shed any light on this,
> I'd be very grateful.

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/diff/?id=v4.8.12&id2=v3.19&dt=2

There's more than 10000 lines changed between 3.19 and 4.8.12. You're
in the right place, but it's a shot in the dark if any developer
remembers the particular problem you're having compared to the massive
pile of bugs fixed and changes made since then. The RHEL 3.10 kernel
as I understand it has v3.19 Btrfs code. You might test with a 4.1.36
kernel to see if your problem reproduces. There was some O_DIRECTbug
fixes a while back but I have no idea what kernel versions those were.

If the fix is relatively recent, it simply may not have been
backported very far because kernels go EOL and backporting just
doesn't happen, not that it can't be done. So you might search the
list for O_DIRECT, and see if anything looks familiar. If that doesn't
pan out you can start doing simple bisects with longterm kernel.org
kernels, starting with the oldest (which is where I got 4.1.36) and
see if you can find a commit that fixes the problem, and then if it
applies cleanly to the much older kernel you're using.



-- 
Chris Murphy