Re: [aarch64] refcount_t: use-after-free in NFS with 64k pages

Cristian Marussi
2019-02-05T12:37:27Z

[-- Attachment #1: Type: text/plain, Size: 2197 bytes --]

Hi

On 05/02/2019 12:14, Benjamin Coddington wrote:
> On 5 Feb 2019, at 7:10, Cristian Marussi wrote:
>
>> Hi Ben
>>
>> On 05/02/2019 11:53, Benjamin Coddington wrote:
>>> Hello Cristian and Punit,
>>>
>>> Did you ever get to the bottom of this one? We just saw this on one
>>> run
>>> of our 4.18.0-era ppc64le, and I'm wondering if we ever found the
>>> root
>>> cause.
>>
>> unfortunately I stopped working actively on finding the root cause,
>> since I've
>> found a viable workaround that let us unblock our broken LTP runs.
>>
>> Setting wsize=65536 in NFS bootparams completely solves the issue with
>> 64k pages
>> (and does NOT break 4k either :D): this confirmed my hyp that there is
>> some sort
>> of race when accounting refcounts during the lifetime of nfs_page
>> structs which
>> leads to a misscounted refcount...but as I said I never looked back
>> into that
>> again (but never say never...)
>>
>> Hope this helps...
>
> Hmm, interesting..
>
> Will you share your reproducer with me? That will save me some time.

Sure.

My reproducer is the attached nfs_stress.sh script; when invoked with the
following params:

./nfs_stress.sh -w 10 -s 160000 -t 10

it leads to a crash within 10secs BUT ONLY with 64KB page Kconfig AND ONLY if
the above wsize workaround is NOT applied. (or the cleanup-code trick mentioned
in the emails) (the choice of the -s size parameter seemed sensible in determine
how quick it will die...)

BUT UNFORTUNATELY this was true ONLY when running on an AEMv8 FastModel (1-cpu
A53) (whose timings are much different from a real board); I've never been able
to reproduce reliably on real ARM64 silicon instead. (or on x86)
So all my debug and triage was made on the model once I was able to quickly
reproduce the same crash (and in fact the workaround worked then fine also on
silicon...)

On real silicon instead the only reproducer was a full LTP run: we had
consistent failures every night with the same exact refcount stacktrace (but
every time on a different LTP test as a trigger...being related to NFS activity
I suppose it's normal); since we applied the wsize workaround we saw no more
crashes.

Thanks

Regards

Cristian

>
> Ben
>

[-- Attachment #2: nfs_stress.sh --]
[-- Type: application/x-shellscript, Size: 2227 bytes --]