Issue found with kernel/net/sunrpc/xdr.c

Issue found with kernel/net/sunrpc/xdr.c

[Mark Young]

I was pointed to this mailing list by Brian Fields.

 We're currently seeing NFS data corruption, which we traced back to memory corruption that happens in the function _shift_data_right_pages in net/sunrpc/xdr.c.
 
 When we see the issue, we're running a 32bit os with systems running with more than 1GB of physical memory.   The errant behavior appears to be that two calls to kmap_atomic (on 32bit systems with highmem present) with the same physical address (on addresses within highmem)  will return two different vaddrs.    In our assessment, this confuses the memmove code into thinking that the two addresses are non-overlapping in spite of the fact that they are overlapping in physical space.  This, in turn, results in corruption.  
 
 A proposed solution to the problem would involve calling kmap_atomic only once in the case that the pgfrom and pgto are identical, and then re-using the resultant vaddr for both vto and vfrom.

Any insight on the issue or the proposed solution would be greatly appreciated.
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

RE: Issue found with kernel/net/sunrpc/xdr.c

[Myklebust, Trond]

> -----Original Message-----
> From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> owner@vger.kernel.org] On Behalf Of Mark Young
> Sent: Tuesday, August 27, 2013 5:17 PM
> To: linux-nfs@vger.kernel.org
> Cc: Matt Craighead
> Subject: Issue found with kernel/net/sunrpc/xdr.c
> 
> I was pointed to this mailing list by Brian Fields.
> 
>  We're currently seeing NFS data corruption, which we traced back to
> memory corruption that happens in the function _shift_data_right_pages in
> net/sunrpc/xdr.c.
> 
>  When we see the issue, we're running a 32bit os with systems running with
> more than 1GB of physical memory.   The errant behavior appears to be that
> two calls to kmap_atomic (on 32bit systems with highmem present) with the
> same physical address (on addresses within highmem)  will return two
> different vaddrs.    In our assessment, this confuses the memmove code into
> thinking that the two addresses are non-overlapping in spite of the fact that
> they are overlapping in physical space.  This, in turn, results in corruption.
> 
>  A proposed solution to the problem would involve calling kmap_atomic only
> once in the case that the pgfrom and pgto are identical, and then re-using
> the resultant vaddr for both vto and vfrom.
> 
> Any insight on the issue or the proposed solution would be greatly
> appreciated.

I'm fine with that solution. Mind sending a duly conformant patch w/ a signed-off-by line?

Thanks
  Trond

Re: Issue found with kernel/net/sunrpc/xdr.c

[J. Bruce Fields]

On Tue, Aug 27, 2013 at 10:01:38PM +0000, Myklebust, Trond wrote:
> > -----Original Message-----
> > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > owner@vger.kernel.org] On Behalf Of Mark Young
> > Sent: Tuesday, August 27, 2013 5:17 PM
> > To: linux-nfs@vger.kernel.org
> > Cc: Matt Craighead
> > Subject: Issue found with kernel/net/sunrpc/xdr.c
> > 
> > I was pointed to this mailing list by Brian Fields.
> > 
> >  We're currently seeing NFS data corruption, which we traced back to
> > memory corruption that happens in the function _shift_data_right_pages in
> > net/sunrpc/xdr.c.
> > 
> >  When we see the issue, we're running a 32bit os with systems running with
> > more than 1GB of physical memory.   The errant behavior appears to be that
> > two calls to kmap_atomic (on 32bit systems with highmem present) with the
> > same physical address (on addresses within highmem)  will return two
> > different vaddrs.    In our assessment, this confuses the memmove code into
> > thinking that the two addresses are non-overlapping in spite of the fact that
> > they are overlapping in physical space.  This, in turn, results in corruption.
> > 
> >  A proposed solution to the problem would involve calling kmap_atomic only
> > once in the case that the pgfrom and pgto are identical, and then re-using
> > the resultant vaddr for both vto and vfrom.
> > 
> > Any insight on the issue or the proposed solution would be greatly
> > appreciated.
> 
> I'm fine with that solution. Mind sending a duly conformant patch w/ a signed-off-by line?

I'm curious why we haven't seen it before: the code's done this for
years.  It looks to me like the two kmap_atomic()s, if they're
non-trivial, should always return different addresses.  And overlapping
copies are probably the normal case for this function.

Shouldn't anyone on 32 bit and high memory have seen filesystem
corruption?

Or maybe memmove is an architecture-specific implementation that happens
to handle left-to-right overlapping copies correctly on common
architectures?

--b.

RE: Issue found with kernel/net/sunrpc/xdr.c

[Matt Craighead]

> I'm curious why we haven't seen it before

I agree, it's slightly mysterious.  We're hitting the bug on a 32-bit ARM system with 1GB or 2GB of memory; surely that's not very far off the beaten path.


> Or maybe memmove is an architecture-specific implementation that happens to handle left-to-right overlapping copies correctly on common architectures?

It's architecture specific, e.g.: http://lxr.linux.no/linux+v3.8/arch/arm/lib/memmove.S

In this particular case, it decides that the memory is non-overlapping (by comparing the virtual addresses), so it incorrectly dispatches to memcpy().  At that point all bets are off in terms of the direction, chunk size, etc. of the copy.

I guess I'd typically expect memcpy() to stride through memory forwards rather than backwards though.  And since this function mandates "from < to", it seems like this would usually fail.


For x86: http://lxr.linux.no/linux+v3.8/arch/x86/lib/memcpy_32.c

In this case, it looks like it doesn't dispatch to memcpy().  It just determines forward/backwards and copies accordingly.  So it would work as long as the "from < to" property was preserved.

If I'm reading the code correctly, kmap_atomic appears to grow downward on x86:
http://lxr.linux.no/linux+v3.8/arch/x86/mm/highmem_32.c
http://lxr.linux.no/linux+v3.8/arch/x86/include/asm/fixmap.h
etc.

Since xdr.c calls kmap_atomic on *pgto first, then *pgfrom, the "from < to" property will therefore be preserved.


Therefore, I suspect you might be able to reproduce the bug on (32-bit) x86 by doing any of the following:
- swapping the order of those kmap_atomic/kunmap_atomic calls 
- modifying kmap_atomic to grow in the opposite direction
- modifying memmove() to dispatch to memcpy() when the virtual regions are non-overlapping

It's still possible that there is something else funny about how memory gets allocated in our setup that makes the bug more likely, but empirically, the bug didn't seem hard to hit.  We didn't need to copy very much data/very many files over NFS before we typically got corruption, and the bug happened on a variety of specific platforms.  It was (unsurprisingly) easier to reproduce with 2GB than with 1GB though.

-----Original Message-----
From: J. Bruce Fields [mailto:bfields@fieldses.org] 
Sent: Wednesday, August 28, 2013 9:33 AM
To: Myklebust, Trond
Cc: Mark Young; linux-nfs@vger.kernel.org; Matt Craighead
Subject: Re: Issue found with kernel/net/sunrpc/xdr.c

On Tue, Aug 27, 2013 at 10:01:38PM +0000, Myklebust, Trond wrote:
> > -----Original Message-----
> > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs- 
> > owner@vger.kernel.org] On Behalf Of Mark Young
> > Sent: Tuesday, August 27, 2013 5:17 PM
> > To: linux-nfs@vger.kernel.org
> > Cc: Matt Craighead
> > Subject: Issue found with kernel/net/sunrpc/xdr.c
> > 
> > I was pointed to this mailing list by Brian Fields.
> > 
> >  We're currently seeing NFS data corruption, which we traced back to 
> > memory corruption that happens in the function 
> > _shift_data_right_pages in net/sunrpc/xdr.c.
> > 
> >  When we see the issue, we're running a 32bit os with systems running with
> > more than 1GB of physical memory.   The errant behavior appears to be that
> > two calls to kmap_atomic (on 32bit systems with highmem present) 
> > with the same physical address (on addresses within highmem)  will return two
> > different vaddrs.    In our assessment, this confuses the memmove code into
> > thinking that the two addresses are non-overlapping in spite of the 
> > fact that they are overlapping in physical space.  This, in turn, results in corruption.
> > 
> >  A proposed solution to the problem would involve calling 
> > kmap_atomic only once in the case that the pgfrom and pgto are 
> > identical, and then re-using the resultant vaddr for both vto and vfrom.
> > 
> > Any insight on the issue or the proposed solution would be greatly 
> > appreciated.
> 
> I'm fine with that solution. Mind sending a duly conformant patch w/ a signed-off-by line?

I'm curious why we haven't seen it before: the code's done this for years.  It looks to me like the two kmap_atomic()s, if they're non-trivial, should always return different addresses.  And overlapping copies are probably the normal case for this function.

Shouldn't anyone on 32 bit and high memory have seen filesystem corruption?

Or maybe memmove is an architecture-specific implementation that happens to handle left-to-right overlapping copies correctly on common architectures?

--b.
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Re: Issue found with kernel/net/sunrpc/xdr.c

[J. Bruce Fields]

On Wed, Aug 28, 2013 at 08:18:10AM -0700, Matt Craighead wrote:
> > I'm curious why we haven't seen it before
> 
> I agree, it's slightly mysterious.  We're hitting the bug on a 32-bit ARM system with 1GB or 2GB of memory; surely that's not very far off the beaten path.

Yeah, it probably gets less testing, but I doubt you're the first to use
32-bit ARM with whatever minimum memory size is required to hit this.
Hm.  (Are you using krb5?  Which version of NFS?  And I guess this
manifested as the client seeing corrupted file data?)

Oh well, just curiosity on my part.  The fix is simple enough.

--b.

> 
> 
> > Or maybe memmove is an architecture-specific implementation that happens to handle left-to-right overlapping copies correctly on common architectures?
> 
> It's architecture specific, e.g.: http://lxr.linux.no/linux+v3.8/arch/arm/lib/memmove.S
> 
> In this particular case, it decides that the memory is non-overlapping (by comparing the virtual addresses), so it incorrectly dispatches to memcpy().  At that point all bets are off in terms of the direction, chunk size, etc. of the copy.
> 
> I guess I'd typically expect memcpy() to stride through memory forwards rather than backwards though.  And since this function mandates "from < to", it seems like this would usually fail.
> 
> 
> For x86: http://lxr.linux.no/linux+v3.8/arch/x86/lib/memcpy_32.c
> 
> In this case, it looks like it doesn't dispatch to memcpy().  It just determines forward/backwards and copies accordingly.  So it would work as long as the "from < to" property was preserved.
> 
> If I'm reading the code correctly, kmap_atomic appears to grow downward on x86:
> http://lxr.linux.no/linux+v3.8/arch/x86/mm/highmem_32.c
> http://lxr.linux.no/linux+v3.8/arch/x86/include/asm/fixmap.h
> etc.
> 
> Since xdr.c calls kmap_atomic on *pgto first, then *pgfrom, the "from < to" property will therefore be preserved.
> 
> 
> Therefore, I suspect you might be able to reproduce the bug on (32-bit) x86 by doing any of the following:
> - swapping the order of those kmap_atomic/kunmap_atomic calls 
> - modifying kmap_atomic to grow in the opposite direction
> - modifying memmove() to dispatch to memcpy() when the virtual regions are non-overlapping
> 
> It's still possible that there is something else funny about how memory gets allocated in our setup that makes the bug more likely, but empirically, the bug didn't seem hard to hit.  We didn't need to copy very much data/very many files over NFS before we typically got corruption, and the bug happened on a variety of specific platforms.  It was (unsurprisingly) easier to reproduce with 2GB than with 1GB though.
> 
> -----Original Message-----
> From: J. Bruce Fields [mailto:bfields@fieldses.org] 
> Sent: Wednesday, August 28, 2013 9:33 AM
> To: Myklebust, Trond
> Cc: Mark Young; linux-nfs@vger.kernel.org; Matt Craighead
> Subject: Re: Issue found with kernel/net/sunrpc/xdr.c
> 
> On Tue, Aug 27, 2013 at 10:01:38PM +0000, Myklebust, Trond wrote:
> > > -----Original Message-----
> > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs- 
> > > owner@vger.kernel.org] On Behalf Of Mark Young
> > > Sent: Tuesday, August 27, 2013 5:17 PM
> > > To: linux-nfs@vger.kernel.org
> > > Cc: Matt Craighead
> > > Subject: Issue found with kernel/net/sunrpc/xdr.c
> > > 
> > > I was pointed to this mailing list by Brian Fields.
> > > 
> > >  We're currently seeing NFS data corruption, which we traced back to 
> > > memory corruption that happens in the function 
> > > _shift_data_right_pages in net/sunrpc/xdr.c.
> > > 
> > >  When we see the issue, we're running a 32bit os with systems running with
> > > more than 1GB of physical memory.   The errant behavior appears to be that
> > > two calls to kmap_atomic (on 32bit systems with highmem present) 
> > > with the same physical address (on addresses within highmem)  will return two
> > > different vaddrs.    In our assessment, this confuses the memmove code into
> > > thinking that the two addresses are non-overlapping in spite of the 
> > > fact that they are overlapping in physical space.  This, in turn, results in corruption.
> > > 
> > >  A proposed solution to the problem would involve calling 
> > > kmap_atomic only once in the case that the pgfrom and pgto are 
> > > identical, and then re-using the resultant vaddr for both vto and vfrom.
> > > 
> > > Any insight on the issue or the proposed solution would be greatly 
> > > appreciated.
> > 
> > I'm fine with that solution. Mind sending a duly conformant patch w/ a signed-off-by line?
> 
> I'm curious why we haven't seen it before: the code's done this for years.  It looks to me like the two kmap_atomic()s, if they're non-trivial, should always return different addresses.  And overlapping copies are probably the normal case for this function.
> 
> Shouldn't anyone on 32 bit and high memory have seen filesystem corruption?
> 
> Or maybe memmove is an architecture-specific implementation that happens to handle left-to-right overlapping copies correctly on common architectures?
> 
> --b.
> -----------------------------------------------------------------------------------
> This email message is for the sole use of the intended recipient(s) and may contain
> confidential information.  Any unauthorized review, use, disclosure or distribution
> is prohibited.  If you are not the intended recipient, please contact the sender by
> reply email and destroy all copies of the original message.
> -----------------------------------------------------------------------------------

RE: Issue found with kernel/net/sunrpc/xdr.c

[Matt Craighead]

NFSv3 over UDP.  No Kerberos or any other authentication of any sort.

There are some non-default mount options, and perhaps some other things that might be unique to this particular test environment, but I don't see anything in particular that seems likely to perturb this code's behavior very much.

-----Original Message-----
From: J. Bruce Fields [mailto:bfields@fieldses.org] 
Sent: Wednesday, August 28, 2013 10:41 AM
To: Matt Craighead
Cc: Myklebust, Trond; Mark Young; linux-nfs@vger.kernel.org
Subject: Re: Issue found with kernel/net/sunrpc/xdr.c

On Wed, Aug 28, 2013 at 08:18:10AM -0700, Matt Craighead wrote:
> > I'm curious why we haven't seen it before
> 
> I agree, it's slightly mysterious.  We're hitting the bug on a 32-bit ARM system with 1GB or 2GB of memory; surely that's not very far off the beaten path.

Yeah, it probably gets less testing, but I doubt you're the first to use 32-bit ARM with whatever minimum memory size is required to hit this.
Hm.  (Are you using krb5?  Which version of NFS?  And I guess this manifested as the client seeing corrupted file data?)

Oh well, just curiosity on my part.  The fix is simple enough.

--b.

> 
> 
> > Or maybe memmove is an architecture-specific implementation that happens to handle left-to-right overlapping copies correctly on common architectures?
> 
> It's architecture specific, e.g.: 
> http://lxr.linux.no/linux+v3.8/arch/arm/lib/memmove.S
> 
> In this particular case, it decides that the memory is non-overlapping (by comparing the virtual addresses), so it incorrectly dispatches to memcpy().  At that point all bets are off in terms of the direction, chunk size, etc. of the copy.
> 
> I guess I'd typically expect memcpy() to stride through memory forwards rather than backwards though.  And since this function mandates "from < to", it seems like this would usually fail.
> 
> 
> For x86: http://lxr.linux.no/linux+v3.8/arch/x86/lib/memcpy_32.c
> 
> In this case, it looks like it doesn't dispatch to memcpy().  It just determines forward/backwards and copies accordingly.  So it would work as long as the "from < to" property was preserved.
> 
> If I'm reading the code correctly, kmap_atomic appears to grow downward on x86:
> http://lxr.linux.no/linux+v3.8/arch/x86/mm/highmem_32.c
> http://lxr.linux.no/linux+v3.8/arch/x86/include/asm/fixmap.h
> etc.
> 
> Since xdr.c calls kmap_atomic on *pgto first, then *pgfrom, the "from < to" property will therefore be preserved.
> 
> 
> Therefore, I suspect you might be able to reproduce the bug on (32-bit) x86 by doing any of the following:
> - swapping the order of those kmap_atomic/kunmap_atomic calls
> - modifying kmap_atomic to grow in the opposite direction
> - modifying memmove() to dispatch to memcpy() when the virtual regions 
> are non-overlapping
> 
> It's still possible that there is something else funny about how memory gets allocated in our setup that makes the bug more likely, but empirically, the bug didn't seem hard to hit.  We didn't need to copy very much data/very many files over NFS before we typically got corruption, and the bug happened on a variety of specific platforms.  It was (unsurprisingly) easier to reproduce with 2GB than with 1GB though.
> 
> -----Original Message-----
> From: J. Bruce Fields [mailto:bfields@fieldses.org]
> Sent: Wednesday, August 28, 2013 9:33 AM
> To: Myklebust, Trond
> Cc: Mark Young; linux-nfs@vger.kernel.org; Matt Craighead
> Subject: Re: Issue found with kernel/net/sunrpc/xdr.c
> 
> On Tue, Aug 27, 2013 at 10:01:38PM +0000, Myklebust, Trond wrote:
> > > -----Original Message-----
> > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs- 
> > > owner@vger.kernel.org] On Behalf Of Mark Young
> > > Sent: Tuesday, August 27, 2013 5:17 PM
> > > To: linux-nfs@vger.kernel.org
> > > Cc: Matt Craighead
> > > Subject: Issue found with kernel/net/sunrpc/xdr.c
> > > 
> > > I was pointed to this mailing list by Brian Fields.
> > > 
> > >  We're currently seeing NFS data corruption, which we traced back 
> > > to memory corruption that happens in the function 
> > > _shift_data_right_pages in net/sunrpc/xdr.c.
> > > 
> > >  When we see the issue, we're running a 32bit os with systems running with
> > > more than 1GB of physical memory.   The errant behavior appears to be that
> > > two calls to kmap_atomic (on 32bit systems with highmem present) 
> > > with the same physical address (on addresses within highmem)  will return two
> > > different vaddrs.    In our assessment, this confuses the memmove code into
> > > thinking that the two addresses are non-overlapping in spite of 
> > > the fact that they are overlapping in physical space.  This, in turn, results in corruption.
> > > 
> > >  A proposed solution to the problem would involve calling 
> > > kmap_atomic only once in the case that the pgfrom and pgto are 
> > > identical, and then re-using the resultant vaddr for both vto and vfrom.
> > > 
> > > Any insight on the issue or the proposed solution would be greatly 
> > > appreciated.
> > 
> > I'm fine with that solution. Mind sending a duly conformant patch w/ a signed-off-by line?
> 
> I'm curious why we haven't seen it before: the code's done this for years.  It looks to me like the two kmap_atomic()s, if they're non-trivial, should always return different addresses.  And overlapping copies are probably the normal case for this function.
> 
> Shouldn't anyone on 32 bit and high memory have seen filesystem corruption?
> 
> Or maybe memmove is an architecture-specific implementation that happens to handle left-to-right overlapping copies correctly on common architectures?
> 
> --b.
> ----------------------------------------------------------------------
> ------------- This email message is for the sole use of the intended 
> recipient(s) and may contain confidential information.  Any 
> unauthorized review, use, disclosure or distribution is prohibited.  
> If you are not the intended recipient, please contact the sender by 
> reply email and destroy all copies of the original message.
> ----------------------------------------------------------------------
> -------------

Re: Issue found with kernel/net/sunrpc/xdr.c

[J. Bruce Fields]

On Wed, Aug 28, 2013 at 08:47:57AM -0700, Matt Craighead wrote:
> NFSv3 over UDP.  No Kerberos or any other authentication of any sort.

Trond, shouldn't the size of the RPC header be entirely predictable in
this case?  I wonder why it was getting to xdr_shrink_bufhead.

--b.

> 
> There are some non-default mount options, and perhaps some other things that might be unique to this particular test environment, but I don't see anything in particular that seems likely to perturb this code's behavior very much.
> 
> -----Original Message-----
> From: J. Bruce Fields [mailto:bfields@fieldses.org] 
> Sent: Wednesday, August 28, 2013 10:41 AM
> To: Matt Craighead
> Cc: Myklebust, Trond; Mark Young; linux-nfs@vger.kernel.org
> Subject: Re: Issue found with kernel/net/sunrpc/xdr.c
> 
> On Wed, Aug 28, 2013 at 08:18:10AM -0700, Matt Craighead wrote:
> > > I'm curious why we haven't seen it before
> > 
> > I agree, it's slightly mysterious.  We're hitting the bug on a 32-bit ARM system with 1GB or 2GB of memory; surely that's not very far off the beaten path.
> 
> Yeah, it probably gets less testing, but I doubt you're the first to use 32-bit ARM with whatever minimum memory size is required to hit this.
> Hm.  (Are you using krb5?  Which version of NFS?  And I guess this manifested as the client seeing corrupted file data?)
> 
> Oh well, just curiosity on my part.  The fix is simple enough.
> 
> --b.
> 
> > 
> > 
> > > Or maybe memmove is an architecture-specific implementation that happens to handle left-to-right overlapping copies correctly on common architectures?
> > 
> > It's architecture specific, e.g.: 
> > http://lxr.linux.no/linux+v3.8/arch/arm/lib/memmove.S
> > 
> > In this particular case, it decides that the memory is non-overlapping (by comparing the virtual addresses), so it incorrectly dispatches to memcpy().  At that point all bets are off in terms of the direction, chunk size, etc. of the copy.
> > 
> > I guess I'd typically expect memcpy() to stride through memory forwards rather than backwards though.  And since this function mandates "from < to", it seems like this would usually fail.
> > 
> > 
> > For x86: http://lxr.linux.no/linux+v3.8/arch/x86/lib/memcpy_32.c
> > 
> > In this case, it looks like it doesn't dispatch to memcpy().  It just determines forward/backwards and copies accordingly.  So it would work as long as the "from < to" property was preserved.
> > 
> > If I'm reading the code correctly, kmap_atomic appears to grow downward on x86:
> > http://lxr.linux.no/linux+v3.8/arch/x86/mm/highmem_32.c
> > http://lxr.linux.no/linux+v3.8/arch/x86/include/asm/fixmap.h
> > etc.
> > 
> > Since xdr.c calls kmap_atomic on *pgto first, then *pgfrom, the "from < to" property will therefore be preserved.
> > 
> > 
> > Therefore, I suspect you might be able to reproduce the bug on (32-bit) x86 by doing any of the following:
> > - swapping the order of those kmap_atomic/kunmap_atomic calls
> > - modifying kmap_atomic to grow in the opposite direction
> > - modifying memmove() to dispatch to memcpy() when the virtual regions 
> > are non-overlapping
> > 
> > It's still possible that there is something else funny about how memory gets allocated in our setup that makes the bug more likely, but empirically, the bug didn't seem hard to hit.  We didn't need to copy very much data/very many files over NFS before we typically got corruption, and the bug happened on a variety of specific platforms.  It was (unsurprisingly) easier to reproduce with 2GB than with 1GB though.
> > 
> > -----Original Message-----
> > From: J. Bruce Fields [mailto:bfields@fieldses.org]
> > Sent: Wednesday, August 28, 2013 9:33 AM
> > To: Myklebust, Trond
> > Cc: Mark Young; linux-nfs@vger.kernel.org; Matt Craighead
> > Subject: Re: Issue found with kernel/net/sunrpc/xdr.c
> > 
> > On Tue, Aug 27, 2013 at 10:01:38PM +0000, Myklebust, Trond wrote:
> > > > -----Original Message-----
> > > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs- 
> > > > owner@vger.kernel.org] On Behalf Of Mark Young
> > > > Sent: Tuesday, August 27, 2013 5:17 PM
> > > > To: linux-nfs@vger.kernel.org
> > > > Cc: Matt Craighead
> > > > Subject: Issue found with kernel/net/sunrpc/xdr.c
> > > > 
> > > > I was pointed to this mailing list by Brian Fields.
> > > > 
> > > >  We're currently seeing NFS data corruption, which we traced back 
> > > > to memory corruption that happens in the function 
> > > > _shift_data_right_pages in net/sunrpc/xdr.c.
> > > > 
> > > >  When we see the issue, we're running a 32bit os with systems running with
> > > > more than 1GB of physical memory.   The errant behavior appears to be that
> > > > two calls to kmap_atomic (on 32bit systems with highmem present) 
> > > > with the same physical address (on addresses within highmem)  will return two
> > > > different vaddrs.    In our assessment, this confuses the memmove code into
> > > > thinking that the two addresses are non-overlapping in spite of 
> > > > the fact that they are overlapping in physical space.  This, in turn, results in corruption.
> > > > 
> > > >  A proposed solution to the problem would involve calling 
> > > > kmap_atomic only once in the case that the pgfrom and pgto are 
> > > > identical, and then re-using the resultant vaddr for both vto and vfrom.
> > > > 
> > > > Any insight on the issue or the proposed solution would be greatly 
> > > > appreciated.
> > > 
> > > I'm fine with that solution. Mind sending a duly conformant patch w/ a signed-off-by line?
> > 
> > I'm curious why we haven't seen it before: the code's done this for years.  It looks to me like the two kmap_atomic()s, if they're non-trivial, should always return different addresses.  And overlapping copies are probably the normal case for this function.
> > 
> > Shouldn't anyone on 32 bit and high memory have seen filesystem corruption?
> > 
> > Or maybe memmove is an architecture-specific implementation that happens to handle left-to-right overlapping copies correctly on common architectures?
> > 
> > --b.
> > ----------------------------------------------------------------------
> > ------------- This email message is for the sole use of the intended 
> > recipient(s) and may contain confidential information.  Any 
> > unauthorized review, use, disclosure or distribution is prohibited.  
> > If you are not the intended recipient, please contact the sender by 
> > reply email and destroy all copies of the original message.
> > ----------------------------------------------------------------------
> > -------------

Re: Issue found with kernel/net/sunrpc/xdr.c

[Myklebust, Trond]

T24gV2VkLCAyMDEzLTA4LTI4IGF0IDEyOjA4IC0wNDAwLCBKLiBCcnVjZSBGaWVsZHMgd3JvdGU6
DQo+IE9uIFdlZCwgQXVnIDI4LCAyMDEzIGF0IDA4OjQ3OjU3QU0gLTA3MDAsIE1hdHQgQ3JhaWdo
ZWFkIHdyb3RlOg0KPiA+IE5GU3YzIG92ZXIgVURQLiAgTm8gS2VyYmVyb3Mgb3IgYW55IG90aGVy
IGF1dGhlbnRpY2F0aW9uIG9mIGFueSBzb3J0Lg0KPiANCj4gVHJvbmQsIHNob3VsZG4ndCB0aGUg
c2l6ZSBvZiB0aGUgUlBDIGhlYWRlciBiZSBlbnRpcmVseSBwcmVkaWN0YWJsZSBpbg0KPiB0aGlz
IGNhc2U/ICBJIHdvbmRlciB3aHkgaXQgd2FzIGdldHRpbmcgdG8geGRyX3Nocmlua19idWZoZWFk
Lg0KDQpORlN2MyBwb3N0LW9wIGF0dHJpYnV0ZXMgYXJlIG9wdGlvbmFsIHRvIHJldHVybiwgc28g
d2hpbGUgdGhlIFJQQyBoZWFkZXINCm1heSBiZSBjb21wbGV0ZWx5IHByZWRpY3RhYmxlLCB0aGUg
TkZTIHJlcGx5IHBheWxvYWQgYmVmb3JlIGEgUkVBRCBvcg0KUkVBRERJUiBjYWxsIHRvIHhkcl9y
ZWFkX3BhZ2VzKCkgaXMgbm90Lg0KDQpDaGVlcnMsDQogIFRyb25kDQoNCi0tIA0KVHJvbmQgTXlr
bGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWts
ZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg==

Re: Issue found with kernel/net/sunrpc/xdr.c

[Myklebust, Trond]

[-- Attachment #1: Type: text/plain, Size: 2539 bytes --]

On Wed, 2013-08-28 at 08:18 -0700, Matt Craighead wrote:
> > I'm curious why we haven't seen it before
> 
> I agree, it's slightly mysterious.  We're hitting the bug on a 32-bit ARM system with 1GB or 2GB of memory; surely that's not very far off the beaten path.
> 
> 
> > Or maybe memmove is an architecture-specific implementation that happens to handle left-to-right overlapping copies correctly on common architectures?
> 
> It's architecture specific, e.g.: http://lxr.linux.no/linux+v3.8/arch/arm/lib/memmove.S
> 
> In this particular case, it decides that the memory is non-overlapping (by comparing the virtual addresses), so it incorrectly dispatches to memcpy().  At that point all bets are off in terms of the direction, chunk size, etc. of the copy.
> 
> I guess I'd typically expect memcpy() to stride through memory forwards rather than backwards though.  And since this function mandates "from < to", it seems like this would usually fail.
> 
> 
> For x86: http://lxr.linux.no/linux+v3.8/arch/x86/lib/memcpy_32.c
> 
> In this case, it looks like it doesn't dispatch to memcpy().  It just determines forward/backwards and copies accordingly.  So it would work as long as the "from < to" property was preserved.
> 
> If I'm reading the code correctly, kmap_atomic appears to grow downward on x86:
> http://lxr.linux.no/linux+v3.8/arch/x86/mm/highmem_32.c
> http://lxr.linux.no/linux+v3.8/arch/x86/include/asm/fixmap.h
> etc.
> 
> Since xdr.c calls kmap_atomic on *pgto first, then *pgfrom, the "from < to" property will therefore be preserved.
> 
> 
> Therefore, I suspect you might be able to reproduce the bug on (32-bit) x86 by doing any of the following:
> - swapping the order of those kmap_atomic/kunmap_atomic calls 
> - modifying kmap_atomic to grow in the opposite direction
> - modifying memmove() to dispatch to memcpy() when the virtual regions are non-overlapping
> 
> It's still possible that there is something else funny about how memory gets allocated in our setup that makes the bug more likely, but empirically, the bug didn't seem hard to hit.  We didn't need to copy very much data/very many files over NFS before we typically got corruption, and the bug happened on a variety of specific platforms.  It was (unsurprisingly) easier to reproduce with 2GB than with 1GB though.

Can you please test that the attached patch fixes the corruption?

Thanks
  Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-SUNRPC-Fix-memory-corruption-issue-on-32-bit-highmem.patch --]
[-- Type: text/x-patch; name="0001-SUNRPC-Fix-memory-corruption-issue-on-32-bit-highmem.patch", Size: 1848 bytes --]

From e9250a18b0a215ec5d970bded81fde01ee8b97aa Mon Sep 17 00:00:00 2001
From: Trond Myklebust <Trond.Myklebust@netapp.com>
Date: Wed, 28 Aug 2013 13:35:13 -0400
Subject: [PATCH] SUNRPC: Fix memory corruption issue on 32-bit highmem systems

Some architectures, such as ARM-32 do not return the same base address
when you call kmap_atomic() twice on the same page.
This causes problems for the memmove() call in the XDR helper routine
"_shift_data_right_pages()", since it defeats the detection of
overlapping memory ranges, and has been seen to corrupt memory.

The fix is to distinguish between the case where we're doing an
inter-page copy or not. In the former case of we know that the memory
ranges cannot possibly overlap, so we can additionally micro-optimise
by replacing memmove() with memcpy().

Reported-by: Mark Young <MYoung@nvidia.com>
Reported-by: Matt Craighead <mcraighead@nvidia.com>
Cc: Bruce Fields <bfields@fieldses.org>
Cc: stable@vger.kernel.org
Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---
 net/sunrpc/xdr.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
index 75edcfa..1504bb1 100644
--- a/net/sunrpc/xdr.c
+++ b/net/sunrpc/xdr.c
@@ -207,10 +207,13 @@ _shift_data_right_pages(struct page **pages, size_t pgto_base,
 		pgfrom_base -= copy;
 
 		vto = kmap_atomic(*pgto);
-		vfrom = kmap_atomic(*pgfrom);
-		memmove(vto + pgto_base, vfrom + pgfrom_base, copy);
+		if (*pgto != *pgfrom) {
+			vfrom = kmap_atomic(*pgfrom);
+			memcpy(vto + pgto_base, vfrom + pgfrom_base, copy);
+			kunmap_atomic(vfrom);
+		} else
+			memmove(vto + pgto_base, vto + pgfrom_base, copy);
 		flush_dcache_page(*pgto);
-		kunmap_atomic(vfrom);
 		kunmap_atomic(vto);
 
 	} while ((len -= copy) != 0);
-- 
1.8.3.1

RE: Issue found with kernel/net/sunrpc/xdr.c

[Matt Craighead]

VHJvbmQsIEkndmUganVzdCBub3cgY29tcGxldGVkIHRlc3Rpbmcgb2YgeW91ciBwYXRjaCAtLSBp
dCBkZWZpbml0ZWx5IGZpeGVzIHRoZSBjb3JydXB0aW9uIGlzc3VlLg0KDQpBbHNvLCB5b3VyIHBh
dGNoIGlzIGNsZWFuZXIgYW5kIHByb2JhYmx5IG1vcmUgZWZmaWNpZW50IHRoYW4gb3Vycy4NCg0K
TGV0IG1lIGFuZCBNYXJrIGtub3cgaWYgeW91IG5lZWQgYW55dGhpbmcgZWxzZSBmcm9tIHVzIG9u
IHRoaXMgaXNzdWUgLS0gdGhhbmtzIGZvciB5b3VyIGhlbHAhDQoNCi0tLS0tT3JpZ2luYWwgTWVz
c2FnZS0tLS0tDQpGcm9tOiBNeWtsZWJ1c3QsIFRyb25kIFttYWlsdG86VHJvbmQuTXlrbGVidXN0
QG5ldGFwcC5jb21dIA0KU2VudDogV2VkbmVzZGF5LCBBdWd1c3QgMjgsIDIwMTMgMTI6NTIgUE0N
ClRvOiBNYXR0IENyYWlnaGVhZA0KQ2M6IEouIEJydWNlIEZpZWxkczsgTWFyayBZb3VuZzsgbGlu
dXgtbmZzQHZnZXIua2VybmVsLm9yZw0KU3ViamVjdDogUmU6IElzc3VlIGZvdW5kIHdpdGgga2Vy
bmVsL25ldC9zdW5ycGMveGRyLmMNCg0KT24gV2VkLCAyMDEzLTA4LTI4IGF0IDA4OjE4IC0wNzAw
LCBNYXR0IENyYWlnaGVhZCB3cm90ZToNCj4gPiBJJ20gY3VyaW91cyB3aHkgd2UgaGF2ZW4ndCBz
ZWVuIGl0IGJlZm9yZQ0KPiANCj4gSSBhZ3JlZSwgaXQncyBzbGlnaHRseSBteXN0ZXJpb3VzLiAg
V2UncmUgaGl0dGluZyB0aGUgYnVnIG9uIGEgMzItYml0IEFSTSBzeXN0ZW0gd2l0aCAxR0Igb3Ig
MkdCIG9mIG1lbW9yeTsgc3VyZWx5IHRoYXQncyBub3QgdmVyeSBmYXIgb2ZmIHRoZSBiZWF0ZW4g
cGF0aC4NCj4gDQo+IA0KPiA+IE9yIG1heWJlIG1lbW1vdmUgaXMgYW4gYXJjaGl0ZWN0dXJlLXNw
ZWNpZmljIGltcGxlbWVudGF0aW9uIHRoYXQgaGFwcGVucyB0byBoYW5kbGUgbGVmdC10by1yaWdo
dCBvdmVybGFwcGluZyBjb3BpZXMgY29ycmVjdGx5IG9uIGNvbW1vbiBhcmNoaXRlY3R1cmVzPw0K
PiANCj4gSXQncyBhcmNoaXRlY3R1cmUgc3BlY2lmaWMsIGUuZy46IGh0dHA6Ly9seHIubGludXgu
bm8vbGludXgrdjMuOC9hcmNoL2FybS9saWIvbWVtbW92ZS5TDQo+IA0KPiBJbiB0aGlzIHBhcnRp
Y3VsYXIgY2FzZSwgaXQgZGVjaWRlcyB0aGF0IHRoZSBtZW1vcnkgaXMgbm9uLW92ZXJsYXBwaW5n
IChieSBjb21wYXJpbmcgdGhlIHZpcnR1YWwgYWRkcmVzc2VzKSwgc28gaXQgaW5jb3JyZWN0bHkg
ZGlzcGF0Y2hlcyB0byBtZW1jcHkoKS4gIEF0IHRoYXQgcG9pbnQgYWxsIGJldHMgYXJlIG9mZiBp
biB0ZXJtcyBvZiB0aGUgZGlyZWN0aW9uLCBjaHVuayBzaXplLCBldGMuIG9mIHRoZSBjb3B5Lg0K
PiANCj4gSSBndWVzcyBJJ2QgdHlwaWNhbGx5IGV4cGVjdCBtZW1jcHkoKSB0byBzdHJpZGUgdGhy
b3VnaCBtZW1vcnkgZm9yd2FyZHMgcmF0aGVyIHRoYW4gYmFja3dhcmRzIHRob3VnaC4gIEFuZCBz
aW5jZSB0aGlzIGZ1bmN0aW9uIG1hbmRhdGVzICJmcm9tIDwgdG8iLCBpdCBzZWVtcyBsaWtlIHRo
aXMgd291bGQgdXN1YWxseSBmYWlsLg0KPiANCj4gDQo+IEZvciB4ODY6IGh0dHA6Ly9seHIubGlu
dXgubm8vbGludXgrdjMuOC9hcmNoL3g4Ni9saWIvbWVtY3B5XzMyLmMNCj4gDQo+IEluIHRoaXMg
Y2FzZSwgaXQgbG9va3MgbGlrZSBpdCBkb2Vzbid0IGRpc3BhdGNoIHRvIG1lbWNweSgpLiAgSXQg
anVzdCBkZXRlcm1pbmVzIGZvcndhcmQvYmFja3dhcmRzIGFuZCBjb3BpZXMgYWNjb3JkaW5nbHku
ICBTbyBpdCB3b3VsZCB3b3JrIGFzIGxvbmcgYXMgdGhlICJmcm9tIDwgdG8iIHByb3BlcnR5IHdh
cyBwcmVzZXJ2ZWQuDQo+IA0KPiBJZiBJJ20gcmVhZGluZyB0aGUgY29kZSBjb3JyZWN0bHksIGtt
YXBfYXRvbWljIGFwcGVhcnMgdG8gZ3JvdyBkb3dud2FyZCBvbiB4ODY6DQo+IGh0dHA6Ly9seHIu
bGludXgubm8vbGludXgrdjMuOC9hcmNoL3g4Ni9tbS9oaWdobWVtXzMyLmMNCj4gaHR0cDovL2x4
ci5saW51eC5uby9saW51eCt2My44L2FyY2gveDg2L2luY2x1ZGUvYXNtL2ZpeG1hcC5oDQo+IGV0
Yy4NCj4gDQo+IFNpbmNlIHhkci5jIGNhbGxzIGttYXBfYXRvbWljIG9uICpwZ3RvIGZpcnN0LCB0
aGVuICpwZ2Zyb20sIHRoZSAiZnJvbSA8IHRvIiBwcm9wZXJ0eSB3aWxsIHRoZXJlZm9yZSBiZSBw
cmVzZXJ2ZWQuDQo+IA0KPiANCj4gVGhlcmVmb3JlLCBJIHN1c3BlY3QgeW91IG1pZ2h0IGJlIGFi
bGUgdG8gcmVwcm9kdWNlIHRoZSBidWcgb24gKDMyLWJpdCkgeDg2IGJ5IGRvaW5nIGFueSBvZiB0
aGUgZm9sbG93aW5nOg0KPiAtIHN3YXBwaW5nIHRoZSBvcmRlciBvZiB0aG9zZSBrbWFwX2F0b21p
Yy9rdW5tYXBfYXRvbWljIGNhbGxzIA0KPiAtIG1vZGlmeWluZyBrbWFwX2F0b21pYyB0byBncm93
IGluIHRoZSBvcHBvc2l0ZSBkaXJlY3Rpb24NCj4gLSBtb2RpZnlpbmcgbWVtbW92ZSgpIHRvIGRp
c3BhdGNoIHRvIG1lbWNweSgpIHdoZW4gdGhlIHZpcnR1YWwgcmVnaW9ucyBhcmUgbm9uLW92ZXJs
YXBwaW5nDQo+IA0KPiBJdCdzIHN0aWxsIHBvc3NpYmxlIHRoYXQgdGhlcmUgaXMgc29tZXRoaW5n
IGVsc2UgZnVubnkgYWJvdXQgaG93IG1lbW9yeSBnZXRzIGFsbG9jYXRlZCBpbiBvdXIgc2V0dXAg
dGhhdCBtYWtlcyB0aGUgYnVnIG1vcmUgbGlrZWx5LCBidXQgZW1waXJpY2FsbHksIHRoZSBidWcg
ZGlkbid0IHNlZW0gaGFyZCB0byBoaXQuICBXZSBkaWRuJ3QgbmVlZCB0byBjb3B5IHZlcnkgbXVj
aCBkYXRhL3ZlcnkgbWFueSBmaWxlcyBvdmVyIE5GUyBiZWZvcmUgd2UgdHlwaWNhbGx5IGdvdCBj
b3JydXB0aW9uLCBhbmQgdGhlIGJ1ZyBoYXBwZW5lZCBvbiBhIHZhcmlldHkgb2Ygc3BlY2lmaWMg
cGxhdGZvcm1zLiAgSXQgd2FzICh1bnN1cnByaXNpbmdseSkgZWFzaWVyIHRvIHJlcHJvZHVjZSB3
aXRoIDJHQiB0aGFuIHdpdGggMUdCIHRob3VnaC4NCg0KQ2FuIHlvdSBwbGVhc2UgdGVzdCB0aGF0
IHRoZSBhdHRhY2hlZCBwYXRjaCBmaXhlcyB0aGUgY29ycnVwdGlvbj8NCg0KVGhhbmtzDQogIFRy
b25kDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXIN
Cg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg0K
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NClRoaXMgZW1haWwgbWVzc2FnZSBpcyBmb3IgdGhl
IHNvbGUgdXNlIG9mIHRoZSBpbnRlbmRlZCByZWNpcGllbnQocykgYW5kIG1heSBjb250YWluDQpj
b25maWRlbnRpYWwgaW5mb3JtYXRpb24uICBBbnkgdW5hdXRob3JpemVkIHJldmlldywgdXNlLCBk
aXNjbG9zdXJlIG9yIGRpc3RyaWJ1dGlvbg0KaXMgcHJvaGliaXRlZC4gIElmIHlvdSBhcmUgbm90
IHRoZSBpbnRlbmRlZCByZWNpcGllbnQsIHBsZWFzZSBjb250YWN0IHRoZSBzZW5kZXIgYnkNCnJl
cGx5IGVtYWlsIGFuZCBkZXN0cm95IGFsbCBjb3BpZXMgb2YgdGhlIG9yaWdpbmFsIG1lc3NhZ2Uu
DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0K

Re: Issue found with kernel/net/sunrpc/xdr.c

[Myklebust, Trond]

T24gV2VkLCAyMDEzLTA4LTI4IGF0IDExOjUzIC0wNzAwLCBNYXR0IENyYWlnaGVhZCB3cm90ZToN
Cj4gVHJvbmQsIEkndmUganVzdCBub3cgY29tcGxldGVkIHRlc3Rpbmcgb2YgeW91ciBwYXRjaCAt
LSBpdCBkZWZpbml0ZWx5IGZpeGVzIHRoZSBjb3JydXB0aW9uIGlzc3VlLg0KPiANCj4gQWxzbywg
eW91ciBwYXRjaCBpcyBjbGVhbmVyIGFuZCBwcm9iYWJseSBtb3JlIGVmZmljaWVudCB0aGFuIG91
cnMuDQo+IA0KPiBMZXQgbWUgYW5kIE1hcmsga25vdyBpZiB5b3UgbmVlZCBhbnl0aGluZyBlbHNl
IGZyb20gdXMgb24gdGhpcyBpc3N1ZSAtLSB0aGFua3MgZm9yIHlvdXIgaGVscCENCg0KVGhhbmtz
IE1BdHQhIERvIHlvdSBtaW5kIGlmIEkgYWRkIGEgJ1Rlc3RlZC1ieTogTWF0dCBDcmFpZ2hlYWQn
IGxpbmU/DQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWlu
ZXINCg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20N
Cg==

RE: Issue found with kernel/net/sunrpc/xdr.c

[Matt Craighead]

U3VyZSwgZ28gYWhlYWQuICBUaGFua3MhDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpG
cm9tOiBNeWtsZWJ1c3QsIFRyb25kIFttYWlsdG86VHJvbmQuTXlrbGVidXN0QG5ldGFwcC5jb21d
IA0KU2VudDogV2VkbmVzZGF5LCBBdWd1c3QgMjgsIDIwMTMgMjozOSBQTQ0KVG86IE1hdHQgQ3Jh
aWdoZWFkDQpDYzogSi4gQnJ1Y2UgRmllbGRzOyBNYXJrIFlvdW5nOyBsaW51eC1uZnNAdmdlci5r
ZXJuZWwub3JnDQpTdWJqZWN0OiBSZTogSXNzdWUgZm91bmQgd2l0aCBrZXJuZWwvbmV0L3N1bnJw
Yy94ZHIuYw0KDQpPbiBXZWQsIDIwMTMtMDgtMjggYXQgMTE6NTMgLTA3MDAsIE1hdHQgQ3JhaWdo
ZWFkIHdyb3RlOg0KPiBUcm9uZCwgSSd2ZSBqdXN0IG5vdyBjb21wbGV0ZWQgdGVzdGluZyBvZiB5
b3VyIHBhdGNoIC0tIGl0IGRlZmluaXRlbHkgZml4ZXMgdGhlIGNvcnJ1cHRpb24gaXNzdWUuDQo+
IA0KPiBBbHNvLCB5b3VyIHBhdGNoIGlzIGNsZWFuZXIgYW5kIHByb2JhYmx5IG1vcmUgZWZmaWNp
ZW50IHRoYW4gb3Vycy4NCj4gDQo+IExldCBtZSBhbmQgTWFyayBrbm93IGlmIHlvdSBuZWVkIGFu
eXRoaW5nIGVsc2UgZnJvbSB1cyBvbiB0aGlzIGlzc3VlIC0tIHRoYW5rcyBmb3IgeW91ciBoZWxw
IQ0KDQpUaGFua3MgTUF0dCEgRG8geW91IG1pbmQgaWYgSSBhZGQgYSAnVGVzdGVkLWJ5OiBNYXR0
IENyYWlnaGVhZCcgbGluZT8NCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGll
bnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29tDQp3d3cu
bmV0YXBwLmNvbQ0KDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KVGhpcyBlbWFpbCBtZXNz
YWdlIGlzIGZvciB0aGUgc29sZSB1c2Ugb2YgdGhlIGludGVuZGVkIHJlY2lwaWVudChzKSBhbmQg
bWF5IGNvbnRhaW4NCmNvbmZpZGVudGlhbCBpbmZvcm1hdGlvbi4gIEFueSB1bmF1dGhvcml6ZWQg
cmV2aWV3LCB1c2UsIGRpc2Nsb3N1cmUgb3IgZGlzdHJpYnV0aW9uDQppcyBwcm9oaWJpdGVkLiAg
SWYgeW91IGFyZSBub3QgdGhlIGludGVuZGVkIHJlY2lwaWVudCwgcGxlYXNlIGNvbnRhY3QgdGhl
IHNlbmRlciBieQ0KcmVwbHkgZW1haWwgYW5kIGRlc3Ryb3kgYWxsIGNvcGllcyBvZiB0aGUgb3Jp
Z2luYWwgbWVzc2FnZS4NCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo=