NFS data corruption on congested network

NFS data corruption on congested network

[Jacek Tomaka]

Hello,
I ran into an issue where the NFS file ends up being corrupted on disk. We started noticing it on certain, quite old hardware after upgrading OS from Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1.

After some investigation we have reasons to believe that the change was introduced by the following commit: 
https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd85805947402d91

We write a number of files on a single thread. Each file is up to 4GB. Before closing we call fdatasync. Sometimes the file ends up being corrupted. The corruptions is in a form of a number ( more than 3k pages in one case) of zero filled pages.
When this happens the file cannot be deleted from the client machine which created the file, even when the process which wrote the file completed successfully.

The machines have about 128GB of memory, i think and probably network that leaves to be desired.

My reproducer is currently tied up to our internal software, but i suspect setting the write_congested flag randomly should allow to reproduce the issue.

Regards.
Jacek Tomaka

Re: NFS data corruption on congested network

[NeilBrown]

On Fri, 23 Feb 2024, Jacek Tomaka wrote:
> Hello,
> I ran into an issue where the NFS file ends up being corrupted on disk. We started noticing it on certain, quite old hardware after upgrading OS from Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1.
> 
> After some investigation we have reasons to believe that the change was introduced by the following commit: 
> https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd85805947402d91

Thanks for the report.
Can you try a change to your kernel?

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index bb79d3a886ae..08a787147bd2 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -668,8 +668,10 @@ static int nfs_writepage_locked(struct folio *folio,
 	int err;
 
 	if (wbc->sync_mode == WB_SYNC_NONE &&
-	    NFS_SERVER(inode)->write_congested)
+	    NFS_SERVER(inode)->write_congested) {
+		folio_redirty_for_writepage(wbc, folio);
 		return AOP_WRITEPAGE_ACTIVATE;
+	}
 
 	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
 	nfs_pageio_init_write(&pgio, inode, 0, false,


though if your kernel is older than 6.3, that will be
         redirty_for_writepage(wbc, page);

Thanks,
NeilBrown


> 
> We write a number of files on a single thread. Each file is up to 4GB. Before closing we call fdatasync. Sometimes the file ends up being corrupted. The corruptions is in a form of a number ( more than 3k pages in one case) of zero filled pages.
> When this happens the file cannot be deleted from the client machine which created the file, even when the process which wrote the file completed successfully.
> 
> The machines have about 128GB of memory, i think and probably network that leaves to be desired.
> 
> My reproducer is currently tied up to our internal software, but i suspect setting the write_congested flag randomly should allow to reproduce the issue.
> 
> Regards.
> Jacek Tomaka
> 

Re: NFS data corruption on congested network

[NeilBrown]

On Mon, 26 Feb 2024, NeilBrown wrote:
> On Fri, 23 Feb 2024, Jacek Tomaka wrote:
> > Hello,
> > I ran into an issue where the NFS file ends up being corrupted on disk. We started noticing it on certain, quite old hardware after upgrading OS from Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1.
> > 
> > After some investigation we have reasons to believe that the change was introduced by the following commit: 
> > https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd85805947402d91
> 
> Thanks for the report.
> Can you try a change to your kernel?
> 
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index bb79d3a886ae..08a787147bd2 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -668,8 +668,10 @@ static int nfs_writepage_locked(struct folio *folio,
>  	int err;
>  
>  	if (wbc->sync_mode == WB_SYNC_NONE &&
> -	    NFS_SERVER(inode)->write_congested)
> +	    NFS_SERVER(inode)->write_congested) {
> +		folio_redirty_for_writepage(wbc, folio);
>  		return AOP_WRITEPAGE_ACTIVATE;
> +	}
>  
>  	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
>  	nfs_pageio_init_write(&pgio, inode, 0, false,

Actually this is only needed before linux 6.8 as only nfs_writepage()
can call nfs_writepage_locked() with sync_mode of WB_SYNC_NONE.
So v5.18 through v6.7 might need fixing.

NeilBrown


> 
> 
> though if your kernel is older than 6.3, that will be
>          redirty_for_writepage(wbc, page);
> 
> Thanks,
> NeilBrown
> 
> 
> > 
> > We write a number of files on a single thread. Each file is up to 4GB. Before closing we call fdatasync. Sometimes the file ends up being corrupted. The corruptions is in a form of a number ( more than 3k pages in one case) of zero filled pages.
> > When this happens the file cannot be deleted from the client machine which created the file, even when the process which wrote the file completed successfully.
> > 
> > The machines have about 128GB of memory, i think and probably network that leaves to be desired.
> > 
> > My reproducer is currently tied up to our internal software, but i suspect setting the write_congested flag randomly should allow to reproduce the issue.
> > 
> > Regards.
> > Jacek Tomaka
> > 
> 
> 
> 

Re: NFS data corruption on congested network

[Cedric Blancher]

On Mon, 26 Feb 2024 at 00:19, NeilBrown <neilb@suse.de> wrote:
>
> On Mon, 26 Feb 2024, NeilBrown wrote:
> > On Fri, 23 Feb 2024, Jacek Tomaka wrote:
> > > Hello,
> > > I ran into an issue where the NFS file ends up being corrupted on disk. We started noticing it on certain, quite old hardware after upgrading OS from Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1.
> > >
> > > After some investigation we have reasons to believe that the change was introduced by the following commit:
> > > https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd85805947402d91
> >
> > Thanks for the report.
> > Can you try a change to your kernel?
> >
> > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > index bb79d3a886ae..08a787147bd2 100644
> > --- a/fs/nfs/write.c
> > +++ b/fs/nfs/write.c
> > @@ -668,8 +668,10 @@ static int nfs_writepage_locked(struct folio *folio,
> >       int err;
> >
> >       if (wbc->sync_mode == WB_SYNC_NONE &&
> > -         NFS_SERVER(inode)->write_congested)
> > +         NFS_SERVER(inode)->write_congested) {
> > +             folio_redirty_for_writepage(wbc, folio);
> >               return AOP_WRITEPAGE_ACTIVATE;
> > +     }
> >
> >       nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
> >       nfs_pageio_init_write(&pgio, inode, 0, false,
>
> Actually this is only needed before linux 6.8 as only nfs_writepage()
> can call nfs_writepage_locked() with sync_mode of WB_SYNC_NONE.
> So v5.18 through v6.7 might need fixing.

Please do not forget the Linux 6.6-stable branch!!

Ced
-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: NFS data corruption on congested network

[Jacek Tomaka]

Hi NeilBrown, 

> though if your kernel is older than 6.3, that will be
>          redirty_for_writepage(wbc, page);

Things are looking good. I have ran it on 15 machines for good couple of hours and i do not see the problem. Usually i would see it after 1-3 iterations but now they are reaching 20 iterations without the problem.

Thank you for the fix.
Regards.
Jacek Tomaka

Temat: Re: NFS data corruption on congested network
Data: 2024-02-26 0:19
Nadawca: "NeilBrown" <neilb@suse.de>
Adresat: "Jacek Tomaka" <Jacek.Tomaka@poczta.fm>; 
DW: trond.myklebust@hammerspace.com; anna.schumaker@netapp.com; linux-nfs@vger.kernel.org; 

> 
>> On Mon, 26 Feb 2024, NeilBrown wrote:
>> On Fri, 23 Feb 2024, Jacek Tomaka wrote:
>>> Hello,
>>> I ran into an issue where the NFS file ends up being corrupted on
disk. We started noticing it on certain, quite old hardware after upgrading
OS from Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1.
>>> 
>>> After some investigation we have reasons to believe that the
change was introduced by the following commit: 
>>>
https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd85805947402d91
>> 
>> Thanks for the report.
>> Can you try a change to your kernel?
>> 
>> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
>> index bb79d3a886ae..08a787147bd2 100644
>> --- a/fs/nfs/write.c
>> +++ b/fs/nfs/write.c
>> @@ -668,8 +668,10 @@ static int nfs_writepage_locked(struct folio
*folio,
>>  	int err;
>>  
>>  	if (wbc->sync_mode == WB_SYNC_NONE &&
>> -	    NFS_SERVER(inode)->write_congested)
>> +	    NFS_SERVER(inode)->write_congested) {
>> +		folio_redirty_for_writepage(wbc, folio);
>>  		return AOP_WRITEPAGE_ACTIVATE;
>> +	}
>>  
>>  	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
>>  	nfs_pageio_init_write(&pgio, inode, 0, false,
> 
> Actually this is only needed before linux 6.8 as only nfs_writepage()
> can call nfs_writepage_locked() with sync_mode of WB_SYNC_NONE.
> So v5.18 through v6.7 might need fixing.
> 
> NeilBrown
> 
> 
>> 
>> 
>> though if your kernel is older than 6.3, that will be
>>          redirty_for_writepage(wbc, page);
>> 
>> Thanks,
>> NeilBrown
>> 
>> 
>>> 
>>> We write a number of files on a single thread. Each file is up to
4GB. Before closing we call fdatasync. Sometimes the file ends up being
corrupted. The corruptions is in a form of a number ( more than 3k pages in
one case) of zero filled pages.
>>> When this happens the file cannot be deleted from the client
machine which created the file, even when the process which wrote the file
completed successfully.
>>> 
>>> The machines have about 128GB of memory, i think and probably
network that leaves to be desired.
>>> 
>>> My reproducer is currently tied up to our internal software, but i
suspect setting the write_congested flag randomly should allow to reproduce
the issue.
>>> 
>>> Regards.
>>> Jacek Tomaka
>>> 
>> 
>> 
>> 
> 
> 
> 

Re: NFS data corruption on congested network

[NeilBrown]

On Mon, 26 Feb 2024, Jacek Tomaka wrote:
> Hi NeilBrown, 
> 
> > though if your kernel is older than 6.3, that will be
> >          redirty_for_writepage(wbc, page);
> 
> Things are looking good. I have ran it on 15 machines for good couple of hours and i do not see the problem. Usually i would see it after 1-3 iterations but now they are reaching 20 iterations without the problem.
> 
> Thank you for the fix.

Thanks for testing!  I'll get the fix submitted.

NeilBrown


> Regards.
> Jacek Tomaka
> 
> Temat: Re: NFS data corruption on congested network
> Data: 2024-02-26 0:19
> Nadawca: "NeilBrown" <neilb@suse.de>
> Adresat: "Jacek Tomaka" <Jacek.Tomaka@poczta.fm>; 
> DW: trond.myklebust@hammerspace.com; anna.schumaker@netapp.com; linux-nfs@vger.kernel.org; 
> 
> > 
> >> On Mon, 26 Feb 2024, NeilBrown wrote:
> >> On Fri, 23 Feb 2024, Jacek Tomaka wrote:
> >>> Hello,
> >>> I ran into an issue where the NFS file ends up being corrupted on
> disk. We started noticing it on certain, quite old hardware after upgrading
> OS from Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1.
> >>> 
> >>> After some investigation we have reasons to believe that the
> change was introduced by the following commit: 
> >>>
> https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd85805947402d91
> >> 
> >> Thanks for the report.
> >> Can you try a change to your kernel?
> >> 
> >> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> >> index bb79d3a886ae..08a787147bd2 100644
> >> --- a/fs/nfs/write.c
> >> +++ b/fs/nfs/write.c
> >> @@ -668,8 +668,10 @@ static int nfs_writepage_locked(struct folio
> *folio,
> >>  	int err;
> >>  
> >>  	if (wbc->sync_mode == WB_SYNC_NONE &&
> >> -	    NFS_SERVER(inode)->write_congested)
> >> +	    NFS_SERVER(inode)->write_congested) {
> >> +		folio_redirty_for_writepage(wbc, folio);
> >>  		return AOP_WRITEPAGE_ACTIVATE;
> >> +	}
> >>  
> >>  	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
> >>  	nfs_pageio_init_write(&pgio, inode, 0, false,
> > 
> > Actually this is only needed before linux 6.8 as only nfs_writepage()
> > can call nfs_writepage_locked() with sync_mode of WB_SYNC_NONE.
> > So v5.18 through v6.7 might need fixing.
> > 
> > NeilBrown
> > 
> > 
> >> 
> >> 
> >> though if your kernel is older than 6.3, that will be
> >>          redirty_for_writepage(wbc, page);
> >> 
> >> Thanks,
> >> NeilBrown
> >> 
> >> 
> >>> 
> >>> We write a number of files on a single thread. Each file is up to
> 4GB. Before closing we call fdatasync. Sometimes the file ends up being
> corrupted. The corruptions is in a form of a number ( more than 3k pages in
> one case) of zero filled pages.
> >>> When this happens the file cannot be deleted from the client
> machine which created the file, even when the process which wrote the file
> completed successfully.
> >>> 
> >>> The machines have about 128GB of memory, i think and probably
> network that leaves to be desired.
> >>> 
> >>> My reproducer is currently tied up to our internal software, but i
> suspect setting the write_congested flag randomly should allow to reproduce
> the issue.
> >>> 
> >>> Regards.
> >>> Jacek Tomaka
> >>> 
> >> 
> >> 
> >> 
> > 
> > 
> > 
>