XFS issues with NVMe SSD

XFS issues with NVMe SSD

[Bond Masuda]

Hi linux-xfs list:

I've been having XFS file system corruption issues when used with Intel 
NVMe SSD in Fedora 25 Linux as reported here:

https://bugzilla.redhat.com/show_bug.cgi?id=1402533

I don't know if this is XFS issue or not, but ext4 on the same Intel 
NVMe SSD doesn't appear to be a problem. Are linux/xfs devs already 
aware of the problem?

Bond

Re: XFS issues with NVMe SSD

[Eric Sandeen]

On 1/21/17 7:20 PM, Bond Masuda wrote:
> Hi linux-xfs list:
> 
> I've been having XFS file system corruption issues when used with
> Intel NVMe SSD in Fedora 25 Linux as reported here:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1402533
> 
> I don't know if this is XFS issue or not, but ext4 on the same Intel
> NVMe SSD doesn't appear to be a problem. Are linux/xfs devs already
> aware of the problem?
> 
> Bond 

Well, the bug languished under anaconda, so no, we haven't seen it.

It's pretty light on details.  Please provide the install logs, dmesg output,
xfs_repair output etc.  I'll update the bug.

-Eric

Re: XFS issues with NVMe SSD

[Brian Foster]

On Mon, Jan 23, 2017 at 07:41:58AM -0600, Eric Sandeen wrote:
> On 1/21/17 7:20 PM, Bond Masuda wrote:
> > Hi linux-xfs list:
> > 
> > I've been having XFS file system corruption issues when used with
> > Intel NVMe SSD in Fedora 25 Linux as reported here:
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1402533
> > 
> > I don't know if this is XFS issue or not, but ext4 on the same Intel
> > NVMe SSD doesn't appear to be a problem. Are linux/xfs devs already
> > aware of the problem?
> > 
> > Bond 
> 
> Well, the bug languished under anaconda, so no, we haven't seen it.
> 
> It's pretty light on details.  Please provide the install logs, dmesg output,
> xfs_repair output etc.  I'll update the bug.
> 

Since this is limited hardware (I don't have access to such hardware, at
least), it might also be helpful to see if you can isolate the problem
from a full distro install. For example, can you format such a device
external to the rootfs, mount, copy some stuff and reproduce the
corruption after some sequence of unmount/remount/reboot operations?

>From the bug, it does seem like a reboot is a consistent factor to
reproduce the corruption. I wonder whether it's enough to have the
device mounted at reboot time, or if being a rootfs device is a primary
factor...

Brian

> -Eric
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: XFS issues with NVMe SSD

[Christoph Hellwig]

On Mon, Jan 23, 2017 at 08:50:48AM -0500, Brian Foster wrote:
> Since this is limited hardware (I don't have access to such hardware, at
> least), it might also be helpful to see if you can isolate the problem
> from a full distro install. For example, can you format such a device
> external to the rootfs, mount, copy some stuff and reproduce the
> corruption after some sequence of unmount/remount/reboot operations?

I have plenty of NVMe hardware, although none of them is Intel and I've
never seen corruption like that.  I also don't run Fedora, though :)

Does anyone know if the Fedora installer does a fstrim run or something
similar?  I remember that most Intel NVMe devices had some pretty severe
deallocate (aka discard on NVMe) bugs.  A firmware update might be a
good start in that case.

Re: XFS issues with NVMe SSD

[Chris Murphy]

On Mon, Jan 23, 2017 at 6:53 AM, Christoph Hellwig <hch@infradead.org> wrote:
> On Mon, Jan 23, 2017 at 08:50:48AM -0500, Brian Foster wrote:
>> Since this is limited hardware (I don't have access to such hardware, at
>> least), it might also be helpful to see if you can isolate the problem
>> from a full distro install. For example, can you format such a device
>> external to the rootfs, mount, copy some stuff and reproduce the
>> corruption after some sequence of unmount/remount/reboot operations?
>
> I have plenty of NVMe hardware, although none of them is Intel and I've
> never seen corruption like that.  I also don't run Fedora, though :)
>
> Does anyone know if the Fedora installer does a fstrim run or something
> similar?  I remember that most Intel NVMe devices had some pretty severe
> deallocate (aka discard on NVMe) bugs.  A firmware update might be a
> good start in that case.

The installer doesn't do fstrim or blkdiscard before installation; but
on my SSDs at least mkfs.xfs and mkfs.btrfs result in trim. So one
possible test would be to manually mkfs with -k, and then if it's a
live media installation, install manually from the booted live
environment with:

rsync -pogAXtlHrDx --exclude /dev/ --exclude /proc/ --exclude /sys/
--exclude /run/ --exclude /boot/*rescue* --exclude /etc/machine-id
/run/install/source/ /mnt/sysimage




-- 
Chris Murphy

Re: XFS issues with NVMe SSD

[Eric Sandeen]

On 1/23/17 3:25 PM, Chris Murphy wrote:
> On Mon, Jan 23, 2017 at 6:53 AM, Christoph Hellwig <hch@infradead.org> wrote:
>> On Mon, Jan 23, 2017 at 08:50:48AM -0500, Brian Foster wrote:
>>> Since this is limited hardware (I don't have access to such hardware, at
>>> least), it might also be helpful to see if you can isolate the problem
>>> from a full distro install. For example, can you format such a device
>>> external to the rootfs, mount, copy some stuff and reproduce the
>>> corruption after some sequence of unmount/remount/reboot operations?
>>
>> I have plenty of NVMe hardware, although none of them is Intel and I've
>> never seen corruption like that.  I also don't run Fedora, though :)
>>
>> Does anyone know if the Fedora installer does a fstrim run or something
>> similar?  I remember that most Intel NVMe devices had some pretty severe
>> deallocate (aka discard on NVMe) bugs.  A firmware update might be a
>> good start in that case.
> 
> The installer doesn't do fstrim or blkdiscard before installation; but
> on my SSDs at least mkfs.xfs and mkfs.btrfs result in trim. So one
> possible test would be to manually mkfs with -k, and then if it's a
> live media installation, install manually from the booted live
> environment with:
> 
> rsync -pogAXtlHrDx --exclude /dev/ --exclude /proc/ --exclude /sys/
> --exclude /run/ --exclude /boot/*rescue* --exclude /etc/machine-id
> /run/install/source/ /mnt/sysimage

Right, mkfs does do trim...

Could also pre-mkfs, and tell anaconda to re-use the partition without
re-mkfs - is that possible?

-Eric

Re: XFS issues with NVMe SSD

[Chris Murphy]

On Mon, Jan 23, 2017 at 2:27 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
>
>
> On 1/23/17 3:25 PM, Chris Murphy wrote:
>> On Mon, Jan 23, 2017 at 6:53 AM, Christoph Hellwig <hch@infradead.org> wrote:
>>> On Mon, Jan 23, 2017 at 08:50:48AM -0500, Brian Foster wrote:
>>>> Since this is limited hardware (I don't have access to such hardware, at
>>>> least), it might also be helpful to see if you can isolate the problem
>>>> from a full distro install. For example, can you format such a device
>>>> external to the rootfs, mount, copy some stuff and reproduce the
>>>> corruption after some sequence of unmount/remount/reboot operations?
>>>
>>> I have plenty of NVMe hardware, although none of them is Intel and I've
>>> never seen corruption like that.  I also don't run Fedora, though :)
>>>
>>> Does anyone know if the Fedora installer does a fstrim run or something
>>> similar?  I remember that most Intel NVMe devices had some pretty severe
>>> deallocate (aka discard on NVMe) bugs.  A firmware update might be a
>>> good start in that case.
>>
>> The installer doesn't do fstrim or blkdiscard before installation; but
>> on my SSDs at least mkfs.xfs and mkfs.btrfs result in trim. So one
>> possible test would be to manually mkfs with -k, and then if it's a
>> live media installation, install manually from the booted live
>> environment with:
>>
>> rsync -pogAXtlHrDx --exclude /dev/ --exclude /proc/ --exclude /sys/
>> --exclude /run/ --exclude /boot/*rescue* --exclude /etc/machine-id
>> /run/install/source/ /mnt/sysimage
>
> Right, mkfs does do trim...
>
> Could also pre-mkfs, and tell anaconda to re-use the partition without
> re-mkfs - is that possible?

Anaconda requires reformat of any partition used for rootfs, except
Btrfs where it requires a new subvolume for rootfs.

I've got one of these with Fedora 25 on it, variably using kernels
4.5.5 through 4.10-rc4 and haven't had any corruption issues with
either XFS or Btrfs
Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802]





-- 
Chris Murphy

Re: XFS issues with NVMe SSD

[Bond Masuda]

On 01/23/2017 01:33 PM, Chris Murphy wrote:
>
> Anaconda requires reformat of any partition used for rootfs, except
> Btrfs where it requires a new subvolume for rootfs.
>
> I've got one of these with Fedora 25 on it, variably using kernels
> 4.5.5 through 4.10-rc4 and haven't had any corruption issues with
> either XFS or Btrfs
> Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802]
>
There are only 2 people I know of that have run into this issue, 
including myself. Maybe the issue is hardware specific? If so, the SSD 
model I had this problem with is Intel SSDPEKKW512G7X1. I'm not sure 
what the "other" person who reported this problem has, other than it too 
was an Intel SSD.

The system I had this issue with got shipped to Japan 2 weeks ago and I 
no longer have possession of it. Ended up not putting the OS install on 
the Intel NVMe SSD, but on a SATA SSD. I may have another system 
similarly configured soon - when I get a chance, I'll try to reproduce 
the problem again and try to collect more information.

Bond

Re: XFS issues with NVMe SSD

[Gabriel VLASIU]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Mon, 23 Jan 2017, Bond Masuda wrote:

> There are only 2 people I know of that have run into this issue, including
> myself.
Well, no. I have the same problem about a week ago.

Intel 6000p (INTEL SSDPEKKF256G7).
RHEL 7 (actually CentOS 7), kernel 3.10.0-514.

I had the root partition on raid 5 for many, many years. I decided to 
switch to nvme ssd for root fs. I copied the content of / partition on 
ssd, made the required changes to fstab and grub, put selinux in 
permissive mode. Reboot. Everything was fine. Touch /.autorelabel, reboot 
again. Everything worked up to relabel point during the boot. Errors. 
Instructed-me to run fsck. Reboot with the old root fs. xfs.fsck was not 
been able to do anything on the ssd partition. In the end, I formatted the 
nvme partition with ext4 and seems to works.

Maybe a selinux problem with xfs again? I don't know.


Sincerely,
Gabriel

- -- 

// Gabriel VLASIU
//
// OpenGPG-KeyID      : 44952F15
// OpenGPG-Fingerprint: 4AC5 7C26 2FE9 02DA 4906  24B2 D32B 7ED7 4495 2F15
// OpenGPG-URL        : http://www.vlasiu.net/public.key

-----BEGIN PGP SIGNATURE-----

iQIbBAEBCgAGBQJYhwzJAAoJENMrftdElS8VEkQP90a6bW8keyLc31ctNFJ/wgcC
DsXuw9SbG9zI62I5TvJVBDU5wVstZX/C70edbrNPDOqRJSNRCbDorJ3AuPhZ6wBm
PfjNXu/Dz4j1CRQBt0cbQ4bGP0U1FgTlMbt5QqNcAwSlXOZ44QIe7uGMbKoiW84S
pfqr1x+FolMPRUMWrNAI0+Yml0gOMmKg7fNpdEkRnB4NfAveD8zaKh2wrDCkglDn
EaoixjQpkj6mVLmCzkMRs6afb90ycEcZCN0+BB/rC2TOQ+YKAWgqppDhf9enB4CF
7rFaUNrjZ4M22S8sA60D/SvnF76XS9hvABlmn81p378qQiiVsG+uW+a7vkYQjz4I
WL0HLWhJdy5HPCgQ4eo36TpTzlMOsMkHaV6aJLH+biRFQo/mxIBgqvr4Fd3oEQMT
+28DqTbAWCZDIG7o8d+tKuU72Cse3rUa2QbhSN5YwzVi4RZOSP56NJvVmXBPq5+6
vplvOPSF7zDIon7ik7J4IHjfg/5dy6JKevx3QekO7vW/0Aeh7GLI3qfzzxuAW8Ww
MjELXSrcFnhFRG/2YxZavlZCtXjYxr6AuaxQT22OWfz9fvulPZWrLPA2G/pDpNzW
njYg5MR2dGaHtq5kg23t2KjpCI/2FxCD/nd3/loeZLXzVrZKeWSRPO8ckxD083+K
cmSHlv8gdk/e6fCr+54=
=dzNp
-----END PGP SIGNATURE-----